| dmsimard|cave | I don't have time to look right now but just came across a 500 error on ask.o.o https://ask.openstack.org/ | 00:01 |
|---|---|---|
| *** jamesmcarthur has joined #openstack-infra | 00:02 | |
| pabelanger | prometheanfire: see issue on test, left -1 | 00:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove final from publish-zuul-website https://review.openstack.org/556681 | 00:03 |
| *** yamamoto has joined #openstack-infra | 00:03 | |
| pabelanger | [6495418.728834] init: askbot-celeryd main process ended, respawning | 00:03 |
| pabelanger | think that is an issue | 00:03 |
| pabelanger | oh | 00:04 |
| pabelanger | -bash: cannot create temp file for here-document: No space left on device | 00:04 |
| pabelanger | that is likely the bigger issue | 00:04 |
| tonyb | ianw: The leagacy jobs failed on that change as it's still looking at tarballs.o.o http://logs.openstack.org/77/554977/2/check/legacy-devstack-dsvm-updown/f12f464/logs/devstacklog.txt.gz#_2018-03-26_23_56_22_485 | 00:05 |
| tonyb | ianw: at least I think that's what's going on | 00:05 |
| *** jamesmcarthur has quit IRC | 00:07 | |
| *** yamamoto has quit IRC | 00:07 | |
| *** gouthamr has joined #openstack-infra | 00:07 | |
| *** chkumar246 has joined #openstack-infra | 00:08 | |
| prometheanfire | pabelanger: I guess that does count as a version, I guess | 00:09 |
| prometheanfire | pabelanger: out of space on my job? | 00:09 |
| ianw | tonyb: yeah, similar i think we can drop the tarballs redirect in d-g i guess. my memory is that we tried using that first, but we had enough issues between remote clouds and tarballs.o.o that we (I) re-implemented the caching bits | 00:09 |
| pabelanger | prometheanfire: no, this is ask.o.o | 00:10 |
| pabelanger | it looks like database and back ups are taking up a large amount of space | 00:10 |
| tonyb | ianw: Ahh okay it's in d-g my grep fu is weak today | 00:10 |
| pabelanger | and apache / jetty logging | 00:10 |
| *** rlandy is now known as rlandy|bbl | 00:10 | |
| ianw | pabelanger: that points to growroot not working ... similar to the other day. probably never tested on gentoo? | 00:10 |
| pabelanger | ianw: no, this is ask.o.o | 00:10 |
| *** dave-mccowan has quit IRC | 00:10 | |
| pabelanger | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2547&rra_id=all | 00:11 |
| ianw | oh haha, sorry i thought you'd been watching the gentoo job fail, should read better! | 00:11 |
| prometheanfire | pabelanger: I'll go ahead and add the 17.0 and fix the variable though | 00:11 |
| prometheanfire | growroot works fine on gentoo :P | 00:11 |
| *** odyssey4me has quit IRC | 00:11 | |
| *** odyssey4me has joined #openstack-infra | 00:11 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add git.zuul-ci.org cgit site https://review.openstack.org/555969 | 00:16 |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: add gentoo systemd image for testing https://review.openstack.org/556661 | 00:17 |
| prometheanfire | pabelanger: mind rebasing? | 00:18 |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: DNM: testing gentoo nodepool-dsvm https://review.openstack.org/556679 | 00:18 |
| prometheanfire | nvm, think I figured how how to do that in gertty | 00:19 |
| *** rcernin has quit IRC | 00:19 | |
| corvus | infra-root: i've verified the cgit changes on git08. i will leave it out of the load balancer for now in case more folks want to test. but i believe it's safe to put back in at this point (and will continue to serve git.o.o correctly). it should also be safe to merge the changes implementing the new behavior (change https://review.openstack.org/555969 ) | 00:19 |
| *** rcernin has joined #openstack-infra | 00:19 | |
| corvus | infra-root: if you want to test it yourself, you can add git08 to your hosts file like this: | 00:19 |
| corvus | 104.239.146.131 git.zuul-ci.org | 00:19 |
| corvus | 104.239.146.131 git.openstack.org | 00:19 |
| corvus | infra-root: be sure to use ports 8080 and 4443, since it's not going through the lb | 00:20 |
| pabelanger | ianw: dmsimard|cave: ask.o.o is back online, but we need to do some maintenance on that host. We are basically out of HDD space because size of database, backups and http logs (apache / jetty). Can't tonight, but we should figure out how to grow server or reduce logs / backups | 00:23 |
| *** salv-orlando has joined #openstack-infra | 00:24 | |
| pabelanger | corvus: wfm | 00:25 |
| pabelanger | I am able to clone over HTTPS | 00:25 |
| *** zhurong has joined #openstack-infra | 00:27 | |
| *** mguiney has joined #openstack-infra | 00:28 | |
| *** salv-orlando has quit IRC | 00:29 | |
| *** HeOS has quit IRC | 00:30 | |
| *** spiffxp has joined #openstack-infra | 00:31 | |
| *** gouthamr has quit IRC | 00:31 | |
| *** vipul has joined #openstack-infra | 00:34 | |
| *** gcb has joined #openstack-infra | 00:36 | |
| *** gcb has quit IRC | 00:36 | |
| *** caphrim007 has joined #openstack-infra | 00:37 | |
| *** efoley has quit IRC | 00:39 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/devstack-gate master: Remove setting of ETCD_DOWNLOAD_URL https://review.openstack.org/556688 | 00:39 |
| *** felipemonteiro has quit IRC | 00:39 | |
| *** spiffxp has quit IRC | 00:39 | |
| *** vipul has quit IRC | 00:40 | |
| *** caphrim007 has quit IRC | 00:41 | |
| *** vipul has joined #openstack-infra | 00:42 | |
| *** spiffxp has joined #openstack-infra | 00:44 | |
| *** hongbin has joined #openstack-infra | 00:51 | |
| *** harlowja has quit IRC | 00:53 | |
| *** mriedem has quit IRC | 00:55 | |
| *** lihi has quit IRC | 00:56 | |
| *** oanson has quit IRC | 00:58 | |
| *** gouthamr has joined #openstack-infra | 00:59 | |
| *** oanson has joined #openstack-infra | 01:04 | |
| *** lihi has joined #openstack-infra | 01:04 | |
| *** tobiash has quit IRC | 01:05 | |
| *** camunoz has quit IRC | 01:08 | |
| *** annp has quit IRC | 01:10 | |
| *** annp has joined #openstack-infra | 01:11 | |
| *** felipemonteiro has joined #openstack-infra | 01:19 | |
| *** salv-orlando has joined #openstack-infra | 01:25 | |
| *** gyankum has joined #openstack-infra | 01:27 | |
| *** salv-orlando has quit IRC | 01:29 | |
| *** cshastri has joined #openstack-infra | 01:31 | |
| openstackgerrit | zhongshengping proposed openstack-infra/project-config master: Remove puppet-ganesha from puppet openstack modules https://review.openstack.org/556696 | 01:37 |
| *** rkukura has quit IRC | 01:37 | |
| ianw | pabelanger: ok ... i feel like jetty logs has been a persistent problem | 01:39 |
| *** bhujay has joined #openstack-infra | 01:49 | |
| *** germs has joined #openstack-infra | 01:52 | |
| *** germs has quit IRC | 01:52 | |
| *** germs has joined #openstack-infra | 01:52 | |
| *** liujiong has joined #openstack-infra | 01:54 | |
| *** germs has quit IRC | 01:57 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: add gentoo systemd image for testing https://review.openstack.org/556661 | 01:58 |
| *** zhurong has quit IRC | 01:58 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: DNM: testing gentoo nodepool-dsvm https://review.openstack.org/556679 | 01:59 |
| prometheanfire | pabelanger: ok, submitted (I think/hope) | 01:59 |
| *** bobh has quit IRC | 02:10 | |
| *** dave-mccowan has joined #openstack-infra | 02:14 | |
| *** namnh has joined #openstack-infra | 02:16 | |
| *** dhajare has joined #openstack-infra | 02:20 | |
| *** felipemonteiro has quit IRC | 02:20 | |
| *** salv-orlando has joined #openstack-infra | 02:25 | |
| *** gyee has quit IRC | 02:26 | |
| *** yamamoto has joined #openstack-infra | 02:26 | |
| ianw | dmsimard|cave: hmm, this ran in limestone and wasn't too happy | 02:27 |
| ianw | http://logs.openstack.org/77/554977/3/check/neutron-grenade/34fbe09/ | 02:27 |
| ianw | i think http://logs.openstack.org/77/554977/3/check/neutron-grenade/34fbe09/job-output.txt.gz#_2018-03-27_00_48_52_995358 is the root cause, but we don't fail gracefully | 02:28 |
| dmsimard|cave | /home/zuul/workspace/devstack-gate/functions.sh: line 184: /home/zuul/workspace/logs/reproduce.sh: No such file or directory | 02:28 |
| dmsimard|cave | chmod: cannot access '/home/zuul/workspace/logs/reproduce.sh': No such file or directory | 02:28 |
| dmsimard|cave | ? | 02:28 |
| ianw | Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /pypi/simple/pbr/ | 02:28 |
| ianw | i think it all starts here where it doesn't get ansible installed | 02:28 |
| *** dmsimard|cave is now known as dmsimard | 02:29 | |
| dmsimard | So an issue with the pypi mirror ? | 02:30 |
| *** salv-orlando has quit IRC | 02:30 | |
| ianw | dmsimard: it all seems to be in place ... http://mirror.regionone.limestone.openstack.org/pypi/simple/pbr/ | 02:33 |
| ianw | either mirror setup, or connectivity to | 02:33 |
| dmsimard | ianw: I just logged on to a random limestone node: http://paste.openstack.org/raw/715298/ | 02:33 |
| *** s-shiono has joined #openstack-infra | 02:33 | |
| *** dhajare has quit IRC | 02:33 | |
| *** VW has joined #openstack-infra | 02:33 | |
| *** daidv has joined #openstack-infra | 02:34 | |
| ianw | dmsimard: was that v4 or v6? | 02:34 |
| dmsimard | ianw: it seems to try v6 by default http://paste.openstack.org/raw/715299/ | 02:35 |
| *** ykarel|away has joined #openstack-infra | 02:35 | |
| dmsimard | but both ping and ping6 work | 02:35 |
| dmsimard | getting late here and I had a long day... if there's a consistent problem I'd suggest we take it out until we can figure it out | 02:40 |
| ianw | dmsimard: a preliminary logstash searching shows a couple of hits, but not persistent | 02:45 |
| ianw | http://logs.openstack.org/81/325381/28/check/nova-multiattach/d2ca11b/job-output.txt was perhaps the last one | 02:48 |
| *** bhujay has quit IRC | 02:50 | |
| *** zhangfei has joined #openstack-infra | 02:51 | |
| *** zhangfei has quit IRC | 02:51 | |
| *** zhangfei has joined #openstack-infra | 02:52 | |
| *** gouthamr has quit IRC | 02:53 | |
| *** ramishra has joined #openstack-infra | 02:53 | |
| pabelanger | dmsimard: ianw: it looks like we might not be setting up ipv6 properly for DNS: http://logs.openstack.org/77/554977/3/check/neutron-grenade/34fbe09/ara/result/dbdb18b6-151d-4efc-ab63-15e1bf886c82/ | 02:56 |
| *** caphrim007 has joined #openstack-infra | 02:56 | |
| pabelanger | ianw: dmsimard: so wondering we we are doing nat from v6tov4 for requests | 02:57 |
| pabelanger | we had this issue with osic1 at the start too, IIRC | 02:57 |
| *** VW has quit IRC | 02:59 | |
| ianw | pabelanger: ahh, that being unreliable? | 03:01 |
| pabelanger | ianw: well, added pressure to network requests because of NATTing. We can confirm with logan- too | 03:02 |
| pabelanger | but looking at configure-unbound, I'd expect ipv6 to work, the task | 03:03 |
| ianw | yeah, i'll have to refresh my mind on how we do ipv6 detetction | 03:04 |
| pabelanger | ack, I'm heading out for the night | 03:04 |
| ianw | ansible_default_ipv6 | 03:05 |
| *** caphrim007_ has joined #openstack-infra | 03:15 | |
| *** ramishra has quit IRC | 03:16 | |
| *** ramishra has joined #openstack-infra | 03:18 | |
| *** caphrim007 has quit IRC | 03:19 | |
| *** rlandy|bbl is now known as rlandy | 03:19 | |
| *** rlandy has quit IRC | 03:19 | |
| *** mguiney has quit IRC | 03:20 | |
| *** valkyrie1 has joined #openstack-infra | 03:20 | |
| *** valkyrie1 is now known as mguiney | 03:20 | |
| *** zhurong has joined #openstack-infra | 03:24 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Use hostvars to get ansible_default_ipv6 https://review.openstack.org/556740 | 03:24 |
| *** salv-orlando has joined #openstack-infra | 03:26 | |
| *** salv-orlando has quit IRC | 03:31 | |
| *** eernst has joined #openstack-infra | 03:32 | |
| ianw | pabelanger / dmsimard : ^ i think i've had this wrong the whole time? seems to detect ipv6 on rax now : http://logs.openstack.org/40/556740/1/check/openstack-infra-base-integration-ubuntu-trusty/b37ad2c/job-output.txt.gz#_2018-03-27_03_29_12_786553 | 03:37 |
| *** andreas_s has joined #openstack-infra | 03:42 | |
| *** sree has joined #openstack-infra | 03:45 | |
| *** zhangfei has quit IRC | 03:46 | |
| *** andreas_s has quit IRC | 03:46 | |
| prometheanfire | pabelanger: looks like a timeout on that gentoo test build | 03:47 |
| *** tobiash has joined #openstack-infra | 03:49 | |
| *** dave-mccowan has quit IRC | 03:49 | |
| *** germs has joined #openstack-infra | 03:53 | |
| *** germs has quit IRC | 03:53 | |
| *** germs has joined #openstack-infra | 03:53 | |
| *** ykarel|away is now known as ykarel | 03:54 | |
| *** udesale has joined #openstack-infra | 03:55 | |
| *** germs has quit IRC | 03:58 | |
| *** bhavik1 has joined #openstack-infra | 03:59 | |
| *** hongbin has quit IRC | 03:59 | |
| *** bhujay has joined #openstack-infra | 04:01 | |
| *** zhangfei has joined #openstack-infra | 04:03 | |
| *** ykarel has quit IRC | 04:04 | |
| *** zhangfei has quit IRC | 04:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Don't assume nodepool.[private|public]_ipv4 https://review.openstack.org/556747 | 04:16 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Don't assume nodepool.[private|public]_ipv4 https://review.openstack.org/556747 | 04:16 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Use hostvars to get ansible_default_ipv6 https://review.openstack.org/556740 | 04:16 |
| openstackgerrit | megan guiney proposed openstack-infra/subunit2sql master: Add delete by uuid functions https://review.openstack.org/537775 | 04:16 |
| *** zhangfei has joined #openstack-infra | 04:22 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: add gentoo systemd image for testing https://review.openstack.org/556661 | 04:23 |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: DNM: testing gentoo nodepool-dsvm https://review.openstack.org/556679 | 04:24 |
| *** yamahata has joined #openstack-infra | 04:24 | |
| *** eernst has quit IRC | 04:24 | |
| *** annp has quit IRC | 04:26 | |
| *** salv-orlando has joined #openstack-infra | 04:27 | |
| *** markvoelker has quit IRC | 04:27 | |
| *** psachin has joined #openstack-infra | 04:27 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Change the GENTOO_PORTAGE_CLEANUP variable default https://review.openstack.org/556749 | 04:28 |
| *** salv-orlando has quit IRC | 04:32 | |
| *** dhajare has joined #openstack-infra | 04:36 | |
| *** zhurong has quit IRC | 04:39 | |
| *** agopi|out has quit IRC | 04:45 | |
| *** agopi|out has joined #openstack-infra | 04:46 | |
| *** ykarel has joined #openstack-infra | 04:46 | |
| *** janki has joined #openstack-infra | 04:48 | |
| *** bhavik1 has quit IRC | 04:57 | |
| *** agopi|out has quit IRC | 05:00 | |
| *** yamamoto_ has joined #openstack-infra | 05:07 | |
| *** yamamoto has quit IRC | 05:11 | |
| *** salv-orlando has joined #openstack-infra | 05:15 | |
| *** markvoelker has joined #openstack-infra | 05:28 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Don't assume nodepool.[private|public]_ipv4 https://review.openstack.org/556747 | 05:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Use hostvars to get ansible_default_ipv6 https://review.openstack.org/556740 | 05:33 |
| *** dhajare has quit IRC | 05:41 | |
| *** dsariel has joined #openstack-infra | 05:50 | |
| *** s-shiono_ has joined #openstack-infra | 05:50 | |
| *** s-shiono has quit IRC | 05:53 | |
| *** germs has joined #openstack-infra | 05:54 | |
| *** germs has quit IRC | 05:54 | |
| *** germs has joined #openstack-infra | 05:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Silence bashate https://review.openstack.org/556366 | 05:55 |
| *** germs has quit IRC | 05:59 | |
| *** sree has quit IRC | 05:59 | |
| *** sree has joined #openstack-infra | 05:59 | |
| *** yamamoto has joined #openstack-infra | 06:01 | |
| prometheanfire | nn | 06:03 |
| *** yamamoto_ has quit IRC | 06:04 | |
| bkero | 06:04 | |
| *** sree has quit IRC | 06:04 | |
| *** aeng has quit IRC | 06:05 | |
| *** armaan has joined #openstack-infra | 06:05 | |
| *** annp has joined #openstack-infra | 06:12 | |
| *** psachin has quit IRC | 06:14 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-osc-placement-dsvm-functional https://review.openstack.org/547814 | 06:15 |
| *** sree has joined #openstack-infra | 06:15 | |
| *** namnh_ has joined #openstack-infra | 06:17 | |
| *** pgadiya has joined #openstack-infra | 06:19 | |
| *** pblaho has joined #openstack-infra | 06:19 | |
| *** sree has quit IRC | 06:19 | |
| *** namnh has quit IRC | 06:20 | |
| *** armaan has quit IRC | 06:20 | |
| *** namnh has joined #openstack-infra | 06:21 | |
| *** namnh_ has quit IRC | 06:21 | |
| *** namnh_ has joined #openstack-infra | 06:22 | |
| *** psachin has joined #openstack-infra | 06:23 | |
| *** namnh has quit IRC | 06:26 | |
| *** auristor1 has joined #openstack-infra | 06:28 | |
| *** dhajare has joined #openstack-infra | 06:28 | |
| *** auristor has quit IRC | 06:28 | |
| *** dbecker has quit IRC | 06:31 | |
| *** armaan has joined #openstack-infra | 06:38 | |
| *** dbecker has joined #openstack-infra | 06:46 | |
| *** zoli is now known as zoli|trng | 06:47 | |
| *** zoli|trng is now known as zoli | 06:48 | |
| *** pcaruana has joined #openstack-infra | 06:53 | |
| *** jaosorior has quit IRC | 07:02 | |
| *** danpawlik has joined #openstack-infra | 07:03 | |
| *** danpawlik has quit IRC | 07:05 | |
| *** dsariel has quit IRC | 07:05 | |
| *** tesseract has joined #openstack-infra | 07:15 | |
| *** jtomasek has joined #openstack-infra | 07:16 | |
| *** yamamoto_ has joined #openstack-infra | 07:22 | |
| *** yamamoto has quit IRC | 07:25 | |
| *** rcernin has quit IRC | 07:25 | |
| *** dsariel has joined #openstack-infra | 07:28 | |
| *** gfidente has joined #openstack-infra | 07:29 | |
| *** andreas_s has joined #openstack-infra | 07:33 | |
| *** Qiming_ has joined #openstack-infra | 07:34 | |
| *** Qiming has quit IRC | 07:35 | |
| *** salv-orlando has quit IRC | 07:36 | |
| *** salv-orlando has joined #openstack-infra | 07:37 | |
| openstackgerrit | zhongshengping proposed openstack-infra/project-config master: Retire puppet-ganesha https://review.openstack.org/556696 | 07:38 |
| *** salv-orlando has quit IRC | 07:42 | |
| *** gongysh has joined #openstack-infra | 07:43 | |
| *** pblaho has quit IRC | 07:44 | |
| *** psachin has quit IRC | 07:46 | |
| *** dims_ has joined #openstack-infra | 07:48 | |
| *** ykarel is now known as ykarel|lunch | 07:49 | |
| *** sree_ has joined #openstack-infra | 07:49 | |
| *** dims has quit IRC | 07:49 | |
| *** sree_ has quit IRC | 07:49 | |
| *** sree has joined #openstack-infra | 07:49 | |
| *** xinliang has joined #openstack-infra | 07:51 | |
| *** xinliang has quit IRC | 07:51 | |
| *** xinliang has joined #openstack-infra | 07:51 | |
| *** bmace has quit IRC | 07:51 | |
| *** bmace has joined #openstack-infra | 07:51 | |
| *** alexchadin has joined #openstack-infra | 07:51 | |
| *** alexchadin has quit IRC | 07:52 | |
| *** alexchadin has joined #openstack-infra | 07:53 | |
| *** ralonsoh has joined #openstack-infra | 07:53 | |
| *** jaosorior has joined #openstack-infra | 07:53 | |
| *** zhurong has joined #openstack-infra | 07:53 | |
| *** tosky has joined #openstack-infra | 07:54 | |
| *** germs has joined #openstack-infra | 07:54 | |
| *** germs has quit IRC | 07:54 | |
| *** germs has joined #openstack-infra | 07:54 | |
| *** numans has quit IRC | 07:57 | |
| *** germs has quit IRC | 07:59 | |
| *** HeOS has joined #openstack-infra | 07:59 | |
| *** numans has joined #openstack-infra | 08:00 | |
| *** lucas-afk is now known as lucasagomes | 08:01 | |
| *** salv-orlando has joined #openstack-infra | 08:04 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Set trusted flag for configure-mirrors test https://review.openstack.org/556776 | 08:04 |
| *** jpich has joined #openstack-infra | 08:05 | |
| frickler | ianw: pabelanger: ^^ this should fix the test failures. not 100% sure whether to just set it globally or whether we should special-case bionic only | 08:06 |
| *** derekh has joined #openstack-infra | 08:09 | |
| *** e0ne has joined #openstack-infra | 08:09 | |
| *** bhujay has quit IRC | 08:13 | |
| *** HeOS has quit IRC | 08:19 | |
| *** alexchad_ has joined #openstack-infra | 08:20 | |
| *** alexchadin has quit IRC | 08:20 | |
| *** kong has quit IRC | 08:21 | |
| openstackgerrit | zhongshengping proposed openstack-infra/project-config master: Retire puppet-ganesha https://review.openstack.org/556696 | 08:22 |
| *** ykarel|lunch is now known as ykarel | 08:30 | |
| *** pgadiya has quit IRC | 08:37 | |
| *** pgadiya has joined #openstack-infra | 08:38 | |
| *** alexchad_ has quit IRC | 08:38 | |
| *** pblaho has joined #openstack-infra | 08:39 | |
| *** alexchadin has joined #openstack-infra | 08:39 | |
| *** gyankum has quit IRC | 08:41 | |
| *** gyan_ has joined #openstack-infra | 08:41 | |
| *** gyan__ has joined #openstack-infra | 08:44 | |
| *** yamahata has quit IRC | 08:44 | |
| *** cshastri has quit IRC | 08:44 | |
| *** gyan_ has quit IRC | 08:47 | |
| *** racciari has joined #openstack-infra | 08:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Handle blank public_ipv4 https://review.openstack.org/556784 | 08:54 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Don't assume nodepool.[private|public]_ipv4 https://review.openstack.org/556747 | 08:54 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Use hostvars to get ansible_default_ipv6 https://review.openstack.org/556740 | 08:54 |
| ianw | pabelanger / dmsimard / clarkb(as the switch guru) : ^ has exposed that a blank public_ipv4 needs some workarounds in our various setup; along with the ipv6 detection bug. i have to go now, feel free to update as required else i'll work on it my time tomorrow ... | 08:55 |
| openstackgerrit | zhongshengping proposed openstack-infra/project-config master: Retire puppet-ganesha(2/2) https://review.openstack.org/556696 | 08:59 |
| openstackgerrit | zhongshengping proposed openstack-infra/project-config master: Retire puppet-ganesha(1/2) https://review.openstack.org/556789 | 08:59 |
| *** zhurong has quit IRC | 09:00 | |
| *** HeOS has joined #openstack-infra | 09:02 | |
| *** jogo has quit IRC | 09:03 | |
| ianw | frickler: yeah, your change failed on something mine did to at one point | 09:05 |
| ianw | secondary | ERROR: Unable to start service openvswitch: A dependency job for openvswitch.service failed. See 'journalctl -xe' for details. | 09:05 |
| *** e0ne has quit IRC | 09:05 | |
| *** e0ne has joined #openstack-infra | 09:05 | |
| openstackgerrit | zhongshengping proposed openstack-infra/project-config master: Retire puppet-ganesha(1/2) https://review.openstack.org/556789 | 09:08 |
| openstackgerrit | zhongshengping proposed openstack-infra/project-config master: Retire puppet-ganesha(1/2) https://review.openstack.org/556789 | 09:11 |
| *** racciari has quit IRC | 09:11 | |
| openstackgerrit | zhongshengping proposed openstack-infra/project-config master: Retire puppet-ganesha(1/2) https://review.openstack.org/556789 | 09:13 |
| *** hashar has joined #openstack-infra | 09:17 | |
| *** bhujay has joined #openstack-infra | 09:18 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Don't assume nodepool.[private|public]_ipv4 https://review.openstack.org/556747 | 09:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Use hostvars to get ansible_default_ipv6 https://review.openstack.org/556740 | 09:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Make tumbleweed integration test non-voting https://review.openstack.org/556792 | 09:18 |
| ianw | frickler / clarkb: ^ from what i can tell, this is an actual problem with the tumbleweed packaging. clarkb; maybe you can file a bug ... i'm not that sure how to do that effectively for suse | 09:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Make tumbleweed integration test non-voting https://review.openstack.org/556792 | 09:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Don't assume nodepool.[private|public]_ipv4 https://review.openstack.org/556747 | 09:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Use hostvars to get ansible_default_ipv6 https://review.openstack.org/556740 | 09:22 |
| frickler | ianw: iiuc dirk had a patch for that in devstack, but I'll have to check whether that is really the same issue | 09:24 |
| frickler | hmm, unclear, see https://bugzilla.suse.com/show_bug.cgi?id=1085971 and https://review.openstack.org/#/c/554907/2/lib/neutron_plugins/ovs_base | 09:26 |
| openstack | bugzilla.suse.com bug 1085971 in Network "openvswitch: Permission denied when starting the ovsdb-server service" [Normal,In_progress] - Assigned to markos.chandras | 09:26 |
| frickler | ianw: oh, you have the journal in https://review.openstack.org/556792 , that shows that it indeed is the same issue | 09:27 |
| *** s-shiono_ has quit IRC | 09:33 | |
| ianw | frickler: oh, thanks for finding that, not sure how i missed that | 09:34 |
| frickler | ianw: well, the title of the patch isn't very informative. I just happened to look at that issue with dirk last week and held a node for debugging | 09:35 |
| AJaeger | hwoarang: backscroll might be interesting for you ^ | 09:35 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Make tumbleweed integration test non-voting https://review.openstack.org/556792 | 09:36 |
| *** jogo has joined #openstack-infra | 09:39 | |
| *** jogo has joined #openstack-infra | 09:39 | |
| frickler | infra-root: more issues with limestone due to failing name resolution, I'm thinking to disable them again until we have a proper solution for this http://logs.openstack.org/88/365388/4/gate/neutron-grenade/4b89ce6/logs/grenade.sh.txt.gz#_2018-03-27_06_01_02_479 | 09:42 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Revert "Bump max-servers limestone nodepool provider to 10" https://review.openstack.org/556804 | 09:44 |
| frickler | ianw: if you have one final click before you fall asleep ;) ^^ | 09:45 |
| hwoarang | AJaeger: i am taking care of that bug | 09:47 |
| hwoarang | need to see what upstream think before backporting | 09:47 |
| *** germs has joined #openstack-infra | 09:55 | |
| *** germs has quit IRC | 10:00 | |
| *** kong has joined #openstack-infra | 10:01 | |
| *** liujiong has quit IRC | 10:03 | |
| *** ykarel_ has joined #openstack-infra | 10:03 | |
| *** dhajare_ has joined #openstack-infra | 10:03 | |
| *** gyan_ has joined #openstack-infra | 10:03 | |
| *** chandankumar has joined #openstack-infra | 10:03 | |
| *** udesale_ has joined #openstack-infra | 10:03 | |
| dirk | frickler: can we merge the workaround patch for now? | 10:04 |
| *** chkumar246 has quit IRC | 10:04 | |
| *** ykarel has quit IRC | 10:04 | |
| *** gyan__ has quit IRC | 10:05 | |
| *** udesale__ has joined #openstack-infra | 10:05 | |
| *** ykarel__ has joined #openstack-infra | 10:05 | |
| *** gyan__ has joined #openstack-infra | 10:05 | |
| dirk | https://review.openstack.org/554907 is the link for that.. | 10:06 |
| frickler | dirk: which patch, https://review.openstack.org/554907 ? if you give it a proper commit message, that should be fine. won't help for the zuul-jobs checks, though | 10:06 |
| dirk | frickler: why does it not help for zuul-jobs? it doesn't use the devstack code? | 10:06 |
| *** dhajare has quit IRC | 10:07 | |
| *** udesale has quit IRC | 10:07 | |
| *** udesale_ has quit IRC | 10:08 | |
| *** gyan_ has quit IRC | 10:08 | |
| dirk | frickler: commit message updated. lets see if the tw job is more happy now | 10:08 |
| *** ykarel_ has quit IRC | 10:08 | |
| *** dhajare_ has quit IRC | 10:08 | |
| *** chandankumar has quit IRC | 10:08 | |
| frickler | dirk: no, they just exercise the roles directly | 10:08 |
| frickler | dirk: see e.g. http://logs.openstack.org/76/556776/1/check/openstack-infra-multinode-integration-opensuse-tumbleweed/83c2b95/ara/result/61d12153-0c71-4fbd-9918-bf1418585302/ | 10:08 |
| *** yamamoto_ has quit IRC | 10:10 | |
| *** gongysh has quit IRC | 10:11 | |
| *** ykarel__ is now known as ykarel | 10:11 | |
| *** derekh is now known as derekh_afk | 10:13 | |
| dmsimard | frickler: +3 | 10:14 |
| dmsimard | I'll re-enable it later on its own custom label so we can isolate tests | 10:16 |
| frickler | dmsimard: good plan, thx | 10:17 |
| *** namnh_ has quit IRC | 10:18 | |
| *** chandankumar has joined #openstack-infra | 10:20 | |
| *** dhajare_ has joined #openstack-infra | 10:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Bump max-servers limestone nodepool provider to 10" https://review.openstack.org/556804 | 10:23 |
| *** tosky__ has joined #openstack-infra | 10:25 | |
| *** tosky is now known as Guest72548 | 10:25 | |
| *** tosky__ is now known as tosky | 10:25 | |
| *** gongysh has joined #openstack-infra | 10:28 | |
| *** zhurong has joined #openstack-infra | 10:31 | |
| *** gyan__ has quit IRC | 10:34 | |
| *** gyan__ has joined #openstack-infra | 10:36 | |
| *** vivsoni has quit IRC | 10:37 | |
| *** vivsoni has joined #openstack-infra | 10:38 | |
| *** e0ne has quit IRC | 10:43 | |
| *** e0ne has joined #openstack-infra | 10:43 | |
| *** pbourke has quit IRC | 10:44 | |
| *** pbourke has joined #openstack-infra | 10:45 | |
| dmsimard | frickler: btw if you have a chance, some easy reviews: https://review.openstack.org/#/q/topic:add-ara-projects :p | 10:45 |
| *** udesale__ has quit IRC | 10:51 | |
| *** yamamoto has joined #openstack-infra | 10:52 | |
| *** zhangfei has quit IRC | 10:53 | |
| *** e0ne has quit IRC | 10:53 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Explicitly read setup.cfg as utf-8 on Python 3 https://review.openstack.org/555366 | 10:54 |
| *** zhipeng has joined #openstack-infra | 10:56 | |
| kashyap | Remind me again: putting a "commit: 1232132" in the Git commit message will make Gerrit link to the commit? | 10:58 |
| kashyap | Or do I _have_ to use Change ID? (Which I know does link..) | 10:58 |
| *** janki has quit IRC | 11:06 | |
| *** fabo has quit IRC | 11:07 | |
| *** zhipeng has quit IRC | 11:08 | |
| *** lucasagomes is now known as lucas-hungry | 11:08 | |
| *** alexchadin has quit IRC | 11:09 | |
| *** e0ne has joined #openstack-infra | 11:09 | |
| frickler | dmsimard: I didn't follow too closely, but there were some issues regarding new project creation. are these resolved now or do will still have to take care of something there? | 11:16 |
| *** Keitaro has quit IRC | 11:17 | |
| AJaeger | frickler: should be resolved | 11:19 |
| AJaeger | frickler: feel free to approve new project creations | 11:20 |
| *** vivsoni_ has joined #openstack-infra | 11:24 | |
| *** vivsoni has quit IRC | 11:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create the ara-server project https://review.openstack.org/556524 | 11:29 |
| *** yamamoto has quit IRC | 11:30 | |
| *** sshnaidm has quit IRC | 11:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create the ara-clients project https://review.openstack.org/556525 | 11:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Create the ara-plugins project https://review.openstack.org/556526 | 11:31 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Fix bashate "local hides exit status" warnings https://review.openstack.org/365388 | 11:31 |
| *** tpsilva has joined #openstack-infra | 11:33 | |
| *** openstackgerrit has quit IRC | 11:33 | |
| *** yamamoto has joined #openstack-infra | 11:34 | |
| *** dtantsur|afk is now known as dtantsur | 11:34 | |
| *** gyan__ has quit IRC | 11:34 | |
| *** openstackgerrit has joined #openstack-infra | 11:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add fixtures-git project https://review.openstack.org/553978 | 11:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Add ansible-role-k8s-rabbitmq https://review.openstack.org/552672 | 11:34 |
| *** rosmaita has joined #openstack-infra | 11:34 | |
| *** yamamoto has quit IRC | 11:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: kolla-cli https://review.openstack.org/555897 | 11:40 |
| *** ldnunes has joined #openstack-infra | 11:40 | |
| *** sree has quit IRC | 11:41 | |
| *** adarazs is now known as adarazs_lunch | 11:46 | |
| *** udesale has joined #openstack-infra | 11:47 | |
| openstackgerrit | Merged openstack-infra/project-config master: add python-cyborgclient to openstack repo https://review.openstack.org/555707 | 11:47 |
| *** jlabarre has joined #openstack-infra | 11:51 | |
| *** yamamoto has joined #openstack-infra | 11:54 | |
| *** sree has joined #openstack-infra | 11:54 | |
| *** gongysh has quit IRC | 11:54 | |
| *** germs has joined #openstack-infra | 11:56 | |
| *** germs has quit IRC | 11:56 | |
| *** germs has joined #openstack-infra | 11:56 | |
| *** sree has quit IRC | 11:58 | |
| *** pgadiya has quit IRC | 12:00 | |
| *** germs has quit IRC | 12:01 | |
| *** bobh has joined #openstack-infra | 12:03 | |
| *** rfolco|off is now known as rfolco|ruckk | 12:03 | |
| *** rfolco|ruckk is now known as rfolco|ruck | 12:03 | |
| *** openstackgerrit has quit IRC | 12:04 | |
| *** sree has joined #openstack-infra | 12:04 | |
| *** alexchadin has joined #openstack-infra | 12:08 | |
| *** janki has joined #openstack-infra | 12:08 | |
| *** sshnaidm has joined #openstack-infra | 12:09 | |
| *** sree has quit IRC | 12:09 | |
| *** bobh has quit IRC | 12:10 | |
| *** edmondsw has joined #openstack-infra | 12:13 | |
| *** dprince has joined #openstack-infra | 12:16 | |
| *** lucas-hungry is now known as lucasagomes | 12:16 | |
| *** armaan has quit IRC | 12:16 | |
| *** sree has joined #openstack-infra | 12:17 | |
| *** openstackgerrit has joined #openstack-infra | 12:27 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/zuul-jobs master: Handle blank public_ipv4 https://review.openstack.org/556784 | 12:27 |
| *** zhurong has quit IRC | 12:27 | |
| *** markvoelker has quit IRC | 12:28 | |
| *** markvoelker has joined #openstack-infra | 12:28 | |
| *** racciari has joined #openstack-infra | 12:36 | |
| *** rlandy has joined #openstack-infra | 12:36 | |
| *** dave-mccowan has joined #openstack-infra | 12:41 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Re-enable Limestone at 10 nodes with custom labels https://review.openstack.org/556841 | 12:42 |
| dmsimard | config-core, infra-root: ^ please fast track this | 12:42 |
| *** odyssey4me has quit IRC | 12:43 | |
| *** odyssey4me has joined #openstack-infra | 12:43 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/irc-meetings master: Remove ajo and ralonsoh from chairing neutron-qos meeting https://review.openstack.org/556842 | 12:43 |
| *** derekh_afk is now known as derekh | 12:48 | |
| *** Keitaro has joined #openstack-infra | 12:54 | |
| *** camunoz has joined #openstack-infra | 12:59 | |
| *** trown|outtypewww is now known as trown | 13:00 | |
| *** panda|off is now known as panda | 13:03 | |
| *** jcoufal has joined #openstack-infra | 13:03 | |
| *** sree has quit IRC | 13:04 | |
| *** bobh has joined #openstack-infra | 13:06 | |
| *** VW has joined #openstack-infra | 13:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Re-enable Limestone at 10 nodes with custom labels https://review.openstack.org/556841 | 13:11 |
| *** vtapia has joined #openstack-infra | 13:12 | |
| *** eharney has joined #openstack-infra | 13:12 | |
| *** Goneri has joined #openstack-infra | 13:17 | |
| *** mriedem has joined #openstack-infra | 13:18 | |
| *** Hal has joined #openstack-infra | 13:26 | |
| *** eernst has joined #openstack-infra | 13:27 | |
| *** Hal has quit IRC | 13:28 | |
| *** Hal has joined #openstack-infra | 13:28 | |
| *** mikal_ has joined #openstack-infra | 13:30 | |
| *** jaosorior has quit IRC | 13:33 | |
| *** mikal has quit IRC | 13:33 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Do not merge: Test limestone nodes https://review.openstack.org/556849 | 13:36 |
| *** gouthamr has joined #openstack-infra | 13:38 | |
| fungi | dmsimard: do you have any theories on the dns query timeouts? | 13:39 |
| fungi | ahh, custom node labels. so i guess that gets us some room to track it down | 13:40 |
| dmsimard | fungi: haven't had the opportunity to check yet | 13:40 |
| *** armaan has joined #openstack-infra | 13:40 | |
| dmsimard | fungi: yes, the plan is to use the custom labels to hunt it down easily without impacting other jobs | 13:40 |
| dmsimard | (and also test the fixes) | 13:40 |
| fungi | my money's on the layer-4 nat table for the gateway filling up with udp state tracking | 13:41 |
| fungi | though i thought when we encountered that in osic we took extra steps to atempt to prefer dns lookups via ipv6 | 13:42 |
| fungi | so maybe not (or maybe that solution broke/got reverted?) | 13:42 |
| *** dtruong2_ has joined #openstack-infra | 13:42 | |
| dmsimard | I believe I may have screwed up the patch to add custom labels | 13:42 |
| *** alexchadin has quit IRC | 13:42 | |
| dmsimard | jobs are queued so I checked nodepool list and it's stack tracing on "KeyError: 'limestone-centos-7'" | 13:43 |
| * dmsimard looks | 13:43 | |
| dmsimard | fungi: do you see anything wrong with https://review.openstack.org/#/c/556841/1/nodepool/nl02.openstack.org.yaml by chance ? | 13:44 |
| dmsimard | (Seems okay to me.. we're using the same *disk* images but with different labels) | 13:44 |
| fungi | what symptoms are you getting? | 13:44 |
| Shrews | dmsimard: fungi: you need those labels in the outer 'labels' section | 13:45 |
| dmsimard | Oh, the labels aren't defined outside the provider pool scope | 13:45 |
| dmsimard | Shrews: yeah, I just realized that | 13:45 |
| * dmsimard sends patch | 13:45 | |
| *** alexchadin has joined #openstack-infra | 13:45 | |
| fungi | ahh, yeah, the labels list at the top of the file | 13:45 |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| *** kgiusti has joined #openstack-infra | 13:46 | |
| dmsimard | Shrews: do you think it would be possible to kind of load the nodepool configuration in a job to test for things like that ? | 13:46 |
| dmsimard | like nginx has nginx -t to test that the configuration "compiles" | 13:47 |
| *** annp_ has joined #openstack-infra | 13:48 | |
| *** vivsoni_ has quit IRC | 13:48 | |
| *** vivsoni_ has joined #openstack-infra | 13:48 | |
| Shrews | config validation checks the format of the yaml, but doesn't test things like cross-references | 13:48 |
| Shrews | we'd need to build that into nodepool | 13:49 |
| dmsimard | checking the format of the yaml would be a start, I don't think we even do that today | 13:49 |
| dmsimard | at least not for nodepool | 13:49 |
| dmsimard | Shrews: I don't need to set a min-ready key right ? It defaults to zero and that's fine ? | 13:50 |
| dmsimard | (In this particular context of testing things) | 13:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add missing custom label declaration for limestone nodepool config https://review.openstack.org/556856 | 13:51 |
| dmsimard | Shrews, fungi ^ | 13:51 |
| Shrews | dmsimard: default is 0 | 13:52 |
| dmsimard | k, cool | 13:52 |
| fungi | dmsimard: is the plan to propose a job which uses one of those labels? otherwise with min-ready 0 i don't think we'll actually build any instances there | 13:53 |
| *** hongbin has joined #openstack-infra | 13:53 | |
| dmsimard | fungi: yes, ianw and frickler have been working on several patches ( https://review.openstack.org/#/q/topic:add-limestone+status:open ) | 13:53 |
| *** dtruong2_ has quit IRC | 13:54 | |
| fungi | okay, then lgtm | 13:54 |
| clarkb | ya we had extra stuff to dns over ipv6 if present but that may have been cleaned up when oaic went away | 13:54 |
| dmsimard | fungi: the idea is to rebase/depends-on those on top of https://review.openstack.org/#/c/556849/ | 13:54 |
| clarkb | was in our dib configs iirc and just ran a script on boot to figure that out in unbound? | 13:54 |
| *** germs has joined #openstack-infra | 13:57 | |
| dmsimard | the failures that I've looked at yesterday was only for pypi | 13:57 |
| dmsimard | rest seemed okay | 13:57 |
| dmsimard | maybe there's been failures elsewhere though | 13:58 |
| *** alexchadin has quit IRC | 13:58 | |
| dmsimard | it was getting late last night after a long day :) | 13:58 |
| *** esberglu has joined #openstack-infra | 13:58 | |
| *** jamielennox has quit IRC | 14:01 | |
| *** germs has quit IRC | 14:02 | |
| *** jamielennox has joined #openstack-infra | 14:02 | |
| *** r-daneel has joined #openstack-infra | 14:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add missing custom label declaration for limestone nodepool config https://review.openstack.org/556856 | 14:02 |
| clarkb | the dns setup would've been part of our nodepool ready script iirc | 14:02 |
| clarkb | because we cant know the ip situation until post boot | 14:03 |
| *** jaosorior has joined #openstack-infra | 14:03 | |
| clarkb | so maybe we lost it in the zuulv3 transition and we need a new pre task in base job | 14:03 |
| clarkb | but basically if ipv6 is routable set unbound to use ipv6 resolvers only | 14:03 |
| dmsimard | Yup. | 14:04 |
| *** ramishra has quit IRC | 14:05 | |
| clarkb | 17d470f60c587e48816d664f6bf7c5679744b679#n25 | 14:06 |
| clarkb | yay mobile copy paste | 14:06 |
| clarkb | thats the sha from when it was in place and the line number at least | 14:06 |
| clarkb | projecf confif at that sha1 in configure_mirror.sh | 14:07 |
| clarkb | so likely want to add that back in fir limestone | 14:07 |
| openstackgerrit | Merged openstack-infra/storyboard master: Finish fixing account duplication on import https://review.openstack.org/555957 | 14:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Switch implicit role not found message to debug https://review.openstack.org/556866 | 14:12 |
| *** mihalis68 has joined #openstack-infra | 14:12 | |
| corvus | infra-root: i'd like to land https://review.openstack.org/555969 and child today: support for git.zuul-ci.org. I've manually tested them on git08. that config is still in place if you want to update your local hosts file; would anyone else like to test it, or review it? | 14:14 |
| clarkb | corvus: this is jeepyb and projects.yaml update? If they work then I'm happy with them having reviewed earlier versions | 14:15 |
| fungi | thanks! i will carve out time to look in a few minutes | 14:15 |
| fungi | clarkb: puppet-cgit and system-config, you haven't seen these yet i don't think | 14:15 |
| corvus | right, the jeepyb change landed, this uses it. | 14:16 |
| fungi | these are the "now let's turn it on" bits | 14:16 |
| clarkb | ah ok probably should take a look real quick then | 14:16 |
| *** dayou has quit IRC | 14:23 | |
| clarkb | corvus: the only thing I'd worry about is the change in how variavles are looked up in the cgit vhost template. I understand why the change was made (because now we have multiple vhosts) but unsure if all those lookups will still work from thr apache module. But if it is teated and working then it must be fine | 14:25 |
| *** ykarel is now known as ykarel|away | 14:29 | |
| corvus | clarkb: yeah, it seems to, and we have a similar vhost template elsewhere | 14:30 |
| *** eharney has quit IRC | 14:31 | |
| *** florianf has joined #openstack-infra | 14:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 14:34 |
| *** kiennt26 has joined #openstack-infra | 14:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Pass whitelabel tenant to zuul::web https://review.openstack.org/556870 | 14:37 |
| *** Goneri has quit IRC | 14:38 | |
| *** dayou has joined #openstack-infra | 14:38 | |
| *** kong has quit IRC | 14:40 | |
| anteaya | clarkb, I see you are listed as absent for today's infra meeting, has anyone volunteered to chair the meeting in your stead? | 14:43 |
| *** lpetrut has joined #openstack-infra | 14:43 | |
| clarkb | anteaya: yes, fungi said he could chair | 14:43 |
| *** ldnunes has quit IRC | 14:43 | |
| *** ldnunes has joined #openstack-infra | 14:44 | |
| anteaya | awesome, thank you fungi | 14:44 |
| fungi | well, i said i'd chair if nobody else wanted to volunteer. i guess that means nobody else wanted to volunteer ;) | 14:44 |
| anteaya | :) | 14:44 |
| * fungi will make sure he's around in time for the meeting in that case | 14:44 | |
| *** eharney has joined #openstack-infra | 14:45 | |
| *** yamahata has joined #openstack-infra | 14:46 | |
| *** ykarel|away has quit IRC | 14:46 | |
| *** kiennt26 has quit IRC | 14:47 | |
| *** dhajare_ has quit IRC | 14:50 | |
| *** gegger has joined #openstack-infra | 14:50 | |
| *** Goneri has joined #openstack-infra | 14:51 | |
| *** felipemonteiro has joined #openstack-infra | 14:57 | |
| *** yamahata has quit IRC | 14:59 | |
| *** rossella_s has joined #openstack-infra | 15:01 | |
| *** hamzy has quit IRC | 15:02 | |
| *** rook is now known as rook-mtg | 15:03 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:04 | |
| *** hamzy has joined #openstack-infra | 15:06 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add a flag to disable no_log on upload-logs https://review.openstack.org/556877 | 15:06 |
| *** florianf has quit IRC | 15:08 | |
| *** bhujay has quit IRC | 15:09 | |
| *** pcaruana has quit IRC | 15:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Test change to upload-logs https://review.openstack.org/556881 | 15:10 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove ajo and ralonsoh from chairing neutron-qos meeting https://review.openstack.org/556842 | 15:10 |
| *** agopi|out has joined #openstack-infra | 15:12 | |
| *** jaosorior has quit IRC | 15:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add a flag to disable no_log on upload-logs https://review.openstack.org/556877 | 15:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: DNM: Test change to upload-logs https://review.openstack.org/556885 | 15:13 |
| *** germs has joined #openstack-infra | 15:13 | |
| *** germs has quit IRC | 15:13 | |
| *** germs has joined #openstack-infra | 15:13 | |
| *** ykarel|away has joined #openstack-infra | 15:15 | |
| *** hemna_ has joined #openstack-infra | 15:17 | |
| *** germs has quit IRC | 15:18 | |
| *** armaan has quit IRC | 15:19 | |
| *** eharney has quit IRC | 15:20 | |
| *** armaan has joined #openstack-infra | 15:20 | |
| *** sambetts|afk is now known as sambetts | 15:20 | |
| *** eharney has joined #openstack-infra | 15:20 | |
| *** ianychoi_ is now known as ianychoi | 15:23 | |
| *** caphrim007_ has quit IRC | 15:25 | |
| *** caphrim007 has joined #openstack-infra | 15:25 | |
| *** imacdonn has quit IRC | 15:26 | |
| *** imacdonn has joined #openstack-infra | 15:26 | |
| *** caphrim007_ has joined #openstack-infra | 15:26 | |
| *** caphrim007 has quit IRC | 15:30 | |
| *** ldnunes has quit IRC | 15:30 | |
| *** ldnunes has joined #openstack-infra | 15:31 | |
| *** rook-mtg is now known as rook | 15:33 | |
| *** immune2iocane has joined #openstack-infra | 15:38 | |
| kashyap | What's the time out for logging me out of wiki.openstack.org? | 15:40 |
| * kashyap just lost 20 minutes of edits as it logged me out mid-way. And didn't give me a way to get the changes back. :-( | 15:40 | |
| * kashyap should've saved it | 15:40 | |
| *** iyamahat has joined #openstack-infra | 15:41 | |
| *** fabo has joined #openstack-infra | 15:42 | |
| *** fabo has joined #openstack-infra | 15:42 | |
| openstackgerrit | Merged openstack-infra/puppet-cgit master: Support multiple git sites https://review.openstack.org/555968 | 15:43 |
| openstackgerrit | Merged openstack-infra/system-config master: Add git.zuul-ci.org cgit site https://review.openstack.org/555969 | 15:43 |
| *** gagehugo has quit IRC | 15:43 | |
| *** andreas_s has quit IRC | 15:44 | |
| *** gagehugo has joined #openstack-infra | 15:44 | |
| *** andreas_s has joined #openstack-infra | 15:44 | |
| *** e0ne has quit IRC | 15:45 | |
| *** armaan has quit IRC | 15:46 | |
| *** HeOS has quit IRC | 15:46 | |
| *** armaan has joined #openstack-infra | 15:46 | |
| *** eharney has quit IRC | 15:46 | |
| *** yamamoto has quit IRC | 15:47 | |
| *** yamamoto has joined #openstack-infra | 15:48 | |
| *** iyamahat has quit IRC | 15:50 | |
| corvus | kashyap: it's really short; if you have any idea how to change it i'm sure you would be awash in the beverage of your choice :( | 15:50 |
| corvus | infra-root: ^ cgit changes merged | 15:50 |
| pabelanger | yay | 15:50 |
| kashyap | corvus: I didn't realize it was so short. Heh, is it MediaWiki limitation? | 15:51 |
| kashyap | If so, I can check w/ some Wiki ninjas perhaps | 15:51 |
| corvus | kashyap: i want to say it's something strange about the openid plugin, but i'm really unsure | 15:51 |
| kashyap | Hmm, OpenID... | 15:51 |
| kashyap | Anyhow, thanks. Now I'm taking backups as I edit them. | 15:52 |
| kashyap | corvus: BTW, I haven't used `gertty` in a while (shame on me; even though I package it in Fedora). | 15:52 |
| kashyap | corvus: What was a show-stopper for me was frequent DB crashes; wonder if you've noticed anything lately | 15:52 |
| kashyap | Or "Works For Me"? | 15:52 |
| *** yamamoto has quit IRC | 15:53 | |
| fungi | kashyap: also we're currently on mw 1.26.3 which may have limited control over some of the session timeout options. i see recommendations for adjusting $wgCookieExpiration (which we don't currently override from its default) but no idea if that actually controls it | 15:53 |
| *** derekh has quit IRC | 15:53 | |
| fungi | er, nevermind, we're on 1.28.2 | 15:53 |
| *** derekh has joined #openstack-infra | 15:53 | |
| fungi | i misread | 15:53 |
| *** eharney has joined #openstack-infra | 15:53 | |
| corvus | kashyap: it's working very reliably for me, so if you can send me any tracebacks, i'd be very happy to fix them | 15:54 |
| *** rossella_s has quit IRC | 15:54 | |
| kashyap | fungi: I see. | 15:54 |
| kashyap | corvus: That's encouraging to hear; I'll do my homework before I post any tracebacks. | 15:55 |
| fungi | though apparently if $wgSessionsInObjectCache is set true (it is for us) then $wgObjectCacheSessionExpiry (which we also don't override from its default) may control it? | 15:55 |
| kashyap | That's one thing I dearly miss, participating in parallel with projects with mailing lists | 15:55 |
| kashyap | Ability to apply patches / review all from command-line in a reliable & offline (I can give in to this, at least) way. | 15:55 |
| kashyap | fungi: What is "mw"? | 15:56 |
| fungi | mediawiki | 15:56 |
| kashyap | Ah, noted | 15:56 |
| fungi | the wiki software powering wiki.openstack.org | 15:56 |
| fungi | developed by the wikimedia foundation, originally for running wikipedia | 15:56 |
| kashyap | Yeah, just didn't connect the small-cased acronym | 15:57 |
| * fungi is too lazy to find his shift key | 15:57 | |
| kashyap | No problem :-) | 15:57 |
| fungi | i'm sure it's under here somewhere | 15:57 |
| *** ralonsoh has quit IRC | 15:57 | |
| fungi | i dig it out when i need to send e-mail | 15:58 |
| *** andreas_s has quit IRC | 15:58 | |
| *** annp_ has quit IRC | 15:59 | |
| *** eharney has quit IRC | 15:59 | |
| fungi | kashyap: https://www.mediawiki.org/wiki/Manual:$wgObjectCacheSessionExpiry indicates that value defaults to an hour. do you think you were likely typing that long? | 16:00 |
| *** gyee has joined #openstack-infra | 16:00 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/glean master: Add networkd support https://review.openstack.org/548604 | 16:00 |
| *** germs has joined #openstack-infra | 16:01 | |
| *** germs has quit IRC | 16:01 | |
| *** germs has joined #openstack-infra | 16:01 | |
| kashyap | fungi: Hmm, that's a good question. I think I was editing the Wiki, then I noticed feedback on a patch, which was much more interesting :P | 16:01 |
| kashyap | And then I got back to Wiki. And then I got to keep the pieces. | 16:01 |
| fungi | https://www.mediawiki.org/wiki/Manual:$wgCookieExpiration looks much less likely to fix it, since the default there is 30 days | 16:01 |
| *** rossella_s has joined #openstack-infra | 16:02 | |
| *** yamamoto has joined #openstack-infra | 16:03 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/glean master: Add networkd support https://review.openstack.org/548604 | 16:04 |
| *** sree has joined #openstack-infra | 16:05 | |
| fungi | #status log set wgObjectCacheSessionExpiry to 86400 in /srv/mediawiki/Settings.php on the wiki to see if it effectively increases session duration to one day | 16:05 |
| openstackstatus | fungi: finished logging | 16:05 |
| fungi | kashyap: ^ | 16:05 |
| kashyap | Oh, nice | 16:05 |
| fungi | time will tell | 16:06 |
| kashyap | Thank you, very much! | 16:06 |
| kashyap | Heh | 16:06 |
| fungi | i've logged back into the wiki and will see if it complains when i go to edit the infra meeting agenda in a few hours | 16:07 |
| *** germs_ has joined #openstack-infra | 16:08 | |
| *** yamamoto has quit IRC | 16:08 | |
| prometheanfire | pabelanger: anything left to do for the glean/nodepool/project-config changes? | 16:09 |
| *** HeOS has joined #openstack-infra | 16:09 | |
| *** sree has quit IRC | 16:09 | |
| *** germs has quit IRC | 16:10 | |
| *** dave-mccowan has quit IRC | 16:10 | |
| *** germs_ has quit IRC | 16:11 | |
| *** germs has joined #openstack-infra | 16:12 | |
| *** germs has quit IRC | 16:12 | |
| *** germs has joined #openstack-infra | 16:12 | |
| *** andreas_s has joined #openstack-infra | 16:13 | |
| fungi | any mysqlites happen to know if simply altering tables in a database to convert to character set utf8mb4 is sufficient, or do i also need to alter database and set the character set there? do i need to change individual columns' character set too? | 16:13 |
| *** rkukura has joined #openstack-infra | 16:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Test change to upload-logs https://review.openstack.org/556881 | 16:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add a flag to disable no_log on upload-logs https://review.openstack.org/556877 | 16:17 |
| Shrews | fungi: table should be enough, iirc | 16:17 |
| corvus | pabelanger, mordred: can you review https://review.openstack.org/556881 asap? | 16:17 |
| fungi | Shrews: for whatever reason it wasn't on the storyboard server... still getting errors like this on valid 4-byte utf-8: http://paste.openstack.org/show/714793/ (trove instance running mysql 2.6) | 16:18 |
| *** yamamoto has joined #openstack-infra | 16:18 | |
| corvus | infra-root: it looks like the cgit changes have merged. i will return git08 to the lb config now | 16:18 |
| fungi | thanks corvus! | 16:19 |
| fungi | Shrews: i'm testing now after also doing alter database set character set utf8mb4 to see if that also turned out to be necessary | 16:19 |
| Shrews | fungi: https://dev.mysql.com/doc/refman/5.7/en/charset-conversion.html | 16:20 |
| fungi | Shrews: i suspected converting individual columns wasn't necessary if also converting their tables, but was finding conflicting (likely clueless) anecdotes on the internet about it | 16:20 |
| Shrews | fungi: last sentence is relevant | 16:20 |
| fungi | Shrews: yep, thanks!!! | 16:21 |
| corvus | #status log git08 removed from emergency file and added back to git loadbalancer | 16:21 |
| openstackstatus | corvus: finished logging | 16:21 |
| *** derekh has quit IRC | 16:21 | |
| *** germs has quit IRC | 16:22 | |
| fungi | Shrews: the one thing i'm not certain of is whether pymysql is deciding the character set of the connection based on the overall database character set, so testing that possibility now | 16:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 16:23 |
| *** yamamoto has quit IRC | 16:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zone-zuul-ci.org master: Add git.zuul-ci.org https://review.openstack.org/556964 | 16:23 |
| Shrews | fungi: there should be a per-connection charset for clients. don't know anything about pymysql | 16:23 |
| Shrews | so maybe that's tripping you up | 16:23 |
| fungi | yeah, still dying on the same exception | 16:24 |
| fungi | Shrews: for reference, this is after applying the migration at http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/db/migration/alembic_migrations/versions/062_convert_to_charset_utf8mb4.py | 16:24 |
| *** pcaruana has joined #openstack-infra | 16:25 | |
| logan- | o/ just reading the backscroll re: limestone fails from last night. going thru logstash i notice one or two of the failed jobs had a failed v4 traceroute in the zuul-info over the snat. fungi is probably on the right track thinking snat port starvation. once we a ready to start doing some jobs on there ping me if some debugging would be helpful at the neutron router, maybe some conntrack data would be helpful to confirm/narrow down snat | 16:25 |
| logan- | contention | 16:25 |
| Shrews | fungi: https://dev.mysql.com/doc/refman/5.6/en/charset-unicode-utf8mb4.html says max is 3-bytes per char | 16:26 |
| *** zoli is now known as zoli|gone | 16:26 | |
| fungi | logan-: yeah, i believe those traceroutes are default unix style (udp-based) and noticed them as well, which is part of what led me down that path | 16:26 |
| *** andreas_s has quit IRC | 16:26 | |
| *** zoli|gone is now known as zoli | 16:26 | |
| Shrews | fungi: so those 4-byte chars are rightly rejected then | 16:26 |
| fungi | Shrews: utf8mb4 should be 4 bytes per char? utf8 is 3 (in mysql parlance at least) | 16:27 |
| Shrews | fungi: oh wait, misread | 16:27 |
| fungi | that page seems to confirm | 16:27 |
| Shrews | yeah | 16:27 |
| *** derekh has joined #openstack-infra | 16:27 | |
| *** felipemonteiro has quit IRC | 16:28 | |
| Shrews | i've always hated dealing w/ charsets | 16:28 |
| Shrews | gimme ascii or gimme death | 16:28 |
| Shrews | :) | 16:28 |
| fungi | here's what mysql says about encoding for those columns right now too: http://paste.openstack.org/show/715428/ | 16:29 |
| logan- | fungi: ya depending on how things look post-unbound fixes I suppose it may be necessary to apply some sysctl changes to the router container to adjust flow timeouts and possibly widen the snat port ranges (not 100% on what the defaults are but i think it may be 32k-64k, so easy to double capacity there if thats the case) | 16:29 |
| *** HeOS has quit IRC | 16:29 | |
| *** bobh has quit IRC | 16:29 | |
| *** germs has joined #openstack-infra | 16:29 | |
| *** germs has quit IRC | 16:29 | |
| *** germs has joined #openstack-infra | 16:29 | |
| fungi | logan-: thanks, i agree with that prognosis | 16:29 |
| * fungi disappears to eat some lunch quickly | 16:30 | |
| logan- | cloudnull: ^ would be interested to hear if you had to apply any conntrack changes on the osic routers to avoid v4 snat port contention | 16:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorganize REST API and dashboard urls https://review.openstack.org/556967 | 16:31 |
| *** germs has quit IRC | 16:32 | |
| *** lucasagomes is now known as lucas-afk | 16:33 | |
| *** yamamoto has joined #openstack-infra | 16:33 | |
| *** andreas_s has joined #openstack-infra | 16:36 | |
| *** pcichy has joined #openstack-infra | 16:37 | |
| *** yamamoto has quit IRC | 16:37 | |
| *** racciari has quit IRC | 16:38 | |
| *** rkukura has quit IRC | 16:39 | |
| *** trown is now known as trown|lunch | 16:40 | |
| clarkb | fungi: logan- fwiw in osic changing dns to go over ipv6 only largely corrected the problem | 16:41 |
| clarkb | I don't recall any failurs after that change that fit this profile. So I think we can start there on the infra side before tuning the cloud itself | 16:41 |
| *** udesale has quit IRC | 16:41 | |
| *** jpich has quit IRC | 16:43 | |
| logan- | cool, wfm | 16:43 |
| *** germs has joined #openstack-infra | 16:43 | |
| *** germs has quit IRC | 16:43 | |
| *** germs has joined #openstack-infra | 16:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 16:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: WIP: Add debian-sid to nodepool-dsvm testing https://review.openstack.org/556974 | 16:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: DNM: testing debian-sid https://review.openstack.org/556975 | 16:45 |
| dmsimard | clarkb, logan-: just got back from running an errand and I'm not caught up with the backlog yet. Did anyone make progress regarding the issues ? | 16:45 |
| *** andreas_s has quit IRC | 16:45 | |
| *** felipemonteiro_ has quit IRC | 16:45 | |
| pabelanger | fungi: zigo: ^first step to see if we can even build debian-sid with diskimage-builder :) | 16:46 |
| *** david-lyle has joined #openstack-infra | 16:46 | |
| clarkb | dmsimard: no I've sat down at conf and was going to look at making this a role in project-config for the base job but happy for someone else to do it too | 16:46 |
| *** armaan has quit IRC | 16:46 | |
| dmsimard | ok, I'll try and figure out if we can fix the DNS issues then | 16:47 |
| *** armaan has joined #openstack-infra | 16:47 | |
| clarkb | dmsimard: were you able to find the old code that did it with my failed pasted info? | 16:47 |
| clarkb | dmsimard: but I think we want ot update the configure-unbound role that base uses | 16:47 |
| *** yamamoto has joined #openstack-infra | 16:48 | |
| corvus | mordred: qq in https://review.openstack.org/556973 | 16:48 |
| clarkb | it is already writing out the config I think it is just not writing a v6 only config when v6 is routable and a v4 only config when v6 is not routable | 16:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: DNM: testing debian-sid https://review.openstack.org/556975 | 16:49 |
| *** janki has quit IRC | 16:49 | |
| clarkb | dmsimard: hrm though it is certainly trying | 16:50 |
| clarkb | dmsimard: maybe have a job log its forwarding config from unbound to double check it | 16:50 |
| *** dhajare_ has joined #openstack-infra | 16:50 | |
| clarkb | or check ara on a fail /me doe sthis | 16:50 |
| dmsimard | clarkb: yeah I haven't had the time to properly sit down for more than 5 minutes and at least look at the patches ianw and frickler have been working on | 16:50 |
| dmsimard | I'll look at what happens, what they sent and figure something out | 16:51 |
| clarkb | dmsimard: http://logs.openstack.org/88/365388/4/gate/neutron-grenade/4b89ce6/ara/result/2460c81a-1e2a-40e3-9256-b13a59091254/ its failing to do the ipv6 check | 16:51 |
| clarkb | dmsimard: so I think that is it, we just have to make the logic work in configure-unbound and I expect this will get much better | 16:51 |
| *** VW_ has joined #openstack-infra | 16:52 | |
| *** hashar is now known as hasharDockerized | 16:52 | |
| clarkb | something wrong with those facts I guess | 16:52 |
| pabelanger | clarkb: dmsimard: yah, I htink ianw fixed that last night | 16:52 |
| pabelanger | https://review.openstack.org/556740/ | 16:53 |
| pabelanger | but, would have expected ansible_default_ipv6 to work | 16:53 |
| pabelanger | guess a scope issue | 16:53 |
| *** jgr is now known as jgrassler | 16:53 | |
| *** yamamoto has quit IRC | 16:53 | |
| *** sree has joined #openstack-infra | 16:53 | |
| *** VW has quit IRC | 16:54 | |
| *** germs_ has joined #openstack-infra | 16:55 | |
| clarkb | looks like firewall setup is broken by that (or something else) | 16:55 |
| pabelanger | think something else | 16:55 |
| clarkb | http://logs.openstack.org/40/556740/6/check/openstack-infra-multinode-integration-ubuntu-xenial/67fb033/ara/result/b99b3931-dbf9-4053-9bf3-c32240a70150/ | 16:55 |
| pabelanger | https://review.openstack.org/556747/ is don't assuming ipv4 | 16:55 |
| *** vivsoni has joined #openstack-infra | 16:56 | |
| *** vivsoni_ has quit IRC | 16:56 | |
| pabelanger | but, for some reason ovs is breaking on centos | 16:56 |
| pabelanger | http://logs.openstack.org/47/556747/6/check/openstack-infra-multinode-integration-centos-7/ca752c3/ara/result/903cc013-c5f0-4af9-be1f-6f2cdb995e06/ | 16:56 |
| *** germs has quit IRC | 16:56 | |
| pabelanger | ovs-vsctl not found? | 16:57 |
| clarkb | that implies the package isn't being installed probably? | 16:57 |
| pabelanger | maybe | 16:57 |
| pabelanger | where do we get it again, RDO? | 16:57 |
| clarkb | that is where we got it in the past at least | 16:57 |
| *** dave-mccowan has joined #openstack-infra | 16:57 | |
| pabelanger | http://logs.openstack.org/47/556747/6/check/openstack-infra-multinode-integration-centos-7/ca752c3/job-output.txt.gz#_2018-03-27_12_35_08_036696 | 16:58 |
| pabelanger | openstack-ocata | 16:58 |
| *** VW_ has quit IRC | 16:58 | |
| *** VW has joined #openstack-infra | 16:59 | |
| *** derekh has quit IRC | 16:59 | |
| *** bobh has joined #openstack-infra | 16:59 | |
| clarkb | ovs-vsctl is one of the management cli tools for ovs the service | 17:00 |
| clarkb | in the past and on other distros its been in the one package together with the server but maybe it isn't there anymore or maybe it installs to a funny path? | 17:00 |
| clarkb | dmsimard: https://review.openstack.org/#/c/556740/6/roles/configure-unbound/tasks/main.yaml would that imply a bug somewhere? its not scoped to a different host so should run and automagically find its own hostvars right? | 17:01 |
| *** bobh has quit IRC | 17:01 | |
| *** bobh has joined #openstack-infra | 17:01 | |
| *** yamamoto has joined #openstack-infra | 17:03 | |
| pabelanger | clarkb: going to hold a centos-7 node and see why that is failing | 17:04 |
| pabelanger | rpm should have that binary | 17:05 |
| clarkb | pabelanger: http://logs.openstack.org/84/556784/2/check/openstack-infra-multinode-integration-ubuntu-bionic/c2e6d2e/ara/result/f1d2926a-8ce7-4bc7-a355-8b0da04e901f/ looks like bionic is not working due to apt get update problems too | 17:05 |
| clarkb | so I think we need to solve ^ and the ovs on centos problem first then we can start moving up the stack and address the actual ansible problems | 17:06 |
| openstackgerrit | Merged openstack-infra/project-config master: Test change to upload-logs https://review.openstack.org/556881 | 17:06 |
| pabelanger | clarkb: let me check reprepro | 17:06 |
| *** Swami has joined #openstack-infra | 17:07 | |
| *** yamamoto has quit IRC | 17:08 | |
| *** annp_ has joined #openstack-infra | 17:09 | |
| pabelanger | okay, have an autohold setup, rechecked 556747 | 17:09 |
| pabelanger | looking at reprepro logs now | 17:09 |
| *** rossella_s has quit IRC | 17:11 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:14 | |
| *** annp_ has quit IRC | 17:14 | |
| openstackgerrit | Merged openstack-infra/zone-zuul-ci.org master: Add git.zuul-ci.org https://review.openstack.org/556964 | 17:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 17:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 17:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Apply caching to entire REST API https://review.openstack.org/556981 | 17:15 |
| *** rkukura has joined #openstack-infra | 17:17 | |
| pabelanger | clarkb: so, i think bionic is update to date with upstream, but will know more when recheck happens | 17:17 |
| logan- | dmsimard: fwiw a different role I wrote does some ipv6 detection stuff that looks similar to the unbound setup. https://github.com/Logan2211/ansible-bird/blob/21d7d8de5af9e73c0853d3434a4b3d3f8dd39a70/tasks/bird_post_install.yml#L25-L30 ...loooking for ['scope'] == 'global' has been important there | 17:18 |
| pabelanger | clarkb: wonder if upstream bionic is broken, will check | 17:18 |
| *** yamamoto has joined #openstack-infra | 17:18 | |
| *** dave-mccowan has quit IRC | 17:19 | |
| clarkb | logan-: ya I think we may want to add in a scope == global check but so far its not even acknowledging it has an ipv6 addr at all | 17:19 |
| logan- | ah gotcha | 17:20 |
| mrhillsman | we had an impromptu | 17:21 |
| mrhillsman | ugh | 17:21 |
| fungi | Shrews: i may have found it. trying now with "?charset=utf8mb4" appended to the dburi it's using and it got past the problem bug. success! (probably! (maybe! (hopefully?))) | 17:21 |
| mrhillsman | meeting and wondering if we can add logs because hash startmeeting was not done | 17:21 |
| fungi | mrhillsman: you can link to the channel log and start timestamp instead | 17:22 |
| *** sambetts is now known as sambetts|afk | 17:23 | |
| mrhillsman | ok | 17:23 |
| *** yamamoto has quit IRC | 17:23 | |
| *** jamesmcarthur has quit IRC | 17:25 | |
| fungi | mrhillsman: we don't really have a tool to turn an irc channel log into meeting minutes after the fact, unfortunately | 17:25 |
| fungi | but the meeting channels are still completely logged | 17:25 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: Reorganzie ZFS document https://review.openstack.org/556988 | 17:26 |
| clarkb | pabelanger: dmsimard ping me if I can help further (but starting to context switch to conference). I have reviewed the two changes to make public ipv4 less required so assuming they don't need fixing I think we can push those once testing works | 17:27 |
| Shrews | fungi: awesome | 17:27 |
| *** rossella_s has joined #openstack-infra | 17:27 | |
| pabelanger | clarkb: ack | 17:30 |
| clarkb | would also be good if someone that understands ansible better than me can chime in if needing hostvars there is a bug | 17:30 |
| clarkb | I feel like it is a bug if necessary | 17:30 |
| pabelanger | yah, same | 17:30 |
| clarkb | corvus: ^ that probably interests you https://review.openstack.org/#/c/556740/6 | 17:30 |
| mrhillsman | yeah, thanks fungi | 17:30 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-storyboard master: Use utf8mb4 for MySQL database charset https://review.openstack.org/555787 | 17:33 |
| *** yamamoto has joined #openstack-infra | 17:33 | |
| *** dave-mccowan has joined #openstack-infra | 17:34 | |
| *** yamamoto has quit IRC | 17:38 | |
| *** yamamoto has joined #openstack-infra | 17:40 | |
| *** yamamoto has quit IRC | 17:40 | |
| *** tesseract has quit IRC | 17:40 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/irc-meetings master: Make Lujin Luo the main chair for Neutron upgrades meeting https://review.openstack.org/556990 | 17:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Rename javascript package to @zuul-ci/dashboard https://review.openstack.org/551999 | 17:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorganize REST API and dashboard urls https://review.openstack.org/556967 | 17:42 |
| *** dhajare_ has quit IRC | 17:43 | |
| pabelanger | clarkb: permissions now setup on https://review.openstack.org/556538/ for IRC logging. Looks like 2000UTC is also clear to merge | 17:45 |
| *** trown|lunch is now known as trown | 17:45 | |
| *** panda is now known as panda|off | 17:48 | |
| *** jamesmcarthur has joined #openstack-infra | 17:49 | |
| *** pcaruana has quit IRC | 17:51 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorganize REST API and dashboard urls https://review.openstack.org/556967 | 17:52 |
| pabelanger | clarkb: dmsimard: this set_fact doesn't look healthy: http://logs.openstack.org/40/556740/6/check/openstack-infra-multinode-integration-centos-7/5af044d/ara/result/9df6ac7e-2d6d-4ac8-af26-d4f4d655830d/ | 17:54 |
| clarkb | pabelanger: updated comment on https://review.openstack.org/#/c/556784/2 | 17:54 |
| clarkb | pabelanger: which is related to your link I think | 17:55 |
| clarkb | pabelanger: 556784 is likely the cause | 17:55 |
| pabelanger | I wonder if we should use ipwrap | 17:55 |
| pabelanger | filter | 17:55 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add a flag to disable no_log on upload-logs https://review.openstack.org/556877 | 17:56 |
| clarkb | pabelanger: oh actually reading the comments maybe those tests ran against ps1 only and ps2 fixed it | 17:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorganize REST API and dashboard urls https://review.openstack.org/556967 | 17:56 |
| clarkb | pabelanger: I've rechecked the hostvars usage change ot test that theory | 17:57 |
| pabelanger | yah | 17:58 |
| pabelanger | maybe | 17:58 |
| dmsimard | pabelanger: indeed, it doesn't look healthy | 17:58 |
| dmsimard | that looks like the inline jinja loop | 17:58 |
| pabelanger | looking at bionic job now | 17:59 |
| dmsimard | it doesn't look like the limestone custom labels are working despite the fix we did earlier this morning? :1/ | 18:00 |
| mordred | :( | 18:00 |
| dmsimard | did a recheck on https://review.openstack.org/#/c/556849/ just to make sure | 18:00 |
| dmsimard | but I got node failures | 18:00 |
| * dmsimard looks at http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nl02.openstack.org.yaml again | 18:01 | |
| dmsimard | nothing jumps at me :( | 18:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Set ubuntu-bionic as non-voting on zuul-jobs https://review.openstack.org/556995 | 18:03 |
| pabelanger | dmsimard: clarkb: disabled bionic on zuul-jobs until we can figure out why it is failing^ | 18:03 |
| dmsimard | clarkb: was infra meeting now ? or is my calendar wrong ? | 18:05 |
| clarkb | 1900UTC, in 55 minutes | 18:05 |
| dmsimard | I hate timezones | 18:06 |
| dmsimard | everyone at the same hour please | 18:06 |
| clarkb | dmsimard: yes thats what we've done by using utc :P | 18:06 |
| clarkb | it never changes | 18:06 |
| pabelanger | have a bionic node held now | 18:07 |
| pabelanger | I thought we added a flag to make bionic apt repos trusted | 18:07 |
| pabelanger | did that get removed? | 18:07 |
| dmsimard | clarkb: yeah but it changes when DST changes :( | 18:08 |
| *** diablo_rojo has quit IRC | 18:08 | |
| *** shardy has quit IRC | 18:09 | |
| *** psachin has joined #openstack-infra | 18:09 | |
| dmsimard | Shrews: do you think we need to restart the launcher or something for the label changes to take effect ? I'm trying to understand why I'm not getting any nodes but it's not obvious to me from the logs | 18:12 |
| pabelanger | clarkb: dmsimard: https://review.openstack.org/556776/ fixes bionic jobs, frickler fixed this morning | 18:12 |
| *** diablo_rojo has joined #openstack-infra | 18:13 | |
| Shrews | dmsimard: config changes are picked up automatically. | 18:13 |
| clarkb | pabelanger: ehy do xenial and trusty not need that? | 18:13 |
| Shrews | dmsimard: if you didn't set a min-ready, you won't get nodes until a request comes in specifying that node type | 18:13 |
| corvus | dmsimard: i have the clock on my irc client set to utc, so the clock i'm looking at when talking to openstacky folk is always utc. fungi has *all* of his clocks set to utc. one or the other of those tricks may help. :) | 18:14 |
| pabelanger | clarkb: new version of APT in bionic, more strictedd | 18:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 18:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 18:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Apply caching to entire REST API https://review.openstack.org/556981 | 18:14 |
| pabelanger | strict* | 18:14 |
| dmsimard | Shrews: I have a review here that attempts to use these labels: https://review.openstack.org/#/c/556849/ | 18:14 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: For utf8mb4 shorten teams.name and users.email https://review.openstack.org/556626 | 18:14 |
| dmsimard | Shrews: and I see requests (I guess?) in the nodepool logs i.e, http://paste.openstack.org/show/715439/ | 18:15 |
| dmsimard | Shrews: but it doesn't look like they get fulfilled | 18:15 |
| fungi | corvus: dmsimard: to be fair, there are a few clocks in my house which are set to local time because otherwise christine complains that she can't tell what time it is | 18:15 |
| dmsimard | Something I noticed, however, is that it seems that limestone has two held opensuse nodes (for a label that is essentially no longer configured for it) .. maybe that triggered something ? | 18:16 |
| pabelanger | dmsimard: what is max-server for limestone? we turned it off | 18:16 |
| fungi | but she's aware it's her responsibility to update them whenever daylight time starts/ends | 18:16 |
| pabelanger | I thought | 18:16 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-limestone | 18:16 |
| dmsimard | pabelanger: we turned it back on with custom labels this morning to test it individually | 18:16 |
| pabelanger | looks off to me | 18:16 |
| Shrews | dmsimard: looks like those requests were declined by nl02-22107-PoolWorker.limestone-regionone-main | 18:16 |
| Shrews | based on that paste | 18:16 |
| dmsimard | Shrews: that's sort of my guess too but I'm not able to figure out why | 18:17 |
| *** jamesmcarthur has quit IRC | 18:17 | |
| dmsimard | I suppose, if anything, those two held nodes are probably not helping | 18:17 |
| dmsimard | since they're using labels that are no longer configured for that provider | 18:18 |
| pabelanger | dmsimard: ah, missed the turn back on part | 18:18 |
| dmsimard | looks like ianw held them for 556740 so they're probably no longer relevant | 18:18 |
| Shrews | dmsimard: my first debugging step is to trace the request. e.g.) fgrep 200-0003336687 launcher-debug.log | 18:18 |
| dmsimard | Shrews: oh, that's good indeed -- I see an error here. | 18:19 |
| dmsimard | it's a trace for tumbleweed, so probably related to the held nodes I mentioned http://paste.openstack.org/raw/715442/ | 18:20 |
| *** felipemonteiro__ has joined #openstack-infra | 18:21 | |
| dmsimard | should I try to see if I can release/delete those ? or do we need to add the label back in first ? | 18:21 |
| pabelanger | I think we might have needed to 2 step the change at https://review.openstack.org/556841 set label: [] first, to delete online nodes, then add limestone specific labels | 18:21 |
| pabelanger | that is usually how we offline things | 18:21 |
| dmsimard | pabelanger: it was already technically offline, max node was at zero -- it's just that I didn't know there were two held nodes | 18:22 |
| pabelanger | dmsimard: right, what I am saying is because we had held nodes, we should have set labels: [] first to delete them. I think nodepool no longer knows which pool they belong too since config is deleted? | 18:23 |
| pabelanger | I think they might be orphaned based on config | 18:24 |
| dmsimard | pabelanger: I'll try and delete them ? What's the worse that can happen ? | 18:24 |
| pabelanger | nodepool doesn't know where the nodes live? | 18:24 |
| pabelanger | limestone-regionone | opensuse-tumbleweed isn't a valid config I think | 18:24 |
| dmsimard | pabelanger: they're showing up in "nodepool list" | 18:24 |
| *** psachin has quit IRC | 18:25 | |
| *** felipemonteiro_ has quit IRC | 18:25 | |
| pabelanger | because that is zookeeper, but shade might not be able to delete it | 18:25 |
| mordred | shad ecan delete anything | 18:25 |
| pabelanger | dmsimard: I mean, try | 18:25 |
| *** dtantsur is now known as dtantsur|afk | 18:25 | |
| dmsimard | pabelanger: looks like they were deleted successfully, I'll double check in the tenant | 18:25 |
| pabelanger | but opensuse-tumbleweed label isn't valid for nl02 | 18:25 |
| pabelanger | dmsimard: cook | 18:26 |
| pabelanger | cool* | 18:26 |
| dmsimard | yup the servers are really gone | 18:27 |
| dmsimard | ok let's try another recheck and see what happens.. | 18:27 |
| dmsimard | Shrews: thanks for the tip! | 18:27 |
| Shrews | np. it's all about the request id. it's the white rabbit... just follow it | 18:28 |
| pabelanger | okay, that looks like it. config was wedged because of held nodes | 18:28 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-limestone | 18:28 |
| pabelanger | looks online now | 18:28 |
| dmsimard | \o/ | 18:30 |
| fungi | infra meeting in t-30 minutes | 18:30 |
| dmsimard | pabelanger: yup plenty of nodes in build | 18:30 |
| pabelanger | dmsimard: yah, it would be worth a test if we did 2 step that change on a held nodes, eg: labels: [], nodepool would actually delete them. Over having to manually do it | 18:32 |
| pabelanger | can see if we have a unit test for that | 18:32 |
| *** jamesmcarthur has joined #openstack-infra | 18:33 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Set trusted flag for configure-mirrors test https://review.openstack.org/556776 | 18:33 |
| *** eharney has joined #openstack-infra | 18:35 | |
| pabelanger | clarkb: https://review.openstack.org/556740/ also looks happy after recheck | 18:35 |
| *** gary_perkins has quit IRC | 18:35 | |
| clarkb | pabelanger: ya I'm rechecking it aain for bionic fixes | 18:35 |
| pabelanger | ack | 18:36 |
| clarkb | pabelanger: can you review the changes it depends on? | 18:36 |
| pabelanger | yah, +3 | 18:36 |
| clarkb | pabelanger: there is one more below that one that also needs review (but currently has a -1 due to bionic that I've rechecked) | 18:37 |
| clarkb | oh you just got it too cool | 18:37 |
| dmsimard | pabelanger: I asked Shrews earlier today if there'd be a way to improve testing of the nodepool config -- out of the box there isn't much we can do right now | 18:39 |
| dmsimard | because when we landed the first patch, I changed the labels but I forgot to add them at the top of the file (i.e, I only changed the provider config) | 18:39 |
| *** rossella_s has quit IRC | 18:40 | |
| *** yamamoto has joined #openstack-infra | 18:40 | |
| clarkb | nodepool has a config validator command | 18:41 |
| clarkb | can we just use that? | 18:41 |
| pabelanger | dmsimard: didn't nodepool config-validate pick up the issue? | 18:41 |
| pabelanger | we should have a tox -enodepool jobs running | 18:41 |
| Shrews | that validates the yaml, but not cross-references | 18:41 |
| clarkb | cross references between what? | 18:42 |
| clarkb | if its content in the same file it should be able to do that | 18:42 |
| clarkb | the schema stuff is pretty flexible | 18:42 |
| Shrews | that provider label names reference the outer labels | 18:42 |
| pabelanger | isn't that something voluptuous could handle? | 18:42 |
| clarkb | ya it can | 18:42 |
| Shrews | if someone knows how to add that, please do | 18:42 |
| pabelanger | k | 18:42 |
| clarkb | you'd have to write a function for that portion of the schema and the nvalidate it in code directly but it is doable | 18:42 |
| clarkb | we did it with jjb outputing a list of jobs then checking zuul wasn't running any jobs outside that list | 18:43 |
| pabelanger | yah, I can make a note to see how to improve it | 18:43 |
| clarkb | so old zuul config validation is probably a good place to look at how to do that | 18:43 |
| pabelanger | +1 | 18:43 |
| dmsimard | Starting getting results back from the limestone nodes on the integration jobs (without any patches) | 18:44 |
| dmsimard | So there's the ipv4 thing we already identified: http://logs.openstack.org/49/556849/1/check/openstack-infra-multinode-integration-centos-7/6f9dffc/ara/result/a9a186cc-0b4c-4d88-a691-f19fdc0bf322/ | 18:44 |
| dmsimard | can already try rebasing that one to see if it fixes it | 18:44 |
| clarkb | dmsimard: we've already rechecked the entire stack and approved most of it | 18:45 |
| clarkb | or at least the ipv4 part of the tsack | 18:45 |
| dmsimard | clarkb: how are you testing it if you don't have the condition that fails it? :/ | 18:45 |
| dmsimard | I mean sure https://review.openstack.org/#/c/556784/ jobs passed but we didn't test it against limestone | 18:45 |
| *** HeOS has joined #openstack-infra | 18:46 | |
| clarkb | dmsimard: the changes treat public and private ipv4 the same | 18:46 |
| clarkb | dmsimard: so unless there is a typo in the private psecific side of things we are testing that the process in general works (because public ips are exercising the same stuff) | 18:46 |
| pabelanger | shouldn't need to rebase, just add a DNM comment with limestone-centos-7 atop of it | 18:46 |
| dmsimard | pabelanger: yeah that's what I was going to do | 18:47 |
| * dmsimard does that | 18:47 | |
| *** yamamoto has quit IRC | 18:47 | |
| clarkb | double checking is definitely good and we want ot positively test the dns stuff since it was "working" while being broken without us knowing | 18:48 |
| dmsimard | +1 | 18:49 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Do not merge: Test lack of ipv4 with limestone nodes https://review.openstack.org/557007 | 18:49 |
| pabelanger | clarkb: Shrews: so maybe instead of doing http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/cmd/config_validator.py move voluptuous into http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/config.py and just do schema validation by default when we build out the newconfig? | 18:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use requests instead of urllib.request in tests https://review.openstack.org/554057 | 18:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorganize REST API and dashboard urls https://review.openstack.org/556967 | 18:51 |
| *** ykarel|away has quit IRC | 18:54 | |
| fungi | ianw: do you still want to revisit the afs+reprepro topic in today's meeting, or should i clean that up? | 18:55 |
| fungi | clarkb: same question for your "clouds update" topic | 18:55 |
| clarkb | fungi: an update on limestone would probably be good | 18:56 |
| clarkb | since things have changed a bit in the last couple days | 18:56 |
| fungi | cool, i'll leave it on the agenda | 18:56 |
| *** gary_perkins has joined #openstack-infra | 18:57 | |
| fungi | dmsimard: is the irc discoverability spec ready for council vote? or is that entry on the agenda stale? | 18:57 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert infra-puppet-apply jobs to non-legacy https://review.openstack.org/557011 | 18:57 |
| clarkb | and now I've got to get lunch before sessions start again at 1pm. Thanks fungi for driving meeting | 18:58 |
| fungi | two minutes to meeting time | 18:58 |
| fungi | clarkb: yw. enjoy lunch! | 18:58 |
| *** immune2iocane has quit IRC | 18:58 | |
| dmsimard | fungi: stale | 18:58 |
| dmsimard | fungi: but worth discussing nonetheless | 18:58 |
| dmsimard | I'm not sure if there has been more comments /me looks | 18:58 |
| fungi | dmsimard: i don't see any yet | 18:58 |
| dmsimard | fungi: nope no comments so we can just mention it in passing :) | 18:59 |
| fungi | cool | 18:59 |
| ianw | fungi: clean them out i think, thanks | 18:59 |
| *** Goneri has quit IRC | 18:59 | |
| pabelanger | fungi: first results of debian-sid: http://logs.openstack.org/75/556975/2/check/nodepool-functional-py35-debian-src/e8a7df7/controller/logs/builds/debian-sid-0000000026_log.txt.gz | 19:00 |
| pabelanger | maybe issue with bootstrap | 19:00 |
| *** germs_ has quit IRC | 19:00 | |
| fungi | meeting time in #openstack-meeting | 19:01 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: make Gentoo use the systemd profile https://review.openstack.org/556652 | 19:01 |
| *** yamamoto has joined #openstack-infra | 19:01 | |
| *** gfidente has quit IRC | 19:02 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: add gentoo systemd image for testing https://review.openstack.org/556661 | 19:02 |
| *** trevormc has joined #openstack-infra | 19:04 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: DNM: testing gentoo nodepool-dsvm https://review.openstack.org/556679 | 19:04 |
| *** yamamoto has quit IRC | 19:08 | |
| *** lpetrut has quit IRC | 19:11 | |
| *** jamesmcarthur has quit IRC | 19:13 | |
| *** jamesmcarthur has joined #openstack-infra | 19:14 | |
| *** jamesmcarthur has quit IRC | 19:14 | |
| *** jamesmcarthur has joined #openstack-infra | 19:14 | |
| *** pcichy has quit IRC | 19:15 | |
| *** VW has quit IRC | 19:15 | |
| *** trown is now known as trown|brb | 19:17 | |
| cmurphy | AJaeger: thanks for the quick feedback on https://review.openstack.org/#/c/557011/ - if i push a puppet-* change that depends-on that, will the new jobs get tested? | 19:17 |
| AJaeger | cmurphy: yes! | 19:17 |
| cmurphy | awesome | 19:18 |
| AJaeger | cmurphy: I should have suggested that - we should not merge such a change without testing ;) | 19:18 |
| cmurphy | you don't live dangerously enough | 19:19 |
| AJaeger | cmurphy: one test;) | 19:19 |
| cmurphy | :P | 19:20 |
| *** gouthamr has quit IRC | 19:21 | |
| *** immune2iocane has joined #openstack-infra | 19:21 | |
| *** VW has joined #openstack-infra | 19:23 | |
| *** hasharDockerized is now known as hashar | 19:25 | |
| *** agopi|out has quit IRC | 19:31 | |
| *** vivsoni has quit IRC | 19:32 | |
| *** vivsoni has joined #openstack-infra | 19:33 | |
| *** harlowja has joined #openstack-infra | 19:34 | |
| *** mlavalle has joined #openstack-infra | 19:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorganize REST API and dashboard urls https://review.openstack.org/556967 | 19:35 |
| mlavalle | Looking here http://stackalytics.com/?module=neutron, it seems that it is not keeping track of the reviews for the stable/queens branch. Is this something that the infra team can help with? | 19:37 |
| *** e0ne has joined #openstack-infra | 19:38 | |
| fungi | mlavalle: nope. you might see if there's a change which needs to be proposed to the stackalytics configuration, but we don't run that service, we just give them a place to host their source code | 19:39 |
| fungi | it's not an official openstack thing, just some devs at mirantis | 19:40 |
| mlavalle | Thanks fungi :-) | 19:42 |
| *** gouthamr has joined #openstack-infra | 19:42 | |
| fungi | kashyap: btw, good news on the wiki session expiration front i think... i logged in, didn't touch it for a few hours, then successfully edited a page and it didn't think i'd been logged out/expired | 19:44 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: DNM: debug some ipv6 variables https://review.openstack.org/557022 | 19:44 |
| ianw | pabelanger: ^ start there i guess... | 19:45 |
| *** Goneri has joined #openstack-infra | 19:45 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Handle blank public_ipv4 https://review.openstack.org/556784 | 19:46 |
| pabelanger | ianw: yah | 19:46 |
| pabelanger | ianw: maybe it is the 2nd when condition that is failing, missing address | 19:46 |
| clarkb | we should add the check for scope global once we understand the variable behavior | 19:47 |
| *** gouthamr has quit IRC | 19:47 | |
| *** gouthamr has joined #openstack-infra | 19:48 | |
| *** diablo_rojo has quit IRC | 19:49 | |
| *** dprince has quit IRC | 19:50 | |
| ianw | corvus: i think it's self testing, here's the debug messages : http://logs.openstack.org/22/557022/1/check/openstack-infra-base-integration-ubuntu-xenial/d67c06f/job-output.txt.gz#_2018-03-27_19_46_03_517672 | 19:50 |
| ianw | but yeah, they're the same | 19:51 |
| pabelanger | ianw: we should be able to what dmsimard is doing to run it on limestone using limestone-ubuntu-xenial node | 19:51 |
| corvus | ianw: yeah, the base-integration tests are designed to test roles like that; i haven't used them to confirm something like this yet. i may just be extra cautious. | 19:51 |
| *** jtomasek has quit IRC | 19:52 | |
| clarkb | coukd it be the instance was not configured properly? | 19:52 |
| clarkb | so the fact wasnt set | 19:52 |
| clarkb | it did show up in the logs though so very confused | 19:52 |
| ianw | clarkb: you can see it skip http://logs.openstack.org/77/554977/3/check/neutron-grenade/34fbe09/ara/result/6868df40-5985-4301-acd7-acfb4209c79b/ , but http://logs.openstack.org/77/554977/3/check/neutron-grenade/34fbe09/zuul-info/host-info.primary.yaml shows ansible_default_ipv6 | 19:53 |
| clarkb | ya | 19:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 19:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 19:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Apply caching to entire REST API https://review.openstack.org/556981 | 19:53 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert infra-puppet-apply jobs to non-legacy https://review.openstack.org/557011 | 19:54 |
| *** germs has joined #openstack-infra | 19:55 | |
| *** germs has quit IRC | 19:55 | |
| *** germs has joined #openstack-infra | 19:55 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard-webclient master: Add details to Search https://review.openstack.org/557024 | 19:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Pass whitelabel tenant to zuul::web https://review.openstack.org/556870 | 19:56 |
| corvus | ianw, pabelanger: i'm satisfied that the base-integration jobs are demonstrating the change is effective. sorry for the distraction. | 19:56 |
| corvus | (basically, all of the '-integration-' jobs on that change will use the role in a self-testing manner, the remaining jobs won't) | 19:56 |
| ianw | here's the dbg messages in multinode; both look ok as well http://logs.openstack.org/22/557022/1/check/openstack-infra-multinode-integration-ubuntu-xenial/8afd432/job-output.txt.gz#_2018-03-27_19_50_25_557124 | 19:57 |
| *** edmondsw has quit IRC | 19:58 | |
| *** vivsoni has quit IRC | 19:58 | |
| *** vivsoni_ has joined #openstack-infra | 19:58 | |
| *** edmondsw has joined #openstack-infra | 19:59 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert infra-puppet-apply jobs to non-legacy https://review.openstack.org/557011 | 19:59 |
| *** bobh has quit IRC | 20:00 | |
| pabelanger | ianw: i think we need to find out which condition in http://logs.openstack.org/77/554977/3/check/neutron-grenade/34fbe09/ara/result/6868df40-5985-4301-acd7-acfb4209c79b/ is false | 20:00 |
| pabelanger | since we have 2 | 20:00 |
| *** edmondsw_ has joined #openstack-infra | 20:01 | |
| pabelanger | ianw: but, lets try doing debug msg on limestone and see what happens | 20:01 |
| *** germs has quit IRC | 20:02 | |
| *** germs has joined #openstack-infra | 20:02 | |
| *** germs has quit IRC | 20:02 | |
| *** germs has joined #openstack-infra | 20:02 | |
| ianw | yeah | 20:03 |
| *** lpetrut has joined #openstack-infra | 20:03 | |
| *** edmondsw has quit IRC | 20:04 | |
| *** edmondsw_ has quit IRC | 20:04 | |
| *** edmondsw has joined #openstack-infra | 20:04 | |
| mlavalle | fungi: in case someone asks again, this is what needs to be done https://review.openstack.org/#/c/557028/ | 20:04 |
| *** edmondsw has quit IRC | 20:04 | |
| *** diablo_rojo has joined #openstack-infra | 20:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: DNM: debug some ipv6 variables https://review.openstack.org/557022 | 20:04 |
| fungi | mlavalle: thanks! | 20:05 |
| *** yamamoto has joined #openstack-infra | 20:05 | |
| fungi | hopefully they're still reviewing changes there regularly | 20:05 |
| mlavalle | it seems so | 20:05 |
| AJaeger | mordred: may I remind you on the email regarding pushing all projects to pypi and the tox_install changes, please? | 20:06 |
| *** bobh has joined #openstack-infra | 20:06 | |
| fungi | clarkb: we're clear to remove the project renames entries for collectd-ceilometer-plugin and python-openstacksdk from the agenda page now, correct? | 20:06 |
| *** kjackal has quit IRC | 20:07 | |
| clarkb | fungi: I think so | 20:07 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config master: Move Cinder sheepdog job to experimental https://review.openstack.org/557031 | 20:07 |
| AJaeger | mordred: midonet confuses me - not sure what's happeing there, see https://review.openstack.org/555788 | 20:08 |
| AJaeger | fungi: yes, hope so ;) | 20:08 |
| ianw | pabelanger: here it is on rax though ... http://logs.openstack.org/77/554977/3/check/neutron-grenade/1ce5b48/job-output.txt.gz#_2018-03-27_15_07_49_295143 ? | 20:08 |
| prometheanfire | pabelanger: everything looks good I guess? | 20:08 |
| pabelanger | prometheanfire: yah, I think so | 20:09 |
| pabelanger | ianw: yah, weird | 20:09 |
| mordred | AJaeger: yes! I shall do that | 20:09 |
| prometheanfire | woo | 20:09 |
| *** yamamoto has quit IRC | 20:11 | |
| AJaeger | thanks, mordred | 20:11 |
| pabelanger | mordred: clarkb: fungi: mind a +3 to land https://review.openstack.org/556538/ I've confirmed no meetings are running | 20:11 |
| pabelanger | and IRC access has been added to bot | 20:12 |
| mordred | infra-root: https://review.openstack.org/#/c/556869 https://review.openstack.org/#/c/556973/ and https://review.openstack.org/#/c/556870/ are the puppet-side changes to go with https://review.openstack.org/#/c/556967/ | 20:12 |
| mordred | rolling it out will mean a small downtime of the zuul-web dashboard, since we need to update the python, puppet and javascript all together - this is the last thing on the v3 release blockers list (that isn't github3.py release) | 20:13 |
| pabelanger | looking | 20:14 |
| AJaeger | andreaf: could you rebase https://review.openstack.org/#/c/540890/ , please? | 20:14 |
| corvus | mordred: test failure on 556869 | 20:16 |
| pabelanger | mordred: you have a test failing on 556973 looks like apache2 failed to start | 20:16 |
| mordred | yay | 20:17 |
| *** jamesmca_ has joined #openstack-infra | 20:20 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: For utf8mb4 shorten teams.name and users.email https://review.openstack.org/556626 | 20:21 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert infra-puppet-apply jobs to non-legacy https://review.openstack.org/557011 | 20:22 |
| mordred | pabelanger: do you know any *good* way for me to run that again somewhere and see what the file it created was? | 20:23 |
| *** eernst has quit IRC | 20:23 | |
| *** vivsoni_ has quit IRC | 20:23 | |
| *** eernst has joined #openstack-infra | 20:23 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| pabelanger | mordred: no, I guess we'd likely need an autohold. or maybe cmurphy can recommend | 20:24 |
| prometheanfire | pabelanger: should I ping clark about those patches then (glean/project-config/nodepool)? | 20:25 |
| mordred | pabelanger: cmurphy knows everything | 20:25 |
| *** gouthamr has quit IRC | 20:25 | |
| *** vivsoni has joined #openstack-infra | 20:25 | |
| *** tosky has quit IRC | 20:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: DNM: debug some ipv6 variables https://review.openstack.org/557022 | 20:25 |
| cmurphy | hrm? | 20:25 |
| *** eernst_ has joined #openstack-infra | 20:26 | |
| *** eernst has quit IRC | 20:27 | |
| mordred | cmurphy: I've got a puppet change failing a beaker test because apache won't start - which means i'm pretty sure the vhost template is getting borked ... any 'good' way you can think of to get a copy of the produced vhost template output? | 20:27 |
| mordred | cmurphy: (other than just holding a build node) | 20:27 |
| pabelanger | prometheanfire: yah, likely. But question on 556661 | 20:28 |
| mordred | cmurphy: oh - wait - I think I found the issue ... | 20:28 |
| prometheanfire | sure | 20:29 |
| pabelanger | mordred: cmurphy: we could have zuul collect apache logs on post-run | 20:29 |
| pabelanger | that should show error | 20:29 |
| clarkb | are we holding off on 556740 until we understand it better or can we get that in and then start pushing jobs to limestone again? | 20:30 |
| cmurphy | mordred: you could add a shell('cat /etc/apache2/sites-enabled/whatever') in http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/spec/acceptance/basic_spec.rb and also remove the catch_failures: true so that the test doesn't abort but that's all hacks, really we should do what pabelanger said | 20:30 |
| *** germs has quit IRC | 20:30 | |
| *** tosky has joined #openstack-infra | 20:30 | |
| mordred | cmurphy: ++ | 20:30 |
| *** eernst_ has quit IRC | 20:31 | |
| cmurphy | mordred: i think you lost your </Directory> | 20:32 |
| prometheanfire | pabelanger: I'll put it in a comment, but we've had instances of pip really messing up the system in gentoo before (we went as far as to disallow system installs in our pip package) | 20:32 |
| prometheanfire | pabelanger: it might be better now, but once bitten, twice shy | 20:33 |
| mordred | cmurphy: I did? | 20:33 |
| openstackgerrit | Merged openstack-infra/zuul master: Rename javascript package to @zuul-ci/dashboard https://review.openstack.org/551999 | 20:34 |
| cmurphy | mordred: in https://review.openstack.org/#/c/556973/4/templates/zuulv3.vhost.erb i don't see a closing </Directory> | 20:34 |
| mordred | cmurphy: oh. how weird | 20:34 |
| mordred | cmurphy: I see it locally ... | 20:34 |
| mordred | maybe I fixed locally without remembering doing so | 20:34 |
| andreaf | AJaeger: sorry that slipped out of my mind, I'll try tomorrow | 20:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Move Cinder sheepdog job to experimental https://review.openstack.org/557031 | 20:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 20:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 20:35 |
| mordred | cmurphy: ah - I was looking at the wrong patch | 20:35 |
| *** trevormc has quit IRC | 20:36 | |
| pabelanger | prometheanfire: yah, I think we want to pip / virtualenv like other images, give that we'd want to do that same with nodepool.o.o. That way we expect the same version of pip / virtuelenv on all images | 20:37 |
| pabelanger | prometheanfire: but, aside from that I think the job is correct, and working | 20:38 |
| dhellmann | AJaeger : please take another look at https://review.openstack.org/#/c/555426/ when you have a minute. I think we need to proceed with that as-is while we change the validation job in the other patch series. | 20:38 |
| prometheanfire | pabelanger: I'm not sure if that will break things, I know it has in the past. | 20:40 |
| clarkb | ianw: re 556740 anything else we need to do before getting it in and then pushing jobs back to limestone? | 20:40 |
| pabelanger | prometheanfire: yah, something we'll have to keep a top of then | 20:40 |
| *** eernst has joined #openstack-infra | 20:40 | |
| *** eernst has quit IRC | 20:41 | |
| *** jamesmca_ has quit IRC | 20:41 | |
| prometheanfire | pabelanger: I'm going to submit a change removing that and also test locally | 20:41 |
| *** eernst has joined #openstack-infra | 20:41 | |
| prometheanfire | I hope it'll work, but testing will say | 20:42 |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: add gentoo systemd image for testing https://review.openstack.org/556661 | 20:43 |
| ianw | clarkb: i don't think so | 20:43 |
| clarkb | ianw: do you want to approve it? | 20:43 |
| *** eernst_ has joined #openstack-infra | 20:43 | |
| *** eernst has quit IRC | 20:43 | |
| pabelanger | prometheanfire: yah, and if broken. We have logs to look at so we can fix :) | 20:44 |
| *** hashar has quit IRC | 20:46 | |
| *** olivierbourdon38 has quit IRC | 20:46 | |
| prometheanfire | ya, don't fix things til you actually need to :P | 20:46 |
| *** olivierbourdon38 has joined #openstack-infra | 20:46 | |
| *** HeOS has quit IRC | 20:47 | |
| *** eernst has joined #openstack-infra | 20:48 | |
| *** eernst_ has quit IRC | 20:48 | |
| *** eernst has quit IRC | 20:49 | |
| *** eernst has joined #openstack-infra | 20:49 | |
| *** eernst has quit IRC | 20:50 | |
| *** eernst has joined #openstack-infra | 20:50 | |
| *** rossella_s has joined #openstack-infra | 20:50 | |
| *** HeOS has joined #openstack-infra | 20:50 | |
| *** eernst has quit IRC | 20:51 | |
| *** eernst has joined #openstack-infra | 20:51 | |
| *** eernst has quit IRC | 20:53 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: DNM: debug some ipv6 variables https://review.openstack.org/557022 | 20:53 |
| *** eernst has joined #openstack-infra | 20:53 | |
| ianw | clarkb: ok, just have to get the patch under it moving again | 20:54 |
| *** Goneri has quit IRC | 20:56 | |
| *** inc0 has quit IRC | 20:57 | |
| *** VW_ has joined #openstack-infra | 20:57 | |
| *** edmondsw has joined #openstack-infra | 20:59 | |
| *** VW has quit IRC | 21:00 | |
| *** edmondsw_ has joined #openstack-infra | 21:00 | |
| *** dave-mccowan has quit IRC | 21:01 | |
| *** rossella_s has quit IRC | 21:02 | |
| *** edmondsw has quit IRC | 21:04 | |
| *** chkumar246 has joined #openstack-infra | 21:04 | |
| *** jamesmcarthur has joined #openstack-infra | 21:04 | |
| *** chandankumar has quit IRC | 21:05 | |
| *** edmondsw_ has quit IRC | 21:05 | |
| *** salv-orl_ has joined #openstack-infra | 21:06 | |
| *** salv-orlando has quit IRC | 21:06 | |
| *** yamamoto has joined #openstack-infra | 21:07 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: DNM: debug some ipv6 variables https://review.openstack.org/557022 | 21:08 |
| *** eharney has quit IRC | 21:08 | |
| *** jamesmcarthur has quit IRC | 21:09 | |
| *** chkumar246 has quit IRC | 21:09 | |
| *** yamamoto has quit IRC | 21:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-cgit master: Fix erroneous serveraliases entry https://review.openstack.org/557054 | 21:13 |
| corvus | infra-root: i've observed some flapping between different vhosts on the git farm. i'm hoping that the blank serveraliases line is the problem. that ^ fixes it. i've tested it manually on git08. | 21:14 |
| corvus | infra-root: i'd like to land that asap to see if it fixes the issue, if not, we may want to revert while we regroup. | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| corvus | er, i just spotted a typo in it | 21:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-cgit master: Fix erroneous serveraliases entry https://review.openstack.org/557054 | 21:16 |
| corvus | that's the typo fixed -- it's in a section we don't use, so shouldn't affect earlier test results | 21:16 |
| *** thingee has quit IRC | 21:18 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Don't assume nodepool.[private|public]_ipv4 https://review.openstack.org/556747 | 21:18 |
| pabelanger | +2 | 21:19 |
| *** chkumar246 has joined #openstack-infra | 21:21 | |
| *** vivsoni has quit IRC | 21:21 | |
| cmurphy | what's the right way to convert http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/infra-puppet-apply-3-centos-7/post.yaml to a zuulv3ish thing? | 21:22 |
| *** lpetrut has quit IRC | 21:23 | |
| ianw | cmurphy: i'd probably just leave it as is? | 21:23 |
| corvus | let me see if i can find something better :) | 21:23 |
| *** r-daneel has quit IRC | 21:25 | |
| cmurphy | :) | 21:25 |
| *** vivsoni has joined #openstack-infra | 21:26 | |
| *** efoley has joined #openstack-infra | 21:27 | |
| *** felipemonteiro__ has quit IRC | 21:28 | |
| corvus | hrm. the stage-output role seems only to be used by the devstack job. i think we talked about moving that into the base job to make things really easy. | 21:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: DNM: debug some ipv6 variables https://review.openstack.org/557022 | 21:28 |
| cmurphy | the linter job doesn't like that one http://logs.openstack.org/11/557011/4/check/openstack-zuul-jobs-linters/e7134b3/job-output.txt.gz#_2018-03-27_20_32_42_214072 | 21:29 |
| corvus | cmurphy: in the mean time, i assume you're making a job that doesn't use "workspace" (as that's a v2 holdover) | 21:29 |
| ianw | corvus: that might explain it; i looked for something more concise when i was updating the gerrit jobs to publish wars, and a variation on the sychronize seemed to work ok | 21:29 |
| corvus | cmurphy: so you probably just want to drop the rsync_opts, and then adjust the src argument to be '{{ ansible_user_dir }}/{{ zuul.projects['system-config'].src_dir }}/applytest/' | 21:31 |
| corvus | cmurphy: basically, an ansible | 21:31 |
| corvus | grr | 21:31 |
| cmurphy | corvus: at the moment it's still using workspace/ but i'm sure i could figure out how to not do that https://review.openstack.org/#/c/557011/4/playbooks/infra-puppet-apply/run.yaml | 21:31 |
| *** immune2iocane has quit IRC | 21:32 | |
| corvus | cmurphy: oh, you're putting stage-output in your playbook :) | 21:32 |
| corvus | in *that* case, you can just use its variables and skip this part | 21:32 |
| * cmurphy doesn't know what she's doing :P | 21:32 | |
| corvus | cmurphy: it's the right thing to do, i just hesitated recommending it because i thought the plan was to put it in the base job and use it everywhere; that might mean this job will need to be adjusted later. but it should probably be okay to run twice if we get there, i think it should be idempotent. | 21:33 |
| corvus | cmurphy: ignore what i've said in irc so far and lemme write some things on that change | 21:33 |
| cmurphy | corvus: okie thank you | 21:34 |
| *** rfolco|ruck is now known as rfolco|off | 21:35 | |
| *** edmondsw has joined #openstack-infra | 21:35 | |
| clarkb | ianw: 556740 apperas ready now | 21:35 |
| *** rossella_s has joined #openstack-infra | 21:36 | |
| *** edmondsw has quit IRC | 21:36 | |
| *** efoley has quit IRC | 21:42 | |
| corvus | cmurphy: ok, left comments on change; let me know if they make sense | 21:47 |
| corvus | or, better, if they don't make sense | 21:47 |
| *** mriedem is now known as mriedem_afk | 21:47 | |
| *** esberglu has quit IRC | 21:47 | |
| openstackgerrit | Fatema Khalid Sherif proposed openstack-infra/storyboard master: Partial-Bug: # 6142 https://review.openstack.org/557070 | 21:47 |
| *** felipemonteiro has joined #openstack-infra | 21:47 | |
| *** HeOS has quit IRC | 21:47 | |
| *** esberglu has joined #openstack-infra | 21:48 | |
| *** esberglu has quit IRC | 21:48 | |
| *** esberglu has joined #openstack-infra | 21:49 | |
| *** e0ne has quit IRC | 21:49 | |
| anteaya | corvus, thanks for the link in the comment on https://review.openstack.org/#/c/349831/ | 21:51 |
| cmurphy | corvus: that seems to make sense, will give it a try | 21:53 |
| *** esberglu has quit IRC | 21:53 | |
| corvus | anteaya: you're welcome | 21:56 |
| corvus | cmurphy: let me know how it goes | 21:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 21:56 |
| mordred | cmurphy: I'm trying one of your hacks now | 21:56 |
| cmurphy | mordred: it would be way easier to just hold the node | 21:57 |
| mordred | cmurphy: oh - actually - I just discovered that one of the jobs actually saves the files | 21:58 |
| mordred | cmurphy: it's almost like someone thoughtof this :) | 21:58 |
| mordred | cmurphy: http://logs.openstack.org/69/556869/6/check/legacy-puppet-beaker-rspec-infra/7a02fbd/logs/apache_config/50-ubuntu-xenial-rax-ord-0003223886.conf.txt.gz | 21:58 |
| openstackgerrit | Merged openstack-infra/puppet-cgit master: Fix erroneous serveraliases entry https://review.openstack.org/557054 | 21:58 |
| *** gouthamr has joined #openstack-infra | 21:59 | |
| cmurphy | mordred: nice :) | 22:00 |
| *** jlabarre has quit IRC | 22:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 22:01 |
| *** salv-orl_ has quit IRC | 22:01 | |
| mordred | cmurphy: yah. and it was easy to find with that | 22:01 |
| *** esberglu has joined #openstack-infra | 22:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 22:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Apply caching to entire REST API https://review.openstack.org/556981 | 22:01 |
| *** salv-orlando has joined #openstack-infra | 22:01 | |
| mordred | pabelanger, corvus: it's going to pass this time - I can feel it | 22:01 |
| *** gouthamr has quit IRC | 22:04 | |
| *** esberglu has quit IRC | 22:05 | |
| *** salv-orlando has quit IRC | 22:06 | |
| *** rkukura has quit IRC | 22:07 | |
| cmurphy | corvus: the linter still doesn't like the stage-output role, "[ANSIBLE0013] Use shell only when shell functionality is required" | 22:08 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to add presentation category group as long as endpoint to manage child collections (associated tracks and allowed groups) https://review.openstack.org/557077 | 22:08 |
| *** yamamoto has joined #openstack-infra | 22:08 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to add presentation category group as long as endpoint to manage child collections (associated tracks and allowed groups) https://review.openstack.org/557077 | 22:09 |
| *** rossella_s has quit IRC | 22:11 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert infra-puppet-apply jobs to non-legacy https://review.openstack.org/557011 | 22:12 |
| *** kgiusti has quit IRC | 22:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 22:12 |
| corvus | cmurphy: neat, are we running a linter on that repo but not zuul-jobs? | 22:13 |
| *** yamamoto has quit IRC | 22:14 | |
| *** hemna_ has quit IRC | 22:14 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert infra-puppet-apply jobs to non-legacy https://review.openstack.org/557011 | 22:15 |
| *** mikal has joined #openstack-infra | 22:15 | |
| *** lbragstad has quit IRC | 22:15 | |
| * mordred hands cmurphy a basket of fluffy bunny rabbits for working on the puppet apply jobs | 22:16 | |
| *** rcernin has joined #openstack-infra | 22:16 | |
| ianw | anyone know the magic to match across consecutive lines in kibana? | 22:17 |
| *** bobh has quit IRC | 22:18 | |
| *** mikal_ has quit IRC | 22:18 | |
| cmurphy | corvus: ah yeah could be http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project.yaml#n28 | 22:18 |
| cmurphy | i don't see that in zuul-jobs | 22:18 |
| *** bobh has joined #openstack-infra | 22:18 | |
| corvus | pabelanger: ^ know what's up with linters on zuul-jobs? | 22:18 |
| * cmurphy snuggles all the bunny rabbits | 22:19 | |
| pabelanger | corvus: i though we had one on zuul-jobs | 22:19 |
| pabelanger | I think set in project-config? | 22:20 |
| corvus | infra-root: with the latest change to the git farm, i'm still seeing cgit occasionally return the wrong site. i believe i've confirmed (by looking at log files) that the request was routed to the correct vhost, but it seems to have returned the wrong data. i'm wondering if somehow we're running the wrong cgit? i could use another pair of eyes on this. can someone help? | 22:20 |
| pabelanger | cmurphy: corvus: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n2177 is where we set openstack-zuul-jobs-linters | 22:21 |
| fungi | corvus: i'm taking a look now. do you see any evidence it could be limited to a specific backend, or does it seem to be randomly spread across them (or can you tell either way)? | 22:23 |
| *** bobh has quit IRC | 22:23 | |
| corvus | fungi: the one i just caught was git01. i'll hit refresh a bunch and see if it hits another. | 22:23 |
| fungi | corvus: also, are you seeing it with requests to git.o.o and git.z.o or just one? just the cgit webui or also breaking clone urls? | 22:24 |
| *** yolanda has quit IRC | 22:25 | |
| pabelanger | oh, I just git the issue just going to https://git.zuul-ci.org/cgit I see openstack data | 22:26 |
| corvus | fungi: both directions | 22:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 22:26 |
| fungi | corvus: any guess at the frequency? i've force-refreshed a couple dozen times and so far haven't hit it | 22:27 |
| corvus | fungi: i just managed to hit all of the backends (only 0-7 are in production; 8 is in maint) and got the correct data on all of them. i'll try some more until i hit the error again. | 22:27 |
| *** yolanda has joined #openstack-infra | 22:27 | |
| corvus | fungi: i have a hunch, unsubstantiated, that somehow if i switch between the sites it might be more frequent? | 22:27 |
| *** rkukura has joined #openstack-infra | 22:27 | |
| fungi | oh, perhaps | 22:28 |
| fungi | because i've been force-refreshing http://git.zuul-ci.org/cgit over and over for a couple minutes and always get the correct content | 22:28 |
| fungi | i wonder if https exhibits it but http doesn't? | 22:28 |
| corvus | okay, i switched between https://zuul and http://openstack (note the protocol difference), and hit the error on https://zuul with git04 as the backend | 22:29 |
| mordred | corvus: wow | 22:29 |
| corvus | and i just hit the error on http://openstack | 22:29 |
| corvus | so it can manifest with either protocol | 22:29 |
| corvus | (that was on git03) | 22:29 |
| fungi | refreshing https://git.zuul-ci.org/cgit over and over now and not seeing any problem either, so the switching between sites thing sounds likely as a (partial?) trigger | 22:29 |
| fungi | nope! i finally got one where https://git.zuul-ci.org/cgit returned git.o.o content (bit without the css) | 22:30 |
| *** dayou has quit IRC | 22:30 | |
| fungi | and that's without having switched between sites at all | 22:30 |
| corvus | i've gotten all 4 combos to return the wrong data | 22:31 |
| pabelanger | I see what corvus is reporting if I switch protocols too | 22:31 |
| corvus | http/https + openstack/zuul | 22:31 |
| *** germs has joined #openstack-infra | 22:31 | |
| *** germs has quit IRC | 22:31 | |
| *** germs has joined #openstack-infra | 22:31 | |
| fungi | so, yeah, switching between the sites doesn't seem to be necessary to trigger it | 22:31 |
| *** yamahata has joined #openstack-infra | 22:31 | |
| corvus | and yeah, that's how it manifests -- the css files are different for the two sites, so if you get wrong data returned for the /cgit url, the css path won't be valid on that host | 22:32 |
| corvus | so the project listing is different and there's no css/logo | 22:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Update docs with better websocket instructions https://review.openstack.org/557083 | 22:32 |
| fungi | maybe the SetEnv directives are sometimes leaking between responses for different vhosts | 22:32 |
| *** lbragstad has joined #openstack-infra | 22:32 | |
| *** lbragstad has quit IRC | 22:32 | |
| corvus | fungi: that seems like a promising avenue | 22:33 |
| *** eernst has quit IRC | 22:33 | |
| corvus | and i wonder if the worker scheme matters | 22:33 |
| *** eernst has joined #openstack-infra | 22:33 | |
| corvus | how do we know which mpm we're using? | 22:34 |
| tosky | uhm, I have this review which adds the lower-contraints job which didn't trigger the gate jobs, even with +2+W and after a recheck | 22:34 |
| fungi | great question--doesn't seem to be separate packages on centos | 22:34 |
| tosky | https://review.openstack.org/#/c/556207/ | 22:34 |
| tosky | do you know what could be wrong? | 22:36 |
| fungi | tosky: it depends on https://review.openstack.org/555402 which hasn't merged yet | 22:36 |
| *** germs has quit IRC | 22:36 | |
| fungi | hasn't even been reviewed, looks like | 22:36 |
| tosky | oh | 22:36 |
| *** kong has joined #openstack-infra | 22:36 | |
| tosky | sill me | 22:36 |
| pabelanger | corvus: fungi: is it maybe keepalive / persistent getting confused when we switch domains? | 22:37 |
| tosky | I was deceived the long commit message | 22:37 |
| corvus | fungi: Server MPM: prefork | 22:37 |
| fungi | corvus: apache documentation doesn't seem to warn against mixing different setenvs on different vhosts (even has it as an explicit example) so if that's the problem then it's likely a bug | 22:38 |
| *** felipemonteiro has quit IRC | 22:38 | |
| fungi | another theory... could the cgi be getting reused between requests for different vhosts? | 22:38 |
| *** eernst has quit IRC | 22:38 | |
| corvus | pabelanger: i believe keepalive is off on these servers | 22:38 |
| pabelanger | ack | 22:39 |
| corvus | fungi, pabelanger: maybe the switching domains comes in to play in that *someone* has to be accessing the other domain. so if i'm switching domains, it's enough to confuse fungi's sessions. | 22:39 |
| openstackgerrit | Matthew Thode proposed openstack-infra/nodepool master: DNM: testing gentoo nodepool-dsvm https://review.openstack.org/556679 | 22:40 |
| corvus | fungi: i wonder; it's using ScriptAlias... i wonder if anything about that is cached | 22:40 |
| *** eernst has joined #openstack-infra | 22:40 | |
| *** eernst has quit IRC | 22:41 | |
| *** eernst has joined #openstack-infra | 22:41 | |
| corvus | hrm. the prefork mpm seems like the least likely to run into problems like this since each request goes to a separate process | 22:42 |
| fungi | looks like under mpm_prefork we're using mod_cgi (whereas the other mpms are configured to use mod_cgid) | 22:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Add new parameters to zuul::web https://review.openstack.org/557084 | 22:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Pass web settings to zuul::web https://review.openstack.org/557085 | 22:43 |
| *** eernst has quit IRC | 22:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 22:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 22:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove zuul::web settings from ::zuul https://review.openstack.org/556870 | 22:47 |
| *** andreas_s has joined #openstack-infra | 22:47 | |
| corvus | cgit has a 'cache-root' argument | 22:47 |
| fungi | mmm | 22:47 |
| pabelanger | fungi: corvus: totally un tested, but maybe something like: SetEnvIf remote_host git.zuul-ci.org GIT_PROJECT_ROOT=/var/lib/git-alias/git.zuul-ci.org | 22:47 |
| fungi | if it's sharing one cache, then that would 'splain some things | 22:47 |
| corvus | 'cache-root-ttl' is: Number which specifies the time-to-live, in minutes, for the cached version of the repository index page | 22:48 |
| corvus | so if it's just caching the rendered page and spitting that back out, that would be consistent with the behavior | 22:48 |
| fungi | agreed | 22:48 |
| corvus | inspecting /var/cache/cgit, that seems to be what the cache files are | 22:50 |
| pabelanger | ack | 22:50 |
| fungi | yep, and cache-root can be set to a unique path per cgitrc if we want | 22:50 |
| corvus | i'll work on a patch to do that ^ | 22:50 |
| fungi | thanks! | 22:50 |
| * fungi is standing by to review what will probably be a fairly trivial patch to puppet-cgit | 22:50 | |
| *** andreas_s has quit IRC | 22:52 | |
| pabelanger | 456779 | 22:52 |
| pabelanger | sorry | 22:52 |
| *** eernst has joined #openstack-infra | 22:54 | |
| pabelanger | fungi: mind a review for eavesdrop.o.o: https://review.openstack.org/556538/ no meetings running currently | 22:54 |
| *** tpsilva has quit IRC | 22:55 | |
| pabelanger | clarkb: ianw: where did we land on https://review.openstack.org/556740/ ready to try it for limestone? | 22:55 |
| ianw | pabelanger: yeah, sorry just had to push through the depends-on under it. let's go with it and keep looking at issues | 22:56 |
| *** claudiub|2 has quit IRC | 22:57 | |
| pabelanger | ianw: wfm | 22:57 |
| ianw | clarkb: i'm currently trying to figure out how to match everything between " {timestamp} | TASK.*" to the blank line after it into one message in logstash-filters. tell me if i'm wasting my time :) | 22:57 |
| *** hongbin has quit IRC | 22:58 | |
| *** eernst has quit IRC | 22:59 | |
| *** gouthamr has joined #openstack-infra | 22:59 | |
| *** eernst has joined #openstack-infra | 23:00 | |
| *** salv-orlando has joined #openstack-infra | 23:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-cgit master: Separate cgit cache by vhost https://review.openstack.org/557091 | 23:03 |
| *** jcoufal has quit IRC | 23:03 | |
| corvus | fungi, pabelanger: ^ i verified that on git08. i also verified that if i hit the /cgit url on 2 vhosts, cgit writes the same cache filename | 23:03 |
| *** eernst has quit IRC | 23:05 | |
| *** eernst has joined #openstack-infra | 23:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove limestone specific labels from nl02.o.o https://review.openstack.org/557092 | 23:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Enable limestone-regione for nodepool.o.o https://review.openstack.org/557093 | 23:06 |
| corvus | ianw: clarkb will know better, but my understanding is that each line ("document" in es parlance) is completely independent, so i don't think it's possible. | 23:07 |
| pabelanger | corvus: nice work | 23:07 |
| pabelanger | +2 | 23:07 |
| *** felipemonteiro has joined #openstack-infra | 23:07 | |
| corvus | ianw: but maybe it's worth a look at the queries in elastic-recheck to see if there's anything like that? | 23:07 |
| corvus | pabelanger, fungi: thank you for your help working through all the possibilities :) | 23:07 |
| *** dayou has joined #openstack-infra | 23:08 | |
| *** salv-orlando has quit IRC | 23:08 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use hostvars to get ansible_default_ipv6 https://review.openstack.org/556740 | 23:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove limestone specific labels from nl02.o.o https://review.openstack.org/557092 | 23:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Enable limestone-regione for nodepool.o.o https://review.openstack.org/557093 | 23:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Enable limestone-regione for nodepool.o.o https://review.openstack.org/557093 | 23:09 |
| pabelanger | ianw: dmsimard: clarkb: I think ^ is how we 2 step the changes to the pool for nodepool | 23:10 |
| pabelanger | without anybody deleting nodes manually | 23:10 |
| ianw | corvus: yeah, i couldn't find anything. that's why my thought is we try to collate tasks into a single message at the logstash level | 23:10 |
| *** yamamoto has joined #openstack-infra | 23:10 | |
| *** eernst has quit IRC | 23:10 | |
| ianw | corvus: because while i can search for message:"configure-unbound : Check for IPv6" ... i can't really corrolate that with " localhost | skipping: Conditional result was False" on the next line | 23:11 |
| corvus | fascinating problem | 23:12 |
| corvus | i wonder if we could do something like add grok filters that know how to annotate lines with task information | 23:12 |
| *** eernst has joined #openstack-infra | 23:12 | |
| corvus | then you could do a structured search for 'message:"Conditional result was False" task:"configure-unbound"' or something | 23:13 |
| ianw | i think as a first pass, if we matched "2018-03-27 16:02:58.091136 | TASK .*" as the start | 23:13 |
| corvus | we'd probably only need to double the size of our es cluster ;) | 23:13 |
| ianw | and then "2018-03-27 16:02:58.207456 | " as the end | 23:13 |
| ianw | into one message, we could mostly search on that | 23:14 |
| corvus | ianw: the downside of that is that the entire output of devstack would be one message | 23:14 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Remove setting of ETCD_DOWNLOAD_URL https://review.openstack.org/556688 | 23:15 |
| *** eernst has quit IRC | 23:15 | |
| *** yamamoto has quit IRC | 23:15 | |
| ianw | urgh ... true. and lots of it is just "2018-03-27 15:11:21.115625 | TASK [shell] ..." with no particular name to negative match on | 23:18 |
| *** eernst has joined #openstack-infra | 23:19 | |
| *** efoley has joined #openstack-infra | 23:20 | |
| corvus | fungi, pabelanger: passed check; into gate now. i'm going to afk for a few; should be back by the time it's on server | 23:21 |
| fungi | sounds good--thanks for tracking it down corvus! | 23:21 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/irc-meetings master: The openstack-infra meeting is now chaired by clarkb https://review.openstack.org/557095 | 23:23 |
| pabelanger | ++ | 23:23 |
| *** eernst has quit IRC | 23:23 | |
| pabelanger | dmsimard: is that your presentation?^ | 23:24 |
| *** eernst has joined #openstack-infra | 23:24 | |
| *** immune2iocane has joined #openstack-infra | 23:24 | |
| *** felipemonteiro has quit IRC | 23:25 | |
| *** eernst has quit IRC | 23:25 | |
| *** chandankumar has joined #openstack-infra | 23:25 | |
| *** tosky has quit IRC | 23:25 | |
| *** eernst has joined #openstack-infra | 23:26 | |
| *** chkumar246 has quit IRC | 23:26 | |
| pabelanger | fungi: https://review.openstack.org/557095/ is an easy review too, and think helps dmsimard demo for his talk tonight :) | 23:27 |
| *** eernst_ has joined #openstack-infra | 23:28 | |
| *** eernst has quit IRC | 23:28 | |
| fungi | approvidado | 23:29 |
| fungi | i don't normally watch that repo, but felt qualified to review that particular change | 23:29 |
| *** chandankumar has quit IRC | 23:30 | |
| pabelanger | :) | 23:31 |
| *** eernst_ has quit IRC | 23:31 | |
| *** eernst has joined #openstack-infra | 23:31 | |
| *** harlowja has quit IRC | 23:32 | |
| *** mlavalle has quit IRC | 23:32 | |
| *** immune2iocane has quit IRC | 23:32 | |
| *** gouthamr has quit IRC | 23:35 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: The openstack-infra meeting is now chaired by clarkb https://review.openstack.org/557095 | 23:35 |
| *** germs has joined #openstack-infra | 23:35 | |
| *** germs has quit IRC | 23:35 | |
| *** germs has joined #openstack-infra | 23:35 | |
| *** germs has quit IRC | 23:36 | |
| *** eernst has quit IRC | 23:36 | |
| *** germs has joined #openstack-infra | 23:36 | |
| *** germs has quit IRC | 23:36 | |
| *** germs has joined #openstack-infra | 23:36 | |
| openstackgerrit | Merged openstack-infra/puppet-cgit master: Separate cgit cache by vhost https://review.openstack.org/557091 | 23:37 |
| openstackgerrit | Merged openstack-infra/system-config master: Add #softwarefactory IRC channel to eavesdrop.o.o https://review.openstack.org/556538 | 23:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Split v2 and v3 web config https://review.openstack.org/556869 | 23:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Support updated REST and HTML layout https://review.openstack.org/556973 | 23:38 |
| mordred | corvus: ok - I think 556869 should pass this time - now that I've actually updated the test case to use the new parameters | 23:40 |
| mordred | corvus: it probably shouldn't have taken me so long to figure out that was my problem | 23:41 |
| *** jamesmcarthur has joined #openstack-infra | 23:43 | |
| *** camunoz has quit IRC | 23:43 | |
| *** chandankumar has joined #openstack-infra | 23:43 | |
| *** mriedem_afk has quit IRC | 23:43 | |
| *** yamahata has quit IRC | 23:44 | |
| mordred | infra-root: https://review.openstack.org/#/c/557084/ and https://review.openstack.org/#/c/557085/ should both be easy and safe to land | 23:44 |
| *** jamesmcarthur has quit IRC | 23:47 | |
| corvus | mordred: both lgtm | 23:48 |
| *** mythicalphoenix has joined #openstack-infra | 23:53 | |
| prometheanfire | pabelanger: well, looks like it passed | 23:55 |
| pabelanger | prometheanfire: nice | 23:55 |
| mordred | corvus: replied on https://review.openstack.org/#/c/556869/ | 23:57 |
| *** germs has quit IRC | 23:58 | |
| *** germs has joined #openstack-infra | 23:59 | |
| *** germs has quit IRC | 23:59 | |
| *** germs has joined #openstack-infra | 23:59 | |
| corvus | mordred: ah. <shrug> :) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!