| *** flepied has quit IRC | 00:02 | |
| *** gfidente has joined #tripleo | 00:33 | |
| *** gfidente has quit IRC | 00:33 | |
| *** gfidente has joined #tripleo | 00:33 | |
| *** jkilpatr has quit IRC | 00:36 | |
| *** yamahata has joined #tripleo | 00:40 | |
| *** flepied has joined #tripleo | 00:49 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: swift/proxy: remove swift::proxy::ceilometer::rabbit_host https://review.openstack.org/394052 | 00:50 |
|---|---|---|
| *** limao has joined #tripleo | 01:01 | |
| *** limao has quit IRC | 01:02 | |
| *** limao has joined #tripleo | 01:02 | |
| *** lblanchard has joined #tripleo | 01:41 | |
| *** lblanchard has quit IRC | 01:46 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Containerized Services for Composable Roles https://review.openstack.org/330659 | 01:53 |
| *** maeca1 has joined #tripleo | 02:14 | |
| *** maeca1 has quit IRC | 02:27 | |
| *** dmacpher has joined #tripleo | 02:36 | |
| *** bkopilov has quit IRC | 02:57 | |
| *** tzumainn has joined #tripleo | 03:08 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Containerized Services for Composable Roles https://review.openstack.org/330659 | 03:18 |
| *** ElCoyote_ has quit IRC | 03:21 | |
| *** links has joined #tripleo | 03:55 | |
| *** coolsvap has joined #tripleo | 04:10 | |
| *** bkopilov has joined #tripleo | 04:17 | |
| *** ayoung has quit IRC | 04:25 | |
| openstackgerrit | Anshul Jain proposed openstack/diskimage-builder: DIB element to support cinder local attach/detach functionality https://review.openstack.org/385880 | 04:36 |
| *** chandankumar has joined #tripleo | 04:43 | |
| *** chlong has joined #tripleo | 04:47 | |
| *** tzumainn has quit IRC | 04:53 | |
| *** tzumainn has joined #tripleo | 04:53 | |
| *** masco has joined #tripleo | 05:04 | |
| *** sudipto has joined #tripleo | 05:05 | |
| *** sudipto_ has joined #tripleo | 05:05 | |
| *** skramaja has joined #tripleo | 05:10 | |
| *** limao has quit IRC | 05:13 | |
| *** limao has joined #tripleo | 05:13 | |
| *** oshvartz has quit IRC | 05:28 | |
| *** ramishra has quit IRC | 05:29 | |
| *** ramishra has joined #tripleo | 05:31 | |
| *** sudswas__ has joined #tripleo | 05:52 | |
| *** sudipto_ has quit IRC | 05:55 | |
| *** sudipto has quit IRC | 05:55 | |
| *** sudipto has joined #tripleo | 05:56 | |
| *** numans has joined #tripleo | 06:01 | |
| *** xuao has joined #tripleo | 06:02 | |
| *** rcernin has joined #tripleo | 06:03 | |
| *** xuao has quit IRC | 06:06 | |
| *** ealcaniz has joined #tripleo | 06:16 | |
| *** mcornea has joined #tripleo | 06:17 | |
| *** abregman has joined #tripleo | 06:18 | |
| *** panda|Zz is now known as panda|sick | 06:32 | |
| *** lmiccini has joined #tripleo | 06:33 | |
| *** pmannidi_ has quit IRC | 06:34 | |
| *** bfournie1 has quit IRC | 06:41 | |
| *** tzumainn has quit IRC | 06:42 | |
| *** Vijayendra has quit IRC | 06:43 | |
| *** pmannidi_ has joined #tripleo | 06:50 | |
| *** oshvartz has joined #tripleo | 06:54 | |
| *** bana_k has joined #tripleo | 06:59 | |
| *** tesseract has joined #tripleo | 07:06 | |
| *** tesseract is now known as Guest71310 | 07:06 | |
| cschwede | Hello! I have a small review request: https://review.openstack.org/#/c/389638/ needs only one more +2/+A for stable/newton - that would be very helpful | 07:15 |
| *** pcaruana has joined #tripleo | 07:17 | |
| *** pmannidi_ has quit IRC | 07:18 | |
| cmyster | cschwede: thats actually pretty nifty for the undercloud. I can see many more options going into needless | 07:19 |
| *** anshul has joined #tripleo | 07:19 | |
| cmyster | but I don't have +2 rights | 07:19 |
| *** limao has quit IRC | 07:19 | |
| cschwede | cmyster: thx for looking at it! yes, might be useful for other things too - curious which services could be disabled on the UC as well? | 07:20 |
| cmyster | cschwede: I can see a dynamic list here actually, since undercloud.conf can set things like telemtry=bool this can make sure its not there (but probably a misuse) | 07:22 |
| *** rasca has joined #tripleo | 07:23 | |
| *** oshvartz has quit IRC | 07:24 | |
| *** limao has joined #tripleo | 07:25 | |
| *** ebarrera has joined #tripleo | 07:26 | |
| *** bana_k has quit IRC | 07:34 | |
| *** ramishra has quit IRC | 07:38 | |
| *** ramishra has joined #tripleo | 07:38 | |
| *** cylopez has joined #tripleo | 07:41 | |
| *** asalkeld has joined #tripleo | 07:43 | |
| *** asalkeld has quit IRC | 07:47 | |
| *** ebalduf has quit IRC | 07:48 | |
| *** florianf has joined #tripleo | 07:49 | |
| *** dsariel has joined #tripleo | 07:50 | |
| *** athomas has joined #tripleo | 07:53 | |
| *** hjensas has joined #tripleo | 07:55 | |
| d0ugal | apetrich: oh, was the question about testing the password patch? | 07:59 |
| apetrich | d0ugal, yeah :) | 07:59 |
| apetrich | d0ugal, no wait at all. :) | 07:59 |
| d0ugal | apetrich: cool, it has actually landed in stable newton | 07:59 |
| d0ugal | apetrich: can you give it a go and see if you run into any other issues? | 08:00 |
| d0ugal | apetrich: so you'll want to make sure you have https://review.openstack.org/393192 and https://review.openstack.org/#/c/394195/ (the second hasn't merged yet) | 08:01 |
| d0ugal | oops, the first hasn't merged - got them the wrong way around. | 08:01 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Pass clients to get the get_password function https://review.openstack.org/393192 | 08:01 |
| *** asalkeld has joined #tripleo | 08:01 | |
| *** chem has joined #tripleo | 08:04 | |
| *** jprovazn has joined #tripleo | 08:05 | |
| *** tremble has joined #tripleo | 08:06 | |
| *** tremble has joined #tripleo | 08:06 | |
| *** asalkeld has quit IRC | 08:06 | |
| apetrich | d0ugal, anyway I check if both are there anyway | 08:08 |
| d0ugal | apetrich: Thanks | 08:09 |
| *** b00tcat has quit IRC | 08:09 | |
| *** b00tcat has joined #tripleo | 08:10 | |
| *** ccamacho has joined #tripleo | 08:10 | |
| *** fzdarsky|afk has joined #tripleo | 08:13 | |
| ccamacho | morning guys! | 08:13 |
| cmyster | morning | 08:13 |
| *** fzdarsky|afk is now known as fzdarsky | 08:14 | |
| *** chem has quit IRC | 08:17 | |
| *** asalkeld has joined #tripleo | 08:19 | |
| *** asalkeld has quit IRC | 08:19 | |
| bandini | morning * | 08:19 |
| *** jlinkes has joined #tripleo | 08:21 | |
| d0ugal | Morning! | 08:26 |
| matbu | o/ | 08:26 |
| matbu | d0ugal: i saw your review merged, thanks | 08:27 |
| matbu | d0ugal: i didn't test it yet, but i will | 08:27 |
| matbu | :) | 08:27 |
| *** Vijayendra has joined #tripleo | 08:28 | |
| *** pmannidi has joined #tripleo | 08:29 | |
| *** aufi has joined #tripleo | 08:31 | |
| *** liverpooler has joined #tripleo | 08:31 | |
| *** abregman_ has joined #tripleo | 08:33 | |
| *** abregman_ has quit IRC | 08:33 | |
| *** abregman has quit IRC | 08:36 | |
| *** Vijayendra has quit IRC | 08:40 | |
| *** amoralej|off is now known as amoralej | 08:41 | |
| *** jpena|off is now known as jpena | 08:41 | |
| *** chlong has quit IRC | 08:41 | |
| *** abregman has joined #tripleo | 08:43 | |
| *** ohamada has joined #tripleo | 08:46 | |
| *** chem has joined #tripleo | 08:46 | |
| -openstackstatus- NOTICE: Gerrit is going to be restarted due to slowness and proxy errors | 08:47 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** openstackgerrit has joined #tripleo | 08:48 | |
| *** jpich has joined #tripleo | 08:48 | |
| *** milan has joined #tripleo | 08:49 | |
| *** dbecker has joined #tripleo | 08:52 | |
| *** percevalbot has quit IRC | 08:55 | |
| *** gchamoul is now known as gchamoul|afk | 08:56 | |
| *** percevalbot has joined #tripleo | 08:56 | |
| *** shardy has joined #tripleo | 09:00 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/389945 | 09:01 |
| *** gchamoul|afk is now known as gchamoul | 09:04 | |
| *** dmacpher has quit IRC | 09:07 | |
| *** abregman is now known as abregman|mtg | 09:07 | |
| *** pblaho has joined #tripleo | 09:08 | |
| *** abregman_ has joined #tripleo | 09:09 | |
| *** abregman|mtg has quit IRC | 09:12 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Add CephClusterFSID to generated passwords https://review.openstack.org/390612 | 09:12 |
| *** lucas-afk is now known as lucasagomes | 09:18 | |
| cmyster | morning lucasagomes | 09:19 |
| lucasagomes | cmyster, morning | 09:19 |
| *** hewbrocca_afk is now known as hewbrocca | 09:25 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/389957 | 09:26 |
| *** dtantsur|afk is now known as dtantsur | 09:28 | |
| *** karthiks has joined #tripleo | 09:29 | |
| *** percevalbot has quit IRC | 09:30 | |
| *** iranzo has joined #tripleo | 09:31 | |
| *** iranzo has joined #tripleo | 09:31 | |
| *** percevalbot has joined #tripleo | 09:34 | |
| hewbrocca | folks, how is CI looking this morning | 09:41 |
| hewbrocca | all unblocked and green and stuff? | 09:41 |
| matbu | hewbrocca: it looks ok (afaik, periodic jobs is green) | 09:44 |
| *** derekh has joined #tripleo | 09:44 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 09:48 |
| shadower | I'm seeing an OVB HA failures -- not sure whether related to the previous problems: http://logs.openstack.org/93/391093/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/a90ef56/console.html | 09:49 |
| openstackgerrit | Merged openstack/tripleo-ui: Refactor *DriverFields components https://review.openstack.org/393289 | 09:53 |
| *** bogdando has quit IRC | 09:54 | |
| *** shardy has quit IRC | 09:55 | |
| therve | shadower, http://logs.openstack.org/93/391093/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/a90ef56/logs/overcloud-controller-0/var/log/gnocchi/metricd.txt.gz | 09:56 |
| therve | Redis issue seems to still be present :/ | 09:56 |
| shadower | yeah :/ | 09:56 |
| therve | matbu, Same for periodic AFAICT: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/cabef21/logs/overcloud-controller-0/var/log/gnocchi/metricd.txt.gz | 09:57 |
| *** shardy has joined #tripleo | 09:57 | |
| *** zoli|gone is now known as zoli | 09:57 | |
| *** zoli is now known as zoliXXL | 09:57 | |
| *** dsariel has quit IRC | 10:00 | |
| d0ugal | Where can I find the reason for a package build failing? | 10:01 |
| d0ugal | I am trying to track down the failure behind: http://logs.openstack.org/92/393192/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/6cb7497/console.html | 10:01 |
| *** Vijayendra has joined #tripleo | 10:03 | |
| zoliXXL | good morning | 10:03 |
| jpich | d0ugal: delorean_repos.tar.xz -> rpmbuild.log | 10:05 |
| jpich | d0ugal: " OSError: [Errno 17] File exists: '/tmp/tht/tripleo-heat-templates'" ...maybe transient? That's weird | 10:05 |
| d0ugal | jpich: aha, thanks! | 10:05 |
| d0ugal | jpich: yeah, that does seem weird. | 10:06 |
| *** akrivoka has joined #tripleo | 10:06 | |
| *** shardy_ has joined #tripleo | 10:07 | |
| *** rickflare has quit IRC | 10:07 | |
| *** shardy has quit IRC | 10:10 | |
| *** yamahata has quit IRC | 10:11 | |
| *** rickflare has joined #tripleo | 10:11 | |
| d0ugal | jpich: I rechecked it, so we shall see. | 10:14 |
| jpich | d0ugal: Thanks! | 10:14 |
| *** shardy_ is now known as shardy | 10:14 | |
| shardy | shadower: Hey, there's a couple of small comments on https://review.openstack.org/#/c/390854 - since it looks like we'll need another recheck do you want to see if we should address them now instead of rechecking this revision? | 10:15 |
| shadower | shardy: all right, sure | 10:16 |
| dtantsur | so, are things worth rechecking now? | 10:18 |
| hewbrocca | seems like we still have a Redis blocker | 10:18 |
| hewbrocca | Who is handling that issue? | 10:18 |
| * hewbrocca mildly disturbed by the lack of responses | 10:21 | |
| shardy | So yeah bug #1638350 has been fixed, but we still seem to have HA test failures | 10:21 |
| openstack | bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,Fix released] https://launchpad.net/bugs/1638350 - Assigned to Gabriele Cerami (gcerami) | 10:21 |
| shardy | I thought that bug had fixed all-the-HA-things :( | 10:21 |
| hewbrocca | damn | 10:21 |
| shardy | panda|sick and bandini were looking at redis things on Friday, let me see what the latest failures look like | 10:21 |
| derekh | hewbrocca: shardy I believe a new redis package has been built http://cbs.centos.org/koji/buildinfo?buildID=13831 | 10:22 |
| hewbrocca | shardy: thanks. Do we also need a telemetry person involved, since they are the only actual Redis consumers? | 10:22 |
| derekh | hewbrocca: shardy but I've just looked at the logs for a recent job and we don't seem to be using it yet | 10:22 |
| shardy | hewbrocca: not sure yet - cistatus actually shows the HA job passing quite regularly earlier today: | 10:23 |
| shardy | http://tripleo.org/cistatus.html | 10:23 |
| derekh | JOBLOGS ]$ grep redis-3 overcloud-controller-1.tar.xz_/var/log/host_info.txt | 10:23 |
| derekh | redis-3.2.4-1.el7.x86_64 | 10:23 |
| hewbrocca | bleh | 10:23 |
| jpich | derekh: I think there's some magic that might need to be done around CBS tags for CI to pick it up? | 10:23 |
| hewbrocca | shardy: So one wonders if it isn't slowness/timeout related | 10:24 |
| derekh | jpich: probably, looking in the current repo's now | 10:24 |
| bandini | well when redis does not come up (for whatever reason), gnocchi services seem to use up loads of CPU | 10:24 |
| hewbrocca | hmmph | 10:25 |
| hewbrocca | *that* is a telemetry issue | 10:25 |
| shardy | yeah we've seen this before, the gnocchi services spin forever eating CPU when there's any kind of issue, instead of eventually failing and declaring failure | 10:26 |
| hewbrocca | Yeah | 10:26 |
| derekh | The new version of redis is in this (pending?) repo http://cbs.centos.org/repos/cloud7-openstack-common-pending/x86_64/os/Packages/ | 10:26 |
| hewbrocca | this is actually a significant bug | 10:26 |
| shardy | http://logs.openstack.org/54/390854/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/cb9e1ed/console.html | 10:26 |
| derekh | do we use that? | 10:26 |
| shardy | so this failed about an hour ago, and it's the badstatusline error | 10:26 |
| hewbrocca | redis becomes temporarily unavailable and gnocchi DOS-es your cloud | 10:26 |
| shardy | So that's https://bugs.launchpad.net/tripleo/+bug/1638908 which isn't yet fixed | 10:27 |
| openstack | Launchpad bug 1638908 in tripleo-quickstart "Overcloud deployment fails in minimal configuration with ('Connection aborted.', BadStatusLine("''",))" [Undecided,In progress] - Assigned to Alfredo Moralejo (amoralej) | 10:27 |
| *** Vijayendra has quit IRC | 10:27 | |
| *** jd__ has joined #tripleo | 10:29 | |
| jd__ | hoy | 10:29 |
| hewbrocca | jd__: ! | 10:29 |
| jd__ | how can I help you gentlemen? | 10:29 |
| hewbrocca | shardy, bandini I have summoned the gnocchi maintainer | 10:29 |
| shardy | https://review.openstack.org/#/c/393876/ is trying to fix the badstatusline thing, but it's been proposed to tripleo-quickstart | 10:29 |
| *** eglynn has joined #tripleo | 10:29 | |
| derekh | The new version of redis is in the delorean-deps repository we use, I don't know how long its been there but it looks like that problem should now be solved http://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-ocata/common/ | 10:29 |
| derekh | shardy: jpich ^ | 10:29 |
| derekh | redis-3.2.4-2.el7.x86_64.rpm | 10:30 |
| jpich | derekh: Great :) | 10:30 |
| shardy | jd__: Hi! | 10:30 |
| shardy | jd__: so we're trying to understand the expected behavior of gnocchi-metricsd when there's some issue on startup | 10:30 |
| shardy | we've had a couple of cases recently when an error (in one case a packaging bug, and most recently a problem with redis) makes the service fail to start | 10:31 |
| shardy | but it spins forever, eating lots of CPU, instead of failing and declaring the service failed | 10:31 |
| jd__ | shardy: it should retry, but not as aggressive as fast | 10:32 |
| jd__ | shardy: do you have any hint where it loops too fast? | 10:32 |
| jd__ | or what was failing maybe? | 10:32 |
| *** milan has quit IRC | 10:32 | |
| *** oshvartz has joined #tripleo | 10:32 | |
| amoralej | but we don't have gnocchi in the undercloud in the cases where we hit https://bugs.launchpad.net/tripleo/+bug/1638908, i'd say | 10:33 |
| openstack | Launchpad bug 1638908 in tripleo-quickstart "Overcloud deployment fails in minimal configuration with ('Connection aborted.', BadStatusLine("''",))" [Undecided,In progress] - Assigned to Alfredo Moralejo (amoralej) | 10:33 |
| shardy | jd__: http://logs.openstack.org/93/391093/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/a90ef56/logs/overcloud-controller-0/var/log/gnocchi/metricd.txt.gz | 10:33 |
| shardy | that's an example | 10:33 |
| shardy | the previous time we hit it this was similar, with a bad version of cotyledon IIRC | 10:34 |
| amoralej | but that's in overcloud | 10:34 |
| amoralej | the bug i reported is referred to the issue in undercloud | 10:34 |
| sshnaidm | derekh, hi | 10:34 |
| shardy | amoralej: Yeah it's two different problems | 10:34 |
| amoralej | i'd say so | 10:34 |
| *** pblaho has quit IRC | 10:35 | |
| amoralej | what i've observed in these cases it's very long response time to heat GET api calls | 10:35 |
| jd__ | shardy: ok, it's a bug, I think I know what it is | 10:35 |
| sshnaidm | derekh, do you know what the timeout here mean? it doesn't seems like timeout for the whole job, right? is it for waiting for a environment only? https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L236 | 10:35 |
| shardy | amoralej: actually | 10:35 |
| shardy | https://github.com/openstack/instack-undercloud/commit/8a738272c7f63ad7e6e00f0836c2d1793ea4f125 | 10:35 |
| jd__ | shardy: I'll write a fix right now :) | 10:35 |
| shardy | we turned telemetry servies back on by default recently | 10:36 |
| hewbrocca | \o/ | 10:36 |
| hewbrocca | jd__: thanks | 10:36 |
| shardy | jd__: great, thanks! :) | 10:36 |
| hewbrocca | shardy: you happen to know is there a bugzilla for this issue | 10:36 |
| jd__ | shardy: thanks for reporting :) | 10:36 |
| *** jaosorior has joined #tripleo | 10:38 | |
| *** limao has quit IRC | 10:38 | |
| amoralej | shardy, we hit the issue even without telemetry enabled, https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-newton-delorean-minimal-72 | 10:38 |
| *** chem has quit IRC | 10:39 | |
| amoralej | let me check last occurrence | 10:39 |
| *** chem has joined #tripleo | 10:39 | |
| jaosorior | Hey guys, just to notify, I'm feeling quite sick, so won't be doing much today :/ | 10:43 |
| derekh | sshnaidm: that timeout should be the max amount of time the testenv exists, iirc it will be deleted if it is hit | 10:43 |
| shardy | hope you feel better soon jaosorior | 10:43 |
| amoralej | shardy, enable_telemetry=True in undercloud doesn't enable gnocchi, only ceilometer services | 10:44 |
| amoralej | with database dispatcher | 10:45 |
| amoralej | https://thirdparty-logs.rdoproject.org/jenkins-tripleo-quickstart-periodic-newton-delorean-ha_192gb-11/undercloud/etc/ceilometer/ceilometer.conf.gz | 10:45 |
| sshnaidm | derekh, and then it should kill the job as I see from the comments, although it doesn't seem to work this way... | 10:45 |
| derekh | sshnaidm: what make you say its not working this way? | 10:46 |
| shardy | amoralej: ah, definitely two issues then, thanks for confirming | 10:46 |
| panda|sick | oh, my still redis problems ? maybe we should merge https://review.openstack.org/392703 ? | 10:47 |
| panda|sick | that was green for me twice yesterday | 10:47 |
| panda|sick | I figured with the new package it wasn't needed anymore | 10:48 |
| sshnaidm | derekh, sorry, found out now that it worked :) | 10:49 |
| sshnaidm | derekh, I'm looking for a way to have the postci function to work before zuul kills everything and doesn't post logs | 10:49 |
| *** Vijayendra has joined #tripleo | 10:52 | |
| derekh | sshnaidm: I'd imagine the way to do this is to add another publisher to the job (I'm not 100% sure though) http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/tripleo.yaml#n183 | 10:52 |
| derekh | panda|sick: as far as I can see the jobs that have run so far are still using python-redis-2.10.3-1.el7.noarch | 10:53 |
| derekh | panda|sick: but the new version is there in the repo now | 10:53 |
| panda|sick | sshnaidm: wasn't arxcruz working on the same thing ? | 10:53 |
| derekh | panda|sick: so maybe the repo just hadn't updated up to now | 10:53 |
| sshnaidm | panda|sick, yeah, I'm trying to look too | 10:54 |
| panda|sick | derekh: so my tests yesterday worked only because of low usage of rh1 ? ... geez | 10:54 |
| sshnaidm | arxcruz, ^^ | 10:54 |
| arxcruz | sshnaidm: panda|sick hey | 10:54 |
| derekh | panda|sick: or maybe the proxy should be restarted in case it has the repo metadata cached | 10:55 |
| * arxcruz reading | 10:55 | |
| arxcruz | sshnaidm: https://review.openstack.org/#/c/393309/ | 10:55 |
| arxcruz | we got a timeout and logs collected on patchset 4 | 10:55 |
| sshnaidm | arxcruz, derekh : but for publisher we need to have the logs already prepared, right? or is it possible to write publisher that will duplicate postci function? | 10:56 |
| shardy | So https://bugs.launchpad.net/tripleo/+bug/1637961 references http://cbs.centos.org/koji/buildinfo?buildID=13831 which is a new version of redis | 10:56 |
| openstack | Launchpad bug 1637961 in tripleo "periodic HA master job pingtest times out" [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami) | 10:56 |
| *** pkovar has joined #tripleo | 10:56 | |
| panda|sick | derekh: have you checked if the package is in testing repo ? asking to apevec in rdo | 10:56 |
| shardy | that exists in http://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-ocata/common/ | 10:56 |
| arxcruz | sshnaidm: publisher will only get the logs in /var/log and upload right? | 10:56 |
| arxcruz | or am i wrong ? | 10:56 |
| shardy | but are we also missing a puppet-redis update? | 10:56 |
| sshnaidm | arxcruz, it's finished, not timeouted | 10:56 |
| * shardy needs more coffee | 10:56 | |
| derekh | sshnaidm: actually now that I think about it that wouldn't work, as the compute nodes would be gone | 10:57 |
| panda|sick | shardy: puppet-redis should not be needed in this case | 10:57 |
| panda|sick | a new * | 10:57 |
| derekh | sshnaidm: *overcloud nodes | 10:57 |
| sshnaidm | derekh, yeah, testenv-client destroys everything | 10:57 |
| *** zoliXXL is now known as zoli|lunch | 10:57 | |
| *** thrash|g0ne is now known as thrash | 10:57 | |
| arxcruz | sshnaidm: http://logs.openstack.org/09/393309/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/13f8c1c/console.html.gz#_2016-11-04_12_07_57_944293 | 10:58 |
| sshnaidm | derekh, arxcruz I thought, maybe to wrap everything into additional timeout in toci_instack_*.sh scripts.. | 10:58 |
| derekh | panda|sick: the latest HA job I'm following has passed, will check to see what version of redis is used when its done pushing logs | 10:58 |
| arxcruz | sshnaidm: but we will still have the global timeout from devstack-gate | 10:58 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo: WIP DO NOT MERGE Initial Composable HA https://review.openstack.org/362389 | 10:59 |
| sshnaidm | arxcruz, this job was timeouted when job finished and postci function started | 11:00 |
| sshnaidm | arxcruz, the problem is when job is timeouted before postci starts | 11:00 |
| openstackgerrit | Robin Cernin proposed openstack/tripleo-validations: Validation stonith device exists in OpenStack Platform HA cluster https://review.openstack.org/360102 | 11:01 |
| shardy | So yeah, the failure I was looking at from a couple of hours ago has the old version of redis still | 11:01 |
| *** lmiccini has quit IRC | 11:01 | |
| jpich | honza: Good morning! There are folks interested in following the work around proper logging for the UI, I'm wondering if you could move/open the blueprint in the TripleO tracker and keep it up to date? I was going to point them to https://blueprints.launchpad.net/tripleo-ui/+spec/websocket-logging but it doesn't look like the existing patches are even linked there :( (If I'm looking in the wrong place, please let me know!) | 11:01 |
| shardy | derekh: I notice the job in question is using a cached overcloud-full which may be related? | 11:02 |
| arxcruz | It would be so more easy if we add a timeout hook function in devstack-gate... :( | 11:02 |
| shardy | http://logs.openstack.org/54/390854/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/cb9e1ed/console.html#_2016-11-07_07_33_11_908396 | 11:02 |
| arxcruz | sshnaidm: well, the timeout will be triggered 13 minutes before the global timeout | 11:02 |
| arxcruz | which is plenty of time to postci script runs | 11:03 |
| panda|sick | shardy: derekh new redis package only arrived in ocata this morning. | 11:03 |
| panda|sick | my tests yesterday likely passed because of the low usage :( | 11:04 |
| panda|sick | the package arrived at about 9am, maybe in time for the periodic jobs, and to upload a new base image them | 11:05 |
| shardy | Ok so we need a periodic promotion to update the cached image | 11:05 |
| *** lmiccini has joined #tripleo | 11:05 | |
| shardy | or temporarily disable using cached images | 11:05 |
| *** ckyriakidou has joined #tripleo | 11:07 | |
| derekh | shardy: the cached image should get the new redis, we do a yum update in it | 11:07 |
| panda|sick | derekh: yeah, right! | 11:08 |
| shardy | derekh: Ok, I'll try rechecking as I guess the job I'm looking at started just before the repo got updated | 11:08 |
| panda|sick | peridoic jobs this morning was still using redis-3.2.4-1.el7.x86_64 | 11:08 |
| sshnaidm | panda|sick, I see redis-3.2.4-2.el7.x86_64 | 11:08 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add an optional extra node admin ssh key parameter https://review.openstack.org/390854 | 11:08 |
| derekh | panda|sick: shardy Ok, the job I've been following has passed and has the new version of redis, so I think we can stop worrying about the redis problem | 11:08 |
| derekh | shardy: panda|sick http://logs.openstack.org/57/383057/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/583269f/logs/overcloud-controller-0/var/log/host_info.txt.gz | 11:09 |
| shardy | shadower: ^^ I pushed an edit just changing the parameter name | 11:09 |
| panda|sick | sshnaidm: in http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/b67570a/logs/overcloud-controller-0/var/log/host_info.txt.gz it was redis-3.2.4-1.el7.x86_64 | 11:09 |
| sshnaidm | panda|sick, http://logs.openstack.org/92/393192/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9232714/logs/overcloud-controller-0/var/log/host_info.txt.gz | 11:09 |
| shardy | we can see if that picks up the latest redis version | 11:09 |
| panda|sick | sshnaidm: this is the most annoying issue ever | 11:10 |
| shadower | shardy: thanks. So should I merge the two resources, into one, too? | 11:10 |
| panda|sick | derekh: great | 11:10 |
| sshnaidm | panda|sick, I see | 11:11 |
| panda|sick | sshnaidm: but yea, since we are updating, everything should be back to normal again ... | 11:11 |
| openstackgerrit | Robin Cernin proposed openstack/tripleo-validations: Validation stonith device exists in OpenStack Platform HA cluster https://review.openstack.org/360102 | 11:12 |
| sshnaidm | shardy, is "bad status" error handled by somebody? | 11:12 |
| panda|sick | sshnaidm: but then again, this is what I said yesterday, and in some degree friday | 11:12 |
| yolanda | hi, i'm starting to implement blueprint to support full disk images https://blueprints.launchpad.net/tripleo/+spec/support-full-disk-images | 11:12 |
| yolanda | part of the blueprint is to allow to don't upload kernel and vmlinuz images, but I wonder the best way to do it on the client | 11:13 |
| sshnaidm | arxcruz, try to talk to clarkb on #openstack-infra, he'll suggest something if it's possible to do on devstack-gate side | 11:13 |
| arxcruz | sshnaidm: so, the removing some minutes from testenv-client timeout won't work ? | 11:13 |
| yolanda | if just skip the kernel and vmlinuz validation, or add some flag in the overcloud image upload command, such as overcloud image upload --full , to ensure that we pass a full disk image | 11:13 |
| yolanda | EmilienM, or other cores, what are your thoughts? | 11:13 |
| sshnaidm | arxcruz, nope, it's just kills everything, but we need to fetch logs *before* this | 11:14 |
| panda|sick | arxcruz: testenv-client is the one responsible of creating and destroying test environments | 11:14 |
| sshnaidm | arxcruz, maybe it's an option to add callback to testenv-client actually | 11:15 |
| sshnaidm | derekh, ^^ | 11:15 |
| arxcruz | sshnaidm: panda|sick yeah, but there's the trap when testenv-client fails | 11:15 |
| arxcruz | so it runs the postci | 11:15 |
| *** chlong has joined #tripleo | 11:15 | |
| sshnaidm | arxcruz, testenv doesn't run postci | 11:15 |
| panda|sick | starting to get dizzy again, be back later. | 11:16 |
| arxcruz | sshnaidm: yeah, I know, but on toci_instack_ovb.sh has the trap on line 50 that will caught the exit from testenv-client right ? | 11:16 |
| sshnaidm | arxcruz, it traps not testenv-client, but everything that is executed in toci_instack_ovb.sh itself | 11:17 |
| sshnaidm | arxcruz, testenv-client runs toci_instack_ovb.sh, it's level above | 11:18 |
| arxcruz | sshnaidm: oh shit, I though was the opposite way, toci_instack_ovb.sh who calls testenv-client | 11:18 |
| arxcruz | yeah, you're right | 11:19 |
| shardy | shadower: I don't mind tbh, I'm fine to merge it as-is, but if you'd like to do that cleanup then feel free | 11:19 |
| shadower | shardy: just noticed that the edit has a bug anyway, so I'll just resubmit it | 11:20 |
| shadower | it'll make teh change smaller, too | 11:20 |
| derekh | sshnaidm: ya, some kind of post-run handler inside of testenv client could work, this wont work if we hit the ZUUL timout but its a lot better then what we have | 11:20 |
| shardy | shadower: ack, thanks! | 11:20 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-heat-templates: Add an optional extra node admin ssh key parameter https://review.openstack.org/390854 | 11:20 |
| b00tcat | hi, how can I get +workflow here? https://review.openstack.org/#/c/383057/ need to commit something else on this project and would be nice to have this commit merged ^^" | 11:22 |
| yolanda | sshnaidm, i'm getting an error on tripleo-quickstart on my change, but looks unrelated: https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-gate-newton-delorean-full-minimal_pacemaker-18/console.txt.gz | 11:22 |
| yolanda | atal: [172.19.2.71]: FAILED! => {"changed": false, "cmd": ["id", "-u", "stack"], "delta": "0:00:00.003234", "end": "2016-11-07 08:59:45.340246", "failed": true, "rc": 1, "start": "2016-11-07 08:59:45.337012", "stderr": "id: stack: no such user" | 11:22 |
| yolanda | i'm not even updating the teardown on that play, is there some known error? | 11:23 |
| *** rhallisey has joined #tripleo | 11:23 | |
| shardy | b00tcat: approved! | 11:23 |
| shardy | b00tcat: apologies for the delay, we had major CI issues last week which blocked merging things for a few days | 11:24 |
| sshnaidm | yolanda, which patch is it? | 11:24 |
| b00tcat | shardy: thanks, and np ;) | 11:24 |
| yolanda | moving to oooq channel... | 11:24 |
| yolanda | sshnaidm, https://review.openstack.org/#/c/384892/ | 11:24 |
| *** athomas has quit IRC | 11:25 | |
| sshnaidm | yolanda, yeah, there is better :) | 11:25 |
| yolanda | pasted there | 11:25 |
| sshnaidm | shardy, is "bad status" error handled by somebody? | 11:27 |
| jpich | Hey folks, https://review.openstack.org/#/c/390612/ had three +2s before needing a rebase. It's a missed parameter from the passwords migration, if the patch could be reviewed again so it can get backported I'd really appreciate it | 11:28 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-common: Fix the validation ssh keys workflow https://review.openstack.org/391093 | 11:28 |
| hewbrocca | jpich: ooh yeah we need that, it's blocking upgrades, no? | 11:29 |
| jpich | hewbrocca: It's blocking Ceph deployments from the UI, as far as I understand the CLI still hardcodes it and is fine | 11:29 |
| jpich | hewbrocca: I could have missed more recent related bugs though | 11:30 |
| hewbrocca | jpich: ahh, maybe not same issue then | 11:31 |
| *** athomas has joined #tripleo | 11:32 | |
| shardy | sshnaidm: https://review.openstack.org/#/c/393876/ has been proposed by amoralej, I'm reworking that so we set those defaults in puppet-tripleo now | 11:32 |
| sshnaidm | shardy, thanks! | 11:32 |
| shardy | if anyone can reproduce it'd be excellent to get confirmation those timeout increases fix things | 11:33 |
| amoralej | shardy, i'm having doubts about it now. I proposed that assuming that the issue is related to slow hardware in my RDO CI, but now we are hitting it in cases with decent hardware also | 11:34 |
| *** sudswas__ has quit IRC | 11:35 | |
| *** sudipto has quit IRC | 11:35 | |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Increase haproxy timeouts https://review.openstack.org/394378 | 11:36 |
| shardy | amoralej: Yeah I'm not sure either but lets recheck ^^ a few times and see if it helps | 11:36 |
| amoralej | ok | 11:36 |
| shardy | amoralej: I did hit this locally, and the requst failed after far less than a minute | 11:37 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-quickstart: Create directories with root https://review.openstack.org/384892 | 11:37 |
| shardy | not managed to reproduce yet tho :( | 11:37 |
| amoralej | are we hitting it in tripleo gate jobs also? | 11:37 |
| shardy | amoralej: yes | 11:37 |
| shardy | http://logs.openstack.org/54/390854/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/cb9e1ed/console.html#_2016-11-07_09_09_32_326606 | 11:38 |
| amoralej | yeah, that looks similar | 11:39 |
| *** dsariel has joined #tripleo | 11:41 | |
| *** tobias-fiberdata has quit IRC | 11:45 | |
| *** nyechiel has joined #tripleo | 11:45 | |
| *** bfournie has joined #tripleo | 11:46 | |
| *** milan has joined #tripleo | 11:48 | |
| slagle | can i get some reviews on https://review.openstack.org/#/c/393948/ and https://review.openstack.org/#/c/393947/ | 11:53 |
| slagle | they fix a newton bug for manila | 11:53 |
| slagle | marios: fyi, ^^ | 11:53 |
| slagle | i know how important that is to you | 11:53 |
| marios | slagle: ack on a call now | 11:55 |
| marios | slagle: in bit will do | 11:55 |
| slagle | thx :) | 11:55 |
| marios | slagle: appreciate the concern i don't miss anything manila | 11:56 |
| slagle | np. you are the manila man | 11:56 |
| *** dprince has joined #tripleo | 11:57 | |
| thrash | better than being the vanilla man I suppose... | 11:58 |
| bandini | lol | 11:59 |
| jd__ | shardy, hewbrocca so that should fix the problem you saw FWIW https://review.openstack.org/#/c/394387/ :) I'll backport it | 11:59 |
| *** jkilpatr has joined #tripleo | 11:59 | |
| *** pradk has joined #tripleo | 11:59 | |
| *** tdasilva has quit IRC | 12:06 | |
| shardy | jd__: thanks for the quick response! :) | 12:08 |
| *** zoli|lunch is now known as zoli | 12:10 | |
| *** zoli is now known as zoliXXL | 12:10 | |
| *** adarazs is now known as adarazs_lunch | 12:12 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Separate Datastax repository from the Midonet one https://review.openstack.org/383057 | 12:13 |
| *** lucasagomes is now known as lucas-hungry | 12:15 | |
| slagle | shadower: hi, saw your reply about the newton release | 12:16 |
| shadower | hey | 12:16 |
| slagle | shadower: tripleo-validations has not been part of any of our previous newton releases | 12:16 |
| *** tdasilva has joined #tripleo | 12:16 | |
| shadower | slagle: these are tht and tripleo-common patches though | 12:16 |
| slagle | yes, i saw the 1 tht patch | 12:17 |
| hewbrocca | jd__: well that is fantastic | 12:17 |
| slagle | shadower: what is the tripleo-common one? | 12:17 |
| hewbrocca | jd__: Once you backport it to stable I guess we'll pull it in automatically | 12:17 |
| shadower | slagle: https://review.openstack.org/#/c/391093/ it depends on the tht one and it's the actual fix | 12:18 |
| slagle | shadower: ok, thanks, for some reason i thought that was a tripleo-validations patch when I looked earlier | 12:20 |
| shadower | slagle: I was afraid I pasted a wrong link there :-) | 12:21 |
| *** bkopilov has quit IRC | 12:21 | |
| *** mburned_out is now known as mburned | 12:22 | |
| *** anshul has quit IRC | 12:22 | |
| slagle | shadower: the tht one lgtm. for the tripleo-common one, try to wrangle up some reviews | 12:22 |
| shadower | slagle: thanks. d0ugal did one in an earlier patchset | 12:23 |
| dtantsur | folks, can I get W+1 on https://review.openstack.org/#/c/392179/ please? 2x +2 and passed CI | 12:27 |
| *** mgould|afk is now known as mgould | 12:28 | |
| *** anshul has joined #tripleo | 12:29 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Validate JSON parameters https://review.openstack.org/393713 | 12:29 |
| *** dmacpher has joined #tripleo | 12:30 | |
| *** maticue has joined #tripleo | 12:35 | |
| honza | jpich: there was some launchpad-linking weirdness when i published the patch --- i certainly tried to link it :) https://review.openstack.org/#/c/376060/ | 12:38 |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: elements: Add new openssh-server element https://review.openstack.org/389171 | 12:39 |
| *** dougbtv has joined #tripleo | 12:39 | |
| *** shardy has quit IRC | 12:41 | |
| *** rcernin has quit IRC | 12:43 | |
| *** rcernin has joined #tripleo | 12:44 | |
| d0ugal | shadower: do you need a re-review somewhere? | 12:44 |
| shadower | d0ugal: yep, here: https://review.openstack.org/#/c/391093/4 | 12:44 |
| d0ugal | shadower: cool, on it | 12:44 |
| shadower | d0ugal: thanks! | 12:44 |
| *** limao has joined #tripleo | 12:49 | |
| *** bfournie has quit IRC | 12:50 | |
| *** sudipto has joined #tripleo | 12:50 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Ensure we update ceph and composable nodes https://review.openstack.org/392260 | 12:50 |
| *** sudswas__ has joined #tripleo | 12:50 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add local template generation tox task https://review.openstack.org/394410 | 12:53 |
| *** limao has quit IRC | 12:53 | |
| slagle | social: you'll need to restore your newton patch for https://review.openstack.org/#/c/392260/ | 12:55 |
| slagle | and update the commit message | 12:55 |
| jpich | honza: Yeah I saw that, that's how I found the blueprint :) Thank you. There's still a number of manual updates that are required for blueprints in general, e.g. what milestone is targetted or what is the progress like, are all the patches done or is there more to come | 12:56 |
| honza | jpich: i'll see what i can do | 12:56 |
| jpich | honza: In that case though, the first step will be to migrate the blueprint to the tripleo tracker (it's still in the ol' tripleo-ui one for now) | 12:56 |
| jpich | honza: Awesome, thank you :) | 12:57 |
| *** ohamada has quit IRC | 12:59 | |
| jaosorior | slagle: will this commit be included in the release? https://review.openstack.org/#/c/393130/ | 12:59 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Make pep8 task run template generation https://review.openstack.org/394415 | 13:00 |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates: Ensure we update ceph and composable nodes https://review.openstack.org/392259 | 13:00 |
| social | slagle: ^^ | 13:00 |
| *** maeca1 has joined #tripleo | 13:00 | |
| dprince | thrash: https://review.openstack.org/#/c/394410/ | 13:01 |
| dprince | thrash: see my comment there about potentially having the Mistral action use this code instead | 13:01 |
| thrash | dprince: ack | 13:02 |
| *** cylopez has quit IRC | 13:03 | |
| slagle | jaosorior: it is merged in stable/newton, so yes :) | 13:03 |
| slagle | social: thanks | 13:04 |
| thrash | dprince: do you think it fits better in tht, or tripleo-common? | 13:04 |
| dprince | thrash: t-h-t I think | 13:04 |
| dprince | thrash: making heat depend on tripleo-common seems way heavy to me | 13:05 |
| social | slagle: could you have look at https://review.openstack.org/#/c/389830/ and https://review.openstack.org/#/c/392593 ? we need them for updates | 13:05 |
| dprince | thrash: but I could be pursuaded | 13:05 |
| thrash | if we want the mistral action to use this, then tripleo-common would need to depend on tht... | 13:05 |
| d0ugal | thrash: it already does, in a way. | 13:06 |
| thrash | not sure if that's already the case. | 13:06 |
| *** rlandy has joined #tripleo | 13:06 | |
| d0ugal | thrash: it expects the templates to be in /usr/share/ | 13:06 |
| d0ugal | thrash: but never actually states the dep AFAIK | 13:06 |
| *** jayg|g0n3 is now known as jayg | 13:06 | |
| thrash | d0ugal: dprince it would have to become an explicit dep. | 13:08 |
| thrash | I'm still forming my thoughts on it... | 13:08 |
| *** ccamacho is now known as ccamacho|lunch | 13:09 | |
| d0ugal | thrash: Yeah, we should maybe add an explicit dep anyway? not sure. | 13:10 |
| *** zoliXXL is now known as zoli|brb | 13:11 | |
| *** amoralej is now known as amoralej|lunch | 13:11 | |
| jpich | d0ugal: Opened https://bugs.launchpad.net/tripleo/+bug/1639787 for the Mistral env persisting issue, feel free to correct any wrong assumption in there :) | 13:11 |
| openstack | Launchpad bug 1639787 in tripleo "Mistral environment not reset between deployments" [High,Triaged] | 13:11 |
| *** cylopez has joined #tripleo | 13:12 | |
| *** maeca1 has quit IRC | 13:12 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Reset the parameter_defaults between deployments via the CLI https://review.openstack.org/394419 | 13:13 |
| d0ugal | jpich: looks good. Initial patch ^ | 13:13 |
| jpich | Quick! :-o | 13:13 |
| openstackgerrit | Thomas Herve proposed openstack/python-tripleoclient: Use a Zaqar queue to get stack events https://review.openstack.org/394420 | 13:13 |
| d0ugal | jpich: haha, it couldn't be much simpler. | 13:13 |
| arxcruz | sshnaidm: http://logs.openstack.org/09/393309/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/13f8c1c/console.html.gz#_2016-11-04_09_31_30_860581 | 13:14 |
| arxcruz | sshnaidm: this is the output from https://github.com/openstack-infra/tripleo-ci/blob/master/testenv-client#L63 | 13:14 |
| jpich | d0ugal: So we're keeping the passwords around, and 'template'/'root_template' don't matter? | 13:15 |
| *** adarazs_lunch is now known as adarazs | 13:15 | |
| *** limao has joined #tripleo | 13:15 | |
| arxcruz | sshnaidm: and the arguments should be https://github.com/openstack-infra/tripleo-ci/blob/master/testenv-client#L170 | 13:15 |
| d0ugal | jpich: Yeah, we need the passwords, so I am glad they got duplicated. The others get updated anyway when the templates are processed IIRC. | 13:15 |
| d0ugal | jpich: but I need to look into this a wee bit more to be sure. | 13:16 |
| panda|sick | is CI stabily good now ? | 13:16 |
| *** limao_ has joined #tripleo | 13:16 | |
| *** tiswanso has quit IRC | 13:16 | |
| jpich | d0ugal: Cool. Thanks a lot!! I'll give the patch a whirl locally as well | 13:16 |
| panda|sick | stably* | 13:17 |
| sshnaidm | panda|sick, more or less | 13:17 |
| shadower | slagle, shardy: can this get a +A before jenkins changes its mind? https://review.openstack.org/#/c/390854/ | 13:18 |
| sshnaidm | panda|sick, I see a row of successes both for ha and nonha, so it looks good atm | 13:19 |
| panda|sick | shadower: what's the "less" part ? | 13:19 |
| panda|sick | sshnaidm: not shadower | 13:19 |
| slagle | shadower: sure | 13:19 |
| *** limao has quit IRC | 13:19 | |
| sshnaidm | arxcruz, it's everything about working with gearman server, I don't think it should be involved here.. | 13:19 |
| sshnaidm | panda|sick, there is always something, y'know | 13:20 |
| openstackgerrit | Merged openstack/tripleo-common: Do not try "manage" actions on nodes that are not in "enroll" state https://review.openstack.org/392179 | 13:21 |
| *** cylopez has quit IRC | 13:21 | |
| panda|sick | sshnaidm: pessimist :) but yeah, after a week of stacking issues I can see why | 13:21 |
| *** bfournie has joined #tripleo | 13:21 | |
| *** lucas-hungry is now known as lucasagomes | 13:21 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Reset the parameter_defaults between deployments via the CLI https://review.openstack.org/394419 | 13:22 |
| slagle | is there anyone that can look at https://bugs.launchpad.net/tripleo/+bug/1634260 ? | 13:22 |
| openstack | Launchpad bug 1634260 in tripleo "Missing get_file files don't cause deploy failures" [High,Triaged] | 13:22 |
| sshnaidm | panda|sick, https://www.youtube.com/watch?v=wJqguBTZGOg | 13:22 |
| therve | panda|sick, I don't know if the redis failure is gone | 13:22 |
| *** rbowen has joined #tripleo | 13:23 | |
| panda|sick | therve: if it's not, I swear I'll become a farmer. | 13:24 |
| therve | Heh | 13:24 |
| openstackgerrit | yolanda.robla proposed openstack/python-tripleoclient: WIP: Support full disk images in TripleO https://review.openstack.org/394426 | 13:24 |
| therve | http://logs.openstack.org/92/393192/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9232714/ looked good on that front though | 13:24 |
| therve | Still failing for other reasons though | 13:24 |
| *** rodrigods has quit IRC | 13:25 | |
| *** zoli|brb is now known as zoli | 13:25 | |
| *** rodrigods has joined #tripleo | 13:25 | |
| *** zoli is now known as zoliXXL | 13:25 | |
| panda|sick | therve: please recheck, that may have been launched before the new redis package was hitting cdn | 13:26 |
| therve | panda|sick, OK. We'll see with the periodic tomorrow too | 13:26 |
| *** fultonj has joined #tripleo | 13:26 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common: Do not try "manage" actions on nodes that are not in "enroll" state https://review.openstack.org/394429 | 13:26 |
| *** rbrady-afk is now known as rbrady | 13:27 | |
| panda|sick | ok, why .. why: 23297:S 07 Nov 10:10:47.236 # Unable to connect to MASTER: Connection timed out | 13:28 |
| panda|sick | 23297:S 07 Nov 10:10:48.277 * Connecting to MASTER no-such-master:6379 | 13:28 |
| *** tiswanso has joined #tripleo | 13:28 | |
| panda|sick | another redis issue ... | 13:29 |
| therve | panda|sick, That is sad. Doesn't impact the ci result I think though | 13:29 |
| panda|sick | redis is up and running, pcs status is good and there is a redis master | 13:29 |
| panda|sick | gnocchi-metricsd is not getting all the CPU | 13:30 |
| panda|sick | so there's another issue somewhere. | 13:30 |
| therve | Yeah cinder is still returning 500 | 13:31 |
| therve | http://logs.openstack.org/92/393192/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9232714/logs/overcloud-novacompute-0/var/log/nova/nova-compute.txt.gz | 13:31 |
| panda|sick | loadaverage is at 6 on controller | 13:31 |
| therve | OSError: [Errno 12] Cannot allocate memory | 13:32 |
| therve | \o/ /o\ | 13:32 |
| panda|sick | I wonder how the crops are doing this year. | 13:32 |
| hewbrocca | Does anybody here know anything about redis | 13:33 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Include keystone authtoken config in manila-share service https://review.openstack.org/393947 | 13:34 |
| panda|sick | I don't see any process taking all the CPU this time in host-info | 13:34 |
| beagles | hewbrocca: only that it seems to be reaching legendary status as a trouble maker | 13:34 |
| therve | panda|sick, http://logs.openstack.org/92/393192/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9232714/logs/overcloud-controller-0/var/log/cinder/cinder-api.txt.gz#_2016-11-07_10_04_39_884 is weird | 13:35 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Move db settings from manila-api to manila-base https://review.openstack.org/393948 | 13:35 |
| therve | Seems to happen after the OOM error | 13:36 |
| *** pradk has quit IRC | 13:37 | |
| *** jprovazn has quit IRC | 13:37 | |
| panda|sick | the max I see in MEM utilization in ps is 2%. we need a live deploy to look at | 13:38 |
| therve | panda|sick, Do we store ps stats of overcloud nodes? | 13:39 |
| *** cylopez has joined #tripleo | 13:39 | |
| panda|sick | therve: we store the output of a single ps command, at the end of the test AFAIK | 13:40 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Refactor addWorkflowExecution https://review.openstack.org/394434 | 13:40 |
| *** links has quit IRC | 13:40 | |
| therve | Ah yeah host_info | 13:40 |
| therve | free 151M | 13:40 |
| therve | That looks a tad small | 13:40 |
| *** jcoufal has joined #tripleo | 13:41 | |
| *** tiswanso has quit IRC | 13:43 | |
| panda|sick | I'll try to launch a deploy in rh1, it will take a while. | 13:43 |
| *** saneax-_-|AFK is now known as saneax | 13:43 | |
| panda|sick | but I'm ot sure I have enough energy today to do debugging. | 13:44 |
| therve | We can tweak down httpd config. cinder as 2 proc/2 threads, and horizong 10/3 | 13:44 |
| therve | 3/10 | 13:44 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move per role Services defaults into environment file https://review.openstack.org/391064 | 13:45 |
| *** jpena is now known as jpena|lunch | 13:49 | |
| panda|sick | ooohh, metricd in controller-0 is clean, *BUT* in controller-1 is again spinning like crazy | 13:51 |
| panda|sick | but it seems to sabilize after a while | 13:52 |
| *** bkopilov has joined #tripleo | 13:52 | |
| *** amoralej|lunch is now known as amoralej | 13:53 | |
| jrist | anyone have trouble logging in with SSL firefox to the UI? | 13:54 |
| *** tzumainn has joined #tripleo | 13:56 | |
| *** Goneri has joined #tripleo | 13:56 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Include keystone authtoken config in manila-share service https://review.openstack.org/394439 | 13:56 |
| *** sshnaidm is now known as sshnaidm|afk | 13:57 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Move db settings from manila-api to manila-base https://review.openstack.org/394440 | 13:58 |
| *** Guest71310 has quit IRC | 13:58 | |
| *** skramaja_ has joined #tripleo | 14:00 | |
| *** skramaja has quit IRC | 14:00 | |
| weshay | panda|sick, you were referring to #15 on the tripleo-ci-status etherpad? | 14:02 |
| panda|sick | live deploy lainched, it will take a while .. | 14:02 |
| *** saneax is now known as saneax-_-|AFK | 14:03 | |
| jpich | jrist: Yeah, because of the self-signed certificate. Unfortunately it looks like it needs to be accepted manually not just for the login page, but also for every URL+port combination too... | 14:03 |
| slagle | therve: any chance you could look at https://bugs.launchpad.net/tripleo/+bug/1634260 ? | 14:03 |
| openstack | Launchpad bug 1634260 in tripleo "Missing get_file files don't cause deploy failures" [High,Triaged] | 14:03 |
| jrist | jpich: :o | 14:03 |
| slagle | or d0ugal perhaps: https://bugs.launchpad.net/tripleo/+bug/1634260 | 14:04 |
| slagle | the problem is in the tripleoclient template processing it looks like | 14:04 |
| jrist | jpich: so we have to go through with each URL/port and accept? | 14:04 |
| jpich | jrist: I wonder if we could add an option to accept self-signed certs to the config file. I don't know how we talk to the services from the UI code, but jtomasek / florianf / honza might have an idea if it'd be possible to implement? | 14:05 |
| panda|sick | weshay: it was general agreement that traces 1 and 2 in that bug were caused by the delays caused by gnocchi-metricsd eating all the CPU | 14:05 |
| *** lblanchard has joined #tripleo | 14:05 | |
| *** cylopez has quit IRC | 14:06 | |
| jpich | jrist: As a workaround for now, it seems like it, or maybe the cert can be added manually to the firefox cert management bits * pokes around * | 14:06 |
| jrist | jpich: interesting. I know there are lots of ways to do it with react-native | 14:06 |
| *** skramaja has joined #tripleo | 14:06 | |
| jpich | jrist: Do you know if there's a bug open for this? Would be good to document workaround(s) there | 14:06 |
| *** skramaja_ has quit IRC | 14:06 | |
| weshay | panda|sick, wonder if prad (pradeep) can help w/ that | 14:06 |
| jrist | jpich: I don't know, just an email from Udi | 14:06 |
| jtomasek | jrist, jpich: I've tried to look for such option in reqwest which is what we use for ajax requests, but I did not find anything like it | 14:08 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move per role Services defaults into environment file https://review.openstack.org/391064 | 14:08 |
| jtomasek | jrist, jpich: why are the certs self signed? isn't that insecure? does the ssl make sense when we then work it around with such option? | 14:09 |
| jpich | jtomasek, jrist: I'm going to open a bug so we can track this information | 14:09 |
| jrist | thanks jpich | 14:09 |
| therve | slagle, Sure looking | 14:09 |
| jrist | jtomasek: lol | 14:09 |
| jrist | jtomasek: technically no | 14:09 |
| jrist | it doesn't make sense | 14:09 |
| jtomasek | jrist: ... | 14:09 |
| jrist | I wonder if we can easily do 'letsencrypt' certificates? | 14:10 |
| beekneemech | jrist: jtomasek: You should talk to jaosorior about this. | 14:10 |
| d0ugal | slagle: I'll take a look and see if I can figure it out | 14:10 |
| jtomasek | jrist: probably, I thought dtrainor was looking into making ssl work | 14:10 |
| *** beekneemech is now known as bnemec | 14:10 | |
| panda|sick | weshay: too much guess work on the logs, when the live env is ready I will start drawing some conclusion again .. *if* i'll be able to think straight, my lunch was a glass of water with sugar. | 14:10 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move per role Services defaults into environment file https://review.openstack.org/394442 | 14:10 |
| weshay | panda|sick, feel free to hand off to sshnaidm|afk and get some sleep | 14:11 |
| slagle | d0ugal: therve : thanks. if only one of you wants to look, that's fine :) just wanted to make sure we have someone on it | 14:11 |
| thrash | dprince: d0ugal so, indirectly, python-tripleoclient -> instack-undercloud -> openstack-tripleo-heat-templates | 14:11 |
| thrash | that's the current deps | 14:11 |
| d0ugal | slagle: oh, I didn't see therve has replied. I'll wait a bit and see how that goes :) | 14:12 |
| *** Guest71310 has joined #tripleo | 14:12 | |
| d0ugal | thrash: hah, nice. | 14:12 |
| thrash | so, shouldn't be a stretch for tripleo-common to depend on tripleo-heat-templates | 14:12 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add an optional extra node admin ssh key parameter https://review.openstack.org/390854 | 14:12 |
| panda|sick | sshnaidm|afk: there is a test env deploying HA in rh1, let's see what happens there | 14:12 |
| panda|sick | sshnaidm|afk: I'll ping you when it's ready | 14:12 |
| thrash | d0ugal: and same for tripleo-common -> instack-undercloud -> openstack-tripleo-heat-templates | 14:12 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Reset the parameter_defaults between deployments via the CLI https://review.openstack.org/394419 | 14:14 |
| slagle | shadower: https://review.openstack.org/#/c/390854/ merged. please backport it | 14:14 |
| shadower | slagle: just got the email. I'm on it. Thanks | 14:14 |
| d0ugal | jpich: Not sure if you started testing https://review.openstack.org/#/c/394419/ | 14:15 |
| *** jprovazn has joined #tripleo | 14:15 | |
| d0ugal | jpich: but I just updated it because I done something silly | 14:15 |
| jpich | d0ugal: No, I hadn't (sorry) - thanks for the update! | 14:15 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-heat-templates: Add an optional extra node admin ssh key parameter https://review.openstack.org/394446 | 14:15 |
| d0ugal | jpich: probably a good thing, it would have failed anyway (I found the error in CI) | 14:15 |
| shadower | slagle: ^ | 14:16 |
| *** noslzzp has quit IRC | 14:16 | |
| slagle | shadower: thanks | 14:17 |
| *** noslzzp has joined #tripleo | 14:17 | |
| *** Guest71310 has quit IRC | 14:18 | |
| *** tesseract has joined #tripleo | 14:19 | |
| hewbrocca | weshay: FWIW we already have a patch from jd__ to fix the gnocchi issue | 14:19 |
| *** tesseract is now known as Guest41117 | 14:19 | |
| hewbrocca | and it has been backported, or is in the process of that | 14:19 |
| hewbrocca | I don't know what else if anything needs to happen to get it pulled into RDO newton | 14:19 |
| hewbrocca | However | 14:19 |
| hewbrocca | it's not clear that fixing gnocchi alone is sufficient | 14:20 |
| *** anshul has quit IRC | 14:20 | |
| weshay | hewbrocca, rockin | 14:20 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Configure civetweb bind address in brackets when IPv6 https://review.openstack.org/390569 | 14:20 |
| *** tiswanso has joined #tripleo | 14:21 | |
| *** Guest41117 has quit IRC | 14:22 | |
| *** dtrainor has joined #tripleo | 14:25 | |
| jpich | jrist, jtomasek: I opened https://bugs.launchpad.net/tripleo/+bug/1639807 . jtomasek could you add the information about reqwest there? I set the severity as high but I don't know how common undercloud deployments with self-signed certificates are, if most folks use their own maybe it's not as bad. I'll see if there might be a "one-step manual workaround" and add it to the bug | 14:27 |
| openstack | Launchpad bug 1639807 in tripleo "Can't login to the UI with SSL when using Firefox" [High,Triaged] | 14:27 |
| *** shardy has joined #tripleo | 14:27 | |
| jtomasek | jpich: thanks, I will | 14:27 |
| *** tesseract- has joined #tripleo | 14:27 | |
| *** anshul has joined #tripleo | 14:30 | |
| *** morazi has joined #tripleo | 14:30 | |
| *** ohamada has joined #tripleo | 14:30 | |
| *** sshnaidm|afk is now known as sshnaidm | 14:32 | |
| *** jaosorior has quit IRC | 14:32 | |
| *** jaosorior has joined #tripleo | 14:33 | |
| sshnaidm | panda|sick, ok | 14:33 |
| *** chlong has quit IRC | 14:34 | |
| weshay | sshnaidm, fyi https://review.openstack.org/#/c/394387/ | 14:34 |
| sshnaidm | weshay, yeah, I know :) | 14:35 |
| weshay | k | 14:35 |
| *** hoobaman has joined #tripleo | 14:36 | |
| hoobaman | hi | 14:36 |
| sshnaidm | panda|sick, what is the current issue you check? the redis failure? | 14:36 |
| hoobaman | currently deploying mitaka | 14:36 |
| hoobaman | however the parameter "CloudDomain" does not seem to work | 14:36 |
| hoobaman | it is normally used to provide your overcloud hosts with a valid fqdn | 14:37 |
| hoobaman | instead it gives us localdomain | 14:37 |
| *** morazi has quit IRC | 14:37 | |
| hoobaman | Is this issue known? any workaround available? | 14:37 |
| bnemec | hoobaman: Did you set the domain on the undercloud neutron as well? I believe CloudDomain just ensures the right domain is added to things like the hosts file on overcloud nodes. | 14:38 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Updated Nuage neutron plugin name https://review.openstack.org/393892 | 14:38 |
| shardy | hoobaman: see https://bugs.launchpad.net/tripleo/+bug/1581472 | 14:38 |
| openstack | Launchpad bug 1581472 in tripleo "CloudDomain doesn't correctly set hostname" [High,Triaged] - Assigned to Giulio Fidente (gfidente) | 14:38 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Clone tripleo-ci in the undercloud https://review.openstack.org/380346 | 14:38 |
| shardy | there is a workaround, which is to set dhcp_domain in nova.conf on the undercloud | 14:38 |
| bnemec | (I think it's neutron.conf) | 14:39 |
| shardy | bnemec: maybe so, perhaps someone can update the bug if that's confirmed, I thought I set it in nova.conf but could be mistaken | 14:39 |
| *** masco has quit IRC | 14:40 | |
| bnemec | I could be wrong, but Nova shouldn't be setting DHCP parameters. | 14:40 |
| hoobaman | bnemec: thx for your reply. Do you mean domain in neutron subnet-update? | 14:40 |
| hoobaman | shardy: thx | 14:40 |
| shardy | #dhcp_domain=novalocal | 14:40 |
| shardy | it's in nova.conf AFAICS | 14:41 |
| *** lmiccini has quit IRC | 14:41 | |
| *** tbonds has joined #tripleo | 14:41 | |
| bnemec | Oh good, they both have settings for this. That's not confusing at all. | 14:42 |
| *** ccamacho|lunch is now known as ccamacho | 14:42 | |
| bnemec | /sarcasm | 14:42 |
| tzumainn | d0ugal, jtomasek hI! not sure who to ask this question to, but here it is: if I use create_default_deployment_plan, should the resulting plan be deployable with no further tinkering? or is tinkering required? | 14:43 |
| hoobaman | shardy:thx | 14:43 |
| hoobaman | bnemec:thx | 14:43 |
| jtomasek | tzumainn: it should afaik (if you have the nodes available) | 14:43 |
| tzumainn | d0ugal, jtomasek, and second question - where do the templates from the default deployment plan come from? I just want to know the baseline used in case I need to create a plan with custom templates | 14:43 |
| *** lmiccini has joined #tripleo | 14:44 | |
| jtomasek | tzumainn: those are templates installed at /usr/share/openstack-tripleo-heat-templates | 14:44 |
| tzumainn | jtomasek, okay, great - thanks! | 14:45 |
| openstackgerrit | Merged openstack/tripleo-common: Add CephClusterFSID to generated passwords https://review.openstack.org/390612 | 14:47 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Change nova ram_allocation_ratio to match puppet-nova https://review.openstack.org/392108 | 14:48 |
| openstackgerrit | Merged openstack/tripleo-ui: Validate JSON parameters https://review.openstack.org/393713 | 14:48 |
| d0ugal | tzumainn: I think it should be possible to deploy it at that point. | 14:48 |
| d0ugal | tzumainn: if not, that is a bug :) | 14:49 |
| *** morazi has joined #tripleo | 14:51 | |
| *** ebalduf has joined #tripleo | 14:52 | |
| openstackgerrit | Thomas Herve proposed openstack/python-tripleoclient: Fix handling of missing environment files https://review.openstack.org/394471 | 14:54 |
| therve | slagle, d0ugal ^^^ | 14:54 |
| tzumainn | d0ugal, \o/ awesome! | 14:54 |
| therve | This may be wrong, but this has a test at least :) | 14:54 |
| d0ugal | tzumainn: jtomasek or jpich will know if there are any required params as I think they have been doing GUI testing | 14:54 |
| d0ugal | therve: looking | 14:54 |
| tzumainn | d0ugal, ah, okay, thanks! | 14:55 |
| slagle | therve: thanks! | 14:56 |
| slagle | shardy: can you review therve's fix? https://review.openstack.org/#/c/394471/ | 14:57 |
| shardy | slagle: will do | 14:57 |
| slagle | thx | 14:59 |
| *** saneax-_-|AFK is now known as saneax | 15:00 | |
| *** sudswas__ has quit IRC | 15:01 | |
| *** sudipto has quit IRC | 15:01 | |
| shardy | ouch, yeah AFAICS the fix is good, thanks therve | 15:01 |
| tzumainn | jtomasek, jpich, I'll be testing this out, but just let me know if you guys think of any required params that need to be set | 15:02 |
| therve | shardy, No problem :). The flush/safe_dump looks clear, I was wondering about the naming (though it shouldn't matter too much) | 15:02 |
| jpich | tzumainn: I think the "count" variables (controllercount, etc) might be needed still somewhere, but I could be misunderstanding | 15:03 |
| *** jpena|lunch is now known as jpena | 15:04 | |
| shardy | Yeah I think it probably doesn't matter too much but this way is more consistent with the file containing env_map | 15:04 |
| tzumainn | jpich, okay, thanks! I'll test it out | 15:04 |
| tzumainn | jpich, would you happen to have an overcloud deployed through the UI available somewhere? if so, would it be possible to paste 'heat stack-show' somewhere so I can compare the parameters against my own? | 15:05 |
| tzumainn | if not, don't worry about it! | 15:05 |
| d0ugal | jpich: I think the counts are all optional - by default you should get 1 control and 1 compute I think | 15:06 |
| jpich | tzumainn: I think the mistral environment might be the relevant bit here, in meetings right now, will let you know after :) | 15:06 |
| tzumainn | jpich, okay, thanks! | 15:07 |
| shardy | tzumainn: you may find "openstack stack environment show overcloud" useful if you want to compare heat environments without introspecting all the stacks | 15:07 |
| jaosorior | bnemec: jrist , what's up? | 15:07 |
| tzumainn | shardy, ahh, thanks! | 15:07 |
| *** jaosorior is now known as jaosorior_sick | 15:09 | |
| *** pblaho has joined #tripleo | 15:10 | |
| Ng | would any kind core folks like to cast an eye on https://review.openstack.org/#/c/393673/1 ? :) | 15:10 |
| jaosorior_sick | jrist, bnemec we're not doing letsencrypt certs, because we would need the undercloud to be publicly accessible (which we can't asure) and the deployer would also need to own a domain (which is no big deal, and we already can support hostnames instead of IPs in the endpoints) | 15:10 |
| jpich | jaosorior_sick: I think the context for SSL is https://bugs.launchpad.net/tripleo/+bug/1639807 at the moment, the self-signed stuff is giving us a couple of issues on the Firefox side | 15:11 |
| openstack | Launchpad bug 1639807 in tripleo "Can't login to the UI with SSL when using Firefox" [High,Triaged] | 15:11 |
| bnemec | jaosorior_sick: Okay, thanks. I knew you had looked into it, so when it came up today I wanted to make sure you were in the loop. | 15:11 |
| *** cylopez has joined #tripleo | 15:12 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Neutron L3 service cleanups for hiera json hook https://review.openstack.org/393262 | 15:13 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Horizon service cleanups for hiera json hook https://review.openstack.org/393258 | 15:13 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Hiera optimization: use a new hiera hook https://review.openstack.org/384757 | 15:13 |
| jpich | jaosorior_sick: Looks like dtrainor is gonna be on it so maybe don't worry about it if you're sick - or maybe it's a different issue you were talking about, sorry about that | 15:13 |
| jaosorior_sick | jpich: well, it's actually quite a normal thing that services that offer SSL by default use self-signed certs (it's the same with FreeIPA). We can't asure that everyone has access to a CA. and in the case of the undercloud deployment, we can't asure that there is external access that the letsencrypt service will use to asure that you own the domain | 15:13 |
| jaosorior_sick | jpich: if someone wants to add letsencrypt support and add it as an option I'm cool with that. I'm just explaining why things are the way they are | 15:14 |
| jaosorior_sick | anyway, Imma go back to the couch, feeling a bit feverish. lets talk about it tomorrow. | 15:14 |
| jpich | jaosorior_sick: Yeah, apparently it's more due to lack of information in the cert itself, looks like what I wrote is mostly wrong and dtrainor has different, better ideas on fixing it :) | 15:14 |
| jpich | jaosorior_sick: Absolutely, take care | 15:14 |
| dtrainor | don't blame me yet until i get this test done :) | 15:14 |
| trozet | shardy: hi. Remember this change https://review.openstack.org/#/c/382926/1/network/service_net_map.j2.yaml ? It looks like something doesn't work when you use network isolation and specify internal_api network...and I cannot figure out what it is | 15:16 |
| trozet | shardy: doing a getattr on the ServiceNetMap for OpenDaylightApiNetwork returns nothing | 15:16 |
| jpich | dtrainor: Thanks for all the information on that thread, I am now full of hope :) | 15:17 |
| dtrainor | I'll have it docu'd in the launchbad bug shortly, just want to make sure i have enough info on it | 15:17 |
| dtrainor | *in it | 15:17 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Make pep8 task run template generation https://review.openstack.org/394415 | 15:18 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add local template generation tox task https://review.openstack.org/394410 | 15:18 |
| *** anshul has quit IRC | 15:19 | |
| jpich | dtrainor: Thank you! | 15:20 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Change nova ram_allocation_ratio to match puppet-nova https://review.openstack.org/394488 | 15:20 |
| *** hoobaman has quit IRC | 15:20 | |
| *** aufi has quit IRC | 15:24 | |
| openstackgerrit | Zane Bitter proposed openstack/tripleo-heat-templates: Configure region correctly for heat-api-cfn service https://review.openstack.org/386705 | 15:27 |
| slagle | can we get another +2 on this backport: https://review.openstack.org/#/c/392259/ ? | 15:28 |
| *** pradk has joined #tripleo | 15:28 | |
| shardy | slagle: done | 15:30 |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Add CephClusterFSID to generated passwords https://review.openstack.org/394493 | 15:31 |
| *** yamahata has joined #tripleo | 15:33 | |
| *** links has joined #tripleo | 15:34 | |
| *** owalsh_ has joined #tripleo | 15:34 | |
| *** numans has quit IRC | 15:36 | |
| *** absubram has joined #tripleo | 15:36 | |
| *** owalsh has quit IRC | 15:38 | |
| *** coolsvap has quit IRC | 15:42 | |
| *** mwhahaha has quit IRC | 15:42 | |
| *** igorbelikov has quit IRC | 15:42 | |
| *** gregwork has quit IRC | 15:42 | |
| *** florianf has quit IRC | 15:43 | |
| *** Ng has quit IRC | 15:43 | |
| *** fungi has quit IRC | 15:43 | |
| *** igorbelikov has joined #tripleo | 15:44 | |
| *** mwhahaha has joined #tripleo | 15:45 | |
| *** rcarrillocruz has quit IRC | 15:45 | |
| *** coolsvap has joined #tripleo | 15:45 | |
| *** Ng has joined #tripleo | 15:46 | |
| *** ChanServ sets mode: +v Ng | 15:46 | |
| *** gregwork has joined #tripleo | 15:46 | |
| shadower | shardy: thanks for your replies on https://review.openstack.org/#/c/393448/ :-) | 15:49 |
| shardy | shadower: np, thanks for the feedback | 15:50 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Properly reload kvm module when trying to set up nested virtualization https://review.openstack.org/386012 | 15:52 |
| *** rcarrillocruz has joined #tripleo | 15:52 | |
| *** radeks has joined #tripleo | 15:54 | |
| *** radeksmg has joined #tripleo | 15:54 | |
| *** florianf has joined #tripleo | 15:54 | |
| *** radeks has quit IRC | 15:54 | |
| *** rcernin has quit IRC | 15:55 | |
| *** fungi has joined #tripleo | 15:55 | |
| bandini | jd__, pradk: I just filed https://bugs.launchpad.net/tripleo/+bug/1639842 are missing any tunable that slows things down or is that rate expected? | 15:55 |
| openstack | Launchpad bug 1639842 in tripleo "Newton - gnocchi-metricd is hammering redis" [Undecided,New] | 15:55 |
| pradk | bandini, was it because redis was down? we had a bug on that | 15:56 |
| pradk | bandini, or redis is up and still getting hammered? | 15:56 |
| bandini | pradk: nope redis is up, that is why I filed a new one | 15:56 |
| jd__ | bandini: metricd uses a lot redis but I am surprised it keeps "reconencting" | 15:56 |
| bandini | jd__, pradk: let me investigate a bit more | 15:57 |
| jd__ | ok :) | 15:58 |
| jrist | jaosorior_sick: noted | 15:58 |
| jrist | jaosorior_sick: thanks | 15:58 |
| jrist | jaosorior_sick: get well! | 15:58 |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-heat-templates: Add constraint for ControllerCount https://review.openstack.org/342272 | 16:00 |
| d0ugal | Mistral meeting time in #openstack-meeting for those interested. | 16:02 |
| d0ugal | rbrady: ^ | 16:02 |
| *** yamahata has quit IRC | 16:06 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Skip new ansible-lint rule until fixing the roles https://review.openstack.org/394506 | 16:06 |
| openstackgerrit | Thomas Herve proposed openstack/python-tripleoclient: Fix handling of missing environment files https://review.openstack.org/394471 | 16:07 |
| *** owalsh_ is now known as owalsh | 16:08 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Move per role Services defaults into environment file https://review.openstack.org/391064 | 16:09 |
| weshay | sshnaidm, you have a minute? | 16:09 |
| sshnaidm | weshay, yep | 16:09 |
| weshay | sshnaidm, join me in bluejeans for a minute | 16:09 |
| *** ebarrera has quit IRC | 16:10 | |
| *** chandankumar has quit IRC | 16:11 | |
| *** maeca1 has joined #tripleo | 16:12 | |
| *** pcaruana has quit IRC | 16:12 | |
| weshay | sshnaidm, https://review.openstack.org/#/c/390569/5 | 16:15 |
| panda|sick | my deployment failed for a different reason than the one we're seeing on the gates. Waiting for postci to complete ... but it's very very slow generally | 16:15 |
| *** bana_k has joined #tripleo | 16:16 | |
| weshay | sshnaidm, https://review.openstack.org/#/c/386080/ | 16:16 |
| weshay | sshnaidm, https://review.openstack.org/#/c/394387/ | 16:17 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Ensure we update ceph and composable nodes https://review.openstack.org/392259 | 16:18 |
| dtantsur | folks, may I get reviews on https://review.openstack.org/#/c/392653/ and https://review.openstack.org/#/c/392148/ please? | 16:19 |
| dtantsur | these are the problems with our workflows that seem to affect real people already :) | 16:19 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Add ability to deploy an overcloud with ssl https://review.openstack.org/382830 | 16:20 |
| *** ealcaniz has quit IRC | 16:22 | |
| *** tremble has quit IRC | 16:24 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Use mistral action to create new containers https://review.openstack.org/392189 | 16:24 |
| openstackgerrit | Martin Mágr proposed openstack/python-tripleoclient: Use correct region value https://review.openstack.org/394515 | 16:25 |
| *** nyechiel has quit IRC | 16:29 | |
| *** oshvartz has quit IRC | 16:29 | |
| *** fpan has quit IRC | 16:30 | |
| *** rhallisey has quit IRC | 16:30 | |
| *** rhallisey has joined #tripleo | 16:30 | |
| sshnaidm | panda|sick, postci on dev env doesn't work so good, you can stop it (if you're on dev env) | 16:31 |
| *** tesseract- has quit IRC | 16:31 | |
| panda|sick | sshnaidm: actually, it was only my connetion that froze | 16:31 |
| jistr | ccamacho: hey looking at https://review.openstack.org/#/c/393644/ i think you might hit what we noticed earlier with bandini and jaosorior that the ControllerPostPuppet (where the restart script belongs too) doesn't get executed at all | 16:33 |
| *** limao_ has quit IRC | 16:33 | |
| jistr | ccamacho: i might have a suggestion how to tackle that, will comment on the review | 16:34 |
| *** cwolferh has joined #tripleo | 16:34 | |
| *** chem has quit IRC | 16:35 | |
| ccamacho | hey jistr thanks man! Im also hitting https://bugs.launchpad.net/tripleo/+bug/1639302 when updating the stack.. All feedback is welcome | 16:35 |
| openstack | Launchpad bug 1639302 in tripleo "Started Mistral Workflow fails due to malformed template" [High,Confirmed] | 16:35 |
| *** fpan has joined #tripleo | 16:37 | |
| *** paramite has quit IRC | 16:37 | |
| *** fpan has quit IRC | 16:38 | |
| *** fpan has joined #tripleo | 16:38 | |
| *** ebarrera has joined #tripleo | 16:39 | |
| *** chem has joined #tripleo | 16:40 | |
| panda|sick | so, I see all controllers with loadavg > 10, beam is usually on top of top, and I think the queue is flooded with requests. Also redis on slaves are unable to contact master | 16:40 |
| *** paramite has joined #tripleo | 16:41 | |
| openstackgerrit | Merged openstack/tripleo-common: Do not try "manage" actions on nodes that are not in "enroll" state https://review.openstack.org/394429 | 16:43 |
| sshnaidm | panda|sick, do I understand right it's swift that going crazy with reconnections to redis? | 16:44 |
| panda|sick | sshnaidm: It's possible, unfortunately things are calming down here, load averages are dropping | 16:45 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Skip new ansible-lint rule until fixing the roles https://review.openstack.org/394506 | 16:46 |
| sshnaidm | panda|sick, hmm... | 16:47 |
| sshnaidm | panda|sick, I see two controllers have the same ip | 16:47 |
| panda|sick | sshnaidm: the weird thing is that loadavg is at 10, cpu is at 60% all the time, but there isn't a single process that is taking all the cpu | 16:47 |
| sshnaidm | panda|sick, seems like HA fail | 16:47 |
| *** paramite has quit IRC | 16:48 | |
| panda|sick | sshnaidm: which IP | 16:48 |
| sshnaidm | panda|sick, yeah, also was searching in top.. | 16:48 |
| *** zoliXXL is now known as zoli|gone | 16:51 | |
| sshnaidm | panda|sick, no, sorry, wrong alert | 16:51 |
| sshnaidm | panda|sick, was looking for IP that swift search for redis in | 16:51 |
| *** jkilpatr_ has joined #tripleo | 16:52 | |
| *** aufi has joined #tripleo | 16:53 | |
| *** michapma_alt has quit IRC | 16:54 | |
| *** bana_k has quit IRC | 16:54 | |
| *** jkilpatr has quit IRC | 16:55 | |
| *** stendulker has joined #tripleo | 16:56 | |
| *** fragatina has joined #tripleo | 16:59 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Ensure heat-domain hiera is in nodes that contain keystone https://review.openstack.org/392519 | 16:59 |
| *** nyechiel has joined #tripleo | 17:00 | |
| *** maeca1 has quit IRC | 17:00 | |
| panda|sick | sshnaidm: I keep seeing Nov 7 17:00:28 localhost proxy-server: STDERR: WARNING:ceilometermiddleware.swift:Send queue FULL: Event dc8dc681-d50c-5476-ac35-05a54f1396c3 not added (txn: txa5336d17dd4d418686de5-005820b32c) (client_ip: 172.18.0.16) | 17:00 |
| panda|sick | Nov 7 17:00:28 localhost proxy-server: 172.18.0.16 172.18.0.16 07/Nov/2016/17/00/28 GET /v1/AUTH_0ce54ff3166a4f639721fda10b83f1d8/measure%3Fformat%3Djson%26limit%3D64%26delimiter%3D/ HTTP/1.0 200 - python-swiftclient-3.1.0 3fa69b2072324712... - 2 - txa5336d17dd4d418686de5-005820b32c - 0.0374 - - 1478538028.511590958 1478538028.549038887 0 | 17:00 |
| *** fragatina has quit IRC | 17:01 | |
| panda|sick | is ceilometer bombarding swift ? | 17:01 |
| *** ealcaniz has joined #tripleo | 17:01 | |
| *** fragatina has joined #tripleo | 17:01 | |
| openstackgerrit | Merged openstack/instack-undercloud: Disable Swift auditors and replicators on the undercloud https://review.openstack.org/389638 | 17:02 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Unset Keystone public_endpoint https://review.openstack.org/386080 | 17:02 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add missing Barbican endpoint from tls-everywhere environment https://review.openstack.org/392473 | 17:02 |
| sshnaidm | panda|sick, I don't know, how are they connected? Does ceilometer keeps its stats in swift? | 17:02 |
| hewbrocca | jd__: ^^^ | 17:02 |
| *** radeksmg has quit IRC | 17:04 | |
| *** abregman_ has quit IRC | 17:04 | |
| sshnaidm | panda|sick, do you have the live setup? | 17:06 |
| panda|sick | sshnaidm: yes | 17:08 |
| *** owalsh has quit IRC | 17:08 | |
| sshnaidm | panda|sick, can you try just the tcp connectivity to redis port from controllers? | 17:08 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add simple-playbook element https://review.openstack.org/385608 | 17:08 |
| sshnaidm | panda|sick, and to virtual ip, in my logs it's 172.17.0.12 | 17:09 |
| *** rbrady is now known as rbrady-afk | 17:11 | |
| panda|sick | from controller-0 to controller-2(master) it's working. But redis log says unable to contact master. | 17:13 |
| jd__ | panda|sick: Gnocchi stores data in Swift, and Ceilometer stores its data in Gnocchi | 17:13 |
| hewbrocca | so if gnocchi is having trouble reaching swift, you'd have a problem | 17:14 |
| *** jlinkes has quit IRC | 17:14 | |
| sshnaidm | jd__, and where is redis in this matryoshka? does swift send messages to it..? | 17:16 |
| panda|sick | also, what happens if the redis servers are unable to sync with the master ? | 17:17 |
| sshnaidm | panda|sick, and what is I/O CPU on these hosts? | 17:17 |
| jd__ | sshnaidm: no, Redis is used by Ceilometer and Gnocchi for caching and/or coordination | 17:18 |
| panda|sick | sshnaidm: %Cpu(s): 65.3 us, 19.0 sy, 0.0 ni, 15.0 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st | 17:18 |
| openstackgerrit | Merged openstack/tripleo-common: Power off new nodes when making them available, not right after enrolling https://review.openstack.org/392148 | 17:18 |
| panda|sick | sshnaidm: almost nothing. | 17:18 |
| *** owalsh has joined #tripleo | 17:18 | |
| sshnaidm | panda|sick, shouldn't it consider itself as a master then..? | 17:18 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common: Power off new nodes when making them available, not right after enrolling https://review.openstack.org/394541 | 17:19 |
| *** links has quit IRC | 17:20 | |
| morazi | pradk, ^^ any thoughts on that connectivity bit re: gnocchi/swift/ceilo ? | 17:20 |
| pradk | reading up | 17:21 |
| akrivoka | florianf: what value to you have listed for swift in tripleo_ui_config.js ? | 17:22 |
| akrivoka | florianf: (trying to test your container patch) | 17:22 |
| *** panda|sick is now known as panda|weak | 17:22 | |
| *** ebarrera has quit IRC | 17:22 | |
| pradk | hmm so regarding ceilometermiddleware and swift, we recently added ceilometer to the swift pipeline | 17:23 |
| pradk | not sure if that has any implications we're seeing here | 17:23 |
| florianf | akrivoka: I use the output of `openstack catalog show swift`, but with the hostname that the UI is using | 17:24 |
| pradk | panda|weak, can you paste me the swift-proxy.conf | 17:24 |
| *** rasca has quit IRC | 17:26 | |
| *** bana_k has joined #tripleo | 17:26 | |
| panda|weak | pradk: You mean swift/proxy-server.conf | 17:26 |
| panda|weak | pradk: ? | 17:26 |
| pradk | panda|weak, yes | 17:27 |
| pradk | panda|weak, I keep seeing Nov 7 17:00:28 localhost proxy-server: STDERR: WARNING:ceilometermiddleware.swift:Send queue FULL: Event dc8dc681-d50c-5476-ac35-05a54f1396c3 not added (txn: txa5336d17dd4d418686de5-005820b32c) (client_ip: 172.18.0.16) | 17:27 |
| pradk | that concerns me | 17:28 |
| *** numans has joined #tripleo | 17:28 | |
| akrivoka | florianf: thanks! | 17:28 |
| panda|weak | pradk: http://paste.openstack.org/show/588293/ | 17:28 |
| *** fragatina has quit IRC | 17:28 | |
| *** nyechiel has quit IRC | 17:28 | |
| pradk | jd__, ^^ can you check as well | 17:29 |
| *** mhenkel has joined #tripleo | 17:30 | |
| florianf | akrivoka: thanks for testing! | 17:30 |
| *** amoralej is now known as amoralej|off | 17:30 | |
| *** sudipto has joined #tripleo | 17:31 | |
| *** sudipto_ has joined #tripleo | 17:31 | |
| *** trown is now known as trown|lunch | 17:31 | |
| jd__ | pradk: something is odd | 17:31 |
| pradk | hm weird, even though nonblocking_notify is false that shows up | 17:32 |
| jpich | CI is green on this patch with three +2s, would it be possible to get the +A? https://review.openstack.org/#/c/393192/ | 17:32 |
| jd__ | pradk: https://github.com/openstack/ceilometermiddleware/blob/master/ceilometermiddleware/swift.py#L278-L289 | 17:32 |
| jd__ | pradk: exactly | 17:32 |
| jpich | d0ugal: ^ It's your name on it ;) | 17:32 |
| pradk | panda|weak, can you paste me the rpm version of python-ceilometermiddleware? | 17:33 |
| jd__ | pradk: the conf option is not translated to a boolean | 17:33 |
| panda|weak | pradk: | 17:33 |
| panda|weak | Version : 0.5.0 | 17:33 |
| pradk | yea the notification should be sent directly as we set that to false | 17:33 |
| jd__ | pradk: so setting it in the conf file enable it | 17:33 |
| panda|weak | Release : 0.20161004113850.7f502e2.el7.centos | 17:34 |
| jd__ | pradk: facepalm | 17:34 |
| pradk | lol | 17:34 |
| jd__ | pradk: conf is just a dict of string I think | 17:34 |
| jd__ | so removing that line should fix that | 17:34 |
| panda|weak | hm, explanation ? please ? | 17:34 |
| pradk | panda|weak, the nonblocking_notify option in the conf | 17:35 |
| pradk | should ideally trigger a bipass and send notify | 17:35 |
| pradk | panda|weak, but its not, instead its enabled by just adding to conf | 17:35 |
| *** stendulker has quit IRC | 17:35 | |
| pradk | panda|weak, a bug in ceilomiddleware imo | 17:35 |
| pradk | but as a work around we can remove the line from conf | 17:36 |
| panda|weak | pradk: what is this option causing in the end ? | 17:36 |
| *** ebarrera has joined #tripleo | 17:36 | |
| pradk | panda|weak, https://github.com/openstack/ceilometermiddleware/blob/master/ceilometermiddleware/swift.py#L278-L289 | 17:36 |
| pradk | that | 17:36 |
| pradk | panda|weak, falling to the except block there | 17:36 |
| d0ugal | jpich: Thanks! | 17:37 |
| jpich | d0ugal: Thank you \o/ | 17:37 |
| panda|weak | pradk: I mean, is this causing performance issue ? | 17:37 |
| jd__ | IIUC https://review.openstack.org/#/c/394548/ should fix this "problem" | 17:37 |
| pradk | panda|weak, well thats why the queue is full | 17:37 |
| panda|weak | jd__: do you know what happes when redis slaves are unable to contact the master ? | 17:37 |
| pradk | and possibly why causing cpu load | 17:38 |
| jd__ | panda|weak: no, googling for "redis sentinel mode" should give explanation | 17:38 |
| panda|weak | jd__: and I can't stop thinking about scrubs. | 17:38 |
| pradk | jd__, i +2'ed it, if you can release asap, i'll rebase ceilomiddleware packages | 17:39 |
| *** ealcaniz has quit IRC | 17:39 | |
| jd__ | panda|weak: I have the chance to share my initial with Dr Dorian so… that's why I picked that nick 10 years ago | 17:39 |
| panda|weak | duckduckgoing "redis sentinel mode" doesn't give much results | 17:39 |
| sshnaidm | jd__, do we run sentinel? isn't it just haproxy? | 17:39 |
| jd__ | pradk: thanks | 17:40 |
| jd__ | sshnaidm: pradk might know better | 17:40 |
| *** dtrainor has quit IRC | 17:40 | |
| jd__ | panda|weak: http://redis.io/topics/sentinel it says it works automagically | 17:41 |
| *** dtrainor has joined #tripleo | 17:41 | |
| pradk | sshnaidm, we do use sentinel in redis | 17:41 |
| *** ebarrera has quit IRC | 17:41 | |
| pradk | sshnaidm, i dont know how the new ng ha architecture changed things, but we do configure sentinel in puppet redis | 17:41 |
| panda|weak | ok, but in this case, is ceilometer affected in any way by this ? or is just tries to contact the master | 17:42 |
| panda|weak | I see gnocchi logs are clean, so it looks like it's happy with redis. | 17:42 |
| pradk | whats the issue with redis if i may ask | 17:42 |
| pradk | we fixed the missing firewall rule issue last week | 17:42 |
| pradk | so all should be fine | 17:42 |
| pradk | afaik | 17:42 |
| panda|weak | pradk: I don't know exactly, the slavess have this output on the logs | 17:42 |
| panda|weak | 23087:S 07 Nov 17:42:46.210 # Unable to connect to MASTER: Connection timed out | 17:43 |
| panda|weak | 23087:S 07 Nov 17:42:47.219 * Connecting to MASTER no-such-master:6379 | 17:43 |
| panda|weak | from a slave host to a master host, telnet to port 6379 is working | 17:43 |
| pradk | hmm suspect the same firewall thing .. | 17:44 |
| pradk | panda|weak, this is with latest build? | 17:44 |
| pradk | panda|weak, can you check iptables -L |grep redis | 17:44 |
| panda|weak | pradk: latest build of what ? redis ? | 17:44 |
| panda|weak | I see gnocchi logs are clean, so it looks like it's happy with redis.ACCEPT tcp -- anywhere anywhere multiport dports 6379,26379 /* 108 redis */ state NEW | 17:44 |
| pradk | panda|weak, assume you're seeing this in osp ? | 17:44 |
| panda|weak | on slave | 17:44 |
| panda|weak | ACCEPT tcp -- anywhere anywhere multiport dports 6379,26379 /* 108 redis */ state NEW | 17:45 |
| panda|weak | on master | 17:45 |
| pradk | yep that looks fine | 17:45 |
| *** lucasagomes is now known as lucas-afk | 17:45 | |
| pradk | panda|weak, if central agent, gnocchi metricd are all coordinating correctly i assume all is fine | 17:45 |
| panda|weak | pradk: rdo, we're trying to track down shy operation after deployment in tripleo CI are timing out | 17:45 |
| panda|weak | s/shy/why | 17:46 |
| *** hewbrocca is now known as hewbrocca_afk | 17:46 | |
| *** jpich has quit IRC | 17:47 | |
| pradk | panda|weak, ok any connection errors to redis in ceilometer/central.log , gnocchi/metricd.log | 17:47 |
| panda|weak | pradk: lots of delays and performance issue, in the past two weeks | 17:47 |
| pradk | panda|weak, understand, anything specific pointing to redis as the reason for performance issue? | 17:47 |
| *** dtantsur is now known as dtantsur|afk | 17:47 | |
| *** athomas has quit IRC | 17:47 | |
| pradk | panda|weak, bandini also mentioned this morning that metricd is taking up some cpu .. not sure if thats the same | 17:48 |
| panda|weak | pradk: that was the main issue last week, redis was not starting properly and metricd was eating all the CPU | 17:49 |
| *** florianf has quit IRC | 17:49 | |
| panda|weak | this may be interesting | 17:49 |
| panda|weak | 2016-11-07 17:38:29.244 16318 INFO swiftclient [-] REQ: curl -i http://10.0.0.19:8080/v1/AUTH_0ce54ff3166a4f639721fda10b83f1d8/measure?format=json&path=1f3075a2-0e19-4ce4-9e74-aaccae264a99 -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: 4cf0a0a7670a4160..." | 17:49 |
| panda|weak | 2016-11-07 17:38:29.245 16318 INFO swiftclient [-] RESP STATUS: 401 Unauthorized | 17:49 |
| sshnaidm | pradk, how can I see that this sentinel is running? I don't see something related in process | 17:49 |
| panda|weak | 2016-11-07 17:38:29.245 16318 INFO swiftclient [-] RESP HEADERS: {u'Content-Length': u'131', u'Www-Authenticate': u'Swift realm="AUTH_0ce54ff3166a4f639721fda10b83f1d8", Keystone uri=\'http://172.17.0.20:5000/v2.0\'', u'X-Trans-Id': u'txbb70c1375ce24056b4388-005820bc15', u'Date': u'Mon, 07 Nov 2016 17:38:30 GMT', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': | 17:49 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add simple-playbook element https://review.openstack.org/385608 | 17:49 |
| panda|weak | u'txbb70c1375ce24056b4388-005820bc15'} | 17:49 |
| panda|weak | 2016-11-07 17:38:29.245 16318 INFO swiftclient [-] RESP BODY: <html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html> | 17:49 |
| panda|weak | in metricsd | 17:49 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Containerized Services for Composable Roles https://review.openstack.org/330659 | 17:50 |
| pradk | hmm that looks like a keystone issue | 17:50 |
| pradk | there was a swift issue where proxy server was not able to talk to account server | 17:51 |
| * panda|weak adds keystone to the stack of troublemakers today | 17:51 | |
| pradk | and hence not finding the account | 17:51 |
| pradk | so this looks like gnocchi is trying to post measures to swift with a token and getting unauthorized | 17:53 |
| pradk | did you check keystone if thats a valid token? | 17:53 |
| panda|weak | pradk: this error does appear regularly, but it's not flooding the logs | 17:54 |
| pradk | ok | 17:55 |
| bnemec | panda|weak: In case you aren't having enough fun today: https://bugs.launchpad.net/tripleo/+bug/1639881 | 17:55 |
| openstack | Launchpad bug 1639881 in tripleo "Bogus rabbit server address with ipv6" [Critical,Triaged] | 17:55 |
| sshnaidm | panda|weak, pradk sorry, but I don't see any sentinel running on hosts.. Do I miss something? | 17:57 |
| *** jkilpatr_ has quit IRC | 17:58 | |
| panda|weak | sshnaidm: maybe that's the problem | 17:58 |
| panda|weak | pradk: I'm trying to look at the token | 17:58 |
| panda|weak | bnemec: yay. | 17:59 |
| gfidente | sshnaidm panda|weak pradk there isnt any redis sentinel with pcmk | 17:59 |
| panda|weak | bnemec: I remember those happy times when I was woking on some features, instead of fixing nested issues ... | 18:00 |
| sshnaidm | gfidente, that's what I suspected.. | 18:00 |
| gfidente | sshnaidm yeah that is expected | 18:00 |
| pradk | yea with ng arch i guess things changed .. but we do try to configure it still .. https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/database/redis.pp#L49-L52 | 18:00 |
| pradk | :) | 18:00 |
| pradk | i guess we havent removed the code yet | 18:00 |
| sshnaidm | gfidente, does something manage redis then? | 18:00 |
| gfidente | sshnaidm pcmk does | 18:01 |
| bnemec | panda|weak: Welcome to tripleo, aka the project that finds everyone else's bugs. :-) | 18:01 |
| gfidente | shadower it uses a RA to set the replica master across the set | 18:01 |
| gfidente | sshnaidm ^ | 18:01 |
| sshnaidm | panda|weak, weshay let's track it in the new issue: https://bugs.launchpad.net/tripleo/+bug/1639885 | 18:01 |
| openstack | Launchpad bug 1639885 in tripleo "CI: pingtest timeouts cause by performance issues (redis, swift, ceiliometer)" [Undecided,New] | 18:01 |
| sshnaidm | gfidente, RA? | 18:02 |
| gfidente | resource agent | 18:02 |
| gfidente | haproxy has a rule to gather what is the redis master too | 18:02 |
| gfidente | which is set by pcmk | 18:02 |
| *** derekh has quit IRC | 18:03 | |
| sshnaidm | gfidente, any link to read about it? or how to trace it | 18:03 |
| sshnaidm | gfidente, I try to understand why redis doesn't connect to its master.. | 18:03 |
| gfidente | start by checking the haproxy listener config | 18:03 |
| sshnaidm | gfidente, seems ok | 18:03 |
| gfidente | so is the problem that non-primary nodes don't connect to the master? | 18:04 |
| *** fragatina has joined #tripleo | 18:04 | |
| panda|weak | pradk: ... python-keystoneclient is installed in undercloud but there is no keystone or openstack identity command ... am I missing something ? | 18:05 |
| pradk | panda|weak, part of openstack cli? | 18:05 |
| pradk | openstack endpoint blah i think | 18:06 |
| *** yamahata has joined #tripleo | 18:06 | |
| sshnaidm | gfidente, yes | 18:07 |
| sshnaidm | gfidente, they don't know about it AFAIU | 18:07 |
| *** shardy has quit IRC | 18:07 | |
| *** jkilpatr_ has joined #tripleo | 18:07 | |
| gfidente | sshnaidm right that's what the pcmk resource agent is meant to control | 18:08 |
| gfidente | sshnaidm is there a bug for this? | 18:08 |
| sshnaidm | gfidente, not special, only here: https://bugs.launchpad.net/tripleo/+bug/1639885 | 18:09 |
| openstack | Launchpad bug 1639885 in tripleo "CI: pingtest timeouts cause by performance issues (redis, swift, ceiliometer)" [High,Triaged] | 18:09 |
| *** rhallisey has quit IRC | 18:09 | |
| panda|weak | pradk: .. ok I don't know how to check the token .. | 18:09 |
| *** jayg is now known as jayg|g0n3 | 18:09 | |
| pradk | yea i dont think that should cause any performance issues though | 18:10 |
| pradk | just sounds like a mis config | 18:10 |
| *** dsariel has quit IRC | 18:10 | |
| *** jcoufal_ has joined #tripleo | 18:10 | |
| *** rhallisey has joined #tripleo | 18:10 | |
| panda|weak | pradk: doesn't that mean that ceilometer is unable to store data on swift ? | 18:10 |
| gfidente | sshnaidm can you paste the output from | 18:10 |
| *** chandankumar has joined #tripleo | 18:10 | |
| gfidente | replication info | 18:10 |
| gfidente | from all nodes? | 18:10 |
| gfidente | (into the bug)] | 18:10 |
| gfidente | haproxy knows the password to send commands to redis, you can take it from there | 18:11 |
| pradk | panda|weak, gnocchi is not able to post measures .. yea which looks like a config issue | 18:11 |
| sshnaidm | gfidente, what is replication info..? | 18:11 |
| gfidente | sshnaidm can I get on the environment? | 18:11 |
| sshnaidm | gfidente, yep | 18:11 |
| gfidente | we do it together | 18:11 |
| panda|weak | [root@overcloud-controller-0 log]# redis-cli replication info | 18:11 |
| *** jayg|g0n3 is now known as jayg | 18:11 | |
| panda|weak | Could not connect to Redis at 127.0.0.1:6379: Connection refused | 18:11 |
| panda|weak | Could not connect to Redis at 127.0.0.1:6379: Connection refused | 18:11 |
| sshnaidm | gfidente, you public key, pleas | 18:11 |
| gfidente | panda|weak nah it's not binding on 127.0.0.1 | 18:11 |
| gfidente | sshnaidm github.com/gfidente.keys | 18:11 |
| *** chandankumar has quit IRC | 18:11 | |
| sshnaidm | panda|weak, where is this log from? | 18:12 |
| *** jcoufal has quit IRC | 18:13 | |
| sshnaidm | gfidente, look at priv | 18:13 |
| panda|weak | so recapping. nonblocking_notify is causing perfomrance issues. Redis is unable to replicate to slaves, and gnocchi is unable to store measurement to swift | 18:13 |
| *** jkilpatr_ has quit IRC | 18:13 | |
| sshnaidm | gfidente, it's panda|weak's environemnt :) | 18:13 |
| panda|weak | did I forgeet something ? | 18:13 |
| *** fzdarsky is now known as fzdarsky|afk | 18:13 | |
| panda|weak | sshnaidm: no logs, just a command | 18:14 |
| *** jaosorior_sick has quit IRC | 18:14 | |
| *** rbrady-afk is now known as rbrady | 18:14 | |
| *** rhallisey has quit IRC | 18:15 | |
| *** akrivoka has quit IRC | 18:15 | |
| *** rhallisey has joined #tripleo | 18:15 | |
| panda|weak | sshnaidm: maybe the wrongest command ever | 18:16 |
| *** akrivoka has joined #tripleo | 18:17 | |
| panda|weak | mh redis-cli -h 172.17.0.13 is taking a lot of time to answer to any command | 18:17 |
| *** dtrainor has quit IRC | 18:18 | |
| sshnaidm | panda|weak, how can I get to overcloud nodes in your env? | 18:18 |
| *** aufi has quit IRC | 18:18 | |
| *** yamahata has quit IRC | 18:19 | |
| panda|weak | sshnaidm: ssh heat-admin@192.0.2.7 | 18:19 |
| panda|weak | 192.0.2.17 192.0.2.16 | 18:20 |
| sshnaidm | panda|weak, omg, it's so sloooooow | 18:20 |
| panda|weak | sshnaidm: dns reverse check has to timeout first ... | 18:20 |
| sshnaidm | panda|weak, and Permission denied (publickey,gssapi-keyex,gssapi-with-mic) | 18:20 |
| sshnaidm | panda|weak, root or jenkins? | 18:21 |
| panda|weak | sshnaidm: should be jenkins | 18:21 |
| sshnaidm | panda|weak, nothing.. if you get access, can you give it to gfidente please? | 18:21 |
| gfidente | I am there | 18:22 |
| sshnaidm | panda|weak, adding jenkins keys there | 18:22 |
| gfidente | waiting for info replication to return | 18:22 |
| sshnaidm | gfidente, great | 18:22 |
| *** yamahata has joined #tripleo | 18:22 | |
| panda|weak | gfidente: it usually takes so much to reply ? | 18:22 |
| gfidente | panda|weak not at all | 18:22 |
| sshnaidm | panda|weak, ssh to overcloud node is extremely slow, box seems very busy | 18:23 |
| gfidente | so yeah this will cause cascading issues I suppose | 18:23 |
| panda|weak | sshnaidm: ssh is slow because it's trying to revers dns before letting you in | 18:23 |
| sshnaidm | panda|weak, still, so many time | 18:24 |
| gfidente | but yes looks like pcmk failed to set the master node | 18:24 |
| panda|weak | gfidente: any ideas why it could be so slow ? | 18:24 |
| gfidente | master_host:no-such-master | 18:24 |
| gfidente | master_port:6379 | 18:24 |
| gfidente | master_link_status:down | 18:24 |
| *** dtrainor has joined #tripleo | 18:24 | |
| *** jpena is now known as jpena|off | 18:24 | |
| *** dougbtv has quit IRC | 18:24 | |
| *** liverpooler has quit IRC | 18:24 | |
| panda|weak | pcs status says differently | 18:24 |
| panda|weak | Master/Slave Set: redis-master [redis] | 18:24 |
| panda|weak | Masters: [ overcloud-controller-2 ] | 18:24 |
| gfidente | right replica in between controller-1 and controller-2 is working fine | 18:26 |
| gfidente | and they are much faster as well | 18:26 |
| *** jkilpatr_ has joined #tripleo | 18:26 | |
| *** sshnaidm is now known as sshnaidm|brb | 18:26 | |
| panda|weak | bnemec: is TripleoCI like this by design ? | 18:27 |
| openstackgerrit | Merged openstack/python-tripleoclient: Pass clients to get the get_password function https://review.openstack.org/393192 | 18:28 |
| panda|weak | gfidente: replica between the two slaves ? | 18:28 |
| gfidente | panda|weak no overcloud-controller-2 is the master | 18:29 |
| *** mcornea has quit IRC | 18:29 | |
| *** sudipto has quit IRC | 18:30 | |
| *** sudipto_ has quit IRC | 18:30 | |
| panda|weak | oh right. | 18:30 |
| *** fragatina has quit IRC | 18:30 | |
| gfidente | so it's really only controller-0 which is so slow | 18:30 |
| *** fragatina has joined #tripleo | 18:31 | |
| *** ohamada has quit IRC | 18:31 | |
| *** yamahata has quit IRC | 18:31 | |
| gfidente | I think -0 is unable to join the replica because it is too slow | 18:31 |
| gfidente | I tried to gave slaveof manually and it didn't work either | 18:32 |
| gfidente | why only -0 is so slow remains to be seen | 18:32 |
| gfidente | it joined the cluster now | 18:34 |
| gfidente | and seems much quicker | 18:34 |
| gfidente | in fact it now goes at same speed of others | 18:34 |
| gfidente | WOW :) | 18:34 |
| panda|weak | gfidente: check now | 18:34 |
| gfidente | yeah it's good now | 18:35 |
| gfidente | I set slaveof | 18:35 |
| gfidente | and as soon as that was set it went to normal speed | 18:35 |
| gfidente | and joined the replica | 18:35 |
| *** mgould is now known as mgould|afk | 18:36 | |
| panda|weak | gfidente: so it's not CPU usage on - the problem ? | 18:36 |
| gfidente | so it seems to be working fine to me now | 18:36 |
| panda|weak | -0 | 18:36 |
| gfidente | no it's not | 18:36 |
| gfidente | cpu usage remains high, but after it joined the cluster redis became responsive | 18:36 |
| gfidente | though I don't think this would cause any issue to the clients | 18:36 |
| gfidente | which are forwarded to the master node only anyway | 18:36 |
| gfidente | sorry guys, going for dinner, be back later | 18:37 |
| panda|weak | gfidente: ok, thanks! | 18:37 |
| panda|weak | sshnaidm|brb: I think it's better to create different bugs for the different issues. | 18:37 |
| openstackgerrit | Ben Nemec proposed openstack/python-tripleoclient: Pass clients to get the get_password function https://review.openstack.org/394573 | 18:38 |
| *** hjensas has quit IRC | 18:39 | |
| dsneddon | A change in disk-image-builder will affect our use of networking in the overcloud images. https://review.openstack.org/#/c/392170/ | 18:39 |
| dsneddon | Once this patch goes in, we have to choose between the network service and NetworkManager, instead of running both. ^^^ | 18:39 |
| dsneddon | I may write an email discussing this to openstack-dev, but I was wondering if anyone had any insights on possible effects of disabling NetworkManager in our overcloud images universally. | 18:40 |
| dsneddon | dprince, gfidente, bnemec, slagle, any thoughts? ^^^ | 18:40 |
| slagle | dsneddon: i don't know of any possible side effects. pretty much the only reason i know of why it's not disabled already is b/c we've understood that you shouldn't have to | 18:42 |
| slagle | but i take it that is no longer true? | 18:42 |
| *** milan has quit IRC | 18:42 | |
| dprince | dsneddon: I suppose our existing approach was mostly around letting distro defaults persist, and simply setting NM managed = no via os-net-config where we needed it | 18:43 |
| dprince | dsneddon: is there a reasons this approach isn't working anymore? | 18:43 |
| dsneddon | slagle, dprince: With the recent release of RHEL 7.3 we started seeing selinux AVC alerts because both services were trying to run dhclient for the same interface. | 18:44 |
| dsneddon | It hasn't caused any problems per se, but it seems like bad behavior. | 18:44 |
| *** akrivoka has quit IRC | 18:44 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Change nova ram_allocation_ratio to match puppet-nova https://review.openstack.org/394488 | 18:44 |
| *** akrivoka has joined #tripleo | 18:45 | |
| dprince | dsneddon: in that case, a setting to control/select which one we want running seems reasonable | 18:45 |
| dsneddon | dprince, What concerns me is that in Noam's DIB patch, he claims that "NetworkManager know how to handle link carrier and network service don't. This crucial for scenarios like nova suspend resume,shelve unshelve,and co. NetworkManager know when this signal received to initiate DHCP" | 18:45 |
| dsneddon | dprince, I hadn't heard that before, but I think his use case is at odds with ours, and we would likely set DIB_NETWORK_MANAGER='network' to disable NM. | 18:46 |
| dprince | dsneddon: I suppose I'd rather not have it baked into an image though. It would be much better to have it config configured dynamically | 18:46 |
| dsneddon | dprince, I agree, it seems like a big hammer to use to address the issue of both services being enabled. | 18:47 |
| slagle | dsneddon: is this the bz? https://bugzilla.redhat.com/show_bug.cgi?id=1390011 | 18:47 |
| openstack | bugzilla.redhat.com bug 1390011 in rhel-osp-director "dhclient related selinux avcs on the overcloud nodes" [Urgent,Assigned] - Assigned to bfournie | 18:47 |
| *** saneax is now known as saneax-_-|AFK | 18:48 | |
| bfournie | slagle: yes the problem with both network and NetworkManager running mainly comes play with dhcp-all-interfaces because NM_CONTROLLED=no is not set there | 18:48 |
| slagle | it sounds like a result of NM + dhcp-all-interfaces | 18:48 |
| slagle | in which case, that's why baking it into the image feels necessary | 18:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/389957 | 18:48 |
| dsneddon | slagle, I think you're right. | 18:49 |
| *** cylopez has left #tripleo | 18:49 | |
| bfournie | slagle: yes, we'd have to either choose a networking service for the image or set NM_CONTROLLED=no, but that the 2nd one causes problems with anyone using dhcp-all-interfaces who wants to use NetworkManager, as Noam patch refers to | 18:51 |
| dsneddon | bfournie: If I understand the patch correctly, there will no longer be an option to have both networking services enabled, so I don't think that just setting NM_CONTROLLED=no and not specifying a DIB_NETWORK_MANAGER is an option. | 18:55 |
| bfournie | dsneddon: yes, I agree | 18:55 |
| bfournie | dsneddon: one that patch goes in we wouldn't have that option, and also should remove the setting of NM_CONTROLLED=no from os-net-config and just rely on 'network' being set for DIB_NETWORK_MANAGER | 18:57 |
| dsneddon | bfournie, I see no reason to change os-net-config, especially as that might affect upgrades (the removal of NM_CONTROLLED=no will cause os-net-config to restart the interfaces). | 18:59 |
| lblanchard | jrist-afk, jtomasek|afk, honza…and anyone else who may be interested…I wanted to show you all some ideas I had on composable roles in the UI. This is future thinking, but just wanted to throw it out there: https://openstack.invisionapp.com/share/J498J5OZX | 18:59 |
| dsneddon | bfournie, Although I do think we could make it an os-net-config parameter, potentially. | 19:00 |
| bfournie | dsneddon: true, ok. It would only matter if DIB_NETWORK_MANa | 19:00 |
| bfournie | GER was set to NetworkManager | 19:00 |
| honza | lblanchard: nom nom nom | 19:00 |
| lblanchard | honza: :) | 19:01 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add simple-playbook element https://review.openstack.org/385608 | 19:04 |
| *** ayoung has joined #tripleo | 19:06 | |
| honza | lblanchard: given how little i know about this feature, i see no issues with the wireframes; it makes senses and it's simple | 19:07 |
| *** rbowen has quit IRC | 19:08 | |
| *** trown|lunch is now known as trown | 19:10 | |
| *** radeksmg has joined #tripleo | 19:12 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker: WIP DO NOT MERGE Allow the creation of resources in disabled state https://review.openstack.org/394198 | 19:13 |
| dsneddon | dprince, I think we may need to call 'systemctl restart network' after os-net-config runs. In which repo does the script live now that calls os-net-config? | 19:20 |
| weshay | panda|weak, sshnaidm|brb thanks guys :) | 19:21 |
| dprince | dsneddon: it still lives in the elements | 19:22 |
| dprince | dsneddon: tripleo-image-elements | 19:22 |
| dsneddon | dprince, Ah, I thought we had moved away from that, thanks. | 19:22 |
| dprince | dsneddon: we need to update steve's review to move it into t-h-t again. I would much prefer if we moved away from elements for this | 19:23 |
| *** rbowen has joined #tripleo | 19:23 | |
| dsneddon | dprince, Yes, I agree, I'll take a look at Steve's review. | 19:23 |
| dprince | dsneddon: he may have abandoned it btw. You might need to dig a bit .... ;) | 19:24 |
| dsneddon | dprince, Ah, yes, found it, I remember reviewing this many moons ago | 19:24 |
| dsneddon | dprince, Which explains the vague memory that we had already moved it from t-i-e | 19:25 |
| slagle | bnemec: can you review https://review.openstack.org/#/c/394471/ | 19:27 |
| *** d0ugal has quit IRC | 19:30 | |
| *** d0ugal has joined #tripleo | 19:31 | |
| *** d0ugal has quit IRC | 19:31 | |
| *** d0ugal has joined #tripleo | 19:31 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add Barbican key order to scenario002 https://review.openstack.org/389057 | 19:32 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Include keystone authtoken config in manila-share service https://review.openstack.org/394439 | 19:38 |
| *** cylopez has joined #tripleo | 19:38 | |
| *** yamahata has joined #tripleo | 19:39 | |
| lblanchard | honza: thanks for reviewing!! I don't know a whole lot about the feature either unfortunately :( Maybe jtomasek|afk can comment more tomorrow and enlighten us. | 19:41 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Move db settings from manila-api to manila-base https://review.openstack.org/394440 | 19:41 |
| *** pkovar has quit IRC | 19:44 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Pass clients to get the get_password function https://review.openstack.org/394573 | 19:46 |
| *** akrivoka has quit IRC | 19:47 | |
| *** akrivoka has joined #tripleo | 19:48 | |
| *** sshnaidm|brb is now known as sshnaidm | 19:52 | |
| openstackgerrit | Leif Madsen proposed openstack/tripleo-docs: Link to RDO built images https://review.openstack.org/394602 | 19:52 |
| leifmadsen | trown: ^^ fyi | 19:53 |
| leifmadsen | thx again | 19:53 |
| *** yamahata has quit IRC | 19:54 | |
| openstackgerrit | Brent Eagles proposed openstack/puppet-tripleo: WIP: Call VF configuration from udev rules https://review.openstack.org/394604 | 19:56 |
| *** dprince has quit IRC | 20:00 | |
| sshnaidm | gfidente, what did you do exactly so controller was back to life? | 20:00 |
| *** cylopez has quit IRC | 20:02 | |
| *** dsneddon_ has joined #tripleo | 20:06 | |
| *** dciabrin has quit IRC | 20:16 | |
| *** dciabrin has joined #tripleo | 20:18 | |
| *** mcornea has joined #tripleo | 20:18 | |
| panda|weak | sshnaidm: issued slaveof command from redis-cli | 20:18 |
| panda|weak | sshnaidm: I'm splitting the issues, one per bug in launchpad | 20:19 |
| sshnaidm | panda|weak, sure, agree | 20:19 |
| *** dougbtv has joined #tripleo | 20:22 | |
| *** noslzzp has quit IRC | 20:24 | |
| *** noslzzp has joined #tripleo | 20:24 | |
| *** akrivoka has quit IRC | 20:25 | |
| *** d0ugal has quit IRC | 20:26 | |
| *** d0ugal has joined #tripleo | 20:27 | |
| *** d0ugal has quit IRC | 20:27 | |
| *** d0ugal has joined #tripleo | 20:27 | |
| *** maeca1 has joined #tripleo | 20:33 | |
| *** dsneddon_ has quit IRC | 20:37 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add an optional extra node admin ssh key parameter https://review.openstack.org/394446 | 20:37 |
| openstackgerrit | Leif Madsen proposed openstack/tripleo-docs: Link to RDO built images https://review.openstack.org/394602 | 20:47 |
| *** mhenkel has quit IRC | 20:53 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Fix handling of missing environment files https://review.openstack.org/394471 | 20:57 |
| *** Goneri has quit IRC | 20:57 | |
| *** dsariel has joined #tripleo | 20:58 | |
| *** mcornea has quit IRC | 21:01 | |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Fix handling of missing environment files https://review.openstack.org/394630 | 21:02 |
| *** ayoung has quit IRC | 21:04 | |
| mwhahaha | anyone have any thoughts as to why an HA deploy fails at the pcs cluster setup with Unable to authenticate to overcloud-controller-0 - (HTTP error: 401) | 21:11 |
| *** rhallisey has quit IRC | 21:12 | |
| mwhahaha | oh i guess not having a corosync.conf might be problemattic | 21:13 |
| trozet | hi can someone tell me what the overcloud parameter outputs for <service>InternalVip are used for? like NovaInternalVip: | 21:13 |
| *** iranzo has quit IRC | 21:14 | |
| *** ebarrera has joined #tripleo | 21:15 | |
| mwhahaha | trozet: aren't they for like internal access between the services? | 21:15 |
| trozet | mwhahaha: I just can't find any reference to the variable in THT, so not sure where it is being used | 21:16 |
| trozet | mwhahaha: the description says stuff like VIP for Neutron API internal endpoint | 21:16 |
| trozet | mwhahaha: I looked at endpoint_map I dont see it using them | 21:17 |
| trozet | dsneddon maybe you know^^^^^? | 21:17 |
| dsneddon | trozet, That gets constructed, based on the setting for NovaApiNetwork in the ServiceNetMap | 21:18 |
| trozet | dsneddon: yeah i see how it gets created, I'm just not sure what it is used for afterwards | 21:19 |
| dsneddon | trozet, You can override which network that lives on, in which case the IP will be different as a result of this line in overcloud.j2.yaml: value: {get_attr: [VipMap, net_ip_map, {get_attr: [ServiceNetMap, service_net_map, NovaApiNetwork]}]} | 21:19 |
| trozet | dsneddon: like I don't see any reference to *InternalVip anywhere in THT | 21:19 |
| *** jayg is now known as jayg|g0n3 | 21:19 | |
| dsneddon | trozet, Yeah, I don't see it used anywhere, either. | 21:19 |
| trozet | dsneddon: I am trying a deployment now and just deleted the ODL one | 21:20 |
| trozet | dsneddon: if it works maybe i will try another deployment and delete all of them? | 21:20 |
| bnemec | trozet: It's possible they aren't being used yet. There's ongoing work to have ssl everywhere that I think those may have been added for. | 21:22 |
| *** rbrady is now known as rbrady-afk | 21:24 | |
| *** ayoung has joined #tripleo | 21:27 | |
| *** jkilpatr_ has quit IRC | 21:27 | |
| *** maeca1 has left #tripleo | 21:29 | |
| trozet | bnemec, dsneddon, mwhahaha: https://review.openstack.org/#/c/199554/ | 21:30 |
| trozet | it's not used anymroe in the dnpoint map | 21:31 |
| trozet | used anymore in endpoint map | 21:31 |
| trozet | I'm going to push a patch to remove them | 21:31 |
| dsneddon | trozet, Sounds good to me | 21:34 |
| bnemec | Ah, interesting. | 21:35 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates: Fixes incorrect reference to OpendaylightApiNetwork https://review.openstack.org/394640 | 21:36 |
| *** rbowen has quit IRC | 21:38 | |
| trozet | dsneddon, bnemec: does it need a bug ID? | 21:38 |
| *** yamahata has joined #tripleo | 21:41 | |
| *** rbowen has joined #tripleo | 21:41 | |
| *** jcoufal_ has quit IRC | 21:41 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Newtonthing to see here https://review.openstack.org/394646 | 21:42 |
| dsneddon | trozet, Hmm, it never hurts, although we do sometimes remove cruft without a bug ID. | 21:43 |
| dsneddon | trozet, For this many lines, I would vote yes on a bug ID | 21:43 |
| trozet | dsneddon: ok | 21:43 |
| trozet | lol on the bnemec commit msg^^^^ | 21:44 |
| *** trown is now known as trown|outtypewww | 21:44 | |
| bnemec | Gotta differentiate from my master Nothing to see here patch. ;-) | 21:45 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates: Removes deprecated overcloud VIP outputs https://review.openstack.org/394651 | 21:49 |
| openstackgerrit | Merged openstack/tripleo-common: Sets defaults in swift connection related to retries https://review.openstack.org/389124 | 21:50 |
| *** jkilpatr has joined #tripleo | 21:58 | |
| *** cylopez has joined #tripleo | 21:59 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker: WIP DO NOT MERGE Allow the creation of resources in disabled state https://review.openstack.org/394198 | 22:04 |
| *** lblanchard has quit IRC | 22:05 | |
| *** yamahata has quit IRC | 22:06 | |
| *** mhenkel has joined #tripleo | 22:07 | |
| *** jprovazn has quit IRC | 22:13 | |
| *** tiswanso has quit IRC | 22:23 | |
| *** cylopez has quit IRC | 22:23 | |
| *** radeksmg has quit IRC | 22:26 | |
| *** absubram has quit IRC | 22:26 | |
| *** pblaho has quit IRC | 22:31 | |
| *** pblaho has joined #tripleo | 22:33 | |
| *** fragatin_ has joined #tripleo | 22:37 | |
| *** eglynn has quit IRC | 22:38 | |
| openstackgerrit | Brent Eagles proposed openstack/os-net-config: WIP: Add support for enabling hotplug on interfaces https://review.openstack.org/394660 | 22:38 |
| openstackgerrit | Brent Eagles proposed openstack/os-net-config: WIP: Add support for enabling hotplug on interfaces https://review.openstack.org/394660 | 22:39 |
| *** fragatina has quit IRC | 22:40 | |
| dsneddon | beagles, Would HOTPLUG=yes/no really apply to *any* interface type? Instead of applying it to BaseOpts, shouldn't it only be added to objects which support hotplug events? | 22:40 |
| beagles | dsneddon: good point... should only be relevant to Interface... will fix | 22:42 |
| dsneddon | beagles, Yeah, interface, and *maybe* Infiniband interfaces, but since we don't have hardware to test, probably just Interface for now. | 22:42 |
| *** tiswanso has joined #tripleo | 22:47 | |
| *** bfournie has quit IRC | 22:51 | |
| *** tiswanso has quit IRC | 22:52 | |
| *** dsariel has quit IRC | 22:54 | |
| panda|weak | sshnaidm: https://bugs.launchpad.net/tripleo/+bug/1639970. I don't think anything we've found so far is the real issue with HA jobs | 22:56 |
| openstack | Launchpad bug 1639970 in tripleo "CI: cinder fails to allocate memory while creating volume for ping test tenant" [Critical,Confirmed] | 22:56 |
| *** limao has joined #tripleo | 22:56 | |
| panda|weak | exhausted, going to bed. | 22:56 |
| *** panda|weak is now known as panda|zZ | 22:56 | |
| sshnaidm | panda|zZ, this is something new | 22:57 |
| sshnaidm | panda|zZ, g'nite! | 22:57 |
| panda|zZ | sshnaidm: therve shomed me a few hours ago, but I was blind | 22:57 |
| panda|zZ | all the failing jobs of the past hours have that message | 22:58 |
| sshnaidm | panda|zZ, you were weak | 22:58 |
| *** limao_ has joined #tripleo | 22:58 | |
| sshnaidm | panda|zZ, arxcruz saw this in tempest already.. | 22:58 |
| panda|zZ | sshnaidm: heh. | 22:58 |
| panda|zZ | maybe 6G is not enough anymore for the overcloud nodes ? | 22:58 |
| openstackgerrit | Brent Eagles proposed openstack/os-net-config: WIP: Add support for enabling hotplug on interfaces https://review.openstack.org/394660 | 22:58 |
| *** limao has quit IRC | 23:01 | |
| sshnaidm | panda|zZ, either to create something less than 1GB image | 23:02 |
| sshnaidm | but seems it's not possible | 23:03 |
| *** saneax-_-|AFK is now known as saneax | 23:07 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Reload haproxy configuration as a post-deployment step https://review.openstack.org/393644 | 23:10 |
| *** morazi has quit IRC | 23:17 | |
| *** gfidente has quit IRC | 23:21 | |
| *** rlandy has quit IRC | 23:24 | |
| *** pradk has quit IRC | 23:25 | |
| *** ayoung has quit IRC | 23:31 | |
| *** tiswanso has joined #tripleo | 23:34 | |
| *** ayoung has joined #tripleo | 23:34 | |
| *** bfournie has joined #tripleo | 23:38 | |
| *** tiswanso has quit IRC | 23:38 | |
| *** limao_ has quit IRC | 23:46 | |
| *** ayoung has quit IRC | 23:49 | |
| *** sshnaidm is now known as sshnaidm|away | 23:53 | |
| *** dciabrin has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!