| *** oneswig has joined #tripleo | 00:04 | |
| *** oneswig has quit IRC | 00:09 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1594732 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1594732 in tripleo "CI: No connected gearman servers" [Critical,Confirmed] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 00:10 |
| *** rook has joined #tripleo | 00:26 | |
| *** limao_ has joined #tripleo | 00:38 | |
| *** akshai has joined #tripleo | 00:45 | |
| *** akshai has quit IRC | 01:06 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1594732 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1594732 in tripleo "CI: No connected gearman servers" [Critical,Confirmed] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 01:10 |
| *** akshai has joined #tripleo | 01:22 | |
| *** akshai_ has joined #tripleo | 01:25 | |
| *** akshai has quit IRC | 01:28 | |
| *** dsariel has quit IRC | 01:29 | |
| *** akshai has joined #tripleo | 01:35 | |
| *** akshai_ has quit IRC | 01:38 | |
| *** hanchao has joined #tripleo | 01:56 | |
| *** dsariel has joined #tripleo | 01:56 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 02:10 |
| *** ebalduf has quit IRC | 02:47 | |
| *** ebalduf has joined #tripleo | 03:03 | |
| *** akshai has quit IRC | 03:05 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 03:10 |
| *** pleia2_ is now known as pleia2 | 03:15 | |
| *** coolsvap has joined #tripleo | 03:17 | |
| *** _milan_ has joined #tripleo | 03:19 | |
| *** akshai has joined #tripleo | 03:20 | |
| *** milan has quit IRC | 03:21 | |
| *** shadower has quit IRC | 03:25 | |
| *** shadower has joined #tripleo | 03:25 | |
| *** lynxman has quit IRC | 03:26 | |
| *** lynxman has joined #tripleo | 03:34 | |
| *** ramishra has joined #tripleo | 03:38 | |
| *** shivrao has joined #tripleo | 03:40 | |
| *** shivrao has quit IRC | 04:02 | |
| *** shivrao has joined #tripleo | 04:04 | |
| *** oneswig has joined #tripleo | 04:06 | |
| *** akshai has quit IRC | 04:09 | |
| *** dsariel has quit IRC | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 04:10 |
| *** oneswig has quit IRC | 04:10 | |
| sshnaidm | EmilienM, hi | 04:10 |
| sshnaidm | EmilienM, where do you see this error? Can you provide a link in bug? https://bugs.launchpad.net/tripleo/+bug/1596390 | 04:11 |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 04:11 |
| *** links has joined #tripleo | 04:15 | |
| *** andreas-f has joined #tripleo | 04:37 | |
| *** ramishra has quit IRC | 04:55 | |
| *** ramishra has joined #tripleo | 04:55 | |
| *** masco has joined #tripleo | 05:06 | |
| *** numans has joined #tripleo | 05:08 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 05:10 |
| *** numans has quit IRC | 05:12 | |
| *** numans has joined #tripleo | 05:12 | |
| *** skramaja has joined #tripleo | 05:17 | |
| *** yamahata has quit IRC | 05:17 | |
| *** yamahata has joined #tripleo | 05:17 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DON'T MERGE, TESTING https://review.openstack.org/334288 | 05:26 |
| *** shivrao has quit IRC | 05:27 | |
| *** saneax_AFK is now known as saneax | 05:29 | |
| *** oshvartz has joined #tripleo | 05:35 | |
| *** rasca has joined #tripleo | 05:35 | |
| *** fragatina has joined #tripleo | 05:40 | |
| *** fragatina has quit IRC | 05:40 | |
| *** fragatina has joined #tripleo | 05:41 | |
| *** florianf has joined #tripleo | 05:42 | |
| openstackgerrit | Merged openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/333673 | 05:48 |
| *** dtantsur|afk is now known as dtantsur | 05:51 | |
| *** jcoufal has joined #tripleo | 05:57 | |
| *** jcoufal has quit IRC | 05:58 | |
| *** masco has left #tripleo | 06:02 | |
| *** bootsha has joined #tripleo | 06:05 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 06:10 |
| *** shivrao has joined #tripleo | 06:15 | |
| *** dsariel has joined #tripleo | 06:17 | |
| *** ramishra_ has joined #tripleo | 06:19 | |
| *** ramishra has quit IRC | 06:21 | |
| *** ramishra_ has quit IRC | 06:25 | |
| *** ramishra has joined #tripleo | 06:25 | |
| *** rcernin has joined #tripleo | 06:27 | |
| *** liverpooler has joined #tripleo | 06:28 | |
| openstackgerrit | Merged openstack/diskimage-builder: Handle locales install on Fedora 24 https://review.openstack.org/333118 | 06:29 |
| *** dixiaoli has joined #tripleo | 06:40 | |
| *** pcaruana has joined #tripleo | 06:41 | |
| *** sthillma has joined #tripleo | 06:44 | |
| *** sthillma has quit IRC | 06:48 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Fix JOB_NAME parameter https://review.openstack.org/334305 | 06:49 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Fix JOB_NAME parameter https://review.openstack.org/334305 | 06:53 |
| *** ccamacho has joined #tripleo | 06:57 | |
| *** d0ugal has joined #tripleo | 06:58 | |
| *** d0ugal has quit IRC | 06:59 | |
| *** tesseract- has joined #tripleo | 06:59 | |
| *** d0ugal has joined #tripleo | 07:00 | |
| *** jprovazn has joined #tripleo | 07:04 | |
| *** yamahata has quit IRC | 07:06 | |
| *** ebalduf has quit IRC | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 07:10 |
| *** jtomasek has joined #tripleo | 07:11 | |
| *** mcornea has joined #tripleo | 07:12 | |
| *** oneswig has joined #tripleo | 07:12 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 07:14 |
| *** jpena|off is now known as jpena | 07:16 | |
| *** bvandenh has joined #tripleo | 07:19 | |
| *** anshul has joined #tripleo | 07:21 | |
| *** anshul is now known as Guest59963 | 07:22 | |
| d0ugal | hrm, looks like CI job are not beings started? | 07:22 |
| *** ebarrera has joined #tripleo | 07:23 | |
| ccamacho | Hey d0ugal, yeahp :( | 07:26 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Fix JOB_NAME parameter https://review.openstack.org/334305 | 07:27 |
| sshnaidm | ccamacho, d0ugal where do you look? | 07:27 |
| *** limao_ has quit IRC | 07:28 | |
| d0ugal | sshnaidm: on gerrit, the jobs are not starting :D | 07:28 |
| d0ugal | sshnaidm: and http://tripleo.org/cistatus.html | 07:28 |
| *** ifarkas has joined #tripleo | 07:28 | |
| *** limao has joined #tripleo | 07:29 | |
| sshnaidm | d0ugal, weird, just submitted to tripleo-ci and it worked | 07:30 |
| d0ugal | sshnaidm: It looks like it might only be working for tripleo-ci? not sure | 07:30 |
| ccamacho | In my case, all my jobs were kiled by timeouts, no hosts available, also this was reported few hours ago https://bugs.launchpad.net/tripleo/+bug/1596390 | 07:30 |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 07:30 |
| sshnaidm | ccamacho, yeah, master jobs fail because of "no hosts" | 07:31 |
| *** dmk0202 has joined #tripleo | 07:31 | |
| *** jpich has joined #tripleo | 07:31 | |
| sshnaidm | ccamacho, although I don't see where is "qemu-img: No space left on device" ? | 07:31 |
| ccamacho | me neither, the bug don't have to many description, but still able to see other different errors... | 07:33 |
| *** panda has quit IRC | 07:34 | |
| sshnaidm | d0ugal, which patch do you submit that doesn't run jobs? | 07:34 |
| *** numans has quit IRC | 07:35 | |
| *** panda has joined #tripleo | 07:35 | |
| d0ugal | sshnaidm: https://review.openstack.org/#/c/332671/ | 07:35 |
| *** aufi has joined #tripleo | 07:36 | |
| sshnaidm | d0ugal, half hour before? | 07:36 |
| d0ugal | sshnaidm: I just rechecked that one, because it should pass and I wanted to test CI | 07:36 |
| *** zaneb has joined #tripleo | 07:36 | |
| d0ugal | sshnaidm: Yeah, I also tried yesterday with one patch | 07:37 |
| * d0ugal looks for it | 07:37 | |
| sshnaidm | d0ugal, I see your patch running: http://status.openstack.org/zuul/ | 07:38 |
| sshnaidm | d0ugal, remaining time: 2 hr 14 min | 07:38 |
| d0ugal | sshnaidm: oh, cool - maybe that problem is resolved then | 07:39 |
| d0ugal | I'll wait a bit and see. | 07:39 |
| *** shivrao has quit IRC | 07:39 | |
| *** olap has joined #tripleo | 07:41 | |
| *** zoli_gone-proxy is now known as zoliXXL | 07:43 | |
| zoliXXL | good morning | 07:45 |
| *** paramite has joined #tripleo | 07:45 | |
| *** shardy has joined #tripleo | 07:47 | |
| *** numans has joined #tripleo | 07:48 | |
| *** lucas-afk is now known as lucasagomes | 08:02 | |
| *** athomas has joined #tripleo | 08:03 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 08:10 |
| *** dixiaoli has quit IRC | 08:16 | |
| *** numans has quit IRC | 08:17 | |
| *** itamarl has joined #tripleo | 08:17 | |
| openstackgerrit | Merged openstack/instack-undercloud: Add net config override https://review.openstack.org/305957 | 08:20 |
| *** andreas-f has quit IRC | 08:28 | |
| *** olap has quit IRC | 08:29 | |
| jistr | marios: morning :) do you think you could add those to your review queue? e.g. 2 of them only need another +2 and they have a green CI https://etherpad.openstack.org/p/tripleo-liberty-mitaka-upgrades | 08:30 |
| marios | jistr: hey man, yeah on it (I am looking at keystone l..m now) | 08:30 |
| marios | lgtm so far | 08:30 |
| jistr | marios: awesome, thanks! :) | 08:30 |
| *** olap has joined #tripleo | 08:31 | |
| *** numans has joined #tripleo | 08:31 | |
| *** electrofelix has joined #tripleo | 08:32 | |
| *** shardy has quit IRC | 08:32 | |
| *** shardy has joined #tripleo | 08:34 | |
| *** derekh has joined #tripleo | 08:36 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Keystone liberty mitaka upgrade step https://review.openstack.org/302235 | 08:42 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 08:57 |
| *** mgould|afk is now known as mgould | 09:04 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Allow pacemaker ports in firewall https://review.openstack.org/334022 | 09:05 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Allow sahara ports in firewall https://review.openstack.org/334023 | 09:08 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Nova needs the proper volumes to use Cinder https://review.openstack.org/301592 | 09:08 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 09:10 |
| *** olap has quit IRC | 09:12 | |
| *** sambetts|afk is now known as sambetts | 09:13 | |
| *** olap has joined #tripleo | 09:14 | |
| *** chem has joined #tripleo | 09:17 | |
| matbu | jistr: hi, i'm looking at the https://review.openstack.org/#/c/325205/22/extraconfig/tasks/major_upgrade_controller_pacemaker_1.sh | 09:18 |
| d0ugal | sshnaidm: I stlll don't see tripleo gates for 332671? | 09:18 |
| matbu | jistr: i'm wondering if we could make a general mariadb upgrade script | 09:18 |
| matbu | s/general/generic/ | 09:19 |
| jistr | matbu: i think that's the way bandini and dciabrin intended it to work | 09:19 |
| matbu | jistr: which can be used by the UC upgrade too | 09:19 |
| *** chem has quit IRC | 09:19 | |
| dciabrin | matbu jistr: hi guys :) | 09:20 |
| *** chem has joined #tripleo | 09:20 | |
| matbu | dciabrin: hello :) | 09:20 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 09:21 |
| *** _milan_ has quit IRC | 09:22 | |
| dciabrin | matbu, we wanted to have it generic eventually, bandini though that unblocking the overcloud upgrade and documenting manual steps on the undercloud would be sufficient for a first step | 09:23 |
| jistr | matbu, dciabrin: i like the idea, there are some practical challenges there (e.g. undercloud doesn't use pacemaker, which would change the procedure quite a bit i think, and there's no easy way currently to share that code between UC and OC in this sense) | 09:24 |
| jistr | so perhaps going step by step here would be good | 09:24 |
| *** limao has quit IRC | 09:24 | |
| sshnaidm | d0ugal, as I see the "upgrades" job failed, others runs, but it will take another 1-2 hours, not sure why the queue is so long.. | 09:25 |
| d0ugal | sshnaidm: how did you find that? I can't see it? | 09:26 |
| sshnaidm | d0ugal, logs from upgrades job: http://logs.openstack.org/71/332671/1/check-tripleo/gate-tripleo-ci-centos-7-upgrades/4deea89/ | 09:26 |
| d0ugal | sshnaidm: Thanks | 09:26 |
| sshnaidm | d0ugal, yeah, it's on http://status.openstack.org/zuul/ | 09:26 |
| *** limao has joined #tripleo | 09:27 | |
| sshnaidm | d0ugal, you can find your patch by ctrl-F nad then click on jobs, you'll see either running log or saved logs. | 09:27 |
| d0ugal | sshnaidm: right, thanks :) | 09:27 |
| *** sshnaidm is now known as sshnaidm|afk | 09:28 | |
| dciabrin | jistr, i agree, we would benefit from a dedicated patch/review for UC upgrade, where we could focus on how to best share code. | 09:28 |
| jistr | +1 | 09:28 |
| *** limao has quit IRC | 09:29 | |
| *** rasca has quit IRC | 09:29 | |
| *** limao has joined #tripleo | 09:29 | |
| matbu | dciabrin: ack | 09:30 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart: Add extra-vars for enable pacamaker to upgrade ci script https://review.openstack.org/334371 | 09:37 |
| *** bootsha has quit IRC | 09:47 | |
| *** tosky has joined #tripleo | 09:47 | |
| *** rasca has joined #tripleo | 09:48 | |
| *** osp has joined #tripleo | 09:57 | |
| *** rcernin has left #tripleo | 10:00 | |
| *** akrivoka has joined #tripleo | 10:00 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Free up more diskspace on jenkins slaves befor CI starts https://review.openstack.org/334383 | 10:01 |
| *** limao has quit IRC | 10:06 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 10:10 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Free up more diskspace on jenkins slaves befor CI starts https://review.openstack.org/334383 | 10:23 |
| derekh | shardy: RE^ , I'm not 100% sure what you meant, does my comment make sense or am I missing something ? | 10:24 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates: Availability monitoring support https://review.openstack.org/254788 | 10:25 |
| *** bootsha has joined #tripleo | 10:26 | |
| shardy | derekh: ah, I think I missed that opt/git is still removed, thanks | 10:26 |
| derekh | shardy: ok | 10:27 |
| amoralej | hi, it seems that https://github.com/openstack/tripleo-quickstart/commit/0d30b47966065ceabca54664cfdc1ad35f6d6944 has broken build-image quickstart ci jobs in RDO | 10:36 |
| amoralej | https://ci.centos.org/job/tripleo-quickstart-promote-master-delorean-build-images/365/console | 10:36 |
| *** Goneri has joined #tripleo | 10:39 | |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Add 'openstack overcloud node provide' command https://review.openstack.org/334411 | 10:40 |
| *** links has quit IRC | 10:43 | |
| *** gfidente has joined #tripleo | 10:47 | |
| *** fragatina has quit IRC | 10:48 | |
| *** fragatina has joined #tripleo | 10:49 | |
| jpich | d0ugal: ^ When you have a few moments, I could use your wisdom! | 10:51 |
| *** links has joined #tripleo | 10:55 | |
| *** Goneri has quit IRC | 10:58 | |
| *** _milan_ has joined #tripleo | 11:03 | |
| *** ccamacho is now known as ccamacho|lunch | 11:03 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 11:10 |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| *** Goneri has joined #tripleo | 11:11 | |
| *** Goneri has quit IRC | 11:18 | |
| *** zaneb has quit IRC | 11:20 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Gnocchi composable roles https://review.openstack.org/318413 | 11:20 |
| openstackgerrit | Keith Schincke proposed openstack/puppet-tripleo: Add RGW to the Ceph mon profile. https://review.openstack.org/334081 | 11:25 |
| *** weshay_afk is now known as weshay | 11:26 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Add gnocchi profiles https://review.openstack.org/315527 | 11:28 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Gnocchi composable roles https://review.openstack.org/318413 | 11:29 |
| *** sshnaidm|afk is now known as sshnaidm | 11:30 | |
| *** jefrite has quit IRC | 11:31 | |
| *** bfournie1 has quit IRC | 11:32 | |
| sshnaidm | derekh, hi | 11:34 |
| sshnaidm | derekh, ping me please when you're available for bluejeans chat | 11:34 |
| *** jefrite has joined #tripleo | 11:35 | |
| derekh | Nice, you can now telnet to running jenkins slaves to get the console log | 11:35 |
| derekh | sshnaidm: ready when you are | 11:35 |
| derekh | sshnaidm: sending you a lin | 11:36 |
| derekh | k | 11:36 |
| sshnaidm | derekh, ok | 11:36 |
| *** lblanchard has joined #tripleo | 11:38 | |
| *** hjensas has joined #tripleo | 11:41 | |
| *** akshai has joined #tripleo | 11:46 | |
| EmilienM | hello | 11:46 |
| *** akshai has quit IRC | 11:48 | |
| EmilienM | derekh, sshnaidm: how is CI going? did we track down space issue? | 11:48 |
| EmilienM | ok I see 334383 | 11:50 |
| EmilienM | everyone: current status of CI is also broken because we need https://review.openstack.org/#/c/333511/ to land | 11:50 |
| *** zoliXXL is now known as zoli|lunch | 11:53 | |
| *** fultonj has joined #tripleo | 11:54 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: fix tripleo-roles after explicit teardown change https://review.openstack.org/334433 | 11:54 |
| *** jpena is now known as jpena|lunch | 11:56 | |
| *** lucasagomes is now known as lucas-hungry | 11:59 | |
| *** rhallisey has joined #tripleo | 12:00 | |
| *** trown is now known as trown|outtypewww | 12:04 | |
| *** rodrigods has quit IRC | 12:06 | |
| *** rodrigods has joined #tripleo | 12:06 | |
| EmilienM | derekh: [Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" | 12:08 |
| EmilienM | seen in zuul for 334383 (nonha job) | 12:08 |
| sshnaidm | EmilienM, this patch should solve "no hosts"? | 12:08 |
| *** jcoufal has joined #tripleo | 12:08 | |
| EmilienM | no | 12:08 |
| EmilienM | it's for space | 12:08 |
| EmilienM | the thing that might help to fix "no hosts found" has been merged in Nova recently | 12:09 |
| EmilienM | but I don't know how updated is our OpenStack that deploy tripleo jobs | 12:09 |
| derekh | EmilienM: and whats breaking that needs this fix? https://review.openstack.org/#/c/333511/ | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 12:10 |
| *** ccamacho|lunch is now known as ccamacho | 12:10 | |
| *** hewbrocca-afk is now known as hewbrocca | 12:10 | |
| *** amoralej is now known as amoralej|lunch | 12:11 | |
| EmilienM | derekh: last week we had a major outage where delorean was not failing in our CI so we deployed THT from current packages and CI was not failing during 1 day, while it should have, on some patches that landed | 12:12 |
| EmilienM | derekh: so we had to revert some work | 12:12 |
| EmilienM | derekh: and this https://review.openstack.org/#/c/333511/ patch is fixing some issues on overcloud | 12:12 |
| EmilienM | derekh: https://review.openstack.org/#/c/334383/2/toci_gate_test.sh | 12:14 |
| EmilienM | why don't rm /opt/stack/cache/files/ubuntu-* ? | 12:14 |
| EmilienM | we'll hit the problem probably with ubuntu 14.04 too | 12:14 |
| derekh | EmilienM: I only went for the files that we're there but ya we should future proof it a bit | 12:16 |
| derekh | EmilienM: once the CI run is finished I'll update it | 12:16 |
| derekh | EmilienM: so, are you saying that even with the disk space issue fixed, still nothing will pass until we merge https://review.openstack.org/#/c/333511/ | 12:17 |
| *** bfournie has joined #tripleo | 12:18 | |
| EmilienM | derekh: maybe not, I've seen different failures over the last 3 days | 12:18 |
| EmilienM | http://logs.openstack.org/83/334383/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/005becd/logs/postci.txt.gz#_2016-06-27_12_09_00_000 | 12:19 |
| EmilienM | Message: No valid host was found. There are not enough hosts available., Code: 500 | 12:19 |
| EmilienM | derekh: this is on your tripleo-ci patch for free space ^ | 12:19 |
| EmilienM | derekh: do we have another problem? | 12:19 |
| derekh | EmilienM: maybe, I havn't been keeping track lately, trying to have rh2 ready for next week | 12:20 |
| *** pradk has joined #tripleo | 12:21 | |
| derekh | EmilienM: want to just merge the disk space fix and then we'll see if anything starts passing? | 12:21 |
| EmilienM | derekh: I can +A it | 12:21 |
| EmilienM | I was waiting for at least one green jo | 12:21 |
| EmilienM | job* | 12:21 |
| derekh | EmilienM: yup, we can wait either, the ha jobs is still running | 12:22 |
| EmilienM | derekh: yeah but I think it's fine, your patch was already executed and it worked | 12:23 |
| *** jayg|g0n3 is now known as jayg | 12:23 | |
| EmilienM | derekh: so I think we can go ahead. Ok? | 12:23 |
| derekh | EmilienM: ok, go for it | 12:23 |
| *** bfournie has quit IRC | 12:23 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: L->M upgrades keystone change of path of the paste_deploy config_file https://review.openstack.org/334044 | 12:23 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Free up more diskspace on jenkins slaves befor CI starts https://review.openstack.org/334383 | 12:23 |
| EmilienM | derekh: ok done ^ | 12:24 |
| *** bfournie has joined #tripleo | 12:24 | |
| derekh | EmilienM: sshnaidm has put a patch on our geard server to solve the problems getting testenvs this seems to be working | 12:24 |
| EmilienM | let's recheck 333511 now | 12:24 |
| EmilienM | and see if it pass CI | 12:24 |
| EmilienM | we'll need it asap too | 12:24 |
| derekh | EmilienM: so we shouldn't have to keep restarting geard | 12:25 |
| EmilienM | derekh: link? | 12:25 |
| derekh | EmilienM: I don't think he has push it to gerrit yet, sshnaidm is the patch anywhere? | 12:25 |
| EmilienM | derekh: btw, did you see https://review.openstack.org/#/c/333419/ ? It should help to stop leaking floating ips | 12:25 |
| sshnaidm | derekh, EmilienM - no, just in server, I'd like to check it before preparing to submit | 12:26 |
| derekh | EmilienM: I think maybe that this may help the "not enough hosts available" problem, as the restarting may have caused testenvs to be used by two jobs simultaneously | 12:27 |
| *** rbrady has quit IRC | 12:27 | |
| EmilienM | derekh: mhh, why do we have it since recently? | 12:28 |
| derekh | EmilienM: yup, I saw it, thanks | 12:28 |
| *** rbrady has joined #tripleo | 12:28 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Drop unused VIP params to controller.yaml https://review.openstack.org/333511 | 12:28 |
| derekh | EmilienM: I have a theory its related to the switch to ZUUL (from jenkins) , but its only a theory | 12:28 |
| derekh | EmilienM: this switch http://lists.openstack.org/pipermail/openstack-dev/2016-June/097584.html | 12:29 |
| *** derekh is now known as derekh_afk | 12:29 | |
| *** jpena|lunch is now known as jpena | 12:29 | |
| sshnaidm | derekh_afk, EmilienM : sort of https://review.openstack.org/#/c/334452 | 12:30 |
| *** rasca has quit IRC | 12:30 | |
| EmilienM | sshnaidm: interesting | 12:30 |
| *** rasca has joined #tripleo | 12:32 | |
| *** wfoster is now known as wfoster_afk | 12:32 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Re-enable Ceilometer composable roles for controller https://review.openstack.org/333498 | 12:35 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: fix tripleo-roles after explicit teardown change https://review.openstack.org/334433 | 12:35 |
| *** snecklifter has joined #tripleo | 12:36 | |
| *** numans has quit IRC | 12:37 | |
| *** fultonj has quit IRC | 12:39 | |
| *** limao has joined #tripleo | 12:40 | |
| *** bootsha has quit IRC | 12:41 | |
| *** wfoster_afk is now known as wfoster | 12:41 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Run post deploy config on force https://review.openstack.org/330096 | 12:42 |
| *** tzumainn has joined #tripleo | 12:45 | |
| *** rlandy has joined #tripleo | 12:46 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-puppet-elements: Install osops-tools-monitoring-oschecks package https://review.openstack.org/324075 | 12:47 |
| pradk | thx marios :) | 12:51 |
| *** akshai has joined #tripleo | 12:51 | |
| *** bfournie has left #tripleo | 12:51 | |
| *** amoralej|lunch is now known as amoralej | 12:56 | |
| marios | pradk: np thanks jistr :) | 12:57 |
| marios | pradk: (he hassles people for reviews in his spare time) | 12:58 |
| jistr | yea that's my hobby | 12:58 |
| *** bfournie has joined #tripleo | 12:59 | |
| *** jprovazn has quit IRC | 13:00 | |
| *** coolsvap has quit IRC | 13:00 | |
| *** derekh_afk is now known as derekh | 13:01 | |
| EmilienM | thrash and other folks: no need to do 'recheck' | 13:04 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Fix openstack-tripleo-common package name https://review.openstack.org/334467 | 13:04 |
| EmilienM | the CI is currently broken again | 13:05 |
| *** akshai has quit IRC | 13:05 | |
| EmilienM | we'll need 333511 probably | 13:05 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 13:06 |
| *** akshai has joined #tripleo | 13:09 | |
| *** lucas-hungry is now known as lucasagomes | 13:09 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/tripleo-quickstart: Fix build-images.yml after teardown change https://review.openstack.org/334472 | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596390 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1596390 in tripleo "qemu-img: No space left on device" [Critical,Confirmed] | 13:10 |
| EmilienM | jistr, marios: can you quickly review https://review.openstack.org/#/c/333347/ ? (It's a Gem change, so don't look functional jobs) | 13:11 |
| EmilienM | derekh: removing alert on the bug ^ | 13:11 |
| derekh | EmilienM: ok | 13:11 |
| marios | EmilienM: looking | 13:12 |
| *** sshnaidm has quit IRC | 13:12 | |
| *** trozet has joined #tripleo | 13:12 | |
| *** [1]cdearborn has joined #tripleo | 13:13 | |
| *** zaneb has joined #tripleo | 13:14 | |
| *** sshnaidm has joined #tripleo | 13:16 | |
| *** zoli|lunch is now known as zoli|wfh | 13:22 | |
| *** limao has quit IRC | 13:25 | |
| *** limao has joined #tripleo | 13:26 | |
| amoralej | weshay, question about https://review.openstack.org/#/c/334472/ | 13:26 |
| amoralej | why "role: tripleo-inventory" is now required? | 13:26 |
| weshay | amoralej, depends if the ansible_ssh_args are set | 13:27 |
| weshay | if ansible is expecting ssh.config.ansible then you'll need to run it, if it's not set.. it should work fine as is | 13:28 |
| weshay | amoralej, you can join #oooq | 13:28 |
| *** skramaja has quit IRC | 13:29 | |
| *** myoung has joined #tripleo | 13:30 | |
| *** xinwu has joined #tripleo | 13:30 | |
| *** weshay is now known as weshay_mtg | 13:31 | |
| *** pradk has quit IRC | 13:31 | |
| *** egafford has joined #tripleo | 13:32 | |
| *** limao_ has joined #tripleo | 13:33 | |
| *** limao_ has quit IRC | 13:34 | |
| *** limao_ has joined #tripleo | 13:35 | |
| *** jprovazn has joined #tripleo | 13:35 | |
| *** limao has quit IRC | 13:37 | |
| *** xinwu has quit IRC | 13:37 | |
| *** ebalduf has joined #tripleo | 13:39 | |
| *** andreas-f has joined #tripleo | 13:40 | |
| *** skramaja has joined #tripleo | 13:43 | |
| *** limao has joined #tripleo | 13:44 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/python-tripleoclient: Run post deploy config on force https://review.openstack.org/334486 | 13:45 |
| *** pradk has joined #tripleo | 13:45 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/python-tripleoclient: Run post deploy config on force https://review.openstack.org/334486 | 13:47 |
| *** limao_ has quit IRC | 13:47 | |
| *** eggmaster has joined #tripleo | 13:47 | |
| pradk | jistr, marios, backport to mitaka https://review.openstack.org/#/c/334486/ | 13:47 |
| *** r-mibu has quit IRC | 13:48 | |
| *** r-mibu has joined #tripleo | 13:48 | |
| *** skramaja has quit IRC | 13:55 | |
| *** tosky has quit IRC | 13:56 | |
| *** bootsha has joined #tripleo | 13:56 | |
| *** akshai has quit IRC | 13:59 | |
| chem | hi, I tryied a quickstart --release liberty and it fails. It looks like packages in the undercloud.qcows conflict with package installed by instack during undercloud step. Does someone could help ? | 13:59 |
| *** fultonj has joined #tripleo | 14:02 | |
| *** fultonj has quit IRC | 14:02 | |
| chem | panda: this is the issue I've got. So is there a way around it ? | 14:02 |
| *** fultonj has joined #tripleo | 14:03 | |
| *** akshai has joined #tripleo | 14:03 | |
| shardy | chem: trown|outtypewww or larsks may be able to help | 14:03 |
| chem | shardy: ack, thanks | 14:04 |
| *** liverpooler has quit IRC | 14:05 | |
| social | matbu: solution is to have cleanup in undercloud puppet for keystone and mariadb | 14:07 |
| *** skramaja has joined #tripleo | 14:10 | |
| matbu | social: yes i saw your review | 14:11 |
| matbu | social: do you have a WIP review for that ? | 14:11 |
| social | I just managed to reproduce the issue so, soon :) | 14:11 |
| matbu | social: ack, you want to remove the "yum update" there also ? | 14:12 |
| shardy | Anyone seeing local delorean builds fail with Error: No Package found for $foo? | 14:12 |
| shardy | I've deleted ~/tripleo/delorean and re-run tripleo.sh --delorean-setup but clearly still have some thing stale or the spec/repo is broken | 14:13 |
| shardy | I'm trying to build openstack/heat | 14:13 |
| *** bootsha has quit IRC | 14:13 | |
| shardy | alternatively, anyone got a link to the location where delorean builds for under-review patches are created? | 14:13 |
| shardy | bnemec: IIRC you linked it to me a few weeks ago, but I'm failing to find it atm | 14:13 |
| trozet | shardy: do you have any update on if composable services are done for neutron yet? | 14:14 |
| openstackgerrit | Alfredo Moralejo proposed openstack/tripleo-quickstart: Fix build-images.yml after teardown change https://review.openstack.org/334472 | 14:16 |
| bnemec | It's https://repos.fedorapeople.org/repos/openstack-m/jenkins/rpm-build/ but that job doesn't run for Heat atm. | 14:16 |
| bnemec | shardy: ^ | 14:16 |
| bnemec | It's also broken for a couple of projects because some of the packaging repos disappeared. :-/ | 14:16 |
| *** rajinir has joined #tripleo | 14:20 | |
| EmilienM | trozet: https://etherpad.openstack.org/p/tripleo-composable-services | 14:20 |
| EmilienM | trozet: we made some progress on controller, but not on compute, it's WIP | 14:20 |
| EmilienM | trozet: progress is really slow because of CI issues that we have every day | 14:20 |
| trozet | EmilienM: thanks for the link. What do you mean compute, that's just openvswitch agent right? | 14:21 |
| ccamacho | Hey EmilienM pradk, quick question, can you check https://review.openstack.org/#/c/318413/ ?? I found the problem in the deployment, is related to ::gnocchi::db::sync when using it the deployment doesn't work, not sure if can be related to https://review.openstack.org/#/c/295944/ | 14:21 |
| EmilienM | ccamacho: right now CI is broken so I'll look once we have valid failures in logs | 14:21 |
| EmilienM | trozet: not only, plugin conf for some drivers that require it | 14:21 |
| pradk | ccamacho, that patch went in a while back and it only skips storage sync.. so should cause issues | 14:22 |
| pradk | ccamacho, any tracebacks in upgrade.log? | 14:22 |
| pradk | shouldnt* | 14:22 |
| *** ayoung has joined #tripleo | 14:22 | |
| *** limao has quit IRC | 14:22 | |
| trozet | EmilienM: I'm not following. What drivers? | 14:23 |
| *** limao has joined #tripleo | 14:23 | |
| trozet | I see TODO: evacuate neutron bits from compute (WIP emilien) | 14:23 |
| EmilienM | trozet: bigswitch, etc | 14:23 |
| EmilienM | trozet: yeah, but there is a bit more, look in the code, there are still some neutron bit | 14:24 |
| trozet | EmilienM: ok will take a look, thanks | 14:24 |
| ccamacho | EmilienM sure, those error are from my local CI. pradk the deployment is not failing in the upgrade, it is failing when deploying the non-ha job locally. Here: https://review.openstack.org/#/c/315527/35/manifests/profile/base/gnocchi/api.pp I have commented the line to see the behavior and the deployment completes when not using it, but not sure, if needed or nor, or if it is an error in the puppet gnocchi manifest | 14:25 |
| *** jschlueter has quit IRC | 14:27 | |
| *** limao_ has joined #tripleo | 14:27 | |
| *** adarazs has quit IRC | 14:29 | |
| *** aufi has quit IRC | 14:29 | |
| *** jmiu_ has quit IRC | 14:30 | |
| social | matbu: /q matbu | 14:30 |
| social | err :) | 14:30 |
| matbu | social: hehe | 14:31 |
| *** limao has quit IRC | 14:31 | |
| *** rook_ has joined #tripleo | 14:32 | |
| *** jschlueter has joined #tripleo | 14:33 | |
| shardy | bnemec: ack, thanks - I'll actually bookmark it this time ;) | 14:33 |
| EmilienM | ccamacho, pradk: about db-sync, do we need to patch puppet-gnocchi or? | 14:34 |
| *** ebarrera has quit IRC | 14:34 | |
| *** adarazs has joined #tripleo | 14:34 | |
| pradk | EmilienM, dont think so.. this is not an issue in packstack or in our tripleo ci.. i think the composable roles patch is failing on something else .. I'll need to look at the gnocchi-upgrade traceback to see what it is | 14:35 |
| *** jmiu_ has joined #tripleo | 14:36 | |
| ccamacho | pradk do you need any log file from my CI? | 14:36 |
| pradk | ccamacho, can you paste me /var/log/gnocchi/gnocchi-upgrade.log | 14:37 |
| ccamacho | sure, just a sec | 14:37 |
| *** weshay_mtg is now known as weshay | 14:39 | |
| ccamacho | pradk that file does not exists, no file in /var/log/gnocchi/* in the OC controller, this is the config file http://paste.openstack.org/show/523583/ | 14:42 |
| ccamacho | pradk is all commented, maybe that's the issue.. | 14:43 |
| pradk | ccamacho, yea thats weird, there is no db connection so db sync will obviously fail.. wonder if there were any other errors before this | 14:44 |
| *** akshai has quit IRC | 14:46 | |
| *** limao_ has quit IRC | 14:47 | |
| *** ayoung_ has joined #tripleo | 14:48 | |
| *** limao has joined #tripleo | 14:48 | |
| ccamacho | pradk not really, but actually, we have those default parameters in the tht submission, so, let me see if they are not passed correctly to the puppet manifests. | 14:48 |
| EmilienM | derekh: why did you revert? | 14:48 |
| EmilienM | https://review.openstack.org/#/c/334516/ | 14:48 |
| *** jmiu_ has quit IRC | 14:49 | |
| derekh | EmilienM: testing, I think it may be causing the current problems with deploying tripleo | 14:49 |
| pradk | ccamacho, did you update the tht after my changes to split the gnocchi services further? | 14:49 |
| pradk | ccamacho, thats possible | 14:49 |
| *** adarazs has quit IRC | 14:49 | |
| *** karts is now known as karthiks | 14:49 | |
| EmilienM | derekh: we don't deploy bigswitch, I don't see how | 14:49 |
| *** jschlueter has quit IRC | 14:49 | |
| derekh | EmilienM: I went comparing CI runs from today to those that worked from last week | 14:49 |
| ccamacho | let me check it as is quite possible that the parameters were empty when executing the puppet mainfest | 14:49 |
| *** limao_ has joined #tripleo | 14:49 | |
| derekh | EmilienM: the ones from today install python-networking-bigswitch on the undercloud | 14:50 |
| EmilienM | on undercloud? | 14:50 |
| derekh | EmilienM: tracback on neutron on the undercloud http://paste.openstack.org/show/523585/ | 14:50 |
| EmilienM | puppet doesn't manage bigswitch on underclout | 14:50 |
| *** limao_ has quit IRC | 14:51 | |
| EmilienM | it's not related to bigswitch | 14:51 |
| *** akshai has joined #tripleo | 14:51 | |
| *** limao_ has joined #tripleo | 14:51 | |
| derekh | EmilienM: actually maybe it isn't puppet, looking elsewhere | 14:51 |
| *** adarazs has joined #tripleo | 14:53 | |
| panda | do you have a link to tripleo meeting agenda ? | 14:53 |
| shardy | https://wiki.openstack.org/wiki/Meetings/TripleO | 14:55 |
| shardy | panda: ^^ | 14:55 |
| *** limao has quit IRC | 14:55 | |
| EmilienM | derekh: do you have a link of the patch that fails? | 14:55 |
| *** jschluet has joined #tripleo | 14:55 | |
| EmilienM | derekh: I have telnet open on 2 different jobs and I passed undercloud | 14:55 |
| EmilienM | derekh: overcloud is being deployed | 14:55 |
| *** jschluet is now known as jschlueter | 14:55 | |
| derekh | EmilienM: ya and the exception I'm seeing is during the overcloud deploy, in the undercloud neutron | 14:55 |
| *** jschlueter has quit IRC | 14:55 | |
| *** jschlueter has joined #tripleo | 14:55 | |
| EmilienM | mhh | 14:55 |
| *** jmiu_ has joined #tripleo | 14:55 | |
| *** adarazs has quit IRC | 14:55 | |
| *** adarazs has joined #tripleo | 14:55 | |
| EmilienM | derekh: any link? | 14:55 |
| *** crinkle_ is now known as crinkle | 14:55 | |
| derekh | EmilienM: http://logs.openstack.org/27/315527/35/check-tripleo/gate-tripleo-ci-centos-7-nonha/35b4723/logs/undercloud/var/log/neutron/server.txt.gz#_2016-06-27_13_16_06_499 | 14:56 |
| *** limao_ has quit IRC | 14:57 | |
| *** limao has joined #tripleo | 14:57 | |
| EmilienM | derekh: is it blocking something? | 14:57 |
| *** itamarl has quit IRC | 14:58 | |
| *** osp has quit IRC | 14:58 | |
| *** zaneb has quit IRC | 14:59 | |
| derekh | EmilienM: So, I think something to do with the bigswitch package is causing the "Message: No valid host was found. There are not enough hosts available" problem | 14:59 |
| derekh | EmilienM: I just don't know yet what / how | 14:59 |
| EmilienM | CREATE_FAILED ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" | 15:00 |
| EmilienM | again | 15:00 |
| derekh | EmilienM: exactly | 15:00 |
| EmilienM | I'm not sure why it's related to bigswitch | 15:00 |
| EmilienM | I don't see any bigswitch thing in logs | 15:01 |
| derekh | EmilienM: console, logs http://logs.openstack.org/27/315527/35/check-tripleo/gate-tripleo-ci-centos-7-nonha/35b4723/console.html#_2016-06-27_11_57_02_879668 | 15:01 |
| EmilienM | jistr: you can safely merge https://review.openstack.org/#/c/333347/ please | 15:01 |
| EmilienM | jistr: it's a Gemfile patch, for lint & syntax jobs | 15:01 |
| derekh | EmilienM: its on the undercloud cached image, somehow | 15:01 |
| derekh | EmilienM: I'm still looking into it | 15:01 |
| EmilienM | derekh: and? | 15:01 |
| EmilienM | derekh: a lot of many other pkgs are updated too | 15:01 |
| *** jistr is now known as jistr|mtg | 15:01 | |
| *** limao has quit IRC | 15:02 | |
| bnemec | derekh: It's a glance error. http://logs.openstack.org/79/329079/4/check-tripleo/gate-tripleo-ci-centos-7-nonha/0ec8e2d/logs/undercloud/var/log/glance/api.txt.gz#_2016-06-27_13_38_16_395 | 15:02 |
| bnemec | Looks like the same thing you were seeing in rh2. | 15:03 |
| EmilienM | ah it looks better | 15:03 |
| derekh | bnemec: so is swift getting OOM'd again? | 15:03 |
| bnemec | derekh: I don't see any ooms, but it's the same error on the Glance side so something is wrong there. | 15:04 |
| derekh | bnemec: Jun 27 15:12:06 instack.localdomain kernel: Out of memory: Kill process 380 (swift-proxy-ser) score 374 or sacrifice child | 15:04 |
| derekh | bnemec: in ther joulan log for the one you just linked | 15:05 |
| bnemec | Okay, apparently I fail at reading logs. | 15:05 |
| *** krotscheck is now known as krotscheck_dcm | 15:05 | |
| bnemec | Oh, I didn't actually pull down the journal. | 15:05 |
| bnemec | Fscking binary log formats. | 15:05 |
| derekh | brb | 15:07 |
| *** skramaja has quit IRC | 15:09 | |
| EmilienM | bnemec: isn't something we had in March? | 15:09 |
| EmilienM | we needed extra RAM iirc and we have it now | 15:09 |
| EmilienM | slagle was also mentionning high CPU usage, maybe should we reduce testenvs on each host | 15:09 |
| *** ayoung has quit IRC | 15:09 | |
| *** ayoung_ is now known as ayoung | 15:09 | |
| *** sshnaidm is now known as sshnaidm|afk | 15:09 | |
| slagle | EmilienM: that was a while ago | 15:10 |
| slagle | EmilienM: pre-upgrade | 15:10 |
| slagle | we've added ram and ssd's since then | 15:11 |
| EmilienM | right, but it seems like we're having a similar situation | 15:11 |
| d0ugal | jpich: Commented on your patch | 15:11 |
| bnemec | The problem derekh was seeing in rh2 is that swift was using absurd amounts of ram. I don't think we could add enough memory to the CI nodes to get around that. | 15:11 |
| jpich | d0ugal: Cheers! | 15:12 |
| EmilienM | bnemec: maybe we have too much swift proxy workers | 15:13 |
| *** osp has joined #tripleo | 15:13 | |
| *** ebarrera has joined #tripleo | 15:13 | |
| derekh | something is causing swift-proxy to use more RAM then it should, we need to figure out what | 15:13 |
| derekh | EmilienM: workers = 2 | 15:13 |
| EmilienM | 2 is not bad | 15:14 |
| bnemec | Yeah, we're not talking about a little too much ram. We're talking about swift eating 80 GB when it's only being asked to serve a single image. | 15:14 |
| bnemec | (this was in rh2, which may or may not be related, but probably is) | 15:15 |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Add gnocchi profiles https://review.openstack.org/315527 | 15:15 |
| *** jistr|mtg is now known as jistr | 15:16 | |
| *** fragatina has quit IRC | 15:17 | |
| pradk | ccamacho, found the issue i assume? i see you uncommented sync | 15:17 |
| *** fragatina has joined #tripleo | 15:17 | |
| *** dsariel has quit IRC | 15:19 | |
| ccamacho | pradk Yeahp, Im debugging the THT side, as the service is not configured at all.. from what we spoke before, the issue should be about the service config, now Im deploying it locally to see if how the params are passed. | 15:19 |
| ccamacho | pradk ill let you know when getting the result | 15:19 |
| pradk | ok cool | 15:19 |
| *** bvandenh has quit IRC | 15:20 | |
| *** yamahata has joined #tripleo | 15:21 | |
| *** saneax is now known as saneax_AFK | 15:24 | |
| *** coolsvap has joined #tripleo | 15:24 | |
| *** weshay is now known as weshay_brb | 15:25 | |
| *** ebarrera has quit IRC | 15:28 | |
| *** ebarrera has joined #tripleo | 15:28 | |
| jistr | EmilienM: re https://review.openstack.org/#/c/333347/ i'm not fan of merging all-red patches unless they are very important for some reason. BTW if we decide to really ignore the CI on that one, you can also +A yourself. I think it's ok to +A any patch (even own patch) if there are two +2s from someone else. | 15:28 |
| derekh | bnemec: EmilienM Jun 27 14:26:52 instack.localdomain kernel: Killed process 21018 (swift-proxy-ser) total-vm:3672584kB, anon-rss:3381984kB, file-rss:1124kB | 15:28 |
| derekh | EmilienM: bnemec if I'm reading that corrently swift is using >3GB of RAM | 15:29 |
| bnemec | Yeah, that's ridiculous. | 15:29 |
| *** dmk0202 has quit IRC | 15:30 | |
| EmilienM | jistr: I'm not a fan either, but this patch has really no impact. As you probably already know, Gemfile is used when running 'bundle install' and 'rake exec' | 15:30 |
| EmilienM | jistr: +A myself | 15:30 |
| derekh | bnemec: EmilienM I'll dig at it a bit later, gotta go for a while, if anybody is looking into it and find anything out can ye email me | 15:31 |
| dtantsur | EmilienM++ it's strange to not merge patches not touching gate at all | 15:31 |
| *** weshay_brb is now known as weshay | 15:31 | |
| bnemec | Gah, my laptop is so slow when it's applying updates. | 15:31 |
| *** derekh is now known as derekh_afk | 15:31 | |
| dtantsur | I remember a ridiculous number of rechecks on a doc-only patch once... | 15:31 |
| EmilienM | dtantsur: functional jobs are not in gate | 15:31 |
| bnemec | Yeah, if it's a unit test-only change like that, I don't wait for tripleo-ci. | 15:32 |
| EmilienM | bnemec: it seems like we have a successful job here https://review.openstack.org/#/c/329504/2 | 15:32 |
| EmilienM | where all 2 jobs are working | 15:32 |
| EmilienM | err, 3 | 15:32 |
| *** noslzzp has joined #tripleo | 15:33 | |
| bnemec | Yeah, there are a handful of passes on the ci status page so _sometimes_ we're not hitting this. | 15:33 |
| bnemec | I think derek couldn't reproduce it in a local environment either. | 15:34 |
| *** noslzzp has quit IRC | 15:34 | |
| *** panda has quit IRC | 15:34 | |
| bnemec | Because of course not. Race bugs never reproduce in an easily debuggable environment. :-) | 15:34 |
| *** panda has joined #tripleo | 15:35 | |
| *** noslzzp has joined #tripleo | 15:35 | |
| EmilienM | I also see some: | 15:37 |
| EmilienM | ResourceInError: resources.Compute.resources[0].resources.NovaCompute: Went to status ERROR due to "Message: Unknown, Code: Unknown" | 15:37 |
| EmilienM | in some other jobs | 15:37 |
| *** shivrao has joined #tripleo | 15:37 | |
| *** shivrao has quit IRC | 15:37 | |
| openstackgerrit | Merged openstack/instack-undercloud: Revert "Pin puppet-lint-absolute_classname-check to 0.1.3" https://review.openstack.org/333347 | 15:38 |
| *** pcaruana has quit IRC | 15:38 | |
| EmilienM | and lot of glance api things | 15:39 |
| EmilienM | http://logs.openstack.org/21/327721/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/3df6467/logs/undercloud/var/log/glance/api.txt.gz#_2016-06-27_13_17_43_114 | 15:39 |
| EmilienM | bnemec: have you seen this one? ^ related to swift too | 15:39 |
| EmilienM | probably same error as you had with proxy going oom | 15:39 |
| *** tesseract- has quit IRC | 15:40 | |
| EmilienM | Out of memory: Kill process 21030 (swift-proxy-ser) score 411 or sacrifice child | 15:42 |
| EmilienM | ok same thing indeed | 15:42 |
| *** bnemec has quit IRC | 15:43 | |
| *** bnemec has joined #tripleo | 15:43 | |
| *** bnemec has quit IRC | 15:50 | |
| *** osp has quit IRC | 15:51 | |
| *** bnemec has joined #tripleo | 15:52 | |
| EmilienM | I don't see anything wrong in netstat -lpn on undercloud (eventual Swift proxy memory leak) | 15:52 |
| *** links has quit IRC | 15:52 | |
| *** tosky has joined #tripleo | 15:52 | |
| EmilienM | bnemec: not only proxy | 15:54 |
| EmilienM | also object service | 15:54 |
| EmilienM | look in top -n 1 -b -o RES | 15:54 |
| EmilienM | swift object is taking 1410380 in virtual size, x2 (2 processes) | 15:54 |
| EmilienM | and proxy takes 3088892 :-O | 15:54 |
| *** shivrao has joined #tripleo | 15:55 | |
| *** shivrao_ has joined #tripleo | 15:56 | |
| *** cschwede has joined #tripleo | 15:57 | |
| EmilienM | cschwede: hey | 15:57 |
| cschwede | EmilienM: hi | 15:57 |
| EmilienM | cschwede: so we're seeing a lot of OOM on swift-proxy-server since super recently | 15:57 |
| EmilienM | we're investigating why but for now, we don't have any idea | 15:58 |
| EmilienM | let me show you some logs for example | 15:58 |
| cschwede | how much memory do you have on the nodes? | 15:58 |
| *** zoli|wfh is now known as zoli|gone | 15:58 | |
| EmilienM | cschwede: 5.8G for undercloud | 15:59 |
| EmilienM | cschwede: if you want, you can download http://logs.openstack.org/21/327721/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/3df6467/logs/undercloud.tar.xz | 15:59 |
| *** zoli|gone is now known as zoli_gone-proxy | 15:59 | |
| *** osp has joined #tripleo | 15:59 | |
| *** shivrao has quit IRC | 15:59 | |
| *** shivrao_ is now known as shivrao | 15:59 | |
| EmilienM | and see Jun 27 09:14:13 instack.localdomain kernel: Out of memory: Kill process 21030 (swift-proxy-ser) score 411 or sacrifice child | 16:00 |
| EmilienM | after running journalctl --file var/log/journal/027a07d166557094b759a31ef1ce5c65/system.journal | 16:00 |
| EmilienM | cschwede: we noticed swift proxy takes insane amount of RAM | 16:00 |
| *** aufi has joined #tripleo | 16:00 | |
| cschwede | what do you mean by insane? | 16:00 |
| *** jpena is now known as jpena|off | 16:01 | |
| EmilienM | cschwede: > 3GB | 16:01 |
| cschwede | that’s not much for a proxy | 16:02 |
| cschwede | does that happen when up- or downloading? | 16:02 |
| EmilienM | 675 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND | 16:02 |
| EmilienM | 676 1062 swift 20 0 3088892 2.692g 1824 R 18.8 46.2 0:07.05 swift-prox+ | 16:02 |
| EmilienM | cschwede: let me look, I think it's downloading | 16:02 |
| EmilienM | because it's when deploying overcloud | 16:02 |
| EmilienM | but let me look again | 16:03 |
| EmilienM | http://logs.openstack.org/21/327721/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/3df6467/logs/undercloud/var/log/glance/api.txt.gz#_2016-06-27_14_03_07_620 | 16:03 |
| EmilienM | eventlet.wsgi.server [req-f741afae-33a7-4fac-b8a5-dcebe03d36f2 5d98f4907aef4ecd974672479562ef5f a287a9f2e7e648739e6eb25560acb740 - - -] 192.0.2.1 - - [27/Jun/2016 14:03:07] "GET /v1/images/4907e1bd-b808-4af1-b6bf-51b18a27dc62 HTTP/1.1" 500 454 214.623023 | 16:03 |
| EmilienM | downloading Swift -> Glance | 16:03 |
| cschwede | hmm, i think it happens at „Jun 27 09:14:13“ ? | 16:06 |
| cschwede | let me check | 16:06 |
| cschwede | the object above looks too small to me | 16:06 |
| EmilienM | lot of 404 though: http://logs.openstack.org/21/327721/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/3df6467/logs/undercloud/var/log/glance/api.txt.gz#_2016-06-27_13_12_19_556 | 16:08 |
| EmilienM | Registry client request GET /images/overcloud-full raised NotFound | 16:08 |
| cschwede | is there no swap on the undercloud? | 16:09 |
| cschwede | there is also a mysqld oom: Jun 27 09:14:13 instack.localdomain kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 | 16:09 |
| EmilienM | cschwede: 2GB | 16:09 |
| *** yamahata has quit IRC | 16:10 | |
| cschwede | this is the request that triggers the oom: | 16:11 |
| cschwede | Jun 27 09:13:18 instack.localdomain object-server[20769]: 192.0.2.1 - - [27/Jun/2016:13:13:18 +0000] "GET /1/153411/AUTH_a287a9f2e7e648739e6eb25560acb 740/glance/4907e1bd-b808-4af1-b6bf-51b18a27dc62" 200 3707895808 "GET http://192.0.2.1:8080/v1/AUTH_a287a9f2e7e648739e6eb25560acb740/glance/4907e1bd-b8 08-4af1-b6bf-51b18a27dc62" "tx6c8f58a4e2d04aa2a7d05-005771266d" "proxy-server 21030" 0.0009 "-" 20769 0 | 16:11 |
| cschwede | 3707895808 bytes ~ 3.5GB | 16:12 |
| cschwede | so if the client doesn’t consume the bytes fast enough, it’s hold in memory | 16:12 |
| cschwede | i suggest to increase ram or swap | 16:12 |
| *** hjensas has quit IRC | 16:13 | |
| shardy | we are resource constrained in both ram and walltime in CI, so neither option is very attractive | 16:13 |
| shardy | we could switch glance to be file-backed for the CI tests instead of swift backed? | 16:13 |
| EmilienM | shardy: yes, in the meantime we figure it out | 16:15 |
| EmilienM | shardy: but it's a testing regression | 16:15 |
| EmilienM | I mean: don't do that forever | 16:15 |
| EmilienM | cschwede: the client == glance api? | 16:16 |
| *** ayoung has quit IRC | 16:16 | |
| cschwede | EmilienM: i think so (whoever fetches that object) | 16:16 |
| shardy | EmilienM: Yup, although we might also consider if there is huge value in swift-backed glance when the swift is an all-in-one setup on the undercloud | 16:16 |
| EmilienM | shardy: do you want me to submit a patch to use file backend until we sort things out? | 16:17 |
| *** ebarrera has quit IRC | 16:17 | |
| shardy | EmilienM: yes that sounds like a good first step, thanks | 16:17 |
| EmilienM | k | 16:17 |
| *** ebarrera has joined #tripleo | 16:20 | |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: glance: disable swift backend https://review.openstack.org/334555 | 16:20 |
| EmilienM | cschwede: maybe there is a timeout parameter somewhere that we can use? | 16:21 |
| EmilienM | in glance config maybe | 16:21 |
| *** mcornea has quit IRC | 16:21 | |
| EmilienM | shardy: /me filing a bug and updating the patch so we don't loose track | 16:22 |
| cschwede | EmilienM: i will think of possible options and get back to you tomorrow if that’s ok | 16:23 |
| shardy | EmilienM: thanks | 16:24 |
| shardy | EmilienM: we should perhaps start a ML thread to discuss if we should leave glance file backed | 16:24 |
| shardy | it seems like swift is just a really expensive way to write to local disk in this particular case | 16:24 |
| EmilienM | https://bugs.launchpad.net/tripleo/+bug/1596604 | 16:25 |
| openstack | Launchpad bug 1596604 in tripleo "swift-proxy-server OOM" [Critical,Confirmed] - Assigned to Emilien Macchi (emilienm) | 16:25 |
| EmilienM | cschwede: can you read it and correct me if needed? ^ | 16:25 |
| shardy | that may have some upgrade implications tho | 16:25 |
| EmilienM | shardy: ok, I'll start it in a few min | 16:25 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: glance: disable swift backend https://review.openstack.org/334555 | 16:26 |
| bnemec | Note that swift memory usage running wild is causing us grief on the overcloud too. | 16:26 |
| *** coolsvap has quit IRC | 16:26 | |
| EmilienM | I didn't disable swift on the undercloud, just change the backend in Glance | 16:27 |
| bnemec | Maybe not in CI because of the tiny cirros image, but in real-world deployments it's OOMing. | 16:27 |
| EmilienM | AFIK some other bits also use swift on undercloud | 16:27 |
| EmilienM | bnemec: in our CI it's failing on the overcloud image download, so definitly a real world thing | 16:27 |
| bnemec | See also https://bugs.launchpad.net/tripleo/+bug/1595916 | 16:28 |
| openstack | Launchpad bug 1595916 in tripleo "Swift memory usage grows until it is killed" [High,New] | 16:28 |
| EmilienM | bnemec, shardy: I'll let you review https://review.openstack.org/#/c/334555/ as soon as you can (+2 for now, we'll +A if CI is passing) | 16:28 |
| cschwede | bnemec: that looks like a memory leak, 60-80GB should not happen | 16:28 |
| bnemec | That's in a baremetal environment with lots of ram. | 16:28 |
| *** coolsvap has joined #tripleo | 16:28 | |
| cschwede | alright, i will have a look into this | 16:29 |
| * EmilienM afk for lunch, back in 45 min | 16:30 | |
| *** ayoung has joined #tripleo | 16:31 | |
| *** jpich has quit IRC | 16:36 | |
| *** coolsvap has quit IRC | 16:37 | |
| tdasilva | cschwede, bnemec: could it be related to this: https://bugs.launchpad.net/cloud-archive/+bug/1493303 | 16:37 |
| openstack | Launchpad bug 1493303 in swift (Ubuntu Wily) "[OSSA 2016-004] Swift proxy memory leak on unfinished read (CVE-2016-0738)" [Undecided,Triaged] | 16:37 |
| openstackgerrit | Merged openstack/tripleo-quickstart: fix tripleo-roles after explicit teardown change https://review.openstack.org/334433 | 16:39 |
| *** shivrao has quit IRC | 16:43 | |
| bnemec | tdasilva: I don't think so. It looks like that should be fixed, and we're running against a very recent build of Swift. | 16:43 |
| *** athomas has quit IRC | 16:46 | |
| *** Guest59963 has quit IRC | 16:46 | |
| *** yamahata has joined #tripleo | 16:52 | |
| *** fultonj has quit IRC | 16:53 | |
| *** paramite has quit IRC | 17:02 | |
| *** aufi has quit IRC | 17:02 | |
| *** ayoung has quit IRC | 17:06 | |
| *** amoralej is now known as amoralej|off | 17:06 | |
| *** ebarrera has quit IRC | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596604 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1596604 in tripleo "swift-proxy-server OOM" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| *** stendulker has joined #tripleo | 17:11 | |
| *** lblanchard has quit IRC | 17:20 | |
| *** ayoung has joined #tripleo | 17:21 | |
| EmilienM | bnemec: ack for your comment on https://review.openstack.org/#/c/334555/ | 17:23 |
| EmilienM | bnemec: I'll update commit message once CI passed and before we land it ok? | 17:23 |
| *** penick has joined #tripleo | 17:24 | |
| bnemec | EmilienM: Yeah, that's what I had in mind. I didn't want to re-trigger CI by editing it at the time. | 17:24 |
| Slower_ | what are people doing to get around the image building repo breakage? | 17:25 |
| *** Slower_ is now known as SLower | 17:25 | |
| *** SLower is now known as Slower | 17:25 | |
| shardy | Slower: hey, what issues are you seeing? | 17:26 |
| shardy | Slower: I'm testing a change to tripleo.sh which modifies the delorean command after a discussion in #rdo | 17:26 |
| shardy | some packages were failing to build for me locally, and it seems it may have been due to the --build-env DELOREAN_DEV=1 here | 17:27 |
| shardy | https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L359 | 17:27 |
| Slower | openstack overcloud image build --all | 17:27 |
| shardy | Slower: ah, Ok, different issue | 17:28 |
| *** akuznetsov has joined #tripleo | 17:28 | |
| Slower | I think the repo has a redirect or something on it | 17:28 |
| Slower | File contains no section headers. | 17:28 |
| Slower | file: file:///etc/yum.repos.d/delorean.repo, line: 1 | 17:28 |
| Slower | '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n' | 17:28 |
| Slower | I've been messing with different ways to give it the repo but haven't found the magic yet | 17:29 |
| shardy | Slower: ah, you might try removing all the delorean repos.d entries, pulling latest tripleo-ci repo then re-running tripleo.sh --repo-setup | 17:29 |
| shardy | there was a recent change to tripleo.sh which makes it follow the redirect IIRC | 17:29 |
| Slower | ah ok | 17:30 |
| rhallisey | Slower, I tried that method recently had the same issue. Doing manual steps worked | 17:30 |
| Slower | thanks guys | 17:30 |
| shardy | https://github.com/openstack-infra/tripleo-ci/commit/8a461f7fdf4855c32ff510f11187c8727c45ee93 | 17:30 |
| *** olap has quit IRC | 17:33 | |
| *** penick_ has joined #tripleo | 17:33 | |
| *** penick has quit IRC | 17:35 | |
| *** penick_ is now known as penick | 17:35 | |
| Slower | doesn't dib get them itself during image build though? | 17:35 |
| Slower | the ones on teh host are fine.. | 17:35 |
| shardy | Slower: DIB_YUM_REPO_CONF points to the local host's /etc/yum.repos.d by default | 17:36 |
| shardy | you can override that by exporting a different REPO_PREFIX | 17:37 |
| Slower | well what the heck | 17:37 |
| EmilienM | bnemec: ok it fails, /me investigating | 17:37 |
| Slower | shardy: ok thx ;) | 17:37 |
| shardy | Slower: it actually works quite well, as for example you can export REPO_PREFIX then re-run --repo-setup | 17:38 |
| shardy | e.g if you want to build images for a stable version one a host configured to use trunk or whatever | 17:38 |
| EmilienM | bnemec, shardy: my patch to disable swift backend on undercloud is failing, http://logs.openstack.org/55/334555/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/925831b/logs/postci.txt.gz#_2016-06-27_17_28_26_000 | 17:40 |
| EmilienM | have you hit this message before? | 17:40 |
| Slower | shardy: oh I'm not dissing that part, that's great. I am wondering why it's not working for me | 17:41 |
| Slower | cause my repos locally are right | 17:41 |
| bnemec | EmilienM: Yes, but that isn't the problem: http://logs.openstack.org/55/334555/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/925831b/console.html#_2016-06-27_17_28_13_198988 | 17:42 |
| EmilienM | oh it's glance | 17:42 |
| shardy | Slower: tried removing ~/.cache/image-create ? | 17:42 |
| bnemec | Yeah | 17:42 |
| Slower | maybe I set some wierd env variable at some point.. new shell seems to be working | 17:43 |
| EmilienM | ok, digging | 17:43 |
| Slower | shardy: I'll try that next | 17:43 |
| *** oshvartz has quit IRC | 17:45 | |
| *** lblanchard has joined #tripleo | 17:46 | |
| bnemec | EmilienM: It looks like glance-api is constantly restarting. :-/ | 17:46 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: glance: disable swift backend https://review.openstack.org/334555 | 17:48 |
| *** ifarkas has quit IRC | 17:48 | |
| *** ramishra has quit IRC | 17:49 | |
| *** fragatina has quit IRC | 17:49 | |
| EmilienM | bnemec: yeah, trying to use "file" instead of long name | 17:50 |
| *** ramishra has joined #tripleo | 17:51 | |
| EmilienM | 2016-06-27 17:11:11.517 25461 DEBUG glance_store.backend [-] Registering store glance.store.filesystem.Store with schemes ('file', 'filesystem') create_stores /usr/lib/python2.7/site-packages/glance_store/backend.py:198 | 17:51 |
| EmilienM | file is the right nae | 17:51 |
| EmilienM | name* | 17:51 |
| *** hewbrocca is now known as hewbrocca-afk | 17:51 | |
| EmilienM | see http://logs.openstack.org/55/334555/2/check-tripleo/gate-tripleo-ci-centos-7-nonha/925831b/logs/undercloud/var/log/glance/api.txt.gz#_2016-06-27_17_06_32_415 | 17:51 |
| *** dsariel has joined #tripleo | 17:53 | |
| *** lucasagomes is now known as lucas-afk | 18:01 | |
| *** egafford has quit IRC | 18:01 | |
| *** dtantsur is now known as dtantsur|afk | 18:02 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient: Don't pass None via UpdateIdentifier https://review.openstack.org/334598 | 18:07 |
| *** egafford has joined #tripleo | 18:07 | |
| *** _milan_ has quit IRC | 18:07 | |
| openstackgerrit | Matt Young proposed openstack/tripleo-quickstart: Fix gate failures due to change in provisioning behavior. https://review.openstack.org/334600 | 18:08 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1596604 | 18:10 |
| openstack | Launchpad bug 1596604 in tripleo "swift-proxy-server OOM" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| *** stendulker has quit IRC | 18:10 | |
| Slower | WARNING: map-services has been deprecated. Please use the svc-map element. | 18:11 |
| Slower | Operation failed: No such file or directory | 18:11 |
| Slower | Unmount /var/tmp/dib_build.CK913eOV/mnt/tmp/yum | 18:11 |
| *** amoralej|off is now known as amoralej | 18:15 | |
| Slower | how do I even debug this thing? | 18:17 |
| *** shardy is now known as shardy_afk | 18:17 | |
| openstackgerrit | Matt Young proposed openstack/tripleo-quickstart: Fix gate failures due to change in provisioning behavior. https://review.openstack.org/334600 | 18:18 |
| openstackgerrit | ayoung proposed openstack/tripleo-quickstart: Provision Identity VM https://review.openstack.org/328335 | 18:24 |
| openstackgerrit | ayoung proposed openstack/tripleo-quickstart: Allow for multiple undercloud nodes https://review.openstack.org/315749 | 18:24 |
| openstackgerrit | ayoung proposed openstack/tripleo-quickstart: Setup IPA server https://review.openstack.org/328373 | 18:24 |
| openstackgerrit | ayoung proposed openstack/tripleo-quickstart: added rhsso https://review.openstack.org/334612 | 18:24 |
| *** ayoung has quit IRC | 18:27 | |
| *** rasca has quit IRC | 18:27 | |
| *** akuznetsov has quit IRC | 18:33 | |
| *** rwsu has quit IRC | 18:33 | |
| *** egafford has quit IRC | 18:34 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/tripleo-quickstart: Fix provision/base after teardown change https://review.openstack.org/334472 | 18:38 |
| *** amoralej is now known as amoralej|off | 18:40 | |
| *** chem` has joined #tripleo | 18:45 | |
| *** chem has quit IRC | 18:47 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Add gnocchi profiles https://review.openstack.org/315527 | 18:51 |
| *** egafford has joined #tripleo | 18:53 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix copyright in docs https://review.openstack.org/333084 | 18:53 |
| *** pradk has quit IRC | 18:54 | |
| *** egafford has quit IRC | 18:54 | |
| *** egafford has joined #tripleo | 18:56 | |
| *** egafford has quit IRC | 18:56 | |
| *** egafford has joined #tripleo | 18:56 | |
| *** pradk has joined #tripleo | 18:56 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Re-enable Ceilometer composable roles for controller https://review.openstack.org/333498 | 18:57 |
| *** xinwu has joined #tripleo | 19:02 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Adds missing line continuation https://review.openstack.org/334624 | 19:03 |
| openstackgerrit | Alfredo Moralejo proposed openstack/tripleo-quickstart: Fix provision/base after teardown change https://review.openstack.org/334472 | 19:04 |
| *** akshai has quit IRC | 19:12 | |
| *** akshai has joined #tripleo | 19:12 | |
| *** jcoufal has quit IRC | 19:14 | |
| *** akrivoka has quit IRC | 19:14 | |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: glance: disable swift backend https://review.openstack.org/334555 | 19:14 |
| EmilienM | ok it should be better now, I tested it | 19:14 |
| EmilienM | I also changed the bug id^ | 19:15 |
| *** pradk has quit IRC | 19:16 | |
| *** akshai has quit IRC | 19:16 | |
| EmilienM | bnemec: did you notice oom also on overclouds? | 19:17 |
| EmilienM | or was it only a supposition? | 19:17 |
| bnemec | EmilienM: I've seen OOM on an overcloud where I was trying to boot a large image, and derekh_afk has seen the same in the upcoming rh2 env on the Mitaka release. | 19:18 |
| EmilienM | ok | 19:19 |
| *** chem` has quit IRC | 19:20 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Add gnocchi profiles https://review.openstack.org/315527 | 19:22 |
| *** florianf has quit IRC | 19:23 | |
| *** pradk has joined #tripleo | 19:28 | |
| *** pradk has quit IRC | 19:29 | |
| derekh_afk | EmilienM: bnemec so here is an interesting observation (maybe a coincidence), the handful of patches on master that have passed today have all been on repo's that don't use the cached image | 19:32 |
| derekh_afk | EmilienM: bnemec going to submit a patch to disable the cached instack image to see what happens, if that helps we can merge it and see if we can figure out the difference | 19:33 |
| EmilienM | derekh_afk: so you mean that a package that is in cache would be part of the reason of our OOM? | 19:33 |
| EmilienM | o | 19:33 |
| EmilienM | derekh_afk: ok thx | 19:33 |
| derekh_afk | EmilienM: in theory they should be the same but something could be going wrong | 19:33 |
| EmilienM | derekh_afk: I alsoo proposed https://review.openstack.org/#/c/334555/ | 19:33 |
| *** panda has quit IRC | 19:34 | |
| *** panda has joined #tripleo | 19:35 | |
| *** akshai has joined #tripleo | 19:36 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Temporarily disable the instack.qcow cached image https://review.openstack.org/334634 | 19:36 |
| *** jefrite has quit IRC | 19:37 | |
| *** pradk has joined #tripleo | 19:39 | |
| derekh_afk | EmilienM: ack +2 until we can figure it out | 19:39 |
| EmilienM | ack | 19:39 |
| *** ayoung has joined #tripleo | 19:42 | |
| *** rajinir has quit IRC | 19:44 | |
| *** derekh_afk is now known as derekh | 19:48 | |
| *** julim has joined #tripleo | 19:51 | |
| *** jayg is now known as jayg|g0n3 | 19:51 | |
| *** akshai_ has joined #tripleo | 19:54 | |
| *** sambetts is now known as sambetts|afk | 19:57 | |
| *** akshai has quit IRC | 19:58 | |
| *** gfidente has quit IRC | 19:59 | |
| *** paramite has joined #tripleo | 20:05 | |
| *** jprovazn has quit IRC | 20:21 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Drop unused VIP params to controller.yaml https://review.openstack.org/333511 | 20:24 |
| EmilienM | bnemec: damn, I still see the "no valid host was found" with file backend | 20:25 |
| EmilienM | I don't have logs yet, I just saw it in zuul/telnet | 20:26 |
| *** derekh has quit IRC | 20:33 | |
| *** xinwu has quit IRC | 20:33 | |
| EmilienM | bnemec: nevermind, I think I was telneting the wrong job | 20:34 |
| bnemec | Sadly I didn't even question the possibility that there was something else wrong too. :-) | 20:38 |
| *** myoung is now known as myoung|brb | 20:39 | |
| *** yamahata has quit IRC | 20:46 | |
| *** yamahata has joined #tripleo | 20:46 | |
| *** yamahata has quit IRC | 20:48 | |
| *** yamahata has joined #tripleo | 20:48 | |
| *** dsariel has quit IRC | 20:48 | |
| EmilienM | bnemec: ok pingtest is running :) | 20:53 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Re-enable Ceilometer composable roles for controller https://review.openstack.org/333498 | 20:54 |
| EmilienM | bnemec: Job complete, result: SUCCESS :) | 21:03 |
| *** milan has joined #tripleo | 21:03 | |
| bnemec | \o/ | 21:03 |
| *** myoung|brb is now known as myoung|biab | 21:05 | |
| Slower | woot! | 21:06 |
| *** lucas-afk has quit IRC | 21:06 | |
| *** lblanchard has quit IRC | 21:09 | |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder: Generalize logic for skipping final image generation https://review.openstack.org/334042 | 21:12 |
| *** lucasagomes has joined #tripleo | 21:13 | |
| *** lucasagomes has quit IRC | 21:19 | |
| *** rhallisey has quit IRC | 21:21 | |
| *** numans has joined #tripleo | 21:26 | |
| *** lucasagomes has joined #tripleo | 21:27 | |
| numans | EmilienM, Hi, can you please add this into your review Q - https://review.openstack.org/#/c/320531/ | 21:28 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Enable IPv4/IPv6 dual-stack Public API endpoints https://review.openstack.org/289279 | 21:35 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Enable IPv4/IPv6 dual-stack Public API endpoints https://review.openstack.org/289279 | 21:39 |
| *** bfournie has quit IRC | 21:54 | |
| *** [1]cdearborn has quit IRC | 22:08 | |
| *** egafford has quit IRC | 22:13 | |
| *** oneswig has quit IRC | 22:13 | |
| *** onovy has quit IRC | 22:14 | |
| openstackgerrit | Merged openstack/instack-undercloud: glance: disable swift backend https://review.openstack.org/334555 | 22:16 |
| *** akshai_ has quit IRC | 22:19 | |
| *** pradk has quit IRC | 22:19 | |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart: Fixes for dlrn gate repo injection while using devmode https://review.openstack.org/334691 | 22:31 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: Additional note for usbkey users regarding the latest images https://review.openstack.org/326419 | 22:38 |
| *** julim has quit IRC | 22:46 | |
| *** rajinir has joined #tripleo | 23:15 | |
| *** myoung|biab is now known as myoung | 23:18 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Release notes for 1.18 https://review.openstack.org/334711 | 23:20 |
| *** thrash is now known as thrash|g0ne | 23:33 | |
| *** onovy has joined #tripleo | 23:39 | |
| *** tosky has quit IRC | 23:51 | |
| *** dmacpher has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!