| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 00:10 |
| *** gbarros has quit IRC | 00:10 | |
| *** gbarros has joined #tripleo | 00:11 | |
| *** funzo has joined #tripleo | 00:13 | |
| *** funzo has quit IRC | 00:18 | |
| *** limao has joined #tripleo | 00:44 | |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Add --set argument to prepare command https://review.openstack.org/489083 | 01:03 |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 01:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| *** funzo has joined #tripleo | 01:14 | |
| *** funzo has quit IRC | 01:18 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: Stop using deprecated --pull-source https://review.openstack.org/493391 | 01:20 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 01:22 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 01:22 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Delete docker-centos-tripleoupstream.yaml https://review.openstack.org/487613 | 01:35 |
| *** itlinux has joined #tripleo | 01:51 | |
| *** tzumainn has joined #tripleo | 02:05 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 02:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| *** leitan_ has quit IRC | 02:14 | |
| *** rbrady has joined #tripleo | 02:17 | |
| *** rbrady has joined #tripleo | 02:17 | |
| *** gvrangan has joined #tripleo | 02:26 | |
| *** rbrady has quit IRC | 02:33 | |
| *** tzumainn has quit IRC | 02:39 | |
| *** tzumainn has joined #tripleo | 02:40 | |
| pabelanger | EmilienM: mwhahaha: bnemec: Container jobs are hitting docker.io directly, and not using reverse proxy cache: http://logs.openstack.org/70/490470/2/gate/gate-tripleo-ci-centos-7-containers-multinode/10c8c84/logs/undercloud/home/jenkins/overcloud_prep_containers.log.txt.gz#_2017-08-14_02_04_28 | 02:47 |
| pabelanger | that should be prioritize to be fixed | 02:47 |
| pabelanger | EmilienM: mwhahaha: bnemec: remote: https://review.openstack.org/493409 Add query for bug 1710533 | 02:59 |
| openstack | bug 1710533 in OpenStack-Gate "Job fails to download container from docker.io" [Undecided,New] https://launchpad.net/bugs/1710533 | 02:59 |
| pabelanger | elastic recheck will start leaving comments now | 02:59 |
| *** gvrangan has quit IRC | 02:59 | |
| *** gvrangan has joined #tripleo | 03:04 | |
| *** stee_3_ has joined #tripleo | 03:04 | |
| *** bugzy_ has joined #tripleo | 03:06 | |
| *** stee_3 has quit IRC | 03:08 | |
| *** bugzy has quit IRC | 03:08 | |
| *** gvrangan has quit IRC | 03:09 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 03:10 |
| *** funzo has joined #tripleo | 03:16 | |
| *** itlinux has quit IRC | 03:17 | |
| *** funzo has quit IRC | 03:20 | |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo stable/ocata: Enable innodb_buffer_pool_size configuration https://review.openstack.org/490577 | 03:23 |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo stable/newton: Enable innodb_buffer_pool_size configuration https://review.openstack.org/490578 | 03:28 |
| *** psachin has joined #tripleo | 03:34 | |
| *** gkadam has joined #tripleo | 03:35 | |
| *** bugzy_ is now known as bugzy | 03:36 | |
| *** udesale has joined #tripleo | 03:37 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: [NEWTON-ONLY] Ignore Ceph healt warning states by default https://review.openstack.org/491847 | 03:39 |
| *** Vijayendra has joined #tripleo | 03:41 | |
| *** psahoo has joined #tripleo | 03:48 | |
| *** itlinux has joined #tripleo | 03:48 | |
| *** gbarros has quit IRC | 03:53 | |
| *** ykarel has joined #tripleo | 03:53 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use prepare command to build container images env file https://review.openstack.org/483178 | 03:59 |
| *** links has joined #tripleo | 04:07 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 04:10 |
| *** pgadiya has joined #tripleo | 04:13 | |
| EmilienM | pabelanger: ok | 04:13 |
| *** funzo has joined #tripleo | 04:16 | |
| *** ramishra has joined #tripleo | 04:17 | |
| *** funzo has quit IRC | 04:21 | |
| *** limao has quit IRC | 04:27 | |
| *** tzumainn has quit IRC | 04:27 | |
| *** limao has joined #tripleo | 04:36 | |
| *** limao has quit IRC | 04:40 | |
| *** shreshtha has joined #tripleo | 04:40 | |
| *** limao has joined #tripleo | 05:00 | |
| *** ramishra has quit IRC | 05:04 | |
| *** ramishra has joined #tripleo | 05:06 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 05:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| *** funzo has joined #tripleo | 05:17 | |
| *** bkopilov has quit IRC | 05:18 | |
| *** jfrancoa has joined #tripleo | 05:18 | |
| *** mdnadeem has joined #tripleo | 05:19 | |
| *** funzo has quit IRC | 05:22 | |
| *** stendulker has joined #tripleo | 05:23 | |
| *** nyechiel has joined #tripleo | 05:26 | |
| *** udesale__ has joined #tripleo | 05:34 | |
| *** udesale has quit IRC | 05:37 | |
| *** shreshtha has quit IRC | 05:37 | |
| *** dsariel has joined #tripleo | 05:41 | |
| *** jaganathan has joined #tripleo | 05:43 | |
| *** florianf has joined #tripleo | 05:45 | |
| *** florianf has quit IRC | 05:48 | |
| *** bkopilov has joined #tripleo | 05:48 | |
| *** links has quit IRC | 05:50 | |
| *** florianf has joined #tripleo | 05:51 | |
| *** gvrangan has joined #tripleo | 05:53 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: WIP copy swift ring files https://review.openstack.org/493437 | 05:54 |
| *** links has joined #tripleo | 05:56 | |
| *** janki has joined #tripleo | 05:56 | |
| *** jprovazn has joined #tripleo | 05:57 | |
| *** marios has joined #tripleo | 05:58 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: WIP check do not review https://review.openstack.org/493439 | 05:59 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 06:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| *** shreshtha has joined #tripleo | 06:12 | |
| *** itlinux has quit IRC | 06:15 | |
| *** udesale__ has quit IRC | 06:16 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat] https://review.openstack.org/491399 | 06:17 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh] https://review.openstack.org/491411 | 06:17 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ceilometer] https://review.openstack.org/490392 | 06:17 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Neutron] https://review.openstack.org/489957 | 06:17 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi] https://review.openstack.org/491433 | 06:17 |
| *** jbadiapa has joined #tripleo | 06:17 | |
| *** yprokule has joined #tripleo | 06:18 | |
| *** shreshtha has quit IRC | 06:21 | |
| *** agurenko has joined #tripleo | 06:22 | |
| *** gvrangan has quit IRC | 06:27 | |
| *** ramishra has quit IRC | 06:27 | |
| *** udesale__ has joined #tripleo | 06:27 | |
| *** dpawar has joined #tripleo | 06:29 | |
| *** owalsh_ has joined #tripleo | 06:29 | |
| *** owalsh has quit IRC | 06:29 | |
| *** rwsu has joined #tripleo | 06:32 | |
| *** shreshtha has joined #tripleo | 06:34 | |
| *** ramishra has joined #tripleo | 06:45 | |
| *** gvrangan has joined #tripleo | 06:46 | |
| *** dpawar has quit IRC | 06:53 | |
| *** rcernin has joined #tripleo | 06:57 | |
| *** ykarel has quit IRC | 06:58 | |
| *** gvrangan has quit IRC | 06:58 | |
| *** masco has joined #tripleo | 07:02 | |
| *** ykarel has joined #tripleo | 07:03 | |
| *** dpawar has joined #tripleo | 07:03 | |
| *** aufi has joined #tripleo | 07:04 | |
| *** mcornea has joined #tripleo | 07:05 | |
| *** dpawar has quit IRC | 07:07 | |
| *** dpawar_ has joined #tripleo | 07:07 | |
| *** dsariel has quit IRC | 07:07 | |
| *** dparkes has joined #tripleo | 07:09 | |
| *** shardy has joined #tripleo | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 07:10 |
| *** jlinkes has joined #tripleo | 07:10 | |
| *** tesseract has joined #tripleo | 07:10 | |
| *** ramishra has quit IRC | 07:11 | |
| *** ebarrera has joined #tripleo | 07:12 | |
| *** ramishra has joined #tripleo | 07:12 | |
| *** brault has joined #tripleo | 07:13 | |
| *** cylopez has joined #tripleo | 07:14 | |
| *** funzo has joined #tripleo | 07:19 | |
| *** jpena|off is now known as jpena | 07:22 | |
| *** funzo has quit IRC | 07:23 | |
| *** oidgar has joined #tripleo | 07:24 | |
| *** ccamacho has joined #tripleo | 07:30 | |
| *** amoralej|off is now known as amoralej | 07:32 | |
| *** jtomasek has joined #tripleo | 07:32 | |
| *** jpich has joined #tripleo | 07:35 | |
| *** egonzalez has joined #tripleo | 07:39 | |
| *** iranzo has joined #tripleo | 07:42 | |
| *** dciabrin has quit IRC | 07:43 | |
| *** mrchao has joined #tripleo | 07:44 | |
| *** owalsh_ is now known as owalsh | 07:46 | |
| *** dpawar_ has quit IRC | 07:52 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Heat] https://review.openstack.org/491399 | 08:03 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh] https://review.openstack.org/491411 | 08:03 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Ceilometer] https://review.openstack.org/490392 | 08:03 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Neutron] https://review.openstack.org/489957 | 08:03 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Gnocchi] https://review.openstack.org/491433 | 08:03 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Mistral] https://review.openstack.org/493471 | 08:03 |
| *** dalvarez has quit IRC | 08:05 | |
| *** brault has quit IRC | 08:06 | |
| *** amoralej has quit IRC | 08:06 | |
| bandini | shardy: Hi (sorry was out the other day). I am fine in deprecating https://review.openstack.org/#/c/474486/ in favour of your https://review.openstack.org/#/c/492550/ | 08:06 |
| bandini | shardy: mind if I rebase https://review.openstack.org/#/c/492550/ as there are conflicts with j2_excludes on master or were you going to do it today? | 08:07 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 08:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| *** flaper87 has quit IRC | 08:10 | |
| *** iranzo has quit IRC | 08:12 | |
| *** paramite has joined #tripleo | 08:14 | |
| shardy | bandini: Hey, np - sure feel free to rebase, I've not got to looking at that patch yet | 08:16 |
| *** flaper87 has joined #tripleo | 08:16 | |
| shardy | bandini: https://review.openstack.org/#/c/492218 needs a review, which is the parent to that patch :) | 08:17 |
| shardy | that one is passing CI so perhaps we can land it to avoid another CI run before the rebase | 08:17 |
| bandini | shardy: oh I see. yes let's get that one in first. /me looks | 08:18 |
| *** flaper87 has quit IRC | 08:18 | |
| *** flaper87 has joined #tripleo | 08:18 | |
| *** limao has quit IRC | 08:19 | |
| *** funzo has joined #tripleo | 08:19 | |
| *** dsariel has joined #tripleo | 08:21 | |
| *** derekh has joined #tripleo | 08:22 | |
| *** limao has joined #tripleo | 08:23 | |
| d0ugal | honza: Hey | 08:23 |
| d0ugal | honza: Not sure what tz you are in atm | 08:23 |
| honza | d0ugal: o/ | 08:23 |
| d0ugal | oh, cool | 08:23 |
| honza | d0ugal: cest :) | 08:23 |
| d0ugal | honza: I was only 50% expecting a reply | 08:23 |
| d0ugal | honza: I seen https://review.openstack.org/#/c/469608/ | 08:23 |
| d0ugal | honza: I think the cron triggers might be broken in mistral master. | 08:24 |
| *** funzo has quit IRC | 08:24 | |
| *** ykarel is now known as ykarel|lunch | 08:24 | |
| honza | d0ugal: with crons themselves? the ci job seems to fail on the reinstall where the mistral workbooks are loaded | 08:25 |
| honza | d0ugal: which is right before my cron code | 08:25 |
| d0ugal | honza: oh, that is interesting. I've not seen an error like that. Got a link to it? | 08:25 |
| honza | d0ugal: http://logs.openstack.org/08/469608/5/check/gate-tripleo-ci-centos-7-undercloud-oooq/d00c9d6/logs/undercloud/home/jenkins/undercloud_reinstall.log.txt.gz#_2017-08-12_08_08_14 | 08:26 |
| d0ugal | honza: I was testing cron triggers as I was helping somebody else use them, and they are added okay but there is an exception every time mistral attempts to run them - something to do with auth tokens I think. | 08:26 |
| honza | d0ugal: ... if i understand this correctly :) | 08:26 |
| d0ugal | huh, interesting. | 08:27 |
| *** yprokule_ has joined #tripleo | 08:27 | |
| honza | d0ugal: that code is fairly new but how did it pass the gate? | 08:27 |
| d0ugal | honza: http://logs.openstack.org/08/469608/5/check/gate-tripleo-ci-centos-7-undercloud-oooq/d00c9d6/logs/undercloud/var/log/mistral/api.log.txt.gz#_2017-08-12_07_59_58_171 | 08:28 |
| d0ugal | honza: good question | 08:28 |
| d0ugal | you are also hitting the cron error I hit. | 08:28 |
| honza | ? | 08:28 |
| honza | is that in the logs? | 08:28 |
| d0ugal | honza: http://logs.openstack.org/08/469608/5/check/gate-tripleo-ci-centos-7-undercloud-oooq/d00c9d6/logs/undercloud/var/log/mistral/api.log.txt.gz#_2017-08-12_07_59_58_171 | 08:28 |
| d0ugal | or just see anywhere in the bottom 50% of that file :) | 08:29 |
| honza | welp | 08:29 |
| d0ugal | FWIW, that error suggests that the cron was added. | 08:29 |
| *** yprokule has quit IRC | 08:30 | |
| *** yprokule_ is now known as yprokule | 08:30 | |
| d0ugal | My understanding of that error is that mistral has a cron trigger that needs executed, so it attempts, but because it never completes it keeps trying | 08:30 |
| *** brault has joined #tripleo | 08:30 | |
| honza | right | 08:30 |
| d0ugal | honza: I am going to be digging into this today, hopefully I can figure it out. | 08:31 |
| d0ugal | but it is quite useful to have a patch that reproduces it in CI :) | 08:32 |
| honza | d0ugal: excellent, i'm happy to help with testing etc | 08:32 |
| d0ugal | honza: I don't understand why the log suggests it fails before that but the cron trigger still seems to be added | 08:34 |
| d0ugal | ... unless this error is just always happening and I'm silly | 08:34 |
| honza | d0ugal: note that it's the reinstall log | 08:34 |
| d0ugal | ohh | 08:34 |
| d0ugal | I've never looked in this log before! | 08:35 |
| *** lucasagomes has joined #tripleo | 08:35 | |
| *** brault has quit IRC | 08:35 | |
| d0ugal | Right, so it is added correctly in the install log. that makes more sense then. | 08:35 |
| d0ugal | I wonder if Mistral is basically DDOSing itself then | 08:36 |
| *** lucasagomes has quit IRC | 08:36 | |
| d0ugal | well, DOS, prob not distributed :) | 08:36 |
| honza | :D | 08:36 |
| *** dalvarez has joined #tripleo | 08:36 | |
| *** lucas-afk is now known as lucasagomes | 08:37 | |
| *** tosky has joined #tripleo | 08:37 | |
| *** jaosorior has joined #tripleo | 08:41 | |
| shardy | Can I get a second reviewer for https://review.openstack.org/#/c/482626/ please? | 08:44 |
| shardy | Final patch converting role templates to be dynamically rendered | 08:45 |
| shardy | for composable networking | 08:45 |
| *** amoralej has joined #tripleo | 08:46 | |
| *** gfidente has joined #tripleo | 08:46 | |
| *** gfidente has quit IRC | 08:46 | |
| *** gfidente has joined #tripleo | 08:46 | |
| *** yprokule_ has joined #tripleo | 08:47 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** yprokule has quit IRC | 08:50 | |
| *** yprokule_ is now known as yprokule | 08:50 | |
| *** dparkes has quit IRC | 08:50 | |
| *** athomas has joined #tripleo | 08:50 | |
| *** limao has quit IRC | 08:52 | |
| *** dbecker has joined #tripleo | 08:53 | |
| *** limao has joined #tripleo | 08:54 | |
| *** florianf_ has joined #tripleo | 08:54 | |
| *** florianf has quit IRC | 08:54 | |
| *** ffiore has joined #tripleo | 08:59 | |
| *** chem has joined #tripleo | 09:01 | |
| *** openstackgerrit has joined #tripleo | 09:01 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Upgrade webpack and webpack-dev-server https://review.openstack.org/475866 | 09:01 |
| *** ffiore has quit IRC | 09:03 | |
| *** ffiore_ has joined #tripleo | 09:03 | |
| *** brault has joined #tripleo | 09:05 | |
| *** brault has quit IRC | 09:09 | |
| *** ramishra has quit IRC | 09:10 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 09:10 |
| *** ramishra has joined #tripleo | 09:11 | |
| *** milan has joined #tripleo | 09:12 | |
| *** ykarel|lunch is now known as ykarel | 09:14 | |
| *** sshnaidm|off has quit IRC | 09:15 | |
| *** funzo has joined #tripleo | 09:20 | |
| *** funzo has quit IRC | 09:25 | |
| openstackgerrit | Michael Henkel proposed openstack/os-net-config master: This patch adds initial support for the Contrai vRouter interface https://review.openstack.org/492492 | 09:25 |
| *** dparkes has joined #tripleo | 09:28 | |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-common master: Add Barbican Config Image to overcloud containers https://review.openstack.org/493493 | 09:31 |
| jaosorior | marios: are you acquainted with the container-upgrades gate? | 09:33 |
| jaosorior | or shardy ^^ | 09:33 |
| jaosorior | or anyone :O | 09:34 |
| shardy | jaosorior: somewhat, but jistr|off implemented most of that job so is a good contact | 09:34 |
| marios | not really jaosorior | 09:34 |
| shardy | I know jistr|off was looking at several issues last week | 09:34 |
| shardy | we keep breaking it because it's not voting :( | 09:35 |
| marios | shardy: yeah jistr|off or matbu i think may be back today and also worked on that | 09:35 |
| marios | err jaosorior ^ | 09:35 |
| jaosorior | ok | 09:35 |
| *** bogdando has joined #tripleo | 09:36 | |
| *** ykarel is now known as ykarel|afk | 09:36 | |
| jaosorior | I think jistr|off is on PTO :/ | 09:36 |
| shardy | jaosorior: yeah looks like it, what's up, lets try to figure it out :) | 09:37 |
| jaosorior | yep | 09:37 |
| jaosorior | opening logs now | 09:37 |
| shardy | I saw a failure earlier that looked quickstart related | 09:37 |
| * shardy looks for open patches | 09:37 | |
| shardy | https://review.openstack.org/#/c/479198/ aimed to make it voting, but there were objections from EmilienM around the promotion pipeline | 09:38 |
| shardy | https://github.com/openstack/tripleo-quickstart/commit/3fa2214be90ec844f181496c6c216423fdda345d fixes the CI failure I saw earlier | 09:38 |
| shardy | but there may be other issues | 09:38 |
| jaosorior | shardy: that merged on friday. But it seems to still be broken | 09:39 |
| jaosorior | shardy: seems to be having issues getting the images http://logs.openstack.org/37/493437/1/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/49329e7/logs/undercloud/var/log/postci.txt.gz | 09:39 |
| jaosorior | http://logs.openstack.org/66/475366/15/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/ac108f3/logs/undercloud/var/log/postci.txt.gz | 09:39 |
| jaosorior | shardy: I think it's related to the fact that we now need to pass an environment that tells the deployment where to fetch the images from, and now it might be the case that that's not being passed on upgrade | 09:40 |
| shardy | jaosorior: Hmm, possibly, although it looks like it's trying to pull images from the undercloud docker-distribution and some images are failing? | 09:41 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Also write an upgrade_(batch)_tasks playbook https://review.openstack.org/490848 | 09:41 |
| jaosorior | shardy: not sure if any actually succeed | 09:41 |
| shardy | Hmm, Ok so we're possibly missing the image prepare environment | 09:42 |
| jaosorior | shardy: what does that job actually do? It upgrades from non-containerized to containered? | 09:42 |
| * shardy looks for patches | 09:42 | |
| shardy | jaosorior: yeah it deploys baremetal ocata then upgrades to containerized pike | 09:42 |
| *** salmankhan has joined #tripleo | 09:42 | |
| jaosorior | oooh okok | 09:43 |
| jaosorior | shardy: you have to run a command to generate taht environment, right? i've been using the one that pulls from upstream that's in the t-h-t repo | 09:45 |
| *** dixiaoli_ has joined #tripleo | 09:48 | |
| *** limao has quit IRC | 09:49 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Consolidate deployment in major-upgrade-composable-steps https://review.openstack.org/493499 | 09:49 |
| *** shardy has quit IRC | 09:50 | |
| lvdombrkr | hi guys, i run openstack undercloud install after it finished i realised that that there is typo mistake in undercloud.conf, is there posibility rerun installation with correct ips? | 09:50 |
| *** iranzo has joined #tripleo | 09:50 | |
| *** iranzo has quit IRC | 09:51 | |
| *** shardy has joined #tripleo | 09:51 | |
| *** akrivoka has joined #tripleo | 09:53 | |
| *** iranzo has joined #tripleo | 09:53 | |
| lvdombrkr | ?? | 09:54 |
| *** iranzo has quit IRC | 09:54 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add kpartx as a requirement to build images https://review.openstack.org/493004 | 09:54 |
| *** dixiaoli_ has quit IRC | 09:55 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Clear up debian element documentation https://review.openstack.org/491971 | 09:55 |
| ccamacho | hey Shardy, can you check plz https://review.openstack.org/493499 ?? This is a fix after merging the change to move templates under the common folder, thanks! | 09:55 |
| shardy | ccamacho: hey, sure looking | 09:56 |
| ccamacho | thanks! | 09:56 |
| shardy | ccamacho: looks good, we can approve when CI has run | 09:57 |
| ccamacho | shardy awesome! | 09:58 |
| lvdombrkr | hi guys, i run openstack undercloud install after it finished i realised that that there is typo mistake in undercloud.conf, is there posibility rerun installation with correct ips? | 09:58 |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates master: Containarise Barbican API https://review.openstack.org/481451 | 09:59 |
| *** snecklifter has joined #tripleo | 10:00 | |
| lvdombrkr | ? | 10:00 |
| shardy | lvdombrkr: yes in most cases you can re-run the install command after changing the config | 10:02 |
| lvdombrkr | shardy: thank you but after re-run i get error: Changing the local_ip is not allowed | 10:03 |
| shardy | lvdombrkr: Oh, yeah I think that's the one thing we prevent, because changing it messes up the swift config | 10:04 |
| lvdombrkr | shardy: that means i need to reinstall os? | 10:05 |
| *** iranzo has joined #tripleo | 10:05 | |
| *** iranzo has quit IRC | 10:05 | |
| *** sshnaidm|off has joined #tripleo | 10:07 | |
| *** sshnaidm|off is now known as sshnaidm | 10:08 | |
| *** iranzo has joined #tripleo | 10:09 | |
| *** iranzo has quit IRC | 10:09 | |
| *** ebarrera has quit IRC | 10:09 | |
| *** ebarrera has joined #tripleo | 10:10 | |
| lvdombrkr | shardy:? | 10:10 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 10:10 |
| shardy | lvdombrkr: yes that's probably best unless you're willing to spend time dealing with the various issues outlined in https://bugs.launchpad.net/tripleo/+bug/1645267 | 10:10 |
| openstack | Launchpad bug 1645267 in tripleo "Undercloud upgrade breaks due to IP range change" [High,Fix released] - Assigned to Ben Nemec (bnemec) | 10:10 |
| shardy | like, it's technically possible to work around without reinstall, but it's probably quicker to start with a clean environment if possible | 10:11 |
| *** iranzo has joined #tripleo | 10:11 | |
| *** iranzo has quit IRC | 10:12 | |
| *** tzumainn has joined #tripleo | 10:12 | |
| lvdombrkr | shardy: thank you! | 10:12 |
| *** iranzo has joined #tripleo | 10:14 | |
| *** iranzo has quit IRC | 10:15 | |
| *** iranzo has joined #tripleo | 10:17 | |
| *** iranzo has quit IRC | 10:17 | |
| *** iranzo has joined #tripleo | 10:18 | |
| *** iranzo has quit IRC | 10:19 | |
| *** iranzo has joined #tripleo | 10:20 | |
| *** iranzo has joined #tripleo | 10:22 | |
| *** iranzo has quit IRC | 10:23 | |
| *** snecklifter has quit IRC | 10:23 | |
| *** snecklifter has joined #tripleo | 10:24 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Containerize virtlogd https://review.openstack.org/469116 | 10:25 |
| *** dtantsur|afk is now known as dtantsur | 10:32 | |
| *** udesale__ has quit IRC | 10:33 | |
| *** florianf_ is now known as florianf | 10:36 | |
| dsariel | sshnaidm, http://10.35.7.44 | 10:39 |
| *** marios has quit IRC | 10:41 | |
| chem | bandini: he, do you need something for removing the WIP on https://review.openstack.org/#/c/491705/ ? | 10:42 |
| bandini | chem: nothing really. Are we 100% good on the container name format requiring a : ? | 10:42 |
| chem | bandini: this is the case currently {up,down}stream, if that change we will bug/adjust, WDYT ? | 10:43 |
| bandini | chem: works for me I guess | 10:44 |
| * bandini removes WIP | 10:44 | |
| *** mdnadeem has quit IRC | 10:44 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Tag the ha containers with 'pcmklatest' at deploy time https://review.openstack.org/491705 | 10:45 |
| bandini | chem: ^ | 10:45 |
| chem | bandini: that was fast :) thanks | 10:46 |
| bandini | chem: at your service, sir ;) | 10:46 |
| *** ykarel|afk is now known as ykarel | 10:48 | |
| *** jkilpatr has quit IRC | 10:49 | |
| *** jkilpatr has joined #tripleo | 10:49 | |
| jaosorior | shardy: hey, would you know how to debug a docker-puppet issue? | 10:50 |
| shardy | jaosorior: possibly, what's up? | 10:52 |
| *** stendulker_ has joined #tripleo | 10:53 | |
| *** ffiore_ has quit IRC | 10:54 | |
| *** ffiore_ has joined #tripleo | 10:54 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for containerized haproxy https://review.openstack.org/489900 | 10:55 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Bind mount tripleo.cnf in transient bootstrap containers https://review.openstack.org/492963 | 10:55 |
| jaosorior | shardy: right, so I'm trying out this change ^^ (the haproxy one) | 10:55 |
| *** zshi_laptop has joined #tripleo | 10:55 | |
| *** stendulker has quit IRC | 10:56 | |
| jaosorior | shardy: and I'm seeing a failure like this http://paste.openstack.org/show/618299/ | 10:56 |
| jaosorior | shardy: however, I don't find anything else that could be related (maybe I just don't know where to look) the logs don't really show any obvious error :/ | 10:57 |
| jaosorior | shardy: I could give you access to my deployment if you have time to take a look | 10:57 |
| openstackgerrit | Merged openstack/tripleo-validations master: Configurable ansible_ssh_user for tripleo-ansible-inventory https://review.openstack.org/493046 | 10:59 |
| *** stendulker_ has quit IRC | 10:59 | |
| *** mdnadeem has joined #tripleo | 11:00 | |
| shardy | jaosorior: Ok looks like a type mismatch, I'd debug it by running docker-puppet.py manually on the CLI, e.g like we do via ansible here: | 11:00 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml#L30 | 11:00 |
| shardy | jaosorior: you export NET_HOST=true and DEBUG=true, then run it via python | 11:01 |
| shardy | jaosorior: you can also hack in some debug to the code which should show either why the input data is bad, or that we have to change something in the script itself | 11:01 |
| shardy | https://review.openstack.org/#/c/489900/8/docker/services/haproxy.yaml | 11:02 |
| shardy | jaosorior: is it because your patch changes volume to be a list, when it was a string before? | 11:03 |
| *** karthiks has joined #tripleo | 11:03 | |
| shardy | volumes I mean, you do list_concat instead of joining the lists via list_join delimited by : | 11:03 |
| jaosorior | shardy: the list_join is still there and it was only for one volume entry | 11:04 |
| jaosorior | but I think it might be related | 11:04 |
| shardy | jaosorior: yeah it's still there, but it's inside a list passed to list_concat | 11:04 |
| shardy | so the resulting data type isn't a string anymore AFAICS | 11:04 |
| jaosorior | shardy: but it used to be a list before too | 11:05 |
| jaosorior | shardy: check the dash before list_join | 11:05 |
| shardy | jaosorior: Hmm, Ok yeah | 11:06 |
| *** ffiore_ has quit IRC | 11:06 | |
| jaosorior | shardy: where can I find the resulting config? | 11:07 |
| *** ffiore_ has joined #tripleo | 11:07 | |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L194 | 11:07 |
| shardy | jaosorior: /var/lib/docker-puppet/docker-puppet.json | 11:08 |
| jaosorior | shardy: found the issue | 11:09 |
| jaosorior | thanks! | 11:09 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for containerized haproxy https://review.openstack.org/489900 | 11:09 |
| jaosorior | shardy: one of the items inside the list_concat had an extra dash, so it was a list. specifically it was the CA cert | 11:10 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 11:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| jaosorior | bandini: got time to check this out https://review.openstack.org/#/c/491437/ ? | 11:11 |
| *** mdnadeem has quit IRC | 11:12 | |
| *** agurenko has quit IRC | 11:14 | |
| *** lucasagomes is now known as lucas-hungry | 11:15 | |
| shardy | jaosorior: nice, good to hear it was an easy fix :) | 11:15 |
| *** shardy is now known as shardy_lunch | 11:15 | |
| *** deadnull has joined #tripleo | 11:20 | |
| *** nyechiel_ has joined #tripleo | 11:21 | |
| *** nyechiel has quit IRC | 11:22 | |
| *** funzo has joined #tripleo | 11:22 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Drop mongo from UC deploy https://review.openstack.org/480063 | 11:23 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Add logrotate-crond container images params https://review.openstack.org/490068 | 11:23 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add logrotate with crond service https://review.openstack.org/490050 | 11:23 |
| *** morazi has quit IRC | 11:25 | |
| *** ffiore_ has quit IRC | 11:25 | |
| *** _milan_ has joined #tripleo | 11:26 | |
| *** rbrady has joined #tripleo | 11:26 | |
| *** funzo has quit IRC | 11:26 | |
| *** mdnadeem has joined #tripleo | 11:27 | |
| jaosorior | marios, ccamacho: can you check this out https://review.openstack.org/#/c/491437/ ? | 11:27 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Persist containerized services httpd logs https://review.openstack.org/489955 | 11:27 |
| ccamacho | jaosorior sure | 11:28 |
| *** adarazs_mtg is now known as adarazs | 11:29 | |
| *** milan has quit IRC | 11:29 | |
| *** nyechiel has joined #tripleo | 11:32 | |
| *** pkovar has joined #tripleo | 11:32 | |
| *** shreshtha has quit IRC | 11:33 | |
| *** nyechiel_ has quit IRC | 11:35 | |
| *** atoth has joined #tripleo | 11:35 | |
| *** jpena is now known as jpena|lunch | 11:38 | |
| jprovazn | gfidente, hi | 11:38 |
| gfidente | jprovazn hey' | 11:38 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Fix metadata_settings in containerized mongodb https://review.openstack.org/492014 | 11:39 |
| jprovazn | gfidente, re https://review.openstack.org/#/c/482496/ - I'm quite confused from CI - it failed because user 'manila' was not there - http://logs.openstack.org/96/482496/7/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq/952b3ac/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-08-11_12_09_25 - this is quite weird because overcloud-full image does have manila user present | 11:39 |
| *** brault has joined #tripleo | 11:40 | |
| *** sshnaidm has quit IRC | 11:41 | |
| *** brault has quit IRC | 11:44 | |
| *** sshnaidm|mtg has joined #tripleo | 11:46 | |
| *** rlandy has joined #tripleo | 11:46 | |
| *** agurenko has joined #tripleo | 11:49 | |
| *** bfournie has quit IRC | 11:50 | |
| *** bfournie has joined #tripleo | 11:50 | |
| *** bfournie has quit IRC | 11:54 | |
| *** dprince has joined #tripleo | 11:56 | |
| jaosorior | dprince: hey, what do we do with this one https://review.openstack.org/#/c/478528/ ? | 11:56 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Add upgrade task to move swift ring files to be reachable by docker https://review.openstack.org/493518 | 11:57 |
| jaosorior | ccamacho, bandini, shardy_lunch: also, just tested this one out. It works :D https://review.openstack.org/#/c/489900/ | 11:57 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Add upgrade task to move swift ring files to be reachable by docker https://review.openstack.org/493518 | 11:59 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Switch to overcloudrc.v3 for running tempest https://review.openstack.org/493030 | 12:00 |
| *** shardy_lunch is now known as shardy | 12:01 | |
| dprince | jaosorior: well, I still haven't tested it | 12:02 |
| dprince | jaosorior: let me try to get it today | 12:02 |
| *** slagle has joined #tripleo | 12:04 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Add upgrade task to move swift ring files to be reachable by docker https://review.openstack.org/493518 | 12:04 |
| *** psachin has quit IRC | 12:08 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 12:10 |
| *** psachin has joined #tripleo | 12:10 | |
| *** thrash|westford is now known as thrash|g0ne | 12:11 | |
| openstackgerrit | Christopher Brown proposed openstack/tripleo-heat-templates master: Removes remaining AdminToken references https://review.openstack.org/493524 | 12:13 |
| *** pgadiya has quit IRC | 12:13 | |
| *** jlabarre has joined #tripleo | 12:16 | |
| *** abishop has joined #tripleo | 12:17 | |
| *** eck`gone is now known as eck` | 12:18 | |
| *** trown|outtypewww is now known as trown | 12:18 | |
| *** lucas-hungry is now known as lucasagomes | 12:21 | |
| *** funzo has joined #tripleo | 12:22 | |
| *** jpena|lunch is now known as jpena | 12:25 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/486567 | 12:26 |
| *** funzo has quit IRC | 12:27 | |
| *** morazi has joined #tripleo | 12:28 | |
| *** liverpooler has joined #tripleo | 12:28 | |
| *** bfournie has joined #tripleo | 12:29 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use testr failing --list to determine tempest status https://review.openstack.org/491189 | 12:31 |
| *** psahoo has quit IRC | 12:32 | |
| *** brault has joined #tripleo | 12:35 | |
| *** gkadam has quit IRC | 12:36 | |
| *** dsavineau has joined #tripleo | 12:38 | |
| chandankumar | arxcruz: EmilienM mwhahaha https://review.openstack.org/#/c/491189/ | 12:38 |
| *** marios has joined #tripleo | 12:42 | |
| *** rbowen has joined #tripleo | 12:43 | |
| *** liverpooler has quit IRC | 12:43 | |
| *** marios_ has joined #tripleo | 12:43 | |
| *** liverpooler has joined #tripleo | 12:43 | |
| arxcruz | chandankumar: I keep my -1 | 12:44 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-common master: Use correct path in healthcheck scripts https://review.openstack.org/493528 | 12:46 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Don't unregister on system/resource delete https://review.openstack.org/492970 | 12:47 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Fix Heat condition for RHEL registration yum update https://review.openstack.org/492632 | 12:47 |
| *** jmelvin has joined #tripleo | 12:50 | |
| *** limao has joined #tripleo | 12:50 | |
| *** shreshtha has joined #tripleo | 12:52 | |
| *** sshnaidm|mtg has quit IRC | 12:52 | |
| *** limao_ has joined #tripleo | 12:53 | |
| *** ansmith has joined #tripleo | 12:54 | |
| *** adarazs is now known as adarazs_brb | 12:54 | |
| *** limao has quit IRC | 12:55 | |
| *** marios_ has quit IRC | 12:57 | |
| *** marios has quit IRC | 12:58 | |
| *** brault has quit IRC | 12:59 | |
| *** marios has joined #tripleo | 12:59 | |
| *** marios_ has joined #tripleo | 13:00 | |
| *** jaganathan has quit IRC | 13:00 | |
| honza | d0ugal: any luck on the cron trigger thing? | 13:01 |
| *** zshi_laptop has quit IRC | 13:03 | |
| *** brault has joined #tripleo | 13:04 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Add 'discover and enroll nodes' workflow https://review.openstack.org/488526 | 13:04 |
| *** psachin has quit IRC | 13:06 | |
| *** jcoufal has joined #tripleo | 13:06 | |
| *** sshnaidm|mtg has joined #tripleo | 13:07 | |
| *** trown is now known as trown|brb | 13:07 | |
| *** deadnull has quit IRC | 13:07 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Add Neutron SR-IOV agent container https://review.openstack.org/469066 | 13:08 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Add 'discover and enroll nodes' workflow https://review.openstack.org/488526 | 13:09 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 13:10 |
| *** marios_ has quit IRC | 13:10 | |
| *** marios_ has joined #tripleo | 13:10 | |
| jaosorior | matbu: you around? | 13:11 |
| *** adarazs_brb is now known as adarazs | 13:11 | |
| *** limao_ has quit IRC | 13:12 | |
| *** limao has joined #tripleo | 13:12 | |
| *** pchavva has joined #tripleo | 13:12 | |
| *** agopi|away has joined #tripleo | 13:13 | |
| shardy | bnemec: Hey, if you get a moment please could you revisit https://review.openstack.org/#/c/474486/ - I'm touching the same files for composable networks so it'd be good to land that if you're happy it all works OK now | 13:14 |
| *** sshnaidm|mtg is now known as sshnaidm | 13:22 | |
| *** funzo has joined #tripleo | 13:23 | |
| *** toure|gone is now known as toure | 13:24 | |
| *** trown|brb is now known as trown | 13:25 | |
| *** brault has quit IRC | 13:25 | |
| *** funzo has quit IRC | 13:28 | |
| apetrich | honza, I have a fix for https://review.openstack.org/#/c/469608/ I'm just checking tests and I will release in a bit | 13:31 |
| honza | apetrich: awesome! | 13:31 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 13:33 |
| *** limao has quit IRC | 13:33 | |
| *** limao has joined #tripleo | 13:34 | |
| *** lblanchard has joined #tripleo | 13:34 | |
| *** jlabarre has quit IRC | 13:36 | |
| *** jlabarre has joined #tripleo | 13:39 | |
| rcernin | shardy: quick question we are seeing timeouts on heat stack-create/stack-delete instantly, we dont see much information at the logs. it works like in 50%. | 13:40 |
| rcernin | shardy: what packages would be relevant here, python-oslo-messages, rabbitmq and anything else? | 13:40 |
| *** ecerquei has joined #tripleo | 13:40 | |
| shardy | rcernin: when you say instantly, the failure happens without waiting for the stack timeout to expire (default is 4 hours) ? | 13:41 |
| shardy | If so what's the error message? | 13:41 |
| *** dixiaoli has joined #tripleo | 13:41 | |
| rcernin | shardy: its overcloud stack, yes | 13:41 |
| rcernin | shardy: its instantly once we run the heat stack-create -f. | 13:42 |
| shardy | rcernin: Ok, I'd check the status of rabbit on the nodes, and that the heat-engine is running | 13:42 |
| shardy | I'd expect the error message to say about an RPC timeout though, any clues in the heat-api logs? | 13:43 |
| *** dixiaoli has quit IRC | 13:43 | |
| *** dixiaoli has joined #tripleo | 13:43 | |
| *** brault has joined #tripleo | 13:45 | |
| rcernin | yeah heat-engine shows down heat-engine | 13:47 |
| *** dixiaoli has quit IRC | 13:47 | |
| rcernin | shardy: heat service-list | 13:47 |
| *** dixiaoli has joined #tripleo | 13:47 | |
| *** ebarrera has quit IRC | 13:47 | |
| *** links has quit IRC | 13:47 | |
| *** ebarrera has joined #tripleo | 13:48 | |
| rcernin | but heat-engine is started on the nodes, time is synced, only SAWarning: Unicode type received non-unicode bind param value 'data-processing-cluster' | 13:48 |
| *** gbarros has joined #tripleo | 13:50 | |
| janki | jaosorior, hey | 13:50 |
| *** myoung|remote has joined #tripleo | 13:52 | |
| *** myoung|remote has quit IRC | 13:53 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-common master: Use correct path in healthcheck scripts https://review.openstack.org/493528 | 13:53 |
| EmilienM | hello | 13:54 |
| *** limao has quit IRC | 13:54 | |
| *** dixiaoli has quit IRC | 13:54 | |
| *** myoung|remote has joined #tripleo | 13:55 | |
| *** limao has joined #tripleo | 13:55 | |
| *** myoung|remote has quit IRC | 13:55 | |
| *** myoung|remote has joined #tripleo | 13:57 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-common master: Use correct path in healthcheck scripts https://review.openstack.org/493528 | 13:57 |
| rcernin | shardy: in service-list we see 8 heat-engines down and 8 up. | 13:57 |
| rcernin | shardy: could that be the reason why it fails in 50%? | 13:57 |
| apetrich | honza, If you can make https://review.openstack.org/#/c/469608/ to depend on https://review.openstack.org/493550 it should work | 13:57 |
| rcernin | shardy: we only have 8 heat-engines, can I get rid of those 8 heat-engines that are down but supposed to be down? | 13:57 |
| *** myoung|remote has quit IRC | 13:58 | |
| honza | apetrich: perfect, doing that now, thank you | 13:58 |
| *** myoung|remote has joined #tripleo | 13:58 | |
| apetrich | honza, no worries. I will keep an eye on it too | 13:58 |
| abishop | jaosorior: are you available to discuss barbican? | 13:58 |
| rcernin | shardy: why it shows 16 heat-engines when the nodes have only 8, 4 per controller | 13:59 |
| d0ugal | honza: woops, I just seen your message here, but as you can see apetrich it doing all the hard work :) | 13:59 |
| honza | :) | 13:59 |
| *** dixiaoli has joined #tripleo | 13:59 | |
| *** myoung|remote has quit IRC | 13:59 | |
| abishop | d0ugal: still hoping you can review https://review.openstack.org/480587 and https://review.openstack.org/489239 | 13:59 |
| honza | apetrich: is that issue severe enough on its to make into pike? my logging patches have an FFE if not | 14:00 |
| *** myoung|remote has joined #tripleo | 14:00 | |
| *** myoung|remote has quit IRC | 14:00 | |
| *** brault has quit IRC | 14:00 | |
| d0ugal | abishop: sure, I'll do that now. | 14:00 |
| *** ramishra has quit IRC | 14:00 | |
| abishop | thx! | 14:01 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add log to validate ipmi role https://review.openstack.org/492117 | 14:01 |
| shardy | rcernin: have you tried restarting the openstack-heat-engine service on all nodes? | 14:02 |
| rcernin | doing it now including heat-api | 14:02 |
| shardy | the inconsistent count does sound wrong - is rabbit running OK? | 14:02 |
| openstackgerrit | Honza Pokorny proposed openstack/instack-undercloud master: Add an hourly cron trigger for tripleo-ui logging https://review.openstack.org/469608 | 14:03 |
| rcernin | shardy: now they are back up | 14:03 |
| *** myoung|remote has joined #tripleo | 14:03 | |
| *** myoung|remote has quit IRC | 14:03 | |
| *** chlong has joined #tripleo | 14:04 | |
| rcernin | shardy: one again after heat stack-create the heat-engines are down, but we have 8 total why it shows 16 ? | 14:04 |
| rcernin | shardy: we have two controllers num_engine_workers = 4 | 14:05 |
| rcernin | that would make it total 8 heat-engine workers | 14:05 |
| jaosorior | janki: hey | 14:05 |
| jaosorior | abishop: sup | 14:05 |
| janki | jaosorior, I need help understanding jenkins error. DO you have a minute? | 14:05 |
| abishop | jaosorior: I/we have an action item to determine what work may be required for Queens | 14:06 |
| apetrich | honza, I think it is. d0ugal what do you think ^^ | 14:06 |
| abishop | jaosorior: here's my starting point... | 14:06 |
| d0ugal | honza: yes, it is a bug. | 14:06 |
| abishop | jaosorior: I fixed the problem that restores the legacy key manager when Barbican is *not* enabled | 14:07 |
| *** ramishra has joined #tripleo | 14:07 | |
| jaosorior | abishop: that's good | 14:07 |
| abishop | jaosorior: I also understand barbican works fine when TripleO deploys it | 14:07 |
| jaosorior | abishop: fine-ish | 14:07 |
| abishop | (btw, that review passed Jenkins, and just needs workflow) | 14:07 |
| jaosorior | abishop: we're using the file-based backend (which is not ideal) | 14:07 |
| jaosorior | abishop: pass me the link | 14:08 |
| abishop | jaosorior: https://review.openstack.org/#/c/488509/ | 14:08 |
| abishop | jaosorior: I don't know about backends, etc. not up to speed on those details. I got involved when cinder guy notified me about their and nova's changes | 14:09 |
| shardy | rcernin: I suspect you have some stale DB objects related to services that aren't running - did a previous engine die uncleanly? | 14:09 |
| abishop | jaosorior: but one thing that I did wonder about is how existing deployments can migrate from fixed key (legacy) manager to barbican | 14:09 |
| shardy | rcernin: you might want to stop heat-* again and try to clean up with heat-manage service clean? | 14:09 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 14:10 |
| jaosorior | abishop: well, there currently is no way of automatically doing a migration, and that's not something the barbican project is looking into either. You would need to manually migrate the key material to now be hosted by barbican. Also, I'm not sure if we even supported encrypted volumes without barbican. | 14:11 |
| abishop | jaosorior: my cinder work-peer is seeing if it's possible to migrate keys entirely within cinder, in which case it won't be an issue for OOO | 14:11 |
| *** sbrzozow has joined #tripleo | 14:13 | |
| *** marios has quit IRC | 14:13 | |
| *** marios_ is now known as marios | 14:14 | |
| *** ecerquei_ has joined #tripleo | 14:14 | |
| *** ecerquei has quit IRC | 14:14 | |
| abishop | jaosorior: my immediate concern is closing OOO meeting action item, which includes determining if something within my purview is needed for PTG | 14:15 |
| abishop | jaosorior: I'm not a barbican/encryption guy, so from my perspective I'm staying in touch with cinder guy on his idea of migrating key within cinder | 14:15 |
| jaosorior | abishop: well, if it's possible to do it via cinder, all that TripleO would need is some steps that we would run on upgrade/update when barbican is added to the mix. | 14:16 |
| *** limao has quit IRC | 14:16 | |
| *** ecerquei_ has quit IRC | 14:16 | |
| *** ecerquei has joined #tripleo | 14:16 | |
| jaosorior | abishop: I guess that's also something that we could do in TripleO, in case the cinder-based option doesn't work out. | 14:16 |
| *** limao has joined #tripleo | 14:16 | |
| jaosorior | abishop: but it wasn't on our backlog. | 14:16 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Automatically retry introspection for failing nodes https://review.openstack.org/462916 | 14:16 |
| abishop | jaosorior: ack on all points | 14:17 |
| jaosorior | janki: Is it the barbican patch? | 14:17 |
| janki | jaosorior, yes | 14:17 |
| jaosorior | janki: I'll check it out later. I'm working on another issue right now :( | 14:17 |
| janki | jaosorior, sure. I have added configimage to overclopud image - https://review.openstack.org/#/c/493493/ | 14:18 |
| abishop | jaosorior: do you agree there's no immediate action, and we see how cinder effort pans out? | 14:18 |
| dalvarez | beagles, ping question about tripleO :) | 14:19 |
| jaosorior | abishop: sounds good to me | 14:20 |
| abishop | jaosorior: cool, thx | 14:21 |
| bfournie | hi shardy, awesome that most composable networks patches merged. When is tripleo pike feature freeze, i.e. I assume those merges go into pike? | 14:21 |
| *** catintheroof has joined #tripleo | 14:21 | |
| beagles | dalvarez, hey, fire away | 14:21 |
| dalvarez | beagles, im trying to work out the necessary changes in tripleO to deploy/configure networking-ovn-metadata-agent. First thing I created a ovn-metadata.yaml from the neutron-metadata.yaml file in tripleo-heat-templates. Then I thought of creating the puppet file in puppet-neutron but since it'll have some parameters from neutron like state_path, do you tihnk it makes more sense to create an ovn_metadata.pp in puppet-neutron repo instead? | 14:21 |
| shardy | bfournie: yeah it's pretty close now, I'm testing the remaining under review patches locally now | 14:21 |
| shardy | bfournie: we're in feature freeze now, but composable-networks has been granted a feature freeze exception, along with a few other things | 14:22 |
| shardy | https://launchpad.net/tripleo/+milestone/pike-rc1 | 14:22 |
| bfournie | shardy: great, thanks | 14:22 |
| beagles | dalvarez, processing ... :) | 14:23 |
| dalvarez | beagles, thanks :) | 14:24 |
| *** funzo has joined #tripleo | 14:24 | |
| beagles | dalvarez, did you mean that "I thought of creating the puppet file in puppet-ovn but since it'll have some parameters from neutron..."? | 14:26 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: TLS for containerized MySQL https://review.openstack.org/493561 | 14:26 |
| dalvarez | beagles, right sorry! | 14:27 |
| dalvarez | i meant puppet-ovn yes | 14:27 |
| numans | beagles, since you are talking about ovn, are you happy workflow+1 this one https://review.openstack.org/#/c/468860/ :) | 14:28 |
| dalvarez | numans++ :) | 14:29 |
| *** funzo has quit IRC | 14:29 | |
| rcernin | shardy: so we did the cleanup of dead agents and yet it is still timing out sometimes :-/ | 14:31 |
| *** dixiaoli has quit IRC | 14:31 | |
| beagles | numans, I can - but it might be more legit to get a containers team member to approve it though | 14:32 |
| numans | beagles, ack | 14:32 |
| *** dixiaoli has joined #tripleo | 14:33 | |
| numans | beagles, thanks for the review | 14:33 |
| *** dixiaoli has quit IRC | 14:33 | |
| beagles | dalvarez, the criteria I would use here is if it is for configuring OVN, then it probably belongs in puppet-ovn, if it is for configuring some kind of integration for neutron, it either goes in puppet-neutron or you move it up a level and do something in puppet-tripleo | 14:33 |
| beagles | dalvarez, it sounds like this is possibly more of a puppet-neutron thing | 14:34 |
| dalvarez | beagles, it's not actually core ovn but the metadata-agent for networking-ovn (integration between ovn <-> neutron) | 14:34 |
| dalvarez | gotcha | 14:34 |
| dalvarez | beagles, thanks, it makes sense! | 14:34 |
| beagles | dalvarez, np | 14:34 |
| *** limao has quit IRC | 14:37 | |
| *** limao has joined #tripleo | 14:37 | |
| beagles | numans, np , bogdando, mandre, please see numans query about workflow for https://review.openstack.org/#/c/468860/ | 14:38 |
| *** mrchao has quit IRC | 14:40 | |
| *** chem has quit IRC | 14:44 | |
| *** chem has joined #tripleo | 14:46 | |
| *** agurenko has quit IRC | 14:55 | |
| oidgar | does anyone here deployed overcloud with containers in the last days successfully? | 14:57 |
| oidgar | it keep failing for me even if I reinstall undercloud | 14:58 |
| *** limao has quit IRC | 14:58 | |
| jaosorior | oidgar: I did | 14:59 |
| *** yamahata has joined #tripleo | 14:59 | |
| *** limao has joined #tripleo | 14:59 | |
| jaosorior | oidgar: what's the error? | 14:59 |
| *** _milan_ has quit IRC | 14:59 | |
| *** ramishra has quit IRC | 15:00 | |
| *** rcernin has quit IRC | 15:00 | |
| *** athomas has quit IRC | 15:00 | |
| oidgar | jaosorior, 2017-08-14 14:49:24,062 ERROR: 19191 -- Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend | 15:00 |
| oidgar | I'm trying to deploy with octavia, which worked for me for a long time and now it seems like something is broken... | 15:01 |
| oidgar | in addition: Failed to get D-Bus connection: Operation not permitted" | 15:01 |
| *** athomas has joined #tripleo | 15:01 | |
| oidgar | I think the problem arises at docker-puppet | 15:01 |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC Isilon Manila Driver https://review.openstack.org/488889 | 15:02 |
| jaosorior | oidgar: those seem to me like red herrings | 15:03 |
| jaosorior | oidgar: I see those issues in successful deployments too | 15:03 |
| *** jmelvin has quit IRC | 15:03 | |
| bogdando | folks, let's merge please https://review.openstack.org/#/c/480063/ | 15:04 |
| bogdando | mandre: hi ^^ | 15:04 |
| *** ebarrera has quit IRC | 15:06 | |
| jaosorior | bogdando: done | 15:06 |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VMAX ISCSI Cinder Driver https://review.openstack.org/489624 | 15:06 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Persist containerized services httpd logs https://review.openstack.org/489955 | 15:07 |
| *** janki has quit IRC | 15:07 | |
| *** oidgar has quit IRC | 15:07 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Drop step_config as top level docker requirement https://review.openstack.org/442716 | 15:09 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name https://review.openstack.org/442755 | 15:09 |
| bogdando | jaosorior: thanks | 15:09 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Add docker templates to configure Ironic inspector https://review.openstack.org/457822 | 15:09 |
| bogdando | this one is ready for merge as well https://review.openstack.org/#/c/490068 | 15:09 |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VMAX ISCSI Cinder Driver https://review.openstack.org/489624 | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 15:10 |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC Unity Manila Driver https://review.openstack.org/490581 | 15:11 |
| mwhahaha | pabelanger: did you create a bug for that docker.io thing? | 15:11 |
| bogdando | https://review.openstack.org/#/c/490048/ needs more reviews please | 15:11 |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VNX Manila Driver https://review.openstack.org/490642 | 15:12 |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VNX Manila Driver https://review.openstack.org/490642 | 15:13 |
| pabelanger | mwhahaha: ya, 1710533 | 15:13 |
| jaosorior | bogdando: commented. | 15:14 |
| bandini | mwhahaha: is it okay to remove the W-1 from https://review.openstack.org/#/c/475457/ now? | 15:14 |
| mwhahaha | bandini: why you no have bug or blueprint reference? :D | 15:15 |
| ansiwen | I constantly get an error while deploying containerized openstack with status code 2 in overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0 | 15:15 |
| mwhahaha | bandini: as we're in rc1 i could not determine importance | 15:15 |
| *** mdnadeem has quit IRC | 15:15 | |
| ansiwen | http://paste.openstack.org/show/618322/ | 15:15 |
| ansiwen | anyone knows that? | 15:15 |
| bandini | mwhahaha: ahah right (in my defence we were stuck on that due to upstream puppet-rabbitmqlabs taking a long time to merge the dependent patch) | 15:15 |
| bandini | mwhahaha: I'll do the bug/commit dance | 15:16 |
| mwhahaha | bandini: thanks | 15:16 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for containerized haproxy https://review.openstack.org/489900 | 15:16 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: TLS for containerized MySQL https://review.openstack.org/493561 | 15:16 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/puppet-tripleo master: Add logrotate-crond configuration https://review.openstack.org/490048 | 15:16 |
| openstackgerrit | Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VMAX Manila Driver https://review.openstack.org/490949 | 15:16 |
| mwhahaha | ansiwen: i usually check syslog for the actual failure (since those logs dont' help) | 15:17 |
| *** ccamacho has quit IRC | 15:19 | |
| *** limao has quit IRC | 15:19 | |
| openstackgerrit | Rajini Karthik proposed openstack/tripleo-heat-templates master: Add support for Dell EMC VMAX ISCSI Backend https://review.openstack.org/489393 | 15:20 |
| *** itlinux has joined #tripleo | 15:20 | |
| *** limao has joined #tripleo | 15:20 | |
| *** jfrancoa has quit IRC | 15:21 | |
| *** shardy has quit IRC | 15:22 | |
| *** jfrancoa has joined #tripleo | 15:22 | |
| *** oidgar has joined #tripleo | 15:22 | |
| *** funzo has joined #tripleo | 15:22 | |
| *** shardy has joined #tripleo | 15:23 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Use rabbitmq ipv6 flag https://review.openstack.org/475457 | 15:25 |
| *** limao has quit IRC | 15:26 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Extend VNC port range https://review.openstack.org/493585 | 15:27 |
| *** dparkes has quit IRC | 15:27 | |
| *** jmelvin has joined #tripleo | 15:31 | |
| *** dprince has quit IRC | 15:35 | |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart-extras master: Fix set_overcloud_workers=False being ignored for >mitaka https://review.openstack.org/493586 | 15:35 |
| *** jlinkes has quit IRC | 15:37 | |
| *** nyechiel has quit IRC | 15:38 | |
| *** yprokule has quit IRC | 15:39 | |
| *** itlinux has quit IRC | 15:40 | |
| *** oidgar_ has joined #tripleo | 15:41 | |
| oidgar_ | jaosorior, so the overcloud deployment worked for you? | 15:42 |
| jaosorior | it did | 15:43 |
| *** chlong_ has joined #tripleo | 15:44 | |
| *** itlinux has joined #tripleo | 15:44 | |
| *** chlong has quit IRC | 15:45 | |
| *** oidgar has quit IRC | 15:45 | |
| *** links has joined #tripleo | 15:46 | |
| *** jaganathan has joined #tripleo | 15:47 | |
| oidgar_ | jaosorior, with the latest code? | 15:48 |
| ansiwen | mwhahaha: you mean syslog of the contoller node? I see a lot of errors there, but no idea if they are related or critical. (docker images not found and some netfilter stuff. yes, I ran the overcloud-prep-containers.sh script before...) | 15:51 |
| *** ffiore has joined #tripleo | 15:52 | |
| *** dprince has joined #tripleo | 15:56 | |
| jaosorior | oidgar_: the latest t-h-t and puppet-tripleo. Yeah | 15:57 |
| *** gbarros has quit IRC | 15:58 | |
| oidgar_ | jaosorior, wierd, hope to fix it soon, thank anyway! | 15:58 |
| *** gbarros has joined #tripleo | 15:59 | |
| *** oidgar_ is now known as oidgar | 16:00 | |
| mwhahaha | ansiwen: yea | 16:00 |
| ansiwen | mwhahaha: what command are you currently using for deploying containerized overcloud? | 16:01 |
| *** brault has joined #tripleo | 16:01 | |
| *** tesseract has quit IRC | 16:01 | |
| ansiwen | mwhahaha: or are you using the overcloud-deploy.sh script, that seems to deploy an containerized deployment as well | 16:02 |
| mwhahaha | ansiwen: no I just use the normal deploy commands | 16:02 |
| ansiwen | which environment files do you include? | 16:02 |
| mwhahaha | ansiwen: I just follow the tripleo docs. So the docker.yaml | 16:03 |
| *** psachin has joined #tripleo | 16:03 | |
| ansiwen | mwhahaha: I used `openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e ./containers-default-parameters.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker-centos-tripleoupstream.yaml` as shardy told me once | 16:04 |
| *** jpena is now known as jpena|off | 16:05 | |
| *** dsneddon has joined #tripleo | 16:05 | |
| *** jpena|off is now known as jpena | 16:08 | |
| mwhahaha | ansiwen: yea same. I do the pulling down of images to a local repo but that's the only difference | 16:08 |
| *** egonzalez has quit IRC | 16:08 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 16:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| *** dsneddon has quit IRC | 16:11 | |
| *** jaganathan has quit IRC | 16:11 | |
| EmilienM | mwhahaha: do we have any action on https://bugs.launchpad.net/openstack-gate/+bug/1710533 ? | 16:12 |
| openstack | Launchpad bug 1710533 in tripleo "Job fails to download container from docker.io" [Critical,Triaged] | 16:12 |
| mwhahaha | EmilienM: not yet that I know of, I just added tripleo on it this morning | 16:14 |
| *** rbowen has quit IRC | 16:14 | |
| *** brault has quit IRC | 16:15 | |
| EmilienM | mwhahaha: ok, I'll look at it today probably | 16:16 |
| EmilienM | I think it's in tripleo-quickstart-extras | 16:17 |
| *** rbowen has joined #tripleo | 16:17 | |
| *** dsariel has quit IRC | 16:18 | |
| EmilienM | https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/extras-common/defaults/main.yml#n8 | 16:18 |
| *** aufi has quit IRC | 16:19 | |
| EmilienM | I think we need some logic to use AFS mirror instead of docker.io - not sure where yet | 16:19 |
| *** akrivoka has quit IRC | 16:23 | |
| tosky | I tried to deploy with SSL last master-tripleo-ci but the post-deploy phase of oooq-extras fails with a ' Certificate did not match expected hostname: 192.168.24.10' (the certificate has 10.0.0.5) | 16:23 |
| tosky | did anyone see something like this? | 16:24 |
| tosky | bigger snipped of the failing part: http://paste.openstack.org/show/618327/ | 16:24 |
| *** dsneddon has joined #tripleo | 16:26 | |
| jaosorior | tosky: did you deploy without network isolation? | 16:28 |
| tosky | jaosorior: I did not use it, correct | 16:28 |
| tosky | jaosorior: is it documentated in the documentation that I didn't check? | 16:29 |
| jaosorior | tosky: right, so it seems that the SSL code in oooq-extras expects net-iso. | 16:29 |
| jaosorior | tosky: I have no idea :D | 16:29 |
| tosky | jaosorior: oh, I see, thanks | 16:29 |
| jaosorior | seems like a oooq-extras bug | 16:29 |
| tosky | I will redeploy | 16:29 |
| jaosorior | use network isolation, that will do the trick probably :) | 16:29 |
| tosky | do I need all 3 network-isolation-absolute.yaml network-environment.yaml and net-iso.yaml ? | 16:29 |
| jaosorior | tosky: else, you gotta look in oooq-extras to see if there's a way to set the IP that's used for the subjectaltname | 16:30 |
| tosky | nah, easier to get something working - I need to test sahara, so on an higher level | 16:30 |
| tosky | as long as the underlying parts work, it's fine | 16:30 |
| *** snecklifter has quit IRC | 16:31 | |
| tosky | and they are not in oooq apparently, but in tripleo-ci | 16:32 |
| tosky | uhm | 16:32 |
| jaosorior | tosky: it's done in a different way in both sides. tripleo-ci has it's own way of doing the public cert. And tripleo-quickstart has it's own way (which is more dynamic as far as I remember() | 16:33 |
| tosky | jaosorior: I was checking the existing featureset where network isolation is implemented, and they refer to the 3 YAMLs above, which are in tripleo-ci | 16:33 |
| tosky | so should I refer to them or is there another way which do not require them? | 16:34 |
| *** snecklifter has joined #tripleo | 16:35 | |
| jaosorior | tosky: don't know :/ | 16:36 |
| tosky | jaosorior: thanks anyway so far | 16:36 |
| *** ykarel has quit IRC | 16:37 | |
| *** nyechiel has joined #tripleo | 16:38 | |
| itlinux | jaosorior: hi are you around? | 16:39 |
| bnemec | tosky: You need net-iso templates from somewhere. tripleo-ci is probably the safest place to get them. | 16:39 |
| jaosorior | itlinux: maybe :D | 16:39 |
| weshay | chandankumar, https://bluejeans.com/u/emacchi/ | 16:39 |
| jaosorior | itlinux: what's up? | 16:39 |
| itlinux | :) sounds good jaosorior: | 16:39 |
| bnemec | tosky: OVB has templates as well, but there is likely some slight variation from the tripleo-ci ones. | 16:40 |
| bnemec | Also I have no idea if either work in a libvirt environment. | 16:40 |
| bnemec | I _think_ so, but no promises. | 16:40 |
| tosky | bnemec: so all the 3 yamls? | 16:40 |
| tosky | also, does it mean that no one tried SSL in a nested libvirt environment? | 16:41 |
| *** jfrancoa has quit IRC | 16:41 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Send logs size to grahite site https://review.openstack.org/481159 | 16:42 |
| bnemec | tosky: Yeah, all three should work. | 16:42 |
| tosky | bnemec: I will check for sure, thanks | 16:42 |
| bnemec | I don't know whether anyone has deployed SSL in libvirt. | 16:42 |
| bnemec | I'm in the process of doing some SSL testing myself, but it's in OVB. | 16:43 |
| *** ebarrera has joined #tripleo | 16:45 | |
| *** stendulker has joined #tripleo | 16:46 | |
| *** oidgar has quit IRC | 16:47 | |
| *** cylopez has quit IRC | 16:48 | |
| *** ecerquei has quit IRC | 16:48 | |
| *** ecerquei has joined #tripleo | 16:49 | |
| *** snecklifter has quit IRC | 16:49 | |
| *** jpena is now known as jpena|off | 16:50 | |
| *** gkadam has joined #tripleo | 16:53 | |
| *** yamahata has quit IRC | 16:54 | |
| *** morazi has quit IRC | 16:54 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/486567 | 16:56 |
| *** milan has joined #tripleo | 16:56 | |
| *** shreshtha has quit IRC | 16:57 | |
| *** derekh has quit IRC | 16:57 | |
| *** rcernin has joined #tripleo | 16:59 | |
| *** jlabarre has quit IRC | 17:05 | |
| *** links has quit IRC | 17:05 | |
| *** jlabarre has joined #tripleo | 17:05 | |
| *** lucasagomes is now known as lucas-afk | 17:06 | |
| *** sshnaidm has quit IRC | 17:06 | |
| *** jpich has quit IRC | 17:06 | |
| *** ffiore has quit IRC | 17:09 | |
| openstackgerrit | Marios Andreou proposed openstack/python-tripleoclient master: Adds when in upgrade_tasks playbook written by config download https://review.openstack.org/491749 | 17:09 |
| pabelanger | mwhahaha: bnemec: EmilienM: we need to update DLRN to use AFS mirrors: http://logs.openstack.org/09/488509/2/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/ffbef54/logs/delorean_logs/2f/a7/2fa7a2ba6dc7f3e0e71695aab8a7f25e0475bd1a_dev/root.log.txt.gz | 17:09 |
| pabelanger | failing to download from buildlogs.centos.org, suggestions on who can fix this? | 17:10 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 17:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/486567 | 17:11 |
| EmilienM | pabelanger: looking now | 17:11 |
| pabelanger | bug 1710678 | 17:11 |
| openstack | bug 1710678 in tripleo "DLRN no configured to use regional mirrors" [Undecided,New] https://launchpad.net/bugs/1710678 | 17:11 |
| EmilienM | amoralej ^ sounds like tricky this one :D | 17:12 |
| pabelanger | IIRC, you can update mock.cfg files and point to new mirrors | 17:12 |
| EmilienM | ah | 17:12 |
| pabelanger | will need to do that for any repo DLRN is using | 17:12 |
| EmilienM | pabelanger: can you file a bug please and assign it to me? | 17:12 |
| EmilienM | ok, I'll work on it | 17:12 |
| pabelanger | EmilienM: not sure how to assign it to you, I get No items matched "emilienm". | 17:14 |
| pabelanger | EmilienM: bug 1710678 | 17:14 |
| openstack | bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [Undecided,New] https://launchpad.net/bugs/1710678 | 17:14 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Also write an upgrade_tasks_playbook https://review.openstack.org/490848 | 17:14 |
| EmilienM | pabelanger: thx, I take this one | 17:15 |
| pabelanger | EmilienM: cool, thanks! | 17:15 |
| amoralej | EmilienM, pabelanger, so you want to patch dlrn config to use infra mirrors? | 17:15 |
| shardy | Hey folks, can I get a second reviewer for https://review.openstack.org/#/c/492218/ if anyone has a moment | 17:15 |
| shardy | jaosorior: If you're still around perhaps you can check it, experimental job passed | 17:15 |
| pabelanger | amoralej: ya | 17:15 |
| EmilienM | shardy: done | 17:16 |
| shardy | EmilienM: thanks! | 17:16 |
| EmilienM | amoralej: yes, maybe we can pair on that, I did some work already for repos | 17:16 |
| EmilienM | amoralej: not sure how we can do it in delorean, not familiar with mock.cfg files | 17:16 |
| amoralej | EmilienM, i'm looking | 17:16 |
| pabelanger | EmilienM: mwhahaha: also replaced mirror.kn1.citycloud.o.o with larger flavor. Hoping that fixes some of the yum download issues we are seeing in that region | 17:16 |
| *** jlabarre has quit IRC | 17:17 | |
| pabelanger | EmilienM: amoralej: you'll likely need to create a new mock.cfg but setup using our mirrors, then pass that .cfg file into DLRN | 17:17 |
| EmilienM | amoralej: maybe you can take https://bugs.launchpad.net/tripleo/+bug/1710678 and I'll take https://bugs.launchpad.net/openstack-gate/+bug/1710533 | 17:18 |
| openstack | Launchpad bug 1710678 in tripleo "DLRN not configured to use regional mirrors" [High,Triaged] - Assigned to Emilien Macchi (emilienm) | 17:18 |
| openstack | Launchpad bug 1710533 in tripleo "Job fails to download container from docker.io" [Critical,Triaged] | 17:18 |
| EmilienM | pabelanger: did we have any other thing beside these 2 issues ^ ? | 17:18 |
| *** jlabarre has joined #tripleo | 17:19 | |
| amoralej | EmilienM, are jobs still using tripleo.sh or we should only take care of oooq? | 17:19 |
| pabelanger | EmilienM: bogdando found https://review.openstack.org/#/c/493520/ this morning, but haven't looked into why yet | 17:19 |
| EmilienM | amoralej: priority is oooq, as majority of jobs are using it - but we need to patch tripleo-ci as well I think | 17:19 |
| pabelanger | EmilienM: Oh, there is the tuned missing package in containers still: http://logs.openstack.org/83/489083/6/gate/gate-tripleo-ci-centos-7-undercloud-containers/dcd7ded/logs/var/log/undercloud_install.txt.gz#_2017-08-14_10_25_53_000 I think mwhahaha suggested a revert of that patch | 17:20 |
| mwhahaha | it doesn't actually fail it but yea | 17:21 |
| mwhahaha | or it needs to figured out how to diable it for containers | 17:21 |
| pabelanger | EmilienM: we also need to have container jobs create a logstash.txt file, so we can properly index their logs, today we are blind in logstash.o.o for most of those failures: http://status.openstack.org/elastic-recheck/data/others.html gate-tripleo-ci-centos-7-undercloud-containers (63) | 17:21 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-log-client.yaml#n613 | 17:22 |
| EmilienM | ok I'm also taking action on the container/logstash thing | 17:23 |
| EmilienM | need help on the tuned though | 17:23 |
| *** gfidente is now known as gfidente|afk | 17:24 | |
| EmilienM | mwhahaha, shardy: since https://review.openstack.org/#/c/489874/ hasn't merged yet, I suggest we backport it in the package manually so we can start testing upgrades for real - wdyt? | 17:24 |
| mwhahaha | probably | 17:24 |
| amoralej | EmilienM, its tricky, dlrn creates a mock config file on the fly taking parts from different sources | 17:25 |
| *** bogdando has quit IRC | 17:26 | |
| bnemec | jaosorior: Why do the new SSL docs recommend setting the CN as "TripleO CA" on the self-signed cert? | 17:27 |
| bnemec | Not using the actual hostname there is causing SSL errors for me. | 17:27 |
| jaosorior | bnemec: cause that's not for the server certificate. That's for the CA certificate. | 17:28 |
| jaosorior | bnemec: the recommendation is to use two certs, one being a self-signed CA certificate. | 17:28 |
| EmilienM | pabelanger: feel free to add details: https://bugs.launchpad.net/tripleo/+bug/1710682 but i'll work on it | 17:28 |
| openstack | Launchpad bug 1710682 in tripleo "CI: container jobs should create logstash.txt" [High,Triaged] - Assigned to Emilien Macchi (emilienm) | 17:28 |
| EmilienM | amoralej: tbh, I don't know | 17:28 |
| bnemec | jaosorior: Oh, I misread the docs. :-( | 17:29 |
| jaosorior | bnemec: if you think they're not clear enough (which they might not be) lets push a fix. This stuff is not so easy :/ so might as well try to remove any ambiguities | 17:29 |
| amoralej | EmilienM, DLRN creates a config file by appending <baseurl>/delorean-deps.repo to https://github.com/softwarefactory-project/DLRN/blob/master/scripts/centos.cfg | 17:30 |
| mwhahaha | jfyi, puppet-tripleo unit tests are broken in stable/newton | 17:30 |
| EmilienM | mwhahaha: what broke? | 17:30 |
| mwhahaha | jaosorior: i know itlinux was struggling with CA issues using self signed certs | 17:30 |
| amoralej | EmilienM, i'll figure out what we can do | 17:30 |
| mwhahaha | EmilienM: update to stdlib it looks like , as well as the selinux unit test thing that we fixed in ocata/master | 17:30 |
| mwhahaha | EmilienM: http://logs.openstack.org/78/490578/2/check/gate-puppet-tripleo-puppet-unit-4.8-centos-7/3d6594a/console.html#_2017-08-14_03_37_57_518831 | 17:31 |
| *** jtomasek has quit IRC | 17:31 | |
| *** ebarrera has quit IRC | 17:31 | |
| EmilienM | mwhahaha: pin + backport? | 17:31 |
| mwhahaha | probably | 17:31 |
| mwhahaha | Stdlib::Absolutepath seems missing | 17:31 |
| * mwhahaha goes looking | 17:31 | |
| mwhahaha | also going to file a bug | 17:31 |
| *** jtomasek has joined #tripleo | 17:32 | |
| bnemec | jaosorior: No, I just mentally skipped over the "not" in "the CN should not be the same as the one we’ll give the leaf certificate." | 17:33 |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/tripleo-heat-templates master: Containerize Manila Share for HA https://review.openstack.org/482680 | 17:33 |
| bnemec | So when I got to the leaf certificate I set it to TripleO CA too. | 17:33 |
| bnemec | Pure user error. :-) | 17:33 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo stable/newton: Fix selinux unit tests https://review.openstack.org/493619 | 17:34 |
| *** tosky has quit IRC | 17:34 | |
| *** gkadam has quit IRC | 17:35 | |
| *** jlabarre has quit IRC | 17:35 | |
| *** yamahata has joined #tripleo | 17:36 | |
| *** jlabarre has joined #tripleo | 17:37 | |
| mwhahaha | EmilienM: it looks like it's because systemd updated recently | 17:38 |
| *** ykarel has joined #tripleo | 17:38 | |
| *** mmethot has joined #tripleo | 17:38 | |
| EmilienM | mwhahaha: puppet-systemd? | 17:38 |
| mwhahaha | yea | 17:39 |
| EmilienM | ah and we don't pin it in Puppetfile_extras? | 17:39 |
| EmilienM | haha! | 17:39 |
| mwhahaha | yea i think so | 17:39 |
| EmilienM | let's pin that | 17:39 |
| mwhahaha | i think in newton we have an older stdlib | 17:39 |
| mwhahaha | let me see | 17:39 |
| mwhahaha | https://github.com/camptocamp/puppet-systemd/blob/6e1414c230ba29473f21e82fa4e85a6fccafa571/manifests/service_limits.pp#L31 | 17:39 |
| mwhahaha | it's this puppet4 stuff | 17:39 |
| mwhahaha | yea we're still tracking master for systemd in newton which is probably not right | 17:40 |
| *** pkovar has quit IRC | 17:41 | |
| EmilienM | :-O | 17:41 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: send graphite stats from ARA https://review.openstack.org/480121 | 17:41 |
| *** masco has quit IRC | 17:42 | |
| *** dtrainor has quit IRC | 17:42 | |
| *** brault has joined #tripleo | 17:42 | |
| *** dtrainor has joined #tripleo | 17:42 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Support initramfs images in modify-image role https://review.openstack.org/479896 | 17:42 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: Add support for updating initramfs images in CI https://review.openstack.org/479900 | 17:42 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo stable/newton: Fix selinux unit tests https://review.openstack.org/493619 | 17:43 |
| mwhahaha | EmilienM: -^ pulled it in with the other fix | 17:43 |
| pabelanger | weshay: left question on ^ | 17:43 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo stable/newton: Enable innodb_buffer_pool_size configuration https://review.openstack.org/490578 | 17:45 |
| *** mmethot has quit IRC | 17:45 | |
| EmilienM | mwhahaha: +2 +A | 17:45 |
| *** marios has quit IRC | 17:46 | |
| *** brault has quit IRC | 17:47 | |
| pabelanger | sadface | 17:48 |
| pabelanger | gate job failed to download from docker.io | 17:48 |
| pabelanger | http://logs.openstack.org/59/491759/3/gate/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container/ff19d03/logs/undercloud/home/jenkins/overcloud_prep_containers.log.txt.gz#_2017-08-14_17_40_26 | 17:48 |
| pabelanger | then gate reset | 17:49 |
| EmilienM | pabelanger: I'm working on it like right now :D | 17:49 |
| EmilienM | I'm preparing a patch | 17:49 |
| pabelanger | EmilienM: cool, ping me when testing and I'll check apache logs | 17:49 |
| EmilienM | I will | 17:50 |
| toure | trown ping | 17:53 |
| *** shardy has quit IRC | 17:54 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for containerized MySQL https://review.openstack.org/493561 | 17:56 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for containerized RabbitMQ https://review.openstack.org/493632 | 17:56 |
| *** mcornea has quit IRC | 17:58 | |
| trown | toure: are the multiple changes in the same repo? | 17:59 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Use ssl-enabled keystone admin endpoint by default https://review.openstack.org/426348 | 18:00 |
| trown | toure: ie multiple tht changes or multiple mistral changes | 18:00 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Use AFS mirrors to download containers instead of docker.io https://review.openstack.org/493634 | 18:01 |
| EmilienM | pabelanger: ^ first patch | 18:01 |
| *** dtantsur is now known as dtantsur|afk | 18:01 | |
| EmilienM | weshay: I'll need your review on that patch, when you have time, please | 18:01 |
| *** ykarel has quit IRC | 18:02 | |
| *** tosky has joined #tripleo | 18:03 | |
| toure | trown yup | 18:03 |
| toure | trown one is dependant of the other | 18:04 |
| trown | toure: ya, you need to have them based on eachother in git, and only point to the top one | 18:04 |
| toure | k | 18:04 |
| toure | thanks for the insight | 18:05 |
| *** stendulker has quit IRC | 18:05 | |
| EmilienM | pabelanger: re: gate-tripleo-ci-centos-7-undercloud-containers, I'm confused why you're asking for this one - what log files do you want to see ? | 18:07 |
| chandankumar | weshay: EmilienM i was not around that time so missed, the Bj call | 18:07 |
| pabelanger | EmilienM: something like http://logs.openstack.org/81/490581/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/62ed68e/logs/undercloud/var/log/extra/logstash.txt That is how we index the logs from tripleo-ci-noha jobs, so we things like elastic-recheck or other logstash.o.o tooks can find patterns. Right now with containers jobs, we are blind | 18:09 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 18:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1710533 in tripleo "Job fails to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| EmilienM | pabelanger: ah ok, I got it now. It's because gate-tripleo-ci-centos-7-undercloud-containers is not run by quickstart | 18:10 |
| pabelanger | EmilienM: Ya, do you know why that is? | 18:11 |
| EmilienM | trown: do you know if we have that in our backlog (far or near) : having gate-tripleo-ci-centos-7-undercloud-containers executed by quickstart instead of tripleo-ci | 18:11 |
| *** sshnaidm has joined #tripleo | 18:13 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Add Neutron SR-IOV agent container https://review.openstack.org/469066 | 18:15 |
| *** salmankhan has quit IRC | 18:15 | |
| trown | EmilienM: hmm... thought we only had updates job left after 3nodes... guess that job was added after our transition plan? in any case I dont see a card for it on https://trello.com/b/U1ITy0cu/tripleo-ci-squad | 18:16 |
| EmilienM | trown: ok, this is not high prio I guess | 18:16 |
| EmilienM | amoralej: can I assign you to the dlrn thing? | 18:17 |
| amoralej | EmilienM, ok | 18:17 |
| EmilienM | thx | 18:17 |
| amoralej | i'm on pto tomorrow | 18:17 |
| EmilienM | ah | 18:17 |
| EmilienM | amoralej: maybe we can email jpena as well | 18:18 |
| amoralej | EmilienM, yes, i need to talk to him about the best way to do it | 18:18 |
| amoralej | but he's on pto also, :), it's local holiday in Madrid | 18:18 |
| EmilienM | ah! | 18:19 |
| *** d0ugal has quit IRC | 18:19 | |
| *** jlabarre has quit IRC | 18:25 | |
| *** jlabarre has joined #tripleo | 18:26 | |
| *** morazi has joined #tripleo | 18:27 | |
| *** psachin has quit IRC | 18:30 | |
| pabelanger | EmilienM: another thing to note about undercloud-containers, its missing DLRN log files too: http://logs.openstack.org/95/487795/1/gate/gate-tripleo-ci-centos-7-undercloud-containers/33b6607/console.html#_2017-08-14_18_26_49_229324 | 18:30 |
| EmilienM | we should probably make this job non voting | 18:30 |
| pabelanger | so, difficult to debug. Look at the output, I don't see any logs being collected | 18:30 |
| EmilienM | it's sad but we don't have much maintainers of the job | 18:31 |
| EmilienM | dprince and others, thoughts ^ | 18:31 |
| pabelanger | EmilienM: ya, I haven't been able to find who is responisble for that job over the last few weeks | 18:31 |
| *** sshnaidm is now known as sshnaidm|off | 18:35 | |
| weshay | pabelanger, dprince can help with that.. Emilien we need to switch that job over to oooq asap | 18:36 |
| EmilienM | and in the meantime non voting? | 18:37 |
| *** mmethot has joined #tripleo | 18:38 | |
| weshay | EmilienM, ya.. let me try to add it non-voting, I'm sure I'll do it incorrectly | 18:39 |
| EmilienM | I'm sure you will :P | 18:40 |
| pabelanger | I'd +1 non-voting | 18:40 |
| *** amoralej is now known as amoralej|off | 18:43 | |
| *** jmelvin has quit IRC | 18:44 | |
| *** mmethot_ has joined #tripleo | 18:44 | |
| *** jlabarre has quit IRC | 18:44 | |
| *** jlabarre has joined #tripleo | 18:45 | |
| *** mmethot has quit IRC | 18:48 | |
| trown | EmilienM: would you be able to look at https://review.openstack.org/487408 so we can close out 3nodes work | 18:49 |
| *** florianf has quit IRC | 18:49 | |
| EmilienM | trown: boom | 18:49 |
| trown | EmilienM: thanks! :) | 18:50 |
| EmilienM | trown: thx for this work, it will really help | 18:50 |
| *** jlabarre has quit IRC | 18:52 | |
| *** nyechiel has quit IRC | 18:54 | |
| *** jlabarre has joined #tripleo | 18:54 | |
| *** jtomasek has quit IRC | 18:58 | |
| *** milan has quit IRC | 18:59 | |
| pabelanger | DLRN breaking in the gate: http://logs.openstack.org/74/489874/7/gate/gate-tripleo-ci-centos-7-scenario003-multinode-oooq/853d7b5/logs/delorean_logs/14/40/14406f8f7ccd7c21b13f70f7a3a6039db3464ad8_dev/rpmbuild.log.txt.gz | 19:00 |
| pabelanger | failing to apply patch it seems | 19:00 |
| dprince | EmilienM: I can take this | 19:00 |
| *** pchavva has quit IRC | 19:02 | |
| toure | trown so the problem was with Selinux | 19:07 |
| *** leitan has joined #tripleo | 19:07 | |
| *** _milan_ has joined #tripleo | 19:08 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| *** jmelvin has joined #tripleo | 19:14 | |
| *** mmethot_ has quit IRC | 19:15 | |
| *** atoth has quit IRC | 19:15 | |
| *** leitan has quit IRC | 19:16 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 19:16 |
| *** karthiks has quit IRC | 19:17 | |
| *** _milan_ has quit IRC | 19:18 | |
| EmilienM | dprince: thx | 19:25 |
| *** karthiks has joined #tripleo | 19:26 | |
| *** nyechiel has joined #tripleo | 19:27 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 19:27 |
| *** salmankhan has joined #tripleo | 19:28 | |
| EmilienM | trown: should this thing work you think? https://review.openstack.org/#/c/493634/1/config/release/tripleo-ci/master.yml logs complain about the unbound variable http://logs.openstack.org/34/493634/1/check/gate-tripleo-ci-centos-7-containers-multinode/9df797f/logs/undercloud/home/jenkins/overcloud_prep_containers.log.txt.gz | 19:29 |
| trown | EmilienM: probably need an "environment" tag on the patch that passes through that var | 19:32 |
| trown | EmilienM: on the task rather | 19:32 |
| *** salmankhan has quit IRC | 19:32 | |
| *** nyechiel has quit IRC | 19:33 | |
| trown | EmilienM: on this task https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/tasks/overcloud-prep-containers.yml#L2 like https://github.com/openstack/tripleo-quickstart-extras/blob/61443d266b490d9f76588783e91209a45c844f20/roles/overcloud-deploy/tasks/deployed-server.yml#L16 | 19:33 |
| trown | EmilienM: the shell environment does not get passed to ansible tasks unless you explicitly pass it | 19:34 |
| trown | EmilienM: hmm I wonder if lookup function will work in the config... might be a shortcut | 19:35 |
| trown | EmilienM: commented on the review with something to try | 19:36 |
| toure | rlandy ping | 19:38 |
| rlandy | toure: hello | 19:38 |
| *** itlinux has quit IRC | 19:38 | |
| *** mmethot_ has joined #tripleo | 19:39 | |
| toure | rlandy I was wondering if you could help me with a quickstart issue in regards to a virtual bmc failure on the undercloud | 19:39 |
| rlandy | toure: you're deploying virt? | 19:39 |
| toure | yup | 19:39 |
| toure | http://paste.openstack.org/show/618336/ | 19:40 |
| rlandy | I can try - virt is not my usual platform of choice | 19:40 |
| toure | rlandy ^^ | 19:40 |
| toure | ah | 19:40 |
| *** jbadiapa has quit IRC | 19:40 | |
| rlandy | toure: did you make any changes or you are just deploying master of everything? | 19:41 |
| toure | nope | 19:41 |
| toure | just deploying master | 19:41 |
| rlandy | ok - lastly - your working dir | 19:41 |
| rlandy | did you clean it out from any previous runs? | 19:41 |
| toure | in regards to location? | 19:41 |
| toure | ooohh | 19:42 |
| toure | nope | 19:42 |
| rlandy | pls clean it out and try again | 19:42 |
| * toure cleans it out | 19:42 | |
| rlandy | let me know if you hit the same error | 19:42 |
| rlandy | it is possible to have cruft in that dir | 19:42 |
| toure | ah cool thanks I hope this is it | 19:42 |
| rlandy | default is ~/.quickstart | 19:42 |
| toure | yup | 19:43 |
| toure | I removed it and restarted | 19:43 |
| toure | and this will probably finish something later this evening | 19:44 |
| rlandy | ok - will be around for a while | 19:44 |
| toure | no need to wait for me... | 19:44 |
| *** agopi has joined #tripleo | 19:44 | |
| toure | thanks though | 19:44 |
| *** agopi has quit IRC | 19:45 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 19:53 |
| *** sbrzozow has quit IRC | 19:55 | |
| *** jprovazn has quit IRC | 20:01 | |
| *** liverpooler has quit IRC | 20:04 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 20:07 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| *** athomas has quit IRC | 20:10 | |
| *** d0ugal has joined #tripleo | 20:13 | |
| EmilienM | trown: in a call, will look shortly, thx | 20:15 |
| *** atoth has joined #tripleo | 20:16 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add settings for OVB in RDO Cloud https://review.openstack.org/480950 | 20:30 |
| *** ecerquei has quit IRC | 20:34 | |
| *** ecerquei has joined #tripleo | 20:34 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Use AFS mirrors to download containers instead of docker.io https://review.openstack.org/493634 | 20:36 |
| pabelanger | EmilienM: mwhahaha: so far, every gate job failure I have seen today is either undercloud-container or container-multinode job :( The good news is the failures are related to what we know already | 20:40 |
| EmilienM | pabelanger: yeah, so let's fix docker.io in my patch ^ and disable voting on undercloud-container | 20:40 |
| pabelanger | 489874 bombed because of packaging issue | 20:40 |
| pabelanger | EmilienM: k, will look at it shortly | 20:41 |
| EmilienM | dprince, weshay: I'm kicking off a patch right now, to disable voting on undercloud-container | 20:41 |
| *** ansmith has quit IRC | 20:41 | |
| *** ecerquei has quit IRC | 20:43 | |
| weshay | EmilienM, k | 20:44 |
| weshay | btw.. way when I say ok to you in irc.. it sounds like... mmmm k | 20:45 |
| EmilienM | weshay: ah, k | 20:45 |
| EmilienM | :) | 20:45 |
| dprince | EmilienM: ack, proceed | 20:46 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/461000 | 20:47 |
| *** gfidente|afk has quit IRC | 20:54 | |
| *** dmsimard|off has quit IRC | 21:03 | |
| *** dprince has quit IRC | 21:03 | |
| *** itlinux has joined #tripleo | 21:04 | |
| *** nyechiel has joined #tripleo | 21:04 | |
| *** lblanchard has quit IRC | 21:04 | |
| *** trown is now known as trown|outtypewww | 21:04 | |
| *** abishop has quit IRC | 21:04 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 21:04 |
| EmilienM | sshnaidm|off, weshay: do you know why we didn't have promotion last night? | 21:09 |
| EmilienM | periodic-tripleo-ci-centos-7-ovb-nonha-containers failed but I'm wondering if it's voting in the pipeline | 21:09 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 21:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| openstackgerrit | James Slagle proposed openstack/tripleo-docs master: Further document disable node updates https://review.openstack.org/493680 | 21:15 |
| *** atoth has quit IRC | 21:16 | |
| itlinux | hello all.. what's the best way to update the BM node when they have been already instacked? | 21:17 |
| itlinux | do I need to remove them and readd them? | 21:17 |
| *** bfournie has quit IRC | 21:18 | |
| EmilienM | michapma: what's the status now on ODL testing? | 21:21 |
| EmilienM | trozet, radez ^ | 21:22 |
| trozet | EmilienM: you mean in OOO CI? | 21:22 |
| EmilienM | trozet: https://review.openstack.org/#/c/486905/ | 21:23 |
| trozet | EmilienM: so that patch uses this one right? https://review.openstack.org/#/c/486731/16/config/general_config/featureset031.yml | 21:25 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Set file mode permission of Ceph keyrings https://review.openstack.org/492303 | 21:26 |
| EmilienM | trozet: yes | 21:27 |
| EmilienM | trozet: I was away last week but no progress :( | 21:27 |
| EmilienM | trozet: last time it was neutron-server not starting | 21:27 |
| EmilienM | any progress on that? | 21:27 |
| trozet | EmilienM: so if it is pulling in the environment file, why are hte variables manually defined on line 62-64 https://review.openstack.org/#/c/486905/16/ci/environments/scenario008-multinode.yaml | 21:28 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 21:28 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Convert scenario001-multinode-containers job to ceph-ansible https://review.openstack.org/479288 | 21:28 |
| EmilienM | trozet: I think we were trying to figure that last time I was working on that | 21:29 |
| EmilienM | but michapma was supposed to look at it, iirc | 21:29 |
| EmilienM | jrist: https://review.openstack.org/#/c/488526/ is one of the RC1 blockers, fyi, if we can make sure it's reviewed by tripleo-ui cores asap, thx | 21:30 |
| *** dmsimard|off has joined #tripleo | 21:30 | |
| jrist | +1 | 21:30 |
| jrist | it's had lots of eyes I think? | 21:30 |
| *** nyechiel has quit IRC | 21:31 | |
| pabelanger | EmilienM: mwhahaha: and gem / bundle client mirror configuration now: https://review.openstack.org/493681/ | 21:31 |
| *** agopi|away has quit IRC | 21:32 | |
| EmilienM | pabelanger: very cool | 21:32 |
| *** agopi|away has joined #tripleo | 21:32 | |
| EmilienM | jrist: yeah, good then. Just checking | 21:32 |
| trozet | EmilienM: I'll ask michapma next time i talk to him | 21:32 |
| EmilienM | jrist: we're cooking rc1 now :D | 21:32 |
| jrist | medium high? | 21:33 |
| EmilienM | trozet: he's working alone? | 21:33 |
| mwhahaha | pabelanger: ah i was wondering if there was a bundler config | 21:33 |
| EmilienM | jrist: high | 21:33 |
| jrist | :o | 21:33 |
| EmilienM | jrist: https://review.openstack.org/#/q/topic:bp/websocket-logging as well | 21:34 |
| EmilienM | https://blueprints.launchpad.net/tripleo/+spec/websocket-logging in FFE, but we need code reviewed & merged asap | 21:34 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-puppet-elements stable/ocata: Add remove-machine-id element https://review.openstack.org/493684 | 21:34 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common stable/ocata: Remove machine-id from image https://review.openstack.org/493685 | 21:35 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-puppet-elements stable/ocata: Add remove-machine-id element https://review.openstack.org/493684 | 21:37 |
| *** agopi|away has quit IRC | 21:39 | |
| weshay | EmilienM, we did get a tripleo promotion | 21:41 |
| weshay | rdo just promoted | 21:41 |
| EmilienM | weshay: how? https://dashboards.rdoproject.org/rdo-dev | 21:41 |
| EmilienM | but cool if that happened | 21:42 |
| weshay | EmilienM, re: master? | 21:44 |
| weshay | EmilienM, it just promoted a few hours ago | 21:44 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment https://review.openstack.org/486905 | 21:44 |
| weshay | rdo that is | 21:44 |
| weshay | this board does not track tripleo | 21:44 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment https://review.openstack.org/486905 | 21:44 |
| jrist | EmilienM: noted. no one left that can do +1 on tripleo-ui except tripleo cores | 21:45 |
| *** gbarros has quit IRC | 21:45 | |
| weshay | EmilienM, tripleo promotions are here: https://ci.centos.org/view/rdo/view/promotion-pipeline/job/tripleo-dlrn-promote/ | 21:45 |
| weshay | and I guess the last time it promoted was the 12th | 21:45 |
| weshay | so I'll go look at the tripleo jobs | 21:45 |
| bnemec | omg, the gate queue | 21:46 |
| EmilienM | weshay: good to know | 21:46 |
| weshay | EmilienM, hrm.. all the tripleo promotion jobs passed on the 14th | 21:47 |
| weshay | we *should* have a promotion, however maybe they picked up different hashes | 21:47 |
| EmilienM | weshay: yeah that's why i'm asking, what failed? | 21:47 |
| EmilienM | ahh this thing | 21:47 |
| weshay | EmilienM, aye | 21:47 |
| EmilienM | I discussed about that with jpena in a car (long story) | 21:48 |
| weshay | will be fixed in the rdo software factory version of these | 21:48 |
| EmilienM | and he thought about using delorean database to store an hash and re-use the same hash stored in this database from the jobs | 21:48 |
| weshay | yes.. that is how it will work | 21:48 |
| EmilienM | weshay: fixed how? | 21:48 |
| EmilienM | ok | 21:48 |
| weshay | ya.. we send a dlrn_api call, set a testing promotion.. so we have the same hash across jobs | 21:49 |
| EmilienM | ok so he was right! lol | 21:49 |
| weshay | then build images, containers... and run *ALL* the tests w/ that same hash | 21:49 |
| weshay | EmilienM, well.. I'm sure you made some good jokes | 21:49 |
| EmilienM | I didn't, I was driving | 21:49 |
| weshay | but that is *not* how it works today in the tripleo jobs, but we're very close to converting these over | 21:49 |
| weshay | EmilienM, yes.. don't joke and drive | 21:50 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Make containerized nova-api run with httpd https://review.openstack.org/475366 | 22:01 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Enable TLS for nova api and placement containers https://review.openstack.org/489885 | 22:02 |
| openstackgerrit | Merged openstack/tripleo-ui master: Notifications fixes https://review.openstack.org/487593 | 22:02 |
| openstackgerrit | Merged openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/493291 | 22:02 |
| *** jmelvin has quit IRC | 22:06 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| *** pradk has quit IRC | 22:10 | |
| mwhahaha | EmilienM: i think your package patch prevented https://review.openstack.org/#/c/489874/ from merging | 22:11 |
| *** itlinux has quit IRC | 22:13 | |
| apetrich | weshay, who is looking after ocata promotion. we have backported a fix for python-mistrallib it landed on the 11th but there's been no passing promotions for ocata since then | 22:14 |
| apetrich | weshay, sorry that was supposed to be a question. it is a tidbit late for me and not all neurons are working at this point :) | 22:14 |
| weshay | apetrich, ya.. ocata is red we've gone 4 days w/o a promotion https://dashboards.rdoproject.org/rdo-dev | 22:15 |
| weshay | apetrich, take a look at the job and see if you can help please | 22:15 |
| weshay | apetrich, https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-ocata-current-tripleo/ | 22:16 |
| apetrich | weshay, I saw it being here https://ci.centos.org/job/weirdo-ocata-promote-packstack-scenario001/ but I could not figure out what was wrong | 22:16 |
| weshay | apetrich, although I think the issue w/ that job was fixed earlier today | 22:16 |
| weshay | apetrich, I think the fix was here https://review.rdoproject.org/r/#/c/8526/ | 22:17 |
| apetrich | weshay, oh nice. I saw it was something packstack | 22:17 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Delete docker-centos-tripleoupstream.yaml https://review.openstack.org/487613 | 22:18 |
| *** itlinux has joined #tripleo | 22:27 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: container ovs-agent, add missing NeutronBridgeMappings https://review.openstack.org/493695 | 22:29 |
| EmilienM | pabelanger: http://logs.openstack.org/34/493634/2/check/gate-tripleo-ci-centos-7-containers-multinode/abaf224/logs/undercloud/home/jenkins/overcloud_prep_containers.log.txt.gz#_2017-08-14_21_26_51 | 22:39 |
| pabelanger | EmilienM: what is overcloud container image upload? | 22:41 |
| pabelanger | are you don't using docker client? | 22:41 |
| EmilienM | pabelanger: not afik, let me show you | 22:42 |
| pabelanger | EmilienM: is that pulling from upstream registry and pusing into another local registry? | 22:43 |
| EmilienM | pabelanger: we use that https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/container_image.py | 22:43 |
| EmilienM | stevebaker: hey, you around? | 22:43 |
| stevebaker | EmilienM: I am | 22:43 |
| stevebaker | pabelanger: yes, the upload command copies images between registries, currently between docker.io and the undercloud registry | 22:44 |
| pabelanger | stevebaker: why? | 22:44 |
| pabelanger | I mean, why do you have the undercloud registry just proxy to docker.io? | 22:45 |
| stevebaker | pabelanger: so that overcloud nodes pull from a local registry instead of DOSing a remote one | 22:45 |
| pabelanger | right, but if you setup local registry to proxy directly to docker.io, you shouldn't need to do pull / push your self. The registry would do it directly? | 22:46 |
| pabelanger | you setup the local registry to cache | 22:46 |
| pabelanger | https://docs.docker.com/registry/recipes/mirror/ | 22:47 |
| stevebaker | pabelanger: setting it up as a pull-through cache would be great if we could also proxy non docker.io registries | 22:49 |
| pabelanger | stevebaker: is there an issue using docker registry to cache non docker.io things? | 22:50 |
| pabelanger | FWIW: you can do this in apache also. it is how we manage http://mirror.regionone.infracloud-vanilla.openstack.org:8081/registry-1.docker It has zero docker things running | 22:51 |
| *** dhill_ has quit IRC | 22:51 | |
| *** dhill_ has joined #tripleo | 22:51 | |
| stevebaker | pabelanger: https://docs.docker.com/registry/recipes/mirror/ the Gotcha section says only docker.io can be mirrored | 22:51 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix metadata_settings in containerized mongodb https://review.openstack.org/492014 | 22:52 |
| pabelanger | stevebaker: right, that is why we did: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb#n184 | 22:52 |
| pabelanger | and didn't depend on docker tools | 22:52 |
| EmilienM | stevebaker, pabelanger: I have to go and I'll be back in a few hours - I'll catch-up with stevebaker . steve: please see the context on https://review.openstack.org/#/c/493634/ and help if you can. TTYL folks | 22:52 |
| pabelanger | EmilienM: stevebaker either way, lookin at the code, you need to drop HTTP in the pull_source and try again | 22:54 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario002 to run Tempest https://review.openstack.org/491102 | 22:54 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Switch scenario002 to run Tempest https://review.openstack.org/491102 | 22:54 |
| stevebaker | pabelanger: which pull_source code? | 22:54 |
| EmilienM | stevebaker: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/container_image.py#L280 maybe | 22:55 |
| EmilienM | so in kolla_builder probably | 22:55 |
| EmilienM | https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/kolla_builder.py | 22:56 |
| * EmilienM afk for real now | 22:56 | |
| stevebaker | pabelanger: btw running a full registry on the undercloud isn't only for CI, it looks like we need to support that fully for downstream customers | 22:57 |
| stevebaker | EmilienM, pabelanger: OK, I think I've finally caught up on what y'all are talking about. | 22:58 |
| pabelanger | stevebaker: yes, looking at the code you are doing it a little different then I expected. The way I tested this all was to setup the docker daemon to use a registry-mirrors in /etc/docker/daemon.json. In your case, it looks like you manually apass the registry url via docker CLI client. So in that cause, pull_source, needs to omit the http part of the url | 23:00 |
| pabelanger | stevebaker: the issue is, you shouldn't be directly hitting docker.io, because depending on the internet for testing is bad | 23:01 |
| pabelanger | and need to use our reverse proxy cache | 23:01 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Fix legacy nova/cinder encryption key manager configuration https://review.openstack.org/488509 | 23:01 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Add featureset023, tests up to overcloud images build https://review.openstack.org/487801 | 23:02 |
| pabelanger | should be 6 merges happening | 23:02 |
| pabelanger | first for today | 23:02 |
| *** rcernin has quit IRC | 23:02 | |
| stevebaker | pabelanger: so yeah, fully qualified image paths don't have protocol prefixes. There are about 3 places that the protocol could be stripped off, maybe it would best happen in https://review.openstack.org/#/c/493634 | 23:02 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Drop mongo from UC deploy https://review.openstack.org/480063 | 23:02 |
| openstackgerrit | Merged openstack/tripleo-common master: Add logrotate-crond container images params https://review.openstack.org/490068 | 23:02 |
| pabelanger | stevebaker: yes, agree | 23:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Convert cephstorage-role.yaml to role.role.j2.yaml https://review.openstack.org/482626 | 23:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Make HA container bundle work on remote nodes https://review.openstack.org/486076 | 23:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Enable TLS configuration for containerized Galera https://review.openstack.org/489963 | 23:03 |
| stevebaker | pabelanger: and yeah, once we can pull from NODEPOOL_DOCKER_REGISTRY_PROXY then in CI we can stop copying the images to the undercloud registry | 23:07 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 23:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710533 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1710533 in tripleo "docker client failed to download container from docker.io" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| stevebaker | pabelanger: hmm, then your next challange will be making sure all the tools treat mirror.ord.rax.openstack.org:8081 as an insecure (not https) registry | 23:12 |
| *** catintheroof has quit IRC | 23:12 | |
| pabelanger | stevebaker: when I tested daemon.json settings, no other settings were needed. What version of docker are you running | 23:13 |
| stevebaker | pabelanger: if you tested thats fine, I just tried to pull on my workstation to see what happebed | 23:13 |
| pabelanger | ya, you should be able to test it locally, we don't have any firewalls in place | 23:14 |
| pabelanger | which mirror are you using? I can look at apache logs | 23:14 |
| pabelanger | stevebaker: dprince also tested this back in Apr and thought he had setup jobs to use it | 23:15 |
| stevebaker | pabelanger: http://paste.openstack.org/show/618346/ | 23:15 |
| pabelanger | ya, we don't have https, just http | 23:15 |
| stevebaker | pabelanger: I've added the insecure registry, now I get this http://paste.openstack.org/show/618347/ | 23:19 |
| pabelanger | right, this isn't a registry, so I doubt it is going to work | 23:20 |
| pabelanger | let me look at logs | 23:20 |
| pabelanger | 1 sec | 23:20 |
| *** tosky has quit IRC | 23:20 | |
| *** limao has joined #tripleo | 23:22 | |
| pabelanger | 23.233.28.9 - - [14/Aug/2017:23:22:10 +0000] "GET /v2/registry-1.docker/tripleoupstream/centos-binary-aodh-api/manifests/latest HTTP/1.1" 403 457 "-" "docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87 kernel/4.11.10-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))" | 23:22 |
| pabelanger | it is adding /v2 | 23:22 |
| pabelanger | which is wrong | 23:22 |
| pabelanger | well, we expect /registry-1.docker/v2 | 23:22 |
| pabelanger | myregistry.local:5000 | 23:24 |
| pabelanger | docs say that is what it only support | 23:24 |
| pabelanger | so it thinks the image is registry-1.docker/tripleoupstream/centos-binary-aodh-api/manifests/latest | 23:25 |
| pabelanger | sigh | 23:25 |
| *** itlinux has quit IRC | 23:26 | |
| *** itlinux has joined #tripleo | 23:28 | |
| stevebaker | pabelanger: I tried setting this in my /etc/sysconfig/docker: HTTP_PROXY='http://mirror.ord.rax.openstack.org:8081/registry-1.docker/' | 23:31 |
| pabelanger | no, that won't work | 23:31 |
| pabelanger | what client are you using? | 23:31 |
| stevebaker | it didn't | 23:31 |
| stevebaker | plain docker pull | 23:31 |
| stevebaker | Get https://registry-1.docker.io/v2/: Forbidden | 23:31 |
| pabelanger | docker --version | 23:31 |
| stevebaker | fedora 26, Docker version 1.13.1 | 23:32 |
| *** karthiks has quit IRC | 23:32 | |
| *** karthiks has joined #tripleo | 23:32 | |
| pabelanger | stevebaker: try the follow: http://git.openstack.org/cgit/openstack-infra/system-config/commit/?id=836cddb2f2242872822a5429f465cb6965c0df69 | 23:33 |
| pabelanger | that _should_ work | 23:33 |
| pabelanger | I have 17.x, which broke something | 23:33 |
| pabelanger | but I know it worked on 1.13.x | 23:33 |
| pabelanger | looking to downgrade now | 23:33 |
| *** limao has quit IRC | 23:33 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] LVM support for dib-block-device https://review.openstack.org/472065 | 23:36 |
| pabelanger | stevebaker: ya, docker 1.12 it works | 23:37 |
| pabelanger | so, if you are using anything but 17.x, we need to setup /etc/docker/daemon.json and use our mirror registry | 23:38 |
| pabelanger | and not use docker pull docker.io/foobar | 23:38 |
| pabelanger | 23.233.28.9 - - [14/Aug/2017:23:37:49 +0000] "GET | 23:39 |
| pabelanger | /cloudfront/registry-v2/docker/registry/v2/blobs/sha256/b4/b4536da7aa245a864871de8a59a4cd6bda3781055ef0cab36b169cbee6bb8b0a/data?Expires=1502755069&Signature=PTvo10n6IJr9jHzOQRONgzMI~my3W4R1JHZ0L39fQrc0SxrJvzl68OPWU37O63ApRP9Ys577iP6Y45e~RnOIfOnWK3dkl8KhG3gMl92lYhGuy9iUOyrdMamh7hd-P5-jN8mpDH72NEn6vrw70-PWp9FS5hz52qKKBYDwkzooRNE_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q HTTP/1.1" 200 23539 | 23:39 |
| pabelanger | "http://mirror.ord.rax.openstack.org:8081/registry-1.docker/v2/tripleoupstream/centos-binary-aodh-api/blobs/sha256:b4536da7aa245a864871de8a59a4cd6bda3781055ef0cab36b169cbee6bb8b0a" "docker/1.12.6 go/go1.7.6 kernel/4.11.10-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" | 23:39 |
| pabelanger | is an example of HTTP request you'll see if you are running tcpdump or wireshark | 23:39 |
| *** dhill_ has quit IRC | 23:51 | |
| *** dtrainor has quit IRC | 23:51 | |
| *** yamahata has quit IRC | 23:51 | |
| *** dsavineau has quit IRC | 23:51 | |
| *** rlandy has quit IRC | 23:51 | |
| *** amoralej|off has quit IRC | 23:51 | |
| *** dalvarez has quit IRC | 23:51 | |
| *** bkopilov has quit IRC | 23:51 | |
| *** artom has quit IRC | 23:51 | |
| *** panda has quit IRC | 23:51 | |
| *** mandre has quit IRC | 23:51 | |
| *** vkhanna has quit IRC | 23:51 | |
| *** morazi has quit IRC | 23:51 | |
| *** kbyrne has quit IRC | 23:51 | |
| *** markmc has quit IRC | 23:51 | |
| *** mjblack has quit IRC | 23:51 | |
| *** bswartz has quit IRC | 23:51 | |
| *** leseb has quit IRC | 23:51 | |
| *** michapma has quit IRC | 23:51 | |
| *** tvignaud has quit IRC | 23:51 | |
| *** stevebaker has quit IRC | 23:51 | |
| *** toure has quit IRC | 23:51 | |
| *** assassin has quit IRC | 23:51 | |
| *** d0ugal has quit IRC | 23:51 | |
| *** dsneddon has quit IRC | 23:51 | |
| *** chem has quit IRC | 23:51 | |
| *** spredzy has quit IRC | 23:51 | |
| *** dr_gogeta86 has quit IRC | 23:51 | |
| *** Tyrantelf has quit IRC | 23:51 | |
| *** zoli has quit IRC | 23:51 | |
| *** mmethot_ has quit IRC | 23:51 | |
| *** sshnaidm|off has quit IRC | 23:51 | |
| *** flaper87 has quit IRC | 23:51 | |
| *** bandini has quit IRC | 23:51 | |
| *** radez has quit IRC | 23:51 | |
| *** eck` has quit IRC | 23:51 | |
| *** colonwq has quit IRC | 23:51 | |
| *** isq_ has quit IRC | 23:51 | |
| *** bcafarel has quit IRC | 23:51 | |
| *** fpan has quit IRC | 23:51 | |
| *** rook has quit IRC | 23:51 | |
| *** timothyb89 has quit IRC | 23:51 | |
| *** EmilienM has quit IRC | 23:51 | |
| *** dmsimard|off has quit IRC | 23:51 | |
| *** jkilpatr has quit IRC | 23:51 | |
| *** openstackgerrit has quit IRC | 23:51 | |
| *** zzzeek has quit IRC | 23:51 | |
| *** zshi has quit IRC | 23:51 | |
| *** saneax_-_ has quit IRC | 23:51 | |
| *** yolanda has quit IRC | 23:51 | |
| *** myoung has quit IRC | 23:51 | |
| *** jschlueter has quit IRC | 23:51 | |
| *** numans has quit IRC | 23:51 | |
| *** arxcruz has quit IRC | 23:51 | |
| *** saneax has quit IRC | 23:51 | |
| *** rbowen has quit IRC | 23:51 | |
| *** slagle has quit IRC | 23:51 | |
| *** ipsecguy_ has quit IRC | 23:51 | |
| *** dobson has quit IRC | 23:51 | |
| *** oanson has quit IRC | 23:51 | |
| *** hamzy has quit IRC | 23:51 | |
| *** hewbrocca_afk has quit IRC | 23:51 | |
| *** tristanC has quit IRC | 23:51 | |
| *** ansiwen has quit IRC | 23:51 | |
| *** tonyb has quit IRC | 23:51 | |
| *** lyarwood has quit IRC | 23:51 | |
| *** mhenkel has quit IRC | 23:51 | |
| *** apetrich has quit IRC | 23:51 | |
| *** jistr|off has quit IRC | 23:51 | |
| *** lifeless_ has quit IRC | 23:51 | |
| *** sdoran has quit IRC | 23:51 | |
| *** lhinds has quit IRC | 23:51 | |
| *** ggillies has quit IRC | 23:51 | |
| *** lucas-afk has quit IRC | 23:51 | |
| *** sai has quit IRC | 23:51 | |
| *** Lokesh_Jain__ has quit IRC | 23:51 | |
| *** sseago has quit IRC | 23:51 | |
| *** greghaynes has quit IRC | 23:51 | |
| *** dtantsur|afk has quit IRC | 23:51 | |
| *** hexo_ has quit IRC | 23:51 | |
| *** StevenK has quit IRC | 23:51 | |
| *** akrzos has quit IRC | 23:51 | |
| *** zul has quit IRC | 23:51 | |
| *** percevalbot` has quit IRC | 23:51 | |
| *** Hazelesque has quit IRC | 23:51 | |
| *** chlong_ has quit IRC | 23:51 | |
| *** stee_3_ has quit IRC | 23:51 | |
| *** ianw has quit IRC | 23:51 | |
| *** remix_tj has quit IRC | 23:51 | |
| *** weshay has quit IRC | 23:51 | |
| *** gchamoul has quit IRC | 23:51 | |
| *** larsks has quit IRC | 23:51 | |
| *** migi has quit IRC | 23:51 | |
| *** trown|outtypewww has quit IRC | 23:51 | |
| *** karthiks has quit IRC | 23:51 | |
| *** itlinux has quit IRC | 23:51 | |
| *** jlabarre has quit IRC | 23:51 | |
| *** jcoufal has quit IRC | 23:51 | |
| *** patrickeast has quit IRC | 23:51 | |
| *** mgkwill has quit IRC | 23:51 | |
| *** vkmc has quit IRC | 23:51 | |
| *** hrybacki has quit IRC | 23:51 | |
| *** v1k0d3n has quit IRC | 23:51 | |
| *** mgagne has quit IRC | 23:51 | |
| *** fungi has quit IRC | 23:51 | |
| *** leifmadsen has quit IRC | 23:51 | |
| *** fabbione has quit IRC | 23:51 | |
| *** CaptTofu has quit IRC | 23:51 | |
| *** NobodyCam has quit IRC | 23:51 | |
| *** rbrady has quit IRC | 23:51 | |
| *** rwsu has quit IRC | 23:51 | |
| *** social has quit IRC | 23:51 | |
| *** SlickNik has quit IRC | 23:51 | |
| *** lvdombrkr has quit IRC | 23:51 | |
| *** portdirect has quit IRC | 23:51 | |
| *** gregwork has quit IRC | 23:51 | |
| *** rodrigods has quit IRC | 23:51 | |
| *** vpickard has quit IRC | 23:51 | |
| *** sdake has quit IRC | 23:51 | |
| *** mrunge has quit IRC | 23:51 | |
| *** jwb has quit IRC | 23:51 | |
| *** zaneb has quit IRC | 23:51 | |
| *** andreaf has quit IRC | 23:51 | |
| *** number80 has quit IRC | 23:51 | |
| *** alanmeadows has quit IRC | 23:51 | |
| *** funzo has quit IRC | 23:51 | |
| *** jaosorior has quit IRC | 23:51 | |
| *** bugzy has quit IRC | 23:51 | |
| *** cinerama` has quit IRC | 23:51 | |
| *** pabelanger has quit IRC | 23:51 | |
| *** soc_off_ has quit IRC | 23:51 | |
| *** jidar has quit IRC | 23:51 | |
| *** mburned_out has quit IRC | 23:51 | |
| *** beagles has quit IRC | 23:51 | |
| *** pliu has quit IRC | 23:51 | |
| *** faceman has quit IRC | 23:51 | |
| *** adarazs has quit IRC | 23:51 | |
| *** rickflare has quit IRC | 23:51 | |
| *** Slower has quit IRC | 23:51 | |
| *** honza has quit IRC | 23:51 | |
| *** ChanServ has quit IRC | 23:51 | |
| *** jrist has quit IRC | 23:51 | |
| *** mwhahaha has quit IRC | 23:51 | |
| *** japestinho has quit IRC | 23:51 | |
| *** shadower has quit IRC | 23:51 | |
| *** tdasilva has quit IRC | 23:51 | |
| *** tbarron has quit IRC | 23:51 | |
| *** melwitt has quit IRC | 23:51 | |
| *** thrash|g0ne has quit IRC | 23:51 | |
| *** rajinir has quit IRC | 23:51 | |
| *** Ng has quit IRC | 23:51 | |
| *** tzumainn has quit IRC | 23:51 | |
| *** dbecker has quit IRC | 23:51 | |
| *** paramite has quit IRC | 23:51 | |
| *** owalsh has quit IRC | 23:51 | |
| *** cmyster has quit IRC | 23:51 | |
| *** therve has quit IRC | 23:51 | |
| *** afazekas has quit IRC | 23:51 | |
| *** rmcadams has quit IRC | 23:51 | |
| *** rmcadams has joined #tripleo | 23:55 | |
| *** owalsh has joined #tripleo | 23:55 | |
| *** jkilpatr has joined #tripleo | 23:55 | |
| *** cmyster has joined #tripleo | 23:55 | |
| *** tzumainn has joined #tripleo | 23:55 | |
| *** therve has joined #tripleo | 23:55 | |
| *** afazekas has joined #tripleo | 23:55 | |
| *** bfournie has joined #tripleo | 23:55 | |
| *** karthiks has joined #tripleo | 23:55 | |
| *** itlinux has joined #tripleo | 23:55 | |
| *** dhill_ has joined #tripleo | 23:55 | |
| *** d0ugal has joined #tripleo | 23:55 | |
| *** mmethot_ has joined #tripleo | 23:55 | |
| *** morazi has joined #tripleo | 23:55 | |
| *** sshnaidm|off has joined #tripleo | 23:55 | |
| *** dtrainor has joined #tripleo | 23:55 | |
| *** yamahata has joined #tripleo | 23:55 | |
| *** dsneddon has joined #tripleo | 23:55 | |
| *** rbowen has joined #tripleo | 23:55 | |
| *** chlong_ has joined #tripleo | 23:55 | |
| *** chem has joined #tripleo | 23:55 | |
| *** jcoufal has joined #tripleo | 23:55 | |
| *** dsavineau has joined #tripleo | 23:55 | |
| *** slagle has joined #tripleo | 23:55 | |
| *** rlandy has joined #tripleo | 23:55 | |
| *** rbrady has joined #tripleo | 23:55 | |
| *** amoralej|off has joined #tripleo | 23:55 | |
| *** dalvarez has joined #tripleo | 23:55 | |
| *** flaper87 has joined #tripleo | 23:55 | |
| *** rwsu has joined #tripleo | 23:55 | |
| *** bkopilov has joined #tripleo | 23:55 | |
| *** stee_3_ has joined #tripleo | 23:55 | |
| *** artom has joined #tripleo | 23:55 | |
| *** panda has joined #tripleo | 23:55 | |
| *** jrist has joined #tripleo | 23:55 | |
| *** mandre has joined #tripleo | 23:55 | |
| *** ipsecguy_ has joined #tripleo | 23:55 | |
| *** spredzy has joined #tripleo | 23:55 | |
| *** social has joined #tripleo | 23:55 | |
| *** dr_gogeta86 has joined #tripleo | 23:55 | |
| *** SlickNik has joined #tripleo | 23:55 | |
| *** lvdombrkr has joined #tripleo | 23:55 | |
| *** kbyrne has joined #tripleo | 23:55 | |
| *** dobson has joined #tripleo | 23:55 | |
| *** vkhanna has joined #tripleo | 23:55 | |
| *** apetrich has joined #tripleo | 23:55 | |
| *** bswartz has joined #tripleo | 23:55 | |
| *** oanson has joined #tripleo | 23:55 | |
| *** mwhahaha has joined #tripleo | 23:55 | |
| *** hamzy has joined #tripleo | 23:55 | |
| *** jistr|off has joined #tripleo | 23:55 | |
| *** lifeless_ has joined #tripleo | 23:55 | |
| *** portdirect has joined #tripleo | 23:55 | |
| *** hewbrocca_afk has joined #tripleo | 23:55 | |
| *** markmc has joined #tripleo | 23:55 | |
| *** mjblack has joined #tripleo | 23:55 | |
| *** sdoran has joined #tripleo | 23:55 | |
| *** gregwork has joined #tripleo | 23:55 | |
| *** lhinds has joined #tripleo | 23:55 | |
| *** ianw has joined #tripleo | 23:55 | |
| *** leseb has joined #tripleo | 23:55 | |
| *** michapma has joined #tripleo | 23:55 | |
| *** ggillies has joined #tripleo | 23:55 | |
| *** tvignaud has joined #tripleo | 23:55 | |
| *** colonwq has joined #tripleo | 23:55 | |
| *** stevebaker has joined #tripleo | 23:55 | |
| *** rodrigods has joined #tripleo | 23:55 | |
| *** vpickard has joined #tripleo | 23:55 | |
| *** remix_tj has joined #tripleo | 23:55 | |
| *** weshay has joined #tripleo | 23:55 | |
| *** sdake has joined #tripleo | 23:55 | |
| *** isq_ has joined #tripleo | 23:55 | |
| *** mrunge has joined #tripleo | 23:55 | |
| *** japestinho has joined #tripleo | 23:55 | |
| *** jwb has joined #tripleo | 23:55 | |
| *** bcafarel has joined #tripleo | 23:55 | |
| *** Tyrantelf has joined #tripleo | 23:55 | |
| *** number80 has joined #tripleo | 23:55 | |
| *** fpan has joined #tripleo | 23:55 | |
| *** toure has joined #tripleo | 23:55 | |
| *** rook has joined #tripleo | 23:55 | |
| *** lucas-afk has joined #tripleo | 23:55 | |
| *** sai has joined #tripleo | 23:55 | |
| *** tristanC has joined #tripleo | 23:55 | |
| *** shadower has joined #tripleo | 23:55 | |
| *** Lokesh_Jain__ has joined #tripleo | 23:55 | |
| *** zaneb has joined #tripleo | 23:55 | |
| *** andreaf has joined #tripleo | 23:55 | |
| *** sseago has joined #tripleo | 23:55 | |
| *** greghaynes has joined #tripleo | 23:55 | |
| *** ansiwen has joined #tripleo | 23:55 | |
| *** assassin has joined #tripleo | 23:55 | |
| *** tdasilva has joined #tripleo | 23:55 | |
| *** tepper.freenode.net sets mode: +o mwhahaha | 23:55 | |
| *** patrickeast has joined #tripleo | 23:55 | |
| *** mgkwill has joined #tripleo | 23:55 | |
| *** tbarron has joined #tripleo | 23:55 | |
| *** dtantsur|afk has joined #tripleo | 23:55 | |
| *** vkmc has joined #tripleo | 23:55 | |
| *** melwitt has joined #tripleo | 23:55 | |
| *** hrybacki has joined #tripleo | 23:55 | |
| *** v1k0d3n has joined #tripleo | 23:55 | |
| *** tonyb has joined #tripleo | 23:55 | |
| *** zoli has joined #tripleo | 23:55 | |
| *** gchamoul has joined #tripleo | 23:55 | |
| *** lyarwood has joined #tripleo | 23:55 | |
| *** mhenkel has joined #tripleo | 23:55 | |
| *** timothyb89 has joined #tripleo | 23:55 | |
| *** thrash|g0ne has joined #tripleo | 23:55 | |
| *** hexo_ has joined #tripleo | 23:55 | |
| *** larsks has joined #tripleo | 23:55 | |
| *** mgagne has joined #tripleo | 23:55 | |
| *** StevenK has joined #tripleo | 23:55 | |
| *** EmilienM has joined #tripleo | 23:55 | |
| *** migi has joined #tripleo | 23:55 | |
| *** akrzos has joined #tripleo | 23:55 | |
| *** fungi has joined #tripleo | 23:55 | |
| *** rajinir has joined #tripleo | 23:55 | |
| *** leifmadsen has joined #tripleo | 23:55 | |
| *** trown|outtypewww has joined #tripleo | 23:55 | |
| *** zul has joined #tripleo | 23:55 | |
| *** percevalbot` has joined #tripleo | 23:55 | |
| *** fabbione has joined #tripleo | 23:55 | |
| *** Hazelesque has joined #tripleo | 23:55 | |
| *** CaptTofu has joined #tripleo | 23:55 | |
| *** Ng has joined #tripleo | 23:55 | |
| *** alanmeadows has joined #tripleo | 23:55 | |
| *** NobodyCam has joined #tripleo | 23:55 | |
| *** tepper.freenode.net sets mode: +oov EmilienM trown|outtypewww Ng | 23:55 | |
| *** rook has quit IRC | 23:55 | |
| *** ChanServ has joined #tripleo | 23:56 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 23:56 | |
| *** dmsimard|off has joined #tripleo | 23:56 | |
| *** openstackgerrit has joined #tripleo | 23:56 | |
| *** zzzeek has joined #tripleo | 23:56 | |
| *** zshi has joined #tripleo | 23:56 | |
| *** saneax_-_ has joined #tripleo | 23:56 | |
| *** yolanda has joined #tripleo | 23:56 | |
| *** myoung has joined #tripleo | 23:56 | |
| *** jschlueter has joined #tripleo | 23:56 | |
| *** numans has joined #tripleo | 23:56 | |
| *** arxcruz has joined #tripleo | 23:56 | |
| *** saneax has joined #tripleo | 23:56 | |
| *** trown|outtypewww has quit IRC | 23:56 | |
| *** bandini has joined #tripleo | 23:56 | |
| *** funzo has joined #tripleo | 23:56 | |
| *** jaosorior has joined #tripleo | 23:56 | |
| *** bugzy has joined #tripleo | 23:56 | |
| *** cinerama` has joined #tripleo | 23:56 | |
| *** pabelanger has joined #tripleo | 23:56 | |
| *** soc_off_ has joined #tripleo | 23:56 | |
| *** jidar has joined #tripleo | 23:56 | |
| *** mburned_out has joined #tripleo | 23:56 | |
| *** beagles has joined #tripleo | 23:56 | |
| *** pliu has joined #tripleo | 23:56 | |
| *** faceman has joined #tripleo | 23:56 | |
| *** adarazs has joined #tripleo | 23:56 | |
| *** rickflare has joined #tripleo | 23:56 | |
| *** Slower has joined #tripleo | 23:56 | |
| *** honza has joined #tripleo | 23:56 | |
| *** ChanServ has quit IRC | 23:56 | |
| *** chlong_ has quit IRC | 23:57 | |
| *** stee_3_ has quit IRC | 23:57 | |
| *** ianw has quit IRC | 23:57 | |
| *** remix_tj has quit IRC | 23:57 | |
| *** weshay has quit IRC | 23:57 | |
| *** gchamoul has quit IRC | 23:57 | |
| *** larsks has quit IRC | 23:57 | |
| *** migi has quit IRC | 23:57 | |
| *** funzo has quit IRC | 23:57 | |
| *** jaosorior has quit IRC | 23:57 | |
| *** bugzy has quit IRC | 23:57 | |
| *** cinerama` has quit IRC | 23:57 | |
| *** pabelanger has quit IRC | 23:57 | |
| *** soc_off_ has quit IRC | 23:57 | |
| *** jidar has quit IRC | 23:57 | |
| *** mburned_out has quit IRC | 23:57 | |
| *** beagles has quit IRC | 23:57 | |
| *** pliu has quit IRC | 23:57 | |
| *** faceman has quit IRC | 23:57 | |
| *** adarazs has quit IRC | 23:57 | |
| *** rickflare has quit IRC | 23:57 | |
| *** Slower has quit IRC | 23:57 | |
| *** honza has quit IRC | 23:57 | |
| *** dmsimard|off has quit IRC | 23:57 | |
| *** openstackgerrit has quit IRC | 23:57 | |
| *** zzzeek has quit IRC | 23:57 | |
| *** zshi has quit IRC | 23:57 | |
| *** saneax_-_ has quit IRC | 23:57 | |
| *** yolanda has quit IRC | 23:57 | |
| *** myoung has quit IRC | 23:57 | |
| *** jschlueter has quit IRC | 23:57 | |
| *** numans has quit IRC | 23:57 | |
| *** arxcruz has quit IRC | 23:57 | |
| *** saneax has quit IRC | 23:57 | |
| *** owalsh has quit IRC | 23:57 | |
| *** cmyster has quit IRC | 23:57 | |
| *** jkilpatr has quit IRC | 23:57 | |
| *** bfournie has quit IRC | 23:57 | |
| *** dhill_ has quit IRC | 23:57 | |
| *** dtrainor has quit IRC | 23:57 | |
| *** yamahata has quit IRC | 23:57 | |
| *** dsavineau has quit IRC | 23:57 | |
| *** rlandy has quit IRC | 23:57 | |
| *** amoralej|off has quit IRC | 23:57 | |
| *** dalvarez has quit IRC | 23:57 | |
| *** bkopilov has quit IRC | 23:57 | |
| *** artom has quit IRC | 23:57 | |
| *** panda has quit IRC | 23:57 | |
| *** mandre has quit IRC | 23:57 | |
| *** vkhanna has quit IRC | 23:57 | |
| *** morazi has quit IRC | 23:57 | |
| *** kbyrne has quit IRC | 23:57 | |
| *** markmc has quit IRC | 23:57 | |
| *** mjblack has quit IRC | 23:57 | |
| *** bswartz has quit IRC | 23:57 | |
| *** leseb has quit IRC | 23:57 | |
| *** michapma has quit IRC | 23:57 | |
| *** tvignaud has quit IRC | 23:57 | |
| *** stevebaker has quit IRC | 23:57 | |
| *** toure has quit IRC | 23:57 | |
| *** assassin has quit IRC | 23:57 | |
| *** tzumainn has quit IRC | 23:57 | |
| *** d0ugal has quit IRC | 23:57 | |
| *** dsneddon has quit IRC | 23:57 | |
| *** chem has quit IRC | 23:57 | |
| *** spredzy has quit IRC | 23:57 | |
| *** dr_gogeta86 has quit IRC | 23:57 | |
| *** Tyrantelf has quit IRC | 23:57 | |
| *** zoli has quit IRC | 23:57 | |
| *** bandini has quit IRC | 23:57 | |
| *** mmethot_ has quit IRC | 23:57 | |
| *** sshnaidm|off has quit IRC | 23:57 | |
| *** flaper87 has quit IRC | 23:57 | |
| *** colonwq has quit IRC | 23:57 | |
| *** isq_ has quit IRC | 23:57 | |
| *** bcafarel has quit IRC | 23:57 | |
| *** fpan has quit IRC | 23:57 | |
| *** timothyb89 has quit IRC | 23:57 | |
| *** EmilienM has quit IRC | 23:57 | |
| *** rbowen has quit IRC | 23:57 | |
| *** slagle has quit IRC | 23:57 | |
| *** ipsecguy_ has quit IRC | 23:57 | |
| *** dobson has quit IRC | 23:57 | |
| *** oanson has quit IRC | 23:57 | |
| *** hamzy has quit IRC | 23:57 | |
| *** hewbrocca_afk has quit IRC | 23:57 | |
| *** tristanC has quit IRC | 23:57 | |
| *** ansiwen has quit IRC | 23:57 | |
| *** tonyb has quit IRC | 23:57 | |
| *** lyarwood has quit IRC | 23:57 | |
| *** mhenkel has quit IRC | 23:57 | |
| *** rmcadams has quit IRC | 23:57 | |
| *** afazekas has quit IRC | 23:57 | |
| *** therve has quit IRC | 23:57 | |
| *** apetrich has quit IRC | 23:57 | |
| *** jistr|off has quit IRC | 23:57 | |
| *** lifeless_ has quit IRC | 23:57 | |
| *** sdoran has quit IRC | 23:57 | |
| *** lhinds has quit IRC | 23:57 | |
| *** ggillies has quit IRC | 23:57 | |
| *** lucas-afk has quit IRC | 23:57 | |
| *** sai has quit IRC | 23:57 | |
| *** Lokesh_Jain__ has quit IRC | 23:57 | |
| *** sseago has quit IRC | 23:57 | |
| *** greghaynes has quit IRC | 23:57 | |
| *** dtantsur|afk has quit IRC | 23:57 | |
| *** hexo_ has quit IRC | 23:57 | |
| *** StevenK has quit IRC | 23:57 | |
| *** akrzos has quit IRC | 23:57 | |
| *** zul has quit IRC | 23:57 | |
| *** percevalbot` has quit IRC | 23:57 | |
| *** Hazelesque has quit IRC | 23:57 | |
| *** karthiks has quit IRC | 23:57 | |
| *** itlinux has quit IRC | 23:57 | |
| *** jcoufal has quit IRC | 23:57 | |
| *** patrickeast has quit IRC | 23:57 | |
| *** mgkwill has quit IRC | 23:57 | |
| *** vkmc has quit IRC | 23:57 | |
| *** hrybacki has quit IRC | 23:57 | |
| *** v1k0d3n has quit IRC | 23:57 | |
| *** mgagne has quit IRC | 23:57 | |
| *** fungi has quit IRC | 23:57 | |
| *** leifmadsen has quit IRC | 23:57 | |
| *** fabbione has quit IRC | 23:57 | |
| *** CaptTofu has quit IRC | 23:57 | |
| *** NobodyCam has quit IRC | 23:57 | |
| *** rbrady has quit IRC | 23:57 | |
| *** rwsu has quit IRC | 23:57 | |
| *** social has quit IRC | 23:57 | |
| *** SlickNik has quit IRC | 23:57 | |
| *** lvdombrkr has quit IRC | 23:57 | |
| *** portdirect has quit IRC | 23:57 | |
| *** gregwork has quit IRC | 23:57 | |
| *** rodrigods has quit IRC | 23:57 | |
| *** vpickard has quit IRC | 23:57 | |
| *** sdake has quit IRC | 23:57 | |
| *** mrunge has quit IRC | 23:57 | |
| *** jwb has quit IRC | 23:57 | |
| *** zaneb has quit IRC | 23:57 | |
| *** andreaf has quit IRC | 23:57 | |
| *** number80 has quit IRC | 23:57 | |
| *** alanmeadows has quit IRC | 23:57 | |
| *** jrist has quit IRC | 23:57 | |
| *** mwhahaha has quit IRC | 23:57 | |
| *** japestinho has quit IRC | 23:57 | |
| *** shadower has quit IRC | 23:57 | |
| *** tdasilva has quit IRC | 23:57 | |
| *** tbarron has quit IRC | 23:57 | |
| *** melwitt has quit IRC | 23:57 | |
| *** thrash|g0ne has quit IRC | 23:57 | |
| *** rajinir has quit IRC | 23:57 | |
| *** Ng has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!