| *** ecerquei__ has quit IRC | 00:05 | |
| *** ecerquei__ has joined #tripleo | 00:05 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| *** bobh has joined #tripleo | 00:32 | |
| *** psahoo has joined #tripleo | 00:42 | |
| *** lblanchard has joined #tripleo | 00:43 | |
| *** tongl has joined #tripleo | 00:50 | |
| *** achadha has joined #tripleo | 01:00 | |
| *** achadha has quit IRC | 01:04 | |
| *** achadha has joined #tripleo | 01:04 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 01:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| *** ecerquei__ has quit IRC | 01:18 | |
| *** psachin has joined #tripleo | 01:19 | |
| *** dmacpher has joined #tripleo | 01:33 | |
| *** archit has joined #tripleo | 01:36 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 02:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| *** tongl has quit IRC | 02:20 | |
| *** Shatadru has joined #tripleo | 02:21 | |
| *** archit has quit IRC | 02:24 | |
| *** atoth has quit IRC | 02:32 | |
| *** Shatadru is now known as Shatadru|coffee| | 02:33 | |
| *** leitan has quit IRC | 02:38 | |
| *** pmannidi has quit IRC | 02:53 | |
| *** pmannidi has joined #tripleo | 02:56 | |
| *** Shatadru|coffee| is now known as Shatadru | 03:05 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 03:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| *** dpawar has joined #tripleo | 03:12 | |
| *** daidv has joined #tripleo | 03:14 | |
| *** gvrangan_ has joined #tripleo | 03:15 | |
| *** lblanchard has quit IRC | 03:17 | |
| *** bobh has quit IRC | 03:33 | |
| *** gkadam has joined #tripleo | 03:35 | |
| *** gbarros has joined #tripleo | 03:35 | |
| *** bobh has joined #tripleo | 03:37 | |
| *** mdnadeem has joined #tripleo | 03:37 | |
| *** gvrangan_ has quit IRC | 03:39 | |
| *** udesale has joined #tripleo | 03:40 | |
| *** gvrangan has joined #tripleo | 03:41 | |
| *** bobh has quit IRC | 03:47 | |
| *** yamahata has joined #tripleo | 03:50 | |
| *** links has joined #tripleo | 03:50 | |
| *** gvrangan has quit IRC | 03:52 | |
| *** gvrangan has joined #tripleo | 03:53 | |
| *** gvrangan has quit IRC | 03:58 | |
| *** ramishra has joined #tripleo | 03:58 | |
| *** Shatadru is now known as Shatadru|afk | 04:04 | |
| *** dpawar has quit IRC | 04:05 | |
| *** jaosorior has joined #tripleo | 04:07 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Use cacertfile option to get CA certificate https://review.openstack.org/507515 | 04:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Don't try to update cloud-init anymore https://review.openstack.org/504850 | 04:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Add unit tests for fluentd profile https://review.openstack.org/505124 | 04:24 |
| *** etingof has quit IRC | 04:27 | |
| *** etingof has joined #tripleo | 04:28 | |
| *** ratailor has joined #tripleo | 04:32 | |
| *** ratailor_ has joined #tripleo | 04:33 | |
| *** Shatadru|afk is now known as Shatadru | 04:34 | |
| *** ratailor_ has quit IRC | 04:35 | |
| *** gbarros has quit IRC | 04:36 | |
| *** ratailor_ has joined #tripleo | 04:37 | |
| *** ratailor__ has joined #tripleo | 04:39 | |
| *** ratailor has quit IRC | 04:40 | |
| *** ykarel_ has joined #tripleo | 04:42 | |
| *** ratailor_ has quit IRC | 04:42 | |
| *** ykarel_ is now known as ykarel | 04:42 | |
| *** stendulker has joined #tripleo | 04:43 | |
| *** bobh has joined #tripleo | 04:48 | |
| *** ratailor_ has joined #tripleo | 04:48 | |
| *** Dinesh_Bhor has joined #tripleo | 04:48 | |
| EmilienM | jaosorior: please review https://review.openstack.org/#/c/510312/ to unblock newton | 04:49 |
| EmilienM | And any core around | 04:49 |
| jaosorior | EmilienM: sure | 04:51 |
| EmilienM | thanks | 04:51 |
| *** ratailor__ has quit IRC | 04:52 | |
| *** dpawar has joined #tripleo | 04:52 | |
| jaosorior | EmilienM: stable/ocata is still timing out, right? | 04:52 |
| *** pdeore has joined #tripleo | 04:52 | |
| jaosorior | EmilienM: got time to review this one https://review.openstack.org/#/c/510404/ ? | 04:52 |
| *** bobh has quit IRC | 04:52 | |
| EmilienM | upgrade jobs yes | 04:52 |
| EmilienM | I'm on my phone now | 04:52 |
| jaosorior | ok, no biggie | 04:52 |
| EmilienM | I'll look asap | 04:52 |
| *** dparkes has joined #tripleo | 04:53 | |
| *** janki has joined #tripleo | 04:53 | |
| *** dparkes has quit IRC | 04:57 | |
| *** ratailor_ has quit IRC | 05:05 | |
| *** ratailor__ has joined #tripleo | 05:05 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| EmilienM | jaosorior: lgtm | 05:11 |
| jaosorior | EmilienM: thanks | 05:11 |
| EmilienM | jaosorior: thanks to you, I go to bed, ttyl | 05:13 |
| jaosorior | enjoy | 05:13 |
| Tengu | hello :) | 05:14 |
| jaosorior | Tengu: hey, how's it going? | 05:14 |
| *** marios has joined #tripleo | 05:14 | |
| Tengu | any way to debug that validation issue ? http://logs.openstack.org/83/510283/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024/58695d5/console.html#_2017-10-08_13_11_59_995862 - apparently it's related to | 05:14 |
| Tengu | http://logs.openstack.org/83/510283/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024/58695d5/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-10-08_13_13_26 but I don't get many logs :) | 05:14 |
| Tengu | jaosorior: still struggling getting a patch into pike, but it's on its way :). What about you? I saw you're doing some stuff with haproxy in order to allow to add new backends? | 05:15 |
| jaosorior | Tengu: I was doing that at some point. But I don't have enough time to continue that work :/ | 05:17 |
| Tengu | jaosorior: hmm, would it be possible I take over that part? | 05:17 |
| jaosorior | Tengu: of course! | 05:17 |
| Tengu | jaosorior: we really need that feature at work, sooo :) | 05:17 |
| Tengu | jaosorior: so I guess I'll need to take your branches and work on them directly, right? | 05:19 |
| jaosorior | Tengu: I guess this could even merge https://review.openstack.org/#/c/474109/ | 05:19 |
| jaosorior | Tengu: which would give you that functionality already | 05:19 |
| jaosorior | I was hoping to get some time to move some services to use that, instead of relying on the huge haproxy.pp file. But that I didn't get time to do :/ | 05:20 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Add resource to create haproxy endpoints dynamically https://review.openstack.org/474109 | 05:20 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Rely on dynamic haproxy endpoints to configure keystone https://review.openstack.org/474110 | 05:20 |
| Tengu | jaosorior: yup, I saw that when mwhahaha pointed it in my own issue (https://bugs.launchpad.net/tripleo/+bug/1721832) | 05:20 |
| openstack | Launchpad bug 1721832 in tripleo "Allow to add custom backends in HAProxy" [Medium,Triaged] | 05:20 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Configure keystone haproxy endpoints through dynamic resource https://review.openstack.org/474107 | 05:20 |
| jaosorior | Tengu: rebased the whole thing ^^ | 05:21 |
| Tengu | jaosorior: what's missing in your current change? | 05:21 |
| jaosorior | Tengu: from the base resource? Nothing I could think of. But I didn't get the time to migrate keystone to use that. That's all | 05:21 |
| *** aditya_r has joined #tripleo | 05:22 | |
| jaosorior | Tengu: you could probably take this patch https://review.openstack.org/#/c/474109/ test it out, and if it does what you need, we could merge it. | 05:22 |
| Tengu | jaosorior: hmm ok. migrating keystone is in a second patch right? | 05:22 |
| jaosorior | Tengu: yep :) and that patch is not needed to enable the feature you want. | 05:22 |
| Tengu | as mwhahaha pointed https://review.openstack.org/#/q/topic:haproxy-dynamic-endpoints+(status:open+OR+status:merged) - I see second and third are related to keystone. | 05:22 |
| Tengu | yup | 05:22 |
| Tengu | I really need the first one :) | 05:22 |
| jaosorior | Tengu: sure, try it out, and if it works for you we merge it. | 05:22 |
| Tengu | if possible, in pike. That part might cause some issue? | 05:23 |
| Tengu | jaosorior: shall I patch the puppet code on my infra, or is there a way to get some "beta packages"? | 05:23 |
| jaosorior | Tengu: damn, not sure if we can move that to pike :/ | 05:24 |
| Tengu | a bit early to get master/queens on a prod infra, right? ;) | 05:24 |
| jaosorior | Tengu: might want to ask other folks about that. I don't really know | 05:24 |
| Tengu | jaosorior: I think if we don't modify the "core" of the class and only allow to add custom backends, it should be fine. | 05:25 |
| jaosorior | Tengu: we could try. Nothing really uses that resource. | 05:25 |
| Tengu | so it should do. | 05:25 |
| *** achadha has quit IRC | 05:27 | |
| *** pmannidi has quit IRC | 05:31 | |
| *** pmannidi has joined #tripleo | 05:34 | |
| *** dpawar has quit IRC | 05:35 | |
| *** stendulker has quit IRC | 05:35 | |
| *** cshastri has joined #tripleo | 05:36 | |
| *** dpawar has joined #tripleo | 05:38 | |
| *** shreshtha has joined #tripleo | 05:39 | |
| *** ratailor_ has joined #tripleo | 05:41 | |
| *** nyechiel_ has joined #tripleo | 05:43 | |
| *** ratailor__ has quit IRC | 05:44 | |
| *** bobh has joined #tripleo | 05:49 | |
| *** etingof has quit IRC | 05:51 | |
| *** pmannidi has quit IRC | 05:51 | |
| *** dpawar has quit IRC | 05:53 | |
| *** bobh has quit IRC | 05:53 | |
| *** spectr has joined #tripleo | 05:54 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/pike: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/510408 | 05:55 |
| *** jtomasek has joined #tripleo | 05:56 | |
| Tengu | jaosorior: small question though: how may I create a new backend with your commit? I suppose I have to add a tripleo.MY_SERVICE.haproxy_endpoints in ExtraConfig, and activate MY_SERVICE somehow in there as well. Probably "service_names" in hiera, which is a list? | 05:59 |
| Tengu | jaosorior: and I guess the server_names list will later be pre-populated or be a merge in order to allow a smooth overriding ? | 06:00 |
| jaosorior | Tengu: so, you can do that, or you can create a service profile, similar to the ones in tripleo-heat-templates/puppet/services/ | 06:01 |
| jaosorior | and add the hieradata that way. | 06:01 |
| Tengu | jaosorior: ok | 06:01 |
| jaosorior | Tengu: doing that (the service profile) will fill up several hieradata entries for you. | 06:02 |
| Tengu | jaosorior: hmm ok. and in order to include that service, I'll need to add something in my local env file I guess. | 06:02 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-quickstart-extras master: Switch vbmc to Systemd's instantiated services https://review.openstack.org/510331 | 06:03 |
| *** udesale__ has joined #tripleo | 06:03 | |
| jaosorior | Tengu: yep | 06:03 |
| jaosorior | Tengu: you will need to add that service to your roles_data.yaml file | 06:03 |
| *** dparkes has joined #tripleo | 06:03 | |
| Tengu | yup. and that's not really convenient in our case, as we actually use the default roles_data content | 06:05 |
| Tengu | duplicate it just to add a new class isn't great :/. | 06:05 |
| *** udesale has quit IRC | 06:05 | |
| Tengu | thus I think I'll do the full-hiera way | 06:05 |
| Tengu | we will also manage the firewall opening through it. | 06:05 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for Glance API containers to log to stdout/stderr https://review.openstack.org/510411 | 06:07 |
| jaosorior | Tengu: ok, if that works for you :) | 06:07 |
| *** aufi has joined #tripleo | 06:08 | |
| *** aditya_r has quit IRC | 06:09 | |
| *** suuuper has joined #tripleo | 06:09 | |
| *** aditya_r has joined #tripleo | 06:10 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| *** etingof has joined #tripleo | 06:12 | |
| lvdombrkr | morning folks | 06:12 |
| *** dpawar has joined #tripleo | 06:14 | |
| *** aditya_r has quit IRC | 06:16 | |
| *** aditya_r has joined #tripleo | 06:16 | |
| *** skramaja has joined #tripleo | 06:21 | |
| jaosorior | morning! | 06:21 |
| *** dsneddon has joined #tripleo | 06:21 | |
| Tengu | hmm. apparently… http://logs.openstack.org/83/510283/1/check-tripleo/legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024/6e52993/job-output.txt.gz#_2017-10-08_14_15_11_967743 there are still some issue in the CI. | 06:23 |
| lvdombrkr | i have question about controller HA, i have three controller nodes and when turn off two, horizon not available | 06:23 |
| lvdombrkr | when turn off 1 controller then i steal have access to dashboard | 06:24 |
| lvdombrkr | it is okey? | 06:24 |
| Tengu | lvdombrkr: you should have a look at pcs status in order to check what's happening | 06:24 |
| *** akane has joined #tripleo | 06:24 | |
| Tengu | as all HA stuff is managed by corosync/pacemaker/pcsd, they are the source of truth in the setup. | 06:25 |
| Tengu | lvdombrkr: small note: the HA is mainly due to a VIP that's attached to one controller - if that controller goes down, it's attached to another controller. When you kill one controller, is it the one with the VIP? | 06:26 |
| Tengu | lvdombrkr: more over, there's a delay for the VIP to be migrated, it might take a minute. | 06:26 |
| *** jprovazn has joined #tripleo | 06:27 | |
| *** yprokule has joined #tripleo | 06:27 | |
| *** achadha has joined #tripleo | 06:28 | |
| *** ykarel_ has joined #tripleo | 06:28 | |
| lvdombrkr | Tengu: thanks, i will look into psc it must be become more clear | 06:30 |
| *** jprovazn_ has joined #tripleo | 06:31 | |
| *** ykarel has quit IRC | 06:31 | |
| *** achadha has quit IRC | 06:32 | |
| *** ratailor_ has quit IRC | 06:35 | |
| *** ratailor__ has joined #tripleo | 06:35 | |
| *** threestrands has quit IRC | 06:37 | |
| lvdombrkr | Tengu: if i stop two of three controllers i see something like : http://paste.openstack.org/raw/623048/ | 06:40 |
| lvdombrkr | and VIP is not pingable | 06:40 |
| lvdombrkr | but all floating ips steel working | 06:40 |
| Tengu | lvdombrkr: well, all is stopped. | 06:40 |
| Tengu | that's normal | 06:40 |
| *** rcernin has joined #tripleo | 06:40 | |
| Tengu | floating go through a virtual router | 06:40 |
| Tengu | as well as the privates, they can still access to internet because of virtual router. Maybe the HA needs to have a least two nodes, I didn't check the policies. | 06:41 |
| *** noslzzp has quit IRC | 06:41 | |
| Tengu | lvdombrkr: and you can see the main issue is apparently galera | 06:42 |
| lvdombrkr | Tengu: in galera logs? | 06:42 |
| Tengu | nope, in the output you pasted. | 06:42 |
| Tengu | galera_monitor_10000 on overcloud-controller-2 'unknown error' | 06:43 |
| Tengu | in the "failed actions" part. | 06:43 |
| lvdombrkr | Tengu: thanks, but in normal if controller-2 is started its must become as primary? if its stay only alive controller | 06:46 |
| Tengu | lvdombrkr: as said, I didn't check who the policies are done in pacemaker/corosync - plus, maybe galera doesn't like having 2 nodes down in a 3-nodes cluster. | 06:47 |
| *** noslzzp has joined #tripleo | 06:47 | |
| Tengu | *how, not who | 06:47 |
| jaosorior | dciabrin: ^^ | 06:48 |
| dciabrin | Tengu, when two out of three galera nodes don't quit the galera cluster cleanly, the remaining one goes into Non-Primary node (loss of galera quorum), and pacemaker make this third node stop | 06:48 |
| *** nyechiel_ has quit IRC | 06:48 | |
| Tengu | dciabrin: ah, so my guess wasn't far from truth :). Thanks for the precision. | 06:48 |
| Tengu | lvdombrkr: does it answer your questions? | 06:49 |
| * dciabrin disappears on a school run | 06:49 | |
| Tengu | :) | 06:49 |
| Tengu | sooo. anyone having a minute in order to help me a bit with the two issues raised by the CI on my review? https://review.openstack.org/#/c/510283/ I'm not sure the issues are in my code, especially since it merged fine into Master earlier… | 06:50 |
| lvdombrkr | Tengu: yes, thanks now its clear... if all three controller not accessible, i can see somewhere else any logs about pacamaker issues? | 06:50 |
| Tengu | it's kind of frustrating sometimes -.-' | 06:50 |
| Tengu | lvdombrkr: hmm nope. unless you actually export your system logs to some remote location. | 06:51 |
| *** masco has joined #tripleo | 06:51 | |
| Tengu | lvdombrkr: as pacemaker is only on the three controllers, at least for those resources, you won't be able to find info on, say, a compute. Nor the undercloud. | 06:51 |
| *** ccamacho has joined #tripleo | 06:52 | |
| lvdombrkr | Tengu: thanks, its clear | 06:54 |
| *** dpawar has quit IRC | 06:55 | |
| Tengu | jaosorior: btw, do you want me to add some release note to your merge request? | 06:57 |
| Tengu | as I understood, it's a good practice :). Doing so will also ensure I understand your changes, so I can take over in order to push them forward. | 06:58 |
| *** udesale has joined #tripleo | 06:59 | |
| *** udesale has quit IRC | 06:59 | |
| *** udesale has joined #tripleo | 06:59 | |
| *** udesale has quit IRC | 07:00 | |
| *** udesale__ has quit IRC | 07:01 | |
| *** jaganathan has joined #tripleo | 07:01 | |
| *** udesale has joined #tripleo | 07:01 | |
| *** jprovazn_ has quit IRC | 07:03 | |
| *** ebarrera has joined #tripleo | 07:05 | |
| jaosorior | Tengu: sure | 07:05 |
| *** dmacpher has quit IRC | 07:05 | |
| Tengu | jaosorior: ok, I let the current run finish, and will submit the change note :). | 07:06 |
| *** aditya_r has quit IRC | 07:06 | |
| *** pcaruana has joined #tripleo | 07:08 | |
| *** aditya_r has joined #tripleo | 07:09 | |
| *** jlinkes has joined #tripleo | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| *** etingof has quit IRC | 07:11 | |
| *** etingof has joined #tripleo | 07:13 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/puppet-tripleo master: Add resource to create haproxy endpoints dynamically https://review.openstack.org/474109 | 07:15 |
| *** ratailor_ has joined #tripleo | 07:18 | |
| *** ratailor__ has quit IRC | 07:22 | |
| *** ratailor__ has joined #tripleo | 07:25 | |
| *** Pranav has joined #tripleo | 07:25 | |
| *** ratailor_ has quit IRC | 07:28 | |
| *** ratailor has joined #tripleo | 07:29 | |
| *** achadha has joined #tripleo | 07:30 | |
| Tengu | http://logs.openstack.org/07/474107/8/check/legacy-tripleo-ci-centos-7-undercloud-containers/7eaa34a/job-output.txt.gz#_2017-10-09_06_11_23_452520 -.- | 07:30 |
| *** ffiore has joined #tripleo | 07:30 | |
| *** ratailor__ has quit IRC | 07:31 | |
| jaosorior | sshnaidm: any idea what's up with that? ^^ | 07:31 |
| Pranav | Hello good folks. Need help with containerized overcloud deployment. | 07:32 |
| Pranav | Image upload fails with error : http://paste.openstack.org/show/623053/ | 07:32 |
| *** dpawar has joined #tripleo | 07:34 | |
| *** achadha has quit IRC | 07:34 | |
| *** Shatadru is now known as Shatadru|brb | 07:34 | |
| jaosorior | mandre: ^^ | 07:35 |
| sshnaidm | jaosorior, http://logs.openstack.org/07/474107/8/check/legacy-tripleo-ci-centos-7-undercloud-containers/7eaa34a/job-output.txt.gz#_2017-10-09_06_11_16_195414 | 07:36 |
| jaosorior | O_o | 07:37 |
| *** pdeore has quit IRC | 07:39 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** egonzalez has joined #tripleo | 07:41 | |
| *** paramite has joined #tripleo | 07:41 | |
| *** ykarel_ is now known as ykarel | 07:45 | |
| *** pdeore has joined #tripleo | 07:47 | |
| *** amoralej|off is now known as amoralej | 07:50 | |
| *** etingof has quit IRC | 07:51 | |
| Tengu | grmbl. apparently,. yes, there are issues in the CI. | 07:54 |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-heat-templates master: Allow configuration Zaqar with Redis https://review.openstack.org/506071 | 07:54 |
| Tengu | sshnaidm: wow, nice error :). | 07:54 |
| Tengu | reminds me of some error "Fatal: no error" X) | 07:55 |
| *** shardy has joined #tripleo | 07:55 | |
| sshnaidm | Tengu, actually it's this one: http://logs.openstack.org/07/474107/8/check/legacy-tripleo-ci-centos-7-undercloud-containers/7eaa34a/job-output.txt.gz#_2017-10-09_06_11_16_195686 | 07:56 |
| Tengu | sshnaidm: hmmm. weird. | 07:56 |
| Tengu | that means haproxy isn't ready, or its backend, right? | 07:56 |
| sshnaidm | Tengu, I dunno.. full log you can see here: http://logs.openstack.org/07/474107/8/check/legacy-tripleo-ci-centos-7-undercloud-containers/7eaa34a/logs/var/log/undercloud_install.txt.gz | 07:57 |
| Tengu | duh… jaosorior I just added the release note file, and not almost all the tests are failing for the haproxy endpoint patch | 07:58 |
| Tengu | sshnaidm: [WARNING] Config file failed schema validation at network_config/0: might not help. | 07:58 |
| Tengu | meaning: no network. | 07:59 |
| Tengu | http://logs.openstack.org/07/474107/8/check/legacy-tripleo-ci-centos-7-undercloud-containers/7eaa34a/logs/var/log/undercloud_install.txt.gz#_2017-10-09_06_11_11_000 | 07:59 |
| jaosorior | uhm.. | 08:00 |
| Tengu | jaosorior: as you say -.- | 08:00 |
| *** jpich has joined #tripleo | 08:02 | |
| *** sshnaidm is now known as sshnaidm_ | 08:03 | |
| *** sshnaidm_ is now known as sshnaidm | 08:03 | |
| *** etingof has joined #tripleo | 08:03 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: /etc/machine-id detection improvements https://review.openstack.org/510312 | 08:06 |
| *** mcornea has joined #tripleo | 08:08 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| *** masco has quit IRC | 08:13 | |
| *** ykarel_ has joined #tripleo | 08:14 | |
| *** ykarel_ has quit IRC | 08:16 | |
| openstackgerrit | Javier Peña proposed openstack/tripleo-quickstart-extras master: Allow pre-installed DLRN https://review.openstack.org/499117 | 08:16 |
| *** masco has joined #tripleo | 08:17 | |
| lvdombrkr | folks, last stable version is ocata at the moment? | 08:20 |
| jaosorior | lvdombrkr: well, there is stable/pike already | 08:23 |
| *** lucas-afk is now known as lucasagomes | 08:24 | |
| lvdombrkr | jaosorior: mhmm in documentation i can see ocata as last | 08:24 |
| Tengu | pike has 1 month now ;) | 08:25 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-docs master: Update documentation for O to P upgrade and update https://review.openstack.org/496223 | 08:30 |
| *** d0ugal_ has quit IRC | 08:31 | |
| *** d0ugal has joined #tripleo | 08:31 | |
| *** d0ugal has quit IRC | 08:31 | |
| *** d0ugal has joined #tripleo | 08:31 | |
| *** derekh has joined #tripleo | 08:32 | |
| *** ratailor_ has joined #tripleo | 08:34 | |
| *** gfidente has joined #tripleo | 08:35 | |
| *** chem has joined #tripleo | 08:36 | |
| *** ratailor has quit IRC | 08:38 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Don't try to update cloud-init anymore https://review.openstack.org/504850 | 08:38 |
| *** Shatadru|brb is now known as Shatadru | 08:39 | |
| *** artom has joined #tripleo | 08:44 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix TripleO CI jobs https://review.openstack.org/508660 | 08:46 |
| *** aditya_r has quit IRC | 08:48 | |
| *** ratailor_ has quit IRC | 08:50 | |
| *** ratailor has joined #tripleo | 08:50 | |
| *** salmankhan has joined #tripleo | 08:51 | |
| *** bobh has joined #tripleo | 08:51 | |
| *** ffiore has quit IRC | 08:52 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: WIP mistral workflow to deploy nodes without nova https://review.openstack.org/313048 | 08:52 |
| *** salmankhan has quit IRC | 08:55 | |
| *** bobh has quit IRC | 08:56 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: WIP mistral workflow to deploy nodes without nova https://review.openstack.org/313048 | 08:56 |
| *** stendulker has joined #tripleo | 08:56 | |
| *** salmankhan has joined #tripleo | 08:57 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Migrate to the new Mistral context class https://review.openstack.org/506186 | 09:04 |
| *** ykarel is now known as ykarel|lunch | 09:05 | |
| *** salmankhan has quit IRC | 09:05 | |
| *** ffiore has joined #tripleo | 09:08 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 09:10 |
| lvdombrkr | folks, is there any command to stop deployment which is in progress? | 09:11 |
| Tengu | lvdombrkr: not really to my knowledge. why would you do that? | 09:11 |
| *** dciabrin has quit IRC | 09:12 | |
| lvdombrkr | i tried add tls to my cluster but deployment stuck for a while | 09:13 |
| lvdombrkr | Tengu: ^ | 09:13 |
| Tengu | duh… ok. hmm in the first deploy, you can't cancel it I think. | 09:13 |
| Tengu | the only way to do so will be to ctrl+c, and stop openstack-heat-engine service - at least that's what we do when we don't want to wait for the "final" failure. | 09:14 |
| *** athomas has joined #tripleo | 09:14 | |
| lvdombrkr | Tengu: its not first deploy im tried updating existing cluster with tls | 09:14 |
| lvdombrkr | and deployment stuck | 09:15 |
| Tengu | ah, you should be able to cancel it then | 09:15 |
| *** tosky has joined #tripleo | 09:15 | |
| Tengu | openstack stack cancel overcloud or something like that. Note that it will do a rollback. | 09:15 |
| Tengu | and it might end with a rollback_failed | 09:15 |
| *** salmankhan has joined #tripleo | 09:15 | |
| *** psachin has quit IRC | 09:20 | |
| *** yamahata has quit IRC | 09:22 | |
| *** psachin has joined #tripleo | 09:23 | |
| lvdombrkr | Tengu: thanks, rollback in progress...im cross my fingers ) | 09:24 |
| *** psahoo has quit IRC | 09:25 | |
| Tengu | lvdombrkr: if you're in a rollback_failed state, you might still try to run an update (without TLS) | 09:27 |
| Tengu | and that should end on an update_complete | 09:27 |
| ratailor | jaosorior, you around ? | 09:28 |
| *** cylopez has joined #tripleo | 09:30 | |
| jaosorior | ratailor: what's up? | 09:30 |
| ratailor | have you encountered https://bugs.launchpad.net/tripleo/+bug/1720137 | 09:31 |
| openstack | Launchpad bug 1720137 in tripleo "Overcloud Deployment fails: No valid host was found. " [High,Confirmed] | 09:31 |
| *** dciabrin has joined #tripleo | 09:31 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates stable/pike: Containerized Fluentd client https://review.openstack.org/507506 | 09:31 |
| *** achadha has joined #tripleo | 09:32 | |
| Tengu | ratailor: heya! I think I did see that one | 09:32 |
| ratailor | I suspect its because of novajoin, I could see AttributeError: 'Env' object has no attribute 'domain' in novajoin-server.log | 09:32 |
| jaosorior | ratailor: IIRC, that one was solved already, and you need an updated version of novajoin | 09:33 |
| ratailor | jaosorior, I think you might have discussed it already with bnemec http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2017-09-19.log.html | 09:33 |
| ratailor | jaosorior, could you provide any reference of bug, which fixed it. | 09:34 |
| jaosorior | ratailor: yeah; and as far as I remember; that was the issue. | 09:34 |
| jaosorior | ratailor: I don't recall. I just remember that when I talked to rob about it. He had actually arleady fixed the issue. But it had been two weeks since last promotion back then, so we had an older version of novajoin available in RDO./ | 09:34 |
| ratailor | jaosorior, Do you know how much time it would take approx for new version of novajoin to become available ? | 09:35 |
| jaosorior | ratailor: depends on promotion | 09:35 |
| jaosorior | ratailor: you could use it if you use tripleo.sh though | 09:36 |
| ratailor | jaosorior, Ohh, so I should be asking to some CI guys :) | 09:36 |
| jaosorior | ratailor: ./tripleo-ci/scripts/tripleo.sh --repo-setup | 09:36 |
| jaosorior | ratailor: and then do sudo yum update -y | 09:36 |
| ratailor | jaosorior, sure, | 09:36 |
| ratailor | jaosorior, Thanks! | 09:36 |
| *** achadha has quit IRC | 09:37 | |
| Tengu | grumpf. | 09:38 |
| jaosorior | Tengu: ?? | 09:38 |
| Tengu | I still don't understand what's wrong in https://review.openstack.org/#/c/510283/ :'( | 09:38 |
| jaosorior | Tengu: well, jenkins is passing there. Seems to be an issue with zuul; but it could merge just with the jenkins vote. | 09:39 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates stable/pike: docker: add logging(source & groups) https://review.openstack.org/507952 | 09:39 |
| Tengu | jaosorior: ah. so I need some +2 and the like in order to go to next step? | 09:40 |
| openstackgerrit | Lukas Bezdicka proposed openstack/python-tripleoclient master: Implement minor update workflow with config download https://review.openstack.org/487488 | 09:40 |
| jaosorior | Tengu: yep | 09:40 |
| Tengu | even with jenkins failing on gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 ? | 09:40 |
| jaosorior | Tengu: I suggest pinging mwhahaha to see what he thinks about that backport | 09:40 |
| Tengu | jaosorior: yup, that was the plan once I get a correct output in the CI. but if you say "it's ok", well, then, ping mwhahaha :)) | 09:41 |
| *** ccamacho has quit IRC | 09:42 | |
| *** ccamacho has joined #tripleo | 09:43 | |
| Tengu | now, regarding the other patch (haproxy endpoints) thats failing more since I added the release note… I'll start a new check. | 09:43 |
| Tengu | maybe there were some job collision. | 09:43 |
| *** milan has joined #tripleo | 09:46 | |
| jaosorior | Tengu: by the way, where are you based on? | 09:48 |
| jaosorior | mandre: hey dude, if you have some time could you take a look at this https://review.openstack.org/#/c/510001/ ? | 09:49 |
| *** egonzalez has quit IRC | 09:50 | |
| *** bobh has joined #tripleo | 09:52 | |
| *** dbecker has joined #tripleo | 09:56 | |
| *** bobh has quit IRC | 09:56 | |
| lvdombrkr | Tengu: how long time rolling back can take? it there any way to look in progress or something like that | 09:58 |
| *** shreshtha has quit IRC | 09:59 | |
| *** cylopez has quit IRC | 10:00 | |
| *** egonzalez has joined #tripleo | 10:03 | |
| *** psachin has quit IRC | 10:05 | |
| *** ykarel|lunch is now known as ykarel | 10:07 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for keystone containers to log to stdout/stderr https://review.openstack.org/508517 | 10:07 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for nova containers to log to stdout/stderr https://review.openstack.org/509157 | 10:07 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for Glance API containers to log to stdout/stderr https://review.openstack.org/510411 | 10:07 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Remove log-dir option from neutron-dhcp execution https://review.openstack.org/510463 | 10:07 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for Neutron containers to log to stdout/stderr https://review.openstack.org/510464 | 10:07 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 10:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for nova containers to log to stdout/stderr https://review.openstack.org/509157 | 10:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for Glance API containers to log to stdout/stderr https://review.openstack.org/510411 | 10:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Remove log-dir option from neutron-dhcp execution https://review.openstack.org/510463 | 10:10 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/instack-undercloud master: Make sure selinux permissions are correct on ~/.ssh. https://review.openstack.org/495157 | 10:11 |
| Tengu | lvdombrkr: hmm, no idea for the time - but you can have some status inside with `openstack stack list --nested | grep -vi complete' | 10:11 |
| Tengu | or something like that. | 10:11 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Remove log-dir option from neutron-dhcp execution https://review.openstack.org/510463 | 10:14 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for Neutron containers to log to stdout/stderr https://review.openstack.org/510464 | 10:14 |
| Tengu | jaosorior: wow, you want to burn the CI? :Þ | 10:15 |
| *** mhenkel has quit IRC | 10:15 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for Neutron containers to log to stdout/stderr https://review.openstack.org/510464 | 10:17 |
| jaosorior | Tengu: that was my evil plan all along :P | 10:18 |
| Tengu | :Þ | 10:18 |
| *** psachin has joined #tripleo | 10:20 | |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-heat-templates master: Allow configuration Zaqar with Redis https://review.openstack.org/506071 | 10:23 |
| *** udesale has quit IRC | 10:25 | |
| *** sid1 has joined #tripleo | 10:26 | |
| *** pblaho has joined #tripleo | 10:26 | |
| *** daidv has quit IRC | 10:29 | |
| *** achadha has joined #tripleo | 10:33 | |
| *** achadha has quit IRC | 10:38 | |
| *** mhenkel has joined #tripleo | 10:40 | |
| *** ed_b has quit IRC | 10:42 | |
| *** jkilpatr has joined #tripleo | 10:43 | |
| lvdombrkr | Tengu: thanks in assist, im back in game ) | 10:48 |
| Tengu | lvdombrkr: good :). I'm happy I can help a bit, that means I understand better how all works ;) | 10:48 |
| *** thrash|g0ne is now known as thrash | 10:51 | |
| *** pdeore has quit IRC | 10:51 | |
| *** jkilpatr has quit IRC | 10:51 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for Neutron containers to log to stdout/stderr https://review.openstack.org/510464 | 10:52 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common stable/pike: Bring the keystone utils up-to-date with Mistral https://review.openstack.org/510527 | 10:52 |
| Tengu | small question: should I apply the resume_guests_state_on_host_boot nova option only on compute nodes, or does it also have to be set on the controllers? | 10:55 |
| *** dtantsur|afk is now known as dtantsur | 10:55 | |
| *** Shatadru is now known as Shatadru|Gone | 10:56 | |
| *** Goneri has joined #tripleo | 10:56 | |
| *** raildo has joined #tripleo | 10:58 | |
| *** morazi has quit IRC | 11:03 | |
| hjensas | In CI I see container jobs fail with "Database schema file with version 85 doesn't exist." when running 'docker', 'run', '--name', 'heat_engine_db_sync' in tripleo jobs that has depends-on a Heat change. Is containers re-built when there is depends-on, or could this be because of a delay in the container images available? | 11:03 |
| hjensas | weshay: ^^ You might know? | 11:04 |
| *** jkilpatr has joined #tripleo | 11:04 | |
| *** jaosorior has quit IRC | 11:05 | |
| *** rhallisey_ has joined #tripleo | 11:05 | |
| *** jaosorior has joined #tripleo | 11:05 | |
| d0ugal | Anyone got time to look at a small Mistral config change? https://review.openstack.org/#/c/509811/ | 11:06 |
| jaosorior | d0ugal: -2 | 11:07 |
| jaosorior | d0ugal: jk; merged. | 11:08 |
| fultonj | LGTM | 11:08 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Revert "latest version of DLRN breaks CI" https://review.openstack.org/510056 | 11:09 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Node config for HA overcloud and FreeIPA node https://review.openstack.org/510012 | 11:09 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Disable cloud-init for IPA supplemental VM https://review.openstack.org/510007 | 11:09 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Fix image locations in roles/libvirt/defaults/main.yml https://review.openstack.org/409464 | 11:09 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 11:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Fix ansible-lint.sh to check playbooks https://review.openstack.org/446525 | 11:17 |
| *** shreshtha has joined #tripleo | 11:18 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: WIP: Set file ACLs for Ceph keyrings for non-containerized deployment https://review.openstack.org/509020 | 11:18 |
| fultonj | colonwq: ^ fyi | 11:19 |
| *** egonzalez has quit IRC | 11:19 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Set empty default for network_isolation_args https://review.openstack.org/497543 | 11:20 |
| *** pblaho has quit IRC | 11:20 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: WIP: Set file ACLs for Ceph keyrings for non-containerized deployment https://review.openstack.org/509020 | 11:21 |
| *** pkovar has joined #tripleo | 11:22 | |
| colonwq | fultonj, ok | 11:22 |
| openstackgerrit | yatin proposed openstack/instack-undercloud master: Use keystone v3 session with novaclient https://review.openstack.org/510535 | 11:23 |
| jaosorior | mandre: is there any reason why we bind-mount /var/log/containers/nova to the nova-libvirt container? | 11:24 |
| ykarel | EmilienM, ^^ | 11:24 |
| *** stendulker has quit IRC | 11:24 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates stable/pike: Fix cold/live migration network config https://review.openstack.org/510536 | 11:24 |
| jaosorior | owalsh: is there any reason why we bind-mount /var/log/containers/nova to the nova-libvirt container? | 11:24 |
| *** Pranav has quit IRC | 11:24 | |
| *** sid1 has quit IRC | 11:25 | |
| *** egonzalez has joined #tripleo | 11:26 | |
| owalsh | jaosorior: not sure, was added in https://review.openstack.org/#/c/442603/27/docker/services/nova-libvirt.yaml | 11:27 |
| *** lucasagomes is now known as lucas-hungry | 11:27 | |
| d0ugal | jaosorior: thanks :) | 11:28 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for nova compute container to log to stdout/stderr https://review.openstack.org/510537 | 11:28 |
| jaosorior | owalsh: probably an accident | 11:28 |
| *** Goneri has quit IRC | 11:28 | |
| jaosorior | jistr: you around? | 11:30 |
| jaosorior | jistr: what logs to /var/log/libvirt/qemu? Seems to me like it's not actually being used. | 11:31 |
| rook | shardy: hey - last time we chatted didn't you want me to try to enable convergence? or was that danp? | 11:31 |
| *** raildo has quit IRC | 11:33 | |
| *** raildo has joined #tripleo | 11:33 | |
| shardy | rook: Hey, yes we've been discussing switching to convergence so it'd be good to enable it in a fairly large deployment and see how it impacts heat resource usage on the undercloud | 11:34 |
| *** dpawar has quit IRC | 11:35 | |
| rook | shardy: I won't have before numbers (unless I monk it out and destroy my sand castle, and redeploy with it turned off) | 11:35 |
| shardy | rook: it's likely to be slightly more memory and disk intensive, but it'd be good to quantify that | 11:35 |
| jaosorior | owalsh: hey, if I want to add an optino to configure logging for libvirtd, in what manifest should I do it? I see a bunch of configs are being done in manifests/migration/libvirt.pp in puppet-nova. But I'm not sure if that's the appropriate place to put it | 11:36 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Remove package if service stopped and disabled https://review.openstack.org/479886 | 11:36 |
| shardy | rook: ack, well comparative numbers would be ideal, but just ensuring it works at scale without any issues would still be helpful | 11:36 |
| shardy | rook: I've tested it but only with fairly small VM based environments | 11:36 |
| rook | shardy: sure. I'll deploy with 32 nodes, destory, then enable, then deploy with 32 nodes again | 11:37 |
| rook | and monitor usage. | 11:37 |
| rook | will 32 nodes suffice? | 11:38 |
| rook | I have ~90 to play with, but that takes a couple of hours to get up totally. | 11:38 |
| owalsh | jaosorior: probably manifests/compute/libvirt.pp | 11:38 |
| shardy | rook: ack yes that sounds great as a first step to validating all is OK with convergence enabled, thanks! | 11:38 |
| rook | shardy do different node types matter (ie cause more utilization?) | 11:38 |
| jaosorior | owalsh: yeah, I guess that makes more sense | 11:39 |
| jaosorior | owalsh: thanks | 11:39 |
| shardy | rook: Not really, the deploy steps are the same for all roles regardless of what's enabled | 11:39 |
| rook | shardy: rgr. | 11:39 |
| shardy | so from the undercloud perspective it should be about the same - it's mostly about the number of total nodes/resources | 11:39 |
| rook | shardy: this is a pretty custom deploy due to the mix bag of nodes i have | 11:40 |
| rook | I think you saw the roles_data | 11:40 |
| shardy | rook: ack, yeah any known working mix of roles should be fine | 11:40 |
| *** shardy is now known as shardy_lunch | 11:40 | |
| *** dougbtv has joined #tripleo | 11:40 | |
| rook | convergence_engine=False <-- shardy_lunch that is the only bit to flip, right? | 11:41 |
| *** sid1 has joined #tripleo | 11:42 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates stable/ocata: Fix cold/live migration network config https://review.openstack.org/510539 | 11:42 |
| *** ansiwen[q] has joined #tripleo | 11:46 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates stable/ocata: Fix cold/live migration network config https://review.openstack.org/510539 | 11:47 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Special treatment for os-net-config upgrade. https://review.openstack.org/509214 | 11:49 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates stable/newton: Fix cold/live migration network config https://review.openstack.org/510543 | 11:50 |
| *** bobh has joined #tripleo | 11:54 | |
| openstackgerrit | yatin proposed openstack/instack-undercloud master: Use keystone v3 session with novaclient https://review.openstack.org/510535 | 11:56 |
| *** fandrieu has quit IRC | 11:57 | |
| jistr | jaosorior: re /var/log/libvirt/qemu -- i think it's for virtlogd | 11:58 |
| *** abishop has joined #tripleo | 11:58 | |
| *** bobh has quit IRC | 11:58 | |
| *** trown|outtypewww is now known as trown | 11:59 | |
| jistr | jaosorior: i don't know the details, but it's possible that there wouldn't be anything until an instance is launched perhaps | 12:00 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/pike: Remove package if service stopped and disabled https://review.openstack.org/510545 | 12:00 |
| jaosorior | jistr: that would make sense | 12:00 |
| jaosorior | jistr: but, does the libvirt container need that bind mount? seems to me like only the virtlogd container would need it. | 12:01 |
| *** aputtur has joined #tripleo | 12:01 | |
| *** aditya_r has joined #tripleo | 12:05 | |
| *** etingof has quit IRC | 12:07 | |
| jistr | jaosorior: hm i don't know for sure, but when i tested it, i think it was necessary to get the pingtest to succeed | 12:07 |
| *** eck`gone is now known as eck` | 12:10 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 12:10 |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| *** ecerquei__ has joined #tripleo | 12:10 | |
| *** udesale has joined #tripleo | 12:13 | |
| *** bobh has joined #tripleo | 12:13 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Create mysql user for non-ha deployments https://review.openstack.org/508504 | 12:14 |
| paramite | Greetings guys, can we please get following two backports merged: https://review.openstack.org/#/c/507506/ https://review.openstack.org/#/c/507952/ ? | 12:16 |
| paramite | and thanks for merging ^^ | 12:17 |
| *** liverpooler has joined #tripleo | 12:18 | |
| *** adarazs is now known as adarazs_off | 12:20 | |
| *** etingof has joined #tripleo | 12:21 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates stable/pike: Create mysql user for non-ha deployments https://review.openstack.org/510550 | 12:21 |
| *** lucas-hungry is now known as lucasagomes | 12:24 | |
| *** ratailor has quit IRC | 12:24 | |
| *** bobh has quit IRC | 12:28 | |
| *** ratailor has joined #tripleo | 12:28 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates stable/pike: List all unhealthy containers https://review.openstack.org/510554 | 12:29 |
| *** pblaho has joined #tripleo | 12:29 | |
| *** bfournie has quit IRC | 12:29 | |
| *** bfournie has joined #tripleo | 12:29 | |
| *** ffiore has quit IRC | 12:31 | |
| *** raildo has quit IRC | 12:32 | |
| *** morazi has joined #tripleo | 12:32 | |
| *** bfournie has quit IRC | 12:34 | |
| *** rlandy has joined #tripleo | 12:34 | |
| *** ratailor_ has joined #tripleo | 12:34 | |
| *** achadha has joined #tripleo | 12:35 | |
| *** gkadam has quit IRC | 12:35 | |
| *** fandrieu has joined #tripleo | 12:36 | |
| *** raildo has joined #tripleo | 12:36 | |
| *** jcoufal has joined #tripleo | 12:36 | |
| *** leitan has joined #tripleo | 12:37 | |
| *** chlong has quit IRC | 12:37 | |
| *** ratailor has quit IRC | 12:38 | |
| *** rbrady_ has joined #tripleo | 12:39 | |
| *** rbrady_ has quit IRC | 12:39 | |
| *** rbrady_ has joined #tripleo | 12:39 | |
| *** rbrady has quit IRC | 12:39 | |
| *** rbrady_ has quit IRC | 12:39 | |
| *** achadha has quit IRC | 12:40 | |
| *** shardy_lunch is now known as shardy | 12:40 | |
| *** catintheroof has joined #tripleo | 12:43 | |
| *** aditya_r has quit IRC | 12:43 | |
| *** jmelvin has joined #tripleo | 12:43 | |
| *** ffiore has joined #tripleo | 12:47 | |
| hrybacki | jaosorior: sshnaidm why https://review.openstack.org/#/c/510007/ over https://review.openstack.org/#/c/508158/ (tied to the actual LP -- which is still open) | 12:48 |
| mandre | jaosorior: I don't have a better answer than jistr about the mounts, maybe it was needed at the time they were added | 12:48 |
| jaosorior | hrybacki: cause I hadn't seen the former. | 12:49 |
| sshnaidm | hrybacki, I didn't see your patch, only this one from jaosorior | 12:49 |
| hrybacki | sshnaidm jaosorior: damn. Is there a proper way to tie that to the LP after it has landed? Or do we just mark that as fix released? https://bugs.launchpad.net/tripleo/+bug/1718712 | 12:50 |
| openstack | Launchpad bug 1718712 in tripleo "cloud-init resets eth0 of supplemental node breaking networking" [High,In progress] - Assigned to Harry Rybacki (hrybacki-h) | 12:50 |
| sshnaidm | hrybacki, no, only mark manually.. | 12:51 |
| *** artom has quit IRC | 12:52 | |
| *** ffiore has quit IRC | 12:52 | |
| hrybacki | ack done | 12:52 |
| mandre | hjensas: containers are not yet rebuilt when there is a Depends-On, so in your case it seems like the container is too old. Where are you seing this error? | 12:54 |
| chandankumar | jaosorior: hello | 12:57 |
| chandankumar | jaosorior: please have a look on this http://logs.openstack.org/28/499928/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/55297dd/logs/undercloud/home/jenkins/tempest_output.log.txt.gz | 12:57 |
| *** jpena is now known as jpena|lunch | 12:58 | |
| jaosorior | chandankumar: looks like a neutron issue | 12:58 |
| *** raildo has quit IRC | 12:58 | |
| chandankumar | container scenario 2 tempest encrytped volume tests are failing for this review https://review.openstack.org/499928 | 12:58 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: [WIP]Add a healt check for overcloud nodes https://review.openstack.org/510560 | 12:58 |
| *** ffiore has joined #tripleo | 12:59 | |
| *** lblanchard has joined #tripleo | 13:00 | |
| trozet | is zuul -1 other people's commit after passing all jobs? | 13:00 |
| *** ansmith has joined #tripleo | 13:01 | |
| *** abregman|afk is now known as abregman | 13:01 | |
| trozet | mwhahaha: can you review https://review.openstack.org/#/c/509834/ please | 13:02 |
| *** bfournie has joined #tripleo | 13:02 | |
| *** rbrady has joined #tripleo | 13:04 | |
| *** rbrady has quit IRC | 13:04 | |
| *** rbrady has joined #tripleo | 13:04 | |
| fultonj | colonwq: https://review.openstack.org/#/c/509021/5/manifests/key.pp@142 | 13:04 |
| fultonj | colonwq: it's getting there, just needs to loop | 13:05 |
| hjensas | mandre: https://review.openstack.org/#/c/473817/ and https://review.openstack.org/#/c/437544/ | 13:05 |
| trown | weshay: ... wont let me comment in launchpad atm... but I think that bug is unrelated to the puppet-firewall issue because we already have it pinned on pike: https://review.rdoproject.org/r/gitweb?p=rdoinfo.git;a=blob;f=rdo.yml;h=9cdc1e0c16668fd420c263f2d531c62023131210;hb=HEAD#l614 | 13:05 |
| hjensas | mandre: http://logs.openstack.org/44/437544/72/check/gate-tripleo-ci-centos-7-containers-multinode/c243aed/logs/subnode-2/var/log/journal.txt.gz#_Oct_05_13_32_12 -- scroll a bit a lot og logs at that second. :) | 13:05 |
| trown | weshay: do you know if it is possible to reproduce that issue on RDO cloud? | 13:05 |
| weshay | trown, k k | 13:05 |
| trown | libvirt wont work, because multinic | 13:05 |
| hjensas | mandre: I guess pushing for that Heat change to be merged is what I should prio. | 13:06 |
| colonwq | fultonj, thanks. Why cannot everything work right the first time? | 13:06 |
| *** pblaho has quit IRC | 13:06 | |
| janki | shardy, hey...there are few comments on that TRIPLEO_CONFIG_HASH patch. we need to cherry-pick it too. its imp for update | 13:07 |
| *** toure_biab is now known as toure | 13:07 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for nova compute container to log to stdout/stderr https://review.openstack.org/510537 | 13:07 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: libvirt: Remove unnecessary binding of /var/log/containers/nova https://review.openstack.org/510561 | 13:07 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for nova-libvirt container to log to stdout/stderr https://review.openstack.org/510562 | 13:07 |
| *** tzumainn has joined #tripleo | 13:08 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Increase the Mistral RPC timeout https://review.openstack.org/509811 | 13:09 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Add release notes for TripleO-Validations patch. https://review.openstack.org/487744 | 13:09 |
| fultonj | colonwq: no, thank you for taking on this puppet patch | 13:09 |
| hrybacki | trown: what is the proper way to trigger a recheck of the zuul gate today? | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1718387 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721366 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1722228 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| openstack | Launchpad bug 1722228 in tripleo "pike ipv6 ping test is failing" [Critical,Triaged] - Assigned to John Trowbridge (trown) | 13:10 |
| *** ratailor_ has quit IRC | 13:10 | |
| *** dprince has joined #tripleo | 13:10 | |
| *** ratailor_ has joined #tripleo | 13:10 | |
| trown | hrybacki: typing 'recheck', but do not worry about "zuul -1" right now, only jenkins matters | 13:10 |
| hrybacki | trown: aye, it was a jenkins kick back but I think it wasn't related to the patchset | 13:11 |
| *** pblaho has joined #tripleo | 13:11 | |
| *** raildo has joined #tripleo | 13:12 | |
| * etingof is looking for advice from a wise owl... | 13:13 | |
| etingof | so at Ironic, we seem to tackle SELinux things via Puppet when setting up Ironic in the undercloud. We only do that via chcon which is not persistent. | 13:14 |
| etingof | I know there is the openstack-selinux package holding SELinux policies for OS projects... | 13:14 |
| etingof | Now, is there a reason why Ironic has no policy present in the openstack-selinux? | 13:14 |
| etingof | If I want to fix some selinux issue with Ironic, should I introduce Ironic policy to openstack-selinux? Or rather keep hacking on Puppet? | 13:15 |
| *** mdnadeem has quit IRC | 13:16 | |
| jaosorior | mwhahaha, jistr, mandre: could you guys take a look at this https://review.openstack.org/#/c/510001/ ? | 13:17 |
| *** skramaja has quit IRC | 13:18 | |
| *** janki has quit IRC | 13:18 | |
| jistr | ack, will try to read soonish | 13:18 |
| *** janki has joined #tripleo | 13:18 | |
| *** rbowen has quit IRC | 13:20 | |
| etingof | rhallisey ^ | 13:21 |
| *** rbowen has joined #tripleo | 13:22 | |
| rhallisey | etingof, I'd ask lon | 13:22 |
| EmilienM | hello | 13:24 |
| etingof | rhallisey, pardon my ignorance, what's lon's nick? | 13:25 |
| EmilienM | weshay: hey good morning, could you please tell why you re-open https://bugs.launchpad.net/tripleo/+bug/1721366 ? | 13:25 |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 13:25 |
| rhallisey | etingof, just lon | 13:25 |
| *** gbarros has joined #tripleo | 13:26 | |
| *** pchavva has joined #tripleo | 13:26 | |
| *** bobh has joined #tripleo | 13:27 | |
| matbu | mwhahaha: EmilienM if you are happy with that: https://review.openstack.org/510231 | 13:27 |
| matbu | feel free to vote :) | 13:27 |
| EmilienM | matbu: why 7.7.0 ? | 13:28 |
| matbu | EmilienM: why not ? :D | 13:28 |
| matbu | EmilienM: idk how the number should be increment | 13:28 |
| matbu | EmilienM: it adds a feature | 13:29 |
| EmilienM | matbu: in pike, we started with 7.0.0.0b1 https://github.com/openstack/releases/blob/master/deliverables/pike/tripleo-common.yaml#L8 | 13:29 |
| EmilienM | matbu: 7.X is pike | 13:29 |
| EmilienM | so we expect 8.X to be queens | 13:29 |
| matbu | EmilienM: hmm weird | 13:29 |
| matbu | master is 7 | 13:29 |
| openstackgerrit | Tom Barron proposed openstack/tripleo-heat-templates stable/newton: manila: set "host" to "hostgroup" https://review.openstack.org/508117 | 13:29 |
| EmilienM | matbu: yes because we didn't release since Pike final tag ;-) | 13:30 |
| weshay | EmilienM, the undercloud install was still failing. TBH it wasn't clear to me if it was a different issue than the one listed in the bug. It seemed to be failing the same way | 13:30 |
| matbu | EmilienM: ha yep make sense :) | 13:30 |
| weshay | same error.. so I reopened | 13:30 |
| weshay | I can open a new one if you think that is better | 13:30 |
| matbu | EmilienM: so yep, im going to fix that | 13:30 |
| EmilienM | weshay: ok, no problem. When that happens, please explain in Launchpad with a comment. Please do not just re-open it without explaining | 13:31 |
| EmilienM | matbu: cool. | 13:31 |
| EmilienM | weshay: also, please show me logs so I can take a look now | 13:31 |
| weshay | sshnaidm, re: https://review.openstack.org/#/c/509660/ I didn't reuse it network-env because that file is json | 13:31 |
| etingof | rhallisey, is there an openstack-selinux repo at openstack.org or it's all at GitHub? | 13:32 |
| rhallisey | etingof, https://github.com/redhat-openstack/openstack-selinux | 13:32 |
| *** abregman is now known as abregman|afk | 13:34 | |
| matbu | EmilienM: should be better now : 8.0.0 | 13:34 |
| EmilienM | matbu: 8.0.0 will be the final tag when we release Queens | 13:34 |
| EmilienM | matbu: let me ask on #openstack-release what's the best tag but 8.0.0 is not what we want I think | 13:35 |
| EmilienM | matbu: let me ask | 13:35 |
| *** ratailor__ has joined #tripleo | 13:35 | |
| *** achadha has joined #tripleo | 13:36 | |
| *** ratailor_ has quit IRC | 13:37 | |
| openstackgerrit | yatin proposed openstack/instack-undercloud master: Use keystone v3 session with novaclient https://review.openstack.org/510535 | 13:38 |
| *** ratailor_ has joined #tripleo | 13:40 | |
| *** bnemec has joined #tripleo | 13:40 | |
| *** achadha has quit IRC | 13:40 | |
| *** janki has quit IRC | 13:42 | |
| matbu | EmilienM: so it should be 8.0.0.b1 ? (if i follow correctly) | 13:43 |
| EmilienM | matbu: maybe, wait a sec | 13:43 |
| *** ratailor__ has quit IRC | 13:43 | |
| matbu | 8.0.0.0b1 (sorry) | 13:43 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart master: DNM TESTING pike promote https://review.openstack.org/510574 | 13:44 |
| openstackgerrit | Merged openstack/tripleo-common master: Add module_path as option for ansible-playbook action https://review.openstack.org/510109 | 13:46 |
| owalsh | jaosorior: just looking at the novajoin issue ratailor_ is hitting... | 13:46 |
| owalsh | jaosorior: this is the fix yea - https://review.openstack.org/408783 | 13:47 |
| Tengu | and who's deploying again an overcloud? :D | 13:47 |
| EmilienM | weshay: I'm investigating but we've made progress, it's now somewhere else further in the steps | 13:48 |
| *** suuuper has quit IRC | 13:48 | |
| owalsh | jaosorior: or is it https://review.openstack.org/502130 | 13:48 |
| EmilienM | weshay: it's in instack_undercloud codebase where we call novaclient | 13:48 |
| EmilienM | jaosorior: I might use your help when you have time | 13:48 |
| *** psachin has quit IRC | 13:48 | |
| jaosorior | owalsh: it's the last one. | 13:49 |
| jaosorior | owalsh: just use the lastest :D | 13:49 |
| jaosorior | EmilienM: what's up? | 13:49 |
| owalsh | jaosorior: ack, thanks | 13:49 |
| EmilienM | jaosorior: I think some code in instack-undercloud still use keystone v2: https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/724f50b/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-10-07_16_28_01 | 13:49 |
| *** etingof has quit IRC | 13:50 | |
| EmilienM | jaosorior: probably v3 context is missing when getting a token | 13:50 |
| jaosorior | EmilienM: are you talking about this https://review.openstack.org/#/c/510535/ ? | 13:50 |
| EmilienM | jaosorior: in https://github.com/openstack/instack-undercloud/blob/master/instack_undercloud/undercloud.py#L1551-L1562 | 13:51 |
| EmilienM | jaosorior: let me click | 13:51 |
| EmilienM | jaosorior: exactly :D | 13:51 |
| EmilienM | ahah I missed it | 13:51 |
| EmilienM | weshay: fyi https://review.openstack.org/#/c/510535/ | 13:51 |
| chandankumar | EmilienM: tripleo scenario2 container tempest volume encryptiontest is failing https://review.openstack.org/#/c/499928/ need some helping hand on this | 13:51 |
| Tengu | small question: is there any example on OVS monitoring through SNMP? | 13:52 |
| EmilienM | chandankumar: what is failing? | 13:52 |
| *** lon has joined #tripleo | 13:53 | |
| jaosorior | EmilienM: neutron it seemed. | 13:53 |
| chandankumar | EmilienM: http://logs.openstack.org/28/499928/15/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/55297dd/logs/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-10-04_09_19_30 | 13:53 |
| lon | whoops, missed etingof - can someone paste me what was going on? | 13:53 |
| lvdombrkr | folks, is there any documentation how to upgrade from ocata to pike? | 13:54 |
| *** aditya_r has joined #tripleo | 13:54 | |
| *** aditya_r has quit IRC | 13:54 | |
| chandankumar | EmilienM: one more thing whom can i catch for starting tempest containerization work on undercloud? | 13:54 |
| EmilienM | lvdombrkr: it's in progress by matbu https://review.openstack.org/#/c/496223/ | 13:54 |
| *** masco has quit IRC | 13:54 | |
| EmilienM | chandankumar: I'll look shortly at the logs | 13:55 |
| chandankumar | EmilienM: thanks :-) | 13:55 |
| EmilienM | chandankumar: not sure who tbh | 13:55 |
| chandankumar | EmilienM: i can work on that, just need guidance where to change to enable tempest container and expose tempest cli on undercloud from containerized tempest | 13:56 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/newton: Allow to configure snmpd_config https://review.openstack.org/510218 | 13:56 |
| *** jpena|lunch is now known as jpena | 13:56 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Special treatment for os-net-config upgrade. https://review.openstack.org/510577 | 13:56 |
| lvdombrkr | EmilienM: thenks i will look into | 13:58 |
| openstackgerrit | Javier Peña proposed openstack/tripleo-quickstart-extras master: Allow pre-installed DLRN https://review.openstack.org/499117 | 13:59 |
| EmilienM | chandankumar: I would ask on mailing-list, I have no clue. | 13:59 |
| chandankumar | EmilienM: sure will drop an email | 13:59 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 14:00 |
| *** links has quit IRC | 14:03 | |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates master: WIP: Set file ACLs for Ceph keyrings for non-containerized deployment https://review.openstack.org/509020 | 14:04 |
| *** morazi has quit IRC | 14:05 | |
| *** ykarel is now known as ykarel|away | 14:07 | |
| *** etingof has joined #tripleo | 14:07 | |
| *** chlong has joined #tripleo | 14:07 | |
| *** morazi has joined #tripleo | 14:07 | |
| *** zaneb has joined #tripleo | 14:08 | |
| openstackgerrit | Merged openstack/tripleo-common master: Take wwn_with_extension into account, when configuring a boot device https://review.openstack.org/508865 | 14:08 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common stable/pike: Take wwn_with_extension into account, when configuring a boot device https://review.openstack.org/510583 | 14:09 |
| *** ratailor__ has joined #tripleo | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721366 | 14:10 |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,In progress] - Assigned to yatin (yatinkarel) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1722228 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1722228 in tripleo "pike ipv6 ping test is failing" [Critical,Triaged] - Assigned to John Trowbridge (trown) | 14:10 |
| *** ratailor__ has quit IRC | 14:11 | |
| *** jaganathan has quit IRC | 14:12 | |
| *** ratailor_ has quit IRC | 14:13 | |
| *** panda|bbl is now known as panda | 14:16 | |
| *** lon has left #tripleo | 14:17 | |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Allow more than one order rule between two resources. https://review.openstack.org/510196 | 14:20 |
| *** jdennis has joined #tripleo | 14:21 | |
| *** chlong_ has joined #tripleo | 14:21 | |
| openstackgerrit | Thomas Herve proposed openstack/instack-undercloud master: Run the swift object expirer in the undercloud https://review.openstack.org/510587 | 14:23 |
| *** dpawar has joined #tripleo | 14:26 | |
| *** spectr has quit IRC | 14:30 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates stable/pike: DNM, testing only https://review.openstack.org/510589 | 14:30 |
| *** sid1 has quit IRC | 14:32 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server https://review.openstack.org/510362 | 14:34 |
| *** akane has quit IRC | 14:34 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Containerized Fluentd client https://review.openstack.org/507506 | 14:34 |
| derekh | Anybody know why gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 is busted on stable/pike ? | 14:35 |
| mwhahaha | yes | 14:36 |
| mwhahaha | would you like to know why? :D | 14:36 |
| *** achadha has joined #tripleo | 14:37 | |
| mwhahaha | https://review.openstack.org/#/c/510234/ | 14:37 |
| *** archit has joined #tripleo | 14:40 | |
| *** achadha has quit IRC | 14:41 | |
| *** morazi has quit IRC | 14:50 | |
| *** morazi has joined #tripleo | 14:51 | |
| derekh | mwhahaha: thanks, will add a depends on to see if it work for me | 14:52 |
| derekh | hmm, its heat will that work... | 14:53 |
| mwhahaha | it might | 14:53 |
| * derekh give it a go | 14:53 | |
| *** cshastri has quit IRC | 14:53 | |
| *** dpawar has quit IRC | 14:54 | |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-heat-templates stable/pike: Add IronicPxe to the default controller https://review.openstack.org/507981 | 14:55 |
| *** links has joined #tripleo | 14:56 | |
| *** aditya_r has joined #tripleo | 14:57 | |
| slagle | EmilienM: have you been able to get traas working with multiple patches in zuul_changes? | 14:57 |
| EmilienM | slagle: I tried and failed, when you submit a comma-separated list, it fails to parse after the first item | 14:58 |
| *** adarazs_off is now known as adarazs | 14:59 | |
| slagle | EmilienM: in the upstream jobs, it's a ^ separated list. but that doesnt seem to get parsed correctly either | 14:59 |
| EmilienM | slagle: can you paste what you tried? | 15:00 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Explicitly list Apache License as 2.0 https://review.openstack.org/510598 | 15:00 |
| EmilienM | slagle: openstack/tripleo-quickstart-extras:master:refs/changes/06/508306/13^openstack/tripleo-heat-templates:master:refs/changes/27/505827/12^openstack/tripleo-common:master:refs/changes/89/508189/7^openstack/tripleo-quickstart:master:refs/changes/07/508307/10 | 15:01 |
| EmilienM | from CI http://logs.openstack.org/07/508307/10/check/gate-tripleo-ci-centos-7-containers-multinode/6c40598/console.html#_2017-10-06_22_00_10_302562 | 15:01 |
| slagle | EmilienM: yea i had soemthing similar: http://paste.openstack.org/show/623116/ | 15:02 |
| slagle | i wonder if Heat is munging the parameter value somehow | 15:02 |
| openstackgerrit | Tim Rozet proposed openstack/os-net-config master: Fix licenses to be explicit with Apache 2.0 https://review.openstack.org/510599 | 15:04 |
| slagle | or does ^ mean something special in yaml. maybe the value should be a scalar | 15:04 |
| EmilienM | slagle: I just re-ran a traas with: http://paste.openstack.org/show/KDEIdwVOdYb5l7oP0sSR/ | 15:04 |
| slagle | EmilienM: in other news, i think i know why the upstream job is failing after NetworkDeployment gets applied | 15:05 |
| *** dmarlin has quit IRC | 15:06 | |
| slagle | EmilienM: ansible is trying to reuse the ssh control socket | 15:06 |
| slagle | and it will no longer work since networking was bounced on the remote node | 15:06 |
| openstackgerrit | Tim Rozet proposed openstack/puppet-tripleo master: Fixes license to explicitly be Apache 2.0 https://review.openstack.org/510600 | 15:06 |
| *** achadha has joined #tripleo | 15:08 | |
| EmilienM | slagle: nice catch | 15:09 |
| *** achadha has quit IRC | 15:09 | |
| *** achadha has joined #tripleo | 15:10 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721366 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1722228 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,In progress] - Assigned to yatin (yatinkarel) | 15:10 |
| openstack | Launchpad bug 1722228 in tripleo "pike ipv6 ping test is failing" [Critical,Triaged] - Assigned to John Trowbridge (trown) | 15:10 |
| EmilienM | bnemec, mwhahaha: re: ocata/upgrade job: the job is failing during pingtest *after* upgrade - which is not too bad. I'm investigating now http://logs.openstack.org/16/510216/1/check/gate-tripleo-ci-centos-7-multinode-upgrades/681c19f/console.html#_2017-10-07_03_44_25_496390 | 15:11 |
| mwhahaha | probably because we never got to that part previously :D | 15:11 |
| EmilienM | mwhahaha: yes we did | 15:11 |
| mwhahaha | oh wait scenario*s were timing out | 15:12 |
| EmilienM | mwhahaha: http://logs.openstack.org/16/510216/1/check/gate-tripleo-ci-centos-7-multinode-upgrades/681c19f/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-10-07_03_40_45_866 | 15:12 |
| sdoran | I don't believe ^ is special to YAML. Should be passed along correctly. | 15:12 |
| EmilienM | mwhahaha: ahah! it should maybe work now | 15:12 |
| mwhahaha | EmilienM: yea i thought that error looked familar | 15:12 |
| *** trown is now known as trown|outtypewww | 15:12 | |
| sdoran | Is the issue Ansible is failing after the host gets bounced for updates and it needs a new control socket? Trying to catch up... | 15:15 |
| sdoran | Because you can update a system mid-play and keep on going after it comes up, but it can be a bit tricky. | 15:16 |
| sdoran | s/update/reboot | 15:16 |
| trozet | mwhahaha: any word on the zuul failures? | 15:17 |
| mwhahaha | trozet: we're still working on v3 | 15:17 |
| trozet | mwhahaha: so what does it mean for the time being? Force verify patches? | 15:18 |
| trozet | mwhahaha: shows -1 even though they all pass | 15:18 |
| mwhahaha | trozet: no the zuul -1 doesn't block | 15:18 |
| mwhahaha | jenkins is the one that matters at teh moment | 15:18 |
| trozet | mwhahaha: oh. Well in that case (they all passed anyway) can you review https://review.openstack.org/#/c/509789/ please | 15:19 |
| mwhahaha | trozet: i already did? | 15:19 |
| trozet | mwhahaha: sorry wrong link https://review.openstack.org/#/c/509834/ | 15:20 |
| mwhahaha | trozet: EmilienM is -1 | 15:20 |
| trozet | mwhahaha: yeah going to ping him next ;) | 15:21 |
| * EmilienM hides | 15:21 | |
| * EmilienM it's thanksgiving here, I'm not ehre | 15:21 | |
| *** chlong has quit IRC | 15:22 | |
| trozet | mwhahaha: see hes not even here we can just ignore his vote :) | 15:22 |
| *** chlong_ has quit IRC | 15:22 | |
| mwhahaha | pffft | 15:22 |
| mwhahaha | gimme a few | 15:22 |
| trozet | mwhahaha: haha I had responded to his comment. It doesnt really matter what that string is, but let me knwo what you think | 15:23 |
| mwhahaha | trozet: so it seems like None is the better value? | 15:24 |
| * mwhahaha shrugs | 15:25 | |
| *** milan has quit IRC | 15:25 | |
| mwhahaha | you tested that it works with false right? | 15:25 |
| *** ed_b has joined #tripleo | 15:26 | |
| trozet | mwhahaha: Rhys did. Does something make you think it wouldnt? | 15:29 |
| mwhahaha | we don't have CI, so as long as it's been tested | 15:29 |
| mwhahaha | setting arbitrary values is always awkward :D | 15:29 |
| trozet | mwhahaha: yeah working on getting ODL CI working | 15:30 |
| trozet | mwhahaha: i can change the ODL CI patch to depend on this one and we can see if it finally passes :) | 15:31 |
| mwhahaha | i'm ok if it's been validated | 15:31 |
| EmilienM | mwhahaha: 'False' is weird | 15:31 |
| EmilienM | but ok | 15:31 |
| mwhahaha | it is weird, but that's an ODL thing not a OOO issue | 15:32 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: docker: add logging(source & groups) https://review.openstack.org/507952 | 15:32 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment https://review.openstack.org/486905 | 15:32 |
| trozet | lets see what happens^ | 15:33 |
| *** rhallisey_ has quit IRC | 15:36 | |
| *** ffiore_ has joined #tripleo | 15:36 | |
| *** ffiore has quit IRC | 15:38 | |
| *** tbarron is now known as tbarron|PTO | 15:39 | |
| *** pblaho has quit IRC | 15:39 | |
| *** liverpooler has quit IRC | 15:41 | |
| sshnaidm | mwhahaha, hi, about https://review.openstack.org/#/c/508660/ - I can't spend time on refactoring functions right now, but I'd like to have it merged to test readiness for zuul v3. Are you fine with creating techdebt card about it in our CI trello and then somebody will pick it up? | 15:43 |
| mwhahaha | not really | 15:44 |
| mwhahaha | but whatever | 15:44 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Add all services to container scenarios https://review.openstack.org/501872 | 15:44 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Switch scenario004-containers to use ceph-ansible https://review.openstack.org/507526 | 15:44 |
| mwhahaha | if somebody had a name, maybe | 15:44 |
| mwhahaha | but it sounds like every other time we point out much we duplicate things and say oh we'll fix it later | 15:44 |
| EmilienM | slagle: hey I think it worked (the zuul_changes) | 15:45 |
| *** jpena is now known as jpena|brb | 15:45 | |
| EmilienM | slagle: I'm installing the undercloud now | 15:45 |
| slagle | EmilienM: i'm not sure it will work | 15:46 |
| slagle | EmilienM: how can oooq test patches against itself? | 15:46 |
| slagle | EmilienM: i dont think it's smart enough to re-exec itself. maybe i'm wrong though | 15:47 |
| slagle | EmilienM: so i think we would have to add support to traas for that if we wanted to test oooq patches specifically | 15:47 |
| EmilienM | slagle: true, oooq and oooq-extras didn't checkout :( | 15:47 |
| EmilienM | no you're right | 15:47 |
| *** catintheroof has quit IRC | 15:48 | |
| EmilienM | slagle: bah, even wasn't correctly deployed with the right zuul change :( | 15:48 |
| EmilienM | slagle: oh wait I found why | 15:49 |
| EmilienM | slagle: for tripleo-common | 15:50 |
| EmilienM | slagle: we don't ship tripleo_common/templates/ | 15:50 |
| slagle | EmilienM: that should get picked up automatically by the python packaging | 15:50 |
| slagle | EmilienM: it's working in the upstream job | 15:51 |
| EmilienM | slagle: ah right, it's in tripleo_common | 15:51 |
| EmilienM | slagle: in my env, none of the zuul_changes worked | 15:52 |
| slagle | EmilienM: right. same as I saw last week | 15:52 |
| slagle | it's not parsed correctly. the value must be invalid or munged by heat perhpas | 15:52 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: snmp: add SnmpdBindHost parameter https://review.openstack.org/510217 | 15:53 |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-common master: Don't wait for stack in progress in delete https://review.openstack.org/510612 | 15:55 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add timestamp output filter https://review.openstack.org/474830 | 15:56 |
| *** dpawar has joined #tripleo | 15:57 | |
| *** dpawar has quit IRC | 15:58 | |
| *** ebarrera has quit IRC | 15:58 | |
| *** dpawar has joined #tripleo | 15:58 | |
| *** egonzalez has quit IRC | 15:58 | |
| *** gbarros has quit IRC | 16:00 | |
| *** achadha_ has joined #tripleo | 16:00 | |
| *** marios has quit IRC | 16:02 | |
| *** dpawar has quit IRC | 16:03 | |
| *** yprokule has quit IRC | 16:03 | |
| *** achadha has quit IRC | 16:04 | |
| *** achadha_ has quit IRC | 16:05 | |
| *** thrash is now known as thrash|biab | 16:06 | |
| *** jlinkes has quit IRC | 16:08 | |
| *** dmarlin has joined #tripleo | 16:09 | |
| openstackgerrit | Merged openstack/tripleo-common master: Enhances Roles List https://review.openstack.org/508514 | 16:09 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721366 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1722228 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,In progress] - Assigned to yatin (yatinkarel) | 16:10 |
| openstack | Launchpad bug 1722228 in tripleo "pike ipv6 ping test is failing" [Critical,Triaged] - Assigned to John Trowbridge (trown) | 16:10 |
| *** gbarros has joined #tripleo | 16:11 | |
| *** ffiore_ has quit IRC | 16:12 | |
| therve | shardy, Do you know if the stack delete workflow is tested? | 16:13 |
| *** jpena|brb is now known as jpena | 16:15 | |
| *** chlong_ has joined #tripleo | 16:16 | |
| *** udesale has quit IRC | 16:16 | |
| *** chlong has joined #tripleo | 16:16 | |
| shardy | therve: at one point it was, but I think it was disabled due to lack of CI walltime | 16:19 |
| shardy | therve: certainly adding coverage for the overcloud delete command would be good | 16:19 |
| shardy | as I think previously we just proved the heat stack could be deleted | 16:20 |
| *** lucasagomes is now known as lucas-afk | 16:21 | |
| *** dparkes has quit IRC | 16:21 | |
| *** nyechiel_ has joined #tripleo | 16:21 | |
| sshnaidm | mwhahaha, https://trello.com/c/uUZ0SI5p/378-make-nodepool-files-handling-more-generic-for-zuul-v3 | 16:24 |
| mwhahaha | sshnaidm: please create a login bug | 16:25 |
| mwhahaha | Err lp | 16:25 |
| sshnaidm | mwhahaha, login bug..? | 16:25 |
| mwhahaha | Stupid auto correct | 16:25 |
| sshnaidm | mwhahaha, mm.. it's not a bug, it's more rfe | 16:25 |
| *** ykarel|away is now known as ykarel | 16:26 | |
| mwhahaha | We don't use Trello to track upstream tripleo work. It needs to even accounted for | 16:26 |
| mwhahaha | Tech-debt is not an rfe | 16:26 |
| mwhahaha | For visibility it need to be in launch pad | 16:27 |
| sshnaidm | mwhahaha, I thought for CI stuff we use more CI trello, but whatever.. | 16:28 |
| sshnaidm | mwhahaha, the problem was solved, but the solution could be improved - is it still a bug? | 16:28 |
| mwhahaha | You can use that for your squad but for the project as a whole it needs to be in launchpad | 16:28 |
| sshnaidm | mwhahaha, ok, np | 16:28 |
| EmilienM | mwhahaha: I plan to release newton/ocata/pike - waiting for some merges today | 16:29 |
| *** milan has joined #tripleo | 16:29 | |
| mwhahaha | Bug: we duplicate the usage of zuul files in tripleo CI that leads to problems when the location changes | 16:29 |
| *** fragatina has joined #tripleo | 16:29 | |
| mwhahaha | Duplication of code is a bug. It can be medium bug but is still a bug | 16:30 |
| sshnaidm | mwhahaha, ok | 16:30 |
| sshnaidm | mwhahaha, https://bugs.launchpad.net/tripleo/+bug/1722344 | 16:30 |
| openstack | Launchpad bug 1722344 in tripleo "Make nodepool files handling more generic for Zuul v3" [Undecided,Triaged] | 16:30 |
| jaosorior | mwhahaha, EmilienM: well, isn't it the problem that we can't merge stuff in stable/ocata cause the upgrade gate is timing out? | 16:30 |
| EmilienM | jaosorior: nope, I haven't seen timeouts lately or I'm wrong | 16:30 |
| jaosorior | mwhahaha, EmilienM: been trying to merge this one for a while https://review.openstack.org/#/c/494947/ | 16:31 |
| jaosorior | at lesat I've seen a bunch of timeouts in stable/ocata | 16:31 |
| EmilienM | jaosorior: let's look at logs | 16:31 |
| *** pcaruana has quit IRC | 16:32 | |
| EmilienM | jaosorior: it wasn't failing on upgrade job | 16:32 |
| *** jaosorior has quit IRC | 16:33 | |
| *** jaosorior has joined #tripleo | 16:34 | |
| *** jpich has quit IRC | 16:34 | |
| mwhahaha | this is why we shouldn't leave broken ci to fester | 16:36 |
| mwhahaha | because it was timeouts but then manifested itself as a new failure that gets assumed timeouts | 16:36 |
| *** jaosorior has quit IRC | 16:38 | |
| *** jaosorior has joined #tripleo | 16:40 | |
| jaosorior | EmilienM: sorry, my internet is failing :/ | 16:41 |
| mwhahaha | jaosorior: it's not timeouts | 16:41 |
| jaosorior | mwhahaha: isn't it? | 16:41 |
| mwhahaha | elastic recheck would report timeouts | 16:41 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server https://review.openstack.org/510362 | 16:41 |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates master: WIP: Set file ACLs for Ceph keyrings for non-containerized deployment https://review.openstack.org/509020 | 16:41 |
| *** aufi has quit IRC | 16:42 | |
| jaosorior | mwhahaha: the failure in gate-tripleo-ci-centos-7-scenario001-multinode-upgrades sure seems like a timeout | 16:42 |
| EmilienM | jaosorior: gate-tripleo-ci-centos-7-scenario001-multinode-upgrades never worked I think | 16:42 |
| EmilienM | jaosorior: but I might be wrong | 16:43 |
| mwhahaha | that one is always timeintout | 16:43 |
| mwhahaha | we need ot -nv the scenarios | 16:43 |
| mwhahaha | http://logs.openstack.org/47/494947/6/check/gate-tripleo-ci-centos-7-multinode-upgrades/7b9093d/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-10-07_10_42_30_089 | 16:43 |
| mwhahaha | that one should work but was failing because of machine-id | 16:43 |
| jaosorior | EmilienM: it's voting | 16:43 |
| mwhahaha | un-related to patch but was failing | 16:43 |
| jaosorior | EmilienM: when do you plan to release updates for newton and ocata? | 16:44 |
| EmilienM | jaosorior: ah this one sorry yes it should work | 16:44 |
| EmilienM | jaosorior: /me was confused | 16:44 |
| EmilienM | jaosorior: this week | 16:44 |
| jaosorior | ok | 16:44 |
| EmilienM | jaosorior: but we do it every 2 weeks | 16:44 |
| mwhahaha | EmilienM: can you propose switching scenario upgrades to nv | 16:44 |
| jaosorior | need to backport that patch to newton too | 16:44 |
| * mwhahaha is over those jobs | 16:45 | |
| EmilienM | mwhahaha: me? oh noe | 16:45 |
| *** zaneb has quit IRC | 16:45 | |
| mwhahaha | oh wait is project-config still frozen | 16:45 |
| mwhahaha | EmilienM: yea you mr. project-config | 16:46 |
| EmilienM | I was | 16:46 |
| EmilienM | until the file reached 50 000 LOC | 16:46 |
| mwhahaha | :o | 16:46 |
| EmilienM | my editor crash now | 16:46 |
| mwhahaha | lol | 16:46 |
| Tengu | hello! | 16:46 |
| Tengu | small question: where do the logs from that class are written? https://github.com/openstack/tripleo-common/blob/78d030cab12a46f5aed0e0cacbaf961edf9e8de0/tripleo_common/filters/capabilities_filter.py | 16:47 |
| mwhahaha | Tengu: nova-scheduler i think | 16:47 |
| Tengu | hmm. | 16:47 |
| Tengu | weird. | 16:47 |
| Tengu | no string matches in that log :/ | 16:47 |
| *** mcornea has quit IRC | 16:48 | |
| Tengu | I also thought it had to go somewhere in nova, but… grep doesn't show anything. As usual I get issues with host matching -.- | 16:48 |
| *** liverpooler has joined #tripleo | 16:48 | |
| *** rcernin has quit IRC | 16:49 | |
| *** jaosorior has quit IRC | 16:53 | |
| *** stendulker has joined #tripleo | 16:54 | |
| mwhahaha | EmilienM: do any of the scenario upgrade jobs (non-containers) work anywhere? or should i just switch them to -nv | 16:57 |
| *** links has quit IRC | 16:58 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/510408 | 16:58 |
| EmilienM | mwhahaha: they should work on ocata | 16:58 |
| mwhahaha | k so should i just switch 001? | 16:59 |
| mwhahaha | that one consistently timesout | 16:59 |
| *** derekh has quit IRC | 17:01 | |
| EmilienM | mwhahaha: I guess you can switch them all | 17:01 |
| EmilienM | unless you see other passing | 17:01 |
| mwhahaha | i believe the plan is to move them all to RDO 3rd party soon | 17:01 |
| mwhahaha | weshay: what's the timeline on 3rd partying the upgrade jobs? | 17:01 |
| Tengu | (still digging in my node placement failure) we're OK that this log line points to the former (in)famous capabilities_filter that searches for host based on the <noderole>SchedulerHints, right? "Filter TripleOCapabilitiesFilter returned 0 hosts" | 17:01 |
| *** achadha has joined #tripleo | 17:01 | |
| EmilienM | mwhahaha: that's the plan iiuc | 17:01 |
| * mwhahaha pulls out http://my1.fr/files/emilien-right-now.jpg to work on project-config | 17:02 | |
| weshay | mwhahaha, my change merged, but the jobs are not registered. Which I have to catch up on why that is.. but asap | 17:02 |
| *** achadha has quit IRC | 17:06 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Take all mounted config_volumes into account https://review.openstack.org/510632 | 17:06 |
| rook | shardy: ping - the convergence work... | 17:09 |
| rook | shardy: what all needs to be enabled, just : convergence_engine=true | 17:09 |
| *** fragatina has quit IRC | 17:09 | |
| mwhahaha | rook: there's an undercloud change that has it | 17:09 |
| *** sshnaidm is now known as sshnaidm|off | 17:10 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721366 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1722228 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,In progress] - Assigned to yatin (yatinkarel) | 17:10 |
| openstack | Launchpad bug 1722228 in tripleo "pike ipv6 ping test is failing" [Critical,Triaged] - Assigned to John Trowbridge (trown) | 17:10 |
| rook | mwhahaha: huh? | 17:10 |
| mwhahaha | rook: https://review.openstack.org/#/c/499283/ | 17:10 |
| rook | to enable the convergence work? | 17:10 |
| rook | ok | 17:10 |
| mwhahaha | to enable it on the undercloud | 17:10 |
| mwhahaha | so you can see what it takes | 17:10 |
| shardy | rook: yeah you can just toggle convergence_engine in heat.conf, or use the undercloud patch mwhahaha mentioned | 17:10 |
| rook | alright, just setting the convergence_engine=true seems like all there is needed | 17:11 |
| rook | shardy: ^ | 17:11 |
| shardy | rook: it's true by default in recent heat releases, but we force it to false in instack-undercloud | 17:11 |
| rook | roger. | 17:11 |
| EmilienM | sshnaidm|off: what are blockers to run zuul v3 legacy jobs? | 17:11 |
| rook | ok | 17:11 |
| rook | kicking off the run | 17:11 |
| shardy | rook: yup that's fine provided you don't then re-run the undercloud install | 17:11 |
| EmilienM | sshnaidm|off: just https://review.openstack.org/#/c/508660/ ? | 17:11 |
| mwhahaha | hrm it seems that the scenario upgrade jobs are not in v3 | 17:13 |
| mwhahaha | or are hiding | 17:14 |
| *** athomas has quit IRC | 17:16 | |
| EmilienM | mwhahaha: https://review.openstack.org/#/c/510215/ | 17:16 |
| EmilienM | mwhahaha: see, upgrade job passed lol | 17:16 |
| EmilienM | SUCCESS in 2h 56m 56s | 17:16 |
| EmilienM | no timeout anymore? nice :D | 17:16 |
| mwhahaha | multinode-upgrades is fine | 17:16 |
| mwhahaha | scenario* always fail | 17:17 |
| mwhahaha | i want to leave multinode-upgrades, but -nv the scenarios | 17:17 |
| mwhahaha | but this ci config makes me cry | 17:17 |
| *** jpena is now known as jpena|away | 17:17 | |
| *** shardy has quit IRC | 17:18 | |
| *** gbarros has quit IRC | 17:22 | |
| *** zaneb has joined #tripleo | 17:22 | |
| *** dbecker has quit IRC | 17:27 | |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart-extras master: Disruption detection and some stack update roles https://review.openstack.org/497950 | 17:28 |
| *** amoralej is now known as amoralej|off | 17:29 | |
| *** jtomasek has quit IRC | 17:30 | |
| *** tongl has joined #tripleo | 17:30 | |
| *** jtomasek has joined #tripleo | 17:31 | |
| Tengu | mwhahaha: ah, I'm not the only one having some bad mood with something related to openstack then :'( | 17:31 |
| Tengu | we can create some club, with beers and snacks. | 17:31 |
| mwhahaha | pretty much | 17:31 |
| mwhahaha | :D | 17:31 |
| Tengu | really, the node placement is a pain. | 17:31 |
| Tengu | … I think I'll replace my beer with some whisky in fact. | 17:32 |
| *** salmankhan has quit IRC | 17:37 | |
| Tengu | pfff. | 17:49 |
| Tengu | no way it's working two times the same, this "advanced placement" :( | 17:49 |
| Tengu | oh. | 17:52 |
| Tengu | maybe I have something, will go to nova chan. | 17:52 |
| *** aditya_r has quit IRC | 17:52 | |
| *** thrash|biab is now known as thrash | 17:53 | |
| *** gbarros has joined #tripleo | 17:55 | |
| Tengu | does anyone understand that log entry? Cannot attach VIF 66511cf9-167a-4312-b599-ca8765467c5d to the node 4aa07b6d-ccf0-4f2b-8938-e20b07ff0156 due to error: Unable to attach VIF 66511cf9-167a-4312-b599-ca8765467c5d, not enough free physical ports. | 17:59 |
| Tengu | it's n nova-compute.log, when I try to add new hosts into the overcloud - and, funnily, the deploy fails because it can't find the new hosts… | 17:59 |
| *** leitan has quit IRC | 18:02 | |
| *** achadha has joined #tripleo | 18:02 | |
| *** leitan has joined #tripleo | 18:02 | |
| openstackgerrit | Wojciech Dec proposed openstack/puppet-tripleo master: Adding manifest for Cisco VTS ML2 mechanism driver configuration https://review.openstack.org/510645 | 18:04 |
| *** tosky__ has joined #tripleo | 18:05 | |
| *** dtantsur is now known as dtantsur|afk | 18:06 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Remove package if service stopped and disabled https://review.openstack.org/510545 | 18:06 |
| *** achadha has quit IRC | 18:06 | |
| *** leitan has quit IRC | 18:07 | |
| *** fandrieu has quit IRC | 18:08 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: overcloud-deploy: add config-download + ansible run feature https://review.openstack.org/508306 | 18:08 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721366 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1722228 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,In progress] - Assigned to yatin (yatinkarel) | 18:10 |
| openstack | Launchpad bug 1722228 in tripleo "pike ipv6 ping test is failing" [Critical,Triaged] - Assigned to John Trowbridge (trown) | 18:10 |
| *** pcaruana has joined #tripleo | 18:11 | |
| *** stendulker has quit IRC | 18:11 | |
| EmilienM | slagle: why don't we have the ssh issue in our local traas? | 18:11 |
| slagle | EmilienM: i think we probably would in a clean environment | 18:13 |
| EmilienM | ok | 18:13 |
| *** ykarel has quit IRC | 18:14 | |
| slagle | EmilienM: i'm working on a patch now | 18:14 |
| *** dprince has quit IRC | 18:16 | |
| *** pkovar has quit IRC | 18:18 | |
| *** dciabrin has quit IRC | 18:19 | |
| *** dciabrin has joined #tripleo | 18:20 | |
| *** sid1 has joined #tripleo | 18:21 | |
| rook | ugh shardy left... mwhahaha do you know how one would check if convergence engine is running? I have it enabled in the config, and rebooted heat... | 18:26 |
| rook | but I wanted a way to validate | 18:26 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart-extras master: overcloud-deploy: add config-download + ansible run feature https://review.openstack.org/508306 | 18:28 |
| mwhahaha | rook: i do not | 18:28 |
| mwhahaha | rook: bnemec might know | 18:28 |
| bnemec | https://cdn.meme.am/instances/250x250/58284552/i-know-nothing.jpg | 18:29 |
| bnemec | I actually don't know for sure. | 18:30 |
| bnemec | The only way I've been able to tell the difference so far is that the dynamic inventory is a lot faster with convergence stacks. | 18:31 |
| *** leitan has joined #tripleo | 18:31 | |
| bnemec | rook: You probably need to talk to the Heat folks about that. | 18:31 |
| rook | 2017-10-09 17:05:15.973 30844 DEBUG oslo_service.service [-] convergence_engine = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2879 | 18:32 |
| rook | i see that in the log | 18:32 |
| rook | so success? | 18:32 |
| rook | :P | 18:32 |
| bnemec | Unless something is horribly wrong in Heat, I would think that should do it. | 18:32 |
| rook | 2017-10-09 17:51:25.910 30879 INFO heat.engine.stack [req-d7509c08-198f-4e96-9e58-069d53d94322 - admin - default default] convergence_dependencies: {(123486, True): {(123487, True)}, (123487, True): {}} | 18:33 |
| rook | interesting | 18:33 |
| slagle | i think you can tell in the Heat db too. there is a "convergence" column on the stack table | 18:33 |
| *** nyechiel_ has quit IRC | 18:40 | |
| rook | bnemec, slagle ok, my understanding is that convergence could help with bad nodes, or nodes that could hold up a overcloud deploy? | 18:48 |
| rook | I am curious if it uses a set of rules in order to validate things? | 18:48 |
| *** dprince has joined #tripleo | 18:49 | |
| slagle | rook: if you mean still continue the deploy if just a few nodes are bad...i don't think it will do that | 18:49 |
| rook | slagle: ah ok https://specs.openstack.org/openstack/heat-specs/specs/juno/convergence.html reading through I got that sense. | 18:50 |
| slagle | rook: well, i think it would probably allow nodes that are already in progress, or other members of the ResourceGroup to finish completing | 18:50 |
| slagle | rook: but i don't see it continuing on to the deploy steps after that | 18:51 |
| slagle | and if it did, it could lead to horrible brokeness, so we would actually not want that, except perhaps for certain roles (e.g., computes) | 18:51 |
| slagle | but Heat has no way of knowing that | 18:51 |
| trozet | dsneddon: do you mind reviewing this? https://review.openstack.org/#/c/509789/ | 18:52 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-docs master: WIP: Add documentation for TripleO/RDO Pipelines https://review.openstack.org/510657 | 18:54 |
| dsneddon | trozet, So, I've actually tried to review this a couple of times. I'm concerned because I am pretty sure that there were places in the templates where we referenced the IP address from the port resource and not from {{role.name}}IpListMap. | 18:59 |
| bnemec | rook: My understanding is that the major benefit of convergence is the ability to run multiple updates concurrently. | 18:59 |
| dsneddon | trozet, Have you tested this with a real deployment? | 19:00 |
| trozet | dsneddon: yeah i did | 19:00 |
| dsneddon | trozet, I wanted to, but I'm having trouble with my oooq right now. | 19:00 |
| bnemec | So if you change a config option, start an update, then realize you wanted to change another config option, you don't have to wait for the first update to complete to kick off another. | 19:00 |
| trozet | dsneddon: i used apex | 19:00 |
| *** milan has quit IRC | 19:01 | |
| dsneddon | trozet, OK, my other concern is that this kind of breaks the intended flow. On the other hand, it's just a fallback mechanism, and this is arguably more elegant than the kludge that we have now. | 19:01 |
| *** fzdarsky has joined #tripleo | 19:01 | |
| trozet | dsneddon: yeah my other patch: https://review.openstack.org/#/c/509190/ is going to need to take into account all of those nuances | 19:01 |
| trozet | dsneddon: but this one doesnt hurt anything, and it fixes it for me | 19:02 |
| dsneddon | trozet, Did you create any custom composable networks in your test, or only use the included networks? | 19:02 |
| trozet | dsneddon: i just used the legacy networks. Some of them were enabled, and some of them were disabled | 19:02 |
| dsneddon | trozet, I gave it a +2, but I'd like Steven to review it before we merge it. | 19:04 |
| trozet | dsneddon: in my 2nd patch, i ran into problems now with jsut what you said. Other things reference the port which doesnt exist | 19:04 |
| trozet | dsneddon: so im going to work on a fix for that | 19:04 |
| dsneddon | trozet, Yeah, thanks. | 19:05 |
| trozet | dsneddon: but i think we just need to not create the port at all if the network is disabled | 19:06 |
| trozet | dsneddon: and try to fix references | 19:06 |
| *** milan has joined #tripleo | 19:06 | |
| dsneddon | trozet, Right, that's what I think too. | 19:06 |
| Tengu | pfff. don't know why, but the advanced placement just doesn't work anympre. | 19:06 |
| trozet | dsneddon: and as you mentioned https://review.openstack.org/#/c/509190/4/overcloud.j2.yaml | 19:07 |
| *** jrist has quit IRC | 19:07 | |
| dsneddon | trozet, The network structure takes a surprising amount of space in the overall Heat resource list, and I don't want to have a bunch of references in all roles for networks that we aren't even using. | 19:07 |
| trozet | dsneddon: render the service net map. This part does allow fallback though if a service is mapped to a disabled network. I.E. if a user did not use the (TBD) rendered service netmap and used their own | 19:07 |
| Tengu | just lost 4 hours this evening trying to make that @|#|¼ advanced placement work, but apparently something changed somewhere and it's ignoring the "node" capabilities… | 19:08 |
| dsneddon | trozet, Yeah, the logic that Steven came up with in service_net_map.j2.yaml is pretty cool, but we could modify that to write out "ctlplane" for any networks that are disabled or unrecognized. | 19:08 |
| trozet | dsneddon: yeah ok | 19:09 |
| *** jmelvin_ has joined #tripleo | 19:09 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721045 | 19:10 |
| openstack | Launchpad bug 1721045 in tripleo "get_watch_server_url in HeatClientPlugin returns incorrect url in IPv6" [Critical,In progress] - Assigned to Alfredo Moralejo (amoralej) | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721366 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,In progress] - Assigned to yatin (yatinkarel) | 19:10 |
| *** jmelvin has quit IRC | 19:11 | |
| Tengu | duh… I can search for the filter logs, they aren't present in pike X( | 19:12 |
| mnaser | is there someone that is working on fixing the tripleo ci for zuulv3? i can maybe help them out? | 19:12 |
| Tengu | that will of course NOT help at all finding what's happening | 19:12 |
| *** jtomasek has quit IRC | 19:13 | |
| mwhahaha | mnaser: there's a patch for it for normal legacy jobs. not sure about the v3 under puppet | 19:14 |
| mnaser | mwhahaha the reason is i still see zuulv3 jobs failing for tripleo (not that they matter now but they will once we'll switch back) | 19:14 |
| mwhahaha | mnaser: yea let me dig up the fix. i think it's recently +A'd | 19:14 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-common stable/pike: Add logging to capabilities filter https://review.openstack.org/510662 | 19:15 |
| Tengu | any way to get some booster on that merge request? -^ | 19:15 |
| mnaser | https://review.openstack.org/#/c/510417/ for example this one failed this morning -- looks like it was looking for /etc/nodepool/sub_nodes (probably wrong nodeset?) | 19:15 |
| Tengu | really, really, really, that one should be in pike. | 19:15 |
| mwhahaha | mnaser: https://review.openstack.org/#/c/508660/ | 19:16 |
| mwhahaha | mnaser: that should clear up some of those issues so let's see what that looks like when it lands | 19:16 |
| EmilienM | rook: zaneb is the best person here to confirm about convergence things | 19:16 |
| mnaser | mwhahaha cool, i'll keep an eye out, thanks! | 19:17 |
| zaneb | rook: slagle's suggestion to look in the DB is the best way | 19:17 |
| *** dciabrin has quit IRC | 19:18 | |
| mwhahaha | mnaser: we do also have https://review.openstack.org/#/c/509704/ outstanding for ocata stuff. not sure it affects puppet jobs though | 19:18 |
| openstackgerrit | James Slagle proposed openstack/instack-undercloud stable/pike: Fix invalid /etc/hosts edit https://review.openstack.org/510664 | 19:19 |
| zaneb | rook: what has been implemented in Heat so far is only one part of that Juno blueprint ('phase 1', although we hadn't come up with that terminology at the time) | 19:19 |
| *** jrist has joined #tripleo | 19:20 | |
| *** jrist has joined #tripleo | 19:20 | |
| *** tosky has quit IRC | 19:20 | |
| rook | zaneb: thanks | 19:20 |
| *** tosky__ is now known as tosky | 19:21 | |
| *** abregman|afk is now known as abregman | 19:21 | |
| rook | zaneb: i know shardy was worried about resource util with convergence enabled... it seem like heat-engine does chew up more CPU, but not memory | 19:21 |
| rook | the deployment hasn't finished up, but I should have some numbers with a 32 node deployment | 19:22 |
| rook | w/ and without convergence. | 19:22 |
| zaneb | rook: cool, that would be great. we did some benchmarking in the gate to get the memory use down, but we are only guessing about how that translates to big deployments | 19:23 |
| rook | yeah zaneb I will scale this up with convergence, but we will at least be able to compare 32 nodes... | 19:27 |
| *** jmelvin__ has joined #tripleo | 19:27 | |
| zaneb | that's a *lot* more nodes that we are comparing in the gate :) | 19:27 |
| rook | :*( | 19:28 |
| rook | someone give this man more OVB | 19:28 |
| zaneb | s/that/than/ | 19:28 |
| *** jmelvin_ has quit IRC | 19:30 | |
| *** pkovar has joined #tripleo | 19:30 | |
| bnemec | rook: I really need to find a public cloud that has all the features we need to run OVB. | 19:32 |
| bnemec | It might be expensive, but for occasional scale testing it's probably cheaper and simpler than having a huge dedicated environment of our own. | 19:32 |
| mnaser | bnemec what's the requirements again? | 19:32 |
| mnaser | i remember a document somewhere | 19:33 |
| *** toure is now known as toure_biab | 19:33 | |
| rook | bnemec does rackspace not have something? | 19:33 |
| rook | bnemec i don't see why they wouldn't... | 19:33 |
| bnemec | mnaser: It's not really documented explicitly, although you could probably figure it out from the docs: http://openstack-virtual-baremetal.readthedocs.io/en/latest/host-cloud/setup.html | 19:33 |
| bnemec | mnaser: I actually created an account on vexx to try this, but haven't had the time yet. | 19:33 |
| Tengu | hmmmm. interesting. Activating the logs for the (in)famous TripleOCapabilitiesFilter I can see only the already-known compute and controller nodes seem to pass through that filter, not the ceph-storages. | 19:34 |
| bnemec | rook: Rax has a bunch of weirdness that I think make it not suitable. | 19:34 |
| mnaser | prevent_arp_spoofing => you can avoid this one | 19:34 |
| rook | fantastic... | 19:34 |
| mnaser | we have port security neutron extension | 19:34 |
| bnemec | Like no direct access to Neutron or something. | 19:34 |
| mnaser | you can turn off port security for an entire network | 19:34 |
| mnaser | so we have that one figured out | 19:34 |
| bnemec | Yeah, prevent_arp_spoofing is not needed with port-security. | 19:34 |
| bnemec | That was a big blocker that doesn't exist in recent releases. | 19:34 |
| mnaser | firewall_driver = neutron.agent.firewall.NoopFirewallDriver <= thats a given for most deployments? or you can just have a security group that allows * ? | 19:35 |
| *** jmelvin__ has quit IRC | 19:35 | |
| bnemec | (assuming port-security is exposed, which it isn't in all public clouds) | 19:35 |
| bnemec | mnaser: That's also obsoleted by port-security. | 19:35 |
| bnemec | I'm not actually aware of any blockers, I just haven't had the time to sit down and try it for real. | 19:35 |
| mnaser | force_config_drive is disabled with us (it was causing a lot of issues when mounting drives where tools would think /dev/vdb is the mounted drive, but in reality thats configdrive) | 19:36 |
| mnaser | oh im just speaking out loud :-P | 19:36 |
| bnemec | And I don't really want to stand up the environment and then leave it sit for a week while I get distracted with other stuff. :-) | 19:36 |
| mnaser | mtu work with us, with newer neutron you can configure whatever mtu you want and our physical network mtu is 9000 so you can run it at 1550 or anything up to 9000-<gre overhead> | 19:36 |
| mnaser | (and we're running pike so you should be good) .. the only thing is that pxe boot patch.. | 19:36 |
| bnemec | Yeah, 9000 should be good. | 19:36 |
| bnemec | Although there are ways around that even if it was only 1500. | 19:36 |
| mnaser | hmm, the patch is a tricky one | 19:37 |
| bnemec | mnaser: That's also not strictly required anymore. | 19:37 |
| bnemec | We still run it on our private clouds, but it's not appropriate for a public cloud. | 19:38 |
| bnemec | The ipxe-boot image will allow enough pxe booting for a TripleO deployment. | 19:38 |
| *** morazi has quit IRC | 19:38 | |
| mnaser | yeah i can imagine that can do the trick | 19:38 |
| bnemec | It has drawbacks and I'd like to find a better way that doesn't require either the image or the patch, but I haven't had time to do that either. :-) | 19:38 |
| mnaser | feel free to reach out if you end up having any time :p | 19:39 |
| Tengu | hello bnemec ! what's this "kolla" thing that failed without real information about its failure for the cherry-pick of your patch in pike? | 19:44 |
| bnemec | Tengu: You can ignore that AFAIK. | 19:46 |
| bnemec | It's a third-party job that's not gating. | 19:46 |
| bnemec | And I'm pretty sure it's failing everywhere at the moment. | 19:46 |
| Tengu | bnemec: ah, ok. | 19:46 |
| Tengu | "great" :) | 19:46 |
| bnemec | We should really address that. It is confusing. | 19:47 |
| Tengu | bnemec: I was a bit surprised your patch wasn't merged in pike in fact. | 19:47 |
| bnemec | Tengu: I just never got around to backporting it. | 19:47 |
| Tengu | and I was turning the logs over and over, I didn't see anything that matched what you added in the filter. I understood once I switched from master to pike on github :P. | 19:47 |
| Tengu | bnemec: well, now it's on the tracks ;) | 19:48 |
| openstackgerrit | Wojciech Dec proposed openstack/tripleo-heat-templates master: Adding Cisco VTS ML2 mechanism driver service template https://review.openstack.org/510673 | 19:48 |
| *** dparkes has joined #tripleo | 19:49 | |
| bnemec | Tengu: If it would help, that patch should be pretty easy to apply to your undercloud locally. | 19:49 |
| bnemec | In case that would help unblock you while the patch is in review. | 19:50 |
| Tengu | bnemec: already done ;) | 19:50 |
| bnemec | Tengu: Cool. Did that help narrow down what was wrong? | 19:50 |
| Tengu | bnemec: more or less. Apparently the new nodes aren't passed in the loop. | 19:51 |
| Tengu | it checks only 5 nodes on the 7 I currently get. | 19:51 |
| bnemec | Tengu: Did you register more nodes with Ironic after the initial set? | 19:51 |
| *** milan has quit IRC | 19:52 | |
| Tengu | bnemec: yup, 2 more | 19:52 |
| Tengu | bnemec: and I'm using the advanced placement, with the "node" capability and related SchedulerHints configuration | 19:52 |
| bnemec | Tengu: And did you use the openstack overcloud node import command to do it? | 19:53 |
| Tengu | bnemec: yup, and introspection as well | 19:54 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Work with containers across builds https://review.openstack.org/509232 | 19:54 |
| bnemec | Huh. | 19:54 |
| Tengu | bnemec: I always have issues with deploying on new nodes. always. | 19:54 |
| bnemec | Tengu: Oh wait, and how did you change their status in Ironic? | 19:54 |
| Tengu | bnemec: openstack overcloud node introspect --all-manageable --provide | 19:55 |
| openstackgerrit | Wojciech Dec proposed openstack/puppet-tripleo master: Adding manifest for Cisco VTS ML2 mechanism driver configuration https://review.openstack.org/510645 | 19:55 |
| bnemec | Hmm, you're shooting holes in my theory. :-P | 19:55 |
| Tengu | sorry ;) | 19:55 |
| bnemec | I've had problems in the past if I manually set nodes to available. | 19:55 |
| Tengu | hmmm ok. so nope, it's done through the openstack overcloud subcommand | 19:56 |
| bnemec | Because there's some magic that happens in the --provide step to make the nodes available to Nova. | 19:56 |
| Tengu | oh? | 19:56 |
| Tengu | that might explain my last failure then (I did try to manually bypass the introspection) | 19:56 |
| Tengu | but not the earlier ones -.- | 19:56 |
| bnemec | Yeah, that may have caused a problem. | 19:57 |
| Tengu | I'm trying for over 4 hours now. | 19:57 |
| * bnemec looks for the code | 19:57 | |
| Tengu | bnemec: note: I'm running Pike. | 19:58 |
| Tengu | might happen the master code has some differences :] | 19:58 |
| bnemec | Tengu: No, this was introduced in Pike: https://github.com/openstack/tripleo-common/blob/master/workbooks/baremetal.yaml#L276 | 19:59 |
| bnemec | version: '2.0' | 19:59 |
| bnemec | name: tripleo.baremetal.v1 | 19:59 |
| bnemec | Well, that's encouraging. | 19:59 |
| Tengu | hmmm ok. I don't know that part of the tripleo deploy. | 19:59 |
| Tengu | so I have some issue to understand what it represents. | 20:00 |
| bnemec | Tengu: That's because it should just happen when you tell TripleO to do the --provide step. | 20:00 |
| Tengu | ah. | 20:00 |
| bnemec | But if you provide the nodes directly in the Ironic cli or something then it doesn't happen. | 20:00 |
| Tengu | erf | 20:00 |
| Tengu | there are many inconsistences in all those commands | 20:01 |
| *** jpena|away is now known as jpena|off | 20:01 | |
| bnemec | It's a really bad user experience, but I'm not sure how to fix it unfortunately. | 20:01 |
| Tengu | ^^ | 20:02 |
| Tengu | sooo. now it should reach the part where it doesn't find my ceph-storage-%index% nodes. | 20:03 |
| bnemec | Ha, I finally found the stupid command: https://github.com/openstack/tripleo-common/blob/master/tripleo_common/utils/nodes.py#L612 | 20:04 |
| bnemec | Tengu: You could try running ^ manually on the undercloud. | 20:04 |
| Tengu | wait, I'm running a deploy, for now it sees all the 7 nodes before filtering out | 20:05 |
| *** achadha has joined #tripleo | 20:05 | |
| Tengu | annnd the TripleOCapabilitiesFilter drop them all. | 20:05 |
| Tengu | duh. | 20:05 |
| mwhahaha | ugh the fix tripleo-ci for v3 patch failed in the gate | 20:05 |
| Tengu | and 2017-10-09 20:05:24Z [overcloud-CephStorage-asca7lbd2o4s-1-zvo46o6aqz5e.CephStorage]: CREATE_FAILED ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: Unknown, Code: Unknown" | 20:05 |
| Tengu | what's that again? new issue? cooool | 20:06 |
| bnemec | Helpful error messages FTW :-/ | 20:06 |
| *** dciabrin has joined #tripleo | 20:06 | |
| Tengu | aahhhhh | 20:06 |
| Tengu | Node tagged ceph-storage-1 matches requested node ceph-storage-1 host_passes /usr/lib/python2.7/site-packages/tripleo_common/filters/capabilities_filter.py:42 | 20:07 |
| Tengu | that time it seems to find its node | 20:07 |
| Tengu | at least the first one | 20:07 |
| Tengu | and I have the second one as well. | 20:07 |
| Tengu | soooo | 20:08 |
| Tengu | apparently… | 20:08 |
| Tengu | bnemec: I suspect the earlier failures were due to some cleanup issues in either ironic and/or nova network (some ports were still present after a cleanup, preventing nova-compute to attache a new port to the MAC addresses). | 20:09 |
| Tengu | hmm, but I still see ERROR in `openstack server list' though | 20:09 |
| *** achadha has quit IRC | 20:09 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721045 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721366 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1721045 in tripleo "get_watch_server_url in HeatClientPlugin returns incorrect url in IPv6" [Critical,In progress] - Assigned to Alfredo Moralejo (amoralej) | 20:10 |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,In progress] - Assigned to yatin (yatinkarel) | 20:10 |
| Tengu | ah, 2 are building. two are from either an earlier test, or due to a first failed try in the current deploy. | 20:10 |
| Tengu | o____O | 20:11 |
| Tengu | I love CREATE_FAILED ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: Unknown, Code: Unknown" (once again) | 20:12 |
| Tengu | followed by CREATE_FAILED ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: No valid host was found. There are not enough ho | 20:12 |
| Tengu | sts available., Code: 500" | 20:12 |
| *** liverpooler has quit IRC | 20:13 | |
| Tengu | ok, failed… youhouu. apparently it didn't find the two nodes in time. | 20:14 |
| *** jprovazn has quit IRC | 20:15 | |
| *** fragatina has joined #tripleo | 20:17 | |
| *** pcaruana has quit IRC | 20:18 | |
| *** dprince has quit IRC | 20:19 | |
| Tengu | anyway. going to sleep a bit, will take that back tomorrow. Hopefully night will make miracles ;). | 20:22 |
| *** morazi has joined #tripleo | 20:22 | |
| openstackgerrit | Andy Smith proposed openstack/tripleo-heat-templates master: WIP Separate rpc and notify messaging backends https://review.openstack.org/507963 | 20:27 |
| openstackgerrit | Andy Smith proposed openstack/puppet-tripleo master: WIP: Updates to separate messaging backends alternative https://review.openstack.org/510684 | 20:30 |
| *** mcornea has joined #tripleo | 20:32 | |
| *** abregman is now known as abregman|afk | 20:33 | |
| *** rhallisey has quit IRC | 20:34 | |
| openstackgerrit | Matthew Flusche proposed openstack/os-apply-config master: fixes how os-apply-config handles invalid json https://review.openstack.org/506328 | 20:39 |
| *** Goneri has joined #tripleo | 20:39 | |
| *** sid1 has quit IRC | 20:40 | |
| *** fandrieu has joined #tripleo | 20:43 | |
| *** florianf has quit IRC | 20:47 | |
| rook | zaneb: it does seem that convergence is taking longer for the deployment to complete. | 20:47 |
| rook | but this is just my butt dyno, no data yet. | 20:47 |
| zaneb | rook: that's not unexpected, it's heavier on the database. | 20:49 |
| rook | ah ha -- the more you know! | 20:49 |
| zaneb | rook: classic scalability vs. speed trade-off. TripleO is an unusual case, in that it has massive stacks but runs only on a single node | 20:49 |
| *** jcoufal has quit IRC | 20:50 | |
| rook | zaneb: right.. however, I am still lost on what convergence buys us | 20:50 |
| zaneb | convergence architecture is more scalable, which is generally the Right Thing, but there will be a price to pay | 20:50 |
| rook | as it seems it hasn't been totally implemented. | 20:50 |
| zaneb | rook: the #1 thing is what bnemec said earlier, you can start a new update when the previous one is still in progress and it Just Works(TM) | 20:51 |
| rook | yup, we can see much higher db usage. | 20:51 |
| zaneb | also handling the cleanup after stuff fails is considerably more robust | 20:51 |
| rook | well, honestly both runs seem heavy at the start... then calm down | 20:52 |
| bnemec | In combination with the blacklist support it may help with robustness in the face of failure too. | 20:52 |
| zaneb | a lot of annoying corner case bugs go away (to be replaced by weird DB concurrency bugs ;) | 20:52 |
| bnemec | If something fails you can immediately blacklist it and fire off another update. | 20:52 |
| zaneb | bnemec: yeah, that one thing alone will change your life ;) | 20:53 |
| *** cylopez has joined #tripleo | 20:53 | |
| rook | zaneb when I say more time, I might run into the timeout :/ | 20:53 |
| rook | with the same amount of nodes, it took about 100 minutes (without convergence) | 20:53 |
| rook | With convergence, i am not going to be shocked if I hit the timeout (245min, or is it 240?).. | 20:54 |
| zaneb | that's slower than we want :/ | 20:55 |
| *** jkilpatr has quit IRC | 20:56 | |
| *** ansmith has quit IRC | 20:58 | |
| rook | yup, getting close to 3 hours... | 20:59 |
| gfidente | slagle so I tested this manually too https://review.openstack.org/#/c/509001 | 21:00 |
| gfidente | seems to work | 21:00 |
| gfidente | given you voted on that already in the past ... :D | 21:00 |
| slagle | gfidente: ok :) | 21:02 |
| rook | zaneb is there something I can query if/when this fails to determine what it is spending so much time on | 21:02 |
| gfidente | slagle thanks | 21:02 |
| gfidente | you know I am not sure how come I saw YAQL issues in previous checks | 21:02 |
| gfidente | when the multinode should not be running the _via_nova task | 21:02 |
| gfidente | have a clue? | 21:02 |
| zaneb | rook: not really, no. I assume it is still making progress (but slowly), not stuck altogether? | 21:03 |
| rook | shit zaneb | 21:06 |
| *** chem has quit IRC | 21:06 | |
| rook | so it seems compute-1 is having my classic issue of the overcloud image was written, the host is no longer in build, but ironic never rebooted the node. | 21:06 |
| rook | so, the timing is fubar, the results are fubar | 21:07 |
| rook | sorry for the noise. :( | 21:07 |
| rook | I will need to kill this, start over, and hope for better results. | 21:07 |
| rook | This happens pretty often | 21:07 |
| bnemec | I wonder if with convergence it will be more possible to resume a failed initial deployment? | 21:08 |
| bnemec | I know in the past we haven't recommended it, but with convergence and the improved fallback networking it might be more doable now. | 21:09 |
| *** yamahata has joined #tripleo | 21:10 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721045 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721366 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1721045 in tripleo "get_watch_server_url in HeatClientPlugin returns incorrect url in IPv6" [Critical,In progress] - Assigned to Alfredo Moralejo (amoralej) | 21:10 |
| openstack | Launchpad bug 1721366 in tripleo "Keystone v2.0 APIs have been removed, TripleO config is incomplete" [Critical,In progress] - Assigned to yatin (yatinkarel) | 21:10 |
| *** cylopez has quit IRC | 21:10 | |
| zaneb | bnemec: yes, I believe it will be safer to recommend. although not when you're trying to benchmark :D | 21:11 |
| rook | truth! | 21:12 |
| bnemec | zaneb: Yeah, good point. The one stuck node just kind of reminded me of it. | 21:12 |
| bnemec | I know it came up as a pain point in Boston. | 21:12 |
| * rook waits for stack to delete, and cleaning to finish | 21:12 | |
| rook | can we make something smart enough to just reboot a node if this happens | 21:13 |
| * rook has a launchpad | 21:13 | |
| zaneb | bnemec: yeah, in theory you can mark the failed node as unhealthy and start another update without even waiting for it to time out | 21:13 |
| zaneb | in theory :D | 21:13 |
| zaneb | rook: sure, a few tweaks to http://git.openstack.org/cgit/openstack/heat-templates/tree/hot/autohealing/autohealing_server.yaml and we'd be set :) | 21:15 |
| *** ecerquei__ has quit IRC | 21:15 | |
| *** apetrich has quit IRC | 21:16 | |
| *** apetrich has joined #tripleo | 21:16 | |
| *** archit has quit IRC | 21:17 | |
| rook | zaneb what are these said tweaks | 21:18 |
| zaneb | you'd have to find a suitable event to trigger from | 21:18 |
| *** abishop has quit IRC | 21:19 | |
| *** lblanchard has quit IRC | 21:20 | |
| zaneb | although actually Giulio's OS::Mistral::ExternalResource might be a better fit for this task, since the problem only occurs during resource creation IIUC | 21:20 |
| rook | zaneb: tagged you in a document | 21:20 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart-extras master: overcloud-deploy: add config-download + ansible run feature https://review.openstack.org/508306 | 21:23 |
| *** leitan has quit IRC | 21:28 | |
| *** leitan has joined #tripleo | 21:28 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: DO-NOT-MERGE Trigger scenario001 https://review.openstack.org/501987 | 21:28 |
| *** leitan has quit IRC | 21:32 | |
| *** jkilpatr has joined #tripleo | 21:33 | |
| *** pchavva has quit IRC | 21:33 | |
| *** dparkes has quit IRC | 21:40 | |
| *** ansmith has joined #tripleo | 21:49 | |
| slagle | EmilienM: pushed some changes to traas to fix zuul_changes, and honor oooq and oooq-extras patches as well | 21:51 |
| slagle | EmilienM: if you see any strangeness, it's probably related to that. | 21:51 |
| slagle | i'm testing it now | 21:51 |
| *** etingof has quit IRC | 21:53 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Disables port status for all ODL deployments https://review.openstack.org/509834 | 21:54 |
| *** threestrands has joined #tripleo | 21:55 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Use keystone v3 session with novaclient https://review.openstack.org/510535 | 21:56 |
| openstackgerrit | Merged openstack/puppet-tripleo master: ovn HA: Enable ip_nonlocal_bind sysctl flag https://review.openstack.org/509470 | 21:56 |
| *** mcornea has quit IRC | 21:58 | |
| *** chlong_ has quit IRC | 21:59 | |
| *** chlong has quit IRC | 21:59 | |
| *** bobh has quit IRC | 21:59 | |
| mwhahaha | larsks: is https://blueprints.launchpad.net/tripleo/+spec/container-healthchecks done? | 21:59 |
| *** etingof has joined #tripleo | 22:06 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721045 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1721045 in tripleo "get_watch_server_url in HeatClientPlugin returns incorrect url in IPv6" [Critical,In progress] - Assigned to Alfredo Moralejo (amoralej) | 22:10 |
| *** fragatina has quit IRC | 22:13 | |
| *** fragatina has joined #tripleo | 22:13 | |
| *** jrist has quit IRC | 22:14 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements https://review.openstack.org/510309 | 22:21 |
| *** pmannidi has joined #tripleo | 22:27 | |
| mwhahaha | bnemec: https://bugs.launchpad.net/tripleo/+bug/1684272 is that fixed since the heat issue was merged? | 22:29 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [High,In progress] | 22:29 |
| *** jdennis has quit IRC | 22:30 | |
| *** fragatina has quit IRC | 22:34 | |
| *** jrist has joined #tripleo | 22:34 | |
| *** bobh has joined #tripleo | 22:34 | |
| *** gfidente has quit IRC | 22:35 | |
| bnemec | mwhahaha: Yeah, at this point I think we can close that one. We're far enough from the initial performance regression that any comparisons would be essentially useless anyway. | 22:36 |
| mwhahaha | k | 22:36 |
| bnemec | Switching to convergence probably nets us a whole new performance profile anyway. | 22:36 |
| * mwhahaha goes through old bugs | 22:36 | |
| *** dmarlin has quit IRC | 22:41 | |
| *** shreshtha has quit IRC | 22:42 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-docs master: WIP: Add documentation for TripleO/RDO Pipelines https://review.openstack.org/510657 | 22:44 |
| *** etingof has quit IRC | 22:47 | |
| *** catintheroof has joined #tripleo | 22:47 | |
| *** achadha has joined #tripleo | 22:54 | |
| *** achadha has quit IRC | 22:54 | |
| *** bfournie has quit IRC | 22:54 | |
| *** achadha has joined #tripleo | 22:54 | |
| *** achadha has quit IRC | 22:58 | |
| *** achadha has joined #tripleo | 22:58 | |
| *** etingof has joined #tripleo | 23:01 | |
| *** bobh has quit IRC | 23:02 | |
| *** tosky has quit IRC | 23:02 | |
| *** dciabrin has quit IRC | 23:04 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1721045 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1721045 in tripleo "get_watch_server_url in HeatClientPlugin returns incorrect url in IPv6" [Critical,In progress] | 23:10 |
| *** gbarros_ has joined #tripleo | 23:10 | |
| owalsh | mwhahaha: hey, https://review.openstack.org/506595 ... do we have promotion & new docker images? | 23:11 |
| *** gbarros has quit IRC | 23:12 | |
| *** tongl has quit IRC | 23:14 | |
| *** Goneri has quit IRC | 23:16 | |
| *** dciabrin has joined #tripleo | 23:16 | |
| *** gbarros_ has quit IRC | 23:25 | |
| *** gbarros has joined #tripleo | 23:26 | |
| *** jmelvin has joined #tripleo | 23:38 | |
| *** bfournie has joined #tripleo | 23:38 | |
| *** jmelvin has quit IRC | 23:39 | |
| *** bfournie has quit IRC | 23:40 | |
| *** bfournie has joined #tripleo | 23:40 | |
| *** raildo has quit IRC | 23:41 | |
| *** gbarros has quit IRC | 23:41 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Config download support for all deployments https://review.openstack.org/508189 | 23:48 |
| *** rlandy is now known as rlandy|bbl | 23:49 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart master: fs10: deploy steps with ansible https://review.openstack.org/508307 | 23:55 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart-extras master: overcloud-deploy: add config-download + ansible run feature https://review.openstack.org/508306 | 23:55 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Change image paths to the images server https://review.openstack.org/510112 | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
