| *** dcain has joined #tripleo | 00:11 | |
| *** akuznetsov has joined #tripleo | 00:11 | |
| *** tiswanso has joined #tripleo | 00:14 | |
| *** dmacpher-afk has quit IRC | 00:16 | |
| *** rhallisey has joined #tripleo | 00:16 | |
| *** akuznetsov has quit IRC | 00:16 | |
| *** dcain1 has joined #tripleo | 00:17 | |
| *** dcain has quit IRC | 00:20 | |
| *** trozet has quit IRC | 00:26 | |
| *** tiswanso has quit IRC | 00:28 | |
| *** dshulyak has joined #tripleo | 00:28 | |
| *** tiswanso has joined #tripleo | 00:28 | |
| *** dshulyak has quit IRC | 00:33 | |
| *** saneax is now known as saneax_AFK | 00:40 | |
| *** dmacpher has joined #tripleo | 01:03 | |
| *** lblanchard has joined #tripleo | 01:07 | |
| *** penick has quit IRC | 01:08 | |
| jidar | shardy: any chance there is work being done to support OS::Heat::SoftwareDeployments with a servers param that takes (or tripleo providing) a Controllers/Computes option? I've alluded to this before but man is it a goofy problem to get around | 01:09 |
|---|---|---|
| jidar | I really don't like having to do `if [[ $HOSTNAME =~ .*controller.* ]]; then` in my scripts | 01:09 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone https://review.openstack.org/286344 | 01:12 |
| slagle | EmilienM: fyi, looks like almost all (if not all) tripleo ha jobs are failing due to what i suspect is this: https://review.openstack.org/#/c/284876/ | 01:31 |
| EmilienM | looking | 01:31 |
| slagle | that passed tripleo ci, but we only run the nonha job there | 01:31 |
| EmilienM | slagle: well, TripleO CI passed it | 01:31 |
| slagle | yea, the nonha job | 01:32 |
| slagle | EmilienM: here's how the ha jobs are failing: http://logs.openstack.org/23/235423/44/check-tripleo/gate-tripleo-ci-f22-ha/7465efa/console.html | 01:32 |
| slagle | search for keystone-manage bootstrap | 01:32 |
| EmilienM | slagle: let me look | 01:33 |
| EmilienM | the exec failed, probably because orchestration | 01:33 |
| EmilienM | let me dig a bit | 01:33 |
| EmilienM | oh but it's undercloud? | 01:34 |
| EmilienM | nevermind | 01:34 |
| EmilienM | slagle: I have a patch to THT | 01:35 |
| slagle | EmilienM: feel free to take the bug i just reported, https://bugs.launchpad.net/tripleo/+bug/1551501 | 01:36 |
| openstack | Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,Triaged] | 01:36 |
| EmilienM | again, this kind of bug will be fixed when we'll use our new pacemaker provider to manage services | 01:36 |
| EmilienM | slagle: https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L989 | 01:36 |
| EmilienM | we need to set that for pacemaker in THT | 01:36 |
| EmilienM | I'm sending a patch now | 01:36 |
| slagle | yea, we only want to run that one time on the bootstrap node i'd guess? | 01:37 |
| EmilienM | yeah | 01:37 |
| EmilienM | I'm taking the bug | 01:37 |
| EmilienM | actually no that's not this dependency | 01:40 |
| *** trozet has joined #tripleo | 01:41 | |
| EmilienM | slagle: do you have the output of the Exec? | 01:42 |
| EmilienM | I suspect Galera is not ready | 01:42 |
| slagle | EmilienM: look at the log i linked, that's all i have | 01:42 |
| *** dcain1 has quit IRC | 01:42 | |
| EmilienM | kk | 01:42 |
| slagle | that was all the stdout from puppet | 01:42 |
| EmilienM | galera should be ready at step 2 and the error is at step 3 | 01:42 |
| *** dcain has joined #tripleo | 01:43 | |
| EmilienM | mhh | 01:43 |
| EmilienM | oh I know | 01:44 |
| EmilienM | slagle: keystone resources are not managed by puppet-keystone on the overcloud | 01:44 |
| EmilienM | so the bootstrap tries to create them | 01:44 |
| EmilienM | but can't I think | 01:45 |
| EmilienM | I have a temporary fix I guess | 01:45 |
| EmilienM | slagle: do you manage default domain, etc already? | 01:45 |
| slagle | is it required? | 01:45 |
| slagle | if it's required, then we must create it via os-cloud-config | 01:45 |
| EmilienM | since a few commits in OpenStack Keystone, yes | 01:45 |
| slagle | oh, possible not then | 01:46 |
| EmilienM | see 7b7fea7a3fe7677981fbf9bac5121bc15601163 | 01:46 |
| EmilienM | slagle: https://github.com/openstack/keystone/commit/f699ca93fc6f2485ec8e76e907572a2f838cd3cb | 01:47 |
| slagle | the repo we're using is from 2016-02-23, so we likely don't have that | 01:49 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: controller/ha: disable keystone-manage bootstrap. https://review.openstack.org/286352 | 01:50 |
| EmilienM | slagle: ^ let's see | 01:50 |
| EmilienM | it should fix the CI | 01:50 |
| *** rhallisey has quit IRC | 02:04 | |
| *** trozet has quit IRC | 02:08 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1551501 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| EmilienM | yes I know | 02:10 |
| slagle | would you like me to remove the alert? | 02:11 |
| EmilienM | lol | 02:11 |
| EmilienM | no please keep it | 02:11 |
| slagle | it's just the "alert" tag | 02:11 |
| EmilienM | it just send notifis to my phone when someone ping me | 02:11 |
| slagle | lol | 02:11 |
| EmilienM | ahah I should drop that | 02:11 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone https://review.openstack.org/286344 | 02:15 |
| *** Marga__ has joined #tripleo | 02:16 | |
| *** akuznetsov has joined #tripleo | 02:17 | |
| *** Marga_ has quit IRC | 02:19 | |
| *** akuznetsov has quit IRC | 02:21 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Make all network ports type to consume FixedIPs https://review.openstack.org/286360 | 02:40 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1551501 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| *** yuanying has joined #tripleo | 03:21 | |
| *** yuanying_ has quit IRC | 03:23 | |
| *** xinwu has quit IRC | 03:23 | |
| *** Marga__ has quit IRC | 03:26 | |
| *** yuanying has quit IRC | 03:27 | |
| *** Marga_ has joined #tripleo | 03:30 | |
| *** Nakato has quit IRC | 03:31 | |
| *** Marga_ has quit IRC | 03:35 | |
| *** lblanchard has quit IRC | 03:38 | |
| *** panda has quit IRC | 03:38 | |
| *** panda has joined #tripleo | 03:38 | |
| mandre | I get an error installing my undercloud with enable_mistral=true because of missing python-mistralclient dependency: http://paste.openstack.org/show/488683/ | 03:39 |
| mandre | where is the best place to add the missing dependency? | 03:39 |
| *** Marga_ has joined #tripleo | 03:43 | |
| *** Nakato has joined #tripleo | 03:45 | |
| *** Marga_ has quit IRC | 03:47 | |
| *** rlandy has quit IRC | 03:48 | |
| *** links has joined #tripleo | 03:49 | |
| *** xinwu has joined #tripleo | 03:51 | |
| *** chlong_ has quit IRC | 04:03 | |
| *** saneax_AFK is now known as saneax | 04:06 | |
| *** yuanying has joined #tripleo | 04:10 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1551501 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| *** yamahata has joined #tripleo | 04:16 | |
| *** Marga_ has joined #tripleo | 04:16 | |
| *** Marga_ has quit IRC | 04:16 | |
| *** Marga_ has joined #tripleo | 04:17 | |
| *** anande has joined #tripleo | 04:35 | |
| *** tiswanso has quit IRC | 04:47 | |
| *** anande has quit IRC | 05:02 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1551501 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| *** dcain has quit IRC | 05:15 | |
| *** Marga_ has quit IRC | 05:20 | |
| *** masco has joined #tripleo | 05:23 | |
| *** dmacpher has quit IRC | 05:24 | |
| *** Marga_ has joined #tripleo | 05:30 | |
| *** jaosorior has joined #tripleo | 05:49 | |
| *** dmacpher has joined #tripleo | 06:01 | |
| *** lucasagomes has quit IRC | 06:02 | |
| *** lucasagomes has joined #tripleo | 06:02 | |
| *** dmacpher_ has joined #tripleo | 06:08 | |
| *** rcernin has joined #tripleo | 06:09 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1551501 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| *** dmacpher has quit IRC | 06:10 | |
| *** dmacpher_ has quit IRC | 06:16 | |
| *** dmacpher_ has joined #tripleo | 06:23 | |
| *** lucasagomes has quit IRC | 06:24 | |
| *** lucasagomes has joined #tripleo | 06:27 | |
| *** thrash|g0ne has quit IRC | 06:33 | |
| *** trown|outtypewww has quit IRC | 06:34 | |
| *** trown has joined #tripleo | 06:35 | |
| *** tzumainn has quit IRC | 06:40 | |
| *** akuznetsov has joined #tripleo | 06:45 | |
| *** akuznetsov has quit IRC | 06:49 | |
| *** ishant has joined #tripleo | 06:54 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/tripleo-heat-templates: Fix typos https://review.openstack.org/265126 | 06:55 |
| *** dmacpher_ has quit IRC | 06:56 | |
| *** Marga_ has quit IRC | 06:57 | |
| *** dshulyak has joined #tripleo | 07:02 | |
| *** dmacpher has joined #tripleo | 07:04 | |
| *** jprovazn has joined #tripleo | 07:05 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1551501 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| *** liverpooler has joined #tripleo | 07:13 | |
| *** mikelk has joined #tripleo | 07:16 | |
| *** dmacpher has quit IRC | 07:20 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Always override X-Forwarded-Proto header for Heat https://review.openstack.org/286420 | 07:36 |
| *** bnemec has quit IRC | 07:36 | |
| *** bnemec has joined #tripleo | 07:39 | |
| *** hjensas has joined #tripleo | 07:48 | |
| *** hjensas has quit IRC | 07:48 | |
| *** hjensas has joined #tripleo | 07:48 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common: Adds a bash script that operators will use during upgrades https://review.openstack.org/286426 | 07:50 |
| *** jtomasek has joined #tripleo | 07:50 | |
| *** fgimenez has joined #tripleo | 08:00 | |
| *** fgimenez has quit IRC | 08:00 | |
| *** fgimenez has joined #tripleo | 08:00 | |
| *** slagle has quit IRC | 08:06 | |
| *** shadower has quit IRC | 08:06 | |
| *** shadower has joined #tripleo | 08:07 | |
| *** devvesa has joined #tripleo | 08:08 | |
| *** pcaruana has joined #tripleo | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1551501 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| *** yamahata has quit IRC | 08:11 | |
| *** paramite has joined #tripleo | 08:18 | |
| *** ccamacho has joined #tripleo | 08:20 | |
| *** rdopiera has joined #tripleo | 08:21 | |
| *** aufi has joined #tripleo | 08:21 | |
| *** sshnaidm has quit IRC | 08:22 | |
| *** mcornea has joined #tripleo | 08:23 | |
| *** ifarkas has joined #tripleo | 08:30 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: controller/ha: disable keystone-manage bootstrap. https://review.openstack.org/286352 | 08:30 |
| *** paramite is now known as paramite|afk | 08:32 | |
| *** ohamada has joined #tripleo | 08:35 | |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-specs: Refactor top level puppet manifests https://review.openstack.org/286439 | 08:37 |
| *** paramite|afk is now known as paramite | 08:37 | |
| *** xinwu has quit IRC | 08:45 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** openstackgerrit has joined #tripleo | 08:48 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: Move ordering constraints from httpd to openstack-core https://review.openstack.org/286446 | 08:48 |
| *** hewbrocca has joined #tripleo | 08:54 | |
| *** paramite is now known as paramite|afk | 08:55 | |
| *** paramite|afk is now known as paramite | 08:57 | |
| *** gfidente has joined #tripleo | 08:58 | |
| *** athomas has joined #tripleo | 09:01 | |
| *** mbound has joined #tripleo | 09:06 | |
| *** jcoufal has joined #tripleo | 09:10 | |
| *** jistr has joined #tripleo | 09:14 | |
| *** sshnaidm has joined #tripleo | 09:18 | |
| *** akrivoka has joined #tripleo | 09:19 | |
| *** olap has quit IRC | 09:29 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list https://review.openstack.org/286062 | 09:29 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 09:29 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: DO NOT MERGE: Print output of brctl show from hosting node https://review.openstack.org/286279 | 09:29 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 09:29 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker: Basic beaker one node test. https://review.openstack.org/281376 | 09:31 |
| hewbrocca | gfidente: good morning! | 09:32 |
| hewbrocca | gfidente: I see you are working on the CI... | 09:32 |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| gfidente | hewbrocca, morning, yes but it seems we're filtering vlan traffic in upstream env | 09:34 |
| gfidente | hewbrocca, I'm still trying to figure why that is | 09:34 |
| gfidente | hewbrocca, bridges on undercloud and overcloud nodes are good so we're down to the one on the baremetal host | 09:35 |
| hewbrocca | OK, cool | 09:36 |
| *** derekh has joined #tripleo | 09:39 | |
| gfidente | derekh, ! | 09:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Make all network ports type to consume FixedIPs https://review.openstack.org/286360 | 09:39 |
| gfidente | maaaaaaaaaaaaaaaaan | 09:40 |
| derekh | gfidente: howdy | 09:40 |
| * gfidente needs help on something | 09:40 | |
| gfidente | we have vsctl in hostinfo for undercloud and overcloud and those look okay | 09:40 |
| gfidente | so I was looking into baremetal and tried https://review.openstack.org/#/c/286279/ | 09:41 |
| gfidente | but neither brctl nor ovs-vsctl is available on HOST_IP ? | 09:41 |
| gfidente | did I ssh on the right host? | 09:41 |
| *** mkovacik has joined #tripleo | 09:42 | |
| derekh | gfidente: that brctl command is being run on the jenkins node, try it further down in the here doc | 09:42 |
| gfidente | HOST_IP is the jenkins node? | 09:43 |
| derekh | gfidente: opps sorry I missed that, HOST_IP is the testenv host | 09:44 |
| gfidente | yeah so I thought one of brctl or ovs-vsctl would work but none did? | 09:44 |
| marios | gfidente: o/ morning, i made a comment on the netiso review as promised earlier this morning | 09:45 |
| marios | gfidente: oh you updated https://review.openstack.org/#/c/273424/ | 09:46 |
| derekh | gfidente: just ran it manually now on the test env host http://paste.openstack.org/show/488723/ | 09:46 |
| *** mgould has joined #tripleo | 09:47 | |
| gfidente | marios, thanks for that :) | 09:47 |
| derekh | gfidente: brctl isn't installed but ovs-vsctl is | 09:47 |
| gfidente | derekh, in patchset 1 was trying ovs-vsctlfwiw | 09:47 |
| openstackgerrit | Merged openstack/tripleo-common: Adds a bash script that operators will use during upgrades https://review.openstack.org/286426 | 09:48 |
| derekh | gfidente: ahhh, ssh is locked down to only allow specific commands, see RE_COMMANDS http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/elements/testenv-worker/bin/ci_commands#n65 | 09:50 |
| gfidente | derekh, ah okay that explains it thanks! | 09:50 |
| gfidente | derekh, so this https://jenkins07.openstack.org/job/gate-tripleo-ci-f22-ha/581/console is using ctlplane for corosync | 09:51 |
| derekh | gfidente: ack, lets see what happens | 09:52 |
| gfidente | derekh, cause ctlplane is untagged, I think the tagged traffic is filtered | 09:52 |
| gfidente | so I expect it to fail anyway, but not on corosync, just to confirm the thesis | 09:53 |
| hewbrocca | gfidente, derekh if you guys need any help don't hesitate to reach out | 10:00 |
| hewbrocca | this is our top priority issue right now | 10:00 |
| derekh | hewbrocca: thanks, will do | 10:01 |
| marios | gerrit really slow or just here? | 10:19 |
| marios | (wasn't too bad earlier) | 10:20 |
| *** olap has joined #tripleo | 10:21 | |
| shardy | also really slow for me :( | 10:27 |
| *** ishant has quit IRC | 10:28 | |
| *** lucasagomes has quit IRC | 10:30 | |
| marios | shardy: thanks | 10:31 |
| *** lucasagomes has joined #tripleo | 10:31 | |
| *** lucasagomes has quit IRC | 10:32 | |
| *** lucasagomes has joined #tripleo | 10:32 | |
| dtantsur | slow for everyone, I've reported to #infra already (it's not that they didn't notice..) | 10:33 |
| *** lucasagomes has quit IRC | 10:37 | |
| *** lucasagomes has joined #tripleo | 10:37 | |
| -openstackstatus- NOTICE: Gerrit is going to be restarted due to poor performance | 10:39 | |
| *** ChanServ changes topic to "Gerrit is going to be restarted due to poor performance" | 10:39 | |
| *** lucasagomes has quit IRC | 10:41 | |
| *** lucasagomes has joined #tripleo | 10:43 | |
| *** ChanServ changes topic to "TripleO | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/" | 10:45 | |
| -openstackstatus- NOTICE: gerrit finished restartign | 10:45 | |
| *** tosky has joined #tripleo | 10:48 | |
| shardy | derekh: are you OK with https://review.openstack.org/#/c/275668 landing now? lgtm | 10:55 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: Move ordering constraints from httpd to openstack-core https://review.openstack.org/286446 | 10:58 |
| derekh | shardy: yup, go for it | 10:59 |
| shardy | derekh: thanks, done | 11:00 |
| *** chlong_ has joined #tripleo | 11:01 | |
| derekh | shardy: the other patch on the series will also need to be approved https://review.openstack.org/#/c/275667/8 | 11:01 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Source undercloud environment variable from a file https://review.openstack.org/275667 | 11:05 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Split the deploy script into its own file https://review.openstack.org/275668 | 11:06 |
| openstackgerrit | Merged openstack/instack-undercloud: mistral: deploy API in WSGI with apache https://review.openstack.org/280719 | 11:06 |
| derekh | shardy: thanks | 11:07 |
| shardy | derekh: Have a question re https://review.openstack.org/#/c/271416/, the Depends-On looks bogus? | 11:09 |
| derekh | shardy: yup, I'll remove it now, i had forgotten about it, it was there so that delorean would build something so I could see if the logs were grabbed | 11:14 |
| shardy | aha, cool | 11:15 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Archive all of the delorean logs https://review.openstack.org/271416 | 11:16 |
| derekh | shardy: ^^ | 11:16 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 11:36 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list https://review.openstack.org/286062 | 11:36 |
| hewbrocca | gfidente: need any additional help with the net-iso tests? | 11:37 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 11:37 |
| *** panda has quit IRC | 11:37 | |
| *** panda has joined #tripleo | 11:38 | |
| gfidente | hewbrocca, I think we haven't figured the root issue yet | 11:41 |
| gfidente | hewbrocca, I was thinking to ping dsneddon if/when he is around | 11:42 |
| gfidente | hewbrocca, we're only dealing with the effects for now :( | 11:42 |
| hewbrocca | gfidente: dsneddon won't be on for another 6 hours or so | 11:42 |
| hewbrocca | hopefully dprince will have some ideas | 11:44 |
| gfidente | yeah it's that we don't see this on local envs | 11:46 |
| gfidente | so it's baiscally troubleshooting we're trying to do on the ci infra to see if we can spot the issue | 11:47 |
| hewbrocca | I can't tell from the gerrit page what the issue actually is? | 11:48 |
| gfidente | well, we can see what goes wrong in the overcloud nodes, but they just don't talk to each other | 11:52 |
| gfidente | at least not via the vlan tagged interfaces which we create in the overcloud nodes to have isolation | 11:52 |
| hewbrocca | hmm is ping working | 11:53 |
| hewbrocca | and if not, are we sure the switch is set up correctly? | 11:53 |
| gfidente | yeah we're trying just that now | 11:55 |
| hewbrocca | cool | 11:56 |
| * hewbrocca backs away | 11:56 | |
| *** tosky has quit IRC | 11:57 | |
| *** Hazelesque_ is now known as Hazelesque | 12:03 | |
| openstackgerrit | Imre Farkas proposed openstack/python-tripleoclient: [WIP] Update baremetal ready state command https://review.openstack.org/285977 | 12:16 |
| jaosorior | marios: Thanks a lot for the review! | 12:21 |
| jaosorior | on the spec | 12:21 |
| *** leanderthal|afk is now known as leanderthal | 12:21 | |
| marios | jaosorior: np, just a first pass | 12:21 |
| *** mcornea has quit IRC | 12:21 | |
| *** mcornea has joined #tripleo | 12:22 | |
| *** saneax is now known as saneax_AFK | 12:22 | |
| *** rasca_ has quit IRC | 12:23 | |
| *** rasca has joined #tripleo | 12:24 | |
| *** tosky has joined #tripleo | 12:27 | |
| *** rasca has quit IRC | 12:28 | |
| *** rasca has joined #tripleo | 12:29 | |
| EmilienM | tripleo reviewers, please look this backport: https://review.openstack.org/#/c/286031/ | 12:29 |
| jistr | and these 2 upgrade fixups https://review.openstack.org/286155 https://review.openstack.org/286156 :) | 12:30 |
| * jistr looking at the IPv6 pingtest | 12:30 | |
| derekh | gfidente: the ci env I've started is ready to run the job, | 12:32 |
| derekh | gfidente: about to run this patch on it "Use netiso in the ha job" | 12:32 |
| derekh | gfidente: are there any others I need to include | 12:32 |
| derekh | ? | 12:32 |
| gfidente | the two it depends upon | 12:32 |
| gfidente | or at least https://review.openstack.org/#/c/280431/ | 12:33 |
| *** anande has joined #tripleo | 12:33 | |
| derekh | gfidente: ok its running | 12:36 |
| derekh | gfidente: ssh jenkins@66.187.229.70 | 12:36 |
| derekh | gfidente: when its finished the testenv will stay around for a few hours | 12:36 |
| *** jaosorior has quit IRC | 12:36 | |
| derekh | gfidente: it running in screen tab 3 | 12:37 |
| gfidente | derekh, I just got on one of the overcloud nodes for the existing run | 12:37 |
| *** jaosorior has joined #tripleo | 12:37 | |
| EmilienM | jistr: that's a backport ... https://review.openstack.org/#/c/286031/ | 12:37 |
| EmilienM | jistr: so I won't modify the code except for a very good reason. | 12:38 |
| jistr | EmilienM: ah right, thanks :) +2 | 12:39 |
| EmilienM | jistr: can you backport https://review.openstack.org/#/c/238097/ too please? | 12:39 |
| jistr | EmilienM: yup i'll do that right now | 12:39 |
| EmilienM | thanks | 12:39 |
| *** lblanchard has joined #tripleo | 12:39 | |
| marios | jistr: EmilienM: wrt jistr question on https://review.openstack.org/#/c/286031/ see https://review.openstack.org/#/c/285858/1 (is a good point jistr) | 12:41 |
| EmilienM | marios: so we need to backport https://review.openstack.org/#/c/285858/ ? | 12:41 |
| marios | jistr: it seems the env it was tested in at the time had it already | 12:41 |
| jistr | marios: haha you asked the same i see :) thanks for the link | 12:42 |
| marios | EmilienM: i think so, are we seeing these validations fail here (like 'ipaddr not available?' if so we should) | 12:42 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-puppet-elements: Add overcloud-base element https://review.openstack.org/286553 | 12:43 |
| EmilienM | marios, jistr ^ | 12:43 |
| marios | EmilienM: ack | 12:43 |
| *** rhallisey has joined #tripleo | 12:43 | |
| EmilienM | gfidente: can you also review https://github.com/puppetlabs/puppetlabs-inifile/pull/199 please ? It will help to have puppet-ceph working for ipv6 | 12:46 |
| *** shardy has quit IRC | 12:47 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: OpenContrail heat templates https://review.openstack.org/286559 | 12:48 |
| EmilienM | I also have 3 more review requests for bigswitch integration: https://review.openstack.org/#/c/280496/ (backport) https://review.openstack.org/#/c/274492 and https://review.openstack.org/#/c/279842/ | 12:48 |
| jistr | EmilienM: contrail backport ^ | 12:48 |
| EmilienM | jistr++ | 12:48 |
| jistr | aha that's where ipaddr comes from :)) | 12:49 |
| jistr | +2 | 12:49 |
| *** rbrady has joined #tripleo | 12:58 | |
| *** lblanchard has quit IRC | 13:06 | |
| *** links has quit IRC | 13:07 | |
| *** rlandy has joined #tripleo | 13:10 | |
| *** julim has joined #tripleo | 13:10 | |
| gfidente | derekh, so ping actually works | 13:11 |
| gfidente | and yet corosync is logging those timeouts and never coming up | 13:11 |
| derekh | gfidente: so the problem isn't the net setup? | 13:11 |
| gfidente | I suspect an corosync/pcmk guy can help here | 13:12 |
| gfidente | derekh, it doesn't look like | 13:12 |
| gfidente | bandini, can you help us? | 13:12 |
| * gfidente having a quick lunch | 13:12 | |
| *** paramite is now known as paramite|afk | 13:13 | |
| *** pradk has joined #tripleo | 13:14 | |
| hewbrocca | gfidente: there's no ipv6 in here yet, is there? | 13:14 |
| gfidente | nope | 13:15 |
| gfidente | actually yes | 13:15 |
| gfidente | I mean, it's netiso on v4 but corosync is using an ipv6 socket | 13:15 |
| hewbrocca | uh... | 13:15 |
| *** lucasagomes is now known as lucas-hungry | 13:15 | |
| hewbrocca | that might be your problem? | 13:15 |
| bandini | gfidente: sure, where, what, when? | 13:16 |
| gfidente | the name resolves to an ipv4 address | 13:16 |
| gfidente | bandini, I think we need to wait for derekh too | 13:16 |
| bandini | gfidente: do we have some logs I can start looking at, to understand the issue? | 13:17 |
| gfidente | bandini, we're only seeing this in the upstream Ci but yes | 13:17 |
| gfidente | http://logs.openstack.org/24/273424/12/check-tripleo/gate-tripleo-ci-f22-ha/a1065dd/logs/overcloud-controller-0.tar.xz | 13:17 |
| gfidente | in there check the pcsd.log it looks like it's not getting to the other overcloud nodes | 13:18 |
| gfidente | and, thanks :) | 13:18 |
| gfidente | bandini, fwiw, we haven't landed this yet https://review.openstack.org/#/c/267073/ | 13:18 |
| gfidente | so corosync is not using --ipv6 | 13:18 |
| gfidente | and we're using ipv4 networks | 13:19 |
| gfidente | but lsof is telling me corosync opened an ipv6 socket to get to the other nodes | 13:19 |
| bandini | oh I see, an ipv6-only socket or an ipv6-mapped-ipv4 one? | 13:19 |
| * bandini downloads the logs | 13:19 | |
| *** aufi has quit IRC | 13:20 | |
| gfidente | ruby 5555 root 11u IPv6 94068 0t0 TCP overcloud-controller-0.localdomain:efi-mg->overcloud-controller-2.localdomain:47524 | 13:21 |
| *** jayg|g0n3 is now known as jayg | 13:23 | |
| *** dprince has joined #tripleo | 13:25 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: compute: include VIR_MIGRATE_TUNNELLED when doing VM shared storage https://review.openstack.org/286584 | 13:27 |
| bandini | gfidente: mmh weird because there seem to be many connections from controller-0 to controller-{1,2} that actually succeed (I cound 85 timed out and 416 successful ones) | 13:28 |
| bandini | gfidente: let me fetch the logs of the other nodes | 13:28 |
| gfidente | exactly | 13:28 |
| gfidente | some are IPv4 | 13:28 |
| *** saneax_AFK is now known as saneax | 13:29 | |
| gfidente | derekh, I wonder if it isn't worth trying with disable ipv6 sysctl? | 13:31 |
| bandini | gfidente: do we have access to these boxes? | 13:32 |
| gfidente | bandini, derek is setting up an env because this only seems to happen there | 13:33 |
| gfidente | so what we saw was by logging on the nodes during deployment but then jobs go away when job completes | 13:33 |
| derekh | bandini: send me a key and I'll add it, the undercloud is still being installed | 13:33 |
| gfidente | *nodes | 13:33 |
| bandini | ah I see | 13:33 |
| bandini | will do | 13:33 |
| *** slagle has joined #tripleo | 13:33 | |
| derekh | the env I'm setting up wont vanish afterwards | 13:33 |
| bandini | perfect, thanks | 13:35 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: compute: include VIR_MIGRATE_TUNNELLED when doing VM shared storage https://review.openstack.org/286584 | 13:35 |
| bandini | gfidente: sorry I sent the key to you as well. I am drunk | 13:37 |
| * hewbrocca checks watch | 13:37 | |
| * bandini whistles innocently | 13:38 | |
| * hewbrocca promotes bandini to manager | 13:38 | |
| bandini | rotfl | 13:38 |
| *** paramite|afk is now known as paramite | 13:39 | |
| derekh | bandini: ssh jenkins@66.187.229.70 | 13:39 |
| gfidente | bandini, you don't need to tell you're drunk everybody | 13:40 |
| derekh | bandini: then ssh root@192.168.1.19 | 13:40 |
| derekh | bandini: thats the undercloud, overcloud images are now being build, so go get a coffee and sober up while you wait ;-) | 13:41 |
| gfidente | building images | 13:41 |
| bandini | derekh: ehehe oki ;) | 13:41 |
| gfidente | derekh, is it using the templates from /usr/share? | 13:42 |
| gfidente | derekh, I wanted to add the sysctl key | 13:42 |
| derekh | gfidente: yes it should be | 13:43 |
| *** liverpooler has quit IRC | 13:43 | |
| *** jtomasek_ has joined #tripleo | 13:43 | |
| *** liverpooler has joined #tripleo | 13:43 | |
| gfidente | done | 13:45 |
| bandini | gfidente: what sysctl are you changing? just so I know | 13:45 |
| gfidente | bandini, yeah there is screen running on the first host | 13:46 |
| gfidente | can you attach to it? | 13:46 |
| gfidente | derekh, ^^ | 13:46 |
| *** Marga_ has joined #tripleo | 13:46 | |
| bandini | sure | 13:47 |
| gfidente | I'm on window 0 | 13:47 |
| bandini | on it | 13:47 |
| derekh | gfidente: sorry, im in | 13:49 |
| gfidente | derekh, fine see screen | 13:50 |
| *** Marga_ has quit IRC | 13:50 | |
| *** shardy has joined #tripleo | 13:50 | |
| *** anande has quit IRC | 13:53 | |
| *** jtomasek_ has quit IRC | 13:54 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Add StackUpgradeManager https://review.openstack.org/286604 | 13:55 |
| * leanderthal sips a bit of bandini 's special wine. offers some to hewbrocca . . . | 13:56 | |
| *** mbound has quit IRC | 13:56 | |
| derekh | dprince: 2 of the periodic jobs passed last night for the first time, the ha one failed | 13:56 |
| bandini | eheh | 13:56 |
| derekh | dprince: Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: keystone-manage bootstrap --bootstrap-password Z9NgcwkR6rpNU2FRKaJuR7TAH returned 1 | 13:56 |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Add 'stack upgrade' command https://review.openstack.org/286606 | 14:00 |
| dprince | derekh: any logs we can get from the failed bootstrap? | 14:00 |
| slagle | derekh: that should be fixed, we were hitting that in the check jobs as well | 14:00 |
| dprince | meeting time | 14:00 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Create new partitioning element. https://review.openstack.org/259881 | 14:00 |
| derekh | dprince: http://tripleo.org/cistatus-periodic.html | 14:00 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-specs: TripleO Deployment Library with Mistral https://review.openstack.org/280407 | 14:00 |
| derekh | dprince: http://logs.openstack.org/periodic/periodic-tripleo-ci-f22-ha/4492ced/ | 14:00 |
| gfidente | shardy, agreed on not doing -n5 for event-list :) | 14:01 |
| gfidente | tks | 14:01 |
| slagle | derekh: https://bugs.launchpad.net/tripleo/+bug/1551501 | 14:01 |
| openstack | Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,Fix released] - Assigned to Emilien Macchi (emilienm) | 14:01 |
| shardy | gfidente: cool - that's actually a really expensive operation, so best avoided unless really needed :) | 14:01 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list https://review.openstack.org/286062 | 14:02 |
| *** derekh is now known as ndipanov_ | 14:04 | |
| *** ndipanov_ is now known as derekh | 14:04 | |
| derekh | slagle: ack, thanks, so it will hopfuly pass tonight | 14:06 |
| openstackgerrit | Merged openstack/instack-undercloud: Unset DIB_YUM_REPO_CONF before running undercloud install https://review.openstack.org/284444 | 14:07 |
| hewbrocca | oh thank god ^^^ | 14:08 |
| *** lucas-hungry is now known as lucasagomes | 14:09 | |
| *** lblanchard has joined #tripleo | 14:14 | |
| *** tzumainn has joined #tripleo | 14:15 | |
| *** mbound has joined #tripleo | 14:15 | |
| dtantsur | hey folks, daily request to review a small patch https://review.openstack.org/#/c/277974/ required for using python-hardware in profile matching :) | 14:17 |
| *** paramite is now known as paramite|afk | 14:21 | |
| *** paramite|afk is now known as paramite | 14:22 | |
| *** eggmaster has joined #tripleo | 14:22 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Convert the container job to a noop https://review.openstack.org/285325 | 14:22 |
| *** tiswanso has joined #tripleo | 14:23 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: compute: include VIR_MIGRATE_TUNNELLED when doing VM shared storage https://review.openstack.org/286584 | 14:28 |
| *** jroll has quit IRC | 14:29 | |
| *** jroll has joined #tripleo | 14:30 | |
| *** jroll has quit IRC | 14:30 | |
| *** jroll has joined #tripleo | 14:30 | |
| *** mbound has quit IRC | 14:30 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: compute: include VIR_MIGRATE_TUNNELLED when doing VM shared storage https://review.openstack.org/286584 | 14:31 |
| *** shardy has quit IRC | 14:33 | |
| *** mbound has joined #tripleo | 14:34 | |
| *** shardy has joined #tripleo | 14:35 | |
| *** mgould has quit IRC | 14:35 | |
| *** sshnaidm has quit IRC | 14:35 | |
| *** weshay has joined #tripleo | 14:36 | |
| *** d0ugal has quit IRC | 14:39 | |
| *** d0ugal has joined #tripleo | 14:40 | |
| *** d0ugal has quit IRC | 14:40 | |
| *** trozet has joined #tripleo | 14:42 | |
| *** akuznetsov has joined #tripleo | 14:49 | |
| *** mgould has joined #tripleo | 14:50 | |
| *** akuznetsov has quit IRC | 14:54 | |
| hewbrocca | gfidente, derekh how's the CI looking? Time is slipping away :) | 14:59 |
| gfidente | hewbrocca, we're looking into it with bandini | 14:59 |
| hewbrocca | tick tock :D | 15:00 |
| *** shardy has quit IRC | 15:00 | |
| * bandini feels the manager pressure | 15:00 | |
| derekh | gfidente: that env is going to dissapear at some stage as the ci command was killed | 15:01 |
| dprince | michchap: hi, so we are (hopefully) getting ready to start work on composable services | 15:01 |
| derekh | gfidente: If it does, I kept the overcloud images so it wont take as long to setup again | 15:01 |
| dprince | michchap: https://review.openstack.org/#/c/245804/ | 15:01 |
| bandini | derekh: how much time do we have? | 15:02 |
| dprince | michchap: that work would move around quite a bit of the puppet code. Likely out of t-h-t and into puppet-tripleo | 15:02 |
| derekh | bandini: no way of knowing, whenever the env gets handed out to a new ci job | 15:02 |
| michchap | dprince: yep that's all I wanted to do initially. I'm on the opnfv team so our interest is mostly being able to easily add/modify the network profiles | 15:02 |
| bnemec | gfidente: Back to the net-iso job, have you looked at the most recent failure? | 15:03 |
| bnemec | Trying to ping default gateway 10.0.0.1...FAILURE\n10.0.0.1 is not pingable. | 15:03 |
| bandini | derekh: it just did :) | 15:03 |
| *** dustins has joined #tripleo | 15:03 | |
| bnemec | With the default single-nic-vlans setup the external network is on vlan 10, with a CIDR of 10.0.0.0/24. | 15:03 |
| dprince | michchap: yep, and one of main benefits of composable services that teams don't have to deal as much with the larger architecture, just their services | 15:03 |
| bnemec | You _have_ add that interface on the undercloud, unless we're going to do a completely custom network-isolation configuration. | 15:03 |
| derekh | gfidente: bandini: ok, restarting it, leave tab 3 running, thats what was keeping the env held | 15:04 |
| dprince | michchap: there is a lot of debate on the spec ATM about how we run puppet though (all in one manifest vs. smaller snippets). I really would like to see the TripleO core team agree on these things before we do any of this work | 15:04 |
| michchap | dprince: I'll have a read through it and leave some feedback there, thanks. | 15:04 |
| slagle | dprince: i'm in agreement now on running all one manifest | 15:05 |
| gfidente | derekh, which screen window shall we use? | 15:05 |
| michchap | dprince: smaller snippets leads to pain | 15:05 |
| dprince | michchap: cool, thanks | 15:05 |
| derekh | gfidente: any one but 3, that were I had run the ./toci_gate_test.sh command that got killed | 15:06 |
| slagle | dprince: it's just i'd like to see the concatenation avoided in the top level overcloud.yaml, but if we can't then we can't | 15:06 |
| dprince | slagle: okay, perhaps it is time I made another pass at fixing some of the other comments. | 15:06 |
| dprince | slagle: sure, I view that as sort of an implementation detail in that so long as it doesn't effect the "interfaces" (how services can plug into our deployment architecture) I think we could refactor it later | 15:06 |
| *** yamahata has joined #tripleo | 15:07 | |
| slagle | ok | 15:08 |
| *** Goneri has joined #tripleo | 15:10 | |
| gfidente | derekh, is instack the same host? | 15:11 |
| slagle | bnemec: success, https://review.openstack.org/#/c/285509/ | 15:11 |
| gfidente | I wanted to edit the images | 15:11 |
| slagle | bnemec: i say we merge that and the previous one | 15:11 |
| bnemec | slagle: Agreed. | 15:12 |
| dprince | gfidente: where are you with debugging the netiso job? | 15:12 |
| derekh | gfidente: 192.168.1.24 | 15:12 |
| dprince | gfidente,derekh: what can I help w/ | 15:12 |
| bnemec | slagle: Do you want to do the honors on https://review.openstack.org/#/c/269827 since I proposed it? | 15:13 |
| derekh | gfidente: images are in /home/jenkins | 15:13 |
| bnemec | There is no way the current netiso patch will ever pass CI. | 15:13 |
| bnemec | Are you guys working on a different version of it or something? | 15:13 |
| slagle | bnemec: done | 15:14 |
| bnemec | gfidente: derekh: ^ | 15:14 |
| derekh | bnemec: this is the patch we're trying, https://review.openstack.org/#/c/273424/ | 15:15 |
| dprince | bnemec: it could work if we used ctlplane for public (tenant) traffic I think | 15:15 |
| bnemec | dprince: Which is not what that patch does. | 15:15 |
| gfidente | bnemec, we can add the interface anyway, we're not there yet | 15:15 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Revert "Revert "neutron: enable nova-event-callback by default"" https://review.openstack.org/269827 | 15:15 |
| derekh | bnemec: but /me doesn't know the details, I've mainy been helping by getting access to various machines for debugging etc.. | 15:15 |
| bnemec | The CI failure on that patch is clear. I'm not sure what needs to be debugged at the moment. | 15:16 |
| dprince | bnemec: last I saw puppet was hanging | 15:16 |
| dprince | bnemec: and we don't know exactly why that was yet | 15:16 |
| bnemec | dprince: That was on the previous version. | 15:16 |
| dprince | gfidente: what was the puppet error? | 15:17 |
| bnemec | The current one is failing on Trying to ping default gateway 10.0.0.1...FAILURE\n10.0.0.1 is not pingable. | 15:17 |
| bnemec | Because 10.0.0.1 is not being configured on the undercloud anymore. | 15:17 |
| *** dmacpher has joined #tripleo | 15:17 | |
| dprince | bnemec: oh, yeah, well that is the public vlan then | 15:17 |
| gfidente | bnemec, dprince so we can add a fake 10.0.0.1 to make validation to pass | 15:18 |
| dprince | bnemec: sorry, I missed something that gfidente did to get puppet output | 15:18 |
| gfidente | we can also change the default gateway to something else so that validation will pass | 15:18 |
| gfidente | I will update the submission adding back 10.0.0.1 | 15:18 |
| gfidente | but what we're debugging is why corosync on the nodes can't talk to each other | 15:18 |
| dprince | gfidente: rather than just focus on HA, could we do this instead? https://review.openstack.org/#/c/285674/4 | 15:19 |
| gfidente | or why computes can't reach rabbit on the controller | 15:19 |
| dprince | gfidente: in my patch the compute was failing to run puppet. But the controller seemed to complete... | 15:19 |
| gfidente | dprince, in nonha the controller is only one | 15:19 |
| dprince | gfidente: I'd like to focus on the simplest case which is a failure first | 15:19 |
| gfidente | dprince, yes agreed | 15:20 |
| gfidente | I am only saying that if controller worked in nonha is probably because there is no communication with other nodes involved | 15:20 |
| gfidente | until it gets to compute | 15:20 |
| dprince | gfidente: agree, that is why | 15:20 |
| gfidente | but it's still simpler case | 15:20 |
| dprince | gfidente: but it is a simpler case so I'd rather start there... | 15:20 |
| gfidente | yeah | 15:20 |
| dprince | gfidente: did you do something to enable the puppet output? | 15:20 |
| gfidente | dprince, no we were looking at the testenv which derekh brought up | 15:21 |
| dprince | gfidente: I see. So you guys are off in the weeds :) | 15:21 |
| gfidente | yes! :( | 15:22 |
| gfidente | I could verify that ping on the vlans work though | 15:22 |
| *** pradk has quit IRC | 15:22 | |
| gfidente | bandini, saw traffic with tcpdump | 15:22 |
| dprince | gfidente: yes, the validation (which is now fixed) tests that | 15:22 |
| *** pradk has joined #tripleo | 15:22 | |
| dprince | gfidente: I didn't question that. I'd like to see more specifically the puppet output from the compute failure. Where it fails and why | 15:23 |
| gfidente | dprince, yes I'll run puppet manually on the new env so we have the puppet output | 15:23 |
| dprince | gfidente: is this iptables on the controller blocking it for example? | 15:23 |
| gfidente | dprince, on the ha scenario iptables was empty cause puppet failed because setting it up | 15:24 |
| *** dustins_ has joined #tripleo | 15:28 | |
| *** dustins has quit IRC | 15:29 | |
| *** rpothier has joined #tripleo | 15:29 | |
| bandini | derekh: just making sure 192.168.1.24 is the undercloud where I can debug the pcsd/corosync issue. correct? | 15:33 |
| bandini | and how much time will I have to play with the env once it starts deploying the overcloud? | 15:33 |
| derekh | bandini: yes correct IP, env should stay up a few hours once tab 3 isn't killed | 15:34 |
| *** sshnaidm has joined #tripleo | 15:34 | |
| derekh | bandini: on a call, back in a few | 15:34 |
| bandini | derekh: ack, thanks | 15:34 |
| *** aufi has joined #tripleo | 15:35 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Set project name for nova -> neutron notifications https://review.openstack.org/285509 | 15:37 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 15:37 |
| gfidente | dprince, bnemec, ^^ I brought back in the NETISO flag | 15:37 |
| *** mcornea_ has joined #tripleo | 15:37 | |
| *** pblaho has quit IRC | 15:37 | |
| *** panda has quit IRC | 15:37 | |
| *** mcornea_ has quit IRC | 15:37 | |
| *** mcornea_ has joined #tripleo | 15:38 | |
| *** panda has joined #tripleo | 15:38 | |
| gfidente | except I did it in the wrong change | 15:39 |
| bnemec | slagle: Did we just break Liberty CI then? How could that have merge conflicted? :-/ | 15:39 |
| *** mbound has quit IRC | 15:40 | |
| *** mbound has joined #tripleo | 15:40 | |
| *** mcornea has quit IRC | 15:40 | |
| slagle | bnemec: not sure. there was no conflict when i rebased it locally | 15:40 |
| slagle | i +A'd it again | 15:40 |
| bnemec | slagle: Annoying. I intentionally approved that patch first so they would merge together. Oh well. | 15:41 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 15:43 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 15:43 |
| gfidente | ^^ better now | 15:43 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Enable network isolation on all the jobs. https://review.openstack.org/285674 | 15:45 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend https://review.openstack.org/252032 | 15:47 |
| *** devvesa has quit IRC | 15:49 | |
| *** masco has quit IRC | 15:49 | |
| *** akuznetsov has joined #tripleo | 15:50 | |
| *** shardy has joined #tripleo | 15:51 | |
| *** yamahata has quit IRC | 15:51 | |
| *** akuznetsov has quit IRC | 15:55 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone https://review.openstack.org/286344 | 15:58 |
| *** rpothier has left #tripleo | 15:59 | |
| hewbrocca | pradk: I keep seeing promising-looking reviews | 16:03 |
| hewbrocca | pradk: are you also working on AODH? | 16:03 |
| pradk | hewbrocca, yea | 16:03 |
| hewbrocca | pradk: do you need help? | 16:03 |
| pradk | hewbrocca, i think its in a pretty decent shape now, if i can get some reviews that would be super helpful. The ci keeps running out of memory on oc nodes though. | 16:04 |
| hewbrocca | Right | 16:04 |
| pradk | hewbrocca, had an email conversation with slagle, derekh and dprince to see if we can bump the memory | 16:05 |
| bnemec | hewbrocca: Can you look at https://etherpad.openstack.org/p/tripleo-mitaka-rc-blockers and let us know if there's anything missing that is a drop-dead requirement in Mitaka? | 16:05 |
| *** masco has joined #tripleo | 16:05 | |
| *** jaosorior is now known as jaosorior_away | 16:06 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Set project name for nova -> neutron notifications https://review.openstack.org/285509 | 16:07 |
| *** chlong_ has quit IRC | 16:10 | |
| *** dmacpher is now known as dmacpher-afk | 16:11 | |
| *** aufi has quit IRC | 16:12 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone https://review.openstack.org/286344 | 16:12 |
| *** paramite has quit IRC | 16:14 | |
| *** liverpooler has quit IRC | 16:15 | |
| gfidente | derekh, dprince so we can ping on the vlans but we can't telnet (ssh) the other nodes it seems | 16:16 |
| *** dcain has joined #tripleo | 16:17 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone https://review.openstack.org/286344 | 16:17 |
| gfidente | derekh, join screen on window 0 | 16:17 |
| dprince | gfidente: which machine? | 16:17 |
| dprince | gfidente: which testenv? | 16:17 |
| gfidente | ssh jenkins@66.187.229.70 | 16:18 |
| gfidente | then screen -x | 16:18 |
| *** leanderthal is now known as leanderthal|afk | 16:19 | |
| dprince | gfidente: add my key? https://launchpad.net/~dan-prince/+sshkeys | 16:19 |
| gfidente | dprince, done | 16:20 |
| hewbrocca | bnemec: added upgrades | 16:21 |
| bnemec | hewbrocca: Good call, thanks. | 16:22 |
| hewbrocca | Oh -- predictable hostnames | 16:24 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker: Basic beaker one node test. https://review.openstack.org/281376 | 16:24 |
| hewbrocca | also keystone + wsgi (requires pcmk constraint changes) | 16:24 |
| *** olap has quit IRC | 16:24 | |
| *** yamahata has joined #tripleo | 16:25 | |
| gfidente | dprince, with bandini we were thinking about packets being propagate on other testenvs on the same baremetal | 16:27 |
| bnemec | hewbrocca: I think that's covered by the HostnameMap change above. And I thought we had the Keystone change merged, is that just fixing the constraints? | 16:27 |
| *** xinwu has joined #tripleo | 16:27 | |
| derekh | gfidente: ping works but ssh doesn't, sounds like what happens to me at home when I have MTU issues | 16:27 |
| hewbrocca | bnemec: yeah, we gotta get the constraints fixed | 16:27 |
| hewbrocca | bandini: is working on that | 16:27 |
| gfidente | derekh, MTU issues good hint | 16:28 |
| *** dustins_ is now known as dustins | 16:28 | |
| bandini | dprince: http://acksyn.org/files/tripleo/pcs-debug.pcap here is the network traffic on vlan20 port 2224 (of pcs's traffic during setup). if you look at it there is something rather fishy (truckloads of retransmits on syn, as if other hosts with the same ip were on the same segment) | 16:28 |
| bandini | but yeah it could be mtu issues as well | 16:29 |
| dprince | derekh, bandini: but we don't change MTU in our CI anywhere that I know of from the defaults | 16:32 |
| *** absubram has joined #tripleo | 16:32 | |
| bandini | dprince: ack. is the hypotesis of other hosts with the same ip addresses being present on the same segment a viable one? | 16:33 |
| dprince | bandini: each test environment has its own bridge | 16:33 |
| *** absubram_ has joined #tripleo | 16:33 | |
| dprince | bandini: which should isolate the traffic from one another... | 16:34 |
| gfidente | dprince, though all underclouds share a nic on the baremetal bridge | 16:34 |
| *** tiswanso has quit IRC | 16:34 | |
| gfidente | but traffic shouldn't go that far | 16:34 |
| bandini | dprince: I see, I am a bit out of ideas ;) it clearly is something fishy with the network (mtu and duplicate hosts were my only hypothesis so far) | 16:35 |
| dprince | bandini: I guess it is possible | 16:35 |
| *** tiswanso has joined #tripleo | 16:35 | |
| dprince | bandini: we can switch the MTU as a test | 16:35 |
| EmilienM | dprince: wdyt about enabling puppet parser future in tripleo? | 16:35 |
| gfidente | dprince, yeah I was looking into the nic templates to try that | 16:35 |
| EmilienM | that would be so nice | 16:35 |
| bandini | dprince: aye, giulio is looking into that | 16:35 |
| EmilienM | dprince: https://docs.puppetlabs.com/puppet/3/reference/experiments_future.html | 16:35 |
| EmilienM | dprince: so we could iterate data within manifests | 16:36 |
| dprince | gfidente: it'll be a custom patch I think to t-h-t | 16:36 |
| *** yamahata has quit IRC | 16:36 | |
| EmilienM | dprince: though we might need puppet 4.1 | 16:36 |
| EmilienM | (at least) | 16:36 |
| dprince | EmilienM: yeah, I'm mixed | 16:37 |
| *** absubram has quit IRC | 16:37 | |
| *** absubram_ is now known as absubram | 16:37 | |
| dprince | EmilienM: is there some language feature you really want to use? | 16:37 |
| EmilienM | dprince: but the features are awesome | 16:37 |
| EmilienM | yeah | 16:37 |
| EmilienM | dprince: example: https://review.openstack.org/286344 | 16:37 |
| EmilienM | dprince: with future parser I could have done a patch in 4 lines | 16:37 |
| gfidente | derekh, can I retrigger overcloud deploy manually or is the env going away? | 16:38 |
| derekh | gfidente: you can retrigger it, as long as the command in tab 3 isn't kill the env will be kept | 16:39 |
| derekh | gfidente: we'll it will timeout eventually but you should be good for a few hours | 16:40 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: endpoint_map: bump nova api from 2.0 to 2.1 https://review.openstack.org/263366 | 16:43 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: WIP: Bump down the MTU for the single-nic-with-vlans https://review.openstack.org/286707 | 16:44 |
| *** dustins has quit IRC | 16:49 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test to periodic job https://review.openstack.org/279218 | 16:52 |
| EmilienM | dprince: what prevents us to use parser? | 16:52 |
| EmilienM | could we give it a try? | 16:52 |
| gfidente | dprince, ack, we did the edits manually and are testing that already | 16:52 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Let Puppet update all packages on non-controllers https://review.openstack.org/284489 | 16:53 |
| dprince | EmilienM: it could be fine. but I'd rather put the effort into composable service roles at this point and do the refactoring there | 16:54 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job https://review.openstack.org/273743 | 16:54 |
| *** rcernin has quit IRC | 16:55 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/268528 | 16:56 |
| EmilienM | dprince: Sure, what can I do? | 16:56 |
| *** mikelk has quit IRC | 16:56 | |
| *** masco has quit IRC | 16:58 | |
| *** mbound has quit IRC | 17:01 | |
| EmilienM | dprince: this is an experiment: https://review.openstack.org/286714 | 17:01 |
| dprince | gfidente: can I make the CI patch depend on this? https://review.openstack.org/#/c/286707/ | 17:02 |
| gfidente | dprince sure but on the host it's set to 1400 on all vlans but telnet still not going | 17:02 |
| gfidente | check screen | 17:02 |
| gfidente | derekh, ^^ | 17:03 |
| derekh | gfidente: which address are you ssh'ing to? | 17:04 |
| gfidente | we're trying ssh over internal_api | 17:04 |
| gfidente | in between controller/compute | 17:04 |
| *** ifarkas has quit IRC | 17:05 | |
| *** jistr has quit IRC | 17:05 | |
| gfidente | derekh, on window 0 | 17:06 |
| *** pcaruana has quit IRC | 17:07 | |
| derekh | gfidente: ack, just looking around on another tab at the moment | 17:07 |
| *** mcornea_ has quit IRC | 17:08 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Enable notifications on the overcloud https://review.openstack.org/286299 | 17:08 |
| EmilienM | slagle: do you know if I can run OOO jobs on openstack/heat-templates patches? | 17:09 |
| EmilienM | I don't see the jobs in the zuul layout, neither in experimental | 17:09 |
| EmilienM | slagle: can I run a patch in THT and use depends-on? | 17:09 |
| hewbrocca | dsneddon: over here ^^^ | 17:10 |
| derekh | gfidente: so I'm running tcpdump on the compute node, it is getting the tcp setup traffic for ssh | 17:11 |
| gfidente | so I never got the ack to telnet though | 17:12 |
| *** stendulker has joined #tripleo | 17:13 | |
| derekh | gfidente: ack, | 17:13 |
| *** xinwu has quit IRC | 17:18 | |
| derekh | gfidente: weird [heat-admin@overcloud-controller-0 ~]$ ssh 172.16.2.6 | 17:20 |
| derekh | Are you sure you want to continue connecting (yes/no)? ^C | 17:20 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Enable per-role SchedulerHints https://review.openstack.org/266930 | 17:20 |
| derekh | gfidente: ^ that what isn't working isn't it ? | 17:20 |
| *** athomas has quit IRC | 17:21 | |
| bandini | derekh: exactly, sometimes it does work (although most of the times it will just hang there) | 17:21 |
| derekh | bandini: ack | 17:21 |
| gfidente | derekh, does it work consistently? | 17:21 |
| derekh | gfidente: nope, I just thought it was weird that it worked once | 17:22 |
| gfidente | ah yeah | 17:22 |
| EmilienM | derekh, dprince: do you know how can I test https://review.openstack.org/#/c/286714/ with tripleo jobs? patching THT with a Depends-On would work? do we use zuul-cloner to pull heat-templates? | 17:25 |
| *** ohamada has quit IRC | 17:26 | |
| *** athomas has joined #tripleo | 17:26 | |
| derekh | EmilienM: a depends on will get you the correct patch on the jenkins node, do we actually us it though | 17:29 |
| *** dtantsur is now known as dtantsur|afk | 17:29 | |
| EmilienM | derekh: the question is: do we use zuul cloner to clone openstack/heat-templates | 17:29 |
| derekh | gfidente: I'm going to have to run in about 10 minutes, if that env gets reclaimed you can rerun the command in tab 3 | 17:30 |
| gfidente | derekh, ok thanks | 17:30 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 17:30 |
| derekh | gfidente: I can join again later tonight (in about 4 or 5 hour) if you want me to do anything send me a mail | 17:30 |
| *** tiswanso has quit IRC | 17:30 | |
| gfidente | derekh, ack | 17:31 |
| derekh | EmilienM: we're using the normal devstack-gate scripts, so I believe the answer to your question is yes, as that uses zuul-cloner doesn't it? | 17:31 |
| gfidente | derekh, last thing, can we get on the baremetal node hosting this environment somehow? | 17:32 |
| EmilienM | ok | 17:32 |
| EmilienM | derekh: fair enough, I'll try | 17:32 |
| *** tiswanso has joined #tripleo | 17:32 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Test Puppet Parser Future - Do not merge https://review.openstack.org/286732 | 17:33 |
| shardy | bnemec: Hey, good to hear you're trying the scheduler hints stuff, perhaps you can leave feedback re your findings on the docs patch https://review.openstack.org/#/c/274217 :) | 17:34 |
| shardy | I tested it a while back and all seemed OK, then I re-tested when I wrote the docs and saw inconsistent results, e.g the nodes weren't pinned as expected | 17:35 |
| derekh | gfidente: the host you want to get to is testenv6-testenv3-t2oqwxetgmbd , dprince can you see if you can help gfidente out with that, I did it earlier by letting him use my account but puppet periodically changes the ssh keys back | 17:35 |
| bnemec | shardy: Yeah, I'm finding that the scheduler hints aren't showing up in the ComputeCapabilitiesFilter. | 17:35 |
| bnemec | Still trying to figure out why. | 17:35 |
| shardy | bnemec: Hmm, that would explain it | 17:35 |
| shardy | bnemec: Are you running master tripleo or an older build? | 17:36 |
| bnemec | shardy: master tripleo | 17:36 |
| shardy | I just rebased the liberty "Enable per-role SchedulerHints" patch, but I've mostly been testing on master | 17:36 |
| *** trown is now known as trown|lunch | 17:37 | |
| openstackgerrit | Richard Su proposed openstack/instack-undercloud: Option to store events in Undercloud Ceilometer https://review.openstack.org/286734 | 17:37 |
| *** derekh has quit IRC | 17:43 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add HostnameMap to allow granular control of hostnames https://review.openstack.org/286739 | 17:44 |
| *** stendulker has quit IRC | 17:45 | |
| *** absubram has quit IRC | 17:48 | |
| openstackgerrit | James Slagle proposed openstack/instack-undercloud: Set max_resources_per_stack to -1 https://review.openstack.org/286746 | 17:55 |
| openstackgerrit | James Slagle proposed openstack/instack-undercloud: Set max_resources_per_stack to -1 https://review.openstack.org/286746 | 17:56 |
| *** fgimenez has quit IRC | 17:57 | |
| *** hjensas has quit IRC | 17:58 | |
| hewbrocca | dprince: So... what's our fallback if we can't get net-iso CI going today? | 18:00 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Update the path to the Docker file https://review.openstack.org/264137 | 18:01 |
| *** hjensas has joined #tripleo | 18:02 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-puppet-elements: Gnocchi Integration https://review.openstack.org/238013 | 18:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Enable heat-manage purge_deleted cron job https://review.openstack.org/279342 | 18:03 |
| *** penick has joined #tripleo | 18:05 | |
| EmilienM | any change to get https://review.openstack.org/#/c/286584/ landed? | 18:05 |
| *** rdopiera has quit IRC | 18:05 | |
| dprince | hewbrocca: manual testing? | 18:06 |
| EmilienM | and https://review.openstack.org/#/c/286031/ | 18:06 |
| openstackgerrit | Merged openstack/instack-undercloud: Set max_resources_per_stack to -1 https://review.openstack.org/286165 | 18:06 |
| dprince | hewbrocca: when people aren't doing AFAIKT. At least not with upstream... | 18:06 |
| dprince | hewbrocca: which people aren't doing | 18:06 |
| hewbrocca | OK | 18:06 |
| dprince | hewbrocca: still working the the CI job in the meantime | 18:07 |
| hewbrocca | EmilienM: ^^^ that's what we're going to have to do then, if we don't get the CI job going | 18:07 |
| hewbrocca | we can't afford to hold off any longer | 18:07 |
| hewbrocca | Thanks guys. I have to head out, I will see you all tomorrow | 18:07 |
| *** xinwu has joined #tripleo | 18:07 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Make debootstrap cache opt-in https://review.openstack.org/285886 | 18:11 |
| *** openstackgerrit has quit IRC | 18:18 | |
| *** openstackgerrit has joined #tripleo | 18:18 | |
| *** lucasagomes is now known as lucas-dinner | 18:22 | |
| gfidente | hewbrocca, dprince so we were trying to do some tcpdump on the baremetal with bandini | 18:24 |
| gfidente | we do see packets going | 18:24 |
| bandini | but stupidly slow...we see like 1 second delay to reply to a syn | 18:25 |
| bandini | and then it increases and and then sometimes it just gets stuck it seems | 18:25 |
| gfidente | and now we tried again and I don't see acks coming back at all | 18:26 |
| gfidente | the only difference I could find from a working environment is that ovs, on the baremetal | 18:26 |
| gfidente | seems to not be forwarding traffic to all ports but only on the selected ports where the sender/receiver are? | 18:27 |
| gfidente | I am not sure if this is a behaviour we can configure nor if it is relevant, I just see it behaving differently | 18:27 |
| *** weshay has quit IRC | 18:27 | |
| gfidente | so I am sorry but basically I still don't know what to blame | 18:28 |
| *** jaosorior_away is now known as jaosorior | 18:31 | |
| dprince | gfidente: the latest CI results for non-ha and Ceph show stack completed successfully | 18:31 |
| gfidente | seriously? | 18:32 |
| dprince | gfidente: CI job still fails but the heat stack was created | 18:32 |
| *** mgould has quit IRC | 18:32 | |
| dprince | gfidente: just noticed this: | 18:33 |
| dprince | 2016-03-01 17:22:59.688 | | tenant-stack | 43a543af-7aab-4d76-bb92-3d4f4ffde482 | Resource CREATE failed: BadRequest: resources.router_interface: Bad router request: Cidr 10.0.0.0/8 of subnet 6278d068-8a92-467b-b9ed-93dac23c2425 overlaps with cidr 10.0.0.0/24 of subnet 984b559e-d9d5-4ea4-91da-44711764b5c3. | CREATE_FAILED | 2016-03-01T17:03:42 | | 18:33 |
| dprince | gfidente: we've got a /8 somewhere? | 18:34 |
| gfidente | the overcloud ping test uses 10.0.0.0/8 it seems | 18:35 |
| gfidente | I suspect it's creating it in the wrong "cloud" | 18:35 |
| *** trown|lunch is now known as trown | 18:35 | |
| dprince | gfidente: still the jobs are getting quite far | 18:35 |
| dprince | gfidente: I'm not sure exactly what you and derekh did this morning to create this custom environment | 18:36 |
| dprince | gfidente: could you run me through that? because, I'm wondering if perhaps it is related to the errors you see now? | 18:36 |
| *** hjensas has quit IRC | 18:37 | |
| gfidente | dprince, I think it's a regular CI job where he purposely set a longer timeout | 18:37 |
| gfidente | it's running on one of the existing testenvs | 18:38 |
| dprince | gfidente: and you are attaching via the jenkins slave? | 18:38 |
| gfidente | dprince, yes | 18:38 |
| *** dustins has joined #tripleo | 18:38 | |
| dprince | gfidente: okay, shouldn't be anything too odd about that | 18:38 |
| gfidente | but are the CI jobs you see passing from https://review.openstack.org/#/c/285674/ ? | 18:39 |
| dprince | gfidente: if we get either ceph or nonha passing I'm going to say lets just go w/ it | 18:39 |
| dprince | http://logs.openstack.org/24/273424/16/check-tripleo/gate-tripleo-ci-f22-nonha/5504090/console.html | 18:40 |
| dprince | gfidente: no, your most recent patch which adopted mine | 18:40 |
| dprince | gfidente: Now that you enabled all jobs I'll ditch mine | 18:40 |
| gfidente | no wait no | 18:40 |
| gfidente | those don't run netiso on nonha and ceph | 18:40 |
| dprince | gfidente: seriously, why? | 18:41 |
| gfidente | I defaulted to 1 NETISO but it only creates the vlan10 on the underclloud | 18:41 |
| gfidente | I was not passing the envs to the other jobs too | 18:41 |
| dprince | gfidente: why are we waisting time and resources on this. PLus it is just really confusing not to look at them all at once | 18:41 |
| dprince | gfidente: lets do them all, until we see one passing | 18:41 |
| dprince | gfidente: you're previous patch did that no? | 18:42 |
| gfidente | so I didn't enable it for all jobs because I thought we want to check non netiso too | 18:42 |
| dprince | gfidente: we do, This is a WIP | 18:42 |
| gfidente | but at least for the purpose of testing your submission is better as it enables it for all | 18:42 |
| gfidente | https://review.openstack.org/#/c/285674 | 18:43 |
| dprince | gfidente: forget about the -1's complaining about enabling it for all jobs and call this a WIP | 18:43 |
| gfidente | so last thing I can do before going is rebase on the MTU change | 18:43 |
| dprince | gfidente: I can do it | 18:43 |
| gfidente | ok | 18:43 |
| dprince | gfidente: if you need to sign off no worries | 18:43 |
| gfidente | I think the forwarding issue on the baremetal is the real thing | 18:44 |
| gfidente | we were looking with wireshark at pcap files | 18:44 |
| gfidente | the syns are retransmitted, the acks too | 18:44 |
| gfidente | as if things were just extremely slow | 18:44 |
| dprince | gfidente: we could just be hitting an old OVS bug or something | 18:45 |
| gfidente | so tcp connections time out, yet ping works | 18:45 |
| dsneddon | gfidente, That sounds like MTU | 18:45 |
| gfidente | dsneddon, hey! | 18:46 |
| gfidente | the environment we're looking at now has mtu set to 1400 on the overcloud nodes | 18:46 |
| dsneddon | gfidente, I've been reading along for a while, just didn't have any helpful suggestions | 18:46 |
| dprince | gfidente: okay, so rebase to try this https://review.openstack.org/#/c/286707/ | 18:46 |
| gfidente | dprince, ack | 18:48 |
| gfidente | then I'll leave guys | 18:48 |
| *** akuznetsov has joined #tripleo | 18:48 | |
| dprince | dsneddon: could you have a gander at https://review.openstack.org/#/c/286707/ and see if it is missing anything. | 18:48 |
| gfidente | I am sorry but ... couldn't figure much more | 18:48 |
| dprince | dsneddon: we only need the 3 nodes | 18:48 |
| gfidente | dprince, we tested that identical change and it works | 18:48 |
| dprince | dsneddon: 3 roles | 18:50 |
| gfidente | the vlan interfaces have mtu 1400 | 18:50 |
| dsneddon | gfidente, You have to set the MTU on the bridge, too | 18:50 |
| gfidente | it just doesn't change the end result | 18:50 |
| gfidente | dsneddon, ok I will | 18:50 |
| *** tserong has quit IRC | 18:52 | |
| dsneddon | gfidente, And the interface, too | 18:52 |
| dsneddon | gfidente, It has to be set at all levels. | 18:52 |
| *** afazekas has quit IRC | 18:52 | |
| *** SpamapS has quit IRC | 18:54 | |
| *** openstack has joined #tripleo | 19:14 | |
| *** SpamapS has joined #tripleo | 19:15 | |
| *** openstackstatus has joined #tripleo | 19:18 | |
| *** ChanServ sets mode: +v openstackstatus | 19:18 | |
| *** openstackgerrit has joined #tripleo | 19:19 | |
| *** hjensas has quit IRC | 19:25 | |
| *** CaptTofu has quit IRC | 19:25 | |
| *** StevenK has quit IRC | 19:25 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Support adding swap to overcloud nodes https://review.openstack.org/273752 | 19:27 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Support adding a swap file to overcloud nodes https://review.openstack.org/273752 | 19:29 |
| *** CaptTofu has joined #tripleo | 19:31 | |
| *** StevenK has joined #tripleo | 19:31 | |
| *** jrist has quit IRC | 19:31 | |
| *** CaptTofu has quit IRC | 19:31 | |
| *** hjensas has joined #tripleo | 19:32 | |
| *** hjensas has quit IRC | 19:32 | |
| *** hjensas has joined #tripleo | 19:32 | |
| *** CaptTofu has joined #tripleo | 19:33 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI https://review.openstack.org/286793 | 19:38 |
| *** panda has quit IRC | 19:38 | |
| *** panda has joined #tripleo | 19:39 | |
| slagle | pradk: this is the patch you can try a Depends-On for the adoh/gnocchi patches, https://review.openstack.org/#/c/286793 | 19:40 |
| slagle | it should result in a 4GB swapfile getting added to the oc nodes | 19:40 |
| pradk | slagle, great, will do now | 19:40 |
| pradk | thx! | 19:40 |
| slagle | np. i hope it works | 19:41 |
| *** akuznetsov has joined #tripleo | 19:42 | |
| *** ccamacho has quit IRC | 19:43 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/241408 | 19:44 |
| *** jprovazn has quit IRC | 19:48 | |
| ayoung | Mad props to you guys...very happy with the tripleo.sh script and triple-common....been setting up and tearing down all day | 19:55 |
| ayoung | I think we might need to up the minimum sizeof a VM, though | 19:55 |
| ayoung | I had deploys failing before setting the node size to 16 G. Trying now with 8, but 4 is too small | 19:55 |
| *** jrist has joined #tripleo | 19:55 | |
| ayoung | I think that was just the controller... | 19:55 |
| *** david-lyle has quit IRC | 19:59 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: WIP: Enable network isolation in all CI jobs https://review.openstack.org/273424 | 20:02 |
| *** xinwu has quit IRC | 20:05 | |
| *** julim has quit IRC | 20:07 | |
| *** julim has joined #tripleo | 20:08 | |
| *** jcoufal_ has quit IRC | 20:08 | |
| *** david-lyle has joined #tripleo | 20:09 | |
| *** julim has quit IRC | 20:10 | |
| *** jaosorior has quit IRC | 20:12 | |
| *** mburned is now known as mburned_out | 20:17 | |
| *** mburned_out is now known as mburned | 20:19 | |
| *** mburned is now known as mburned_out | 20:20 | |
| *** yamahata has quit IRC | 20:21 | |
| *** ccamacho has joined #tripleo | 20:27 | |
| *** akuznetsov has quit IRC | 20:29 | |
| *** penick has quit IRC | 20:38 | |
| *** ccamacho has quit IRC | 20:38 | |
| *** ccamacho has joined #tripleo | 20:38 | |
| *** yamahata has joined #tripleo | 20:39 | |
| *** mburned_out is now known as mburned | 20:40 | |
| *** pcaruana has joined #tripleo | 20:44 | |
| *** xinwu has joined #tripleo | 20:47 | |
| *** yamahata has quit IRC | 20:47 | |
| openstackgerrit | Merged openstack/instack-undercloud: Increase scheduler_max_attempts on undercloud and make it configurable https://review.openstack.org/286008 | 20:50 |
| *** julim has joined #tripleo | 20:55 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI https://review.openstack.org/286793 | 20:58 |
| openstackgerrit | John Trowbridge proposed openstack/instack-undercloud: Deploy Monitoring on the undercloud with Puppet https://review.openstack.org/276127 | 20:58 |
| *** ccamacho has quit IRC | 20:58 | |
| *** ccamacho has joined #tripleo | 21:04 | |
| *** weshay has joined #tripleo | 21:04 | |
| dprince | This would be helpful to the UI developers: https://review.openstack.org/#/c/283221/ | 21:04 |
| dsneddon | EmilienM, I'm currently developing two alternate solutions to outputting both IPv4 and IPv6 IPs for external network and Public API, not sure which will end up being the best. | 21:08 |
| dsneddon | EmilienM, In one case, I create a second external network, and create new ports everywhere for that network (but they default to noop,yaml, so will output the control plane IP/VIP). | 21:08 |
| EmilienM | dsneddon: cool. I just need an array whatever your alternative :P | 21:08 |
| *** david-lyle has quit IRC | 21:08 | |
| dsneddon | EmilienM, In the other case, I create two subnets on the same network. Neutron then gives the port 2 IPs, one IPv6 and one IPv4. I just haven't figured out how to make the external load balancer work with that, where we preselect the controller IPs. | 21:09 |
| EmilienM | first solution seems ok to me | 21:10 |
| dsneddon | EmilienM, Yeah, but then the first solution creates a lot more cruft for the default case (extra ports and parameters in more places) | 21:11 |
| dsneddon | EmilienM, I'll probably end up posting early WIP reviews for both soon. | 21:11 |
| EmilienM | cool | 21:11 |
| openstackgerrit | John Trowbridge proposed openstack/instack-undercloud: Deploy Monitoring on the undercloud with Puppet https://review.openstack.org/276127 | 21:13 |
| trown | EmilienM: hopefully its ok if I rebase/recheck/generally catherd your opstools patches :) | 21:14 |
| EmilienM | trown: go ahead ! | 21:15 |
| *** david-lyle has joined #tripleo | 21:17 | |
| *** jayg is now known as jayg|g0n3 | 21:19 | |
| *** mburned is now known as mburned_out | 21:27 | |
| *** mburned_out is now known as mburned | 21:29 | |
| openstackgerrit | Gabriele Cerami proposed openstack/instack-undercloud: mysqld config: set innodb_file_per_table to ON https://review.openstack.org/285227 | 21:31 |
| *** yamahata has joined #tripleo | 21:32 | |
| *** mburned is now known as mburned_out | 21:33 | |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-heat-templates: mysqld config: set innodb_file_per_table to ON https://review.openstack.org/285224 | 21:34 |
| *** mburned_out is now known as mburned | 21:36 | |
| *** derekh has joined #tripleo | 21:41 | |
| *** dshulyak has quit IRC | 21:43 | |
| *** dshulyak has joined #tripleo | 21:44 | |
| *** r-mibu has quit IRC | 21:46 | |
| *** r-mibu has joined #tripleo | 21:47 | |
| *** dshulyak has quit IRC | 21:48 | |
| *** rhallisey has quit IRC | 21:55 | |
| *** shivrao has joined #tripleo | 21:55 | |
| *** ccamacho has quit IRC | 21:58 | |
| *** ccamacho has joined #tripleo | 21:59 | |
| *** rhallisey has joined #tripleo | 21:59 | |
| derekh | bandini: did ye find the problem? | 21:59 |
| *** trown is now known as trown|outtypewww | 21:59 | |
| *** pcaruana has quit IRC | 22:01 | |
| bandini | derekh: nope. it is definitely a network issue...it's as if everything is slow, or packets dropped and sometimes duplicated | 22:01 |
| bandini | I will discuss it again with giulio tomorrow | 22:01 |
| derekh | bandini: ok, is anybody poking at the env? If not I'll see if I can figure anything out | 22:02 |
| shardy | derekh: Hey, I just noticed that the undercloud install pulls from https://github.com/derekhiggins/puppet-vlan | 22:03 |
| shardy | is that expected? | 22:03 |
| shardy | (I'm assuming it's an old stray element dep somewhere) | 22:04 |
| derekh | shardy: yes, thats where the puppet module lives is it used? iirc I wrote it for packstack at one stage | 22:04 |
| shardy | derekh: I guess I was surprised to see us pulling from a personal repo, I'm not sure if it's used tbh | 22:05 |
| * shardy will investigate tomorrow | 22:05 | |
| derekh | shardy: I wouldn't be surprised if it isn't used, somebody probably took the list of repos that packstack uses or something like that | 22:05 |
| shardy | yeah, sounds likely, thanks for the info! | 22:06 |
| * derekh didn't put it there | 22:06 | |
| *** dprince has quit IRC | 22:08 | |
| derekh | shardy: its got impressive download stats, https://github.com/derekhiggins/puppet-vlan/graphs/traffic | 22:09 |
| shardy | hehe, I can't actually see the stats, but I can imagine ;) | 22:12 |
| *** tiswanso has quit IRC | 22:12 | |
| openstackgerrit | Merged openstack/instack-undercloud: Enable AMT driver https://review.openstack.org/286252 | 22:13 |
| *** dshulyak has joined #tripleo | 22:14 | |
| *** shardy has quit IRC | 22:15 | |
| *** ccamacho has quit IRC | 22:15 | |
| dmsimard | fyi, trown's out so I figured I'd let you know .. it looks like there are some issues around swift backend commits merged in glance_store today | 22:16 |
| dmsimard | we had to fix stuff in packstack and puppet-openstack, I added a note on the etherpad https://etherpad.openstack.org/p/delorean_master_current_issues | 22:16 |
| *** penick has joined #tripleo | 22:16 | |
| dmsimard | I know you guys are generally pinned but just in case | 22:17 |
| *** dshulyak has quit IRC | 22:19 | |
| *** shivrao has quit IRC | 22:19 | |
| *** lblanchard has quit IRC | 22:21 | |
| openstackgerrit | Merged openstack/os-collect-config: Replace assertEqual(None, *) with assertIsNone in tests https://review.openstack.org/263273 | 22:28 |
| *** david-lyle has quit IRC | 22:29 | |
| *** shivrao has joined #tripleo | 22:30 | |
| *** yamahata has quit IRC | 22:30 | |
| *** rbrady has quit IRC | 22:34 | |
| *** jtomasek has quit IRC | 22:41 | |
| *** rhallisey has quit IRC | 22:43 | |
| *** trozet has quit IRC | 22:49 | |
| *** david-lyle has joined #tripleo | 23:00 | |
| *** ianw has quit IRC | 23:07 | |
| *** chlong_ has joined #tripleo | 23:13 | |
| *** dustins has quit IRC | 23:15 | |
| *** dcain has quit IRC | 23:22 | |
| *** lblanchard has joined #tripleo | 23:30 | |
| pradk | slagle, \o/ looks like the swap patch worked, the nonha ci now passes https://review.openstack.org/#/c/241408/ | 23:35 |
| *** panda has quit IRC | 23:37 | |
| *** panda has joined #tripleo | 23:38 | |
| *** david-lyle has quit IRC | 23:41 | |
| EmilienM | pradk: nice, still issues with HA? did my suggestion helped? | 23:59 |
| * EmilienM looks logs | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!