| EmilienM | I don't see Puppet errors | 00:00 |
|---|---|---|
| *** yamahata has joined #tripleo | 00:00 | |
| *** dshulyak has joined #tripleo | 00:02 | |
| *** dshulyak has quit IRC | 00:07 | |
| *** yamahata has quit IRC | 00:11 | |
| *** prometheanfire has joined #tripleo | 00:13 | |
| prometheanfire | SpamapS: ping, why didn't this get +w? https://review.openstack.org/#/c/281960 | 00:13 |
| pradk | EmilienM, the memory issue was with non-ha case, which is working now | 00:19 |
| pradk | EmilienM, the ha thing fails with "deploy_stdout": "Trying to ping 192.0.2.10 for local network 192.0.2.0/24...FAILURE\n" | 00:19 |
| EmilienM | weird | 00:19 |
| *** saneax is now known as saneax_AFK | 00:38 | |
| *** derekh has quit IRC | 00:38 | |
| ayoung | slagle, sorry to bother you with a provisioning quyestion, but: trying to create a vm for a virt install, and having trouble with the virt-install parameters for the bridge: --network bridge=brbm,model=virtio does not seem to work. What am I missing? | 00:43 |
| ayoung | looking at the XML, I see : <virtualport type='openvswitch'> with a UUID. But I can't see how to pass that. | 00:44 |
| openstackgerrit | Merged openstack/instack-undercloud: Switch default keystone auth_uri to v3 https://review.openstack.org/265380 | 00:58 |
| *** dmacpher-afk has quit IRC | 01:17 | |
| *** penick has quit IRC | 01:43 | |
| *** tiswanso has joined #tripleo | 01:47 | |
| *** tiswanso has quit IRC | 01:48 | |
| *** tiswanso has joined #tripleo | 01:48 | |
| *** tiswanso has quit IRC | 01:48 | |
| *** tiswanso has joined #tripleo | 01:49 | |
| *** rhallisey has joined #tripleo | 01:49 | |
| *** tiswanso has quit IRC | 01:49 | |
| *** tiswanso has joined #tripleo | 01:50 | |
| *** rhallisey has quit IRC | 02:11 | |
| *** xinwu has quit IRC | 02:38 | |
| *** dmacpher has joined #tripleo | 02:42 | |
| *** penick has joined #tripleo | 02:47 | |
| *** anande has joined #tripleo | 02:53 | |
| *** rlandy has quit IRC | 02:54 | |
| *** shivrao has quit IRC | 02:58 | |
| *** lblanchard has quit IRC | 03:04 | |
| *** weshay has quit IRC | 03:22 | |
| *** tzumainn has quit IRC | 03:22 | |
| *** david-lyle has joined #tripleo | 03:24 | |
| *** penick has quit IRC | 03:27 | |
| *** stendulker has joined #tripleo | 03:27 | |
| *** penick has joined #tripleo | 03:28 | |
| *** panda has quit IRC | 03:37 | |
| *** panda has joined #tripleo | 03:38 | |
| *** dshulyak has joined #tripleo | 03:38 | |
| *** dshulyak has quit IRC | 03:43 | |
| *** yuanying has quit IRC | 03:44 | |
| *** masco has joined #tripleo | 03:50 | |
| *** anande has quit IRC | 03:56 | |
| *** anande has joined #tripleo | 03:57 | |
| *** anande has quit IRC | 04:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-collect-config: Updated from global requirements https://review.openstack.org/264446 | 04:03 |
| *** xinwu has joined #tripleo | 04:05 | |
| *** yuanying has joined #tripleo | 04:07 | |
| *** ianw has joined #tripleo | 04:29 | |
| *** shivrao has joined #tripleo | 04:32 | |
| *** bkero has joined #tripleo | 04:37 | |
| bkero | mandre: ping. I'm trying to get tripleo-quickstart running locally, but the ansible playbook for overcloud only seems to be being used for the ci.centos.org provisioned playbook. | 04:38 |
| bkero | Is there any way I can use tripleo-quickstart to deploy an overcloud at home? | 04:38 |
| *** yamahata has joined #tripleo | 04:41 | |
| *** cmyster has quit IRC | 04:44 | |
| *** saneax_AFK is now known as saneax | 04:46 | |
| *** tiswanso has quit IRC | 04:49 | |
| *** links has joined #tripleo | 04:51 | |
| mandre | bkero: I'm using it myself like that, just set the virthost variable to your target node | 04:51 |
| mandre | bkero: something like "ansible-playbook playbooks/centosci/minimal.yml -e virthost=192.168.0.2" | 04:53 |
| mandre | bkero: I missed the part "locally" in your original question, but that should work the same way | 04:55 |
| mandre | I suggest you read http://blog.oddbit.com/2016/02/19/deploy-an-ha-openstack-development-envir/ if you haven't already | 04:57 |
| bkero | mandre: lemme try that real quick | 04:58 |
| bkero | mandre: Hmm, in the centosci minimal playbook it's referencing an 'undercloud' host that's not getting defined I think | 04:59 |
| bkero | Ah, I think it might be working (didn't have ssh access to undercloud in that terminal) | 05:00 |
| mandre | bkero, the undercloud is being defined in the libvirt/setup/undercloud play | 05:01 |
| bkero | Okay | 05:01 |
| mandre | that got me confused too | 05:01 |
| bkero | mandre: That's a very useful blog post, thank you! | 05:03 |
| *** apetrich_ has quit IRC | 05:11 | |
| *** apetrich_ has joined #tripleo | 05:26 | |
| *** dshulyak has joined #tripleo | 05:27 | |
| *** dshulyak has quit IRC | 05:31 | |
| *** pcaruana has joined #tripleo | 05:36 | |
| *** ayoung has quit IRC | 05:38 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix spurios = in dib-python readme https://review.openstack.org/286991 | 05:38 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Reorder developer quickstart docs https://review.openstack.org/286997 | 05:45 |
| *** jdob has quit IRC | 05:46 | |
| *** pcaruana has quit IRC | 05:49 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix cloud-init-disable-resizefs README title https://review.openstack.org/286999 | 05:49 |
| *** rcernin has joined #tripleo | 05:51 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add ability to check for mount leaks https://review.openstack.org/280018 | 05:56 |
| *** jdob has joined #tripleo | 05:58 | |
| *** jaosorior has joined #tripleo | 06:06 | |
| *** jprovazn has joined #tripleo | 06:31 | |
| *** hjensas has quit IRC | 06:33 | |
| *** aufi has joined #tripleo | 06:39 | |
| *** leanderthal|afk is now known as leanderthal | 06:52 | |
| *** akuznetsov has joined #tripleo | 06:54 | |
| *** shardy has joined #tripleo | 07:00 | |
| *** saneax is now known as saneax_AFK | 07:02 | |
| *** akuznetsov has quit IRC | 07:03 | |
| *** akuznetsov has joined #tripleo | 07:08 | |
| *** akuznetsov has quit IRC | 07:10 | |
| *** dshulyak has joined #tripleo | 07:10 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add HostnameMap to allow granular control of hostnames https://review.openstack.org/275605 | 07:15 |
| *** ccamacho has joined #tripleo | 07:18 | |
| *** jtomasek has joined #tripleo | 07:19 | |
| jaosorior | shardy: Hey man, how's it going? hey, question: With the recent addition of the overriding of the nova server that doesn't get replaced as a consecuence of the user_data changing (https://review.openstack.org/#/c/283832/ ); If one now updates some cloud init resource, that won't take effect, right? | 07:20 |
| shardy | jaosorior: Hey, not bad thanks, yeah that's the intention | 07:23 |
| shardy | e.g any change to user_data will only affect new servers (rebuild, replace, scale-out) | 07:24 |
| shardy | existing servers should be left alone | 07:24 |
| jaosorior | I see | 07:24 |
| *** panda has quit IRC | 07:37 | |
| *** panda has joined #tripleo | 07:38 | |
| *** liverpooler has joined #tripleo | 07:44 | |
| *** mikelk has joined #tripleo | 07:47 | |
| *** shivrao_ has joined #tripleo | 07:51 | |
| *** mcornea has joined #tripleo | 07:51 | |
| *** shivrao has quit IRC | 07:52 | |
| *** shivrao_ is now known as shivrao | 07:52 | |
| *** dmacpher has quit IRC | 07:58 | |
| *** jcoufal has joined #tripleo | 07:58 | |
| *** pcaruana has joined #tripleo | 08:02 | |
| *** gfidente has joined #tripleo | 08:05 | |
| *** gfidente has quit IRC | 08:05 | |
| *** gfidente has joined #tripleo | 08:05 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 08:07 |
| *** hjensas has joined #tripleo | 08:09 | |
| *** rdopiera has joined #tripleo | 08:16 | |
| *** ohamada has joined #tripleo | 08:19 | |
| *** fgimenez has joined #tripleo | 08:23 | |
| *** NobodyCa1 has joined #tripleo | 08:28 | |
| *** NobodyCa1 has quit IRC | 08:33 | |
| *** NobodyCa1 has joined #tripleo | 08:34 | |
| *** akuznetsov has joined #tripleo | 08:35 | |
| *** devvesa has joined #tripleo | 08:35 | |
| hewbrocca | gfidente: morning! how is it looking | 08:37 |
| gfidente | hewbrocca, it isn't going | 08:37 |
| gfidente | there is traffic but is super slow, connections time out | 08:37 |
| hewbrocca | so, same as yesterday still | 08:38 |
| hewbrocca | did anyone grab dsneddon to take a look? | 08:38 |
| gfidente | yeah yesterday we had a quick chat | 08:38 |
| dsneddon | hewbrocca: look at what? | 08:38 |
| hewbrocca | dsneddon: morning! | 08:38 |
| gfidente | dsneddon, hey :) | 08:38 |
| hewbrocca | network isolation in the TripleO CI rack | 08:38 |
| hewbrocca | this is like a blocking mission critical thing and we can't get it going :( | 08:39 |
| dsneddon | hewbrocca: Oh, the CI environment? Yeah, there was some thinking that it could be MTU, but I do kind of doubt that's the problem. | 08:39 |
| hewbrocca | Seems unlikely | 08:39 |
| *** NobodyCa1 has quit IRC | 08:39 | |
| dsneddon | Not that I know what the problem is. I hope the problem isn't the underlying switches, because I don't know who has visibility. | 08:39 |
| hewbrocca | dsneddon: it's the same hardware as in the rest of the scale lab | 08:40 |
| hewbrocca | for whatever that's worth | 08:40 |
| dsneddon | hewbrocca: Hmm, I'm not aware of what's going on in the scale lab | 08:40 |
| *** ifarkas has joined #tripleo | 08:40 | |
| hewbrocca | Dell 630s, Juniper (I think) switches | 08:40 |
| gfidente | dsneddon, though traffic should stay within the ovs bridge on the baremetal | 08:41 |
| gfidente | which doesn't have any physical nic from the baremetal plugged in it | 08:41 |
| gfidente | so in theory this isn't leaving the baremetal kernel | 08:42 |
| gfidente | unless ovs is doing something complex in br-int | 08:42 |
| *** mbound has joined #tripleo | 08:42 | |
| dsneddon | gfidente, I guess I was thinking this was a bare metal job earlier. If I had realized it was virt I wouldn't have wasted time on the MTU | 08:42 |
| gfidente | yeah it's all VMs | 08:43 |
| hewbrocca | dsneddon: that's the design of the TripleO CI rack | 08:43 |
| hewbrocca | there are a bunch of long-lived VMs. Undercloud gets deployed into one of them, | 08:43 |
| gfidente | dsneddon, so we host multiple testenvs each on an isolated bridge where the overcloud nics are plugged | 08:43 |
| dsneddon | gfidente, Well, I guess we'd want to make sure that the libvirt network bridges were set up correctly, but I expect that it's just using instack to do that, right? | 08:43 |
| hewbrocca | and then it deploys an overcloud into others | 08:43 |
| gfidente | dsneddon, man, the libvirt network bridge is indeed the only difference I can see compared to the usual 'development' environment | 08:44 |
| gfidente | and no it's not using instack | 08:44 |
| hewbrocca | it's a custom hack by derekh and dprince | 08:44 |
| gfidente | so that isn't a regular linux bridge | 08:44 |
| gfidente | it's an additional ovs bridge | 08:44 |
| gfidente | but I checked it 1000times yesterday and couldn't find anything wrong with it | 08:44 |
| *** akuznetsov has quit IRC | 08:45 | |
| dsneddon | gfidente, Yeah, I mean the obvious thing would be to make sure that all VLANs are allowed on the bridge, but if we are getting some traffic through that wouldn't be the issue. That would cause no traffic, not slow traffic. | 08:45 |
| gfidente | yeah I *think* the reason why it's slow is that it is actually taking the wrong route | 08:46 |
| gfidente | probably in OVS because we were looking into pcap with bandini yesterday | 08:46 |
| gfidente | and we noticed duplicates and retransmissions | 08:46 |
| gfidente | a lot of them | 08:46 |
| gfidente | as if things were first get lost, and then finally getting to receiver | 08:48 |
| *** onovy has quit IRC | 08:48 | |
| gfidente | which we translated into 'stupidly slow' | 08:48 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Use service tenant for ceilometer https://review.openstack.org/231461 | 08:48 |
| gfidente | I think if you could give a look to the baremetal bridging that would be nice | 08:49 |
| gfidente | but we need to wait for derek, I don't have credentials to login on the baremetal | 08:49 |
| *** shivrao has quit IRC | 08:51 | |
| gfidente | dsneddon, fwiw, we're also using an older version of ovs | 08:52 |
| gfidente | something like 2.3 from F20 | 08:52 |
| gfidente | not sure if that rings any bell to you | 08:52 |
| openstackgerrit | Merged openstack/puppet-tripleo: Always override X-Forwarded-Proto header for Heat https://review.openstack.org/286420 | 08:52 |
| jaosorior | yay :D | 08:54 |
| gfidente | jaosorior, what was it, merge fast fix fast? | 08:54 |
| jaosorior | gfidente: huh? | 08:55 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Always override X-Forwarded-Proto header for Heat https://review.openstack.org/287047 | 08:56 |
| jaosorior | gfidente: That's actually even an upstream issue, not something we enable | 08:58 |
| *** chem has joined #tripleo | 08:58 | |
| jaosorior | * not something we particularle enable | 08:58 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Write the compute upgrade script for tripleo major upgrade workflow https://review.openstack.org/285989 | 08:59 |
| *** paramite has joined #tripleo | 09:01 | |
| marios | thanks gfidente | 09:01 |
| *** lucas-dinner is now known as lucasagomes | 09:06 | |
| hewbrocca | dsneddon: I know it's super-late for you, but if you'll be around a bit longer and can take a look once derekh gets on, it would be super-helpful | 09:08 |
| *** devvesa has quit IRC | 09:09 | |
| dsneddon | hewbrocca: Yeah, I was going to get up super early, but if Derek is coming on soon I'd rather wait | 09:09 |
| hewbrocca | Anybody know if slagle got a patch together to add swap to the CI instances | 09:09 |
| hewbrocca | for aodh/gnocchi testing? | 09:09 |
| *** derekh has joined #tripleo | 09:10 | |
| derekh | gfidente: howdy, I jumped on the env last night to see if I could figure anything out | 09:11 |
| derekh | gfidente: no luck | 09:11 |
| gfidente | derekh, yeah | 09:12 |
| gfidente | I am lost as well | 09:12 |
| derekh | gfidente: I did notice some weird stuff, e.g. tcpdump -i eth0 -nn -e port 5000 | 09:12 |
| derekh | gfidente: ^^ only show the incoming traffic, not the traffic in both directions, can you see if thats the case in a working deployment? | 09:13 |
| *** ishant has joined #tripleo | 09:13 | |
| marios | hewbrocca: https://review.openstack.org/#/c/273752/ might be this | 09:13 |
| gfidente | derekh, oh you see the masquerading rule forwarding stuff on eth0 when it shouldn't? | 09:13 |
| gfidente | derekh, on which host did you run that? | 09:13 |
| gfidente | derekh, also I think we might get dsneddon to check the bridges on the baremetal | 09:13 |
| marios | hewbrocca: and https://review.openstack.org/#/c/286793 | 09:14 |
| gfidente | derekh, can you pass him credentials to login on the hosting node? | 09:14 |
| *** devvesa has joined #tripleo | 09:14 | |
| derekh | gfidente: from the comute node [root@overcloud-novacompute-0 ~]# curl http://172.16.2.7:5000 | 09:14 |
| derekh | tcpdump on the controller | 09:15 |
| hewbrocca | derekh: we have dsneddon on hand | 09:15 |
| hewbrocca | he can help | 09:15 |
| hewbrocca | marios: that's great. I'd really support getting it merged, it will help unwedge a bunch of stuff (if it in fact works) | 09:16 |
| derekh | hewbrocca: cool, is he here now? | 09:16 |
| derekh | hewbrocca: or later | 09:17 |
| hewbrocca | he is indeed | 09:17 |
| hewbrocca | up very very late, explicitly to help you :) | 09:17 |
| dsneddon | derekh, Hello o/ | 09:17 |
| derekh | dsneddon: hi, have you been on the instance to see whats happening yet? | 09:18 |
| dsneddon | derekh, No, you'll have to give me instructions and creds to check it out | 09:18 |
| *** jistr has joined #tripleo | 09:19 | |
| derekh | gfidente: can you do that^^ I gotta do a school run then will be back, I'll then get you on the baremetal node also | 09:19 |
| hewbrocca | derekh: gfidente doesn't have the creds, he says | 09:19 |
| dsneddon | derekh, No, he said he didn't know | 09:19 |
| hewbrocca | oh maybe that was only for the bare metal | 09:20 |
| gfidente | derekh, yeah the ssh key on the baremetal | 09:20 |
| derekh | hewbrocca: the env we setup yesterday, dsneddon will want to start there I'd assume so he can see whats happening | 09:20 |
| dsneddon | derekh, Exactly | 09:20 |
| hewbrocca | gfidente: ^^^ | 09:20 |
| gfidente | derekh, the baremetal not the jenkins node | 09:20 |
| derekh | gfidente: I know that but wouldn't he want to start with the jenkins node | 09:21 |
| dsneddon | gfidente, Were you seeing the same behavior in bare metal? I want to see what's happening in Jenkins. | 09:21 |
| gfidente | dsneddon, derekh wait | 09:21 |
| gfidente | it's the *same* deployment | 09:21 |
| derekh | gfidente: you key is back on the same place, so you can log in now | 09:21 |
| gfidente | but dsneddon is interested in checking the ovs bridges on the baremetal node hosting the testenv | 09:22 |
| gfidente | dsneddon, jenkins is driving the deployment but it's not where the bridges are | 09:22 |
| derekh | gfidente: add the key for dsneddon, back in a few minutes | 09:22 |
| gfidente | derekh, ack! | 09:22 |
| dsneddon | gfidente: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjzU2KBRZsBIf53iWv4Zhbr//aIxkAfii9G6nTOKcKTVYKuC9rYtKU2h7XqkGQ1x1tXV5REzGxZEyARciDvaK6ip2nIcV/aeVzFTMuoU+vR3pSEhR8UvrRNgokDHkAdp9wzZGG+l0NaJD/TZmycnuNT9zsFWqR74XWS9TZ/hHRal6/8iC4Llcf7MKTMAz5ai/wxUyEe9EErLJ/uqhvD11Qpfsw7aGIdHgGAkwFeJNCFOb26PLiiaXKC2FuN8s06LdlJxd+BYKhonBJFKo+vozEjIvoSlFR14B+tThkGdodFUIf+MOAcjXb8fJ9qj9viKoBOt2GFyr/TKeLF0V1d2dl dsneddon@dsneddon-t540p | 09:22 |
| *** mkovacik has joined #tripleo | 09:28 | |
| shardy | Anyone else seeing a REPOREFS don't match error with puppet-concat installing the undercloud? | 09:32 |
| *** pblaho has joined #tripleo | 09:33 | |
| jaosorior | shardy: At least I didn't see that yesterday | 09:33 |
| jaosorior | shardy: Nevermind, it was on monday that I did my last deployment | 09:34 |
| *** akrivoka has joined #tripleo | 09:35 | |
| derekh | gfidente: dsneddon back | 09:41 |
| gfidente | derekh, hey we're both logged on the BM in screen | 09:42 |
| gfidente | derekh, any clue what 172.16.2.4 is? | 09:42 |
| derekh | gfidente: dsneddon so one thing I tried last night, was upgrading ovs a little on the baremetal node, bug its still F20 so it only went from 2.3.0-1 to 2.3.1-3 | 09:42 |
| gfidente | derekh, overlcoud has 2.6 and 2.7 that I can tell | 09:42 |
| dsneddon | derekh, OK, thanks for letting me know | 09:42 |
| gfidente | derekh, I'm deploying on another host to reproduce your tcpdump too, will pass credentials | 09:43 |
| *** apetrich_ has quit IRC | 09:43 | |
| derekh | gfidente: | fc18b9ff-f43e-4b2d-84ef-e7aa7178eb7f | redis_virtual_ip | fa:16:3e:e7:ca:0b | {"subnet_id": "50bd1464-db38-467a-9dda-5dca3b4aa54b", "ip_address": "172.16.2.4"} | | 09:45 |
| derekh | gfidente: ack, thanks | 09:45 |
| gfidente | derekh, ah missed it :) | 09:45 |
| *** mgould has joined #tripleo | 09:57 | |
| openstackgerrit | Javier Peña proposed openstack/tripleo-heat-templates: Fix vncproxy_host for IPv6 https://review.openstack.org/287068 | 10:00 |
| hewbrocca | derekh: anything interesting come to light? | 10:02 |
| *** electrofelix has joined #tripleo | 10:02 | |
| derekh | hewbrocca: dsneddon is poking around at the moment | 10:02 |
| dsneddon | derekh, I haven't found anything yet, just confirmed this weird behavior where ICMP works but TCP is cripplingly slow | 10:03 |
| derekh | dsneddon: ack | 10:05 |
| gfidente | dsneddon, you're pluggin an if | 10:06 |
| gfidente | on the baremetal in the bridge which gets to overcloud right? | 10:07 |
| derekh | dsneddon: also traceroute between the controller and the compute node was reporting all kinds of hop numbers, so its dropping udp packets also | 10:07 |
| dsneddon | Right | 10:07 |
| gfidente | ok just wanted to make sure I was following :P | 10:07 |
| dsneddon | Connectivity between the undercloud and overcloud is fine, it's just between overcloud nodes. | 10:08 |
| derekh | dsneddon: can you try that ssh a coupple of times, we've had it work between the overcloud nodes also sometimews | 10:09 |
| gfidente | dsneddon, that's not undercloud that's baremetal | 10:09 |
| dsneddon | Right, that's what I meant. | 10:09 |
| *** onovy has joined #tripleo | 10:10 | |
| gfidente | derekh, looks like it's timing out when retrying it though | 10:10 |
| dsneddon | gfidente, The ssh seems to be working, slowly, but then it might be the target trying to do DNS lookups that's adding the delay. It's working, though, just not authenticating. | 10:10 |
| *** saneax_AFK is now known as saneax | 10:11 | |
| gfidente | or that's probably dns | 10:11 |
| gfidente | right | 10:11 |
| gfidente | ack, let's add something into hosts to see how it goes | 10:11 |
| dsneddon | gfidente, You should set up keys between the overcloud nodes, and see if you can scp anything between them. | 10:11 |
| *** apetrich has joined #tripleo | 10:14 | |
| dsneddon | gfidente, derekh: This couldn't be something like the wrong network driver in the libvirt VM definition, could it? | 10:15 |
| *** xinwu has quit IRC | 10:16 | |
| gfidente | dsneddon, it's not slow on dns though it seems | 10:17 |
| gfidente | the sshd debug says it took 12secs for keys exchange | 10:17 |
| gfidente | Mar 02 10:02:37 overcloud-controller-0 sshd[26747]: debug1: SSH2_MSG_NEWKEYS sent [preauth] | 10:17 |
| gfidente | Mar 02 10:02:37 overcloud-controller-0 sshd[26747]: debug1: expecting SSH2_MSG_NEWKEYS [preauth] | 10:17 |
| gfidente | Mar 02 10:02:50 overcloud-controller-0 sshd[26747]: debug1: SSH2_MSG_NEWKEYS received [preauth] | 10:17 |
| gfidente | Mar 02 10:02:50 overcloud-controller-0 sshd[26747]: debug1: KEX done [preauth] | 10:17 |
| bandini | also a telnet to port 22 would hang most of the time | 10:18 |
| bandini | iirc | 10:18 |
| derekh | dsneddon: as in using virtio or something else? here is what we got http://paste.openstack.org/show/488921/ | 10:18 |
| gfidente | but for some reason, at least this is consistently working | 10:18 |
| dsneddon | derekh, virtio is good, wanted to make sure it wasn't something else. | 10:18 |
| derekh | dsneddon: vlan20 on the bm node that were testing with has a mtu of 1500, should it be lower to match the other nodes? | 10:22 |
| gfidente | derekh, the "working" overcloud is deployed now fwiw | 10:24 |
| gfidente | derekh, so we can try the tcpdump | 10:24 |
| derekh | gfidente: thanks, will give it a go | 10:24 |
| *** dtantsur|afk is now known as dtantsur | 10:25 | |
| derekh | dsneddon: tab 9 on the screen session is also on the bm host if you need it for the tcpdump | 10:25 |
| derekh | gfidente: will poke around your other env now, I'll have tabs 4&5 on the controller and 6&7 on the compute node | 10:29 |
| dsneddon | derekh, gfidente: OK, we've confirmed that we have good connectivity between the virt-host and the overcloud controller | 10:30 |
| gfidente | dsneddon, so it's weird though | 10:31 |
| dsneddon | derekh, gfidente: Maybe. That was weird, we did an scp file transfer, and the file transferred quickly, but the setup and teardown were too long | 10:31 |
| gfidente | ssh says 3.3 megs/sec but it took 52 secs? | 10:31 |
| dtantsur | morning folks! | 10:31 |
| dtantsur | I have a 2 line liberty backport with 1x +2 and gate passed! https://review.openstack.org/#/c/284746/ please | 10:32 |
| shardy | dtantsur: g'morning - done | 10:34 |
| dtantsur | thnx! | 10:34 |
| derekh | gfidente: The same thing happens with my tcpdump on you env, it seems weird but its the same for both envs so not the problem# | 10:34 |
| dtantsur | anyone here knowing or caring about networks? just a sanity check: does https://review.openstack.org/#/c/286617/ make sense to you? | 10:35 |
| dtantsur | maybe gfidente or EmilienM, you're in some networking team now ;) ^^ | 10:35 |
| gfidente | derekh, can you try snooping the bridge? | 10:37 |
| openstackgerrit | Merged openstack/python-tripleoclient: Replace python-hardware with smaller python-hardware-detect https://review.openstack.org/284746 | 10:38 |
| *** bvandenh has joined #tripleo | 10:38 | |
| gfidente | derekh, you do see it when snooping vlan20 | 10:39 |
| gfidente | derekh, looks like ovs is only bridging one direction on eth0 | 10:39 |
| derekh | gfidente: ok | 10:40 |
| gfidente | derekh, I think you know better how to get the pcap file off of the baremetal node? | 10:40 |
| gfidente | dsneddon, ^^ | 10:40 |
| derekh | gfidente: dsneddon yup, I'll grab it | 10:40 |
| derekh | dsneddon: http://goodsquishy.com/downloads/172_16_2_7.cap | 10:42 |
| dsneddon | derekh, Thank you | 10:43 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Change the default value for NetworkNexusVxlanGlobalConfig https://review.openstack.org/287096 | 10:44 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: install zaqarclient https://review.openstack.org/286156 | 10:47 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: quiet yum update https://review.openstack.org/286155 | 10:47 |
| jistr | marios: trying to get the upgrade e2e as much as i can, i realized that we need the yum & zaqarclient changes on compute too, so updated the patches ^^ | 10:48 |
| *** leanderthal has quit IRC | 10:48 | |
| jistr | marios: or maybe not the yum change necessarily, because it doesn't go through Heat. But the zaqarclient is needed for os-collect-config to keep working. | 10:49 |
| jistr | marios: do you think i should keep the `yum -q` option for the computes as well? ^^ | 10:49 |
| marios | jistr: looking | 10:50 |
| *** d0ugal has joined #tripleo | 10:50 | |
| *** yamahata has quit IRC | 10:51 | |
| *** tosky has joined #tripleo | 10:51 | |
| marios | jistr: yeah we can add -q in the compute script if we need (but if it is the output to heat that is too big then as you say not that big a deal in that case) | 10:51 |
| dsneddon | derekh, I made a couple of changes on the virt host. I added a vlan20 interface which is on the overcloud internal API network for testing. We also were messing with the MTU, which is now set at 1300 on the controller's vlan20 interface (which is why tcpdump shows the 1260 mss) | 10:53 |
| jistr | marios: yeah i don't think we need it in the compute actually. Would it be better if the operator saw the yum output as it's happening? I'm now a bit more inclined to restore the `yum -q` patch to affect only controllers. | 10:53 |
| marios | jistr: yeah exactly (am looking/voting on the review now) this was my thought too (for the operator) | 10:54 |
| jistr | marios: ack i'll restore it, thx | 10:54 |
| derekh | dsneddon: ok | 10:54 |
| dsneddon | derekh, I'm about out of ideas, except maybe trying to install OVS 2.4, if that's even possible. | 10:54 |
| dsneddon | derekh, We have 2.4.0 on RHEL 7 | 10:55 |
| gfidente | dsneddon, you mentioned a restart, maybe we can restart just OVS to start with? | 10:55 |
| dsneddon | gfidente, That would have been done when derekh upgraded OVS to version 2.3.1 (or whatever) last night. | 10:56 |
| derekh | dsneddon: I was afraid you might say something like that ;-) | 10:56 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: install zaqarclient https://review.openstack.org/286156 | 10:57 |
| dsneddon | derekh, I have been expecting that the F20 base would bite us sooner or later. I'm surprised it hasn't bitten us harder so far. | 10:58 |
| derekh | dsneddon: ok, I'll see if I can build an image for it, with centos 7 will that have the version needed? /me checking | 10:58 |
| hewbrocca | *oofá | 10:58 |
| hewbrocca | * | 10:58 |
| dsneddon | Which reminds me | 10:58 |
| * dsneddon still hasn't upgraded his laptop from F20 | 10:59 | |
| hewbrocca | So, changing the OS on all the TripleO CI hosts seems like a very risky undertaking | 10:59 |
| dsneddon | hewbrocca, I hope it won't come to that. | 10:59 |
| hewbrocca | me too :) | 10:59 |
| hewbrocca | getting lunch, back shortly | 11:00 |
| marios | jistr: did something go wrong with gerrit (the zaqar one went to v3 but no update on the quiet yum one) | 11:00 |
| jistr | marios: yea it's rejecting the yum -q change for me | 11:00 |
| derekh | hewbrocca: it would only be the testenvs, the overcloud wouldn't be touched, I can try it on one host and see what happens, | 11:01 |
| jistr | marios: i wonder if there's some safety measure against reverting patches inadvertently, which is kicking in. Googling how to solve it | 11:01 |
| gfidente | derekh, I see this guy here | 11:01 |
| gfidente | http://koji.fedoraproject.org/koji/buildinfo?buildID=680581 | 11:01 |
| gfidente | it's 2.4 for f21 maybe close enough | 11:01 |
| gfidente | and can be cleanly installed on f20 | 11:02 |
| gfidente | maybe | 11:02 |
| marios | jistr: shortlog look ok to me https://review.openstack.org/gitweb?p=openstack/tripleo-heat-templates.git;a=shortlog;h=59430030f0147991bed882ff9d1466771a37d46d (from zaqar) ...weird | 11:02 |
| derekh | hewbrocca: if trying a new image is the onyl way forward then dsneddon may as well go to bed, this will take all day at best I'd imaging, there is now way the various elemenst will work forst time rount | 11:02 |
| derekh | *round | 11:03 |
| gfidente | derekh, oh can't we upgrade in place on this baremetal? | 11:03 |
| dsneddon | derekh, I don't know for sure that a newer OVS will work, but I know that I've never seen this behavior on a system with OVS 2.4. | 11:03 |
| derekh | gfidente: yup, we can try that | 11:04 |
| hewbrocca | we think we can get to OVS 2.4 without a new image? | 11:05 |
| dsneddon | derekh, gfidente: OVS 2.4.0 is available starting in F21: http://comments.gmane.org/gmane.linux.network.openvswitch.general/9984 | 11:05 |
| *** tosky has quit IRC | 11:05 | |
| derekh | dsneddon: hewbrocca: going to try an upgrade if I can | 11:06 |
| * hewbrocca PH345 | 11:06 | |
| hewbrocca | er | 11:06 |
| hewbrocca | PH34R5 | 11:06 |
| dsneddon | derekh, Best of luck, I'll check in after I get a modicum of sleep. | 11:06 |
| hewbrocca | dsneddon: thanks for all your help | 11:06 |
| derekh | dsneddon: ok, talk to you later and thanks | 11:06 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: install zaqarclient https://review.openstack.org/286156 | 11:08 |
| *** ohamada has quit IRC | 11:11 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: install zaqarclient https://review.openstack.org/286156 | 11:11 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: quiet yum update https://review.openstack.org/286155 | 11:11 |
| jistr | ok finally :) | 11:11 |
| *** tosky has joined #tripleo | 11:11 | |
| jistr | marios: ^ FYI the trick was that gerrit will indeed reject older patch set submission, probably as a safety measure. So one needs to `rebase -i`, `edit` the commit with not doing anything else than `git commit --amend`, this will change the timestamps on the commit (and the commit hash) and then gerrit will not reject it. | 11:13 |
| jistr | i even got my +2s back, heh | 11:13 |
| *** olap has joined #tripleo | 11:14 | |
| derekh | gfidente: hewbrocca 2.4.0-1.fc21 doesn't appear to have helped, although it didn't pull in any dependencies , just tried an update of the openvswitch packages | 11:14 |
| marios | jistr: thx | 11:15 |
| hewbrocca | glorious :( | 11:15 |
| hewbrocca | derekh: and that's OVS 2.4? | 11:16 |
| hewbrocca | I think we have probably lost enough time on this | 11:16 |
| hewbrocca | derekh: is there *any way* we can get manual testing on these patches so that we can get them merged | 11:17 |
| derekh | hewbrocca: 2.4.0-1.fc21 | 11:17 |
| derekh | hewbrocca: as in manually test and approve it, I'd say so, probably need to be brought to the other cores though but it seams reasonable | 11:23 |
| *** stendulker_ has joined #tripleo | 11:33 | |
| *** cmyster has joined #tripleo | 11:33 | |
| *** stendulker has quit IRC | 11:36 | |
| *** panda has quit IRC | 11:37 | |
| *** stendulker_ has quit IRC | 11:38 | |
| *** panda has joined #tripleo | 11:38 | |
| *** mbound has quit IRC | 11:41 | |
| *** masco has quit IRC | 11:43 | |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-docs: Add advanced docs about images https://review.openstack.org/259451 | 11:46 |
| gfidente | derekh, so it's not like it's behaving the same though with 2.4 | 11:49 |
| gfidente | as if we are getting somewhere it seems | 11:49 |
| derekh | gfidente: yup, things are happening slightly differently | 11:49 |
| gfidente | but consistenly | 11:50 |
| gfidente | I mean we repoduced it | 11:50 |
| gfidente | maybe this really is something we fixed in rhel | 11:50 |
| derekh | yup | 11:52 |
| jaosorior | this had been fixed already, right? 31mError: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: keystone-manage bootstrap --bootstrap-password MQ67mevhB4RRReeR6bnz2ujqz returned 1 instead of one of [0]\u001b | 11:53 |
| *** trown|outtypewww is now known as trown | 11:53 | |
| *** thrash|g0ne has joined #tripleo | 11:54 | |
| *** thrash|g0ne has quit IRC | 11:54 | |
| *** thrash|g0ne has joined #tripleo | 11:54 | |
| *** masco has joined #tripleo | 12:00 | |
| *** apetrich has quit IRC | 12:04 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI https://review.openstack.org/286793 | 12:04 |
| hewbrocca | gfidente, derekh So, it is worth trying to reproduce this on Centos 7 | 12:09 |
| hewbrocca | I almost don't care, since there is no possible way we can reimage the whole damn CI rack right now | 12:09 |
| derekh | hewbrocca: I've started looking into building a F21 image, we wouldn't need to reimage all the rack, just the test envs, the overcloud would remain untouched | 12:10 |
| hewbrocca | OK | 12:11 |
| derekh | hewbrocca: its still a chunk of work but not as risky as zapping the overcloud | 12:11 |
| hewbrocca | right | 12:11 |
| hewbrocca | Well, before we spend the effort on that, can we somehow test that it helps? | 12:12 |
| gfidente | derekh, in tripleo meaning of the world, I am going to ask something very stupid | 12:12 |
| gfidente | which overcloud is it that won't be zapped? | 12:13 |
| gfidente | hewbrocca, ack I am moving on manual testing as well | 12:13 |
| derekh | gfidente: the one hosting the jekins slaves | 12:14 |
| gfidente | oh the one hosting the jenkins slaves | 12:14 |
| gfidente | makes sense now, thanks | 12:14 |
| hewbrocca | gfidente: OK | 12:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Support adding a swap file to overcloud nodes https://review.openstack.org/273752 | 12:14 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Support adding a swap file to overcloud nodes https://review.openstack.org/287158 | 12:15 |
| derekh | hewbrocca: we'll I was going to try it on one node first, but still have to figure out if I can build an image etc... | 12:15 |
| hewbrocca | OK, I will leave you guys to it, then | 12:15 |
| hewbrocca | I was just trying to sort a way that we could verify F21 works before putting the effort into building images etc. | 12:16 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Adds the upgrade of any stand-alone swift nodes before controllers https://review.openstack.org/287159 | 12:16 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Adds the upgrade of any stand-alone swift nodes before controllers https://review.openstack.org/287159 | 12:18 |
| *** mbound has joined #tripleo | 12:19 | |
| shardy | dtantsur: Hey, just raised bug #1552209 - do you happen to know if there's a fix already? | 12:21 |
| openstack | bug 1552209 in tripleo "introspection outputs a keystoneclient deprecation warning" [Undecided,New] https://launchpad.net/bugs/1552209 | 12:21 |
| shardy | "Passing auth_token to client objects is deprecated..." | 12:22 |
| dtantsur | shardy, there is! lemme find a link | 12:22 |
| shardy | nice! :) | 12:22 |
| dtantsur | shardy, https://review.openstack.org/284723 for your reviewing pleasure :) | 12:22 |
| shardy | I just ran introspection on some desktop baremetal machines with the fake_pxe driver and it all works pretty well except for that :) | 12:23 |
| shardy | dtantsur: thanks, will check it out | 12:23 |
| *** mgould has quit IRC | 12:23 | |
| shardy | I also noticed there's a bunch of connection errors on the node consoles about connecting to vendor_passthrough/lookup | 12:24 |
| shardy | is that a known issue also? | 12:24 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Adds v6 capability to the deploy validation test (pings) https://review.openstack.org/286031 | 12:24 |
| shardy | it doesn't seem to be a fatal error, but initially I thought the ramdisk couldn't post back to the undercloud at all | 12:24 |
| dtantsur | shardy, I suspect it's something fixed by https://review.gerrithub.io/#/c/264885/ | 12:24 |
| * dtantsur is shardy's personal magician today :D | 12:25 | |
| shardy | nice :) | 12:25 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/241408 | 12:27 |
| *** fgimenez has quit IRC | 12:27 | |
| slagle | gfidente: network isolation is going to be broken on stable/liberty until we land https://review.openstack.org/#/c/286553/ won't it? | 12:28 |
| *** ishant has quit IRC | 12:29 | |
| slagle | since https://review.openstack.org/#/c/286031/ got backported | 12:29 |
| hewbrocca | slagle: morning! | 12:30 |
| gfidente | slagle, right | 12:30 |
| gfidente | "house of cards" | 12:30 |
| gfidente | I was going to update the ipv6 entire set in a dependency order | 12:31 |
| gfidente | the order in which I pulled them in | 12:31 |
| *** fgimenez has joined #tripleo | 12:31 | |
| slagle | hewbrocca: mornin | 12:32 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Enable per-role SchedulerHints https://review.openstack.org/266930 | 12:33 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Bind Galera on a hostname for compat with IPv6 addresses https://review.openstack.org/285997 | 12:34 |
| dtantsur | shardy, in return, could you please check https://review.openstack.org/277974 as well? :) | 12:35 |
| *** jaosorior has quit IRC | 12:37 | |
| *** jaosorior has joined #tripleo | 12:38 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Nova RPC unpinning https://review.openstack.org/285462 | 12:39 |
| *** jaosorior has quit IRC | 12:39 | |
| *** jaosorior has joined #tripleo | 12:40 | |
| hewbrocca | So love to see things merging | 12:40 |
| *** dshulyak has quit IRC | 12:41 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 12:42 |
| dtantsur | hewbrocca++ | 12:43 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Enable notifications on the overcloud https://review.openstack.org/286299 | 12:44 |
| trown | lol hewbrocca I tend to have the opposite feeling :p, if we dont merge anything it cant break :p | 12:45 |
| hewbrocca | LOL | 12:45 |
| trown | although that would be all of openstack not just tripleo | 12:45 |
| *** cmyster has quit IRC | 12:48 | |
| shardy | dtantsur: sure, will do | 12:51 |
| *** lucasagomes is now known as lucas-hungry | 12:51 | |
| gfidente | shardy, resources[0]: | 12:57 |
| gfidente | "str_replace" parameters must be a mapping validate | 12:57 |
| gfidente | how can I get some more infos on which str_replace ctually failed? | 12:57 |
| gfidente | I think this is coming out of the str_split in https://review.openstack.org/#/c/235423/ | 12:58 |
| shardy | gfidente: are you using https://review.openstack.org/#/c/275605/ (HostnameMap), or templates that contain it, on an older heat without https://review.openstack.org/#/c/282403/ ? | 12:59 |
| *** ohamada has joined #tripleo | 12:59 | |
| gfidente | shardy, yes | 12:59 |
| *** links has quit IRC | 12:59 | |
| shardy | gfidente: Ok, for that hostname map patch, we need the latest heat with the associated str_replace fix | 12:59 |
| shardy | it's landed on master and liberty now | 12:59 |
| *** dprince has joined #tripleo | 13:00 | |
| gfidente | shardy, ack updating from delorean | 13:01 |
| derekh | gfidente: you finished on that baremetal hosts? about to try a new image, so we will loose it | 13:01 |
| gfidente | derekh, yes go ahead | 13:01 |
| gfidente | thanks | 13:01 |
| *** thrash|g0ne is now known as thrash | 13:01 | |
| *** thrash has joined #tripleo | 13:01 | |
| hewbrocca | woop new image | 13:02 |
| gfidente | shardy, works with the updated version, thanks | 13:05 |
| *** rhallisey has joined #tripleo | 13:06 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Upgrades: Add StackUpgradeManager https://review.openstack.org/286604 | 13:07 |
| *** mgould has joined #tripleo | 13:10 | |
| *** mkovacik has quit IRC | 13:11 | |
| *** mkovacik has joined #tripleo | 13:12 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add mirror server https://review.openstack.org/238414 | 13:12 |
| *** mkovacik has quit IRC | 13:12 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Used the mirror server for the centos image and git repos https://review.openstack.org/285257 | 13:12 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use instack-qcow2 from cache if available and appropriate https://review.openstack.org/285259 | 13:12 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Upload the instack qcow image after periodic jobs https://review.openstack.org/285258 | 13:12 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job https://review.openstack.org/229789 | 13:12 |
| *** jayg|g0n3 is now known as jayg | 13:16 | |
| *** rbrady has joined #tripleo | 13:19 | |
| *** rbrady has quit IRC | 13:20 | |
| *** rbrady has joined #tripleo | 13:21 | |
| *** rain has joined #tripleo | 13:21 | |
| *** rain is now known as Guest23862 | 13:21 | |
| *** rbrady has quit IRC | 13:25 | |
| *** Guest23862 is now known as leanderthal | 13:29 | |
| derekh | F21 image booted but can't see nic's , anybody remember if we had to do anything to load specific kernel modules or something..? | 13:29 |
| *** rbrady has joined #tripleo | 13:30 | |
| *** pradk has quit IRC | 13:32 | |
| *** dshulyak has joined #tripleo | 13:32 | |
| *** apetrich has joined #tripleo | 13:33 | |
| *** pleia2 has quit IRC | 13:33 | |
| *** paramite has quit IRC | 13:33 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend https://review.openstack.org/252032 | 13:34 |
| openstackgerrit | Jaume Devesa proposed openstack/tripleo-docs: Extending the image build information https://review.openstack.org/270290 | 13:37 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add Rabbit IPv6 only support https://review.openstack.org/269058 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 Support to Isolated Networks https://review.openstack.org/235423 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes https://review.openstack.org/270831 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6 https://review.openstack.org/270110 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6 https://review.openstack.org/270154 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Allow to enable IPv6 on Corosync https://review.openstack.org/267073 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Set /64 cidr_netmask for pcmk VIPs when IPv6 https://review.openstack.org/267647 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fix vncproxy_host for IPv6 https://review.openstack.org/287068 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 versions of the Controller NIC configs https://review.openstack.org/269872 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fixup swift device string to delimit the ipv6 address with [] https://review.openstack.org/267523 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Make the Neutron subnet ipv6_{ra,address}_mode configurable https://review.openstack.org/271208 | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Support the deployment of Ceph over IPv6 https://review.openstack.org/272089 | 13:40 |
| gfidente | jprovazn, shadower so https://review.openstack.org/272089 is the tip | 13:40 |
| gfidente | applies on master | 13:41 |
| *** masco has quit IRC | 13:46 | |
| EmilienM | gfidente: https://bugs.launchpad.net/tripleo/+bug/1536103 | 13:47 |
| openstack | Launchpad bug 1536103 in tripleo "nova-consoleauth and nova-api fail to start for ipv6" [Medium,In progress] - Assigned to Giulio Fidente (gfidente) | 13:47 |
| EmilienM | I have a fix in puppet-nova | 13:47 |
| EmilienM | gfidente: just fyi, for your reviews, we should stop adding brackets ourselves, we have now the way to do it in our puppet modules, let's use it | 13:48 |
| gfidente | EmilienM, ack that patch is part of the tree | 13:48 |
| hewbrocca | derekh: you get yer nics working? | 13:48 |
| gfidente | EmilienM, yes I think it's time to update where necessary | 13:48 |
| gfidente | EmilienM, so I am testing the TIP | 13:48 |
| gfidente | if it passes I think the best approach is to go where we need the updates and update the specific patch, not add more patches on top | 13:49 |
| gfidente | but I want to see it passing with the 'known' patches first | 13:49 |
| jprovazn | gfidente, so applying https://review.openstack.org/#/c/272089/ and then the remaining IPv6 patches on current master is the right approach, right? | 13:50 |
| gfidente | jprovazn, ack, just checkout the tree from 272089 | 13:50 |
| derekh | hewbrocca: nope, trying a new image now, not too optimistic at the moment | 13:50 |
| gfidente | it'll bring in all 11 deps | 13:50 |
| * hewbrocca facepalm | 13:50 | |
| jprovazn | ack | 13:50 |
| hewbrocca | hmm | 13:51 |
| hewbrocca | derekh: they *are* somewhat wacky 10Gbe nics | 13:51 |
| EmilienM | gfidente: I disagree actually, we can land patches that work and iterate later. One big problem I've noticed in TripleO is that patches stand for very long time, instead of iterations on multiple patches. | 13:51 |
| hewbrocca | It's not out of the question that a special driver was required | 13:51 |
| gfidente | EmilienM, oh you'd merge the 'suboptimal' versions if they work you mean? | 13:51 |
| gfidente | EmilienM, let's see them working first | 13:52 |
| EmilienM | gfidente: yes | 13:53 |
| gfidente | agreed | 13:53 |
| shadower | gfidente: you mean the whole patchset applies or just that https://review.openstack.org/#/c/272089/ | 13:53 |
| EmilienM | the ipv6 network isolation patch is a proof of non-iteration | 13:53 |
| EmilienM | where we continue to send patchsets | 13:53 |
| gfidente | shadower, if you do checkout and not cherry-pick, it'll bring in the tree with all the 12 patches | 13:53 |
| shadower | right yeah | 13:53 |
| gfidente | the new ui is not nice, but on the right where it says 'related' it is actually a tree | 13:54 |
| shadower | hey so is the main ipv6 patch (https://review.openstack.org/#/c/235423/) blocked from merging until we fix net iso in the ci? | 13:54 |
| *** rlandy has joined #tripleo | 13:54 | |
| derekh | hewbrocca: yup | 13:54 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make OpenStack service ports configurable in HAProxy https://review.openstack.org/287199 | 13:55 |
| *** masco has joined #tripleo | 13:58 | |
| *** leanderthal has quit IRC | 13:58 | |
| EmilienM | gfidente: just to make sure i understand about https://review.openstack.org/#/c/270110/4/puppet/all-nodes-config.yaml | 13:58 |
| *** rleander_ has joined #tripleo | 13:58 | |
| *** rleander_ is now known as leanderthal | 13:59 | |
| EmilienM | memcache_node_ips_v6 is in fact memcache_node_ips with brackets, right? | 13:59 |
| dprince | derekh: which F21 image? From nodepool? | 14:00 |
| dprince | derekh: does nodepool even support F21 anymore (I'm not sure it does) | 14:01 |
| derekh | dprince: no a new one I've build for the testenvs, we had been using F20, I'm trying F21 to see if it fixes the problem we were seeing with the netiso test | 14:01 |
| dprince | EmilienM: regarding landing patches and iterating. In general I agree, but the issue in TripleO is we continue to add new features without adding any CI coverage on our existing features | 14:02 |
| dprince | EmilienM: so the reason why patches sit so long is we don't actually know if they break things... we have to manually test | 14:03 |
| gfidente | EmilienM, not exactly it's got the inet6: prefix too | 14:03 |
| *** lucas-hungry is now known as lucasagomes | 14:03 | |
| dprince | EmilienM: ideally we'd have CI coverage on some of the features we added like 6 months ago... and then we wouldn't be in this situation | 14:03 |
| gfidente | EmilienM, so it wasn't "just" brackets in this case :( | 14:03 |
| *** olap has quit IRC | 14:04 | |
| EmilienM | gfidente: ok, so I have no patch in puppet-nova to do. | 14:04 |
| dprince | derekh: no kernal module was required that I'm aware of. We did set some sysctl settings to disable ARP flux | 14:05 |
| dprince | derekh: but that shouldn't impact you actually seeing the NICs | 14:06 |
| *** olap has joined #tripleo | 14:06 | |
| *** masco has quit IRC | 14:06 | |
| dprince | derekh: f20 is quite old. So I do agree trying a newer Fedora is worth a shot. This could just be some odd OVS bug that has been fixed | 14:07 |
| derekh | dprince: yup, i'll be adding those, ya didn't think we added anything special. | 14:07 |
| *** weshay has joined #tripleo | 14:07 | |
| derekh | dprince: yup, thats what we're thinking (hoping) | 14:07 |
| dprince | derekh: would F22 be a better candidate though. I at least have one local box and can confirm that it does work with regards to OVS bridge vlan trunking, etc. | 14:08 |
| hewbrocca | Is there any virtue in doing Centos 7? | 14:08 |
| hewbrocca | it's closer to a real deployment target... | 14:08 |
| hewbrocca | although I guess that doesn't matter for this exercise | 14:08 |
| dprince | hewbrocca: I'm not sure the tripleo-incubator scripts all support Centos 7. They are older... | 14:08 |
| derekh | dprince: it would but I'm trying to change as little as possible | 14:09 |
| hewbrocca | OK, as long as we've considered it | 14:09 |
| dprince | derekh: sure, well a new OS is a new OS I think | 14:09 |
| derekh | dprince: that not exactly true, trying to get a bunch of stuff working with a new os may show p 3 problems on average | 14:10 |
| derekh | dprince: jumping 2 stpes is 6 problems to deal with | 14:10 |
| derekh | I'm making up numbers but the point I think is valid | 14:11 |
| derekh | has anybody dies nova bare matel with F22 for example | 14:11 |
| dprince | derekh: being cautious is fine. I'm just pointing out that I can't remember if I ever testing network iso on F21 | 14:11 |
| derekh | *dies -> tried | 14:11 |
| *** dmacpher has joined #tripleo | 14:11 | |
| dprince | derekh: but I know it works on F22... | 14:11 |
| dprince | derekh: in virt w/ vlans, etc. | 14:12 |
| dtantsur | trown, is promotion job blocked by something now? | 14:12 |
| derekh | dprince: yup, if I can actually get F21 to boot, I'll try F22 aswell | 14:12 |
| *** ibravo has joined #tripleo | 14:12 | |
| gfidente | EmilienM, maybe we can use your puppet function for the mongodb patch? | 14:13 |
| trown | dtantsur: promotion job is always blocked :p, but we have mitaka up to a few days old | 14:13 |
| dtantsur | trown, yeah, but not the inspector packaging change fixing acl on /var/log/ironic-inspector.. | 14:14 |
| gfidente | going for lunch, ttyl | 14:14 |
| EmilienM | gfidente: yes maybe | 14:14 |
| trown | dtantsur: ah got it. I am hoping for a promote today in RDO... tripleo uses there own process | 14:15 |
| dtantsur | sigh.. | 14:15 |
| dtantsur | ETHINGSTAKETOOLONG | 14:15 |
| *** tzumainn has joined #tripleo | 14:17 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Upgrades: Add 'stack upgrade' command https://review.openstack.org/286606 | 14:17 |
| *** bandini has quit IRC | 14:18 | |
| trown | dtantsur: in this case I blame glance :P, they broke the swift store when using keystone v2, however they quickly fixed it too so a bit of a wash | 14:18 |
| *** bandini has joined #tripleo | 14:18 | |
| *** lblanchard has joined #tripleo | 14:18 | |
| hewbrocca | -EOPENSTACK | 14:19 |
| *** bandini has quit IRC | 14:19 | |
| *** masco has joined #tripleo | 14:19 | |
| *** bandini has joined #tripleo | 14:19 | |
| trown | ya who knew integrating all of openstack is kind of hard | 14:19 |
| *** cmyster has joined #tripleo | 14:19 | |
| *** cmyster has quit IRC | 14:19 | |
| *** cmyster has joined #tripleo | 14:19 | |
| *** tiswanso has joined #tripleo | 14:20 | |
| dtantsur | trown, glance broke swift breaking keystone... in the house that hewbrocca built | 14:20 |
| hewbrocca | *kaboom* | 14:20 |
| dtantsur | (sorry, could not resist) | 14:20 |
| trown | :) | 14:20 |
| bandini | /win 7 | 14:20 |
| hewbrocca | ě+šýí+ | 14:20 |
| dtantsur | bandini, no such window | 14:20 |
| shardy | Another review for https://review.openstack.org/#/c/284723 would be good, tested locally and it fixes bug #1552209 | 14:20 |
| openstack | bug 1552209 in tripleo "introspection outputs a keystoneclient deprecation warning" [Medium,In progress] https://launchpad.net/bugs/1552209 - Assigned to Dmitry Tantsur (divius) | 14:20 |
| hewbrocca | as long as we're injecting random stuff into the channel | 14:20 |
| bandini | dtantsur: :) | 14:21 |
| *** mkovacik has joined #tripleo | 14:24 | |
| dprince | gfidente, derekh: hey, an idea. Since we are in the business of trying to rebuild the testenv's | 14:30 |
| dprince | gfidente, derekh: back when we initially did IPv4 network iso I had thought we might use the multiple-nics templates for upstream CI | 14:30 |
| dprince | gfidente, derekh: so rather than trying to get VLANs to work with our existing (single nic) environments would it make sense to just add more NICs to the each virtual machine? | 14:31 |
| dprince | gfidente, derekh: and then we can use 'multiple-nics' instead | 14:31 |
| *** tiswanso has quit IRC | 14:31 | |
| hewbrocca | dprince: that seems very sensible | 14:32 |
| derekh | dprince: would work for me, the version of devtest_testenv we have been using is v. old so we may hit a few snags that need to be worked out | 14:32 |
| dprince | gfidente: in other words lets just skip the fancy Vlans for now. We'd still get a lot of the architecture tested by just using multiple NICs | 14:33 |
| *** tiswanso has joined #tripleo | 14:33 | |
| hewbrocca | It seems less risky/complex | 14:33 |
| hewbrocca | but, what do I know | 14:34 |
| *** ayoung has joined #tripleo | 14:39 | |
| gfidente | dprince, derekh, ack | 14:42 |
| gfidente | derekh, that means we'll have to rebuild all the testenvs though | 14:43 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Add overcloud-base element https://review.openstack.org/286553 | 14:43 |
| dprince | gfidente: FWIW, this is what I had always envisioned we'd have to do anyways. And was why I never pushed the job yet... | 14:44 |
| gfidente | dprince yes it's probably even closer to real world use case | 14:44 |
| gfidente | EmilienM, I have some invalid parameter enable_bootstrap on Keyston | 14:45 |
| gfidente | EmilienM, do I miss a patch for puppet-keystone which went in recently? | 14:45 |
| *** rbrady has quit IRC | 14:48 | |
| EmilienM | gfidente: the only one thing is https://github.com/openstack/puppet-keystone/commit/f3efc1295893bba5e1fb5664fbb9537d89c2fd0c | 14:48 |
| EmilienM | but HA job des not use it | 14:48 |
| gfidente | oh it's exactly the param I miss though | 14:49 |
| gfidente | what do you mean HA does not use it? | 14:49 |
| EmilienM | gfidente: https://github.com/openstack/tripleo-heat-templates/commit/f752e1fe5301abdbf2e8405c595a05e08f0c609b | 14:52 |
| EmilienM | related to LP/1551501 | 14:52 |
| gfidente | okay so it's disabled but I need the module update, gotcha | 14:53 |
| gfidente | one more nice thing about not calling classes from the modules | 14:53 |
| gfidente | is that hiera automatic binding does not fail if the param is not available in the class | 14:53 |
| EmilienM | gfidente: no, we need to manage keystone resources with puppet | 14:53 |
| EmilienM | and not with os cloud config | 14:54 |
| EmilienM | there is a long standing patch about it | 14:54 |
| *** pradk has joined #tripleo | 14:55 | |
| *** sshnaidm_ has joined #tripleo | 14:56 | |
| *** liverpooler has quit IRC | 14:56 | |
| *** dustins has joined #tripleo | 14:56 | |
| *** trozet has joined #tripleo | 14:56 | |
| *** sshnaidm has quit IRC | 14:56 | |
| *** sshnaidm_ has quit IRC | 14:56 | |
| gfidente | EmilienM, but the module change is going into OPM right? | 14:57 |
| *** dmsimard is now known as dmsimard|afk | 14:58 | |
| EmilienM | it is in OPM | 14:58 |
| gfidente | ack | 14:58 |
| gfidente | I had it from sources | 14:58 |
| *** tosky has quit IRC | 15:00 | |
| *** tosky has joined #tripleo | 15:01 | |
| marios | jistr: we forgot about the benefits of being able to pass the rpc pin into the compute upgrade script | 15:03 |
| jistr | marios: ah, right... | 15:03 |
| *** sshnaidm has joined #tripleo | 15:03 | |
| jistr | marios: so do we just handle swift nodes different than the rest, or do we do the script delivery in a separate stack-update? | 15:04 |
| marios | jistr: so is a delivery workflow worth it for that? otherwise we hard code it and update it in the tripleo-common script when needed | 15:04 |
| marios | jistr: heh good question. :) | 15:05 |
| jistr | marios: ok hardcoding is an option too. | 15:05 |
| jistr | marios: tbh i don't like "handle swift nodes different than the rest" very much | 15:05 |
| marios | jistr: so the initial 'delivery' worfklow should finish *very* quickly fwiw right (just sending the upgrade scripts) | 15:05 |
| jistr | marios: inclined to select between hardcoding and a separate delivery workflow for all | 15:05 |
| marios | jistr: agreed on the choice | 15:06 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: WIP: Enable network isolation in all CI jobs https://review.openstack.org/273424 | 15:06 |
| jistr | marios: yeah i think we'd do the same thing as with upgrades, we'd disable the -post.yaml files (Puppet runs) | 15:06 |
| jistr | that makes it considerably quicker | 15:06 |
| marios | jistr: let's see what it looks like on paper and take it from there (I'll start from splitting the wofklow and then go to the tripleo-common script). as we said we already have the compute landed (and as we forgot for the rpm pin passing) | 15:07 |
| marios | rmp/rpc | 15:07 |
| jistr | marios: yeah. and with the separate script delivery workflow, we don't need any node type detection logic in tripleo-common bash, which is a plus too imho | 15:08 |
| marios | jistr: well exactly, you know my resistance to growing that script | 15:08 |
| jistr | marios: yea, +1 | 15:08 |
| *** tosky has quit IRC | 15:10 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Support adding a swap file to overcloud nodes https://review.openstack.org/287158 | 15:10 |
| hewbrocca | woo more merges! | 15:12 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/241408 | 15:12 |
| *** rbrady has joined #tripleo | 15:13 | |
| jrist | guys I've been working on a new logo on my off time. | 15:15 |
| jrist | http://wixagrid.com/tripleo/tripleo_svg.html | 15:15 |
| marios | jrist: that is fantastic | 15:15 |
| *** tosky has joined #tripleo | 15:15 | |
| marios | jrist: HOW ARE YOU?! | 15:15 |
| jrist | marios: thansk! good how are you? | 15:15 |
| jrist | should I propose it on the mailing list? | 15:16 |
| marios | +1 | 15:16 |
| *** jaosorior has quit IRC | 15:16 | |
| shadower | I like it | 15:16 |
| hewbrocca | totally | 15:16 |
| jrist | woot | 15:16 |
| hewbrocca | dprince: ^^^ | 15:16 |
| dprince | jrist: nice, FWIW I have an actual svg file of my old clunky fatty owl too | 15:18 |
| dprince | jrist: your own has no feet? | 15:18 |
| jrist | haha | 15:18 |
| jrist | everyone says that | 15:18 |
| jrist | :) | 15:18 |
| dprince | jrist: I guess you when with ears instead | 15:18 |
| jrist | dprince: I can add some if it is contentious | 15:19 |
| dprince | jrist: we want our deployment solution to be well grounded I think | 15:19 |
| derekh | lol | 15:19 |
| jrist | :) | 15:19 |
| gfidente | jrist, I think it's totally nice but but but but but but, the eyes of that owl are a little to close and in the mid-height so it looks like this is a stoned owl | 15:19 |
| gfidente | jrist, not that I have anything against getting stoned but | 15:19 |
| gfidente | jrist, it doesn't make justice to tripleo | 15:19 |
| jrist | haha | 15:19 |
| jrist | it's supposed to be "cute" | 15:19 |
| shadower | "you've got to be stoned to try tripleo" | 15:20 |
| jrist | theoretically the eyes can move since it's svg :) | 15:20 |
| jrist | easter egg? | 15:20 |
| marios | lol gfidente | 15:20 |
| marios | gfidente: though i think it would be worse if the eyes were pointing outwards | 15:21 |
| gfidente | it's just the way he is looking at me | 15:21 |
| marios | i mean if they were moved it may look weird | 15:21 |
| jrist | lol | 15:22 |
| jrist | gfidente: https://i.imgur.com/pHJwJg7.png | 15:22 |
| jrist | gfidente: any better? | 15:22 |
| gfidente | jrist, I think it's better yes, can they go a little bit down too? | 15:23 |
| jrist | oh man picky picky | 15:23 |
| gfidente | and thanks a lot for taking care of me | 15:23 |
| gfidente | yes indeed | 15:23 |
| gfidente | :( | 15:23 |
| jrist | anything for you gfidente | 15:24 |
| jrist | https://i.imgur.com/POIDamx.png | 15:24 |
| jrist | gfidente: ^ | 15:24 |
| gfidente | oh this is very clever owl | 15:24 |
| gfidente | marios, ^^ | 15:24 |
| gfidente | thanks :) | 15:24 |
| jrist | gfidente: you like it that way? | 15:25 |
| gfidente | yes | 15:25 |
| hewbrocca | excellent | 15:25 |
| gfidente | hewbrocca, sorry | 15:25 |
| openstackgerrit | Merged openstack/instack-undercloud: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/265382 | 15:26 |
| hewbrocca | woop woop woop! | 15:26 |
| hewbrocca | ^^^ | 15:26 |
| marios | gfidente: yeah it works well | 15:27 |
| bnemec | Even better, that clears the way for the tuskar removal patch. :-) | 15:27 |
| jrist | \o/ | 15:27 |
| jrist | bnemec: the logo for the merged patch? :) | 15:27 |
| *** cmyster has quit IRC | 15:27 | |
| jrist | s/for/or/g | 15:28 |
| *** masco has quit IRC | 15:28 | |
| bnemec | :-) | 15:28 |
| EmilienM | merging a patch is like a celebration \o/ | 15:28 |
| dtantsur | jrist, awesome owl! :) | 15:29 |
| gfidente | EmilienM, we'll merge 12 and celebrate ^12 bugs | 15:29 |
| jrist | dtantsur: is it good as Lucas' bear? | 15:29 |
| bnemec | I'm still jealous of the Heat team's fire Elmo. | 15:29 |
| jrist | gfidente: svg updated http://wixagrid.com/tripleo/tripleo_svg.html | 15:30 |
| gfidente | jrist, serious guy | 15:30 |
| gfidente | it's looking at me to actually deploy something now | 15:30 |
| jrist | :) | 15:30 |
| bnemec | http://data.whicdn.com/images/115236260/large.gif | 15:30 |
| dtantsur | jrist, a bit less drive, but still cool :) | 15:30 |
| jrist | haha | 15:30 |
| jrist | I'll take it | 15:30 |
| trown | he does look expectant of something happening | 15:30 |
| gfidente | trown++ | 15:31 |
| gfidente | jrist++ | 15:31 |
| jrist | I'm going to put an easter egg in - if you do up up down down left right b a enter it will make the eyes move independently and blink | 15:32 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Upgrades: quiet yum update https://review.openstack.org/286155 | 15:32 |
| bnemec | Would somebody like to do the honors on https://review.openstack.org/#/c/280431 ? | 15:33 |
| bnemec | We have at least three different CI patches dependent on it now. | 15:33 |
| EmilienM | dprince, gfidente: I have a dual stack patch for puppet-tripleo, can you look at it when possible? it's backward compatible | 15:33 |
| dprince | EmilienM: "dual stack"? | 15:34 |
| EmilienM | dprince: ipv6 dual stack | 15:34 |
| dprince | EmilienM: oh, I see | 15:34 |
| derekh | hewbrocca: gfidente - dprince is going to try the multi nic thing as I'm on PTO tomorrow and probably wont get it done this evening | 15:34 |
| EmilienM | dprince: a very common use case for people deploying ipv6 | 15:35 |
| gfidente | we should probably also try having 'external' on ipv4 | 15:35 |
| EmilienM | dprince: everything is in the commit message | 15:35 |
| gfidente | while keeping the others on ipv6 | 15:35 |
| hewbrocca | derekh: OK | 15:36 |
| hewbrocca | dprince: any chance you'd have that rolling today? | 15:36 |
| dprince | hewbrocca: perhaps, it'll be later this evening most likely when I deploy it. And then we'd have to test the patches to try it | 15:37 |
| hewbrocca | dprince: rock on | 15:37 |
| hewbrocca | anything you need from anyone to help | 15:37 |
| dprince | hewbrocca: rebuilding the testenv's is a bit disruptive in the middle of the day | 15:37 |
| *** panda has quit IRC | 15:37 | |
| *** cmyster has joined #tripleo | 15:37 | |
| *** olap has quit IRC | 15:38 | |
| hewbrocca | dprince: makes sense | 15:38 |
| *** panda has joined #tripleo | 15:38 | |
| shadower | bnemec: looks good but I don't think I should be the one to +2/A the patch... | 15:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Upgrades: install zaqarclient https://review.openstack.org/286156 | 15:39 |
| *** yamahata has joined #tripleo | 15:39 | |
| gfidente | shadower, go shadower we're using it already | 15:39 |
| *** olap has joined #tripleo | 15:39 | |
| jistr | gfidente: not sure if we need another CI cycle because of the commit message comment. https://review.openstack.org/#/c/283712 personally i think we should land it | 15:40 |
| shadower | gfidente: where? Also, I haven't done a ton of reviews in literally a year so the fact I can still +2 is an overshight rather than anything eles | 15:40 |
| jistr | (jaosorior is offline though) | 15:40 |
| dprince | derekh: oh BTW. I hit this yesterday afternoon, looks like all the nightly CI jobs failed due to it as well https://bugs.launchpad.net/tripleo/+bug/1552211 | 15:41 |
| openstack | Launchpad bug 1552211 in tripleo "glance images fail to upload (500 errors)" [Undecided,New] | 15:41 |
| derekh | dprince: thanks, trown is that the error I saw ye talking about on #rdo ? | 15:42 |
| gfidente | jistr, it's a trap, i wanted it to land https://review.openstack.org/#/c/256670/ | 15:42 |
| trown | dprince: derekh, yep it is fixed | 15:42 |
| trown | but just a couple hours ago | 15:42 |
| dprince | trown: was there a ticket? if not can you update that one with the details? | 15:43 |
| trown | https://review.openstack.org/#/c/287050 was the fix | 15:43 |
| trown | dprince: ya, I will update the launchpad | 15:43 |
| dprince | trown: thanks | 15:44 |
| *** xinwu has joined #tripleo | 15:44 | |
| *** fgimenez has quit IRC | 15:45 | |
| trown | dprince: derekh, I am actually fairly optimistic https://ci.centos.org/view/rdo/job/rdo-delorean-promote-mitaka/ will pass | 15:46 |
| derekh | trown: cool beans | 15:46 |
| trown | which would give us a hash to test tripleoci against as well | 15:46 |
| *** fgimenez has joined #tripleo | 15:46 | |
| *** fgimenez has quit IRC | 15:46 | |
| *** fgimenez has joined #tripleo | 15:46 | |
| gfidente | jistr, commented on the change to see if we can still 'avoid' it | 15:47 |
| shardy | I just hit https://bugs.launchpad.net/tripleo/+bug/1552288 which will probably mean the periodic job breaks as soon as we land https://review.openstack.org/#/c/279218/ | 15:47 |
| openstack | Launchpad bug 1552288 in tripleo "Re-running undercloud install fails with neutron error" [Undecided,New] | 15:47 |
| EmilienM | dprince: where is your patch that added swap in OOO CI? | 15:48 |
| derekh | dprince: I forgot to mention, when you want to test the testenvs, you'll probably want to kill the 4 python/gear processes on the testenv , then manually grab one of the json files and use it on the jenkins slave I spun up yesterday | 15:48 |
| *** trown is now known as trown|brb | 15:48 | |
| gfidente | AND WHY IS MY UNDERCLOUD FAILING TO BIND PORTS NOW :( | 15:49 |
| EmilienM | dprince: nevermind, https://github.com/openstack-infra/tripleo-ci/commit/39a0d985b87f9d32260d9bf032ae5705fc6ca2a3 | 15:49 |
| dprince | derekh: no need, I'm gonna ninja this in so it works right off the bat ;) | 15:49 |
| hewbrocca | excellent | 15:50 |
| dprince | derekh: good to know, I'll resort to that when I break it | 15:50 |
| derekh | dprince: happy days | 15:50 |
| *** jistr has quit IRC | 15:51 | |
| derekh | dprince: on jenkins slave tab 3, jimmy rig ./toci_gate_test.sh to use your json file and then run "ZUUL_REF=None ZUUL_UUID= ZUUL_CHANGES=tripleo-ci WORKSPACE=/home/jenkins/workspace/job DEVSTACK_GATE_TIMEOUT=500 NODE_NAME=derekh-test.novalocal ./toci_gate_test.sh" | 15:52 |
| *** trown|brb is now known as trown | 15:53 | |
| *** hjensas has quit IRC | 15:54 | |
| *** dmacpher is now known as dmacpher-afk | 15:55 | |
| *** paramite has joined #tripleo | 16:00 | |
| *** rhallisey has quit IRC | 16:01 | |
| *** apetrich has quit IRC | 16:01 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Allow container template to recognize an update https://review.openstack.org/272212 | 16:01 |
| *** trown is now known as trown|meeting | 16:01 | |
| *** rhallisey has joined #tripleo | 16:02 | |
| *** bvandenh has quit IRC | 16:04 | |
| *** rhallisey has quit IRC | 16:04 | |
| *** tiswanso has quit IRC | 16:04 | |
| *** pcaruana has quit IRC | 16:05 | |
| *** rhallisey has joined #tripleo | 16:05 | |
| *** tiswanso has joined #tripleo | 16:09 | |
| *** mbound has quit IRC | 16:15 | |
| *** xinwu has quit IRC | 16:15 | |
| *** ohamada has quit IRC | 16:15 | |
| dtantsur | sorry folks, but I'll keep on trying: please review https://review.openstack.org/#/c/277974/ please please. only 40 LoC of changes, will help me with getting rid of this "profile matching" topic | 16:15 |
| *** mbound has joined #tripleo | 16:15 | |
| *** ohamada has joined #tripleo | 16:15 | |
| *** athomas has quit IRC | 16:16 | |
| *** athomas has joined #tripleo | 16:16 | |
| Erming_ | Hi, after deployed overcloud, I ran Tempest and got the following error. what does this mean? | 16:16 |
| Erming_ | tempest_lib.exceptions.IdentityError: Got identity error | 16:16 |
| shardy | dtantsur: I was actually testing that when I discovered re-running undercloud install is broken :( | 16:16 |
| shardy | the patch looks fine tho | 16:16 |
| dtantsur | :) | 16:17 |
| shardy | dtantsur: one question, what happens if inspection_extras=true and the image doesn't contain python-hardware? | 16:18 |
| shardy | does introspection fail, or just return an error in the extra blob? | 16:18 |
| dtantsur | shardy, fail | 16:19 |
| shardy | ack, thanks | 16:19 |
| *** bvandenh has joined #tripleo | 16:25 | |
| *** lzachery has joined #tripleo | 16:31 | |
| *** electrofelix has quit IRC | 16:34 | |
| *** tiswanso has quit IRC | 16:34 | |
| *** tiswanso has joined #tripleo | 16:34 | |
| *** dshulyak has quit IRC | 16:36 | |
| *** david-lyle has quit IRC | 16:36 | |
| *** Marga__ has joined #tripleo | 16:36 | |
| *** david-lyle has joined #tripleo | 16:38 | |
| *** yamahata has quit IRC | 16:39 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 16:45 |
| *** rlandy has quit IRC | 16:51 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades https://review.openstack.org/287318 | 16:58 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Moves the swift start/stop into the common_functions.sh file https://review.openstack.org/287320 | 16:59 |
| *** lzachery has quit IRC | 17:00 | |
| *** penick has quit IRC | 17:01 | |
| *** trown|meeting is now known as trown | 17:01 | |
| trown | has anyone noticed the ping test hanging indefinitely after success, before cleanup? | 17:01 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades https://review.openstack.org/287318 | 17:02 |
| *** rlandy has joined #tripleo | 17:04 | |
| *** leanderthal is now known as leanderthal|afk | 17:04 | |
| *** rhallisey has quit IRC | 17:05 | |
| *** rhallisey has joined #tripleo | 17:06 | |
| *** xinwu has joined #tripleo | 17:06 | |
| *** olap has quit IRC | 17:06 | |
| *** trown is now known as trown|lunch | 17:08 | |
| *** yamahata has joined #tripleo | 17:09 | |
| *** electrofelix has joined #tripleo | 17:10 | |
| *** rdopiera has quit IRC | 17:10 | |
| *** fgimenez has quit IRC | 17:11 | |
| *** pleia2 has joined #tripleo | 17:12 | |
| *** aufi has quit IRC | 17:12 | |
| *** ohamada has quit IRC | 17:13 | |
| *** ccamacho has quit IRC | 17:13 | |
| *** dshulyak has joined #tripleo | 17:15 | |
| *** dtantsur is now known as dtantsur|afk | 17:17 | |
| *** devvesa has quit IRC | 17:17 | |
| *** dshulyak has quit IRC | 17:20 | |
| *** masco has joined #tripleo | 17:22 | |
| *** ccamacho has joined #tripleo | 17:23 | |
| *** electrofelix has quit IRC | 17:25 | |
| gfidente | EmilienM, fwiw, I'm hitting this on centos https://bugzilla.redhat.com/show_bug.cgi?id=1298401 | 17:25 |
| openstack | bugzilla.redhat.com bug 1298401 in memcached "Memcached fails to start in ipv6 environment" [High,Closed: errata] - Assigned to hguemar | 17:25 |
| EmilienM | gfidente: how can I help? | 17:25 |
| gfidente | on centos we have memcached-1.4.15-9.el7.x86_64 | 17:26 |
| gfidente | maybe mburned is our guy | 17:26 |
| gfidente | mbound, I think the version of memcached we have in centos is too old for https://bugzilla.redhat.com/show_bug.cgi?id=1298401 | 17:27 |
| openstack | bugzilla.redhat.com bug 1298401 in memcached "Memcached fails to start in ipv6 environment" [High,Closed: errata] - Assigned to hguemar | 17:27 |
| gfidente | that'll make ipv6 to fail on rdo | 17:27 |
| *** xinwu has quit IRC | 17:28 | |
| shardy | Anyone got advice on how best to debug baremetal boxes when ironic deploys the image but it's not bootable afterwards? | 17:29 |
| hewbrocca | lucasagomes: ^^^ | 17:30 |
| lucasagomes | shardy, hi there! | 17:30 |
| *** apetrich has joined #tripleo | 17:30 | |
| shardy | lucasagomes: hey! | 17:30 |
| lucasagomes | shardy, what error you are see there? | 17:31 |
| shardy | lucasagomes: so, I've got two desktop boxes, deploying OK via fake_pxe, but after one of them is deployed it ends up with an un-bootable image (the other, a mini-dell works fine) | 17:31 |
| *** dprince has quit IRC | 17:31 | |
| shardy | is it a case of booting with a livecd and looking at the bootloader and disk config? | 17:31 |
| shardy | It looks like it's not installed the bootloader at all AFAICS | 17:32 |
| lucasagomes | shardy, oh right it's the tenant's requested image then | 17:32 |
| lucasagomes | shardy, right, yeah... Do you see any grub problems when you try to boot it up? | 17:32 |
| shardy | lucasagomes: Yeah, the deploy ramdisk runs, then halts both boxes, but the overcloud image won't boot on the uefi one | 17:33 |
| lucasagomes | shardy, right... you sure the image being deployed contains the grub efi modules? | 17:33 |
| lucasagomes | shardy, if it was built with that grub2-deprecated element it won't contain it | 17:33 |
| lucasagomes | you have to use the grub2 (without the -deprecated) | 17:34 |
| shardy | lucasagomes: aha, I'm not sure tbh, it's our stock tripleo overcloud-full image | 17:34 |
| shardy | can I create an image which works for both uefi and non uefi? | 17:34 |
| lucasagomes | shardy, right that may be the case, i put a patch up for tripleo-ci relatively new that was changing it | 17:34 |
| lucasagomes | shardy, lemme grab the link | 17:34 |
| lucasagomes | shardy, yes | 17:34 |
| shardy | cool, I have a mixture of old/new boxes here :) | 17:35 |
| lucasagomes | shardy, https://review.openstack.org/#/c/263846/ | 17:35 |
| lucasagomes | shardy, yeah just make sure the image contains these packages installed in it https://github.com/openstack/diskimage-builder/blob/master/elements/grub2/pkg-map#L16-L18 | 17:36 |
| lucasagomes | then it should work on both uefi and bios legacy machines | 17:36 |
| shardy | lucasagomes: aha, yeah I should have that as I'm running latest master tripleoclient | 17:36 |
| lucasagomes | grub2-install is smart enough to detect if it's uefi or bios and copy the right modules over | 17:36 |
| shardy | I'll double check what's in the image tho | 17:37 |
| lucasagomes | shardy, alright, so it may be something more complex | 17:37 |
| lucasagomes | shardy, ack, also, when you deploy the node in Ironic did you pass boot_mode:uefi capability to it? | 17:37 |
| lucasagomes | if you do, ironic will create the EFI partition for that UEFI machine | 17:37 |
| *** dmsimard|afk is now known as dmsimard | 17:38 | |
| lucasagomes | shardy, the downside is that you will need 2 flavors 1 for uefi and 1 for bios | 17:38 |
| shardy | lucasagomes: ah, no I just passed boot_option:local | 17:38 |
| shardy | having two flavors is OK, I've already got a flavor for each type of machine | 17:39 |
| shardy | so I just add boot_mode:uefi to the capabilities? | 17:39 |
| lucasagomes | shardy, yeah | 17:39 |
| shardy | lucasagomes: great, thanks, let me try that | 17:39 |
| lucasagomes | shardy, it's also strange that iPXE booted the machine without using the efi ROM | 17:39 |
| lucasagomes | but as for local disk, I think that just passing boot_mode:uefi may solve the case | 17:40 |
| *** dshulyak has joined #tripleo | 17:40 | |
| lucasagomes | shardy, for ref http://docs.openstack.org/developer/ironic/deploy/install-guide.html?highlight=uefi#boot-mode-support | 17:40 |
| lucasagomes | it says pxe_ipmitool only but it works for pxe_* (/me needs to correct the docs) | 17:41 |
| shardy | lucasagomes: I enabled the uefi network stack with ipv4 PXE support in the bios, could that explain it? | 17:41 |
| lucasagomes | shardy, oh right, yeah | 17:41 |
| lucasagomes | shardy, cause in mitaka we added support for booting from uefi and iPXE if you need it | 17:41 |
| lucasagomes | shardy, https://review.openstack.org/#/c/257497/ (that's the patch for it) | 17:42 |
| *** xinwu has joined #tripleo | 17:42 | |
| lucasagomes | shardy, ok try with the boot_mode see if works, if not I we can debug it | 17:43 |
| *** mbound has quit IRC | 17:44 | |
| lucasagomes | the boot_mode uefi will also make sure you have a GPT partiton table | 17:44 |
| shardy | lucasagomes: ack, thanks for the help, rebooting to try boot_mode now :) | 17:44 |
| lucasagomes | which may be required for some UEFI systems | 17:44 |
| lucasagomes | shardy, cool | 17:44 |
| shardy | Ok, this is probably something we should reference/mention in the baremetal part of the tripleo docs I guess | 17:44 |
| * shardy reads ironic docs | 17:45 | |
| lucasagomes | indeed... many of this UEFI stuff is landing this cycle | 17:45 |
| lucasagomes | I current have some patches on the fly allowing operators to choose the disk label they want for the disks (say GPT even if it's BIOS to support partitions > 2TB) | 17:45 |
| lucasagomes | loads of things to document :-( | 17:46 |
| *** tosky_ has joined #tripleo | 17:47 | |
| *** bvandenh has quit IRC | 17:48 | |
| *** tosky has quit IRC | 17:48 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 17:48 |
| *** trown|lunch is now known as trown | 17:49 | |
| *** paramite is now known as paramite|afk | 17:51 | |
| *** electrofelix has joined #tripleo | 17:53 | |
| *** electrofelix has quit IRC | 17:53 | |
| trown | EmilienM: could you help me parse the commit message for https://review.openstack.org/#/c/286352/ ... what does it mean "when Keystone resources will be managed by Puppet"? for context promote job now fails HA because of this bootstrap issue | 17:58 |
| lucasagomes | shardy, I'm about to call it a day, is it ok if we continue tomorrow? | 17:58 |
| lucasagomes | (in case it doesn't work) | 17:59 |
| *** ibravo has quit IRC | 17:59 | |
| shardy | lucasagomes: sure - the local boot is now working, but the pxe boot of the deployment ramdisk isn't anymore | 17:59 |
| EmilienM | trown: overcloud keystone resources (endpoints, etc) are managed by os cloud config and not puppet | 17:59 |
| shardy | I'm also going to drop off soon, will ping you tomorrow if that's OK, thanks for the help! | 17:59 |
| *** tosky_ is now known as tosky | 17:59 | |
| EmilienM | trown: so puppet fails to bootstrap the db | 17:59 |
| EmilienM | because some data is already in there I presume | 18:00 |
| lucasagomes | shardy, odd... I wonder if it has to do with the now partitioned local disk?! | 18:00 |
| *** dprince has joined #tripleo | 18:00 | |
| lucasagomes | tho the ramdisk shouldn't depend on the disk at all | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| lucasagomes | shardy, ack, yeah tomorrow we sort it out | 18:00 |
| lucasagomes | shardy, g'night! | 18:00 |
| shardy | o/ | 18:00 |
| trown | EmilienM: ick... so that means we have to fix this new issue in os cloud config? | 18:01 |
| trown | EmilienM: it is only in the HA case | 18:01 |
| EmilienM | I'm sorry I have no idea and not investigated this bug | 18:01 |
| *** lucasagomes is now known as lucas-dinner | 18:01 | |
| *** shardy has quit IRC | 18:06 | |
| *** athomas has quit IRC | 18:06 | |
| *** xinwu has quit IRC | 18:08 | |
| *** david-lyle has quit IRC | 18:12 | |
| *** david-lyle has joined #tripleo | 18:13 | |
| *** absubram has joined #tripleo | 18:13 | |
| *** gfidente has quit IRC | 18:14 | |
| *** dprince has quit IRC | 18:16 | |
| *** dprince has joined #tripleo | 18:16 | |
| *** mikelk has quit IRC | 18:17 | |
| *** mcornea has quit IRC | 18:20 | |
| *** penick has joined #tripleo | 18:21 | |
| bnemec | Shameless plug: https://review.openstack.org/279822 needs a +2A and should unblock the work to have Puppet manage Keystone, which might help with the bootstrap issue. | 18:27 |
| *** shivrao has joined #tripleo | 18:28 | |
| *** ccamacho has quit IRC | 18:30 | |
| *** saneax is now known as saneax_AFK | 18:30 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/241408 | 18:34 |
| *** xinwu has joined #tripleo | 18:46 | |
| *** masco has quit IRC | 18:47 | |
| *** jcoufal has quit IRC | 18:50 | |
| *** tosky has quit IRC | 18:55 | |
| *** jcoufal has joined #tripleo | 18:56 | |
| slagle | bnemec: the deprecation warning there is premature though isn't it? | 19:02 |
| slagle | bnemec: if we land that...we're saying the os-cloud-config method is deprecated | 19:02 |
| slagle | with no guarantee the puppet bit ever gets landed | 19:02 |
| *** mgould has quit IRC | 19:03 | |
| bnemec | slagle: True, although we've actually been logging that deprecation warning for months now. :-/ | 19:05 |
| slagle | oh, hah | 19:05 |
| slagle | nm me then | 19:05 |
| slagle | there ya go, +A | 19:06 |
| bnemec | Well, you're right. Actually logging the deprecation should have been a follow-up change with a depends-on of the puppet keystone change. | 19:06 |
| slagle | bnemec: i'll trade you, an easy backport: https://review.openstack.org/#/c/286746/ | 19:07 |
| bnemec | slagle: Nice, thanks | 19:08 |
| trown | slagle: bnemec, is there a patch for the puppet managing keystone? | 19:15 |
| slagle | yea... | 19:15 |
| bnemec | trown: https://review.openstack.org/#/c/244162/ | 19:15 |
| trown | whoa double revert | 19:15 |
| openstackgerrit | Merged openstack/instack-undercloud: Set max_resources_per_stack to -1 https://review.openstack.org/286746 | 19:16 |
| bnemec | Yeah, it merged, we found out it broke upgrades so we reverted it, but we still want to do it -> another revert revert. | 19:16 |
| trown | bnemec: did anything change wrt upgrades? or just needs more testing? | 19:17 |
| slagle | there were several bugs fixed on the puppet side | 19:17 |
| openstackgerrit | Merged openstack/python-tripleoclient: Check Keystone init before running it https://review.openstack.org/279822 | 19:17 |
| trown | bnemec: mind if I fix up the merge conflict? I am interested if that allows us to get trunk working | 19:18 |
| openstackgerrit | Merged openstack/puppet-tripleo: Always override X-Forwarded-Proto header for Heat https://review.openstack.org/287047 | 19:18 |
| bnemec | trown: No, go for it. The only reason I'm the owner is that I submitted the revert, so I also submitted a followup to undo it. | 19:19 |
| bnemec | trown: Don't miss Juan's comment on the previous patch set either. I think there was a problem with the last rebase of the patch. | 19:19 |
| trown | bnemec: cool, will check that | 19:19 |
| bnemec | Err, maybe it was a couple of rebases ago. Hard to keep track. :-) | 19:20 |
| trown | ya I see it | 19:20 |
| slagle | before we merge it, we should let jistr test it in an upgrade attempt | 19:20 |
| *** rwsu has quit IRC | 19:20 | |
| *** rwsu has joined #tripleo | 19:21 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-cloud-config: Updated from global requirements https://review.openstack.org/285049 | 19:22 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/268528 | 19:24 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests https://review.openstack.org/244162 | 19:26 |
| trown | bnemec: your rpm builder will build with the depends-on? | 19:28 |
| bnemec | trown: No, but the client patch has merged so you should just need to pull down the latest one. | 19:29 |
| trown | bnemec: cool I will give it a go | 19:29 |
| trown | I get why heat added it, but the confirmation on whether I want to delete my overcloud is annoying | 19:31 |
| *** rwsu has quit IRC | 19:32 | |
| dprince | bnemec: could you merge this https://review.openstack.org/#/c/283221/ | 19:32 |
| trown | hmm... heat stack-delete is misbehaving quite a bit for me today... I wonder if the two things are related | 19:32 |
| bnemec | dprince: Done | 19:34 |
| * bnemec needs food | 19:34 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests https://review.openstack.org/244162 | 19:36 |
| dprince | bnemec: thanks | 19:37 |
| *** panda has quit IRC | 19:37 | |
| *** rwsu has joined #tripleo | 19:38 | |
| *** panda has joined #tripleo | 19:38 | |
| openstackgerrit | Merged openstack/instack-undercloud: Wire enable_zaqar into hiera template https://review.openstack.org/283221 | 19:42 |
| rhallisey | has anyone deployed network isolation lately been unable to delete their stack after? | 19:42 |
| ayoung | after tripleo.sh --overcloud-deploy I don't see the overcloudrc file anywhere...what happened to it? | 19:42 |
| trown | rhallisey: I have been having issues today deleting stacks w/o network isolation | 19:44 |
| *** xinwu has quit IRC | 19:44 | |
| rhallisey | basically heat-engine is on a loop where it says it's destroying something then sleeps | 19:45 |
| trown | rhallisey: yep, the ping test actually hangs on trunk when trying to delete the stack on cleanup | 19:45 |
| trown | rhallisey: are you using heat/heatclient from trunk? or the tripleoci repo setup? | 19:46 |
| rhallisey | ya I'm using trunk | 19:46 |
| trown | cool, so we are likely hitting the same issue | 19:46 |
| rhallisey | strength in numbers | 19:46 |
| rhallisey | trown, I've tried destroying it manually | 19:47 |
| rhallisey | deleting neutron ports nova instances... | 19:47 |
| rhallisey | still didn't work | 19:47 |
| trown | ya same | 19:47 |
| rhallisey | crap | 19:47 |
| trown | heat had a bit of a merge frezy yesterday too | 19:48 |
| trown | nothing jumps out as being related to stack-delete though | 19:49 |
| trown | although this heatclient patch looks suspiciously related https://github.com/openstack/python-heatclient/commit/a3c32990a81b202ebac7408a9da08dcf03c8306c | 19:50 |
| *** akrivoka has quit IRC | 19:51 | |
| trown | oh... but that is for `openstack stack delete` | 19:51 |
| trown | rhallisey: does `openstack stack delete` behave differently? I just started a redeploy of my env to test something else | 19:52 |
| rhallisey | that works | 19:53 |
| rhallisey | no | 19:53 |
| rhallisey | no it doesn't.. | 19:53 |
| trown | thats cold | 19:53 |
| trown | I was excited for a few seconds | 19:53 |
| rhallisey | me too | 19:54 |
| Slower | haha | 19:56 |
| *** penick has quit IRC | 19:57 | |
| *** tiswanso has left #tripleo | 19:58 | |
| *** tiswanso has joined #tripleo | 19:58 | |
| tiswanso | Hi all, I am looking to try and make use of the openstack-puppet $::os_service_default param capability from a tripleo heat template | 20:00 |
| *** penick has joined #tripleo | 20:01 | |
| tiswanso | does anyone have a pattern I could follow? ie. having a hiera datafile only write a setting if it's not defaulted | 20:01 |
| rhallisey | trown, trying to clean it by hand then run the overcloud with a different name | 20:02 |
| tiswanso | the end result is I don't want tripleo deploy to result in a .conf .ini file setting if it's not explicitly set by the user template params | 20:02 |
| jdob | anyone have a second for some quick questions on tripleo.sh ? | 20:04 |
| dmsimard | trown: usually the osc commands leverage the client, it doesn't re-implement stuff | 20:05 |
| trown | dmsimard: ya, I am going to try with an older heatclient and see if it is the client that broke stack-delete | 20:06 |
| dmsimard | but that's the first time I see an implementation "in-tree" instead of in python-openstackclient proper | 20:06 |
| trown | cascading.fail-- | 20:06 |
| dmsimard | I didn't know projects were doing that | 20:06 |
| trown | dmsimard: ya, ironic has an osc plugin too | 20:07 |
| trown | python-tripleoclient is also an osc plugin... | 20:07 |
| dmsimard | man everything is moving out of tempest and osc, what is this | 20:07 |
| dmsimard | it might be better this way I guess | 20:07 |
| trown | rhallisey: I just checked using liberty heatclient, and same behavior... so probably heat itself | 20:11 |
| rhallisey | ya I think so | 20:12 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests https://review.openstack.org/244162 | 20:12 |
| rhallisey | the 'workaround' I mentioned earlier will do for now | 20:12 |
| trown | for you :P | 20:12 |
| trown | doesnt help me make the ping test work on trunk | 20:13 |
| rhallisey | ya :) | 20:13 |
| *** penick has quit IRC | 20:21 | |
| bnemec | If someone will merge this trivial backport https://review.openstack.org/#/c/279226 we can enable the YAML validation job on t-h-t. | 20:24 |
| *** jprovazn has quit IRC | 20:35 | |
| *** rcernin has quit IRC | 20:36 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 20:37 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: TEST: Run the ssl ci change against stable https://review.openstack.org/287425 | 20:42 |
| *** Marga__ has quit IRC | 20:43 | |
| *** Marga_ has joined #tripleo | 20:43 | |
| *** david-lyle has quit IRC | 20:45 | |
| *** david-lyle has joined #tripleo | 20:47 | |
| *** Marga_ has quit IRC | 20:50 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 20:54 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: set -o pipefail in deploy.sh https://review.openstack.org/287433 | 20:54 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: set -o pipefail in deploy.sh https://review.openstack.org/287433 | 20:56 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 20:56 |
| *** mbound has joined #tripleo | 21:02 | |
| *** dshulyak has quit IRC | 21:06 | |
| openstackgerrit | Merged openstack/os-net-config: launchpad bug 1537330, fix _is_active_nic https://review.openstack.org/279842 | 21:08 |
| openstack | Launchpad bug 1537330 in os-net-config "os_net_config.utils._is_active_nic gives wrong result for linux bond" [Undecided,New] https://launchpad.net/bugs/1537330 | 21:08 |
| *** julim has quit IRC | 21:11 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Rename tox env to pep8 https://review.openstack.org/279226 | 21:26 |
| *** lblanchard has quit IRC | 21:28 | |
| *** paramite|afk is now known as paramite | 21:30 | |
| *** mbound has quit IRC | 21:30 | |
| *** mbound has joined #tripleo | 21:31 | |
| *** trozet has quit IRC | 21:36 | |
| *** mkovacik has quit IRC | 21:39 | |
| *** r-mibu has quit IRC | 21:46 | |
| *** trozet has joined #tripleo | 21:46 | |
| *** r-mibu has joined #tripleo | 21:47 | |
| *** ccamacho has joined #tripleo | 21:47 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Update enable-tls.yaml with new endpoints https://review.openstack.org/287465 | 21:49 |
| *** jtomasek has quit IRC | 21:50 | |
| trown | EmilienM: https://review.openstack.org/#/c/244162/ is very close to fixing tripleo trunk HA, but the keystone bootstrap is happening before the database is initialized... any ideas? | 21:50 |
| EmilienM | trown: oh yeah? | 21:51 |
| EmilienM | let me check | 21:51 |
| trown | EmilienM: If I run deploy again after it fails the first time, it works, because database is up the second time | 21:51 |
| EmilienM | that's not possible https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L976 | 21:51 |
| trown | hmm... | 21:52 |
| trown | EmilienM: yet I have this on the controller keystone log http://fpaste.org/332459/ | 21:53 |
| trown | which very much looks like keystone bootstrap | 21:53 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 21:55 |
| *** Marga_ has joined #tripleo | 21:56 | |
| *** dprince has quit IRC | 22:03 | |
| openstackgerrit | Merged openstack/instack-undercloud: Remove option of installing tuskar https://review.openstack.org/265383 | 22:04 |
| *** xinwu has joined #tripleo | 22:08 | |
| rhallisey | I need idea.. libvirtError: internal error: Unable to add port tap8fd56434-ca to OVS bridge br-int | 22:09 |
| *** jayg is now known as jayg|g0n3 | 22:09 | |
| *** paramite has quit IRC | 22:12 | |
| rhallisey | s | 22:12 |
| openstackgerrit | Richard Su proposed openstack/instack-undercloud: Option to store events in Undercloud Ceilometer https://review.openstack.org/286734 | 22:13 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: OpenContrail heat templates https://review.openstack.org/286559 | 22:18 |
| *** trown is now known as trown|outtypewww | 22:19 | |
| *** weshay has quit IRC | 22:19 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 22:29 |
| * bnemec is failing hard at rebasing ^ correctly | 22:30 | |
| *** Goneri has quit IRC | 22:32 | |
| *** rwsu has quit IRC | 22:32 | |
| *** rwsu has joined #tripleo | 22:33 | |
| *** lzachery has joined #tripleo | 22:36 | |
| slagle | did we make any changes to the testenv's to try and get net-iso working? | 22:37 |
| *** ayoung has quit IRC | 22:37 | |
| slagle | i've seen a lot of jobs failing today trying to ping the controller ip's during validation | 22:38 |
| slagle | http://logs.openstack.org/93/286793/3/check-tripleo/gate-tripleo-ci-f22-ha/495ab96/console.html | 22:38 |
| *** lzachery has quit IRC | 22:42 | |
| *** trozet has quit IRC | 22:45 | |
| EmilienM | trown|outtypewww: I'm investigating bootstrap errors too | 22:47 |
| EmilienM | slagle: in master or stable/liberty? | 22:48 |
| EmilienM | or both | 22:48 |
| pradk | i have been seeing those errors at random in ci as well | 22:50 |
| *** dustins has quit IRC | 22:52 | |
| *** mbound has quit IRC | 22:56 | |
| *** thrash is now known as thrash|g0ne | 23:01 | |
| *** saneax_AFK is now known as saneax | 23:01 | |
| *** jcoufal has quit IRC | 23:02 | |
| *** tiswanso has quit IRC | 23:12 | |
| *** tiswanso has joined #tripleo | 23:12 | |
| *** tiswanso has quit IRC | 23:12 | |
| *** absubram has quit IRC | 23:25 | |
| pradk | Aodh overcloud is passing ci now.. if i can get some reviews and +2's we can get it in https://review.openstack.org/#/c/241408/ | 23:26 |
| pradk | slagle, bnemec, ^^ | 23:27 |
| *** panda has quit IRC | 23:37 | |
| *** panda has joined #tripleo | 23:38 | |
| *** Goneri has joined #tripleo | 23:47 | |
| *** ayoung has joined #tripleo | 23:50 | |
| *** pradk has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
