Wednesday, 2016-03-02

EmilienMI don't see Puppet errors00:00
*** yamahata has joined #tripleo00:00
*** dshulyak has joined #tripleo00:02
*** dshulyak has quit IRC00:07
*** yamahata has quit IRC00:11
*** prometheanfire has joined #tripleo00:13
prometheanfireSpamapS: ping, why didn't this get +w?
pradkEmilienM, the memory issue was with non-ha case, which is working now00:19
pradkEmilienM, the ha thing fails with "deploy_stdout": "Trying to ping for local network\n"00:19
*** saneax is now known as saneax_AFK00:38
*** derekh has quit IRC00:38
ayoungslagle, sorry to bother you with a provisioning quyestion, but:  trying to create a vm for a virt install, and having trouble with the virt-install parameters for the bridge:   --network bridge=brbm,model=virtio  does not seem to work.  What am I missing?00:43
ayounglooking at the XML, I see : <virtualport type='openvswitch'> with a UUID.  But I can't see how to pass that.00:44
openstackgerritMerged openstack/instack-undercloud: Switch default keystone auth_uri to v3
*** dmacpher-afk has quit IRC01:17
*** penick has quit IRC01:43
*** tiswanso has joined #tripleo01:47
*** tiswanso has quit IRC01:48
*** tiswanso has joined #tripleo01:48
*** tiswanso has quit IRC01:48
*** tiswanso has joined #tripleo01:49
*** rhallisey has joined #tripleo01:49
*** tiswanso has quit IRC01:49
*** tiswanso has joined #tripleo01:50
*** rhallisey has quit IRC02:11
*** xinwu has quit IRC02:38
*** dmacpher has joined #tripleo02:42
*** penick has joined #tripleo02:47
*** anande has joined #tripleo02:53
*** rlandy has quit IRC02:54
*** shivrao has quit IRC02:58
*** lblanchard has quit IRC03:04
*** weshay has quit IRC03:22
*** tzumainn has quit IRC03:22
*** david-lyle has joined #tripleo03:24
*** penick has quit IRC03:27
*** stendulker has joined #tripleo03:27
*** penick has joined #tripleo03:28
*** panda has quit IRC03:37
*** panda has joined #tripleo03:38
*** dshulyak has joined #tripleo03:38
*** dshulyak has quit IRC03:43
*** yuanying has quit IRC03:44
*** masco has joined #tripleo03:50
*** anande has quit IRC03:56
*** anande has joined #tripleo03:57
*** anande has quit IRC04:01
openstackgerritOpenStack Proposal Bot proposed openstack/os-collect-config: Updated from global requirements
*** xinwu has joined #tripleo04:05
*** yuanying has joined #tripleo04:07
*** ianw has joined #tripleo04:29
*** shivrao has joined #tripleo04:32
*** bkero has joined #tripleo04:37
bkeromandre: ping. I'm trying to get tripleo-quickstart running locally, but the ansible playbook for overcloud only seems to be being used for the provisioned playbook.04:38
bkeroIs there any way I can use tripleo-quickstart to deploy an overcloud at home?04:38
*** yamahata has joined #tripleo04:41
*** cmyster has quit IRC04:44
*** saneax_AFK is now known as saneax04:46
*** tiswanso has quit IRC04:49
*** links has joined #tripleo04:51
mandrebkero: I'm using it myself like that, just set the virthost variable to your target node04:51
mandrebkero: something like "ansible-playbook playbooks/centosci/minimal.yml -e virthost="04:53
mandrebkero: I missed the part "locally" in your original question, but that should work the same way04:55
mandreI suggest you read if you haven't already04:57
bkeromandre: lemme try that real quick04:58
bkeromandre: Hmm, in the centosci minimal playbook it's referencing an 'undercloud' host that's not getting defined I think04:59
bkeroAh, I think it might be working (didn't have ssh access to undercloud in that terminal)05:00
mandrebkero, the undercloud is being defined in the libvirt/setup/undercloud play05:01
mandrethat got me confused too05:01
bkeromandre: That's a very useful blog post, thank you!05:03
*** apetrich_ has quit IRC05:11
*** apetrich_ has joined #tripleo05:26
*** dshulyak has joined #tripleo05:27
*** dshulyak has quit IRC05:31
*** pcaruana has joined #tripleo05:36
*** ayoung has quit IRC05:38
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fix spurios = in dib-python readme
openstackgerritgreghaynes proposed openstack/diskimage-builder: Reorder developer quickstart docs
*** jdob has quit IRC05:46
*** pcaruana has quit IRC05:49
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fix cloud-init-disable-resizefs README title
*** rcernin has joined #tripleo05:51
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add ability to check for mount leaks
*** jdob has joined #tripleo05:58
*** jaosorior has joined #tripleo06:06
*** jprovazn has joined #tripleo06:31
*** hjensas has quit IRC06:33
*** aufi has joined #tripleo06:39
*** leanderthal|afk is now known as leanderthal06:52
*** akuznetsov has joined #tripleo06:54
*** shardy has joined #tripleo07:00
*** saneax is now known as saneax_AFK07:02
*** akuznetsov has quit IRC07:03
*** akuznetsov has joined #tripleo07:08
*** akuznetsov has quit IRC07:10
*** dshulyak has joined #tripleo07:10
openstackgerritMerged openstack/tripleo-heat-templates: Add HostnameMap to allow granular control of hostnames
*** ccamacho has joined #tripleo07:18
*** jtomasek has joined #tripleo07:19
jaosoriorshardy: Hey man, how's it going? hey, question: With the recent addition of the overriding of the nova server that doesn't get replaced as a consecuence of the user_data changing ( ); If one now updates some cloud init resource, that won't take effect, right?07:20
shardyjaosorior: Hey, not bad thanks, yeah that's the intention07:23
shardye.g any change to user_data will only affect new servers (rebuild, replace, scale-out)07:24
shardyexisting servers should be left alone07:24
jaosoriorI see07:24
*** panda has quit IRC07:37
*** panda has joined #tripleo07:38
*** liverpooler has joined #tripleo07:44
*** mikelk has joined #tripleo07:47
*** shivrao_ has joined #tripleo07:51
*** mcornea has joined #tripleo07:51
*** shivrao has quit IRC07:52
*** shivrao_ is now known as shivrao07:52
*** dmacpher has quit IRC07:58
*** jcoufal has joined #tripleo07:58
*** pcaruana has joined #tripleo08:02
*** gfidente has joined #tripleo08:05
*** gfidente has quit IRC08:05
*** gfidente has joined #tripleo08:05
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job
*** hjensas has joined #tripleo08:09
*** rdopiera has joined #tripleo08:16
*** ohamada has joined #tripleo08:19
*** fgimenez has joined #tripleo08:23
*** NobodyCa1 has joined #tripleo08:28
*** NobodyCa1 has quit IRC08:33
*** NobodyCa1 has joined #tripleo08:34
*** akuznetsov has joined #tripleo08:35
*** devvesa has joined #tripleo08:35
hewbroccagfidente: morning! how is it looking08:37
gfidentehewbrocca, it isn't going08:37
gfidentethere is traffic but is super slow, connections time out08:37
hewbroccaso, same as yesterday still08:38
hewbroccadid anyone grab dsneddon to take a look?08:38
gfidenteyeah yesterday we had a quick chat08:38
dsneddonhewbrocca: look at what?08:38
hewbroccadsneddon: morning!08:38
gfidentedsneddon, hey :)08:38
hewbroccanetwork isolation in the TripleO CI rack08:38
hewbroccathis is like a blocking mission critical thing and we can't get it going :(08:39
dsneddonhewbrocca: Oh, the CI environment? Yeah, there was some thinking that it could be MTU, but I do kind of doubt that's the problem.08:39
hewbroccaSeems unlikely08:39
*** NobodyCa1 has quit IRC08:39
dsneddonNot that I know what the problem is. I hope the problem isn't the underlying switches, because I don't know who has visibility.08:39
hewbroccadsneddon: it's the same hardware as in the rest of the scale lab08:40
hewbroccafor whatever that's worth08:40
dsneddonhewbrocca: Hmm, I'm not aware of what's going on in the scale lab08:40
*** ifarkas has joined #tripleo08:40
hewbroccaDell 630s, Juniper (I think) switches08:40
gfidentedsneddon, though traffic should stay within the ovs bridge on the baremetal08:41
gfidentewhich doesn't have any physical nic from the baremetal plugged in it08:41
gfidenteso in theory this isn't leaving the baremetal kernel08:42
gfidenteunless ovs is doing something complex in br-int08:42
*** mbound has joined #tripleo08:42
dsneddongfidente, I guess I was thinking this was a bare metal job earlier. If I had realized it was virt I wouldn't have wasted time on the MTU08:42
gfidenteyeah it's all VMs08:43
hewbroccadsneddon: that's the design of the TripleO CI rack08:43
hewbroccathere are a bunch of long-lived VMs. Undercloud gets deployed into one of them,08:43
gfidentedsneddon, so we host multiple testenvs each on an isolated bridge where the overcloud nics are plugged08:43
dsneddongfidente, Well, I guess we'd want to make sure that the libvirt network bridges were set up correctly, but I expect that it's just using instack to do that, right?08:43
hewbroccaand then it deploys an overcloud into others08:43
gfidentedsneddon, man, the libvirt network bridge is indeed the only difference I can see compared to the usual 'development' environment08:44
gfidenteand no it's not using instack08:44
hewbroccait's a custom hack by derekh and dprince08:44
gfidenteso that isn't a regular linux bridge08:44
gfidenteit's an additional ovs bridge08:44
gfidentebut I checked it 1000times yesterday and couldn't find anything wrong with it08:44
*** akuznetsov has quit IRC08:45
dsneddongfidente, Yeah, I mean the obvious thing would be to make sure that all VLANs are allowed on the bridge, but if we are getting some traffic through that wouldn't be the issue. That would cause no traffic, not slow traffic.08:45
gfidenteyeah I *think* the reason why it's slow is that it is actually taking the wrong route08:46
gfidenteprobably in OVS because we were looking into pcap with bandini yesterday08:46
gfidenteand we noticed duplicates and retransmissions08:46
gfidentea lot of them08:46
gfidenteas if things were first get lost, and then finally getting to receiver08:48
*** onovy has quit IRC08:48
gfidentewhich we translated into 'stupidly slow'08:48
openstackgerritMerged openstack/tripleo-heat-templates: Use service tenant for ceilometer
gfidenteI think if you could give a look to the baremetal bridging that would be nice08:49
gfidentebut we need to wait for derek, I don't have credentials to login on the baremetal08:49
*** shivrao has quit IRC08:51
gfidentedsneddon, fwiw, we're also using an older version of ovs08:52
gfidentesomething like 2.3 from F2008:52
gfidentenot sure if that rings any bell to you08:52
openstackgerritMerged openstack/puppet-tripleo: Always override X-Forwarded-Proto header for Heat
jaosorioryay :D08:54
gfidentejaosorior, what was it, merge fast fix fast?08:54
jaosoriorgfidente: huh?08:55
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Always override X-Forwarded-Proto header for Heat
jaosoriorgfidente: That's actually even an upstream issue, not something we enable08:58
*** chem has joined #tripleo08:58
jaosorior* not something we particularle enable08:58
openstackgerritMerged openstack/tripleo-heat-templates: Write the compute upgrade script for tripleo major upgrade workflow
*** paramite has joined #tripleo09:01
mariosthanks gfidente09:01
*** lucas-dinner is now known as lucasagomes09:06
hewbroccadsneddon: I know it's super-late for you, but if you'll be around a bit longer and can take a look once derekh gets on, it would be super-helpful09:08
*** devvesa has quit IRC09:09
dsneddonhewbrocca: Yeah, I was going to get up super early, but if Derek is coming on soon I'd rather wait09:09
hewbroccaAnybody know if slagle got a patch together to add swap to the CI instances09:09
hewbroccafor aodh/gnocchi testing?09:09
*** derekh has joined #tripleo09:10
derekhgfidente: howdy, I jumped on the env last night to see if I could figure anything out09:11
derekhgfidente: no luck09:11
gfidentederekh, yeah09:12
gfidenteI am lost as well09:12
derekhgfidente: I did notice some weird stuff, e.g. tcpdump -i eth0 -nn -e  port 500009:12
derekhgfidente: ^^ only show the incoming traffic, not the traffic in both directions, can you see if thats the case  in a working deployment?09:13
*** ishant has joined #tripleo09:13
marioshewbrocca: might be this09:13
gfidentederekh, oh you see the masquerading rule forwarding stuff on eth0 when it shouldn't?09:13
gfidentederekh, on which host did you run that?09:13
gfidentederekh, also I think we might get dsneddon to check the bridges on the baremetal09:13
marioshewbrocca: and
gfidentederekh, can you pass him credentials to login on the hosting node?09:14
*** devvesa has joined #tripleo09:14
derekhgfidente: from the comute node [root@overcloud-novacompute-0 ~]# curl
derekhtcpdump on the controller09:15
hewbroccaderekh: we have dsneddon on hand09:15
hewbroccahe can help09:15
hewbroccamarios: that's great. I'd really support getting it merged, it will help unwedge a bunch of stuff (if it in fact works)09:16
derekhhewbrocca: cool, is he here now?09:16
derekhhewbrocca: or later09:17
hewbroccahe is indeed09:17
hewbroccaup very very late, explicitly to help you :)09:17
dsneddonderekh, Hello o/09:17
derekhdsneddon: hi, have you been on the instance to see whats happening yet?09:18
dsneddonderekh, No, you'll have to give me instructions and creds to check it out09:18
*** jistr has joined #tripleo09:19
derekhgfidente: can you do that^^ I gotta do a school run then will be back, I'll then get you on the baremetal node also09:19
hewbroccaderekh: gfidente doesn't have the creds, he says09:19
dsneddonderekh, No, he said he didn't know09:19
hewbroccaoh maybe that was only for the bare metal09:20
gfidentederekh, yeah the ssh key on the baremetal09:20
derekhhewbrocca: the env we setup yesterday, dsneddon will want to start there I'd assume so he can see whats happening09:20
dsneddonderekh, Exactly09:20
hewbroccagfidente: ^^^09:20
gfidentederekh, the baremetal not the jenkins node09:20
derekhgfidente: I know that but wouldn't he want to start with the jenkins node09:21
dsneddongfidente, Were you seeing the same behavior in bare metal? I want to see what's happening in Jenkins.09:21
gfidentedsneddon, derekh wait09:21
gfidenteit's the *same* deployment09:21
derekhgfidente: you key is back on the same place, so you can log in now09:21
gfidentebut dsneddon is interested in checking the ovs bridges on the baremetal node hosting the testenv09:22
gfidentedsneddon, jenkins is driving the deployment but it's not where the bridges are09:22
derekhgfidente: add the key for dsneddon, back in a few minutes09:22
gfidentederekh, ack!09:22
dsneddongfidente: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjzU2KBRZsBIf53iWv4Zhbr//aIxkAfii9G6nTOKcKTVYKuC9rYtKU2h7XqkGQ1x1tXV5REzGxZEyARciDvaK6ip2nIcV/aeVzFTMuoU+vR3pSEhR8UvrRNgokDHkAdp9wzZGG+l0NaJD/TZmycnuNT9zsFWqR74XWS9TZ/hHRal6/8iC4Llcf7MKTMAz5ai/wxUyEe9EErLJ/uqhvD11Qpfsw7aGIdHgGAkwFeJNCFOb26PLiiaXKC2FuN8s06LdlJxd+BYKhonBJFKo+vozEjIvoSlFR14B+tThkGdodFUIf+MOAcjXb8fJ9qj9viKoBOt2GFyr/TKeLF0V1d2dl dsneddon@dsneddon-t540p09:22
*** mkovacik has joined #tripleo09:28
shardyAnyone else seeing a REPOREFS don't match error with puppet-concat installing the undercloud?09:32
*** pblaho has joined #tripleo09:33
jaosoriorshardy: At least I didn't see that yesterday09:33
jaosoriorshardy: Nevermind, it was on monday that I did my last deployment09:34
*** akrivoka has joined #tripleo09:35
derekhgfidente: dsneddon back09:41
gfidentederekh, hey we're both logged on the BM in screen09:42
gfidentederekh, any clue what is?09:42
derekhgfidente: dsneddon so one thing I tried last night, was upgrading ovs a little on the baremetal node, bug its still F20 so it only went from 2.3.0-1 to 2.3.1-309:42
gfidentederekh, overlcoud has 2.6 and 2.7 that I can tell09:42
dsneddonderekh, OK, thanks for letting me know09:42
gfidentederekh, I'm deploying on another host to reproduce your tcpdump too, will pass credentials09:43
*** apetrich_ has quit IRC09:43
derekhgfidente: | fc18b9ff-f43e-4b2d-84ef-e7aa7178eb7f | redis_virtual_ip              | fa:16:3e:e7:ca:0b | {"subnet_id": "50bd1464-db38-467a-9dda-5dca3b4aa54b", "ip_address": ""} |09:45
derekhgfidente: ack, thanks09:45
gfidentederekh, ah missed it :)09:45
*** mgould has joined #tripleo09:57
openstackgerritJavier Peña proposed openstack/tripleo-heat-templates: Fix vncproxy_host for IPv6
hewbroccaderekh: anything interesting come to light?10:02
*** electrofelix has joined #tripleo10:02
derekhhewbrocca: dsneddon is poking around at the moment10:02
dsneddonderekh, I haven't found anything yet, just confirmed this weird behavior where ICMP works but TCP is cripplingly slow10:03
derekhdsneddon: ack10:05
gfidentedsneddon, you're pluggin an if10:06
gfidenteon the baremetal in the bridge which gets to overcloud right?10:07
derekhdsneddon: also traceroute between the controller and the compute node was reporting all kinds of hop numbers, so its dropping udp packets also10:07
gfidenteok just wanted to make sure I was following :P10:07
dsneddonConnectivity between the undercloud and overcloud is fine, it's just between overcloud nodes.10:08
derekhdsneddon: can you try that ssh a coupple of times, we've had it work between the overcloud nodes also sometimews10:09
gfidentedsneddon, that's not undercloud that's baremetal10:09
dsneddonRight, that's what I meant.10:09
*** onovy has joined #tripleo10:10
gfidentederekh, looks like it's timing out when retrying it though10:10
dsneddongfidente, The ssh seems to be working, slowly, but then it might be the target trying to do DNS lookups that's adding the delay. It's working, though, just not authenticating.10:10
*** saneax_AFK is now known as saneax10:11
gfidenteor that's probably dns10:11
gfidenteack, let's add something into hosts to see how it goes10:11
dsneddongfidente, You should set up keys between the overcloud nodes, and see if you can scp anything between them.10:11
*** apetrich has joined #tripleo10:14
dsneddongfidente, derekh: This couldn't be something like the wrong network driver in the libvirt VM definition, could it?10:15
*** xinwu has quit IRC10:16
gfidentedsneddon, it's not slow on dns though it seems10:17
gfidentethe sshd debug says it took 12secs for keys exchange10:17
gfidenteMar 02 10:02:37 overcloud-controller-0 sshd[26747]: debug1: SSH2_MSG_NEWKEYS sent [preauth]10:17
gfidenteMar 02 10:02:37 overcloud-controller-0 sshd[26747]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]10:17
gfidenteMar 02 10:02:50 overcloud-controller-0 sshd[26747]: debug1: SSH2_MSG_NEWKEYS received [preauth]10:17
gfidenteMar 02 10:02:50 overcloud-controller-0 sshd[26747]: debug1: KEX done [preauth]10:17
bandinialso a telnet to port 22 would hang most of the time10:18
derekhdsneddon: as in using virtio or something else? here is what we got
gfidentebut for some reason, at least this is consistently working10:18
dsneddonderekh, virtio is good, wanted to make sure it wasn't something else.10:18
derekhdsneddon: vlan20 on the bm node that were testing with has a mtu of 1500, should it be lower to match the other nodes?10:22
gfidentederekh, the "working" overcloud is deployed now fwiw10:24
gfidentederekh, so we can try the tcpdump10:24
derekhgfidente: thanks, will give it a go10:24
*** dtantsur|afk is now known as dtantsur10:25
derekhdsneddon: tab 9 on the screen session is also on the bm host if you need it for the tcpdump10:25
derekhgfidente: will poke around your other env now, I'll have tabs 4&5 on the controller and 6&7 on the compute node10:29
dsneddonderekh, gfidente: OK, we've confirmed that we have good connectivity between the virt-host and the overcloud controller10:30
gfidentedsneddon, so it's weird though10:31
dsneddonderekh, gfidente: Maybe. That was weird, we did an scp file transfer, and the file transferred quickly, but the setup and teardown were too long10:31
gfidentessh says 3.3 megs/sec but it took 52 secs?10:31
dtantsurmorning folks!10:31
dtantsurI have a 2 line liberty backport with 1x +2 and gate passed! please10:32
shardydtantsur: g'morning - done10:34
derekhgfidente: The same thing happens with my tcpdump on you env, it seems weird but its the same for both envs so not the problem#10:34
dtantsuranyone here knowing or caring about networks? just a sanity check: does make sense to you?10:35
dtantsurmaybe gfidente or EmilienM, you're in some networking team now ;) ^^10:35
gfidentederekh, can you try snooping the bridge?10:37
openstackgerritMerged openstack/python-tripleoclient: Replace python-hardware with smaller python-hardware-detect
*** bvandenh has joined #tripleo10:38
gfidentederekh, you do see it when snooping vlan2010:39
gfidentederekh, looks like ovs is only bridging one direction on eth010:39
derekhgfidente: ok10:40
gfidentederekh, I think you know better how to get the pcap file off of the baremetal node?10:40
gfidentedsneddon, ^^10:40
derekhgfidente: dsneddon yup, I'll grab it10:40
dsneddonderekh, Thank you10:43
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Change the default value for NetworkNexusVxlanGlobalConfig
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: install zaqarclient
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: quiet yum update
jistrmarios: trying to get the upgrade e2e as much as i can, i realized that we need the yum & zaqarclient changes on compute too, so updated the patches ^^10:48
*** leanderthal has quit IRC10:48
jistrmarios: or maybe not the yum change necessarily, because it doesn't go through Heat. But the zaqarclient is needed for os-collect-config to keep working.10:49
jistrmarios: do you think i should keep the `yum -q` option for the computes as well? ^^10:49
mariosjistr: looking10:50
*** d0ugal has joined #tripleo10:50
*** yamahata has quit IRC10:51
*** tosky has joined #tripleo10:51
mariosjistr: yeah we can add -q in the compute script if we need (but if it is the output to heat that is too big then as you say not that big a deal in that case)10:51
dsneddonderekh, I made a couple of changes on the virt host. I added a vlan20 interface which is on the overcloud internal API network for testing. We also were messing with the MTU, which is now set at 1300 on the controller's vlan20 interface (which is why tcpdump shows the 1260 mss)10:53
jistrmarios: yeah i don't think we need it in the compute actually. Would it be better if the operator saw the yum output as it's happening? I'm now a bit more inclined to restore the `yum -q` patch to affect only controllers.10:53
mariosjistr: yeah exactly (am looking/voting on the review now) this was my thought too (for the operator)10:54
jistrmarios: ack i'll restore it, thx10:54
derekhdsneddon: ok10:54
dsneddonderekh, I'm about out of ideas, except maybe trying to install OVS 2.4, if that's even possible.10:54
dsneddonderekh, We have 2.4.0 on RHEL 710:55
gfidentedsneddon, you mentioned a restart, maybe we can restart just OVS to start with?10:55
dsneddongfidente, That would have been done when derekh upgraded OVS to version 2.3.1 (or whatever) last night.10:56
derekhdsneddon: I was afraid you might say something like that ;-)10:56
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: install zaqarclient
dsneddonderekh, I have been expecting that the F20 base would bite us sooner or later. I'm surprised it hasn't bitten us harder so far.10:58
derekhdsneddon: ok, I'll see if I can build an image for it, with centos 7 will that have the version needed? /me checking10:58
dsneddonWhich reminds me10:58
* dsneddon still hasn't upgraded his laptop from F2010:59
hewbroccaSo, changing the OS on all the TripleO CI hosts seems like a very risky undertaking10:59
dsneddonhewbrocca, I hope it won't come to that.10:59
hewbroccame too :)10:59
hewbroccagetting lunch, back shortly11:00
mariosjistr: did something go wrong with gerrit (the zaqar one went to v3 but no update on the quiet yum one)11:00
jistrmarios: yea it's rejecting the yum -q change for me11:00
derekhhewbrocca: it would only be the testenvs, the overcloud wouldn't be touched, I can try it on one host and see what happens,11:01
jistrmarios: i wonder if there's some safety measure against reverting patches inadvertently, which is kicking in. Googling how to solve it11:01
gfidentederekh, I see this guy here11:01
gfidenteit's 2.4 for f21 maybe close enough11:01
gfidenteand can be cleanly installed on f2011:02
mariosjistr: shortlog look ok to me;a=shortlog;h=59430030f0147991bed882ff9d1466771a37d46d (from zaqar) ...weird11:02
derekhhewbrocca: if trying a new image is the onyl way forward then dsneddon may as well go to bed, this will take all day at best I'd imaging, there is now way the various elemenst will work forst time rount11:02
gfidentederekh, oh can't we upgrade in place on this baremetal?11:03
dsneddonderekh, I don't know for sure that a newer OVS will work, but I know that I've never seen this behavior on a system with OVS 2.4.11:03
derekhgfidente: yup, we can try that11:04
hewbroccawe think we can get to OVS 2.4 without a new image?11:05
dsneddonderekh, gfidente: OVS 2.4.0 is available starting in F21:
*** tosky has quit IRC11:05
derekhdsneddon: hewbrocca: going to try an upgrade if I can11:06
* hewbrocca PH34511:06
dsneddonderekh, Best of luck, I'll check in after I get a modicum of sleep.11:06
hewbroccadsneddon: thanks for all your help11:06
derekhdsneddon: ok, talk to you later and thanks11:06
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: install zaqarclient
*** ohamada has quit IRC11:11
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: install zaqarclient
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: quiet yum update
jistrok finally :)11:11
*** tosky has joined #tripleo11:11
jistrmarios: ^ FYI the trick was that gerrit will indeed reject older patch set submission, probably as a safety measure. So one needs to `rebase -i`, `edit` the commit with not doing anything else than `git commit --amend`, this will change the timestamps on the commit (and the commit hash) and then gerrit will not reject it.11:13
jistri even got my +2s back, heh11:13
*** olap has joined #tripleo11:14
derekhgfidente: hewbrocca 2.4.0-1.fc21 doesn't appear to have helped, although it didn't pull in any dependencies , just tried an update of the openvswitch packages11:14
mariosjistr: thx11:15
hewbroccaglorious :(11:15
hewbroccaderekh: and that's OVS 2.4?11:16
hewbroccaI think we have probably lost enough time on this11:16
hewbroccaderekh: is there *any way* we can get manual testing on these patches so that we can get them merged11:17
derekhhewbrocca: 2.4.0-1.fc2111:17
derekhhewbrocca: as in manually test and approve it, I'd say so, probably need to be brought to the other cores though but it seams reasonable11:23
*** stendulker_ has joined #tripleo11:33
*** cmyster has joined #tripleo11:33
*** stendulker has quit IRC11:36
*** panda has quit IRC11:37
*** stendulker_ has quit IRC11:38
*** panda has joined #tripleo11:38
*** mbound has quit IRC11:41
*** masco has quit IRC11:43
openstackgerritAna Krivokapic proposed openstack/tripleo-docs: Add advanced docs about images
gfidentederekh, so it's not like it's behaving the same though with 2.411:49
gfidenteas if we are getting somewhere it seems11:49
derekhgfidente: yup, things are happening slightly differently11:49
gfidentebut consistenly11:50
gfidenteI mean we repoduced it11:50
gfidentemaybe this really is something we fixed in rhel11:50
jaosoriorthis had been fixed already, right? 31mError: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: keystone-manage bootstrap --bootstrap-password MQ67mevhB4RRReeR6bnz2ujqz returned 1 instead of one of [0]\u001b11:53
*** trown|outtypewww is now known as trown11:53
*** thrash|g0ne has joined #tripleo11:54
*** thrash|g0ne has quit IRC11:54
*** thrash|g0ne has joined #tripleo11:54
*** masco has joined #tripleo12:00
*** apetrich has quit IRC12:04
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI
hewbroccagfidente, derekh So, it is worth trying to reproduce this on Centos 712:09
hewbroccaI almost don't care, since there is no possible way we can reimage the whole damn CI rack right now12:09
derekhhewbrocca: I've started looking into building a F21 image, we wouldn't need to reimage all the rack, just the test envs, the overcloud would remain untouched12:10
derekhhewbrocca: its still a chunk of work but not as risky as zapping the overcloud12:11
hewbroccaWell, before we spend the effort on that, can we somehow test that it helps?12:12
gfidentederekh, in tripleo meaning of the world, I am going to ask something very stupid12:12
gfidentewhich overcloud is it that won't be zapped?12:13
gfidentehewbrocca, ack I am moving on manual testing as well12:13
derekhgfidente: the one hosting the jekins slaves12:14
gfidenteoh the one hosting the jenkins slaves12:14
gfidentemakes sense now, thanks12:14
hewbroccagfidente: OK12:14
openstackgerritMerged openstack/tripleo-heat-templates: Support adding a swap file to overcloud nodes
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Support adding a swap file to overcloud nodes
derekhhewbrocca: we'll I was going to try it on one node first, but still have to figure out if I can build an image etc...12:15
hewbroccaOK, I will leave you guys to it, then12:15
hewbroccaI was just trying to sort a way that we could verify F21 works before putting the effort into building images etc.12:16
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Adds the upgrade of any stand-alone swift nodes before controllers
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Adds the upgrade of any stand-alone swift nodes before controllers
*** mbound has joined #tripleo12:19
shardydtantsur: Hey, just raised bug #1552209 - do you happen to know if there's a fix already?12:21
openstackbug 1552209 in tripleo "introspection outputs a keystoneclient deprecation warning" [Undecided,New]
shardy"Passing auth_token to client objects is deprecated..."12:22
dtantsurshardy, there is! lemme find a link12:22
shardynice! :)12:22
dtantsurshardy, for your reviewing pleasure :)12:22
shardyI just ran introspection on some desktop baremetal machines with the fake_pxe driver and it all works pretty well except for that :)12:23
shardydtantsur: thanks, will check it out12:23
*** mgould has quit IRC12:23
shardyI also noticed there's a bunch of connection errors on the node consoles about connecting to vendor_passthrough/lookup12:24
shardyis that a known issue also?12:24
openstackgerritMerged openstack/tripleo-heat-templates: Adds v6 capability to the deploy validation test (pings)
shardyit doesn't seem to be a fatal error, but initially I thought the ramdisk couldn't post back to the undercloud at all12:24
dtantsurshardy, I suspect it's something fixed by
* dtantsur is shardy's personal magician today :D12:25
shardynice :)12:25
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm
*** fgimenez has quit IRC12:27
slaglegfidente: network isolation is going to be broken on stable/liberty until we land won't it?12:28
*** ishant has quit IRC12:29
slaglesince got backported12:29
hewbroccaslagle: morning!12:30
gfidenteslagle, right12:30
gfidente"house of cards"12:30
gfidenteI was going to update the ipv6 entire set in a dependency order12:31
gfidentethe order in which I pulled them in12:31
*** fgimenez has joined #tripleo12:31
slaglehewbrocca: mornin12:32
openstackgerritMerged openstack/tripleo-heat-templates: Enable per-role SchedulerHints
openstackgerritMerged openstack/tripleo-heat-templates: Bind Galera on a hostname for compat with IPv6 addresses
dtantsurshardy, in return, could you please check as well? :)12:35
*** jaosorior has quit IRC12:37
*** jaosorior has joined #tripleo12:38
openstackgerritMerged openstack/tripleo-heat-templates: Nova RPC unpinning
*** jaosorior has quit IRC12:39
*** jaosorior has joined #tripleo12:40
hewbroccaSo love to see things merging12:40
*** dshulyak has quit IRC12:41
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib
openstackgerritMerged openstack/tripleo-heat-templates: Enable notifications on the overcloud
trownlol hewbrocca I tend to have the opposite feeling :p, if we dont merge anything it cant break :p12:45
trownalthough that would be all of openstack not just tripleo12:45
*** cmyster has quit IRC12:48
shardydtantsur: sure, will do12:51
*** lucasagomes is now known as lucas-hungry12:51
gfidenteshardy, resources[0]:12:57
gfidente "str_replace" parameters must be a mapping validate12:57
gfidentehow can I get some more infos on which str_replace ctually failed?12:57
gfidenteI think this is coming out of the str_split in
shardygfidente: are you using (HostnameMap), or templates that contain it, on an older heat without ?12:59
*** ohamada has joined #tripleo12:59
gfidenteshardy, yes12:59
*** links has quit IRC12:59
shardygfidente: Ok, for that hostname map patch, we need the latest heat with the associated str_replace fix12:59
shardyit's landed on master and liberty now12:59
*** dprince has joined #tripleo13:00
gfidenteshardy, ack updating from delorean13:01
derekhgfidente: you finished on that baremetal hosts? about to try a new image, so we will loose it13:01
gfidentederekh, yes go ahead13:01
*** thrash|g0ne is now known as thrash13:01
*** thrash has joined #tripleo13:01
hewbroccawoop new image13:02
gfidenteshardy, works with the updated version, thanks13:05
*** rhallisey has joined #tripleo13:06
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Upgrades: Add StackUpgradeManager
*** mgould has joined #tripleo13:10
*** mkovacik has quit IRC13:11
*** mkovacik has joined #tripleo13:12
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Add mirror server
*** mkovacik has quit IRC13:12
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Used the mirror server for the centos image and git repos
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Use instack-qcow2 from cache if available and appropriate
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Upload the instack qcow image after periodic jobs
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job
*** jayg|g0n3 is now known as jayg13:16
*** rbrady has joined #tripleo13:19
*** rbrady has quit IRC13:20
*** rbrady has joined #tripleo13:21
*** rain has joined #tripleo13:21
*** rain is now known as Guest2386213:21
*** rbrady has quit IRC13:25
*** Guest23862 is now known as leanderthal13:29
derekhF21 image booted but can't see nic's , anybody remember if we had to do anything to load specific kernel modules or something..?13:29
*** rbrady has joined #tripleo13:30
*** pradk has quit IRC13:32
*** dshulyak has joined #tripleo13:32
*** apetrich has joined #tripleo13:33
*** pleia2 has quit IRC13:33
*** paramite has quit IRC13:33
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend
openstackgerritJaume Devesa proposed openstack/tripleo-docs: Extending the image build information
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Rabbit IPv6 only support
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 Support to Isolated Networks
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Allow to enable IPv6 on Corosync
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Set /64 cidr_netmask for pcmk VIPs when IPv6
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Fix vncproxy_host for IPv6
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 versions of the Controller NIC configs
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Fixup swift device string to delimit the ipv6 address with []
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Make the Neutron subnet ipv6_{ra,address}_mode configurable
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Support the deployment of Ceph over IPv6
gfidentejprovazn, shadower so is the tip13:40
gfidenteapplies on master13:41
*** masco has quit IRC13:46
openstackLaunchpad bug 1536103 in tripleo "nova-consoleauth and nova-api fail to start for ipv6" [Medium,In progress] - Assigned to Giulio Fidente (gfidente)13:47
EmilienMI have a fix in puppet-nova13:47
EmilienMgfidente: just fyi, for your reviews, we should stop adding brackets ourselves, we have now the way to do it in our puppet modules, let's use it13:48
gfidenteEmilienM, ack that patch is part of the tree13:48
hewbroccaderekh: you get yer nics working?13:48
gfidenteEmilienM, yes I think it's time to update where necessary13:48
gfidenteEmilienM, so I am testing the TIP13:48
gfidenteif it passes I think the best approach is to go where we need the updates and update the specific patch, not add more patches on top13:49
gfidentebut I want to see it passing with the 'known' patches first13:49
jprovazngfidente, so applying and then the remaining IPv6 patches on current master is the right approach, right?13:50
gfidentejprovazn, ack, just checkout the tree from 27208913:50
derekhhewbrocca: nope, trying a new image now, not too optimistic at the moment13:50
gfidenteit'll bring in all 11 deps13:50
* hewbrocca facepalm13:50
hewbroccaderekh: they *are* somewhat wacky 10Gbe nics13:51
EmilienMgfidente: I disagree actually, we can land patches that work and iterate later. One big problem I've noticed in TripleO is that patches stand for very long time, instead of iterations on multiple patches.13:51
hewbroccaIt's not out of the question that a special driver was required13:51
gfidenteEmilienM, oh you'd merge the 'suboptimal' versions if they work you mean?13:51
gfidenteEmilienM, let's see them working first13:52
EmilienMgfidente: yes13:53
shadowergfidente: you mean the whole patchset applies or just that
EmilienMthe ipv6 network isolation patch is a proof of non-iteration13:53
EmilienMwhere we continue to send patchsets13:53
gfidenteshadower, if you do checkout and not cherry-pick, it'll bring in the tree with all the 12 patches13:53
shadowerright yeah13:53
gfidentethe new ui is not nice, but on the right where it says 'related' it is actually a tree13:54
shadowerhey so is the main ipv6 patch (  blocked from merging until we fix net iso in the ci?13:54
*** rlandy has joined #tripleo13:54
derekhhewbrocca: yup13:54
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make OpenStack service ports configurable in HAProxy
*** masco has joined #tripleo13:58
*** leanderthal has quit IRC13:58
EmilienMgfidente: just to make sure i understand about
*** rleander_ has joined #tripleo13:58
*** rleander_ is now known as leanderthal13:59
EmilienMmemcache_node_ips_v6 is in fact memcache_node_ips with brackets, right?13:59
dprincederekh: which F21 image? From nodepool?14:00
dprincederekh: does nodepool even support F21 anymore (I'm not sure it does)14:01
derekhdprince: no a new one I've build for the testenvs, we had been using F20, I'm trying F21 to see if it fixes the problem we were seeing with the netiso test14:01
dprinceEmilienM: regarding landing patches and iterating. In general I agree, but the issue in TripleO is we continue to add new features without adding any CI coverage on our existing features14:02
dprinceEmilienM: so the reason why patches sit so long is we don't actually know if they break things... we have to manually test14:03
gfidenteEmilienM, not exactly it's got the inet6: prefix too14:03
*** lucas-hungry is now known as lucasagomes14:03
dprinceEmilienM: ideally we'd have CI coverage on some of the features we added like 6 months ago... and then we wouldn't be in this situation14:03
gfidenteEmilienM, so it wasn't "just" brackets in this case :(14:03
*** olap has quit IRC14:04
EmilienMgfidente: ok, so I have no patch in puppet-nova to do.14:04
dprincederekh: no kernal module was required that I'm aware of. We did set some sysctl settings to disable ARP flux14:05
dprincederekh: but that shouldn't impact you actually seeing the NICs14:06
*** olap has joined #tripleo14:06
*** masco has quit IRC14:06
dprincederekh: f20 is quite old. So I do agree trying a newer Fedora is worth a shot. This could just be some odd OVS bug that has been fixed14:07
derekhdprince: yup, i'll be adding those, ya didn't think we added anything special.14:07
*** weshay has joined #tripleo14:07
derekhdprince: yup, thats what we're thinking (hoping)14:07
dprincederekh: would F22 be a better candidate though. I at least have one local box and can confirm that it does work with regards to OVS bridge vlan trunking, etc.14:08
hewbroccaIs there any virtue in doing Centos 7?14:08
hewbroccait's closer to a real deployment target...14:08
hewbroccaalthough I guess that doesn't matter for this exercise14:08
dprincehewbrocca: I'm not sure the tripleo-incubator scripts all support Centos 7. They are older...14:08
derekhdprince: it would but I'm trying to change as little as possible14:09
hewbroccaOK, as long as we've considered it14:09
dprincederekh: sure, well a new OS is a new OS I think14:09
derekhdprince: that not exactly true, trying to get a bunch of stuff working with a new os may show p 3 problems on average14:10
derekhdprince: jumping 2 stpes is 6 problems to deal with14:10
derekhI'm making up numbers but the point I think is valid14:11
derekhhas anybody dies nova bare matel with F22 for example14:11
dprincederekh: being cautious is fine. I'm just pointing out that I can't remember if I ever testing network iso on F2114:11
derekh*dies -> tried14:11
*** dmacpher has joined #tripleo14:11
dprincederekh: but I know it works on F22...14:11
dprincederekh: in virt w/ vlans, etc.14:12
dtantsurtrown, is promotion job blocked by something now?14:12
derekhdprince: yup, if I can actually get F21 to boot, I'll try F22 aswell14:12
*** ibravo has joined #tripleo14:12
gfidenteEmilienM, maybe we can use your puppet function for the mongodb patch?14:13
trowndtantsur: promotion job is always blocked :p, but we have mitaka up to a few days old14:13
dtantsurtrown, yeah, but not the inspector packaging change fixing acl on /var/log/ironic-inspector..14:14
gfidentegoing for lunch, ttyl14:14
EmilienMgfidente: yes maybe14:14
trowndtantsur: ah got it. I am hoping for a promote today in RDO... tripleo uses there own process14:15
*** tzumainn has joined #tripleo14:17
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Upgrades: Add 'stack upgrade' command
*** bandini has quit IRC14:18
trowndtantsur: in this case I blame glance :P, they broke the swift store when using keystone v2, however they quickly fixed it too so a bit of a wash14:18
*** bandini has joined #tripleo14:18
*** lblanchard has joined #tripleo14:18
*** bandini has quit IRC14:19
*** masco has joined #tripleo14:19
*** bandini has joined #tripleo14:19
trownya who knew integrating all of openstack is kind of hard14:19
*** cmyster has joined #tripleo14:19
*** cmyster has quit IRC14:19
*** cmyster has joined #tripleo14:19
*** tiswanso has joined #tripleo14:20
dtantsurtrown, glance broke swift breaking keystone... in the house that hewbrocca built14:20
dtantsur(sorry, could not resist)14:20
bandini /win 714:20
dtantsurbandini, no such window14:20
shardyAnother review for would be good, tested locally and it fixes bug #155220914:20
openstackbug 1552209 in tripleo "introspection outputs a keystoneclient deprecation warning" [Medium,In progress] - Assigned to Dmitry Tantsur (divius)14:20
hewbroccaas long as we're injecting random stuff into the channel14:20
bandinidtantsur: :)14:21
*** mkovacik has joined #tripleo14:24
dprincegfidente, derekh: hey, an idea. Since we are in the business of trying to rebuild the testenv's14:30
dprincegfidente, derekh: back when we initially did IPv4 network iso I had thought we might use the multiple-nics templates for upstream CI14:30
dprincegfidente, derekh: so rather than trying to get VLANs to work with our existing (single nic) environments would it make sense to just add more NICs to the each virtual machine?14:31
dprincegfidente, derekh: and then we can use 'multiple-nics' instead14:31
*** tiswanso has quit IRC14:31
hewbroccadprince: that seems very sensible14:32
derekhdprince: would work for me, the version of devtest_testenv we have been using is v. old so we may hit a few snags that need to be worked out14:32
dprincegfidente: in other words lets just skip the fancy Vlans for now. We'd still get a lot of the architecture tested by just using multiple NICs14:33
*** tiswanso has joined #tripleo14:33
hewbroccaIt seems less risky/complex14:33
hewbroccabut, what do I know14:34
*** ayoung has joined #tripleo14:39
gfidentedprince, derekh, ack14:42
gfidentederekh, that means we'll have to rebuild all the testenvs though14:43
openstackgerritMerged openstack/tripleo-puppet-elements: Add overcloud-base element
dprincegfidente: FWIW, this is what I had always envisioned we'd have to do anyways. And was why I never pushed the job yet...14:44
gfidentedprince yes it's probably even closer to real world use case14:44
gfidenteEmilienM, I have some invalid parameter enable_bootstrap on Keyston14:45
gfidenteEmilienM, do I miss a patch for puppet-keystone which went in recently?14:45
*** rbrady has quit IRC14:48
EmilienMgfidente: the only one thing is
EmilienMbut HA job des not use it14:48
gfidenteoh it's exactly the param I miss though14:49
gfidentewhat do you mean HA does not use it?14:49
EmilienMrelated to LP/155150114:52
gfidenteokay so it's disabled but I need the module update, gotcha14:53
gfidenteone more nice thing about not calling classes from the modules14:53
gfidenteis that hiera automatic binding does not fail if the param is not available in the class14:53
EmilienMgfidente: no, we need to manage keystone resources with puppet14:53
EmilienMand not with os cloud config14:54
EmilienMthere is a long standing patch about it14:54
*** pradk has joined #tripleo14:55
*** sshnaidm_ has joined #tripleo14:56
*** liverpooler has quit IRC14:56
*** dustins has joined #tripleo14:56
*** trozet has joined #tripleo14:56
*** sshnaidm has quit IRC14:56
*** sshnaidm_ has quit IRC14:56
gfidenteEmilienM, but the module change is going into OPM right?14:57
*** dmsimard is now known as dmsimard|afk14:58
EmilienMit is in OPM14:58
gfidenteI had it from sources14:58
*** tosky has quit IRC15:00
*** tosky has joined #tripleo15:01
mariosjistr: we forgot about the benefits of being able to pass the rpc pin into the compute upgrade script15:03
jistrmarios: ah, right...15:03
*** sshnaidm has joined #tripleo15:03
jistrmarios: so do we just handle swift nodes different than the rest, or do we do the script delivery in a separate stack-update?15:04
mariosjistr: so is a delivery workflow worth it for that? otherwise we hard code it and update it in the tripleo-common script when needed15:04
mariosjistr: heh good question. :)15:05
jistrmarios: ok hardcoding is an option too.15:05
jistrmarios: tbh i don't like "handle swift nodes different than the rest" very much15:05
mariosjistr: so the initial 'delivery' worfklow should finish *very* quickly fwiw right (just sending the upgrade scripts)15:05
jistrmarios: inclined to select between hardcoding and a separate delivery workflow for all15:05
mariosjistr: agreed on the choice15:06
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: WIP: Enable network isolation in all CI jobs
jistrmarios: yeah i think we'd do the same thing as with upgrades, we'd disable the -post.yaml files (Puppet runs)15:06
jistrthat makes it considerably quicker15:06
mariosjistr: let's see what it looks like on paper and take it from there (I'll start from splitting the wofklow and then go to the tripleo-common script). as we said we already have the compute landed (and as we forgot for the rpm pin passing)15:07
jistrmarios: yeah. and with the separate script delivery workflow, we don't need any node type detection logic in tripleo-common bash, which is a plus too imho15:08
mariosjistr: well exactly, you know my resistance to growing that script15:08
jistrmarios: yea, +115:08
*** tosky has quit IRC15:10
openstackgerritMerged openstack/tripleo-heat-templates: Support adding a swap file to overcloud nodes
hewbroccawoo more merges!15:12
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm
*** rbrady has joined #tripleo15:13
jristguys I've been working on a new logo on my off time.15:15
mariosjrist: that is fantastic15:15
*** tosky has joined #tripleo15:15
mariosjrist: HOW ARE YOU?!15:15
jristmarios: thansk! good how are you?15:15
jristshould I propose it on the mailing list?15:16
*** jaosorior has quit IRC15:16
shadowerI like it15:16
hewbroccadprince: ^^^15:16
dprincejrist: nice, FWIW I have an actual svg file of my old clunky fatty owl too15:18
dprincejrist: your own has no feet?15:18
jristeveryone says that15:18
dprincejrist: I guess you when with ears instead15:18
jristdprince: I can add some if it is contentious15:19
dprincejrist: we want our deployment solution to be well grounded I think15:19
gfidentejrist, I think it's totally nice but but but but but but, the eyes of that owl are a little to close and in the mid-height so it looks like this is a stoned owl15:19
gfidentejrist, not that I have anything against getting stoned but15:19
gfidentejrist, it doesn't make justice to tripleo15:19
jristit's supposed to be "cute"15:19
shadower"you've got to be stoned to try tripleo"15:20
jristtheoretically the eyes can move since it's svg :)15:20
jristeaster egg?15:20
marioslol gfidente15:20
mariosgfidente: though i think it would be worse if the eyes were pointing outwards15:21
gfidenteit's just the way he is looking at me15:21
mariosi mean if they were moved it may look weird15:21
jristgfidente: any better?15:22
gfidentejrist, I think it's better yes, can they go a little bit down too?15:23
jristoh man picky picky15:23
gfidenteand thanks a lot for taking care of me15:23
gfidenteyes indeed15:23
jristanything for you gfidente15:24
jristgfidente: ^15:24
gfidenteoh this is very clever owl15:24
gfidentemarios, ^^15:24
gfidentethanks :)15:24
jristgfidente: you like it that way?15:25
gfidentehewbrocca, sorry15:25
openstackgerritMerged openstack/instack-undercloud: Deploy Aodh services, replacing Ceilometer Alarm
hewbroccawoop woop woop!15:26
mariosgfidente: yeah it works well15:27
bnemecEven better, that clears the way for the tuskar removal patch. :-)15:27
jristbnemec: the logo for the merged patch? :)15:27
*** cmyster has quit IRC15:27
*** masco has quit IRC15:28
EmilienMmerging a patch is like a celebration \o/15:28
dtantsurjrist, awesome owl! :)15:29
gfidenteEmilienM, we'll merge 12 and celebrate ^12 bugs15:29
jristdtantsur: is it good as Lucas' bear?15:29
bnemecI'm still jealous of the Heat team's fire Elmo.15:29
jristgfidente: svg updated
gfidentejrist, serious guy15:30
gfidenteit's looking at me to actually deploy something now15:30
dtantsurjrist, a bit less drive, but still cool :)15:30
jristI'll take it15:30
trownhe does look expectant of something happening15:30
jristI'm going to put an easter egg in - if you do up up down down left right b a enter it will make the eyes move independently and blink15:32
openstackgerritMerged openstack/tripleo-heat-templates: Upgrades: quiet yum update
bnemecWould somebody like to do the honors on ?15:33
bnemecWe have at least three different CI patches dependent on it now.15:33
EmilienMdprince, gfidente: I have a dual stack patch for puppet-tripleo, can you look at it when possible? it's backward compatible15:33
dprinceEmilienM: "dual stack"?15:34
EmilienMdprince: ipv6 dual stack15:34
dprinceEmilienM: oh, I see15:34
derekhhewbrocca: gfidente - dprince is going to try the multi nic thing as I'm on PTO tomorrow and probably wont get it done this evening15:34
EmilienMdprince: a very common use case for people deploying ipv615:35
gfidentewe should probably also try having 'external' on ipv415:35
EmilienMdprince: everything is in the commit message15:35
gfidentewhile keeping the others on ipv615:35
hewbroccaderekh: OK15:36
hewbroccadprince: any chance you'd have that rolling today?15:36
dprincehewbrocca: perhaps, it'll be later this evening most likely when I deploy it. And then we'd have to test the patches to try it15:37
hewbroccadprince: rock on15:37
hewbroccaanything you need from anyone to help15:37
dprincehewbrocca: rebuilding the testenv's is a bit disruptive in the middle of the day15:37
*** panda has quit IRC15:37
*** cmyster has joined #tripleo15:37
*** olap has quit IRC15:38
hewbroccadprince: makes sense15:38
*** panda has joined #tripleo15:38
shadowerbnemec: looks good but I don't think I should be the one to +2/A the patch...15:39
openstackgerritMerged openstack/tripleo-heat-templates: Upgrades: install zaqarclient
*** yamahata has joined #tripleo15:39
gfidenteshadower, go shadower we're using it already15:39
*** olap has joined #tripleo15:39
jistrgfidente: not sure if we need another CI cycle because of the commit message comment. personally i think we should land it15:40
shadowergfidente: where? Also, I haven't done a ton of reviews in literally a year so the fact I can still +2 is an overshight rather than anything eles15:40
jistr(jaosorior is offline though)15:40
dprincederekh: oh BTW. I hit this yesterday afternoon, looks like all the nightly CI jobs failed due to it as well
openstackLaunchpad bug 1552211 in tripleo "glance images fail to upload (500 errors)" [Undecided,New]15:41
derekhdprince: thanks, trown is that the error I saw ye talking about on #rdo ?15:42
gfidentejistr, it's a trap, i wanted it to land
trowndprince: derekh, yep it is fixed15:42
trownbut just a couple hours ago15:42
dprincetrown: was there a ticket? if not can you update that one with the details?15:43
trown was the fix15:43
trowndprince: ya, I will update the launchpad15:43
dprincetrown: thanks15:44
*** xinwu has joined #tripleo15:44
*** fgimenez has quit IRC15:45
trowndprince: derekh, I am actually fairly optimistic will pass15:46
derekhtrown: cool beans15:46
trownwhich would give us a hash to test tripleoci against as well15:46
*** fgimenez has joined #tripleo15:46
*** fgimenez has quit IRC15:46
*** fgimenez has joined #tripleo15:46
gfidentejistr, commented on the change to see if we can still 'avoid' it15:47
shardyI just hit which will probably mean the periodic job breaks as soon as we land
openstackLaunchpad bug 1552288 in tripleo "Re-running undercloud install fails with neutron error" [Undecided,New]15:47
EmilienMdprince: where is your patch that added swap in OOO CI?15:48
derekhdprince: I forgot to mention, when you want to test the testenvs, you'll probably want to kill the 4 python/gear processes on the testenv , then manually grab one of the json files and use it on the jenkins slave I spun up yesterday15:48
*** trown is now known as trown|brb15:48
EmilienMdprince: nevermind,
dprincederekh: no need, I'm gonna ninja this in so it works right off the bat ;)15:49
dprincederekh: good to know, I'll resort to that when I break it15:50
derekhdprince: happy days15:50
*** jistr has quit IRC15:51
derekhdprince: on jenkins slave tab 3, jimmy rig ./ to use your json file and then run "ZUUL_REF=None ZUUL_UUID=  ZUUL_CHANGES=tripleo-ci WORKSPACE=/home/jenkins/workspace/job DEVSTACK_GATE_TIMEOUT=500 NODE_NAME=derekh-test.novalocal  ./"15:52
*** trown|brb is now known as trown15:53
*** hjensas has quit IRC15:54
*** dmacpher is now known as dmacpher-afk15:55
*** paramite has joined #tripleo16:00
*** rhallisey has quit IRC16:01
*** apetrich has quit IRC16:01
openstackgerritMerged openstack/tripleo-heat-templates: Allow container template to recognize an update
*** trown is now known as trown|meeting16:01
*** rhallisey has joined #tripleo16:02
*** bvandenh has quit IRC16:04
*** rhallisey has quit IRC16:04
*** tiswanso has quit IRC16:04
*** pcaruana has quit IRC16:05
*** rhallisey has joined #tripleo16:05
*** tiswanso has joined #tripleo16:09
*** mbound has quit IRC16:15
*** xinwu has quit IRC16:15
*** ohamada has quit IRC16:15
dtantsursorry folks, but I'll keep on trying: please review please please. only 40 LoC of changes, will help me with getting rid of this "profile matching" topic16:15
*** mbound has joined #tripleo16:15
*** ohamada has joined #tripleo16:15
*** athomas has quit IRC16:16
*** athomas has joined #tripleo16:16
Erming_Hi, after deployed overcloud, I ran Tempest and got the following error. what does this mean?16:16
Erming_    tempest_lib.exceptions.IdentityError: Got identity error16:16
shardydtantsur: I was actually testing that when I discovered re-running undercloud install is broken :(16:16
shardythe patch looks fine tho16:16
shardydtantsur: one question, what happens if inspection_extras=true and the image doesn't contain python-hardware?16:18
shardydoes introspection fail, or just return an error in the extra blob?16:18
dtantsurshardy, fail16:19
shardyack, thanks16:19
*** bvandenh has joined #tripleo16:25
*** lzachery has joined #tripleo16:31
*** electrofelix has quit IRC16:34
*** tiswanso has quit IRC16:34
*** tiswanso has joined #tripleo16:34
*** dshulyak has quit IRC16:36
*** david-lyle has quit IRC16:36
*** Marga__ has joined #tripleo16:36
*** david-lyle has joined #tripleo16:38
*** yamahata has quit IRC16:39
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib
*** rlandy has quit IRC16:51
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Moves the swift start/stop into the file
*** lzachery has quit IRC17:00
*** penick has quit IRC17:01
*** trown|meeting is now known as trown17:01
trownhas anyone noticed the ping test hanging indefinitely after success, before cleanup?17:01
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades
*** rlandy has joined #tripleo17:04
*** leanderthal is now known as leanderthal|afk17:04
*** rhallisey has quit IRC17:05
*** rhallisey has joined #tripleo17:06
*** xinwu has joined #tripleo17:06
*** olap has quit IRC17:06
*** trown is now known as trown|lunch17:08
*** yamahata has joined #tripleo17:09
*** electrofelix has joined #tripleo17:10
*** rdopiera has quit IRC17:10
*** fgimenez has quit IRC17:11
*** pleia2 has joined #tripleo17:12
*** aufi has quit IRC17:12
*** ohamada has quit IRC17:13
*** ccamacho has quit IRC17:13
*** dshulyak has joined #tripleo17:15
*** dtantsur is now known as dtantsur|afk17:17
*** devvesa has quit IRC17:17
*** dshulyak has quit IRC17:20
*** masco has joined #tripleo17:22
*** ccamacho has joined #tripleo17:23
*** electrofelix has quit IRC17:25
gfidenteEmilienM, fwiw, I'm hitting this on centos bug 1298401 in memcached "Memcached fails to start in ipv6 environment" [High,Closed: errata] - Assigned to hguemar17:25
EmilienMgfidente: how can I help?17:25
gfidenteon centos we have memcached-1.4.15-9.el7.x86_6417:26
gfidentemaybe mburned is our guy17:26
gfidentembound, I think the version of memcached we have in centos is too old for bug 1298401 in memcached "Memcached fails to start in ipv6 environment" [High,Closed: errata] - Assigned to hguemar17:27
gfidentethat'll make ipv6 to fail on rdo17:27
*** xinwu has quit IRC17:28
shardyAnyone got advice on how best to debug baremetal boxes when ironic deploys the image but it's not bootable afterwards?17:29
hewbroccalucasagomes: ^^^17:30
lucasagomesshardy, hi there!17:30
*** apetrich has joined #tripleo17:30
shardylucasagomes: hey!17:30
lucasagomesshardy, what error you are see there?17:31
shardylucasagomes: so, I've got two desktop boxes, deploying OK via fake_pxe, but after one of them is deployed it ends up with an un-bootable image (the other, a mini-dell works fine)17:31
*** dprince has quit IRC17:31
shardyis it a case of booting with a livecd and looking at the bootloader and disk config?17:31
shardyIt looks like it's not installed the bootloader at all AFAICS17:32
lucasagomesshardy, oh right it's the tenant's requested image then17:32
lucasagomesshardy, right, yeah... Do you see any grub problems when you try to boot it up?17:32
shardylucasagomes: Yeah, the deploy ramdisk runs, then halts both boxes, but the overcloud image won't boot on the uefi one17:33
lucasagomesshardy, right... you sure the image being deployed contains the grub efi modules?17:33
lucasagomesshardy, if it was built with that grub2-deprecated element it won't contain it17:33
lucasagomesyou have to use the grub2 (without the -deprecated)17:34
shardylucasagomes: aha, I'm not sure tbh, it's our stock tripleo overcloud-full image17:34
shardycan I create an image which works for both uefi and non uefi?17:34
lucasagomesshardy, right that may be the case, i put a patch up for tripleo-ci relatively new that was changing it17:34
lucasagomesshardy, lemme grab the link17:34
lucasagomesshardy, yes17:34
shardycool, I have a mixture of old/new boxes here :)17:35
lucasagomesshardy, yeah just make sure the image contains these packages installed in it
lucasagomesthen it should work on both uefi and bios legacy machines17:36
shardylucasagomes: aha, yeah I should have that as I'm running latest master tripleoclient17:36
lucasagomesgrub2-install is smart enough to detect if it's uefi or bios and copy the right modules over17:36
shardyI'll double check what's in the image tho17:37
lucasagomesshardy, alright, so it may be something more complex17:37
lucasagomesshardy, ack, also, when you deploy the node in Ironic did you pass boot_mode:uefi capability to it?17:37
lucasagomesif you do, ironic will create the EFI partition for that UEFI machine17:37
*** dmsimard|afk is now known as dmsimard17:38
lucasagomesshardy, the downside is that you will need 2 flavors 1 for uefi and 1 for bios17:38
shardylucasagomes: ah, no I just passed boot_option:local17:38
shardyhaving two flavors is OK, I've already got a flavor for each type of machine17:39
shardyso I just add boot_mode:uefi to the capabilities?17:39
lucasagomesshardy, yeah17:39
shardylucasagomes: great, thanks, let me try that17:39
lucasagomesshardy, it's also strange that iPXE booted the machine without using the efi ROM17:39
lucasagomesbut as for local disk, I think that just passing boot_mode:uefi may solve the case17:40
*** dshulyak has joined #tripleo17:40
lucasagomesshardy, for ref
lucasagomesit says pxe_ipmitool only but it works for pxe_* (/me needs to correct the docs)17:41
shardylucasagomes: I enabled the uefi network stack with ipv4 PXE support in the bios, could that explain it?17:41
lucasagomesshardy, oh right, yeah17:41
lucasagomesshardy, cause in mitaka we added support for booting from uefi and iPXE if you need it17:41
lucasagomesshardy, (that's the patch for it)17:42
*** xinwu has joined #tripleo17:42
lucasagomesshardy, ok try with the boot_mode see if works, if not I we can debug it17:43
*** mbound has quit IRC17:44
lucasagomesthe boot_mode uefi will also make sure you have a GPT partiton table17:44
shardylucasagomes: ack, thanks for the help, rebooting to try boot_mode now :)17:44
lucasagomeswhich may be required for some UEFI systems17:44
lucasagomesshardy, cool17:44
shardyOk, this is probably something we should reference/mention in the baremetal part of the tripleo docs I guess17:44
* shardy reads ironic docs17:45
lucasagomesindeed... many of this UEFI stuff is landing this cycle17:45
lucasagomesI current have some patches on the fly allowing operators to choose the disk label they want for the disks (say GPT even if it's BIOS to support partitions > 2TB)17:45
lucasagomesloads of things to document :-(17:46
*** tosky_ has joined #tripleo17:47
*** bvandenh has quit IRC17:48
*** tosky has quit IRC17:48
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test
*** trown|lunch is now known as trown17:49
*** paramite is now known as paramite|afk17:51
*** electrofelix has joined #tripleo17:53
*** electrofelix has quit IRC17:53
trownEmilienM: could you help me parse the commit message for ... what does it mean "when Keystone resources will be managed by Puppet"? for context promote job now fails HA because of this bootstrap issue17:58
lucasagomesshardy, I'm about to call it a day, is it ok if we continue tomorrow?17:58
lucasagomes(in case it doesn't work)17:59
*** ibravo has quit IRC17:59
shardylucasagomes: sure - the local boot is now working, but the pxe boot of the deployment ramdisk isn't anymore17:59
EmilienMtrown: overcloud keystone resources (endpoints, etc) are managed by os cloud config and not puppet17:59
shardyI'm also going to drop off soon, will ping you tomorrow if that's OK, thanks for the help!17:59
*** tosky_ is now known as tosky17:59
EmilienMtrown: so puppet fails to bootstrap the db17:59
EmilienMbecause some data is already in there I presume18:00
lucasagomesshardy, odd... I wonder if it has to do with the now partitioned local disk?!18:00
*** dprince has joined #tripleo18:00
lucasagomestho the ramdisk shouldn't depend on the disk at all18:00
*** derekh has quit IRC18:00
lucasagomesshardy, ack, yeah tomorrow we sort it out18:00
lucasagomesshardy, g'night!18:00
trownEmilienM: ick... so that means we have to fix this new issue in os cloud config?18:01
trownEmilienM: it is only in the HA case18:01
EmilienMI'm sorry I have no idea and not investigated this bug18:01
*** lucasagomes is now known as lucas-dinner18:01
*** shardy has quit IRC18:06
*** athomas has quit IRC18:06
*** xinwu has quit IRC18:08
*** david-lyle has quit IRC18:12
*** david-lyle has joined #tripleo18:13
*** absubram has joined #tripleo18:13
*** gfidente has quit IRC18:14
*** dprince has quit IRC18:16
*** dprince has joined #tripleo18:16
*** mikelk has quit IRC18:17
*** mcornea has quit IRC18:20
*** penick has joined #tripleo18:21
bnemecShameless plug: needs a +2A and should unblock the work to have Puppet manage Keystone, which might help with the bootstrap issue.18:27
*** shivrao has joined #tripleo18:28
*** ccamacho has quit IRC18:30
*** saneax is now known as saneax_AFK18:30
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm
*** xinwu has joined #tripleo18:46
*** masco has quit IRC18:47
*** jcoufal has quit IRC18:50
*** tosky has quit IRC18:55
*** jcoufal has joined #tripleo18:56
slaglebnemec: the deprecation warning there is premature though isn't it?19:02
slaglebnemec: if we land that...we're saying the os-cloud-config method is deprecated19:02
slaglewith no guarantee the puppet bit ever gets landed19:02
*** mgould has quit IRC19:03
bnemecslagle: True, although we've actually been logging that deprecation warning for months now. :-/19:05
slagleoh, hah19:05
slaglenm me then19:05
slaglethere ya go, +A19:06
bnemecWell, you're right.  Actually logging the deprecation should have been a follow-up change with a depends-on of the puppet keystone change.19:06
slaglebnemec: i'll trade you, an easy backport:
bnemecslagle: Nice, thanks19:08
trownslagle: bnemec, is there a patch for the puppet managing keystone?19:15
trownwhoa double revert19:15
openstackgerritMerged openstack/instack-undercloud: Set max_resources_per_stack to -1
bnemecYeah, it merged, we found out it broke upgrades so we reverted it, but we still want to do it -> another revert revert.19:16
trownbnemec: did anything change wrt upgrades? or just needs more testing?19:17
slaglethere were several bugs fixed on the puppet side19:17
openstackgerritMerged openstack/python-tripleoclient: Check Keystone init before running it
trownbnemec: mind if I fix up the merge conflict? I am interested if that allows us to get trunk working19:18
openstackgerritMerged openstack/puppet-tripleo: Always override X-Forwarded-Proto header for Heat
bnemectrown: No, go for it.  The only reason I'm the owner is that I submitted the revert, so I also submitted a followup to undo it.19:19
bnemectrown: Don't miss Juan's comment on the previous patch set either.  I think there was a problem with the last rebase of the patch.19:19
trownbnemec: cool, will check that19:19
bnemecErr, maybe it was a couple of rebases ago.  Hard to keep track. :-)19:20
trownya I see it19:20
slaglebefore we merge it, we should let jistr test it in an upgrade attempt19:20
*** rwsu has quit IRC19:20
*** rwsu has joined #tripleo19:21
openstackgerritOpenStack Proposal Bot proposed openstack/os-cloud-config: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements
openstackgerritJohn Trowbridge proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests
trownbnemec: your rpm builder will build with the depends-on?19:28
bnemectrown: No, but the client patch has merged so you should just need to pull down the latest one.19:29
trownbnemec: cool I will give it a go19:29
trownI get why heat added it, but the confirmation on whether I want to delete my overcloud is annoying19:31
*** rwsu has quit IRC19:32
dprincebnemec: could you merge this
trownhmm... heat stack-delete is misbehaving quite a bit for me today... I wonder if the two things are related19:32
bnemecdprince: Done19:34
* bnemec needs food19:34
openstackgerritJohn Trowbridge proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests
dprincebnemec: thanks19:37
*** panda has quit IRC19:37
*** rwsu has joined #tripleo19:38
*** panda has joined #tripleo19:38
openstackgerritMerged openstack/instack-undercloud: Wire enable_zaqar into hiera template
rhalliseyhas anyone deployed network isolation lately been unable to delete their stack after?19:42
ayoungafter --overcloud-deploy I don't see the overcloudrc file anywhere...what happened to it?19:42
trownrhallisey: I have been having issues today deleting stacks w/o network isolation19:44
*** xinwu has quit IRC19:44
rhalliseybasically heat-engine is on a loop where it says it's destroying something then sleeps19:45
trownrhallisey: yep, the ping test actually hangs on trunk when trying to delete the stack on cleanup19:45
trownrhallisey: are you using heat/heatclient from trunk? or the tripleoci repo setup?19:46
rhalliseyya I'm using trunk19:46
trowncool, so we are likely hitting the same issue19:46
rhalliseystrength in numbers19:46
rhalliseytrown, I've tried destroying it manually19:47
rhalliseydeleting neutron ports nova instances...19:47
rhalliseystill didn't work19:47
trownya same19:47
trownheat had a bit of a merge frezy yesterday too19:48
trownnothing jumps out as being related to stack-delete though19:49
trownalthough this heatclient patch looks suspiciously related
*** akrivoka has quit IRC19:51
trownoh... but that is for `openstack stack delete`19:51
trownrhallisey: does `openstack stack delete` behave differently? I just started a redeploy of my env to test something else19:52
rhalliseythat works19:53
rhalliseyno it doesn't..19:53
trownthats cold19:53
trownI was excited for a few seconds19:53
rhalliseyme too19:54
*** penick has quit IRC19:57
*** tiswanso has left #tripleo19:58
*** tiswanso has joined #tripleo19:58
tiswansoHi all, I am looking to try and make use of the openstack-puppet $::os_service_default param capability from a tripleo heat template20:00
*** penick has joined #tripleo20:01
tiswansodoes anyone have a pattern I could follow?  ie. having a hiera datafile only write a setting if it's not defaulted20:01
rhalliseytrown, trying to clean it by hand then run the overcloud with a different name20:02
tiswansothe end result is I don't want tripleo deploy to result in a .conf .ini file setting if it's not explicitly set by the user template params20:02
jdobanyone have a second for some quick questions on ?20:04
dmsimardtrown: usually the osc commands leverage the client, it doesn't re-implement stuff20:05
trowndmsimard: ya, I am going to try with an older heatclient and see if it is the client that broke stack-delete20:06
dmsimardbut that's the first time I see an implementation "in-tree" instead of in python-openstackclient proper20:06
dmsimardI didn't know projects were doing that20:06
trowndmsimard: ya, ironic has an osc plugin too20:07
trownpython-tripleoclient is also an osc plugin...20:07
dmsimardman everything is moving out of tempest and osc, what is this20:07
dmsimardit might be better this way I guess20:07
trownrhallisey: I just checked using liberty heatclient, and same behavior... so probably heat itself20:11
rhalliseyya I think so20:12
openstackgerritJohn Trowbridge proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests
rhalliseythe 'workaround' I mentioned earlier will do for now20:12
trownfor you :P20:12
trowndoesnt help me make the ping test work on trunk20:13
rhalliseyya :)20:13
*** penick has quit IRC20:21
bnemecIf someone will merge this trivial backport we can enable the YAML validation job on t-h-t.20:24
*** jprovazn has quit IRC20:35
*** rcernin has quit IRC20:36
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL
openstackgerritBen Nemec proposed openstack/instack-undercloud: TEST: Run the ssl ci change against stable
*** Marga__ has quit IRC20:43
*** Marga_ has joined #tripleo20:43
*** david-lyle has quit IRC20:45
*** david-lyle has joined #tripleo20:47
*** Marga_ has quit IRC20:50
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: set -o pipefail in
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: set -o pipefail in
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test
*** mbound has joined #tripleo21:02
*** dshulyak has quit IRC21:06
openstackgerritMerged openstack/os-net-config: launchpad bug 1537330, fix _is_active_nic
openstackLaunchpad bug 1537330 in os-net-config "os_net_config.utils._is_active_nic gives wrong result for linux bond" [Undecided,New]
*** julim has quit IRC21:11
openstackgerritMerged openstack/tripleo-heat-templates: Rename tox env to pep8
*** lblanchard has quit IRC21:28
*** paramite|afk is now known as paramite21:30
*** mbound has quit IRC21:30
*** mbound has joined #tripleo21:31
*** trozet has quit IRC21:36
*** mkovacik has quit IRC21:39
*** r-mibu has quit IRC21:46
*** trozet has joined #tripleo21:46
*** r-mibu has joined #tripleo21:47
*** ccamacho has joined #tripleo21:47
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Update enable-tls.yaml with new endpoints
*** jtomasek has quit IRC21:50
trownEmilienM: is very close to fixing tripleo trunk HA, but the keystone bootstrap is happening before the database is initialized... any ideas?21:50
EmilienMtrown: oh yeah?21:51
EmilienMlet me check21:51
trownEmilienM: If I run deploy again after it fails the first time, it works, because database is up the second time21:51
EmilienMthat's not possible
trownEmilienM: yet I have this on the controller keystone log
trownwhich very much looks like keystone bootstrap21:53
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL
*** Marga_ has joined #tripleo21:56
*** dprince has quit IRC22:03
openstackgerritMerged openstack/instack-undercloud: Remove option of installing tuskar
*** xinwu has joined #tripleo22:08
rhalliseyI need idea.. libvirtError: internal error: Unable to add port tap8fd56434-ca to OVS bridge br-int22:09
*** jayg is now known as jayg|g0n322:09
*** paramite has quit IRC22:12
openstackgerritRichard Su proposed openstack/instack-undercloud: Option to store events in Undercloud Ceilometer
openstackgerritMerged openstack/tripleo-heat-templates: OpenContrail heat templates
*** trown is now known as trown|outtypewww22:19
*** weshay has quit IRC22:19
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL
* bnemec is failing hard at rebasing ^ correctly22:30
*** Goneri has quit IRC22:32
*** rwsu has quit IRC22:32
*** rwsu has joined #tripleo22:33
*** lzachery has joined #tripleo22:36
slagledid we make any changes to the testenv's to try and get net-iso working?22:37
*** ayoung has quit IRC22:37
slaglei've seen a lot of jobs failing today trying to ping the controller ip's during validation22:38
*** lzachery has quit IRC22:42
*** trozet has quit IRC22:45
EmilienMtrown|outtypewww: I'm investigating bootstrap errors too22:47
EmilienMslagle: in master or stable/liberty?22:48
EmilienMor both22:48
pradki have been seeing those errors at random in ci as well22:50
*** dustins has quit IRC22:52
*** mbound has quit IRC22:56
*** thrash is now known as thrash|g0ne23:01
*** saneax_AFK is now known as saneax23:01
*** jcoufal has quit IRC23:02
*** tiswanso has quit IRC23:12
*** tiswanso has joined #tripleo23:12
*** tiswanso has quit IRC23:12
*** absubram has quit IRC23:25
pradkAodh overcloud is passing ci now.. if i can get some reviews and +2's we can get it in
pradkslagle, bnemec,  ^^23:27
*** panda has quit IRC23:37
*** panda has joined #tripleo23:38
*** Goneri has joined #tripleo23:47
*** ayoung has joined #tripleo23:50
*** pradk has quit IRC23:53

Generated by 2.14.0 by Marius Gedminas - find it at!