Tuesday, 2015-06-16

« Monday, 2015-06-15 Index Wednesday, 2015-06-17 »
*** david-lyle has quit IRC00:24
*** dsneddon has quit IRC00:51
*** Slower has quit IRC01:00
*** Marga_ has quit IRC01:00
*** Marga_ has joined #tripleo01:01
*** saurabhs has left #tripleo01:04
*** Slower has joined #tripleo01:07
*** sthillma has quit IRC01:16
*** cwolferh has joined #tripleo01:31
*** Marga_ has quit IRC01:50
*** david-lyle has joined #tripleo02:08
*** cwolferh has quit IRC02:11
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Include Redis VIP in example environment  https://review.openstack.org/19203602:13
*** spzala has quit IRC02:22
*** yog__ has joined #tripleo02:50
*** Marga_ has joined #tripleo02:51
*** Marga_ has quit IRC02:52
*** Marga_ has joined #tripleo02:53
*** penick has joined #tripleo02:53
*** untriaged-bot has joined #tripleo03:01
untriaged-botUntriaged bugs so far:03:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146161503:01
openstackLaunchpad bug 1461615 in diskimage-builder "Yum vars directory doesn't exist" [Undecided,In progress] - Assigned to Abel Lopez (al592b)03:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146160103:01
openstackLaunchpad bug 1461601 in diskimage-builder "CentOS 6 not resizing" [Undecided,Fix committed] - Assigned to Abel Lopez (al592b)03:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145946503:01
openstackLaunchpad bug 1459465 in tripleo "devtest fails to build ramdisk on Fedora or CentOS due to Yum multilib error" [Undecided,Triaged]03:01
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/146340903:01
openstackLaunchpad bug 1463409 in os-collect-config "Do not tolerate world readable os-collect-config.conf" [Undecided,New]03:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146335703:01
openstackLaunchpad bug 1463357 in diskimage-builder "missing symlink support for ppc64 Fedora deployment ramdisk " [Undecided,In progress] - Assigned to yogananth subramanian (anantyog-o)03:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146163103:01
openstackLaunchpad bug 1461631 in diskimage-builder "New Fedora 22 cloud images fail to build" [Undecided,New]03:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/146198403:01
openstackLaunchpad bug 1461984 in tripleo "VM element: /usr/sbin/grub2-install: error: will not proceed with blocklists." [Undecided,New]03:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/146469203:01
openstackLaunchpad bug 1464692 in tripleo "debian build broken" [Undecided,New]03:01
*** untriaged-bot has quit IRC03:01
*** yuanying_ has joined #tripleo03:18
*** yuanying has quit IRC03:20
*** marios has quit IRC03:32
*** sdake has quit IRC03:32
*** marios has joined #tripleo03:32
*** penick has quit IRC03:37
*** yuanying_ has quit IRC03:47
*** Marga_ has quit IRC03:48
*** yuanying has joined #tripleo03:52
*** radek_ has joined #tripleo04:14
*** radek__ has quit IRC04:15
*** sdake has joined #tripleo04:17
*** panda has quit IRC04:47
*** panda has joined #tripleo04:48
*** ukalifon1 has joined #tripleo04:53
*** lazy_prince has joined #tripleo05:24
*** yamahata has joined #tripleo05:28
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates: Ensure present/latest for puppet driven package updates  https://review.openstack.org/19091805:30
*** ishant has joined #tripleo05:31
*** gfidente has joined #tripleo05:36
*** gfidente has quit IRC05:36
*** gfidente has joined #tripleo05:36
*** jtomasek has joined #tripleo05:57
*** shadower has quit IRC06:06
*** shadower has joined #tripleo06:06
*** sdake has quit IRC06:14
*** masco has joined #tripleo06:21
*** dasm|afk is now known as dasm06:21
*** pino|work has joined #tripleo06:30
*** dtantsur|afk is now known as dtantsur06:32
shardymorning all!06:39
*** jprovazn has joined #tripleo06:49
*** masco has quit IRC07:00
*** ukalifon1 has quit IRC07:02
*** greghaynes has quit IRC07:08
shardyanyone know why https://review.openstack.org/#/c/191726/ hasn't triggered CI?07:09
*** greghaynes has joined #tripleo07:10
*** paramite has joined #tripleo07:10
*** shardy_ has joined #tripleo07:15
*** ukalifon has joined #tripleo07:16
*** shardy has quit IRC07:16
*** ifarkas has joined #tripleo07:17
*** chlong has quit IRC07:18
*** ifarkas has quit IRC07:21
*** shardy_ has quit IRC07:21
*** ifarkas has joined #tripleo07:21
*** shardy has joined #tripleo07:21
*** adrianopetrich has joined #tripleo07:29
openstackgerritMerged openstack/tripleo-heat-templates: Isolate network traffic for Nova VNC proxy  https://review.openstack.org/19040907:38
*** regebro has quit IRC07:42
*** regebro has joined #tripleo07:43
openstackgerritMerged openstack/tripleo-heat-templates: Remove external bridge from Compute nodes  https://review.openstack.org/19084907:47
openstackgerritMerged openstack/tripleo-heat-templates: Add support for isolating swift storage nets  https://review.openstack.org/18865507:48
openstackgerritMerged openstack/tripleo-heat-templates: Flexible IP allocations for isolated nets  https://review.openstack.org/19116907:49
openstackgerritMerged openstack/tripleo-heat-templates: network/config storage role fixes  https://review.openstack.org/18823407:50
openstackgerritMerged openstack/tripleo-heat-templates: Add BondInterfaceOvsOptions parameter  https://review.openstack.org/18982707:50
jprovaznHi, anyone in review mood? If so, could you please take a look at https://review.openstack.org/#/c/178150/ and https://review.openstack.org/#/c/179862/07:51
*** shardy_ has joined #tripleo07:52
*** shardy has quit IRC07:53
*** al has quit IRC07:56
*** al has joined #tripleo07:57
*** shardy_ has quit IRC07:57
*** shardy has joined #tripleo07:58
*** shardy has quit IRC07:58
*** jcoufal has joined #tripleo08:00
*** shardy has joined #tripleo08:01
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Allow control of hostname formatting  https://review.openstack.org/19172608:02
gfidenteshardy, ^^ rebased08:02
shardygfidente: aha, thanks, I was just about to do that! :)08:05
gfidenteah okay I won't if you're keeping eyes on it08:05
gfidentehopefully we don't need to do it anymore though08:06
*** Goneri has joined #tripleo08:07
gfidenteshardy, actually I forgot to remove the tab as well ... so if you want to update it now that has no votes ...08:07
shardygfidente: the most recent revision didn't have that line in it at all?08:10
shardyI'll rebase from my local branch08:10
*** derekh has joined #tripleo08:10
gfidenteshardy, I downloaded last revision08:11
gfidenteso take care I think the tab will still be there08:11
shardygfidente: Ok let me check, your comment was on the n-1'th revision I think08:11
gfidenteps#5 yes08:11
gfidentewell, comment was much older, but ps#5 still has the tab08:12
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Make puppet-applying *Post resources depend on hieradata  https://review.openstack.org/19114608:12
*** lucasagomes has joined #tripleo08:13
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Allow control of hostname formatting  https://review.openstack.org/19172608:15
shardygfidente: thanks, that should fix it I think08:16
shardyI'm not sure why CI hasn't been triggering on that change though?08:16
*** Marga_ has joined #tripleo08:21
*** ukalifon has quit IRC08:23
*** ukalifon has joined #tripleo08:24
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch to using pypi  https://review.openstack.org/19210108:27
*** yamahata has quit IRC08:29
*** ukalifon has quit IRC08:37
*** ukalifon has joined #tripleo08:37
*** AzherKhan has joined #tripleo08:39
*** panda has quit IRC08:48
*** panda has joined #tripleo08:48
*** yamahata has joined #tripleo08:52
*** dtantsur is now known as dtantsur|brb08:52
*** yamahata has quit IRC08:57
*** mcornea has joined #tripleo08:57
*** chlong has joined #tripleo08:59
*** untriaged-bot has joined #tripleo09:01
untriaged-botUntriaged bugs so far:09:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146161509:01
openstackLaunchpad bug 1461615 in diskimage-builder "Yum vars directory doesn't exist" [Undecided,In progress] - Assigned to Abel Lopez (al592b)09:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146160109:01
openstackLaunchpad bug 1461601 in diskimage-builder "CentOS 6 not resizing" [Undecided,Fix committed] - Assigned to Abel Lopez (al592b)09:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145946509:01
openstackLaunchpad bug 1459465 in tripleo "devtest fails to build ramdisk on Fedora or CentOS due to Yum multilib error" [Undecided,Triaged]09:01
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/146340909:01
openstackLaunchpad bug 1463409 in os-collect-config "Do not tolerate world readable os-collect-config.conf" [Undecided,New]09:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146335709:01
openstackLaunchpad bug 1463357 in diskimage-builder "missing symlink support for ppc64 Fedora deployment ramdisk " [Undecided,In progress] - Assigned to yogananth subramanian (anantyog-o)09:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146163109:01
openstackLaunchpad bug 1461631 in diskimage-builder "New Fedora 22 cloud images fail to build" [Undecided,New]09:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/146198409:01
openstackLaunchpad bug 1461984 in tripleo "VM element: /usr/sbin/grub2-install: error: will not proceed with blocklists." [Undecided,New]09:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/146469209:01
openstackLaunchpad bug 1464692 in tripleo "debian build broken" [Undecided,New]09:01
*** untriaged-bot has quit IRC09:01
*** hewbrocca has joined #tripleo09:01
hewbroccashardy: morning!09:02
hewbroccashardy: could you remind me of the location of your excellent write-up on debugging heat stack-deploy failures?09:02
shardyhewbrocca: morning, sure, sec09:02
shardyhttp://hardysteven.blogspot.co.uk/2015/04/debugging-tripleo-heat-templates.html09:02
hewbroccathanks!09:02
hewbroccajcoufal: when are we going to have an index of all this stuff, so I can stop asking shardy where they are?09:03
shardyhewbrocca: tl;dr heat resource-list -n5 overcloud | grep FAILED09:03
shardythen heat deployment-show <failing deployment ID>09:03
hewbroccaCool09:04
hewbroccathanks09:04
hewbroccamarios: dobre rano09:04
jcoufalhewbrocca: we synced with shardy yesterday, and as we discussed last week, you should have something by thursday09:04
hewbroccajcoufal: excellent09:04
shardyjcoufal: FWIW I'd like to help refactor much of this stuff into proper docs at some point, but if linking the blog posts is the quickest approach for now that's probably fine09:07
*** chlong has quit IRC09:07
jcoufalshardy: depends on time pressure09:07
shardyI need to finish the series on heat template internals, there's at least one more post required to complete the deconstruction09:07
jcoufalfor now I linked that blog post from the docs09:08
shardyjcoufal: agreed, sounds good :)09:08
jcoufalif you manage to transfer it to proper docs right away it would be better solution09:08
jcoufalbut really depends on the time conditions09:08
shardyjcoufal: Yeah, I'm happy to do that but I was a bit put off after spending time then discovering we had overlap09:08
shardyif we can get a matrix of docs gaps then assign them to folks that will help avoid future duplication :)09:09
jcoufalshardy: sure09:09
*** pelix has joined #tripleo09:13
*** chlong has joined #tripleo09:20
hewbroccayeah, that matrix thing09:20
hewbroccathat is *exactly* what we need09:21
hewbroccajcoufal: is working on it I do believe09:21
jcoufalhewbrocca: yes, we agreed yesterday that I will send it to shardy for review09:21
hewbroccasuperb09:24
shardyWow jq really makes my head hurt09:25
hewbroccastupid javascript09:25
shardyAnyone super expert at it around here fancy a challenge?09:25
* hewbrocca whistles, looks at floor09:26
* shardy needs moar coffee09:26
*** akrivoka has joined #tripleo09:27
*** paramite is now known as paramite|afk09:34
dasmhello o/09:47
hewbroccadasm: morning!09:49
*** ishant has quit IRC09:56
*** ishant has joined #tripleo09:57
*** Marga_ has quit IRC10:00
*** Marga_ has joined #tripleo10:06
*** dtantsur|brb is now known as dtantsur10:08
akrivokawe still have a lot of tuskarclient patches that are waiting to be merged https://review.openstack.org/#/q/status:open+project:openstack/python-tuskarclient,n,z10:13
*** Marga_ has quit IRC10:20
hewbroccahelp, anyone?10:21
hewbrocca^^^10:21
*** yog__ has quit IRC10:36
*** paramite|afk is now known as paramite10:37
openstackgerritTan Lin proposed openstack/diskimage-builder: Add oat-client element  https://review.openstack.org/19214010:39
openstackgerritMerged openstack/python-tuskarclient: Handle creation of plan with existing name  https://review.openstack.org/18714910:44
*** AzherKhan has quit IRC10:46
openstackgerritMerged openstack/python-tuskarclient: Updated from global requirements  https://review.openstack.org/18733310:47
*** yuanying has quit IRC10:47
openstackgerritMerged openstack/python-tuskarclient: Enable SSL-related CLI opts  https://review.openstack.org/19025110:49
openstackgerritMerged openstack/python-tuskarclient: Filter and format parameters for plan role in OSC  https://review.openstack.org/18672810:53
akrivokagfidente: thanks!11:03
gfidentethe remaining two I am less comfortable with though11:04
akrivokagfidente: due to jenkins failing or some other reason?11:05
gfidentethe contents!11:05
akrivokaregebro: ^11:05
gfidenteoh that means I don't feel I understand that the minimum required to +211:07
gfidentenot that they are anyway broken11:08
akrivokaah fair enough :)11:08
*** lucasagomes is now known as lucas-hungry11:08
regebrogfidente: Anything I can do how to help?11:09
*** rhallisey has joined #tripleo11:14
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch to using pypi  https://review.openstack.org/19210111:24
gfidentederekh, ^^ I see jobs failing on getting the fedora cloud image as well with no route to host11:25
derekhgfidente: ya thanks, dan was working on this yesterday we have a few problems still outstanding that I'm trying to work through11:27
gfidentebut you want to update pypi meantime anyway?11:27
derekhgfidente: yup, we need to install a new geard broker, pypi index and proxy and probbaly add some compute nodes11:28
derekhgfidente: I only pushed that patch back up with a mottit message change to see if gets pasted the download fedora image erorr11:29
derekhgfidente: which I think should now be ok11:29
derekhgfidente: it was a problem with the proxy server11:29
gfidentethat would be great11:29
*** dprince has joined #tripleo11:29
derekhdprince: morning, I think I now have squick working, moving onto te-broker11:31
dprincederekh: cool. Want me to add your keys there?11:31
dprincederekh: what did I do wrong to the previous squid?11:32
derekhdprince: I was just going to delete and reinstall11:32
derekhdprince not sure, I couldn't log in because I didn't have the key11:33
dprincederekh: right, and now I can't login because I don't have the key ;)11:33
derekhdprince: my running theory was that there was some complication due to it being started by a different tennant to usual11:33
dprincederekh: to the squid that is...11:33
derekhdprince: its the key on the seed host11:34
dprincederekh: what tenant are you using? I was putting these in 'admin'.11:34
derekhdprince: openstack-nodepool11:34
*** jcoufal has quit IRC11:35
dprincederekh: lets put them all in the same place then11:36
derekhdprince: anyways I may have been wrong, when I redid a new squid instance, the dns server in /etc/resolv.conf was set to 192.168..... which wasn't working, I've manually set it to 8.8.8.811:36
dprincederekh: so kill my te-broker and you fire up one then...11:36
dprincederekh: all the traffic to the squid should be on the 192.168. network though. So inter-tenant traffic shouldn't be the issue here. That would be the 10. network....11:37
dprincederekh: btw. I think the root cause of all this was  /mnt/state/var/lib/neutron/keystone-signing being exactly 1 year old or something.11:38
derekhdprince: I have a feeling that securuty groups are getting applied, getting something similar with traffic to the tebroker,11:38
derekhdprince: ahh ok11:39
derekhdprince: want to debug the tebroker thing first to see if it is security groups ?11:39
derekhdprince: or just blow it away and create a new one11:39
dprincederekh: not really. I'd rather just put them in the same tenant anyway :)11:39
derekhdprince: ok, will do it now11:40
dprincederekh: 2 more computes would give us 12? shall I create those?11:44
derekhdprince: yup, I think so, it will give us a bit of head room if we loose one11:45
dprincederekh: the nodepool tenant doesn't ahve a keypair? How are you logging in? And if so I'd like my key added too...11:48
*** paramite is now known as paramite|afk11:49
derekhdprince: ahh, is keypair tied to user ?11:49
derekhdprince: I'm using the tripleo-ci user in the openstack-nodepool teant , default key11:50
hewbroccawhat on earth are you characters up to11:50
dprincederekh: that would explain why we aren't seeing the same things.11:50
dprincehewbrocca: fixing the TripleO ci cloud. https://etherpad.openstack.org/p/tripleo-cloud-outage11:51
derekhhewbrocca: fixing stuff11:51
dprincederekh: I'm really enjoying the devtest_overcloud_derekhisawsom.sh btw. I've just made it depend on overcloud.yaml_dan btw ;)11:54
derekhdprince: just confirmed that default key is /root/.ssh/id_rsa on the seed host, I've just copied it locally11:54
derekhdprince: nice ;-)11:54
*** athomas has quit IRC11:56
hewbroccawell someone's got to do it11:56
dprincederekh: did you delete my te-broker yet?11:57
hewbroccadprince: while you're in there -- did you guys ever get multi-network environments together to test the network isolation code?11:57
derekhdprince: yup, new instance is up11:57
dprincehewbrocca: I've been testing the network isolation stuff... but not in this environment11:57
derekhdprince: manaully installing geard now, we didn't have an image built11:58
dprincederekh: use used fedora, pip installed it, and then just ran in screen...11:58
hewbroccadprince: roger that. was just curious11:58
*** morazi has joined #tripleo11:59
derekhdprince: was going to run through whats in tripleo-image-elements/elements/geard so it comes up on reboot etc..12:00
dprincederekh: even better12:00
*** paramite|afk is now known as paramite12:04
*** athomas has joined #tripleo12:05
*** lucas-hungry is now known as lucasagomes12:08
openstackgerritMerged openstack-infra/tripleo-ci: Switch to using pypi  https://review.openstack.org/19210112:11
*** trown|outttypeww is now known as trown12:15
*** jayg|g0n3 is now known as jayg12:20
derekhdprince: tebroker is now running, I don't see any other problem at the moment but will be back in about 10 minutes to look around and start on the pypi mirror12:20
dprincederekh: we've got 12 computes now12:21
*** pblaho has joined #tripleo12:36
dprincederekh: ./toci_devtest.sh: line 193: export: `https://pypi.python.org/simple/': not a valid identifier12:44
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Fix PIP_EXTRA_INDEX_URL.  https://review.openstack.org/19218012:47
derekhdprince: opps, sorry, are your fixing it or will I?12:47
dprincederekh: just pushed https://review.openstack.org/19218012:48
*** panda has quit IRC12:48
dprincederekh: might as well wait for it to complete I guess12:48
derekhdprince: ack12:48
*** panda has joined #tripleo12:48
dprincederekh: if it gets us further we may go ahead and send it even if we don't get a full pass though...12:49
*** thrash|g0ne is now known as thrash12:50
derekhdprince: ack, yup, once it builds an image, or pip installs anything12:50
dprincederekh: https://jenkins06.openstack.org/job/check-tripleo-ironic-overcloud-f21-nonha/68/console12:56
*** jehb has quit IRC13:00
*** stendulker has joined #tripleo13:00
*** yamahata has joined #tripleo13:01
derekhdprince: its looking good,  I reckon we should send it13:05
dprincederekh: agree, it is at least progress13:07
openstackgerritMerged openstack-infra/tripleo-ci: Fix PIP_EXTRA_INDEX_URL.  https://review.openstack.org/19218013:07
*** Marga_ has joined #tripleo13:17
*** dasm is now known as dasm|afk13:20
*** julim has joined #tripleo13:21
dprincederekh: it is hanging on connecting to the seed vm13:25
*** jehb has joined #tripleo13:27
derekhdprince: this could be a "regular" non cloud problem, it connected to the seed ok to crate the baremetal flavor13:28
* derekh checks the undercloud job13:28
*** lblanchard has joined #tripleo13:31
derekhdprince: looks like its not stuck anymore, was just taking the breather for 5 minutes13:32
*** sdake has joined #tripleo13:32
*** sdake has quit IRC13:32
derekhdprince: fyi, bandersnatch index is now being built but it will take a few hours to complete13:33
*** sdake has joined #tripleo13:33
*** Marga_ has quit IRC13:34
dprincederekh: cool. would it help if we took a snapshot periodically? We'd likely just have space for just one or two though...13:36
*** Goneri has quit IRC13:37
*** Marga_ has joined #tripleo13:39
*** Marga_ has quit IRC13:40
*** Marga_ has joined #tripleo13:40
derekhdprince: maybe, I'm trying to remember how big it is, iirc around 100G13:41
* derekh wonders if a volume would be more suitable for that13:42
dprincederekh: volume works too... but I'd soon like to tear all this down and rev it to a more recent codebase13:43
dprincederekh: primarily, because I'd like to get a more recent Heat template structure13:43
derekhdprince: ack, yup makes sense13:43
dprincederekh: also, I think a working 'nova rebuild' would help our use case for stuff like this13:43
dprincederekh: in that we could simply issue nova rebuilds for all the compute nodes...13:44
dprincederekh: step at a time. But more recent code gives us a few new tricks to troublshoot with I think13:44
derekhdprince: yup, it would be a  lot better then having to being up a new stack of compute nodes13:44
*** Goneri has joined #tripleo13:44
*** ishant has quit IRC13:45
*** gfidente is now known as gfidente|afk13:52
derekhdprince: https://jenkins06.openstack.org/job/check-tripleo-ironic-undercloud-precise-nonha/340/13:53
*** lazy_prince has quit IRC13:54
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Implement Advanced Firewalling support  https://review.openstack.org/19119513:59
EmilienMgfidente|afk: ^13:59
dprincederekh: yep, I think we are back up again14:01
dprincederekh: less than 24 hours almost this time. Maybe we just need to do it once a month so we don't forget things14:02
derekhdprince: yup, although hopefully the new deployment should sort it out14:03
*** Marga_ has quit IRC14:03
derekhdprince: btw did you see the arp flood this time ? or was it just a traceback on a few compute nodes14:04
dprincederekh: 4-5 compute hosts were un-pingable.14:06
dprincederekh: all the ones I checked on the DRAC were in kernel panic14:06
openstackgerritSteven Hardy proposed openstack/tripleo-image-elements: Return derived config id when signalling deployments  https://review.openstack.org/19028214:06
dprincederekh: I figure a bunch of neutron ports maybe couldn't get cleaned up because our neutron-server had the keystone auth issue14:07
shardydprince: ^^ after battling with jq for longer than I care to admit, that I think should now work for signalling the derived config ID via 99-refresh-completed14:07
derekhdprince: ok, its sounds different to the reason we redeployed the computes nodes a few months ago14:07
dprincederekh: then when I tried rebooting the servers I think we got ARP flooded, because the machines then wouldn't stay booted14:07
derekhdprince: ahh ok14:08
dprincederekh: so I think rebooting them causes problems...14:08
derekhdprince: ack, not surprised14:08
dprincederekh: at that point I was like... lets just spin up some fresh computes...14:08
derekhdprince: yup, better then spending days trying to figure it all out14:09
dprincederekh: well, I'm pretty sure the stale neutron keystone-signing dir was the root of all our problems this time14:09
derekhdprince: ok14:10
dprincederekh: because once the new computes were up I still couldn't manage ports or anything until I deleted that one14:10
dprincederekh: I deleted a bunch of ports and reset the floatingips too14:11
dprincederekh: just to keep things clean14:11
derekhdprince: ack14:12
*** regebro has quit IRC14:21
*** jprovazn has quit IRC14:22
*** directxman12 has left #tripleo14:24
*** Marga_ has joined #tripleo14:26
openstackgerritJulia Kreger proposed openstack/diskimage-builder: Change simple-init to use a PATH variable  https://review.openstack.org/19222414:28
*** lazy_prince has joined #tripleo14:30
*** Marga_ has quit IRC14:35
*** Marga_ has joined #tripleo14:35
*** stendulker has quit IRC14:50
*** stendulker has joined #tripleo14:59
*** cwolferh has joined #tripleo14:59
*** rlandy has joined #tripleo15:00
*** untriaged-bot has joined #tripleo15:01
untriaged-botUntriaged bugs so far:15:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146161515:01
openstackLaunchpad bug 1461615 in diskimage-builder "Yum vars directory doesn't exist" [Undecided,In progress] - Assigned to Abel Lopez (al592b)15:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146160115:01
openstackLaunchpad bug 1461601 in diskimage-builder "CentOS 6 not resizing" [Undecided,Fix committed] - Assigned to Abel Lopez (al592b)15:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145946515:01
openstackLaunchpad bug 1459465 in tripleo "devtest fails to build ramdisk on Fedora or CentOS due to Yum multilib error" [Undecided,Triaged]15:01
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/146340915:01
openstackLaunchpad bug 1463409 in os-collect-config "Do not tolerate world readable os-collect-config.conf" [Undecided,New]15:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146335715:01
openstackLaunchpad bug 1463357 in diskimage-builder "missing symlink support for ppc64 Fedora deployment ramdisk " [Undecided,In progress] - Assigned to yogananth subramanian (anantyog-o)15:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146163115:01
openstackLaunchpad bug 1461631 in diskimage-builder "New Fedora 22 cloud images fail to build" [Undecided,New]15:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/146198415:01
openstackLaunchpad bug 1461984 in tripleo "VM element: /usr/sbin/grub2-install: error: will not proceed with blocklists." [Undecided,New]15:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/146469215:01
openstackLaunchpad bug 1464692 in tripleo "debian build broken" [Undecided,New]15:01
*** untriaged-bot has quit IRC15:01
*** jcoufal has joined #tripleo15:04
*** Marga_ has quit IRC15:05
*** paramite has quit IRC15:05
*** Marga_ has joined #tripleo15:06
*** Marga__ has joined #tripleo15:10
*** Marga_ has quit IRC15:10
*** yamahata has quit IRC15:12
*** Marga__ has quit IRC15:19
*** gfidente|afk is now known as gfidente15:24
*** spzala has joined #tripleo15:25
*** Goneri has quit IRC15:28
*** Marga_ has joined #tripleo15:30
*** rhallisey has quit IRC15:51
*** rhallisey has joined #tripleo15:51
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Configurable vlan_id's for isolated networks  https://review.openstack.org/18983515:52
*** alop has joined #tripleo15:55
*** sthillma has joined #tripleo15:57
*** sthillma_ has joined #tripleo16:00
*** sthillma has quit IRC16:02
*** sthillma_ is now known as sthillma16:02
*** daneyon has joined #tripleo16:11
*** daneyon_ has joined #tripleo16:14
*** daneyon has quit IRC16:17
*** dasm|afk is now known as dasm16:19
*** mcornea has quit IRC16:22
*** daneyon_ has quit IRC16:25
*** daneyon has joined #tripleo16:25
*** masco has joined #tripleo16:27
*** jcoufal has quit IRC16:28
*** trown is now known as trown|lunch16:30
*** Marga_ has quit IRC16:32
*** Marga_ has joined #tripleo16:33
*** alop has quit IRC16:33
*** stendulker has quit IRC16:35
*** penick has joined #tripleo16:36
*** dtantsur is now known as dtantsur|afk16:40
*** alop has joined #tripleo16:41
*** Marga_ has quit IRC16:44
*** Marga_ has joined #tripleo16:44
*** Marga_ has quit IRC16:45
*** Marga_ has joined #tripleo16:46
*** hewbrocca has quit IRC16:47
*** panda has quit IRC16:48
*** panda has joined #tripleo16:49
*** Marga_ has quit IRC16:49
*** Marga_ has joined #tripleo16:49
*** Marga_ has quit IRC16:49
*** dsneddon has joined #tripleo16:50
*** lucasagomes is now known as lucas-dinner16:56
*** sthillma has quit IRC16:56
*** penick has quit IRC16:58
*** masco has quit IRC16:59
EmilienMdprince: I see a lot of overlap with what I'm doing on https://review.openstack.org/191195 and os-refresh-config/pre-configure.d/97-fedora-iptables17:02
EmilienMgfidente: ^17:02
EmilienMdo we have a status of the CI now? It looks broken to me17:03
dprinceEmilienM: should be up again now. We had a cloud outage yesterday and only got things back online around 10am this morning17:06
dprinceEmilienM: there is probably overlap between those two yes. The 97-fedora-iptables should go away if you are using puppet17:06
dprinceEmilienM: should probably go away that is17:06
*** derekh has quit IRC17:06
EmilienMdprince: ok I'll take care of it17:07
EmilienMdprince: can I run recheck now?17:08
dprinceEmilienM: we've had a really bad week almost for TripleO CI. Hoping things will go better now. I'm still digging out of some of the issues from last week still17:08
dprinceEmilienM: yes, recheck away17:08
EmilienMdprince: good luck :(17:08
*** Marga_ has joined #tripleo17:12
openstackgerritgreghaynes proposed openstack/diskimage-builder: Document only exectuables in phase dirs policy  https://review.openstack.org/19229217:16
*** Marga_ has quit IRC17:17
*** sdake has quit IRC17:19
*** sdake_ has joined #tripleo17:19
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/19229617:19
*** Marga_ has joined #tripleo17:20
*** Marga_ has quit IRC17:25
openstackgerritOpenStack Proposal Bot proposed openstack/os-cloud-config: Updated from global requirements  https://review.openstack.org/18848317:25
openstackgerritOpenStack Proposal Bot proposed openstack/os-refresh-config: Updated from global requirements  https://review.openstack.org/19230217:25
*** pino|work has quit IRC17:26
openstackgerritOpenStack Proposal Bot proposed openstack/tuskar-ui: Updated from global requirements  https://review.openstack.org/19232417:27
*** dsneddon has quit IRC17:28
*** dsneddon has joined #tripleo17:34
*** yamahata has joined #tripleo17:36
*** pelix has quit IRC17:38
*** athomas has quit IRC17:41
*** Marga_ has joined #tripleo17:46
*** trown|lunch is now known as trown17:52
*** daneyon_ has joined #tripleo17:57
*** marzif_ has joined #tripleo18:00
*** daneyon has quit IRC18:00
*** marzif has quit IRC18:01
*** Marga_ has quit IRC18:07
*** Marga_ has joined #tripleo18:07
*** lazy_prince has quit IRC18:07
*** sthillma has joined #tripleo18:08
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add init-scripts directory support  https://review.openstack.org/19234118:16
*** dprince has quit IRC18:20
openstackgerritgreghaynes proposed openstack/diskimage-builder: Move simple-init data files out of phase subdirs  https://review.openstack.org/19234218:22
*** gfidente is now known as gfidente|afk18:24
*** Marga_ has quit IRC18:25
*** Marga_ has joined #tripleo18:26
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Fix allocation pool in external.yaml  https://review.openstack.org/19234918:28
*** rlandy has quit IRC18:32
*** rlandy has joined #tripleo18:33
*** penick has joined #tripleo18:35
*** daneyon has joined #tripleo18:42
*** daneyon_ has quit IRC18:45
*** spredzy|afk is now known as spredzy18:50
*** weshay has quit IRC18:50
*** yamahata has quit IRC19:06
*** Marga_ has quit IRC19:07
*** sdake_ is now known as sdake19:13
*** sdake has quit IRC19:15
*** sdake has joined #tripleo19:15
*** ukalifon has quit IRC19:27
*** dasm is now known as dasm|afk19:42
*** trown is now known as trown|outttypeww19:46
EmilienMI'm trying to debug https://review.openstack.org/#/c/191195/19:52
EmilienMbut I don't see failure in logs19:52
EmilienMany thoughts?19:52
slagleEmilienM: you want to look in the os-collect-config logs19:54
slaglehttp://logs.openstack.org/95/191195/3/check-tripleo/check-tripleo-ironic-overcloud-precise-nonha/d76d785/logs/overcloud-controller0_logs/os-collect-config.txt.gz19:54
slaglelooks like horizon failed to start:19:55
slagle[ALERT] 166/191044 (9350) : Starting proxy horizon: cannot bind socket19:55
EmilienMslagle: I did, but puppet is not run on ubuntu19:55
EmilienMI'm looking at fedora logs19:55
slagleok...19:56
*** jcoufal has joined #tripleo19:57
*** ifarkas has quit IRC19:58
slagleso in this one http://logs.openstack.org/95/191195/3/check-tripleo/check-tripleo-ironic-overcloud-f21puppet-ha/36574a0/console.html19:58
slaglei see:19:58
slagle2015-06-16 18:20:56.002 | {"Status" : "SUCCESS","Reason" : "Configuration Complete","UniqueId" : "seedImageID","Data" : "Finished os-refresh-config."}19:58
slagle2015-06-16 18:20:56.003 | + wait_for -w 20 --delay 1 -- ping -c 1 192.0.2.119:58
slagle2015-06-16 18:21:17.021 | Timing out after 20 seconds:19:58
slagleso it failed to ping the instance on the overcloud at the end19:58
slagleand yea, i don't see any overcloud logs19:59
*** julim has quit IRC20:00
*** Marga_ has joined #tripleo20:01
*** sthillma has quit IRC20:12
*** sdake_ has joined #tripleo20:16
*** julim has joined #tripleo20:18
*** sdake has quit IRC20:20
*** jcoufal has quit IRC20:23
*** spzala has quit IRC20:23
*** jayg is now known as jayg|g0n320:30
*** radek_ has quit IRC20:30
openstackgerritimain proposed openstack/tripleo-incubator: Add support for using containers.  https://review.openstack.org/18821820:36
*** lblanchard has quit IRC20:40
*** rlandy has quit IRC20:40
lifeless.oO20:42
*** rlandy has joined #tripleo20:46
openstackgerritimain proposed openstack/tripleo-image-elements: Add support for full images.  https://review.openstack.org/19030320:51
*** akrivoka has quit IRC20:59
*** penick has quit IRC21:00
*** untriaged-bot has joined #tripleo21:01
untriaged-botUntriaged bugs so far:21:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146161521:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146160121:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145946521:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146335721:01
openstackLaunchpad bug 1461615 in diskimage-builder "Yum vars directory doesn't exist" [Undecided,In progress] - Assigned to Abel Lopez (al592b)21:01
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/146163121:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/146198421:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/146469221:01
openstackLaunchpad bug 1461601 in diskimage-builder "CentOS 6 not resizing" [Undecided,Fix committed] - Assigned to Abel Lopez (al592b)21:01
*** untriaged-bot has quit IRC21:01
openstackLaunchpad bug 1459465 in tripleo "devtest fails to build ramdisk on Fedora or CentOS due to Yum multilib error" [Undecided,Triaged]21:01
openstackLaunchpad bug 1463357 in diskimage-builder "missing symlink support for ppc64 Fedora deployment ramdisk " [Undecided,In progress] - Assigned to yogananth subramanian (anantyog-o)21:01
openstackLaunchpad bug 1461631 in diskimage-builder "New Fedora 22 cloud images fail to build" [Undecided,Fix committed]21:01
openstackLaunchpad bug 1461984 in tripleo "VM element: /usr/sbin/grub2-install: error: will not proceed with blocklists." [Undecided,Triaged]21:01
openstackLaunchpad bug 1464692 in tripleo "debian build broken" [Undecided,Triaged]21:01
*** penick has joined #tripleo21:02
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates: Ensure present/latest for puppet driven package updates  https://review.openstack.org/19091821:09
*** Marga_ has quit IRC21:09
*** Marga_ has joined #tripleo21:10
*** cwolferh has quit IRC21:20
*** julim has quit IRC21:21
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add test for centos 6  https://review.openstack.org/19244021:26
*** Marga_ has quit IRC21:30
*** Marga_ has joined #tripleo21:30
*** Marga_ has quit IRC21:40
*** Marga_ has joined #tripleo21:40
*** gfidente|afk has quit IRC21:47
*** rlandy has quit IRC21:51
*** rlandy has joined #tripleo21:51
*** penick has quit IRC21:59
*** sdake_ is now known as sdake22:02
*** penick has joined #tripleo22:18
*** rhallisey has quit IRC22:22
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates: Allow control of hostname formatting  https://review.openstack.org/19172622:23
*** rlandy has quit IRC22:25
*** rlandy has joined #tripleo22:26
openstackgerritBen Nemec proposed openstack/tripleo-incubator: Make sure configure-vm mac doesn't start negative  https://review.openstack.org/19244922:34
*** openstackgerrit has quit IRC22:38
*** openstackgerrit has joined #tripleo22:39
openstackgerritMerged openstack/tripleo-incubator: Generate multiple baremetal MACs in sequence  https://review.openstack.org/18350322:40
*** pbourke has quit IRC22:44
*** pbourke has joined #tripleo22:44
*** Marga_ has quit IRC22:55
*** rlandy has quit IRC22:59
*** daneyon has left #tripleo23:01
*** shardy has quit IRC23:05
*** Goneri has joined #tripleo23:11
*** chlong has quit IRC23:15
*** Marga_ has joined #tripleo23:19
openstackgerritIan Wienand proposed openstack/diskimage-builder: Update default Fedora to 22  https://review.openstack.org/18584323:39
openstackgerritIan Wienand proposed openstack/diskimage-builder: Cleanup yum downloading  https://review.openstack.org/18662023:39
*** Marga_ has quit IRC23:41
*** rhefner has quit IRC23:41
*** lucas-dinner has quit IRC23:41
*** Goneri has quit IRC23:44
*** Marga_ has joined #tripleo23:53
openstackgerritimain proposed openstack/tripleo-heat-templates: Containerized TripleO Overcloud.  https://review.openstack.org/17884023:57
« Monday, 2015-06-15 Index Wednesday, 2015-06-17 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!