Wednesday, 2015-01-07

*** david-lyle is now known as david-lyle_afk00:01
*** barra204 has joined #tripleo00:05
*** dsneddon has joined #tripleo00:15
*** dsneddon has quit IRC00:15
*** dsneddon has joined #tripleo00:15
*** barra204 has quit IRC00:37
*** avozza is now known as zz_avozza01:06
*** zz_avozza is now known as avozza01:10
*** larstobi has quit IRC01:14
*** thrash is now known as thrash|g0ne01:14
*** larstobi has joined #tripleo01:16
*** nosnos has joined #tripleo01:27
*** wuhg has joined #tripleo01:40
*** mandre has joined #tripleo01:56
*** pradeep has joined #tripleo02:00
*** Yanyanhu has joined #tripleo02:00
*** dsneddon has quit IRC02:01
*** pradeep1 has quit IRC02:01
*** dsneddon has joined #tripleo02:01
*** pradeep1 has joined #tripleo02:20
*** pradeep has quit IRC02:22
*** dsneddon has quit IRC02:22
*** lucasagomes has quit IRC02:24
*** pradeep1 has quit IRC02:28
*** jerryz has joined #tripleo02:29
*** Yanyan has joined #tripleo02:46
*** Yanyanhu has quit IRC02:49
*** julim has quit IRC03:00
*** avozza is now known as zz_avozza03:02
*** mandre has quit IRC03:02
*** nosnos has quit IRC03:20
*** noslzzp has joined #tripleo03:23
*** chlong has quit IRC03:41
*** chlong has joined #tripleo03:42
*** chlong has quit IRC03:42
*** chlong has joined #tripleo03:44
*** pcrews has quit IRC03:44
*** chlong has quit IRC03:48
*** tserong_ has joined #tripleo03:48
*** tserong_ has joined #tripleo03:48
*** chlong has joined #tripleo03:48
*** tserong has quit IRC03:50
*** madhuri has joined #tripleo03:51
*** tserong_ is now known as tserong03:52
*** untriaged-bot has joined #tripleo04:00
untriaged-botUntriaged bugs so far:04:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/140551304:00
uvirtbotLaunchpad bug 1405513 in tripleo "[tripleo-heat-templates] haproxy option added for controller caused noisy debug log in keystone" [Undecided,New]04:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/140810804:00
uvirtbotLaunchpad bug 1408108 in tripleo "horizon/apache2 image element logging set to debug; useless mod_authz_core messages" [Undecided,New]04:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/140279504:00
untriaged-bothttps://bugs.launchpad.net/tuskar/+bug/138748704:00
uvirtbotLaunchpad bug 1402795 in tripleo "Keystone token_flush  cron job running at the same time in the controller node causes deadlock" [Undecided,In progress]04:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/140408504:00
uvirtbotLaunchpad bug 1387487 in tuskar "tripleo-image-elements/elements/tuskar-ui /os-refresh-config /post-configure.d/101-tuskar-ui  needs authentication from Keystone" [Undecided,New]04:00
uvirtbotLaunchpad bug 1404085 in tripleo "l3 agent failed to spawn radvd due to no filter matched" [Undecided,In progress]04:00
*** untriaged-bot has quit IRC04:00
*** nosnos has joined #tripleo04:15
*** Marga_ has quit IRC04:15
*** Marga_ has joined #tripleo04:16
*** Nakato has quit IRC04:16
*** Nakato has joined #tripleo04:16
*** Nakato has quit IRC04:18
*** Nakato has joined #tripleo04:19
*** Marga_ has quit IRC04:20
*** pradeep has joined #tripleo04:21
*** rlandy has quit IRC04:23
*** noslzzp has quit IRC04:28
*** mandre has joined #tripleo04:29
*** masco has joined #tripleo04:37
*** radez is now known as radez_g0n304:54
*** Marga_ has joined #tripleo05:07
*** Marga_ has quit IRC05:08
*** Marga_ has joined #tripleo05:08
*** tzumainn has quit IRC05:09
*** vinsh has joined #tripleo05:13
*** vinsh has quit IRC05:25
*** pensu has joined #tripleo05:33
*** daneyon has joined #tripleo05:45
*** eghobo has joined #tripleo05:56
*** zz_avozza is now known as avozza06:01
*** pradeep has quit IRC06:09
*** avozza is now known as zz_avozza06:11
*** daneyon_ has joined #tripleo06:15
*** madhuri has quit IRC06:16
*** daneyon has quit IRC06:18
*** pradeep has joined #tripleo06:29
*** Marga_ has quit IRC06:31
*** Marga_ has joined #tripleo06:31
*** eghobo has quit IRC06:33
*** ishant has joined #tripleo06:48
greghaynestchaypo: ohai - not ignoring you re: the local env vars, was a travel day for me today06:52
greghaynestchaypo: can chat about it tomorrow though?06:52
*** daneyon_ has quit IRC06:59
*** mrunge has joined #tripleo07:28
*** mandre has quit IRC07:30
*** madhuri_ has joined #tripleo07:31
madhuri_Hi. I am facing issue while trying to add new compute node to tripleo environment.07:32
madhuri_Can anyone please help?07:32
madhuri_Compute node is not able to communicate to the controller node.07:33
*** mandre has joined #tripleo07:35
*** ifarkas has joined #tripleo07:45
tchaypomorning07:48
tchaypogreghaynes:07:48
tchaypogreghaynes: sure :) sorry, I didn’t mean to imply you were ignoring me - I’m not certain I’m correct and I don’t want to be holding up the change if I’m wrong :)07:48
tchaypomadhuri_: that sounds annoying. I’m not sure if I can help, but if you can tell me more I might possibly...07:50
madhuri_My tripleo environment is setup on single machine with one controller node and two compute node07:51
madhuri_And I tried to add a new compute node on different machine to it07:51
madhuri_While trying to start the compute service on my new node, it failed. And failing at point where it is trying to call compute_node_create API of controller.07:53
madhuri_Giving connection time out error07:53
StevenKConnection timed out sounds like a firewalling issue07:53
madhuri_Logs: http://pastebin.com/9NiVbjvM07:54
madhuri_It is not so I have checked.07:54
tchaypomorning StevenK07:54
tchaypolong time no see07:54
StevenKtchaypo: That's what happens when you bugger off to Europe over Christmas07:55
tchayposince you’re here, I’ve been wanting to pick your brain about the the file init-neutron-ovs produces07:55
*** zz_avozza is now known as avozza07:55
tchaypohttps://review.openstack.org/#/c/135727/07:56
tchaypoI think the vlan change I’ve made is correct, but I had another change in up until revision 8, described in the commit message07:56
tchaypohttps://review.openstack.org/#/c/135727/8//COMMIT_MSG07:56
StevenKtchaypo: Heh, you know that I've been staring into a text editor for hours this week trying to finish my talk for the CI miniconf07:56
tchayponope. makes sense though. How soon is LCA?07:57
StevenKNext week07:57
tchaypomeeting in 2 minutes...07:57
*** subscope has joined #tripleo07:57
madhuri_tchaypo: ^^08:03
*** pcaruana|afk| is now known as pcaruana08:04
*** rdopiera has joined #tripleo08:06
*** rdopiera has quit IRC08:07
*** rdopiera has joined #tripleo08:07
*** chlong has quit IRC08:07
*** jprovazn has joined #tripleo08:09
*** kfox1111 has joined #tripleo08:12
madhuri_tchaypo: In compute_node_create RPC API, message body i.e the information of new compute node was large. I tried sending small value and it was successful.\08:14
*** yuanying has joined #tripleo08:16
madhuri_tchaypo: I checked and increased memory and even changed the arch to amd64 of controller node thinking that it might be problem of memory. But still facing the issue.08:16
tchaypomadhuri_: apologies for lack of response, in a meeting right now, but should be able to get back to you soonish08:17
madhuri_Np :)08:17
*** yuanying_ has quit IRC08:17
*** gmann__ has joined #tripleo08:19
*** rha has joined #tripleo08:19
*** gmann__ has left #tripleo08:20
tchaypookay, back08:27
tchayposo to summarise - ou know it’s not a firewall issue, because you can connect and send data through, and get a response - provided the response is small08:27
tchayposorry, provided the request data is small08:27
*** jtomasek has joined #tripleo08:27
tchaypobut the real request has a big blob’o’data and that doesn’t seem to make it through?08:27
*** avozza is now known as zz_avozza08:28
*** gmann has joined #tripleo08:28
*** gmann has left #tripleo08:29
madhuri_Yes might be.08:30
*** mandre has quit IRC08:31
tchaypoIf that’s the case, my first thoughts are that it could be a TCP fragmentation issue - I’ve seen cases where the request data doesn’t fit inside one TCP packet, and the usual process of re-assembling the packet doesn’t work. That can be because firewalls drop the second packet, or because the http server doesn’t reassemble it properly08:32
tchayposo I’d probably start by looking at the very first http server or proxy that the request is hitting and seeing if you can get it to log the full contents of the request, just to see if that’s coming through okay08:33
madhuri_How can I make it logged?08:37
*** dtantsur|afk is now known as dtantsur08:42
*** gfidente has joined #tripleo08:42
*** gfidente has joined #tripleo08:42
*** gmann has joined #tripleo08:56
*** pcaruana has quit IRC08:59
*** nosnos has quit IRC09:00
*** nosnos has joined #tripleo09:01
*** jcoufal has joined #tripleo09:07
*** nosnos has quit IRC09:08
*** nosnos has joined #tripleo09:10
tchayposdake: nice update09:10
*** pcaruana has joined #tripleo09:12
*** jistr has joined #tripleo09:14
*** larstobi has quit IRC09:17
*** athomas has joined #tripleo09:17
*** larstobi has joined #tripleo09:19
*** mrunge has quit IRC09:21
*** GheRivero has quit IRC09:23
*** GheRivero has joined #tripleo09:23
tchaypomadhuri_: depends what your server/proxy is. I’d have to go looking into the manual to figure it out09:25
tchaypoas a first step i might run tcpdump (with "-s 0” to make sure it gets the whole packet) just to confirm if the data is even coming in the network interface. If it’s not even getting that far it sounds like a firewall issue; if it’s getting in but the web server/proxy isn’t logging it then it sounds like a web server issue09:26
*** mrunge has joined #tripleo09:29
*** mrunge has quit IRC09:29
*** mrunge has joined #tripleo09:29
tchaypoDerekh isn’t around..09:35
tchayposo I won’t talk to him09:35
madhuri_tchaypo: I will check it and get back to you09:41
madhuri_Thanks for your help. I will catch you later09:41
*** madhuri_ has quit IRC09:45
*** mrunge_ has joined #tripleo09:57
*** mrunge has quit IRC09:59
*** untriaged-bot has joined #tripleo10:00
untriaged-botUntriaged bugs so far:10:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/140551310:00
untriaged-bothttps://bugs.launchpad.net/tuskar/+bug/138748710:00
uvirtbotLaunchpad bug 1405513 in tripleo "[tripleo-heat-templates] haproxy option added for controller caused noisy debug log in keystone" [Undecided,New]10:00
*** untriaged-bot has quit IRC10:00
uvirtbotLaunchpad bug 1387487 in tuskar "tripleo-image-elements/elements/tuskar-ui /os-refresh-config /post-configure.d/101-tuskar-ui  needs authentication from Keystone" [Undecided,New]10:00
*** mcornea has joined #tripleo10:00
*** zaro has quit IRC10:05
*** derekh has joined #tripleo10:05
*** zaro has joined #tripleo10:06
*** pradeep has quit IRC10:06
*** bvandenh has joined #tripleo10:11
*** Yanyan has quit IRC10:28
*** mrunge_ is now known as mrunge10:33
*** jp_at_hp has joined #tripleo10:37
*** chlong has joined #tripleo10:38
*** zz_avozza is now known as avozza10:50
*** avozza is now known as zz_avozza10:56
*** pradeep has joined #tripleo11:02
*** lucasagomes has joined #tripleo11:05
*** chlong has quit IRC11:08
*** pelix has joined #tripleo11:09
*** zz_avozza is now known as avozza11:09
*** GheRivero has quit IRC11:15
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Cleanup VMs at the end of each CI test  https://review.openstack.org/14546911:17
*** GheRivero has joined #tripleo11:21
openstackgerritMerged openstack-infra/tripleo-ci: Remove chown .cache workaround  https://review.openstack.org/14450011:23
*** GheRivero has quit IRC11:33
*** GheRivero has joined #tripleo11:34
*** andreaf has joined #tripleo11:40
*** GheRivero has quit IRC11:44
*** jcoufal_ has joined #tripleo11:45
*** jcoufal has quit IRC11:49
*** GheRivero has joined #tripleo11:50
*** david-lyle_afk is now known as david-lyle11:58
*** morazi has quit IRC12:01
*** avozza is now known as zz_avozza12:08
*** rhallisey has joined #tripleo12:24
*** pensu has quit IRC12:30
*** dprince has joined #tripleo12:33
*** athomas has quit IRC12:35
*** athomas has joined #tripleo12:38
*** lazy_prince is now known as killer_prince12:38
*** athomas has quit IRC12:38
*** pblaho_ has joined #tripleo12:38
*** athomas has joined #tripleo12:40
*** pblaho_ has quit IRC12:43
*** nosnos has quit IRC12:43
*** morazi has joined #tripleo12:49
*** GheRivero has quit IRC12:53
*** GheRivero has joined #tripleo12:53
*** pradeep has quit IRC12:56
*** weshay has joined #tripleo13:01
*** masco has quit IRC13:04
*** thrash|g0ne is now known as thrash13:07
*** david-lyle is now known as david-lyle_afk13:08
*** ishant|2 has joined #tripleo13:09
*** ishant|2 has quit IRC13:09
derekhdprince: I missed the meeting, was on a train and had no power, sent mail to the list instead13:10
dprincederekh: ah. no worries. maybe it was for the best... I'll check the list for your mail13:11
*** ishant has quit IRC13:11
*** noslzzp has joined #tripleo13:12
*** rlandy has joined #tripleo13:13
*** noslzzp has quit IRC13:15
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Cleanup VMs at the end of each CI test  https://review.openstack.org/14546913:16
*** tzumainn has joined #tripleo13:17
*** GheRivero has quit IRC13:20
*** radez_g0n3 is now known as radez13:26
*** GheRivero has joined #tripleo13:26
*** pradeep has joined #tripleo13:26
Ngmjorning13:26
dprinceNg: morning.13:28
Ngdprince: hey dan :) good winterval break?13:28
dprinceNg: yes. mostly. Got back to something I hadn't done in a while: woodworking :)13:29
dprinceNg: you?13:29
Ngdprince: nice! Mine was very relaxing for the bits where we had lots of family around (because free childcare) and crazy intense for the bits when it was just us, so in some ways I'm quite glad to be back to sitting still on my own in a silent room ;)13:30
*** pblaho has joined #tripleo13:30
*** jcoufal_ has quit IRC13:31
*** jcoufal has joined #tripleo13:31
dprinceNg: exactly. This is the time of year when work == vacation (of sorts)13:31
Nghehe, yep13:31
dprinceotoolee: hi, I'm looking at something you did with regards to swift.zones about a year back. You around?13:32
*** jdob has joined #tripleo13:32
dprinceotoolee: Specicially, the place_in_zone function here : http://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements/swift/os-refresh-config/configure.d/73-swift#n5313:33
*** kfox1111 has quit IRC13:36
dprinceotoolee: I'd like to understand how that works. The comment mentions a server number but I'm not sure we have that13:36
*** pblaho has quit IRC13:45
*** pblaho_ has joined #tripleo13:45
derekhis devtest on fedora defaulting to selinux -> enforcing now ? ... or did I make it happen somehow...13:47
slaglederekh: for fedora? should still be permissive afaik13:48
*** noslzzp has joined #tripleo13:50
derekhslagle: ok, the seed image I'm running has selinux enforcing, maybe I've locally screwed something up somewhere, its just tooo damn secure13:50
*** kbyrne has quit IRC13:50
dprinceotoolee: ah. So it looks like we'd need this too: https://review.openstack.org/#/c/97758/13:55
*** noslzzp has quit IRC13:58
*** julim has joined #tripleo13:59
dprincederekh: ^^14:00
dprincederekh: we are missing a heat-template "feature" so no wonder it thought it was broken14:00
derekhdprince: ahh ok14:01
*** noslzzp has joined #tripleo14:02
*** ukalifon1 has joined #tripleo14:05
derekhslagle: found my problem, don't set NODE_DIST=fedora , if you do selinux-permissive doesn't get added14:06
derekhhttp://git.openstack.org/cgit/openstack/tripleo-incubator/tree/scripts/devtest_variables.sh#n16014:06
derekhsomebody has been adding environment variables to my .devtestrc file over the holidays14:08
dprincederekh: I did it14:09
*** killer_prince is now known as lazy_prince14:10
derekhdprince: sounds about right, I did give you access to my network at some stage in the past14:10
dprincederekh: I mean... maybe :). You need a revokation list14:11
derekh;-)14:11
dprincederekh: anyways, I'm low risk. The likelyhood that I would actually remember where I put your IP address is secure enough14:11
derekhdprince: yup, I guessed as much14:12
*** lblanchard has joined #tripleo14:16
*** lucasagomes is now known as lucas-hungry14:20
*** pradeep1 has joined #tripleo14:26
*** pradeep has quit IRC14:26
*** jerryz has quit IRC14:30
sdakemorning14:40
sdakethanks tchaypo14:40
*** jerryz has joined #tripleo14:42
*** rhallisey_ has joined #tripleo14:45
*** rhallisey has quit IRC14:46
*** rhallisey_ has quit IRC14:48
*** rhallisey has joined #tripleo14:48
*** kbyrne has joined #tripleo14:52
*** pradeep2 has joined #tripleo14:56
*** jistr has quit IRC14:57
*** pradeep1 has quit IRC14:57
*** lucas-hungry is now known as lucasagomes14:57
*** jistr has joined #tripleo14:59
*** Marga_ has quit IRC15:10
*** lazy_prince is now known as killer_prince15:14
*** pradk has joined #tripleo15:20
*** jprovazn has quit IRC15:22
openstackgerritRyan Brady proposed openstack/diskimage-builder: Fix for RHEL6  https://review.openstack.org/14354415:28
openstackgerritRyan Brady proposed openstack/diskimage-builder: Allow for disabling rhel registration  https://review.openstack.org/14044615:28
*** dkingshott has joined #tripleo15:35
*** pradeep has joined #tripleo15:36
*** pradeep2 has quit IRC15:38
*** subscope has quit IRC15:44
*** rdopiera has quit IRC15:47
*** ukalifon1 has quit IRC15:48
*** Marga_ has joined #tripleo15:49
*** andrearosa has quit IRC15:52
*** jtomasek has quit IRC15:58
*** yamahata has joined #tripleo15:59
*** untriaged-bot has joined #tripleo16:00
untriaged-botUntriaged bugs so far:16:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/140551316:00
uvirtbotLaunchpad bug 1405513 in tripleo "[tripleo-heat-templates] haproxy option added for controller caused noisy debug log in keystone" [Undecided,New]16:00
untriaged-bothttps://bugs.launchpad.net/tuskar/+bug/138748716:00
uvirtbotLaunchpad bug 1387487 in tuskar "tripleo-image-elements/elements/tuskar-ui /os-refresh-config /post-configure.d/101-tuskar-ui  needs authentication from Keystone" [Undecided,New]16:00
*** untriaged-bot has quit IRC16:00
*** ukalifon1 has joined #tripleo16:01
*** jprovazn has joined #tripleo16:08
*** pcrews has joined #tripleo16:09
*** ukalifon1 has quit IRC16:11
*** jistr has quit IRC16:18
*** jrist has quit IRC16:18
*** dsneddon has joined #tripleo16:23
*** jrist has joined #tripleo16:24
*** jrist has quit IRC16:27
*** jrist has joined #tripleo16:27
*** jerryz has quit IRC16:30
*** phschwartz_ has quit IRC16:33
*** phschwartz has joined #tripleo16:33
*** jistr has joined #tripleo16:34
*** dtantsur is now known as dtantsur|afk16:41
*** ChuckC has quit IRC16:42
*** andrearosa has joined #tripleo16:47
*** cwolferh has joined #tripleo16:49
*** jtomasek has joined #tripleo16:50
*** rhallisey has quit IRC16:52
*** andrearosa has quit IRC16:52
*** andrearosa has joined #tripleo16:53
*** dsneddon has quit IRC16:57
*** dsneddon has joined #tripleo16:58
*** Marga_ has quit IRC16:59
*** Marga_ has joined #tripleo17:00
*** mrunge has quit IRC17:02
*** jcoufal has quit IRC17:05
SpamapSderekh: hi. So if I wanted to make a Debian/Ubuntu delorean flavor... would that be crazy or is there already an abstraction layer?17:05
*** rhallisey has joined #tripleo17:07
derekhSpamapS: not crazy, we currently don't have a good abstraction layer so would need to add one, delorean itself isn't very big so it shouldn't be a huge amount of work.17:08
*** sbadia has quit IRC17:12
*** sbadia has joined #tripleo17:13
*** penick has joined #tripleo17:14
SpamapSderekh: ty. Just tired of pip derailing everything. ;)17:15
derekhSpamapS: yup, I hear ya17:16
*** Marga_ has quit IRC17:17
*** Marga_ has joined #tripleo17:17
*** Marga_ has quit IRC17:21
*** pradeep1 has joined #tripleo17:21
*** Marga_ has joined #tripleo17:22
*** Marga_ has quit IRC17:23
*** Marga_ has joined #tripleo17:24
*** pradeep has quit IRC17:24
*** Marga_ has quit IRC17:24
*** Marga_ has joined #tripleo17:25
*** jistr has quit IRC17:26
*** penick has quit IRC17:26
lxsliwhat's this delorean thing? It's hard to search for17:26
derekhlxsli: its building upstream packages for F20 and putting them here http://209.132.178.33/repos/report.html17:29
derekhlxsli: a little doc https://openstack.redhat.com/packaging/rdo-packaging.html17:29
*** penick has joined #tripleo17:30
derekhlxsli: and the delorean code along with the packaging for upstream https://github.com/openstack-packages17:30
SpamapSI kind of wonder if we could just reuse either Debian or Ubuntu's work. If nothing else, their dependencies might prove useful.17:32
derekhSpamapS: for the packaging ?17:32
derekhspamapS: or the building17:32
shardyAnyone have any clues as to how I force DiB to use os-apply-config from latest master (or a fork)?17:33
tchaypoderekh: so i had a question for you17:33
SpamapSderekh: for the packaging.17:33
SpamapSderekh: we still need to build for each commit right?17:33
shardyThe venv seems to always end up with a pip derived version17:33
tchaypoderekh: oh, and i even managed to remember what it was. Etherpad says we’ve had nothing since the 26th of december, so i’m about to send an email saying “all quiet on the western front"17:33
derekhSpamapS: so the idea is that the delorean packaging would feed into fedora (i.e. delorean packaging is its upstream),17:33
derekhSpamapS: yup, we're building for each commit17:34
tchayposhardy: with the thing.17:34
derekhtchaypo: I know of nothing, but havn't been aqround over holidays so other may know of problems that happened17:34
derekh*others17:35
*** Marga_ has quit IRC17:35
*** chuckC_ has joined #tripleo17:35
tchaypoderekh: i’ve been around every day or two, i haven’t seen anything17:35
*** Marga_ has joined #tripleo17:35
derekhtchaypo: ok, sounds like we're good so17:36
tchaypoi thought i’d found something, but it was just one particular change that was broken17:36
tchayposhardy: what are you trying so far?17:36
SpamapSderekh: well AFAIK Ubuntu doesn't want a new upstream (though we could ask) and Debian probably would love upstream help.17:37
tchaypoI think that using the DIB_REPO* variables as per http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/source-repositories/README.md is a start, but maybe the os-apply-config element also needs to be told to do a source install?17:37
*** ifarkas has quit IRC17:37
derekhSpamapS: ack17:37
bnemectchaypo: derekh: Are you talking CI breakages?  We had one or two from the setuptools/pip issues recently.17:37
bnemecThey were mostly fixed without our involvement though.17:37
*** daneyon has joined #tripleo17:38
SpamapSderekh: either way, we're talking about packages that just lay the software down. We're still going to configure it using other tools and start/stop it using service control defined elsewhere, yeah?17:38
tchaypobnemec: yeah, the last one of those I17:38
tchaypo‘m aware of was17:38
derekhbnemec: yup ci breakages, iirc those were covered by the last mail from tchaypo17:38
*** dsneddon has quit IRC17:38
tchaypo26th dec, https://bugs.launchpad.net/tripleo/+bug/1405732, resolved by17:39
uvirtbotLaunchpad bug 1405732 in tripleo "rm: cannot remove `/home/jenkins/.cache/image-create/pypi/mirror/': Permission denied" [Critical,Fix released]17:39
shardytchaypo: Yeah, I'm trying DIB_INSTALLTYPE_os_apply_config=source, but I'm confused as the docs say that is the default anyway17:39
bnemectchaypo: This one also broke us: https://bugs.launchpad.net/tripleo/+bug/140716517:39
*** Marga_ has quit IRC17:39
uvirtbotLaunchpad bug 1407165 in tripleo "setuptools 10.2 breaks mako install" [Critical,Fix released]17:39
tchaypohttps://review.openstack.org/#/c/144184/ on dec 2817:39
bnemecBut very briefly.17:40
*** Marga_ has joined #tripleo17:40
*** dsneddon has joined #tripleo17:40
tchaypobnemec: would you mind popping some details on that into https://etherpad.openstack.org/p/tripleo-ci-breakages ?17:40
*** Marga_ has quit IRC17:41
bnemectchaypo: Yep, I forgot about that17:41
*** Marga_ has joined #tripleo17:41
derekhSpamapS: Yup, other consumers of the packages can use them as they please.17:41
tchaypobnemec: <317:42
tchayposhardy: i have vague memories that some things are just blindly installed from package regardless17:43
tchaypobut maybe that’s a ci-specific thing17:43
tchaypo i know recently i was making some changes to something (os-net-config?) but i couldn’t test my changes as ci kept installing the previous package version17:43
*** pensu has joined #tripleo17:44
shardytchaypo: Yeah, something like that seems to be happening, although this is running DiB locally17:45
derekhtchaypo: when you say recently do you mean before Nov 18 https://review.openstack.org/#/c/133047/17:45
shardyI just want to build an image with latest-master os-apply-config and a forked version of os-refresh-config17:45
*** EmilienM is now known as EmilienM|afk17:45
shardyI guess I'll just hack up the image I have instead17:45
derekhtchaypo: that would have been causing pip into intall the released version and ignore your patch17:46
*** dsneddon has quit IRC17:47
*** david-lyle_afk is now known as david-lyle17:47
*** daneyon_ has joined #tripleo17:52
tchaypoderekh: actually i tink it was https://review.openstack.org/#/c/136811/17:54
*** dsneddon has joined #tripleo17:54
*** daneyon has quit IRC17:55
tchaypoat least, past-me said on https://review.openstack.org/#/c/136558/ that 136811 was needed before 136558 would stop failing17:55
*** zz_avozza is now known as avozza17:55
derekhtchaypo: k17:55
tchaypolooking at it now i’m confused about why it was required17:56
*** athomas has quit IRC18:00
*** derekh has quit IRC18:02
*** wuhg has quit IRC18:04
openstackgerritJames Polley proposed openstack/tripleo-incubator: Fix EXTRA_ROUTE instantiation  https://review.openstack.org/14554918:08
*** pblaho_ has quit IRC18:16
*** pelix has quit IRC18:17
*** yamahata has quit IRC18:20
*** jang2 has joined #tripleo18:22
*** mrunge has joined #tripleo18:22
*** mrunge has quit IRC18:23
greghaynesI keep seeing jobs where jenkins kills the job due to timeout18:26
greghaynesnot sure what changed, im surprised were close to that limit at all actuallyd18:26
*** mcornea has quit IRC18:26
openstackgerritgreghaynes proposed openstack/tripleo-image-elements: Enable debug logs  https://review.openstack.org/14252918:30
tchaypoI *think* that what changed was heat retry support18:31
greghaynesoh!?18:31
tchaypopreviously we’d have random failures because $RANDOM happened and a vm blew up18:31
greghaynesI didnt realise there was an automated retry element18:32
tchayponow heat will retry the action, so we don’t see those so much18:32
greghayneswow18:32
greghaynesok, thats totally plausible18:32
tchaypoinstead we see systemic failures, where nova can’t provision a VM because it’s requesting too much RAM, or because networking is breaking, or because it’s trying to install a package that doesn’t exist18:32
tchaypoand heat just sits there and retries18:32
greghaynesok, so seems like we should be setting some sort of retry limit18:32
tchaypoI *think* that is the case, anyway. I haven’t dug very far to prove that18:33
tchayposure, I think that’s called a “timeout” ;)18:33
greghaynesnono18:33
greghayneswe need to do that too18:33
tchaypoyeah, true, a retry limite of 2 or 3 is probably a good idea18:33
tchaypobut as well, I’d like to see better surfacing of the underlying error18:33
greghaynesyea. Right now it does the worst possible thing which consumes resources until jenkins times it out18:34
tchaypojust seeing “retry count exceeded” is a start, but then we need to be able to find the thing that caused the failure18:34
greghaynesok, I also have a patch to make our CI timeouts a lot more sane so I should prioritize that ;)18:34
tchaypowhen I’ve done that tracking, you usually see neutron saying “no available host” first, then you have to dig further to find out why there was no available host18:34
greghaynesawesome18:34
tchaypowhich in some of my cases was because the networking config was all wrong so they never talked back, for instance18:34
openstackgerritTom Hancock proposed openstack/tripleo-incubator: add --uri flag for remote hypervisor  https://review.openstack.org/14526318:35
bnemecdprince: Okay, so I've finally got my devtest environment sane enough to actually try your puppet changes.18:39
bnemecI have images built and and overcloud deployed, but it doesn't look like puppet ever actually ran on the nodes.18:39
bnemecAny idea what I might have missed?18:39
dprinceCan you post your os-collect-config log somewhere?18:40
dprinceDo you see a 'puppet apply' in there anywhere?18:40
*** eghobo has joined #tripleo18:42
dprincebnemec: In your undercloud (or seed) do you have a nameserver setup?18:42
dprincebnemec: neutron subnet-show ... and look for dns nameserver18:42
bnemecdprince: Yeah, I've got the nameserver and I can ping out to google et al from the overcloud nodes, so I think that should be fine.18:43
bnemecLet me grab the o-c-c log18:43
dprincebnemec: great.18:43
*** regebro has quit IRC18:43
bnemecdprince: http://paste.openstack.org/show/155884/18:44
bnemecAnd here are the elements included in the overcloud-control node:18:44
bnemecBuilding elements: base  fedora selinux-permissive hosts baremetal dhcp-all-interfaces os-collect-config heat-config-puppet puppet-modules hiera os-net-config delorean-repo rdo-release stackuser pip-cache common-venv use-ephemeral use-ephemeral snmpd18:44
dprincebnemec: did you swap in the puppet resource registry?18:45
dprincebnemec: for Heat?18:45
sdakeSpamapS there?18:45
dprincebnemec: yes.18:45
dprincebnemec: export RESOURCE_REGISTRY_PATH="$TRIPLEO_ROOT/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml"18:46
dprincebnemec: I set that18:46
bnemecdprince: Oh, wth?  It looks like the resource registry path env var isn't set.18:46
dprincebnemec: that is probably it.18:46
bnemecBut it's in my list of env var exports.18:46
bnemecWonder how I missed it.18:46
bnemecdprince: Alright, will try again with that.  Thanks for your help.18:46
* greghaynes is suddenly very confused by http://paste.ubuntu.com/9689038/18:46
dprincebnemec: that is essentially the thing that flips our Heat templates into Puppet mode18:46
dprincebnemec: yep, good luck18:47
greghaynesthat doesnt really add up18:47
sdakemaybe someone else besides SpamapS can answer18:47
sdakedo folks think introducing containers to tripleo could be done by 1) making a tool that builds contaniers out of tripleo image elements and 2) modify how orc works to deploy containers rather then refresh on disk configs?18:47
bnemecOh, I forgot to export it.  No wonder it didn't stick. :-)18:48
*** Marga_ has quit IRC18:49
*** Marga_ has joined #tripleo18:50
greghaynessdake: That seems backwards to me. o-r-c doesnt really have any smarts for managing nodes18:50
dprincebnemec: if there are changes missing in https://review.openstack.org/#/c/143516/ (Document TripleO PUppet...) then feel free to add them18:50
greghaynessdake: that is a good question though...18:50
dprincebnemec: FWIW I did use an export in the doc though ;)18:51
bnemecdprince: Will do.  I think this was just a pebkac though.  For some reason I didn't have an export on my resource registry env var command.18:51
bnemecYeah, no idea how I screwed that up. :-)18:51
greghaynessdake: as for the building, dib should be able to output container image formats and then 'just use dib'18:51
greghaynessdake: have you looked at having heat spawn the containers?18:53
*** Marga__ has joined #tripleo18:53
*** Marga_ has quit IRC18:53
*** jehb has joined #tripleo18:58
greghayneshrm. I wonder if heat stack-show takes long enough to run that our wait_for 360 10 is actually something like a 2hr timeout instead of a 1hr19:00
*** jprovazn has quit IRC19:00
sdakegreghaynes re dib, perhaps that is the best place for the code, but limitations of docker (all files must be in one directory for example) mean using the existing DIB code will be impossible19:03
sdakeit would definately have to be something new that created dockerfiles out of image elements19:03
greghaynessdake: why? Cant you have dib output a tar with a dockerfile?19:03
greghaynesalso I thought they had some kind of binary image format thing19:04
greghaynes(my docker knowledge is actually pretty weak)19:04
sdakethey have their own binary image format, that is correct19:04
sdakebasically8 when you do "docker run"19:04
sdakeit loads the binary image into a unique address space (using namespaces)19:04
sdakebut it is still part of the host os scheduler19:05
sdakeso no virtualization, just a bit of isolation19:05
greghaynesok, so its not really portable?19:05
sdaketo create the binary image format you use a tool called "docker build"19:05
*** subscope has joined #tripleo19:05
sdakedocker build takes a file which looks alot like a shell script19:05
greghaynesalso, mordred was working on making dib do exactly this (build docker images)19:05
sdakecool I'll definately hook up with him19:06
sdakethankks for the pointer!19:06
greghaynesnp19:06
sdakeso orc manages only one node right?19:06
sdakeis that the thing that applies the refreshed updated config?19:06
greghaynesyes, its basically a script to apply a given state locally19:06
greghayneswell, a script runner to do that19:07
sdakeso a container refresh works by stopping the container, loading the new binary, starting the container19:07
sdakeassuming there are new binary bits19:07
greghaynesor just make new container, nuke old container19:07
sdakeif there are no binary bits, but only config options, those are passed like an argument list to a function19:07
sdakeyup that would work too19:07
sdakesince the container gets all the config options as "cli" input19:08
mordredwhat did I do?19:08
sdakeit uses that to configure the container19:08
sdakehey mordred \o19:08
mordredhey!19:08
sdakeyour working on dib of image elements into container?19:08
mordredyeah - Dockerfiles are woefully inadequate for complex things19:08
greghaynesyeeep19:08
greghaynesmy two day experimenting with docker resulted in my image formats being tar + makefile that gen'd a dockerfile19:09
greghaynesnot sure how crazy that was though19:09
mordredI am, although it's a little stalled right now beacuse I'm working on something else - but at the end of the day teaching dib how to do docker instead of qemu-nbd should be about 1 day's worth of work I think19:09
sdakeinteresitng, I thought it would be more complicated19:09
mordredgreghaynes: yah - that's another way - just use dib like normal, then export the filesystem and do a one-line dockerfile that uses the tarball as a base19:09
*** yamahata has joined #tripleo19:09
mordredsdake: nah - many things are conceptually very similar19:10
mordredI just haven't had time to finish it off19:10
* mordred waves hands - promises to hack more on this in a couple of weeks after he's done with dib-nodepool19:11
sdakecool thanks19:15
openstackgerritJames Polley proposed openstack/tripleo-image-elements: Reposition vlan type  https://review.openstack.org/14556019:18
*** jp_at_hp has quit IRC19:18
*** GheRivero has quit IRC19:21
*** dank has joined #tripleo19:37
*** dkingshott has quit IRC19:41
*** jehb has quit IRC19:41
*** GheRivero has joined #tripleo19:43
*** Will____ has joined #tripleo19:46
*** auld has joined #tripleo19:48
tchaypogreghaynes: nice math in the paste19:50
*** GheRivero has quit IRC19:51
*** lucasagomes is now known as lucas-dinner19:52
*** lucas-dinner has quit IRC19:52
openstackgerritgreghaynes proposed openstack/tripleo-incubator: WIP: Use walltime as wait_for  https://review.openstack.org/14556619:54
greghaynestchaypo: yep ^19:54
*** cwolferh has quit IRC19:54
*** spzala has joined #tripleo20:03
*** GheRivero has joined #tripleo20:03
greghaynesI wrote a one-off script to try and see where were spending time in devtest based on console.log's and heres the output http://paste.ubuntu.com/9689476/20:20
greghaynes(the number on the left is seconds)20:21
greghaynesits funny how long we spend installing lxml ;)20:21
greghaynesabout 10 mins20:21
openstackgerritMerged openstack/tripleo-heat-templates: Pass Horizon port through to controller nodes  https://review.openstack.org/14370020:23
clarkbyes lxml is a long compile20:24
openstackgerritgreghaynes proposed openstack/tripleo-incubator: Use walltime as wait_for  https://review.openstack.org/14556620:24
*** EmilienM|afk is now known as EmilienM20:25
*** pensu has quit IRC20:29
openstackgerritMerged openstack/os-cloud-config: Add support for creating Keystone v3 clients  https://review.openstack.org/13388520:33
openstackgerritMerged openstack/os-cloud-config: Do not use https for keystone v2 admin  https://review.openstack.org/14258020:38
*** GheRivero has quit IRC20:38
*** subscope has quit IRC20:41
SpamapSbut we shouldn't be spending any time recompiling lxml20:42
SpamapSmust not work well with ccache20:42
SpamapSgreghaynes: perhaps we should reconsider using --system-site-packages until we figure out if we want to do the packages thing.20:42
*** GheRivero has joined #tripleo20:56
*** GheRivero has quit IRC21:00
*** GheRivero has joined #tripleo21:00
*** openstackgerrit has quit IRC21:05
*** GheRivero has quit IRC21:05
*** openstackgerrit has joined #tripleo21:06
*** auld has quit IRC21:07
*** Will____ has quit IRC21:10
SpamapSStevenK: FYI, I rebased as requests here https://review.openstack.org/#/c/141217/21:11
*** daneyon_ has quit IRC21:16
*** daneyon has joined #tripleo21:17
*** nealph has joined #tripleo21:17
nealphlsmola: ping? looking to bounce some ideas around on how to make Ceilometer's "pipeline.yaml" config file more dynamic.21:19
nealphpassthrough-like functionality, for a config other than the standard "*.conf" file.21:20
*** daneyon has quit IRC21:21
*** Hefeweizen has quit IRC21:21
*** GheRivero has joined #tripleo21:22
openstackgerritJonathan Brownell proposed openstack/tripleo-image-elements: Adding iptable rules for swift-storage DIB element  https://review.openstack.org/11140921:22
*** julim has quit IRC21:25
SpamapSnealph: you could just make a json parameter that is directly written to pipeline.yaml if set.21:26
SpamapSnealph: JSON is valid yaml. :)21:26
nealphSpamapS: was actually thinking along those lines. Appropriate place for the param would be in overcloud-source.yaml, methinks?21:27
openstackgerritMerged openstack/tripleo-heat-templates: Drop the MysqlClusterUniquePart validation  https://review.openstack.org/14474621:27
SpamapSnealph: yes, it's basically just another form of passthrough.21:28
*** GheRivero has quit IRC21:33
*** GheRivero has joined #tripleo21:34
nealphSpamapS: the question for me is on the "receiving" end, i.e in Ceilometer's pipeline.yaml. I suppose I would shift that to be a HOT template, vs the current static file. Yes?21:35
nealph(https://github.com/openstack/tripleo-image-elements/blob/master/elements/ceilometer-overcloud-config/os-apply-config/etc/ceilometer/pipeline.yaml)21:35
*** daneyon has joined #tripleo21:36
*** dprince has quit IRC21:47
SpamapSnealph: not HOT, HOT is for templating OpenStack resources. You want an os-apply-config template.21:49
SpamapSnealph: oh! it already is.21:50
SpamapSnealph: so yeah, what you could do is wrap that in this {{#ceilometer.passthrough.pipeline}}{{.}}{{/ceilometer.passthrough.pipeline}}{{^ceilometer.passthrough.pipeline}}...current static content goes here...{{/ceilometer.passthrough.pipeline}}21:52
nealphPerfect. Yes, effectively an optional one, which fits the bill nicely.21:52
nealphit's an optional template, that is.21:53
SpamapSnealph: you can probably also just make it a string then.21:53
nealphthat was my next thought...whether I could avoid JSON. :)21:55
*** GheRivero has quit IRC21:56
*** chlong has joined #tripleo21:56
SpamapSnealph: you could always introduce a yaml parameter type to HOT. :)21:56
SpamapSnealph: the benefit of using JSON is it will be validated by Heat before acceptance.21:57
SpamapSWhich is nice.21:57
nealphvalidated for JSON validity? or something more?21:57
SpamapSnealph: just json22:00
*** untriaged-bot has joined #tripleo22:00
untriaged-botUntriaged bugs so far:22:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/140551322:00
uvirtbotLaunchpad bug 1405513 in tripleo "[tripleo-heat-templates] haproxy option added for controller caused noisy debug log in keystone" [Undecided,New]22:00
untriaged-bothttps://bugs.launchpad.net/tuskar/+bug/138748722:00
SpamapSstill helpful in some instances22:00
uvirtbotLaunchpad bug 1387487 in tuskar "tripleo-image-elements/elements/tuskar-ui /os-refresh-config /post-configure.d/101-tuskar-ui  needs authentication from Keystone" [Undecided,New]22:00
*** untriaged-bot has quit IRC22:00
*** rhallisey has quit IRC22:00
nealphSpamapS: agreed, there's some level of "stupid typo" checking there.22:01
*** GheRivero has joined #tripleo22:02
*** chlong has quit IRC22:03
nealphSpamapS: cool, I'm off to try it out. thanks for the guidance.22:03
*** jdob has quit IRC22:03
greghaynesSpamapS: Yea, --site-packages might be the way to go...22:04
greghaynesThe time calculated in that paste doesnt account for dependencies, etc too so its likely a lot longer22:04
*** ChuckC has joined #tripleo22:06
openstackgerritgreghaynes proposed openstack/tripleo-incubator: Use walltime as wait_for  https://review.openstack.org/14556622:06
greghaynesShameless plug for reviews on ^ - fixes an issue I keep seeing in CI where we end up not having logs for a failure... so cant really debug the original failure well22:07
greghaynesIt also already passed CI, just had a commit msg change22:07
*** auld has joined #tripleo22:09
*** GheRivero has quit IRC22:11
nealphgreghaynes: just a quibble, but should your deprecation warning string start with "deprecation warning"?22:13
greghaynesSure, why not :)22:13
nealphquibble < nitpick.22:13
openstackgerritgreghaynes proposed openstack/tripleo-incubator: Use walltime as wait_for  https://review.openstack.org/14556622:14
greghaynesargh, I should not edit commit msgs in gerrit, I end up losing them when I push a new rev22:15
openstackgerritgreghaynes proposed openstack/tripleo-incubator: Use walltime as wait_for  https://review.openstack.org/14556622:15
greghaynesWe also seem to be hitting the nodes going into deleting state in heat failure *a lot*. https://review.openstack.org/#/c/142529/ would be very helpful in debugging22:16
*** lblanchard has quit IRC22:16
clarkbgreghaynes: always git review -d22:16
clarkbgreghaynes: then you will be fine22:16
greghaynesooo22:17
greghaynesI wonder if I can just alias that22:17
*** GheRivero has joined #tripleo22:17
*** shardy is now known as shardy_z22:19
*** thrash is now known as thrash|g0ne22:22
*** GheRivero has quit IRC22:32
*** GheRivero has joined #tripleo22:38
*** jang2 has quit IRC22:41
*** gfidente has quit IRC22:42
*** chuckC_ has quit IRC22:48
*** GheRivero has quit IRC22:57
*** jtomasek has quit IRC22:58
*** pradk has quit IRC23:01
*** auld has quit IRC23:02
*** noslzzp has quit IRC23:03
*** noslzzp has joined #tripleo23:04
*** auld has joined #tripleo23:07
SpamapSgreghaynes: so..23:07
*** auld has quit IRC23:08
SpamapSgreghaynes: I'm not really understanding why you need a new wait_for23:08
SpamapSgreghaynes: the existing one doesn't intentionally work that way.23:08
SpamapSgreghaynes: why not just add a --walltime to wait_for, and deprecate expecting it to be off?23:09
greghaynesYea, I should. I went into writing it expecting it to come out a lot more different than it did23:14
greghaynesSpamapS: Do you remember why we intentionally dont use walltime in wait_for?23:14
*** Marga__ has quit IRC23:16
*** Marga_ has joined #tripleo23:16
bnemecgreghaynes: So https://review.openstack.org/#/c/142529/6 failed on exactly the bug it's trying to find, but I don't see the /var/log/debug directory in the seed logs.23:16
*** auld has joined #tripleo23:18
SpamapSJan  7 22:18:09 ubuntu ironic-conductor: 2015-01-07 22:18:09,205 WARNING Error in deploy of node a81fbfee-684e-4933-852d-3874514de24f: Failed to execute command via SSH: LC_ALL=C /usr/bin/virsh --connect qemu:///system destroy baremetalbrbm4_1.23:19
SpamapSJan  7 22:20:50 ubuntu ironic-conductor: 2015-01-07 22:20:50,829 WARNING During sync_power_state, node a81fbfee-684e-4933-852d-3874514de24f state 'power on' does not match expected state. Changing hardware state to 'power off'.23:19
SpamapSJan  7 22:21:20 ubuntu ironic-conductor: 2015-01-07 22:21:20,757 ERROR Failed to change power state of node a81fbfee-684e-4933-852d-3874514de24f to 'power off'. Attempts left: 2.23:19
SpamapSHmmm23:19
SpamapSgreghaynes: I don't think we intentionally don't use wall time. I think we want a timeout for the whole section.23:19
SpamapSThat failure looks like I'd need virsh logs from the testenv. :-P23:19
SpamapSWhich is not currently possible. I'll file a medium bug and move on.23:20
bnemecSpamapS: There's a bug open for that already.23:20
bnemecHigh priority because it hits us a _lot_ in CI.23:21
bnemecSpamapS: https://bugs.launchpad.net/tripleo/+bug/138078223:21
uvirtbotLaunchpad bug 1380782 in tripleo "InstanceDeployFailure in overcloud job" [High,Triaged]23:21
SpamapSbnemec: oh doh23:22
*** chlong has joined #tripleo23:22
bnemecI'm wondering if our CI virsh command filter is disallowing that for some reason.23:22
SpamapSbnemec: thanks, marked as dupe23:23
*** Hefeweizen has joined #tripleo23:23
greghaynesbnemec: hrm, corect23:24
bnemecI've actually got that bug number in a sticky note on my desktop so I can copy paste it.23:24
bnemecWhich is a rather bad approach to actually solving it...23:25
*** cody-somerville has quit IRC23:26
*** noslzzp has quit IRC23:28
bnemecOkay, doesn't appear to be the ci_commands filter script.23:31
* bnemec updates the bug23:32
*** ChuckC has quit IRC23:37
*** ChuckC has joined #tripleo23:38
*** andreaf has quit IRC23:38
*** cody-somerville has joined #tripleo23:39
*** cody-somerville has joined #tripleo23:39
*** ChuckC has quit IRC23:40
*** eghobo has quit IRC23:42
*** morazi has quit IRC23:44
*** Marga_ has quit IRC23:44
*** Marga_ has joined #tripleo23:45
*** Marga_ has quit IRC23:46
*** Marga_ has joined #tripleo23:47
*** Marga_ has quit IRC23:48
*** Marga_ has joined #tripleo23:48
*** Marga_ has quit IRC23:49
*** Marga_ has joined #tripleo23:49
*** Marga_ has quit IRC23:49
*** Marga_ has joined #tripleo23:50
*** auld has quit IRC23:50
*** penick has quit IRC23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!