Monday, 2014-03-03

*** e0ne has quit IRC00:04
*** jcooley_ has joined #tripleo00:15
*** jcooley_ has quit IRC00:19
*** cd-undercloud has joined #tripleo00:25
cd-undercloud************** overcloud complete status=1 ************00:25
*** cd-undercloud has quit IRC00:25
openstackgerrittom-howley proposed a change to openstack/tripleo-incubator: WIP - Add script to build all images.  https://review.openstack.org/7746200:52
*** e0ne has joined #tripleo00:59
*** e0ne has quit IRC01:04
*** cd-undercloud has joined #tripleo01:05
cd-undercloud************** overcloud complete status=1 ************01:05
*** cd-undercloud has quit IRC01:05
*** cd-undercloud has joined #tripleo01:07
cd-undercloud************** overcloud complete status=128 ************01:07
*** cd-undercloud has quit IRC01:07
*** CaptTofu has joined #tripleo01:07
*** CaptTofu has quit IRC01:10
*** CaptTofu has joined #tripleo01:11
*** CaptTofu has quit IRC01:15
*** CaptTofu has joined #tripleo01:16
*** CaptTofu has quit IRC01:20
*** CaptTofu has joined #tripleo01:24
*** nosnos has joined #tripleo01:28
*** yamahata_ has quit IRC01:37
*** yamahata_ has joined #tripleo01:37
*** xuhanp has joined #tripleo01:43
*** cd-undercloud has joined #tripleo01:52
cd-undercloud************** overcloud complete status=1 ************01:52
*** cd-undercloud has quit IRC01:52
*** xuhanp has left #tripleo01:52
*** weshay has quit IRC01:55
*** e0ne has joined #tripleo01:58
*** e0ne has quit IRC02:03
*** cd-undercloud has joined #tripleo02:26
cd-undercloud************** overcloud complete status=128 ************02:26
*** cd-undercloud has quit IRC02:26
*** ramishra_ has joined #tripleo02:27
*** ramishra_ has quit IRC02:38
*** ramishra has joined #tripleo02:39
*** CaptTofu has quit IRC02:47
*** e0ne has joined #tripleo02:58
*** cd-undercloud has joined #tripleo03:00
cd-undercloud************** overcloud complete status=1 ************03:00
*** cd-undercloud has quit IRC03:00
*** e0ne has quit IRC03:03
*** jprovazn has joined #tripleo03:11
* SpamapS heads to airport03:20
*** ramishra_ has joined #tripleo03:29
*** ramishra has quit IRC03:32
openstackgerritJenkins proposed a change to openstack/os-refresh-config: Updated from global requirements  https://review.openstack.org/7557503:33
*** jcooley_ has joined #tripleo03:49
*** ramishra_ has quit IRC03:56
*** e0ne has joined #tripleo03:58
*** e0ne has quit IRC04:03
*** akuznetsov has quit IRC04:18
*** akuznetsov has joined #tripleo04:19
*** nosnos has quit IRC04:28
*** sseago has quit IRC04:29
*** sseago has joined #tripleo04:32
*** nosnos has joined #tripleo04:34
*** CaptTofu has joined #tripleo04:48
*** CaptTofu has quit IRC04:52
*** e0ne has joined #tripleo04:58
*** jang1 has joined #tripleo05:02
*** e0ne has quit IRC05:03
*** jang1 has quit IRC05:08
*** jprovazn has quit IRC05:08
*** ramishra has joined #tripleo05:26
tchaypolifeless: the new message is05:36
tchaypo2014-03-02 23:01:32.289 3264 TRACE nova.compute.manager [instance: 7d5ebc10-5132-4f3a-b43e-e826bfebaadd] InstanceDeployFailure: Timeout reached while waiting for PXE deploy of instance 7d5ebc10-5132-4f3a-b43e-e826bfebaadd05:36
tchaypobut it did get much further this time05:38
*** killer_prince has joined #tripleo05:52
*** e0ne has joined #tripleo05:58
*** nosnos has quit IRC06:01
*** e0ne has quit IRC06:03
*** nosnos has joined #tripleo06:06
lifelesstchaypo: ok so06:15
lifelesstchaypo: CI is failing jobs06:15
lifelesstchaypo: methinks systemically06:16
lifelessah06:17
lifeless2014-03-03 00:08:09.661 | Installed openstack client tool symlinks in /opt/stack/new/tripleo-incubator/scripts06:17
lifeless2014-03-03 00:08:09.662 | libvirtd:x:117:ubuntu,fungi,mordred,corvus,jenkins,clarkb06:17
lifeless2014-03-03 00:08:09.699 | adding jenkins to group libvirtd06:17
lifeless2014-03-03 00:08:09.699 | jenkins was just added the the libvirtd.  Devtest will not06:17
lifelessI think we need to do that in node prep06:17
lifelesshmm no06:19
lifelesswe do the su -l thing06:19
*** ramishra has quit IRC06:19
lifelessoh, broken06:22
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Revert "Stop devtest if user was not in libvirtd group"  https://review.openstack.org/7750606:23
*** lsmola has joined #tripleo06:28
*** saju_m has joined #tripleo06:31
*** e0ne has joined #tripleo06:35
*** rlandy has joined #tripleo06:38
*** nosnos has quit IRC06:40
lifelessSpamapS: if you check in before  sleep ^ - you can see in https://jenkins07.openstack.org/job/check-tripleo-seed-precise/142/console that it fixed the issue06:41
*** rdopieralski has joined #tripleo06:46
*** CaptTofu has joined #tripleo06:49
*** CaptTofu has quit IRC06:54
*** nosnos has joined #tripleo07:04
*** e0ne has quit IRC07:09
killer_princeHi all.. I am trying to test one of my custome DIB elements for RHEL 6.5 using ironic. Since meta data service is broken in ironic, does it mean that os-apply-config will not execute os-refresh-config..?07:11
*** mrunge has joined #tripleo07:13
*** akrivoka has joined #tripleo07:19
fabbionelifeless: go back to sleep my friend :D07:23
*** akrivoka has quit IRC07:24
* SpamapS arrives07:29
*** akrivoka has joined #tripleo07:29
*** ramishra has joined #tripleo07:30
SpamapSkiller_prince: metadata service is broken in ironic?07:31
SpamapSkiller_prince: os-collect-config tries very hard to work even if one of its' sources is not available.. so ec2 will just not be present in the tree..07:31
SpamapSfabbione: lifeless is in California ATM.. so pacific time.. perfectly fine to be awake right now :)07:32
fabbioneSpamapS: ahh eheh.. i thought he was downunder :)07:32
fabbionenot a problem, was just saying hello to an old friend ;)07:33
killer_princeso I just want to verify since i am not sure if os-* services depend on cloud-init and correspondingly the metadata service?07:33
SpamapSfabbione: aye. He's graced our presence this week. :)07:36
SpamapSkiller_prince: they do need cloud-init to work07:36
openstackgerritA change was merged to openstack/tripleo-incubator: Revert "Stop devtest if user was not in libvirtd group"  https://review.openstack.org/7750607:36
SpamapSkiller_prince: because if not, there won't be any local configuration to tell os-collect-config where to find the Heat configuration07:36
*** ramishra has quit IRC07:45
*** akrivoka has quit IRC07:53
*** akrivoka has joined #tripleo07:55
*** jtomasek has joined #tripleo08:00
*** e0ne has joined #tripleo08:06
*** e0ne has quit IRC08:13
*** bauzas has joined #tripleo08:15
marioslsmola: morning08:16
lsmolamarios: hello08:17
marioslsmola: hey man i am looking at https://review.openstack.org/#/c/76267/6/tuskar/api/controllers/v1/overcloud.py - on line 171, do you mean the overcloud configuration?08:19
marioslsmola: (in the comments)08:19
lsmolamarios: yes08:20
lsmolamarios: the thing jistr will be doing08:20
lsmolamarios: though we are not bringing any workflow afaik, so it will be probably another action you will run manually08:20
marioslsmola: ok cool. but right now for this review in the code you are capturing the stack_id in anticipation of <however we handle that config> *(we spoke about maybe a new thread for handle it but we didn't work out the api).08:22
marioslsmola: in other words, there isn't anything here in the code which we couldn't merge today.08:22
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Using flavors from Roles  https://review.openstack.org/7682608:23
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Create stack in right order  https://review.openstack.org/7626708:23
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Fix of Overcloud Update  https://review.openstack.org/7685008:23
lsmolamarios: not sure if i get it :-)08:26
lsmolamarios: so what is the question here? :-D08:26
marioslsmola: :) its ok, basically just confirming that this review doesn't actually try and track the configuration. we just grab the stack id so we could use it later08:27
lsmolamarios: yes08:27
lsmolamarios: it the stack_id should be correctly filled08:27
lsmolamarios: as create method returns stack, we can do that now08:27
marioslsmola: ok, one other q. i asked on review v 2 but seems you missed it. in https://review.openstack.org/#/c/76267/7/tuskar/api/controllers/v1/overcloud.py line 52, i'm not 100% clear when we would have the overcloudrolecount defined in database and when we wouldn't08:29
marioslsmola: i.e. why do we need to check here if overcloud_roles is None08:30
lsmolamarios: yeah, that piece will be most probably refactored08:31
lsmolamarios: but right now it is stored as relation and for loading that, i need to have it in db08:31
lsmolamarios: which i don't before i create it08:31
lsmolamarios: I will rewrite it to some better way, i just haven't figured it out yet08:32
*** d0ugal has joined #tripleo08:32
*** d0ugal has joined #tripleo08:32
marioslsmola: thanks - still not 100% (sorry, monday morning) but lets discuss on the review will be easier.08:34
lsmolamarios: basically the .overcloud_role association is there as relation on models level08:35
lsmolamarios: but it works only if i query if from the db08:35
lsmolamarios: but I need to use it before it is stored to db, so it is not very good way08:36
lsmolamarios: that is why there is the second way, when the roles are loaded separately08:36
lsmolamarios: but it can be refactored to one way08:36
*** pbelanyi has joined #tripleo08:37
openstackgerritAna Krivokapic proposed a change to openstack/tuskar-ui: Use distribution pie charts for nodes  https://review.openstack.org/7721408:37
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Using flavors from Roles  https://review.openstack.org/7682608:39
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Create stack in right order  https://review.openstack.org/7626708:39
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Fix of Overcloud Update  https://review.openstack.org/7685008:39
*** saju_m has quit IRC08:40
*** saju_m has joined #tripleo08:42
*** ifarkas has joined #tripleo08:44
*** jistr has joined #tripleo08:47
lsmolamarios: hard to put todo there, when I don't know how  iwill fix it yet08:48
marioslsmola: that's not what todo is for. more like 'reminder' - here we need to do #something08:49
*** CaptTofu has joined #tripleo08:50
*** mrunge has quit IRC08:51
lsmolamarios: btw the explanation is in the overcloud_role parameter doc08:52
lsmolamarios: ok then, I will put there some todo08:52
*** CaptTofu has quit IRC08:54
*** gcha has joined #tripleo08:55
*** e0ne has joined #tripleo08:55
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Using flavors from Roles  https://review.openstack.org/7682609:00
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Create stack in right order  https://review.openstack.org/7626709:00
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Fix of Overcloud Update  https://review.openstack.org/7685009:00
lsmolamarios: added09:00
*** mrunge has joined #tripleo09:01
*** saju_m has quit IRC09:02
*** hashar_ has joined #tripleo09:07
*** athomas has joined #tripleo09:09
*** jcooley_ has quit IRC09:10
*** pblaho has joined #tripleo09:21
*** coolsvap has joined #tripleo09:30
*** saju_m has joined #tripleo09:34
*** e0ne_ has joined #tripleo09:47
*** e0ne has quit IRC09:50
*** ramishra has joined #tripleo09:53
*** jp_at_hp has joined #tripleo09:56
*** adrian17od has joined #tripleo10:02
*** martyntaylor has joined #tripleo10:03
*** hashar_ is now known as hashar10:09
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Return template attributes  https://review.openstack.org/7755510:25
*** saju_m has quit IRC10:25
openstackgerritDougal Matthews proposed a change to openstack/python-tuskarclient: Remove multivalue attributes and roles from CLI  https://review.openstack.org/7755610:25
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Error handling in the API  https://review.openstack.org/7681910:28
*** saju_m has joined #tripleo10:41
*** e0ne has joined #tripleo10:45
*** coolsvap has quit IRC10:47
*** e0ne_ has quit IRC10:49
*** CaptTofu has joined #tripleo10:51
openstackgerritstephen-mulcahy proposed a change to openstack/tripleo-incubator: Adding basic instrumentation to devtest to gain an understanding of the time used for each step, and the impact of adding elements such as pip-cache. Also allows comparison of runtime for different network environments.  https://review.openstack.org/7756410:54
*** CaptTofu has quit IRC10:55
*** athomas has quit IRC10:56
*** andreaf has joined #tripleo10:58
openstackgerritA change was merged to openstack/tuskar: Add six module dependency in requirements.txt  https://review.openstack.org/7733611:02
*** saju_m has quit IRC11:11
*** gcha has quit IRC11:11
*** athomas has joined #tripleo11:13
openstackgerritDmitry Tantsur proposed a change to openstack/tuskar-ui: Create node profile details page  https://review.openstack.org/7720811:16
*** nosnos has quit IRC11:21
*** saju_m has joined #tripleo11:25
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Return template attributes  https://review.openstack.org/7755511:31
*** bogdando has quit IRC11:38
openstackgerritLadislav Smola proposed a change to openstack/tuskar-ui: Real Overcloud Horizon url  https://review.openstack.org/7719711:39
openstackgerritIvan Berezovskiy proposed a change to openstack/diskimage-builder: Fixed missing of EXTRA_ARGS parameter in install-packages  https://review.openstack.org/7757011:45
*** jprovazn has joined #tripleo12:03
*** hashar has quit IRC12:25
*** e0ne has quit IRC12:36
*** gcha has joined #tripleo12:40
*** e0ne has joined #tripleo12:46
*** ramishra has quit IRC12:47
*** mrunge has quit IRC12:47
openstackgerritA change was merged to openstack/os-cloud-config: First post.  https://review.openstack.org/7737812:47
*** CaptTofu has joined #tripleo12:52
*** saju_m has quit IRC12:52
openstackgerritMichael Kerrin proposed a change to openstack/tripleo-incubator: Move wait_for for checking exit status to checking for condition to be true  https://review.openstack.org/7653712:53
openstackgerritMichael Kerrin proposed a change to openstack/tripleo-incubator: Add FAIL_MATCH_OUTPUT to wait_for...  https://review.openstack.org/7624012:53
*** CaptTofu has quit IRC12:56
*** lsmola has quit IRC12:57
*** hashar has joined #tripleo12:59
*** ramishra has joined #tripleo12:59
*** CaptTofu has joined #tripleo13:04
*** lsmola has joined #tripleo13:12
*** lsmola has quit IRC13:13
*** lsmola has joined #tripleo13:13
*** ramishra_ has joined #tripleo13:15
*** ramishra has quit IRC13:18
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Return template attributes  https://review.openstack.org/7755513:19
*** e0ne_ has joined #tripleo13:24
*** e0ne has quit IRC13:27
*** Shrews_ has joined #tripleo13:29
*** Shrews has quit IRC13:30
lifelessfabbione: o/13:31
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Return template attributes  https://review.openstack.org/7755513:32
*** juice has quit IRC13:33
*** juice has joined #tripleo13:35
*** kevinbenton has quit IRC13:35
*** kevinbenton has joined #tripleo13:35
*** e0ne has joined #tripleo13:36
SpamapSlifeless: 'morning13:39
fabbionelifeless: eheh morning...13:39
*** e0ne has quit IRC13:39
*** e0ne has joined #tripleo13:39
*** e0ne_ has quit IRC13:40
*** sballe_ has joined #tripleo13:44
*** weshay has joined #tripleo13:44
*** clarkb has quit IRC13:44
*** rdopieralski has quit IRC13:46
*** sballe has quit IRC13:48
*** jp_at_hp has quit IRC13:49
*** jp_at_hp has joined #tripleo13:50
*** ramishra_ has quit IRC13:50
openstackgerritLadislav Smola proposed a change to openstack/tuskar: Return template attributes  https://review.openstack.org/7755513:51
fungilifeless: SpamapS: i take it from the /topic that you're aware no tripleo jobs are running once again13:54
*** funzo_ has quit IRC13:54
lifelessSpamapS: dude, morning13:54
*** funzo has joined #tripleo13:55
lifelessfungi: I wasn't, we hadn't reset it.13:55
fungioh, too fun13:55
lifelessfungi: will do zomg morning things and then look into it13:55
fungilifeless: have fun! (i'm still in the middle of zomgmorning things myself)13:55
SpamapSfungi: when we are all talking and not zomgmorning-ing.. I'd like to know what we should monitor, as I think we're about to start alerting people more forcibly and it would be good to monitor infra's perception of our cloud as well as us monitoring the basics.13:57
* SpamapS zomgmornings13:57
*** tteggel has quit IRC13:58
*** tteggel has joined #tripleo13:59
fungiSpamapS: well, my first sign is usually when i skim the zuul status page looking for any sign of problems i'm about to need to fix, and happen to notice that there are pending tripleo jobs and no workers getting assigned to them. there's almost certainly something better which could spot the problem sooner, but would need some brainstorming14:00
*** morazi has joined #tripleo14:02
*** ftcjeff has joined #tripleo14:03
SpamapSlifeless: so, I guess we'll be getting a new motherboard for that box... and now I think it is time we deploy a new notcompute and perhaps practice our dd/takeovernode skills :-P14:05
SpamapSfungi: yeah in this case, the cloud is unable to reply to traffic.. so even the most basic monitor would show problems..14:06
SpamapSfungi: though previously, the quota issues would have been harder to spot14:06
openstackgerritstephen-mulcahy proposed a change to openstack/tripleo-incubator: Add basic timing instrumentation to devtest.sh  https://review.openstack.org/7756414:07
fungiSpamapS: agreed... perhaps a quick compare of the enrtries returned by nova list and show limits (or whatever it's called) to see whether they're in the same ballpark would work?14:09
fungibut again, that's checking for recurrence of a specific buggy behavior, which presumably will get fixed (and also was fallout from the outage itself, it seems)14:09
SpamapSfungi: right, that is why I'd rather have a monitor which looks to see that zuul and nodepool are able to use the cloud, rather than try to chase every detail and symptom14:11
SpamapSseems right now we have the most famous of all monitoring systems: "the users will yell if it breaks"14:12
fungipart of the problem is that it's a bit subjective. we expect to hit provider quotas sometimes, have brief periods where we can't reach things, have nodes built which are unresponsive, and so on14:12
fungiso tuning it to reflect actual impact to performance but doing so before it impacts performance is a little bit of a guessing game14:13
SpamapSfungi: Indeed, so queue length and error rate seem like the indicator, and I agree, tuning those to be "not too squawky" but "not too laissez faire" is tough.14:14
sdagueSpamapS: so you could poll http://zuul.openstack.org/status.json and ensure you are making progress14:14
sdaguebasically is there a running job14:14
sdaguein the tripleo queues14:14
sdagueand give some window of time that it's acceptable to not have a running job, like 15 minutes14:15
sdagueto account for transition14:15
fungione thing which we might be able to look at is if we were to breakout the graphite stats for nodes in a particular state per provider/subprovider, we could check for <x% nodes in a ready or used state for some predetermined period14:15
sdaguehonestly, you could probably make a sane nagios check for that in a couple hours14:15
SpamapSright, if len(queue): if not running_jobs: sqawk14:15
fungiand then have nodepool grow the ability to periodically test the reachability of nodes it thinks are ready, and switch them to delete if they cease to respond14:15
SpamapSOk.. some good ideas there. Now I really will zomgmorning as my tummy just reminded me that I'm currently starving myself and thus have not had calories in many moons.14:16
*** julim has joined #tripleo14:18
*** lblanchard has joined #tripleo14:19
*** e0ne has quit IRC14:22
*** e0ne has joined #tripleo14:22
SpamapSrebooting ci-overcloud controller14:22
SpamapSNMI panic14:22
SpamapS:(14:22
pbelanyi29153314:22
*** vkozhukalov has joined #tripleo14:28
adrian17odHi folks14:31
adrian17odI'm just starting to take a look at TripleO so as a first step I'd like to stand up a local env14:32
adrian17odIs is possible to deploy everything on VMs (following these instructions http://docs.openstack.org/developer/tripleo-incubator/devtest.html)14:32
SpamapSadrian17od: yes that produces 5 VMs14:33
SpamapSadrian17od: you can do those steps manually, or just clone openstack/tripleo-incubator and run scripts/devtest.sh --trash-my-machine and it will do them all automatically14:34
adrian17odDoes --trash-my-machine trash my workstation or a VM?14:35
lifelessSpamapS: 'yay'14:35
SpamapSadrian17od: it adds some bridges and installs some packages14:35
lifelessSpamapS: JIRAing it ?14:35
SpamapSlifeless: I will, first wanted to get it back into a working state so we can start moving to a new box14:36
SpamapSit is back up now... just running occ --force --one14:37
*** ccrouch has joined #tripleo14:38
adrian17odThanks SpamapS14:39
lifelessSpamapS: ack14:40
SpamapSgah .. hotel network was melting my brain14:48
SpamapS06:42:52.485488 IP 10.0.244.182.44616 > 184.105.137.237.22: Flags [.], ack 256, win 345, options [nop,nop,TS val 182222332 ecr 920315666], length 014:48
SpamapSnote .. window 345 .. no response.. :-P14:49
SpamapStethered on 4G now so much nicer14:49
SpamapS:P14:49
SpamapSfungi: things should be flowing again.. but I believe we'll be migrating to new hardware which will likely cause some interruption later today14:51
SpamapSanyway.. tummy really wants to not be empty :-P14:52
fungiSpamapS: GO EAT14:53
fungii'll check to see if we've got any nodes successfully building now14:54
*** jpeeler has quit IRC14:54
*** jpeeler has joined #tripleo14:54
*** jpeeler has joined #tripleo14:54
GheRiveromorning TripleO!14:55
lifelessfungi: I wouldn't call whats here 'food'14:55
jeblair14:18 <+fungi> and then have nodepool grow the ability to periodically test the reachability of nodes it thinks are ready, and switch them to delete if they cease to respond14:56
jeblairfungi: nodepool does that14:56
lifelessGheRivero: bon jorno!14:56
fungijeblair: it already does?14:56
lifelessit does14:56
fungii missed that, but awesome14:56
jeblairfungi: it grew that capability in response to hpcloud.14:56
fungiahh...14:56
lifelessof course it presumes thast node unreachable doesn't imply cloud unreachable14:56
jeblairfungi: which used to regularly "disappear" servers14:56
jeblairlifeless: correct14:57
lifelessthe real issue here is we aren't deploying a HA control plane yet14:57
* lifeless cracks the whip14:57
fungiso really, just monitoring to make sure there are some slaves in ready/used state over long enough to deal with a gate reset fully depleting reserves should be enough of a health check14:57
jeblair14:18 <+fungi> one thing which we might be able to look at is if we were to breakout the graphite stats for nodes in a particular state per provider/subprovider, we could check for <x% nodes in a ready or used14:57
jeblairfungi: i think it does that too14:57
*** ccorrigan has quit IRC14:58
fungii meant break out the stats we're looking at (right now we have a summary graph across all providers and image types)14:58
fungibut it did seem that we already generate those stats14:58
jeblairhttp://graphite.openstack.org/render/?width=586&height=308&_salt=1393858716.68&target=stats.gauges.nodepool.target.jenkins01.tripleo-precise.tripleo-test-cloud.ready14:58
jeblairexample ^14:58
fungiso the data is present today to be able to detect a condition wherein there are no nodes running jobs and none available to run jobs14:59
jeblairhttp://graphite.openstack.org/render/?width=586&height=308&_salt=1393858716.68&target=sumSeries%28stats.gauges.nodepool.target.jenkins*.tripleo-precise.tripleo-test-cloud.ready%2914:59
jeblairbetter example ^14:59
fungiand if that persists beyond the expected upper bound for a new node build, then we have a problem with the provider15:00
*** killer_prince has quit IRC15:00
fungiit would also point out situations like on friday when we were getting overlimit responses to nova create in rax-iad15:00
jeblairfungi: so, i think the correct infra responses to both of those events is to ignore them and watch tv.15:02
fungijeblair: touché15:03
jeblair?15:03
jeblairfungi: i mean, nodepool is designed to work around things like the rax-iad issue (because clouds fail all the time), so i don't really want to be notified whetever a cloud fails15:06
lifelessyou shouldn't be15:06
lifelessjeblair: the context here was monitoring to alert tripleo-cd-admins of failures15:06
lifelessjeblair: and picking a metric that is useful vs just being precise15:06
fungijeblair: i agree as far as notification is concerned. more that it's something we might be interested in tracking so we know how often we're hitting issues like that15:07
jeblairok15:07
fungi"oh, look, that dip in the summary graph is because rax-iad has no usable nodes--i bet we need to talk to them if this keeps happening and see what we can do to improve on that"15:08
jeblairgraphite data are available in json too, so you can get the near-instantaneous value for all this stuff15:08
*** AndreaRosa has joined #tripleo15:13
*** vkozhukalov has quit IRC15:18
openstackgerritStuart McLaren proposed a change to openstack/tripleo-image-elements: Glance: Make API delayed delete configurable  https://review.openstack.org/7761015:19
*** vkozhukalov has joined #tripleo15:21
*** lblanchard has quit IRC15:23
*** jang1 has joined #tripleo15:23
* SpamapS finally wanders down to try the pseudo-breakfast15:28
*** vkozhukalov has quit IRC15:29
*** jang1 has quit IRC15:32
openstackgerritStuart McLaren proposed a change to openstack/tripleo-image-elements: Glance: Make API bind host and port configurable  https://review.openstack.org/7761715:36
openstackgerritStuart McLaren proposed a change to openstack/tripleo-image-elements: Glance: Make API bind host and port configurable  https://review.openstack.org/7761715:38
*** coolsvap has joined #tripleo15:40
*** clarkb has joined #tripleo15:41
*** beekneemech is now known as bnemec15:43
*** lblanchard has joined #tripleo15:44
GheRiverowhat's the plan for today?15:54
*** rlandy has quit IRC15:55
GheRiverotaxi, shuttle, "small" walk?15:55
*** rdopieralski has joined #tripleo15:58
*** jcooley_ has joined #tripleo16:00
*** jcooley_ has quit IRC16:05
*** ramishra has joined #tripleo16:08
*** Shrews_ is now known as Shrews16:10
*** lazy_prince is now known as killer_prince16:12
*** ifarkas has quit IRC16:13
*** akrivoka has quit IRC16:22
*** jprovazn has quit IRC16:24
*** ramishra has quit IRC16:25
*** andreaf has quit IRC16:26
*** rbrady-afk is now known as rbrady16:28
*** ramishra has joined #tripleo16:29
openstackgerritDougal Matthews proposed a change to openstack/python-tuskarclient: Add API binding for the template_parameters action  https://review.openstack.org/7763716:34
*** ifarkas has joined #tripleo16:54
*** jcooley_ has joined #tripleo16:54
*** gcha has quit IRC16:57
*** bauzas has quit IRC16:57
*** e0ne has quit IRC16:58
*** spzala has joined #tripleo16:58
*** jcooley_ has quit IRC16:59
*** saurabhs has joined #tripleo16:59
*** jcooley_ has joined #tripleo17:01
*** lsmola has quit IRC17:03
*** cwolferh has joined #tripleo17:03
*** pbelanyi has quit IRC17:05
openstackgerritDougal Matthews proposed a change to openstack/python-tuskarclient: Add CLI support for the template_parameters action  https://review.openstack.org/7764917:06
*** adrian17od has quit IRC17:06
*** jistr has quit IRC17:08
openstackgerritDougal Matthews proposed a change to openstack/python-tuskarclient: Add docstrings to the CLI actions  https://review.openstack.org/7765117:09
*** coolsvap has quit IRC17:10
*** coolsvap has joined #tripleo17:11
openstackgerritA change was merged to openstack/diskimage-builder: Update the image name for RHEL Guest Image  https://review.openstack.org/7735317:11
*** gcha has joined #tripleo17:13
*** rdopieralski has quit IRC17:13
*** pblaho has quit IRC17:24
*** martyntaylor has quit IRC17:27
*** newell has joined #tripleo17:28
*** bauzas has joined #tripleo17:34
*** martyntaylor has joined #tripleo17:40
openstackgerritDerek Higgins proposed a change to openstack/diskimage-builder: Add $EXTRA_ARGS back to yum call  https://review.openstack.org/7718417:44
*** 1JTAAFJP3 has quit IRC17:45
*** killer_prince is now known as lazy_prince17:45
*** hashar has quit IRC17:53
*** rwsu has joined #tripleo17:55
*** tzumainn has joined #tripleo17:56
*** jang1 has joined #tripleo17:57
*** jdob has joined #tripleo17:57
*** vkozhukalov has joined #tripleo17:57
jp_at_hpCan any core reviewers take a look at https://review.openstack.org/#/c/75829/ and https://review.openstack.org/#/c/75831/ and https://review.openstack.org/#/c/77158/ please?17:59
*** slagle has joined #tripleo18:00
*** ramishra has quit IRC18:00
*** jang1 has quit IRC18:02
*** lucasagomes has joined #tripleo18:02
*** jang1 has joined #tripleo18:03
*** derekh has joined #tripleo18:03
*** dprince has joined #tripleo18:05
openstackgerritJonathan Brownell proposed a change to openstack/diskimage-builder: Fix spelling error in "vm" element README.md  https://review.openstack.org/7700718:06
*** dprince has quit IRC18:06
*** jang1 has quit IRC18:07
*** jang1 has joined #tripleo18:07
*** dprince has joined #tripleo18:08
derekhSpamapS: https://review.openstack.org/#/c/77184/18:08
*** tomh_ has joined #tripleo18:09
*** jprovazn has joined #tripleo18:11
*** athomas has quit IRC18:12
*** rollerj has joined #tripleo18:14
*** jcoufal has joined #tripleo18:15
lifelessSpamapS: 10.105.40.159/2118:16
slagleshadower: can you remove your +A from https://review.openstack.org/#/c/74866/ ?18:17
slagleshadower: we aren't merging stuff (except for critical fixes) until the CI is back18:17
derekhshadower: If you want to A+ something https://review.openstack.org/#/c/77184/18:22
derekhFedora builds are pretty much busted without it18:22
*** tzumainn has quit IRC18:23
*** tzumainn has joined #tripleo18:23
*** UtahDave has joined #tripleo18:24
*** rainya has joined #tripleo18:24
*** weshay has quit IRC18:31
tchaypoAny aussies feeling homesick should take a look at http://www.brisbanetimes.com.au/queensland/snake-eats-crocodile-after-epic-fight-in-queensland-20140303-33xz8.html18:33
jp_at_hpdprince: does this address your wishes for getting the sha1's used in builds out?  https://review.openstack.org/#/c/75829/18:33
jprovazns]18:34
openstackgerritA change was merged to openstack/diskimage-builder: Add $EXTRA_ARGS back to yum call  https://review.openstack.org/7718418:37
rollerjif the undercloud fails to deploy, were is the best place to debug why it failed?  i've looked for nova and heat logs on the seed without help.18:46
SpamapStchaypo: horrifying18:50
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Store node details in the JSON datafile  https://review.openstack.org/7314518:52
*** cody-somerville has joined #tripleo18:53
*** cody-somerville has joined #tripleo18:53
StevenKGheRivero: https://review.openstack.org/#/c/73145/18:55
*** e0ne has joined #tripleo18:59
*** cwolferh has quit IRC18:59
*** cwolferh has joined #tripleo19:01
*** cwolferh has quit IRC19:01
*** cwolferh has joined #tripleo19:01
*** matty_dubs|gone is now known as matty_dubs|sfo19:02
*** martyntaylor has left #tripleo19:05
*** rcarrillocruz has joined #tripleo19:06
*** weshay has joined #tripleo19:07
*** rcarrillocruz1 has quit IRC19:07
*** gcha has quit IRC19:12
*** tomh_ has quit IRC19:12
*** pblaho has joined #tripleo19:15
derekhSpamapS lifeless we reboot the overcloud controller , looks down :-(19:18
SpamapSderekh: agreed, on it19:18
*** jcoufal has quit IRC19:20
derekhSpamapS: k, will do19:20
*** jcoufal has joined #tripleo19:21
*** jcoufal has quit IRC19:26
*** mrunge has joined #tripleo19:26
derekhlifeless: SpamapS:  is that one of ye? "ci-overcloud TEXTCONS is already in use by a different client."19:27
*** cody-somerville has quit IRC19:27
StevenKGheRivero: Has my branch caused yours eyes to bleed yet?19:27
GheRiveroyes19:27
StevenKGheRivero: :-(19:27
lifelessderekh: there is a screen thing19:28
GheRiveromainly the setup-baremetal and backward compat19:28
lifelessderekh: so yes, sspis probaably in it19:28
derekhlifeless: ok, screen session on the cd-undercloud, I see 319:29
*** coolsvap has quit IRC19:29
pleia2dprince: https://review.openstack.org/#/c/75538/19:30
SpamapSderekh: screen -Ax robertc/ilos219:32
SpamapSderekh: you may need to be added to an acl if that doesn't work19:32
SpamapSderekh: box is rebooting now19:32
SpamapSderekh: first PXE failed19:33
SpamapSderekh: sorry for not being more responsive I had to switch networks19:33
derekhSpamapS: thanks, no prob19:33
*** e0ne has quit IRC19:35
*** rlandy has joined #tripleo19:35
*** e0ne has joined #tripleo19:36
SpamapS2nd attempt fails :-(19:38
*** cody-somerville has joined #tripleo19:39
SpamapSJira created DC-406119:39
*** sballe has joined #tripleo19:39
*** sballe_ has quit IRC19:40
*** jcoufal has joined #tripleo19:42
SpamapSbooting19:43
*** pblaho has quit IRC19:43
*** markmc has joined #tripleo19:44
SpamapSrunning occ19:52
StevenKNg: Now I find myself very curious about the $1 expansion ...19:53
SpamapSderekh: ping, ci overcloud back up19:53
derekhSpamapS: great19:54
SpamapSperhaps we should just cron reboot...19:54
SpamapS*/2 hours :-P19:54
derekh:-)19:54
SpamapSthat would work.. if reboot didn't kernel panic :-/19:55
*** jp_at_hp has quit IRC19:55
jprovaznhehe19:56
*** e0ne has quit IRC19:58
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Store node details in the JSON datafile  https://review.openstack.org/7314520:00
openstackgerritDougal Matthews proposed a change to openstack/python-tuskarclient: Add docstrings to the CLI actions  https://review.openstack.org/7765120:00
lifelessderekh: its on the one machine you don't have access to yet (we need to wipe it...)20:05
derekhlifeless: ok20:06
openstackgerritGregory Haynes proposed a change to openstack/tripleo-incubator: Add Ironic bits to _undercloud  https://review.openstack.org/7296920:07
GheRiveroStevenK: ping20:09
*** adrian17od has joined #tripleo20:14
*** e0ne has joined #tripleo20:15
openstackgerritGregory Haynes proposed a change to openstack/tripleo-incubator: Add Ironic bits to _undercloud  https://review.openstack.org/7296920:16
*** dprince has quit IRC20:25
*** hashar has joined #tripleo20:25
*** adrian17od has left #tripleo20:26
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Store node details in the JSON datafile  https://review.openstack.org/7314520:26
saurabhsdiskimage builder says 'first-boot.d is DEPRECATED'. so what is the replacement of it20:32
fungirandom question for the audience... if someone wanted to set up their own small-scale dev demo of tripleo right now, what's the hardware recommendation? (i'm going to assume that's mostly falling on the shoulders of what ipmi support exists within ironic)?20:37
*** ifarkas has quit IRC20:38
*** sparkycollier has joined #tripleo20:40
*** jcoufal has quit IRC20:42
*** mrunge has quit IRC20:45
openstackgerritBen Kero proposed a change to openstack/tripleo-incubator: add check to create ~/.ssh/ to install-dependencies  https://review.openstack.org/7744520:50
fungimainly curious whether there's any more public detail about hardware known to work well (for recent definitions of well) beyond the general mention in https://wiki.openstack.org/wiki/TripleO/TripleOCloud#Hardware_configuration20:51
*** vkozhukalov has quit IRC20:51
*** rcarrillocruz has quit IRC20:54
*** rcarrillocruz has joined #tripleo20:54
*** rcarrillocruz has quit IRC20:59
rbradyrwsu: ping21:07
*** rcarrillocruz has joined #tripleo21:13
*** markmc has quit IRC21:20
*** e0ne has quit IRC21:21
*** hashar has quit IRC21:30
*** julim has quit IRC21:35
rwsurbrady: ack21:35
*** hashar has joined #tripleo21:38
rbradyrwsu: on the nfs-server element, would you suggest sticking the code to create the folder in post-configure.d?21:38
rwsurbrady: yup I think so, neutron-server does something similar21:42
rbradyrwsu: thanks21:42
greghaynesfungi: devtest (which is what devs seem to run) doesnt even ironic ATM21:43
rwsurbrady: but you may need to update a config file to use the new path21:44
fungigreghaynes: more curious about running something pseudo-functional, like a scaled-down version of the current tripleo-ci undercloud21:44
greghaynesaha21:45
* greghaynes is currently picking some ironic brains at meetup to learn how to do that21:45
greghaynesbut its a slow process21:46
rbradyrwsu: the config file is likely going to be templated and I'll check existence / create the dir right before the command that consumes the config in the os-refresh-config file21:46
* tchaypo has finally managed to run devtest all the way though21:47
* tchaypo has been told to contact lifeless for a cigar21:47
greghaynes\O/21:47
greghaynesor at least a beer21:47
*** rbrady is now known as rbrady-bbl21:48
*** rlandy has quit IRC21:53
tchaypoStevenK: https://review.openstack.org/#/c/77048/21:53
*** CaptTofu has quit IRC21:54
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Add a ci cloud spam bot  https://review.openstack.org/7770922:01
*** ftcjeff has quit IRC22:03
*** tomh_ has joined #tripleo22:04
rwsurbrady: so was looking at https://review.openstack.org/#/c/74563/3/elements/cinder/install.d/cinder-source-install/72-cinder with Clint22:06
rwsurbrady: he pointed out that you can't run os-apply-config during image build time because the metadata isn't available at that time22:06
rwsurbrady: do we need that if clause at all?22:07
*** hrushig has joined #tripleo22:07
openstackgerritjames Polley proposed a change to openstack/tripleo-incubator: Clarify the platforms on which devtest has CI-testing currently.  https://review.openstack.org/7704822:07
rwsurbrady-bbl: ^22:07
openstackgerritA change was merged to openstack/diskimage-builder: Fix spelling error in "vm" element README.md  https://review.openstack.org/7700722:12
*** jcoufal has joined #tripleo22:12
*** dprince has joined #tripleo22:15
fungilifeless: devananda: Ng: SpamapS_: jog0_: GheRivero: derekh: dprince: slagle: just a heads up that it looks like the ci overcloud fell offline again around 17:00 utc22:18
tchaypofungi: when the overcloud isn't soaking your time, I'd love it if gerrit would notice that I've changed my name on launchpad22:18
fungiwhich i think was probably only a couple hours after you fixed it again22:18
*** jcooley_ has quit IRC22:19
fungitchaypo: you can change your full name at https://review.openstack.org/#/settings/contact22:19
fungigerrit doesn't update that from the openid, i don't think22:20
tchaypofungi: why yes, yes, I can22:20
*** markmc has joined #tripleo22:21
fungiabout the only thing you can't change is your ssh username. you're pretty much stuck with whatever it picked up initially (but at least ssh gives you a good way to specify the username for a given remote host in ~/.ssh/config)22:23
*** CLOUDOUTAGE has joined #tripleo22:24
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- ci-overcloud down22:24
*** CLOUDOUTAGE has quit IRC22:24
SpamapSci-overcloud controller motherboard being replaced right now \o/22:29
derekhwoot22:30
*** CLOUDOUTAGE has joined #tripleo22:30
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- ci-overcloud down (motherboard being changed)22:30
*** CLOUDOUTAGE has quit IRC22:30
slagle:)22:30
tchaypoThanks fungi22:32
tchaypoSo now I'm at a loose end.22:32
tchaypoMaybe time for me to do some triage and look for bugs I think I can bash on22:33
*** lblanchard has quit IRC22:34
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Clarify the platforms on which devtest currently runs CI testing.  https://review.openstack.org/7704822:39
fungiSpamapS: awesome22:43
fungii love the CLOUDOUTAGE bot, btw ;)22:43
tchaypoit's very annoying and spammy22:43
tchaypo+122:43
derekhannoying and spammy was the goal22:44
SpamapSderekh: seems you've a talent for that :)22:44
derekhSpamapS: thanks22:44
*** epim has joined #tripleo22:48
tchayposo what I think I want to work on now is getting `virsh console` to work for the seed and undercloud so that I don't have to tunnel vnc across the pacific.22:50
StevenKYou mean tunnelling vnc across the pacific isn't fun?22:50
*** jcooley_ has joined #tripleo22:51
slaglelifeless: i'm trying this, which must not be right: https://etherpad.openstack.org/p/tripleo-rationalise-ui22:52
tchaypono, no it is not.22:52
*** jcooley_ has quit IRC22:53
*** sparkycollier has quit IRC22:56
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Turn the toci seed and undercloud job into noop's  https://review.openstack.org/7772122:58
lifelessslagle:22:59
lifelesshttps://etherpad.openstack.org/p/tripleo-incubator-rationalise-ui22:59
*** CLOUDOUTAGE has joined #tripleo23:00
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- ci-overcloud down (motherboard being changed)23:00
*** CLOUDOUTAGE has quit IRC23:00
*** jcooley_ has joined #tripleo23:00
*** julim has joined #tripleo23:07
*** CaptTofu has joined #tripleo23:09
*** tomh_ has quit IRC23:15
*** tzumainn has quit IRC23:16
lifelesshttps://etherpad.openstack.org/p/tripleo-incubator-rationalise-ui23:24
SpamapSFYI, JIRA ticket has changed to SYS-5563 now as we wait for ILO to get DHCP23:29
*** hrushig has quit IRC23:30
*** hashar has quit IRC23:30
*** CLOUDOUTAGE has joined #tripleo23:31
CLOUDOUTAGElifeless devananda Ng SpamapS jog0 GheRivero derekh dprince slagle  -- ci-overcloud down (motherboard being changed)23:31
*** CLOUDOUTAGE has quit IRC23:31
*** geerdest has joined #tripleo23:32
*** slagle has quit IRC23:33
*** slagle has joined #tripleo23:34
*** slagle has quit IRC23:38
*** jcooley_ has quit IRC23:38
*** slagle has joined #tripleo23:38
*** jcooley_ has joined #tripleo23:38
*** slagle has quit IRC23:40
*** slagle has joined #tripleo23:40
*** jcooley_ has quit IRC23:43
*** jcooley_ has joined #tripleo23:43
*** jcooley_ has quit IRC23:45
*** andreaf has joined #tripleo23:48
*** jcooley_ has joined #tripleo23:51
*** UtahDave has quit IRC23:53
*** UtahDave has joined #tripleo23:58
*** ramishra has joined #tripleo23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!