Tuesday, 2014-02-04

bnemecomgwhitespace -1. ;-)00:00
bnemecI could live with that though.00:00
openstackgerritlifeless proposed a change to openstack/tripleo-heat-templates: Make make overcloud.yaml DTRT always.  https://review.openstack.org/7087400:00
bnemecIt can't be copy-pasted, but at least the correct value for JSONFILE is documented.00:00
lifelessSpamapS: ^00:01
*** ftcjeff has quit IRC00:01
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Attempt to make devtest_testenv docs work better.  https://review.openstack.org/7087300:01
*** boris-42_ has quit IRC00:03
*** boris-4__ has quit IRC00:07
openstackgerritlifeless proposed a change to openstack/tripleo-heat-templates: Support neutron.flat-networks in overclouds.  https://review.openstack.org/7087600:07
*** marun has joined #tripleo00:10
*** matsuhashi has joined #tripleo00:12
* SpamapS -> kids00:13
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Use new cleaner setting for OVERCLOUD_LIBVIRT_TYPE  https://review.openstack.org/7087800:14
*** marun has quit IRC00:15
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Use new cleaner setting for OVERCLOUD_LIBVIRT_TYPE  https://review.openstack.org/7087800:18
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Long overdue change to use multiple -P's with heat  https://review.openstack.org/7087900:18
lifelesslunchtime00:22
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Support flat-networks for the overcloud.  https://review.openstack.org/7088000:22
*** mikal_ is now known as mikal00:29
*** nosnos has joined #tripleo00:33
*** tserong has joined #tripleo00:33
lifelesstserong: are suse planning on sending anyone to the tripleo meetup in march ?00:34
slaglelifeless: the doc syntax errors aside, it makes it not obvious whether you're supposed to run the devtest_* scripts or following the commands on the individual pages instead00:37
lifelessslagle: ok, lets add some prose about that.00:37
lifelessslagle: have you captured the syntax errors in the review?00:37
slaglelifeless: not yet. really, just generate the docs and have a look00:38
lifelessslagle: heh, I need to futz with my venvs a bit for that00:39
tseronglifeless, not that i'm aware of.  I'll email our internal dev list though just in case00:39
tserongthis is in Sunnyvale, right?00:41
lifelesstserong: yes00:43
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Support flat-networks for the overcloud.  https://review.openstack.org/7088000:43
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Don't source devtest_end.sh.  https://review.openstack.org/7086100:43
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Reintegrate devtest actions with docs.  https://review.openstack.org/7086000:43
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Long overdue change to use multiple -P's with heat  https://review.openstack.org/7087900:43
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Attempt to make devtest_testenv docs work better.  https://review.openstack.org/7087300:43
*** nosnos_ has joined #tripleo00:44
lifelessslagle: I've put some prose in there; would love tweaks00:46
tseronglifeless, email sent, i'll let you know if/when I hear back from anyone (but they're probably all asleep right now because most of them are in europe)00:46
lifelesskk00:46
lifelesstserong: just arranging room sizes etc00:46
tserongsure00:46
*** nosnos has quit IRC00:47
*** weshay has joined #tripleo00:52
*** nosnos_ has quit IRC00:54
NobodyCamI have a DiskImage builder question with regard to DIB_REPOREF_<name> is it possible to pull two patchs?00:54
*** nosnos has joined #tripleo00:54
clarkbNobodyCam: at the very least you can construct a new ref with the two refs stacked00:55
NobodyCam:) I can try that :)00:56
NobodyCamTy clarkb00:56
*** CaptTofu has joined #tripleo00:56
clarkbnp, sorry I don't have a better answer than that, not super familiar with dib00:58
lifelessNobodyCam: no, you need to make a ref which includes both01:11
lifelessNobodyCam: if they are in a series, grabbing the to one will include the other transitively01:12
*** lblanchard has joined #tripleo01:12
NobodyCamahh ok :) Ty lifeless01:12
openstackgerritA change was merged to openstack/tuskar-ui: Rename Openstack to OpenStack  https://review.openstack.org/7048001:16
*** ccrouch has joined #tripleo01:20
*** mestery has joined #tripleo01:25
*** mestery has quit IRC01:26
*** mestery has joined #tripleo01:26
*** TravT has quit IRC01:27
*** marun has joined #tripleo01:28
*** marun has quit IRC01:33
*** peoplemerge has quit IRC01:37
*** mestery has quit IRC01:47
lifelessStevenK: oh hai01:50
lifelessStevenK: you're very quiet today :)01:50
lifelessgreghaynes: is your COMPUTESCALE thing working ?01:52
greghaynesgood question, havent tried it rebased off master and its been there for a bit01:53
*** markmcclain1 has joined #tripleo01:54
*** markmcclain has quit IRC01:54
lifelessCause, I need it for the ci-overcloud redeploy :)01:55
lifelessslagle: any chance of a +2 on https://review.openstack.org/#/c/70207/ ?01:55
*** markmcclain1 has quit IRC01:56
StevenKlifeless: I had to rebase 69060 losing all my votes, and I've been glaring at 6501401:59
lifelessStevenK: can I throw a puzzle your way ?02:00
*** markmcclain has joined #tripleo02:01
*** CaptTofu has quit IRC02:02
StevenKlifeless: I am just about to nom on lunch, but certainly02:02
lifelessok so02:02
*** CaptTofu has joined #tripleo02:03
lifelesshttps://review.openstack.org/#/c/70239/ and https://review.openstack.org/#/c/70252/02:03
lifelessare patches to make file injection start down the path to DIAF02:03
lifeless239 failed in grenade, apparently by nova-compute exiting silently02:03
lifelesswhich is a bit WTF02:03
lifelessbut then it passed02:03
lifelesswhich we thought was because russell ninjad the codepaht02:03
lifelessbut his patch to devstack was broken02:03
lifelessso it seems to genuinely pass02:04
lifelessbut for some unit tests02:04
lifelessso the puzzle is02:04
lifelessis it really good02:04
lifelessor not02:04
lifeless(and unittests need fixing)02:04
lifelessit probably needs 4-5 stabs at it with recheck no bug02:04
lifelessto be confident02:04
lifeless +02:04
lifelessthe unittests fixed02:04
lifelessbefore we can say 'land this shizzle'02:04
StevenKlifeless: recheck no bug now requires a core to do so, IIRC?02:05
lifelessclarkb: ^ wut?02:05
StevenKlifeless: And what is it with you not writing tests? :-)02:05
lifelessStevenK: I'm context switching like crazy right now02:06
StevenKlifeless: What do you mean 'right now'? :-P02:06
clarkbno just reverify02:06
lifelessStevenK: one of the things I"m hoping to ask you / greghaynes / wendar_ is to take more stuff off of me so I can context switch less02:06
lifelesswhich you're doing as you get up to speed02:06
StevenKlifeless: When you're bored, I'd love comments on 65014, around the .unittests change (so I can add coverage back), and the dsvm-swift failure (no module subunit, WTF?)02:07
lifelessdata point - my latest trunk builds have failed to bring up the ovs agent proerly; service neutron-openvswitch-agent restart was needed before nova boot of bm's worked02:08
greghaynesMaybe we make a lifeless ctxt switch graph - number of projects with open reviews / number of reviews02:08
lifelessStevenK: the functional failure ?02:08
lifelessStevenK: what do you mean w.r.t. coverage?02:08
lifelessgreghaynes: https://review.openstack.org/#/dashboard/419002:08
greghayneshaha02:09
lifelessgreghaynes: I know, right?02:10
lifelesshow to make btrfs DIAF: $ sync02:12
lifelesspossibly I need to run a trim02:14
*** epim has quit IRC02:17
lifeless 2307 root      20   0     0    0    0 R  83.4  0.0  35:19.96 flush-btrfs-102:28
lifelessI do so regret using btrfs02:28
lifelessDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util02:29
lifelesssda               0.00  1820.00    0.00  123.20     0.00  7911.20   128.43     2.12   17.25    0.00   17.25   8.06  99.3602:29
StevenKlifeless: The functional failure is http://logs.openstack.org/14/65014/7/check/check-swift-dsvm-functional/b098e3e ; WRT coverage look at the diff for .unittests02:35
*** lblanchard has quit IRC02:42
lifelessoh gawd02:51
lifelesslooks like chattr +C is even more insidious than I thought02:51
lifelessdoesn't seem to stick if you run it on existing non-0-length files.02:51
lifelessSTAB02:51
lifelessno freaking wonder this has been so appallingly slow02:52
*** vkozhukalov has joined #tripleo02:53
*** markmcclain has quit IRC02:56
lifelessStevenK: sorry, dealing with locked up browser basically, due to fs writes being stupendously slow atm02:56
lifelessStevenK: irc is fine though, so if you can describe with words...02:56
*** dkehn_ has joined #tripleo03:03
*** spzala has quit IRC03:05
*** boris-42_ has joined #tripleo03:12
lifelessbbiaw, kid time03:19
*** CaptTofu has quit IRC04:32
lifelessSpamapS: one thing on seed restarts - mysql gets confused ;)04:37
StevenKlifeless: So the diff explains the .unittests thing.04:41
StevenK2014-02-03 04:57:53.362 | 2014-02-03 04:57:53 + [[ g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,horizon,mysql,rabbit,sysstat,pidstat,s-proxy,s-account,s-container,s-object,cinder,c-api,c-vol,c-sch,n-cond,c-bak,heat,h-api,h-api-cfn,h-api-cw,h-eng,ceilometer-acompute,ceilometer-acentral,ceilometer-collector,ceilometer-api,ceilometer-alarm-notifier,ceilometer-alarm-evaluator,ceilometer-anotification,n-net =~ baremetal ]]04:41
StevenK2014-02-03 04:57:53.362 | /usr/bin/python: No module named subunit04:41
StevenKThat just makes me WTF04:41
lifelessStevenK: whats the .unittests thing ?04:41
lifelessStevenK: thats a different job, it might not be installing test-requires04:42
lifelessStevenK: or it might be not using a venv but subunit is installed in a venv04:42
StevenKHm.04:42
StevenKlifeless: Is your browser still an unhappy pile of fail?04:43
StevenK-nosetests --exe --with-coverage --cover-package swift. --cover-erase $cover_bra04:43
StevenK+testr run04:43
StevenKTrying 'python -m subunit.run discover -t ./ ./test/unit' rather than testr ends up with it listing the tests and complaining04:45
lifelessStevenK: yes04:46
lifelessStevenK: or rather sync is04:46
lifelessStevenK: which my browser uses like a hamster with its first wheel04:46
StevenKlifeless: Which is more or less than dpkg? :-)04:46
lifelessStevenK: more04:46
lifelessStevenK: sqlite ...04:47
lifelessStevenK: because everyone needs a transactional DB for their url history and bookmarks04:47
lifelessits really getting in the way of restoring ci-overcloud04:47
StevenKSure they do. Mozilla says they do.04:47
StevenK*cough*04:47
lifeless/dev/mapper/lifelesshp-root  168G  166G  239M 100% /04:48
lifelessGC dammit, GC04:48
StevenKlifeless: So my question is how do I turn on coverage with either testr or python -m subunit.run ?04:48
lifelessStevenK: have you looked at the way coverage is done in nova?04:49
StevenKlifeless: Trying to, but it's not clear what it runs under the venv with --coverage04:52
lifelessStevenK: it runs testr then fixes up the results from each runner separately04:53
lifelessStevenK: that might be abstracted away via setup.py testr04:53
StevenKlifeless: Yeah, nova's run_tests.sh uses setup.py testr05:10
lifelessStevenK: so thats all you should need then - copy that glue across05:12
StevenKlifeless: Except that python setup.py testr gives tracebacks05:13
greghayneslifeless: Looks like the COMPUTESCALE patch does work :)05:16
lifelessStevenK: greghaynes cool05:16
greghaynesAlso, fixed my overcloud: PEBKAC05:16
StevenK  File "/home/steven/openstack/openstack/swift/.tox/py27/local/lib/python2.7/site-packages/testtools/testsuite.py", line 299, in filter_by_ids05:17
StevenK    suite_or_case._tests[:] = filtered05:17
StevenKTypeError: 'TestSuite' object does not support item assignment05:17
StevenKA whole bunch of tracebacks ending like that05:17
*** coolsvap has joined #tripleo05:21
*** markmcclain has joined #tripleo05:28
*** rwsu has quit IRC05:41
lifelessgreghaynes: what was it?05:49
lifelessStevenK: interesting05:49
lifelessStevenK: ._tests should be a list, well defined on TestSuite05:49
greghayneshad the IP as the seed vm, so was totally trying to use seed for heat under and over cloud05:49
lifelessStevenK: are you returning a custom TestSuite or something?05:50
lifelessgreghaynes: heh05:50
greghayneswhen I did a heat list with all seed vm stuff sourced and saw both over and under I had a good laugh05:50
*** killer_prince has joined #tripleo05:50
StevenKlifeless: Not that I'm aware of. Note that a bare 'testr run' works fine05:50
lifelessStevenK: does testr run --parallel work fine?05:51
lifelessStevenK: in the py27 venv?05:51
StevenKlifeless: Nope, that gives the same tracebacks05:52
lifelessStevenK: ok, so thats strange :)05:52
lifelessStevenK: TESTR_PDB=1 testr run --parallel05:53
lifelessand have a poke05:53
StevenKlifeless: That doesn't drop to pdb05:54
lifelessoh05:55
lifelessits a plumbing failure05:55
lifelesstestr list > foo05:55
lifelesspython -m testtools.run <same args that subunit.run is run with but replace the tmp listing file with foo>05:56
lifelesserm might be list-tests, but whatever05:56
StevenKAh ha, so using concurrency 1 helps, but it looks like I get a raw subunit stream to stdout05:56
lifelessStevenK: from subunit.run? of course.05:58
StevenKThat's setup.py testr, but there was a --subunit in --testr-args05:59
*** nosnos_ has joined #tripleo06:00
lifelessoh right, thats for the custom display06:00
lifelessanyhow, you need to do what I described above to run it in testtools.run06:00
lifelessonce you can reproduce there06:00
lifelessthen you can do06:00
lifelesspython -m pdb /path/to/testtools/run.py <...>06:01
lifelessmmm, may need to pass the args in as 'run <...>' within pdb06:01
StevenKlifeless: python setup.py testr --coverage --testr-args='--concurrency 1' works06:02
lifelessright06:02
lifelessbecause you're avoiding the filtering06:02
lifelessI gave instructions on doing the filtering by hand at the layer below, to trigger the fault where you can inspect it06:02
lifelessStevenK: start at 'its a plumbing failure'06:03
*** nosnos has quit IRC06:03
StevenKlifeless: Except that testr list gives "KeyError: 'Could not import command module testrepository.commands.list'"06:04
*** jcoufal has joined #tripleo06:04
lifelessStevenK: tried list-tests ?06:16
*** tzumainn has quit IRC06:20
*** e0ne has joined #tripleo06:23
*** jcooley_ has joined #tripleo06:26
*** e0ne has quit IRC06:27
*** rdopieralski has joined #tripleo06:31
*** Shrews_ has joined #tripleo06:32
*** coolsvap_away has joined #tripleo06:32
*** matsuhashi has quit IRC06:33
*** matsuhashi has joined #tripleo06:33
*** SpamapS_ has joined #tripleo06:33
*** shadower_ has joined #tripleo06:34
*** jayg|g0n` has joined #tripleo06:38
*** vkozhukalov has quit IRC06:38
*** coolsvap has quit IRC06:38
*** tserong has quit IRC06:38
*** marios has quit IRC06:38
*** SpamapS has quit IRC06:38
*** w_ has quit IRC06:38
*** d0ugal has quit IRC06:38
*** Shrews has quit IRC06:38
*** jayg|g0n3 has quit IRC06:38
*** shadower has quit IRC06:38
*** mordred has quit IRC06:38
*** tserong has joined #tripleo06:45
*** tserong has joined #tripleo06:45
*** d0ugal has joined #tripleo06:45
*** d0ugal has quit IRC06:45
*** d0ugal has joined #tripleo06:45
*** vkozhukalov has joined #tripleo06:45
*** mordred has joined #tripleo06:54
*** marios has joined #tripleo07:00
*** olaph has joined #tripleo07:01
*** coolsvap_away has quit IRC07:04
*** markmcclain has quit IRC07:10
*** jcooley_ has quit IRC07:12
*** saju_m has joined #tripleo07:12
*** coolsvap has joined #tripleo07:26
*** bcrochet has joined #tripleo07:29
*** mrunge has joined #tripleo07:31
*** akuznetsov has joined #tripleo07:37
GheRiveromorning TripleO07:38
*** rwsu has joined #tripleo07:42
*** rlandy has joined #tripleo07:46
*** jcooley_ has joined #tripleo07:48
*** pblaho has joined #tripleo07:48
*** rwsu has quit IRC07:51
*** rwsu has joined #tripleo07:53
*** akuznetsov has quit IRC07:54
*** jcooley_ has quit IRC07:56
*** e0ne has joined #tripleo07:59
*** jtomasek has joined #tripleo07:59
*** bauzas has joined #tripleo08:04
*** jprovazn has joined #tripleo08:07
*** killer_prince is now known as lazy_prince08:08
*** jistr has joined #tripleo08:11
*** akuznetsov has joined #tripleo08:13
openstackgerritA change was merged to openstack/tripleo-image-elements: Document shortcomings of pypi-openstack element  https://review.openstack.org/7079108:15
*** jtomasek has quit IRC08:17
*** jtomasek has joined #tripleo08:17
*** jtomasek has quit IRC08:22
*** jtomasek has joined #tripleo08:22
*** pblaho has quit IRC08:34
*** pblaho has joined #tripleo08:35
*** coolsvap has quit IRC08:36
lifelessruh roh08:40
lifeless| 6ce42318-1bbe-41ba-b2ca-71f16b10a705 | DHCP agent         | ubuntu             | xxx   | True           |08:40
lifeless| aac87c16-da4f-4a9c-9339-a0c198ab21e8 | DHCP agent         | ubuntu.localdomain | :-)   | True           |08:40
lifelesstwo agents08:40
lifelesshmmm08:40
*** ifarkas has joined #tripleo08:41
lifelessok thats fun08:41
lifelessI think i've tracked down at least one cause of multiple neutron agents08:42
lifelesshostname changes in the seed :/08:42
*** hashar has joined #tripleo08:46
*** vkozhukalov has quit IRC08:49
*** lsmola has joined #tripleo08:56
*** gcha has joined #tripleo08:57
*** athomas has joined #tripleo09:00
*** markmc has joined #tripleo09:13
*** dtantsur has joined #tripleo09:16
*** dtantsur has left #tripleo09:19
*** coolsvap has joined #tripleo09:23
*** lucasagomes has joined #tripleo09:34
*** dtantsur has joined #tripleo09:43
*** e0ne has quit IRC09:46
*** e0ne has joined #tripleo09:46
*** e0ne_ has joined #tripleo09:50
*** e0ne has quit IRC09:50
* Ng not around today. Family birthday things to be done :)09:57
*** e0ne_ has quit IRC10:02
*** e0ne has joined #tripleo10:02
*** e0ne has quit IRC10:06
*** e0ne has joined #tripleo10:07
*** e0ne has quit IRC10:10
*** e0ne has joined #tripleo10:10
*** martyntaylor has joined #tripleo10:14
openstackgerritA change was merged to openstack/tripleo-image-elements: Add a helper to deploy testenvironments.  https://review.openstack.org/7028510:16
*** jcooley_ has joined #tripleo10:17
*** jcooley_ has quit IRC10:23
*** jistr has quit IRC10:27
*** d0ugal has quit IRC10:29
*** vkozhukalov has joined #tripleo10:30
*** coolsvap has quit IRC10:30
*** d0ugal has joined #tripleo10:32
*** d0ugal has joined #tripleo10:32
*** boris-42_ has quit IRC10:43
openstackgerritJiri Tomasek proposed a change to openstack/tuskar-ui: Updated Nodes Overview page  https://review.openstack.org/6938010:44
*** lazy_prince has quit IRC10:46
openstackgerritA change was merged to openstack/tripleo-heat-templates: Make make overcloud.yaml DTRT always.  https://review.openstack.org/7087410:46
openstackgerritA change was merged to openstack/tripleo-heat-templates: Support neutron.flat-networks in overclouds.  https://review.openstack.org/7087610:46
*** cody-somerville has joined #tripleo10:46
*** jistr has joined #tripleo10:50
*** e0ne has quit IRC10:50
*** e0ne has joined #tripleo10:50
*** hashar has quit IRC10:58
*** CaptTofu has joined #tripleo11:02
openstackgerritJiri Tomasek proposed a change to openstack/tuskar-ui: Updated Nodes Overview page  https://review.openstack.org/6938011:05
*** CaptTofu has quit IRC11:07
*** e0ne has quit IRC11:10
*** jcooley_ has joined #tripleo11:13
*** jcooley_ has quit IRC11:19
*** akrivoka has joined #tripleo11:22
*** jcoufal has quit IRC11:26
*** nosnos_ has quit IRC11:30
*** CaptTofu has joined #tripleo11:32
*** gcha has quit IRC11:39
openstackgerritA change was merged to openstack/tripleo-incubator: Make devtest use COMPUTESCALE for overcloud  https://review.openstack.org/7020712:07
*** e0ne has joined #tripleo12:17
*** CaptTofu has quit IRC12:22
*** e0ne_ has joined #tripleo12:24
*** e0ne has quit IRC12:24
lsmolahello12:27
lsmolaanybopdy has ERROR: Unknown resource Type : OS::Heat::UpdateWaitConditionHandle , running devtest.sh ?12:27
*** e0ne has joined #tripleo12:29
openstackgerritA change was merged to openstack/tripleo-image-elements: Fix mysql element on Fedora 19  https://review.openstack.org/7072912:29
*** e0ne__ has joined #tripleo12:30
*** e0ne___ has joined #tripleo12:31
*** e0ne__ has quit IRC12:31
*** boris-42_ has joined #tripleo12:32
*** e0ne_ has quit IRC12:32
*** e0ne___ has quit IRC12:32
*** e0ne_ has joined #tripleo12:32
openstackgerritJenkins proposed a change to openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/7096412:33
*** e0ne has quit IRC12:34
*** matsuhashi has quit IRC12:36
*** matsuhashi has joined #tripleo12:36
*** matsuhashi has quit IRC12:41
*** hashar has joined #tripleo12:49
*** weshay has quit IRC12:56
*** markmcclain has joined #tripleo12:59
*** gcha has joined #tripleo13:00
*** gcha has quit IRC13:03
*** e0ne has joined #tripleo13:05
*** gcha has joined #tripleo13:06
*** e0ne_ has quit IRC13:08
*** e0ne_ has joined #tripleo13:13
*** michchap has quit IRC13:14
*** e0ne has quit IRC13:16
*** jprovazn has quit IRC13:20
*** jcoufal has joined #tripleo13:21
*** CaptTofu has joined #tripleo13:22
*** rainya has joined #tripleo13:27
*** jdob has joined #tripleo13:30
*** weshay has joined #tripleo13:32
*** athomas has quit IRC13:33
*** athomas has joined #tripleo13:41
*** saju_m has quit IRC13:41
*** jprovazn has joined #tripleo13:44
*** ifarkas has quit IRC13:51
*** ifarkas has joined #tripleo13:52
*** ftcjeff has joined #tripleo14:00
*** dprince has joined #tripleo14:05
*** lsmola has quit IRC14:05
*** Shrews_ is now known as Shrews14:06
*** hashar has quit IRC14:10
*** hashar has joined #tripleo14:10
*** markmcclain has quit IRC14:15
*** lblanchard has joined #tripleo14:20
*** hashar_ has joined #tripleo14:20
*** hashar has quit IRC14:21
*** jayg|g0n` is now known as jayg14:22
*** lsmola has joined #tripleo14:22
*** pblaho has quit IRC14:22
*** hashar_ is now known as hashar14:23
*** tzumainn has joined #tripleo14:27
dprinceAnyone else seeing PXE ARP timeout issues today?14:29
*** matty_dubs|gone is now known as matty_dubs14:29
dprinceJust tried bringing up an undercloud twice and I see it both times... The second time I directly connected my cable (avoiding any router issues)14:29
openstackgerritA change was merged to openstack/tripleo-image-elements: Add dependency on source-repositories  https://review.openstack.org/6985414:30
*** spzala has joined #tripleo14:34
*** akuznetsov has quit IRC14:35
fungidoes anybody know if lifeless got the tripleo-ci cloud back on line? i'd like to reenable it in our nodepool but not if it's still down (since we apparently still have lingering bugs in nodepool which don't deal well with prolonged provider outages)14:38
*** lsmola has quit IRC14:41
*** coolsvap has joined #tripleo14:42
*** jdob_ has joined #tripleo14:49
*** jdob has quit IRC14:49
*** jcooley_ has joined #tripleo14:51
*** morazi has joined #tripleo14:52
*** lsmola has joined #tripleo14:53
*** boris-42_ has quit IRC14:54
*** jcooley_ has quit IRC14:57
*** lsmola has quit IRC15:05
*** rbrady has quit IRC15:07
*** rbrady has joined #tripleo15:07
*** jdob has joined #tripleo15:13
*** lsmola has joined #tripleo15:18
*** hashar has quit IRC15:19
*** jdob_ has quit IRC15:19
*** jistr has quit IRC15:23
*** jistr has joined #tripleo15:23
jprovaznrwsu, this is debug output on seed machine when calling os-collect-config on undercloud: http://paste.openstack.org/show/62446/15:35
jprovaznrwsu, "Could not find credential.." - are you hitting the same issue?15:35
*** TravT has joined #tripleo15:36
*** rdopieralski has quit IRC15:41
*** hashar has joined #tripleo15:45
*** jcooley_ has joined #tripleo15:46
jprovaznbbl15:51
*** jprovazn is now known as jprovazn_afk15:51
rwsujprovazn: seed comes up for me, but devtest can't ping it15:51
*** jcooley_ has quit IRC15:51
*** CaptTofu has quit IRC15:57
*** CaptTofu has joined #tripleo16:02
*** jcooley_ has joined #tripleo16:09
openstackgerritMarios Andreou proposed a change to openstack/tuskar: WIP: Using merge.py from tuskar to generate overcloud.yaml  https://review.openstack.org/5204516:17
*** dtantsur has quit IRC16:20
*** vkozhukalov has quit IRC16:31
*** max_lobur has quit IRC16:33
*** michchap has joined #tripleo16:34
*** max_lobur has joined #tripleo16:35
rwsujprovazn_afk: this is what I'm seeing: http://paste.openstack.org/show/62458/16:38
rwsuget a redirect in ping and that is causing the wait_for to timeout.16:38
rwsuanyone else seeing this?16:38
*** michchap has quit IRC16:38
*** lsmola has quit IRC16:39
*** jcooley_ has quit IRC16:40
*** rlandy is now known as rlandy|bbl16:42
*** markmcclain has joined #tripleo16:42
*** jistr has quit IRC16:47
*** edmund has joined #tripleo16:49
*** marun has joined #tripleo16:50
*** mestery has joined #tripleo16:51
*** mestery has quit IRC16:51
*** lsmola has joined #tripleo16:51
*** dprince has quit IRC16:52
*** mestery has joined #tripleo16:53
*** CaptTofu has quit IRC16:53
*** akuznetsov has joined #tripleo16:56
bnemecrwsu: I'm seeing the same thing.  Can't ping the seed at 192.0.2.1.16:56
*** boris-42_ has joined #tripleo16:57
rwsubnemec: thanks, was wondering if it was just me or a beaker issue16:58
bnemecrwsu: Yeah, I'm not running in beaker so it's definitely not that.16:59
bnemecStrangely, I see 192.0.2.1 configured on the seed, so I'm not sure why it's not responding.16:59
*** e0ne_ has quit IRC16:59
*** boris-42_ is now known as boris-4217:00
*** akuznetsov has quit IRC17:00
*** CaptTofu has joined #tripleo17:01
*** jcoufal_ has joined #tripleo17:02
*** jcoufal has quit IRC17:02
*** mestery has quit IRC17:02
*** akuznetsov has joined #tripleo17:02
*** mestery has joined #tripleo17:03
*** matty_dubs is now known as matty_dubs|lunch17:03
*** boris-42 is now known as boris-42_17:04
*** mestery has quit IRC17:05
*** markmc has quit IRC17:05
*** mestery has joined #tripleo17:06
*** mestery_ has joined #tripleo17:10
*** mestery_ has quit IRC17:11
*** mestery has quit IRC17:11
*** mestery has joined #tripleo17:12
*** cwolferh has joined #tripleo17:12
*** ifarkas has quit IRC17:12
*** slagle has quit IRC17:15
*** peoplemerge has joined #tripleo17:17
*** mrunge has quit IRC17:21
*** bauzas has quit IRC17:24
bnemecrwsu: I think I see the problem: http://paste.openstack.org/show/62461/17:33
bnemeceth0 is the configured interface, but br-ctlplane is on eth1.17:33
*** mestery_ has joined #tripleo17:34
*** jcooley_ has joined #tripleo17:34
*** jrist has quit IRC17:35
bnemecNope, I was wrong.17:36
bnemecThe route didn't get created on my host system correctly for some reason.17:36
*** athomas has quit IRC17:37
*** mestery has quit IRC17:38
bnemecAh, I ran the route command in the wrong terminal and didn't have SEED_IP set.17:38
bnemecrwsu: ^17:39
bnemecThat seems to have gotten communication with my seed working again.17:39
rwsucool i'll try that17:40
*** TravT has quit IRC17:40
*** TravT has joined #tripleo17:40
* SpamapS_ lives17:43
*** SpamapS_ is now known as SpamapS17:43
*** jcooley_ has quit IRC17:44
*** akuznetsov has quit IRC17:44
*** jrist has joined #tripleo17:48
*** akuznetsov has joined #tripleo17:50
*** akuznetsov has quit IRC17:51
*** dprince has joined #tripleo17:54
lifelessSpamapS: oh hi17:55
*** lucasagomes has quit IRC17:55
*** akuznetsov has joined #tripleo17:55
lifelessI seem to have found a nasty little race locally17:55
*** jrist has quit IRC17:56
lifelesshmm, no17:56
lifelesssomething else17:56
lifeless2 nodes registered, can only boot one17:56
lifeless2014-02-04 17:57:36.607 6431 DEBUG nova.scheduler.filters.ram_filter [req-f738f041-bf63-45cf-b914-209931329f60 1d4dbedad355438888dcb1e1af82f64d 962e421a84504665b6bc4e3603ab2805] (undercloud, 6bd5945d-6bcf-4e22-a3e7-f31c6af203dc) ram:0 disk:0 io_ops:0 instances:1 does not have 2048 MB usable ram, it only has 0.0 MB usable ram. host_passes /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/scheduler/filters/ram_filter.py:6017:58
lifeless2014-02-04 17:57:36.610 6431 INFO nova.filters [req-f738f041-bf63-45cf-b914-209931329f60 1d4dbedad355438888dcb1e1af82f64d 962e421a84504665b6bc4e3603ab2805] Filter RamFilter returned 0 hosts17:58
lifeless| 2  | undercloud | 1    | 2048      | 20      |             | None       | None        |             | None          |18:01
lifeless| 3  | undercloud | 1    | 2048      | 20      |             | None       | None        |             | None18:01
lifeless2048 for both of those.. curiouser and curioser18:01
*** gcha has quit IRC18:01
*** hashar has quit IRC18:01
*** matty_dubs|lunch is now known as matty_dubs18:01
*** TravT has quit IRC18:03
bnemecI've had nova-bm get in a bad state where it thinks a node is allocated but it shouldn't be.18:04
*** max_lobur is now known as max_lobur_afk18:05
*** rbrady is now known as rbrady-afk18:05
lifelessbnemec: yah, found that here18:05
lifelessthere's a bug still open I believe on that situation18:05
*** e0ne has joined #tripleo18:08
*** jrist has joined #tripleo18:09
*** akuznetsov has quit IRC18:11
*** coolsvap is now known as coolsvap_away18:16
*** michchap has joined #tripleo18:24
*** jcooley_ has joined #tripleo18:24
openstackgerritAna Krivokapic proposed a change to openstack/tripleo-incubator: Make sure the path of default storage pool is used  https://review.openstack.org/7105218:25
*** jcooley_ has quit IRC18:26
*** dkehn has quit IRC18:27
*** martyntaylor has left #tripleo18:28
*** akuznetsov has joined #tripleo18:28
*** dkehn has joined #tripleo18:28
*** michchap has quit IRC18:28
*** akuznetsov has quit IRC18:29
*** vkozhukalov has joined #tripleo18:30
*** wendar_ is now known as wendar18:31
*** e0ne_ has joined #tripleo18:33
*** akuznetsov has joined #tripleo18:34
*** e0ne has quit IRC18:35
*** akuznetsov has quit IRC18:35
*** akuznetsov has joined #tripleo18:35
*** akuznetsov has quit IRC18:35
*** akuznetsov has joined #tripleo18:35
*** akuznetsov has quit IRC18:36
*** akuznetsov has joined #tripleo18:36
*** mestery_ has quit IRC18:38
*** dprince has quit IRC18:39
*** markmcclain has quit IRC18:40
*** sdake has quit IRC18:42
*** jistr has joined #tripleo18:42
*** akuznetsov has quit IRC18:42
*** sdake has joined #tripleo18:43
*** sdake has joined #tripleo18:43
*** ccrouch has quit IRC18:46
*** slagle has joined #tripleo18:47
*** jprovazn_afk is now known as jprovazn18:50
*** rlandy|bbl is now known as rlandy18:51
*** jcooley_ has joined #tripleo18:55
*** sdake has quit IRC18:55
*** markmcclain1 has joined #tripleo19:00
*** markmcclain1 has quit IRC19:01
*** mestery has joined #tripleo19:04
*** jcoufal_ is now known as jcoufal19:06
*** ccrouch has joined #tripleo19:06
*** markmcclain has joined #tripleo19:06
*** dprince has joined #tripleo19:06
*** e0ne_ has quit IRC19:09
*** akrivoka has quit IRC19:14
*** jcooley_ has quit IRC19:17
*** shadower_ is now known as shadower19:18
*** sdake has joined #tripleo19:19
*** sdake has joined #tripleo19:19
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Fix chattr +C of VM images.  https://review.openstack.org/7107519:23
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Support flat-networks for the overcloud.  https://review.openstack.org/7088019:23
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Don't source devtest_end.sh.  https://review.openstack.org/7086119:23
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Reintegrate devtest actions with docs.  https://review.openstack.org/7086019:23
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Long overdue change to use multiple -P's with heat  https://review.openstack.org/7087919:23
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Minor syntax cleanup in devtest.sh.  https://review.openstack.org/7085719:23
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: When debugging setup-endpoints set -x as well.  https://review.openstack.org/7084619:23
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Attempt to make devtest_testenv docs work better.  https://review.openstack.org/7087319:23
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Stop sourcing devtest_undercloud.  https://review.openstack.org/6979719:23
*** TravT has joined #tripleo19:26
*** pblaho has joined #tripleo19:27
*** jamezpolley_ is now known as jamezpolley19:32
*** jamezpolley is now known as tchaypo19:34
*** jcooley_ has joined #tripleo19:36
openstackgerritDan Prince proposed a change to openstack/tripleo-incubator: Add devtest support for power management  https://review.openstack.org/6792619:37
*** pblaho has quit IRC19:37
*** ccrouch1 has joined #tripleo19:38
*** ccrouch has quit IRC19:38
lifelessLOL19:39
lifelessdeploying a 3 node cloud locally19:39
lifelesshit the PXE timeout on the third node19:39
lifelessits now re-converting and re-dding :/19:39
SpamapSlifeless: ah, timer starts for all 3 at one time?19:39
lifelessSpamapS: no, each has their own timer, started on the nova boot call19:40
lifelessSpamapS: but, heat triggers nova boot for them all at once19:40
lifelessSpamapS: then we thrash doing the converts in parallel, then we serialise on dd19:40
*** sdake has quit IRC19:42
*** sdake has joined #tripleo19:43
*** sdake has quit IRC19:44
*** sdake has joined #tripleo19:44
*** sdake has quit IRC19:44
*** sdake has joined #tripleo19:44
*** sdake has joined #tripleo19:44
*** jcooley_ has quit IRC19:44
greghaynesSpamapS: Trying to self triage that bug - do I need any special perms to do that? The importance edit icon has a red slash through it :(19:44
lifelessSpamapS: and I'm seeing the poor dd performance you were, but I haven't tracked down to whether that is poor IO in the target VM or bridge issues19:44
lifelessgreghaynes: have you joined ~tripleo on launchpad ?19:45
greghaynesDont think so19:45
lifelessthat would be it then19:45
greghaynesrequest set19:45
greghayness/set/sent19:45
*** jcooley_ has joined #tripleo19:49
*** rlandy has quit IRC19:51
*** bauzas has joined #tripleo19:58
*** jtomasek has quit IRC20:00
*** markmc has joined #tripleo20:00
*** jistr has quit IRC20:03
*** jcooley_ has quit IRC20:03
*** CaptTofu has quit IRC20:06
*** jcooley_ has joined #tripleo20:07
*** gcha has joined #tripleo20:15
*** hashar has joined #tripleo20:17
*** panda__ has joined #tripleo20:23
*** mestery has quit IRC20:23
*** UtahDave has joined #tripleo20:25
*** panda has quit IRC20:26
jprovaznhm, my access_key_id in cfn.json doesn't match access key id in keystone db on seed machine: http://paste.openstack.org/show/62474/20:26
jprovaznany idea how this could happen?20:26
jprovaznundercloud template uses:20:28
jprovazn        cfn:                                                                                                                                                                                 |20:28
jprovazn          access_key_id:                                                                                                                                                                     |20:28
jprovazn            Ref: notCompute0Key20:28
jprovaznwhich seems to be ok20:28
*** e0ne has joined #tripleo20:28
SpamapSjprovazn: how did you look up the key in keystone?20:31
jprovaznSpamapS, I suppose \"access\"... should match access_key_id (I could be wrong but on another deployment these 2 equal)20:32
jprovaznSpamapS, or do you mean something else? query was: select * from credential;20:33
SpamapSjprovazn: ok, I guess there should only be a few there. :)20:34
SpamapSjprovazn: did you do an update of any kind?20:34
SpamapSjprovazn: I mean, a stack-update20:34
jprovaznSpamapS, no stack-update20:34
jprovaznI ran devtest script20:35
SpamapSjprovazn: that is indeed strange.20:37
SpamapSjprovazn: if you do a 'heat resource-metadata notCompute0Config' do you see the right value?20:38
*** markmcclain has quit IRC20:38
*** weshay has quit IRC20:42
jprovaznSpamapS, http://paste.openstack.org/show/62478/  - maybe it's because the ERROR state (which I didn't check before20:43
*** markmcclain has joined #tripleo20:45
*** weshay has joined #tripleo20:46
*** sgrasley has joined #tripleo20:47
jprovaznactually not - ERROR state was set after undercloud unsuccessfully tried to fetch metadata20:48
*** sdake has quit IRC20:48
*** slagle has quit IRC20:49
*** sdake has joined #tripleo20:50
*** sdake has joined #tripleo20:50
SpamapSjprovazn: hm20:52
* jprovazn retries devtest again20:52
*** panda has joined #tripleo20:56
jog0do we use ceilometer  in tripleo yet?20:57
jog0I ask because of http://lists.openstack.org/pipermail/openstack-dev/2014-February/026226.html20:58
*** jrist has quit IRC20:58
ccrouch1jog0: I'm not sure its used, but i know we have an element for adding it to the overcloud20:58
ccrouch1and rwsu has looked into running on both under and over cloud20:59
*** panda__ has quit IRC20:59
jog0ccrouch1: thanks20:59
SpamapSit would be quite sexy to have graphs in Tuskar. :)21:01
SpamapSbtw something has been making our keystone purge fail.. so we had ~2.2 million tokens21:01
SpamapS(hoo-ray for me fixing the token list to be fast so we wouldn't detect it... ;)21:01
*** vkozhukalov has quit IRC21:02
*** sdake has quit IRC21:02
rwsujog0: there is a known issue with the ceilometer elements https://bugs.launchpad.net/tripleo/+bug/127260521:03
rwsuthere is some post setup you need to do in order for the services to come up correctly21:03
*** NearlyFunctional has quit IRC21:06
ccrouch1someone should assign that to tsedovic :-)21:06
*** michchap has joined #tripleo21:06
*** jrist has joined #tripleo21:11
*** michchap has quit IRC21:11
*** sdake has joined #tripleo21:17
*** sdake has quit IRC21:19
*** sdake has joined #tripleo21:19
*** sdake has joined #tripleo21:19
*** NearlyFunctional has joined #tripleo21:20
*** CaptTofu has joined #tripleo21:22
*** rbrady-afk is now known as rbrady21:23
*** panda__ has joined #tripleo21:24
rbradySpamapS: I've been running into that token issue for a bit. Because my undercloud vm is smemory constrained, if I let it run overnight it causes mysql to attempt to grab too much memory.  I usually fix it by stopping a few services, starting mysqld, truncating the token table and restarting.21:25
rbradySpamapS: I have tried to use the keystone-manage token_flush method with varying levels of success.21:26
SpamapSrbrady: the element installs a purge very two hours21:27
SpamapSrbrady: a purge cron job that runs every two hours I mean21:28
*** panda has quit IRC21:28
SpamapSrbrady: and it does it "gently" .. 500 rows at a time..21:28
SpamapSrbrady: but I think the cron job may be broken21:28
*** jcooley_ has quit IRC21:31
dprinceslagle: are you running F20 yet? On your seed hosts too?21:33
rbradySpamapS: I will confirm that for you tonight and file a bug21:35
*** mestery has joined #tripleo21:39
*** mestery has quit IRC21:40
*** mestery has joined #tripleo21:40
SpamapS/usr/local/bin/cleanup-keystone-tokens.sh: 4: /usr/local/bin/cleanup-keystone-tokens.sh: os-config-applier: not found21:40
SpamapSold script21:41
*** panda has joined #tripleo21:42
*** dprince has quit IRC21:45
*** panda__ has quit IRC21:45
*** jdob has quit IRC21:53
*** sdake has quit IRC21:55
*** panda has quit IRC21:56
rbradyhey SpamapS: about my question friday, "How do I access external services from the overcloud".  I'm not sure if I framed that question correctly.  Trying again: I need to be able to run "curl -l google.com" and have it resolve.  What do I need to do to the tripleo network setup to achieve that?21:57
*** hashar has quit IRC21:59
*** jprovazn has quit IRC21:59
lifelessrbrady: from a VM, the control plane, or a hypervisor ?22:00
rbradyfrom a VM running on the overcloud22:01
*** lblanchard has quit IRC22:01
rbradyovercloud blockstorage022:01
*** markmc has quit IRC22:01
lifelessrbrady: a block storage node isn't a VM22:01
lifelessrbrady: given you're probably working in a devtest emulated-baremetal envrionment, let me rephrase.22:02
lifelessrbrady: 'a VM created by nova boot against the overcloud' or 'part of the overcloud stack deployed by the undercloud' ?22:02
rbrady'part of the overcloud stack deployed by the undercloud'22:02
lifelessok22:03
lifelessso, by design, you can't22:03
lifelessonly the nodes that /need/ external connectivity - API servers and network routers - are setup to have it.22:03
*** UtahDave has quit IRC22:03
lifelessthat said, we may have bugs that stop servers that do need external connectivity from being fully able to use that22:03
*** e0ne has quit IRC22:04
rbradylifeless: I need it for cinder to reach NFS share using cinder's NFS volume driver.22:04
lifelesssuch as bug  1196938 and bug 1211510 and bug 121441522:04
lifelessrbrady: I don't follow22:04
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: keystone package install support (openSUSE)  https://review.openstack.org/7112222:05
rbradylifeless: say you have a blockstorage node deployed as overcloud from undercloud.  cinder-volume service can use different backends.  Thos backends are established servers/devices22:06
rbradylifeless: so if you want to have cinder create volumes on the NFS shares, you need to route to them22:06
lifelessrbrady: so you want to use an externally deployed storage backend?22:07
rbradylifeless: yes22:07
lifelessrbrady: they should be deployed in the same backend network, behind the firewall though - right ?22:07
lifelessrbrady: I don't see folk having 3par or netapp filers sitting on the public internet22:07
rbradyin the context of me testing this, using devtest, how do I accomplish this short of creating an NFS node on the overcloud?22:08
lifelessrbrady: create an NFS node on the brbm switch22:08
lifelessrbrady: through whatever means you like :)22:09
*** michchap has joined #tripleo22:09
lifelessrbrady: note that we can't test it in CI if its not able to be brought up via devtest...22:09
*** panda__ has joined #tripleo22:10
*** jcooley_ has joined #tripleo22:10
lifelessrbrady: I presume you're working on bringing up cinder?22:13
rbradylifeless: thanks22:13
rbradylifeless: I have cinder working with the fake lvm backing file.  I'd like to flesh out what it takes to do real lvm and other backends.  See what needs to change in the element.22:14
lifelessrbrady: perhaps glusterfs :)22:15
lifelessrbrady: cinder on top of NFS is IMO a reallllllllllllllllllllllllllly bad idea22:15
rbradylifeless: I've been looking here: http://www.openstack.org/blog/2013/11/openstack-user-survey-statistics-november-2013/22:15
*** sdake has joined #tripleo22:16
*** jcooley_ has quit IRC22:16
rbradylifeless: which shows LVM, Ceph, Other...with top "other" is NFS22:16
*** jcooley_ has joined #tripleo22:17
lifelessthose images are terribly hard to read22:17
rbradylifeless: tell me about it.  and they're MS Powerpoint specific.22:18
ccrouch1+1, they are actually scaled *larger* on that page22:18
SpamapSlifeless: I think rbrady is saying he has an NFS server that is not on the same L2 segment as the cinder volume server.22:18
SpamapSoh dear22:18
SpamapSscrollback strikes again22:18
lifelessisn't rbrady ronelle ?22:18
SpamapSignore me22:18
SpamapSshe?22:18
hewbroccano that would be rlandy22:18
lifelessah,ack.22:19
SpamapSEither way.. DOH22:19
lifelessGET BETTER NICKS PEOPEL22:19
ccrouch1ha22:19
lifelesssays the person with an opaque nick22:19
rbradylifeless: not ronelle22:19
ccrouch1thats rich coming from lifeless :-)22:19
SpamapSSpamapS is so clearly masculine. :)22:19
lifelessccrouch1: bet you to it :P22:19
rbradylifeless: Ryan Brady22:19
ccrouch1:-)22:19
lifelessrbrady: one thing you can do is set your real name field in your client22:20
lifelessanyhow22:20
lifelessrbrady: so, NFS seems to be popular.22:20
SpamapSrbrady: I attended a talk at InkTank/Dreamhost about how DreamCompute does CEPH backed volumes..22:20
lifelessrbrady: I suggest that to support it with testing we'll need a NFS server in the environment; I'd put that on the control node for devtest stories22:20
SpamapSrbrady: the compute nodes and cinder nodes have a dedicated L2 network for that.22:20
lifelessrbrady: but it should be opt-in, as zomg not a file system + block level semantics terrifies me22:21
*** rbrady is now known as Ryan_Brady_Supre22:21
*** Ryan_Brady_Supre is now known as Ryan_Brady22:21
lifelessRyan_Brady_Supre: oh, I didn't mean change your nick.22:21
lifelessRyan_Brady: there is a separate field which /whois <nick> shows22:21
lifelessif you /whois me, in the ircname field, you can see who I am.22:22
lifelessyours says 'purple'22:22
lifelessrbray as a nick was just fine22:22
greghaynesIm not sure you can change that without reconnecting :)22:22
lifelessrbrady22:22
lifelessgreghaynes: you can change the config though :P22:22
ccrouch1purple is his pseudonym22:22
Ryan_Bradylifeless: i probably just need to use a better client22:22
greghaynesYep22:22
lifelessRyan_Brady: what client are you using ?22:23
SpamapSWe could just use a gender neutral pronoun.22:23
*** CaptTofu has quit IRC22:23
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: glance package install support (openSUSE)  https://review.openstack.org/7112622:23
Ryan_Bradylifeless: pidgin22:23
hewbroccalifeless: are you disparaging the Notworking File System?22:23
lifelesshewbrocca: I am22:23
lifelesshewbrocca: fav hobby22:23
hewbroccait's a depressingly easy target22:23
lifelessRyan_Brady: it has the setting somewhere, 'real name' or something.22:23
lifelesshewbrocca: aye22:23
ccrouch1lifeless: i have slapping NFS on the over cloud control node is the easy option for testing22:23
*** mestery has quit IRC22:24
lifelessccrouch1: yah22:24
ccrouch1it doesnt help with the "I've got an existing NFS setup somewhere" option which I think ryan was getting at22:24
ccrouch1but i guess thats what your brbm comment was for22:24
lifelessccrouch1: yeah22:25
Ryan_Bradythere is just a lot of places with networking stuff and my iptables-fu didn't work well.  I know now to look at it from the brbm switch22:25
*** panda___ has joined #tripleo22:25
lifelessccrouch1: so the issue with a real NFS setup and brbm test environments is that we use TEST_NET_222:26
lifelesswhich is nonroutable22:26
lifelessyou can only get out via NAT22:26
*** markmcclain has quit IRC22:26
StevenKlifeless: Holy crap, re: -nova backlog22:26
lifelessRyan_Brady: so yeah, part of it will be the DNS setup for you22:26
lifelessRyan_Brady: but also you'll be messing with your NAT on virbr0 if you do want an external server, but...22:27
lifelessRyan_Brady: using a NFS server on the overcloud control plane server will be about a 1000 times easier22:27
*** slagle has joined #tripleo22:28
Ryan_Bradylifeless: thanks22:28
ccrouch1Ryan_Brady: so i guess you can divide the problem into two: Getting NFS+Cinder working through Heat  and B getting it work when not deployed to the overcloud22:28
lifelessI have a q though22:28
*** panda__ has quit IRC22:28
lifelesswhy do NFS at all?22:29
lifelessif we're deploying and scaling everything22:29
lifelessand ceph/glusterfs/sheepdog are better options22:29
lifelessor perhaps s/'at all'/'right now'/22:29
ccrouch1really lower hanging fruit. This should all be working already iiuc22:30
Ryan_Bradylifeless, ccrouch1: I have to step out for a bit to go to a meeting.22:30
lifelessciao22:30
*** Ryan_Brady is now known as rbrady-backlater22:30
lifelessccrouch1: I'm not meaning to push back, clearly eventually we want to support what users want22:30
ccrouch1ha22:30
lifelessccrouch1: just given time frames and so on I would have expected glusterfs in the deployment to be first off the rank from RH22:30
*** michchap has quit IRC22:32
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Package install support for python-ironicclient  https://review.openstack.org/7112722:32
ccrouch1we'll definitely get to glusterfs. The thinking was NFS would be a quick win. simple to setup, configure and test22:32
ccrouch1used by a tonne of people already22:32
ccrouch1it doesnt indicate an institutional move away from gluster in preference for NFS :-)22:33
*** markmcclain has joined #tripleo22:33
lifelessccrouch1: My experience with NFS must be vastly different to yours :)22:35
SpamapSsame here22:36
SpamapSMo NFS Mo Problems is what we always used to say up in the DC22:36
* SpamapS </street>22:36
*** matty_dubs is now known as matty_dubs|gone22:36
ccrouch1well someone is using it, per the survey22:37
greghaynesNot For S(peed, ecurity, tability) is what we would say depending on our mood :p22:37
lifelessccrouch1: I know :> - but the largest fraction AFAICT is amongst small deploys22:38
StevenKI tend to expand it to "No File Security"22:38
*** gcha has quit IRC22:39
SpamapSNFSv4 has some kind of crypto IIRC22:42
SpamapSbut you have to bring kerberos to make that work.. and.. there went simple..22:42
greghaynesYep, one does not simply gssapi22:43
lifelessbut its a simple api22:43
lifelessit says so right there in the name22:43
greghayneshaha22:43
hewbroccaIn some language, gssapi *must* mean "Now you have two problems"22:43
lifelesslike LDAP is lightweight.22:43
ccrouch1lifeless: that could very well be the case. TBH I'm more concerned about22:43
ccrouch1(4:27:19 PM) lifeless: Ryan_Brady: using a NFS server on the overcloud control plane server will be about a 1000 times easier22:43
ccrouch1if there is little chance of getting cinder to talk to an NFS server external to the overcloud then, that reduces its value massively imho22:43
lifelessccrouch1: in devtest22:44
lifelessccrouch1: you snipped the context.22:44
lifelessccrouch1: devtest uses an unroutable network so that we don't have to have folk do network allocations22:44
ccrouch1ah ok, i see22:44
ccrouch1i agree on the overcloud makes perfect sense for devtest22:45
ccrouch1i was meaning in the non-devtest context22:45
*** jdurgin has joined #tripleo22:46
jog0lifeless: so ceph in ad tripleo, and immediate plans? if so I dragged jdurgin in here22:47
jog0is it on the table for the this MVP22:47
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Fix python-ceilometerclient package install  https://review.openstack.org/7113622:48
*** michchap has joined #tripleo22:49
lifelessjog0: not something I'm pushing people towards, HA is the current thing22:50
lifelessHA of enough services to take the current offering (keystone/glance/nova/neutron) through a deploy without interrupting already deployed VM's22:51
lifelessstep after that would be accepting nova boot etc etc while doing that22:51
SpamapSheh22:51
SpamapSceph can back glance, though I suspect we'd probably rather do swift since we already have it there22:52
jog0lifeless: ahh so no cinder, right now and we don't want to have a shared filesystem for ephemeral?22:52
jog0which means ceph isn't on the table at the moment22:52
lifelessjog0: nobody wants a shared file system for ephemeral, those that do don't know what they want :)22:52
lifelessjog0: right22:52
jog0lifeless: ack, wasn't sure if we had cinder or not yet.22:53
lifelessjog0: we do but its not in the tested set of stuff22:53
jog0jdurgin: I guess we will have to revist later22:53
jog0lifeless: to me thats the equivalent of we don't do it.22:53
lifelessjog0: me too :)22:54
jdurginok, thanks. you've certainly got plenty to do as it is :)22:54
lifelessjdurgin: if you want to hack on ceph support by all means, it just will be a lower priority for folk pushing the main wheelbarrow22:54
SpamapSwe have talked about ceph and/or gluster for backing cinder though22:54
lifelessright, ideal use case22:54
jog0so for this MVP we won't support volumes. which sounds reasonable  to me22:54
jog0lifeless: I brought jdurgin over because I figured once we are ready to dive into cinder HA he can help out a little22:55
jog0but I was premature22:55
SpamapSok, I installed an MTA on cd-undercloud22:55
*** weshay has quit IRC22:55
SpamapSCRON was spewing stuff to /dev/null22:55
SpamapSlifeless: ideas on who should get root mail?22:55
jdurgincould certainly be interesting in the coming cycles, but it will probably need a new cinder driver for it anyway22:56
jog0jdurgin: oh? do tell22:56
SpamapSwe can leave it on the box..  I don't mind that..22:56
jog0I thought ceph had good cinder support22:56
jdurginbaremetal will probably want to use kernel rbd22:56
jdurginall the management stuff can be the same, but attach/detach will be different22:56
jog0jdurgin: oh rightm, i was just thinking overcloud ceph22:56
jog0cinder22:56
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Nova package install support (openSUSE)  https://review.openstack.org/7113922:57
SpamapSjdurgin: how do vms work right now?22:58
SpamapSjdurgin: kvm rbd?22:58
jdurginyup22:58
SpamapSok, well we can still do ceph for the overcloud22:58
lifelessjdurgin: jog0: cool22:59
SpamapSwhich is probably all we need it for actually22:59
lifelessSpamapS: you should22:59
lifelessjdurgin: I'm not sure undercloud will want ceph at all22:59
lifelessjdurgin: simpler we keep that layer the better I think22:59
SpamapSlifeless: yeah I am sending rootmail to me for now. Just thinking more of "the team" than "just me" :-P22:59
jdurginlifeless: yeah, I agree it makes more sense in overcloud, but if you've already running ceph, maybe you'd rather have fewer disks on the undercloud compute nodes23:01
SpamapSjdurgin: Thats one of those philosophical questions, isn't it? ;)23:02
jdurgin:)23:02
lifelessjdurgin: how many fewer than 1 can you get ?23:04
lifelessunbound variable on line 46 of os-svc-enable23:05
lifeless74-cinder-volume blew up23:05
SpamapSwoot, the 'data' fix merged23:12
*** panda___ has quit IRC23:14
*** mestery has joined #tripleo23:17
*** mestery has quit IRC23:17
*** mestery has joined #tripleo23:17
*** jayg is now known as jayg|g0n323:18
*** markmcclain has quit IRC23:25
*** panda___ has joined #tripleo23:28
SpamapSoh another interesting fact23:29
SpamapSour pypi local mirror has been broken since Dec 1623:29
SpamapSwhen we installed new setuptools in it as root23:29
lifelessSpamapS: data?23:30
SpamapSlifeless: the image cache thing23:30
lifelesshttps://review.openstack.org/#/c/69060/ has merged23:30
lifelessSpamapS: image cache thing?23:30
lifelesswe can run trunk nova again23:30
SpamapSlifeless: right23:35
SpamapSlifeless: so I'm purging keystone tokens..23:35
lifelessStevenK: so I60a108ef1f0e46ad4da94d2a730426e255011476 and Icff1304fc816acc843f8962727aef8bbbc7bbaa3 should be much more amenable to hacking on now23:36
SpamapSlifeless: the cron job is still not working right.. I don't know why just yet.. so I'm just running the cleanup script in a terminal23:36
SpamapSlifeless: we were at 2.2 million earlier.. down to 1.8 million now23:36
lifelessStevenK: I've rebased and pushed, you should get a clean test run (errors are genuine) now23:37
SpamapSlifeless: seems that with the improved efficiency of keystones sql backend we only get slow in the 2 million token range. :-P23:37
lifelessSpamapS: e.g. 10 minutes of use ? :P23:37
SpamapSsrrsly23:37
SpamapSlifeless: oh, and cron mail also revealed this gem:23:38
SpamapSERROR: (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`stack_lock`, CONSTRAINT `stack_lock_ibfk_1`23:38
SpamapSFOREIGN KEY (`stack_id`) REFERENCES `stack` (`id`))') 'DELETE FROM stack WHERE stack.id = %s' ('1002e2fb-c558-481a-a357-98c360c8ab15',)23:38
SpamapShttps://bugs.launchpad.net/heat/+bug/127633623:38
mordredyay! foreign keys are awesome23:39
lifelessStevenK: and yes, holy fuck w.r.t. nova backlog23:39
SpamapSawesome like Soundwave's tapes that turn into murderous dogs and birds? :-P23:39
StevenKlifeless: That SIGUSR1 patch needs to land, but I understand the concerns around it -- sounds like some of the issues we addressed in oopstools23:41
* jog0 needs to have a one man nova review party23:41
*** bauzas has quit IRC23:42
lifelessso htf is os-svc-enable failing23:47
lifelessdib-run-parts Tue Feb 4 22:55:24 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/74-cinder-volume23:48
lifelessUsage: os-svc-restart -n SERVICENAME23:48
lifeless23:48
lifeless  -h             Show help and exit23:48
lifeless  -n SERVICENAME Name of job/service file.23:48
lifeless23:48
lifeless/usr/local/bin/os-svc-restart: line 46: $1: unbound variable23:48
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Add a script to deploy the ci-overcloud.  https://review.openstack.org/7115423:50
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: os-svc-enable should error when no svcname given.  https://review.openstack.org/7115523:50
*** markmcclain has joined #tripleo23:54
lifelessoh, os-svc-restart...23:54
ccrouch1i was *just* about to mention that :-)23:54
ccrouch1but thought I must be missing something23:54
ccrouch1https://github.com/openstack/tripleo-image-elements/blob/master/elements/os-svc-install/bin/os-svc-restart#L46 makes much more sense23:55
*** weshay has joined #tripleo23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!