Tuesday, 2014-03-11

*** lblanchard has joined #tripleo00:15
*** weshay has joined #tripleo00:16
*** blamar has quit IRC00:17
*** weshay has quit IRC00:21
*** matsuhashi has joined #tripleo00:21
SpamapSlifeless: nova failing seems like it may have been a race we haven't identified yet... re-doing the overcloud owrked..00:22
SpamapSdib-run-parts Tue Mar 11 00:21:12 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/74-swift-proxy00:22
SpamapSstop: Unknown instance:00:22
SpamapSstart: Job failed to start00:22
SpamapSexcept it didn't00:22
SpamapSError: [swift-hash]: both swift_hash_path_suffix and swift_hash_path_prefix are missing from /etc/swift/swift.conf00:22
giulivoguys anyone around using devtest on f20?00:23
giulivoI think I'm facing a problem with the seed where "nova db sync" fails00:23
SpamapSswift.hash is ""00:23
giulivowith "OperationalError: (OperationalError) (2006, 'MySQL server has gone away') None None"00:23
giulivoanyone has seen that before?00:24
SpamapSgiulivo: check your mysql logs.. that would be a disconnect of some kind..00:24
SpamapSgiulivo: either mysqld crashed or something else00:24
giulivoSpamapS, yeah I was checking that and it is actually running but yes00:24
giulivosometimes I get disconnected during long queries00:24
giulivoany idea what could be going on?00:24
SpamapSgiulivo: could be a wait_timeout too long00:24
giulivoI basically reproduced this three times with three attempts to devtest_seed.sh00:25
SpamapSthere are a few timeouts :p00:25
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Allow the testenv client to dictate the worker timeout  https://review.openstack.org/7617100:25
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Set the worker timeout  https://review.openstack.org/7617200:25
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Kill commands that outlive test env  https://review.openstack.org/7540200:25
SpamapSlifeless: ok I just forgot to source tripleo-overcloud-passwords before deploying...00:27
giulivoSpamapS, just to make sure, why too long?00:28
SpamapS... this is where retry would come in really handy :-/00:28
SpamapSgiulivo: strike that. Reverse it. ;)00:28
giulivook :)00:28
giulivoSpamapS, but it is set to 28800 so ... :(00:29
SpamapSgiulivo: ok so that's one thing down :-P00:30
giulivoSpamapS, interestingly I got the "gone down" even in the interactive shell while doing "show variables"00:31
SpamapSlifeless: side note, this _might_ be causing havoc: https://bugs.launchpad.net/keystone/+bug/128993500:33
SpamapSgiulivo: yeah are you positive your mysqld isn't dying and being resurrected by systemd or mysqld_safe ?00:34
giulivoSpamapS, nah I'm on your side00:34
*** blamar has joined #tripleo00:34
giulivoit is probably crashing00:34
*** CaptTofu has quit IRC00:39
*** lucasagomes has quit IRC00:53
*** spzala has quit IRC00:55
derekhSpamapS: are you looking into random overcloud jobs taking longer then expected ?01:04
derekhSpamapS: if so, I just hit the same thing here while running other tests01:05
derekhSpamapS: In my case I *think* I know the problem01:05
derekhSpamapS: we're leaving baremetal nodes lieing around running, which may have IP's conflicting with the new testenv being setup01:06
SpamapSderekh: ew01:06
SpamapSderekh: don't we use cleanup-env or something like that?01:07
derekhSpamapS: nope, we don't have the access needed for cleanup-env01:07
derekhSpamapS: the domains are setup once when the testenv host comes up01:08
SpamapSright first boot thingy01:09
SpamapSderekh: so, we're racing with still-shutting-down hosts, or forgetting to shut down hosts?01:10
derekhSpamapS: we were only shutting down the seed vm, not the others at all01:10
SpamapSderekh: that does sound suspicious01:11
*** cwolferh_ has quit IRC01:13
derekhSpamapS: gonna push it so we can see what happens01:14
*** rbrady-afk is now known as rbrady01:14
openstackgerritA change was merged to openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/7567001:14
SpamapSderekh: wheee01:14
rbradywoot!  my patch finally passed CI!01:14
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Destroy all ci testenv domains  https://review.openstack.org/7950201:14
SpamapSrbrady: feels like the heavens opening up and shining on you doesn't it?01:15
lifelessSpamapS: https://review.openstack.org/#/c/79480/ +A please01:15
lifelessSpamapS: will let me test the faster CI path01:15
rbradySpamapS: yes!01:15
*** CaptTofu has joined #tripleo01:15
derekhthe more I think about it the more I think this would cause all kinds of problems....01:16
lifelessderekh: what would ?01:16
lifelessderekh: 79502? nova bm / ironic should take care of it already01:16
derekhlifeless: it would only assert state of the node its about to try and boot, wouldn't it?01:17
SpamapSshouldn't we be carefully doing a 'stack-delete overcloud' and then 'stack-delete undercloud' ?01:17
lifelessderekh: nope01:17
lifelessderekh: anything registered with it01:17
openstackgerritA change was merged to openstack/tripleo-incubator: Give the seed and undercloud more IPs to work with  https://review.openstack.org/7948001:18
SpamapSThat exercises a ton of really poorly exercised paths in OpenStack and it would turn all the testenv machines off01:18
SpamapSexcept it might not turn off CREATE_FAILED instances.. so hm01:18
derekhlifeless: ok, well something is wrong with nove-bm then, I've been watching the nodes not get destroyed01:18
SpamapSIf we're just ripping down the undercloud and overcloud what is powering off those machines?01:19
derekhSpamapS: but we're not ripping down the undercloud or overcloud, all we do is "virsh destroy seed_x"01:20
tchaypodestroy == power down01:20
*** retr0h has joined #tripleo01:20
*** lblanchard has quit IRC01:20
derekhtchaypo: yup, my point was we only do it for the seed, the rest are left running01:21
tchaypoyeah, I'm not sure what SpamapS' was asking though01:21
lifelessderekh: ok, so - its possible that we have state asserting turned off01:22
lifelessderekh: or maybe its an ironic only feature (but I didn't think so)01:22
lifelessdevananda: ^01:22
lifelessdevananda: does nova-bm force off nodes without an instance on them ?01:22
lifelessdevananda: as we were discussing for Ironic the other day ?01:23
*** rpodolyaka has quit IRC01:23
*** blamar has quit IRC01:23
* SpamapS wonders if this is finally the thing that will drive us onto ironic ;)01:23
devanandalifeless: ironic certainly does01:23
derekhlifeless: even if it did, the seed only knows about one of the nodes, so would only shut that one down01:23
devanandalifeless: lemme take a look at current nova-bm code01:24
lifelessderekh: yes, but that one is always the undercloud, no ?01:24
openstackgerritA change was merged to openstack/tripleo-image-elements: Adds nfs-server element  https://review.openstack.org/7471201:24
devanandalifeless: is the context here a newly enrolled node (no prior instance) or something else?01:25
derekhlifeless: ya, thats probably true01:25
derekhlifeless: will rerun my test01:25
*** rpodolyaka has joined #tripleo01:25
lifelessdevananda: previous test VM01:25
lifelessdevananda: so we deploy new seed, baremetal_N vm may still be running01:26
devanandalifeless: and baremetal_N is enrolled with new seed, I presume01:26
devanandaso from seed's POV, that node had no prior instance01:26
devanandalifeless: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L510801:27
devanandaignore that, still digging01:29
devanandaderekh: do you get this log message? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L509301:29
jogolifeless: ((message:"Build timed out (after" AND  message:"minutes). Marking the build as failed.") OR (message:"./safe-devstack-vm-gate-wrap.sh: line" AND  message:"Killed timeout -s 9 ${DEVSTACK_GATE_TIMEOUT}m $BASE/new/devstack-gate/devstack-vm-gate.sh"))AND filename:"console.html" AND build_queue:"check-tripleo"01:30
derekhdevananda: new seed booting up, will know in a few minutes01:30
jogobug Bug 125868201:31
jogoseeing a lot of hits for tripleo, which makes sense01:31
giulivoSpamapS, in case you're still around for some help01:31
giulivobasically I'm installing using qpidd and qpid.username qpid.password are not defined in os-collect-config/heat_local.json01:32
giulivoso it is failing and executing again and again every 30secs (restarting mysql too)01:32
giulivonow, in the seed, I see heat_local.json is recreated every 30sec with the os-collect-config run01:33
giulivowhere does the data comes from?01:33
derekhdevananda: no sign of that log message yet01:35
devanandaderekh: anything interesting in n-cpu?01:35
derekhdevananda: lifeless none of the vm were killed on register bm nodes, only one restarted on heat stack-create overcloud (the rest are left untouched running)01:36
derekhsecond vm restarted now01:37
derekhdevananda: nothing much, only logging on warn level http://paste.openstack.org/show/73092/01:38
devanandaderekh: ok. so a VM restarted ~1 minute later?01:39
derekhand now 3rd01:39
devanandaderekh: not stopped -- it restarted?01:39
*** nosnos has joined #tripleo01:40
derekhdevananda: actually probably destory/start the virsh id changed01:40
derekhso here is what I see , seed gets 8 bm nodes registered (all running VM's), all are left running, the I do a "heat stack-create overcloud", one of the vm is destroyed/started (based on virsh id changing), then when that nodes becomes active, the second is destroted/started and the 3rd01:42
derekhso we need something to kill those , weither its nova-bm or something like https://review.openstack.org/#/c/79502/101:44
lifelessgiulivo: is this your seed?01:45
lifelessgiulivo: or undercloud/overcloud?01:45
giulivolifeless, hi thanks for pinging back, my seed yes01:45
lifelessgiulivo: are you following devtest?I believe we have examples in there for changing config.json to use qpidd01:46
giulivoI'm following devtest but I might have missed bits :(01:46
devanandaderekh: what does 'nova list' and 'nova baremetal-node-list' show?01:47
devanandaderekh: and: mysql nova -e "select count(*) from instances"01:47
giulivolifeless, when you say config.json you mean $TE_DATAFILE ?01:47
lifelessgiulivo: no, the seed stack config file01:48
lifelessgiulivo: which replaces heat since yuo can't deploy the seed with heat01:48
*** tserong has quit IRC01:48
derekhdevananda: http://paste.openstack.org/show/73093/01:49
devanandaderekh: fwiw, spawn should be rebooting them anyway: https://github.com/openstack/nova/blob/master/nova/virt/baremetal/driver.py#L29401:49
devanandaderekh: what exactly is the problem?01:49
derekhdevananda: some of the VM are left behind (running) from previous ci runs, so unless they are all killed they will have existing IP's that conflict with a new ci run01:50
devanandayou should kill them before tearing down the old seed01:51
devanandanova.compute.manager's sync_power_state logic only works on instances01:51
devanandathose left over VMs don't have an instance uuid in the new seed, therefor they aren't synced01:52
giulivolifeless, so that's the config.json created by the seed-stack-config element, correct?01:52
*** rpodolyaka has quit IRC01:52
devanandaand the bm virt driver can't run its own periodic tasks01:52
*** rpodolyaka has joined #tripleo01:52
devanandalifeless: options seem to be: kill VMs before shutdown, add a new periodic task or init hook to nova.compute.manager and push the power state down to the driver, or use ironic01:53
devanandaor an external script01:53
devanandathat might work too01:54
devanandarun just after you enroll the VMs and before heat stack-create01:54
devanandaoh, and add a new API hook for nova-bm to expose a set-power-state01:54
devanandai forgot nova doesn't have that01:55
lifelessderekh: I'll +2 your patch given that, if you could  please add a note that we don't need it in Ironic ? And perhaps a check for the variable devananda has set for Ironic use...01:56
*** tserong has joined #tripleo01:56
*** tserong has joined #tripleo01:56
derekhlifeless: will do01:57
giulivolifeless, hacking cfn-init-data worked, maybe we should just add the qpid section here https://github.com/openstack/tripleo-image-elements/blob/master/elements/seed-stack-config/config.json ?01:59
derekhdevananda: what can I check to see if ironic is being used? I only see IRONIC_PASSWORD in tripleo-incubator , suppose I could check for ironic in the element list02:02
devanandalifeless: fwiw, https://github.com/openstack/ironic/blob/master/ironic/conductor/manager.py#L9502:02
* StevenK has a sad about people linking github urls02:02
*** Qiming has joined #tripleo02:02
*** antonym has quit IRC02:03
lifelessderekh: devananda has a patch set up for incubator02:03
derekhlifeless: k02:03
lifelessderekh: I'm suggesting using the same variable name that uses02:03
*** antonym has joined #tripleo02:03
lifelessgiulivo: we can add qpid just-in-time rather than statically, was more my point02:03
*** Kim-Chi-San has quit IRC02:04
giulivolifeless, I'm not there yet, don't know the process very well, need guidance or if you can do it I would check the submission02:04
*** spzala has joined #tripleo02:04
*** Kim-Chi-San has joined #tripleo02:04
devanandaStevenK: SpamapS: what's the status of the ironic undercloud work?02:05
lifelessgiulivo: I've no idea about qpidd :)02:06
StevenKdevananda: The JSON nodes work has landed again, I'll be pushing up a new patchset for toci after lunch02:06
devanandaStevenK: awesome02:06
giulivolifeless, ok so basically this is working fine https://github.com/openstack/tripleo-image-elements/blob/master/elements/qpidd/os-refresh-config/post-configure.d/73-qpidd but in my seed failed because in cfn-init-data I did not have qpid.username and qpid.password02:07
devanandaStevenK: did the bits to start using the nova-ironic driver from our tree make it anywhere yet?02:07
giulivothey aren't defined in the seed-stack-config/config.json02:07
StevenKdevananda: I don't think so, I think the patch in incubator needs a rework now that nova-ironic is in ironic02:07
giulivobut I don't know how those could be put in place just-in-time02:07
lifelessgiulivo: righto, so it looks like the rh folk didn't get through to incubator seed setup with their qpid work02:07
StevenKdevananda: I can recall words to that effect in the stream of e-mail I injested this morning.02:08
devanandaStevenK: heh :)02:08
lifelessdevananda: I -1'd the patch yesterday02:08
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Destroy all ci testenv domains  https://review.openstack.org/7950202:08
lifelessdevananda: needs fixing02:08
lifelessderekh: reviewed02:09
*** rpodolyaka has quit IRC02:10
derekhlifeless: replied02:11
lifelessderekh: replied :)02:18
*** lazy_prince is now known as killer_prince02:18
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Destroy all testenv domains  https://review.openstack.org/7950202:24
lifelessI think gear has falled over again02:24
derekhlifeless: updated02:25
*** spzala has quit IRC02:26
derekhlifeless: possibly, I don't think we've fixed the problem yet, this has to make it to testenv hosts https://review.openstack.org/#/c/76546/02:26
lifelessbouncing it02:26
derekhlifeless: and this will be needed to https://review.openstack.org/#/c/76588/ to avoid a problem we would hit at a later stage02:27
lifelessderekh: wait, what time is it for you ? :)02:27
derekhlifeless: 2:30 AM02:27
derekhlifeless: which makes it sleep time :-)02:27
lifelessgeard restarted02:27
lifelessderekh: fairly sure we applied https://review.openstack.org/#/c/76546/02:28
derekhlifeless: nope, doesn;t look like it on testenv-testenv3-mo3qqxo7373c02:29
lifelessderekh: ack, I've just pinged infra about the gear fix02:29
derekhlifeless: I gotta go sleep but can do that tomorrow if ya want (but we should probably wait for the fix to gear)02:29
lifelessderekh: would be good to test (and deploy) stevenks worker change to add nodes to the JSON02:30
lifelessderekh: cleans up a bunch of stuff02:30
*** rpodolyaka has joined #tripleo02:30
lifelessdevananda: whats the status of the Ironic nova driver, usability wise ?02:31
derekhlifeless: ok, I'll test a new local test env tomorrow and if alls good with push to production02:31
derekhif the gear change is in that is, would also be nice to get https://review.openstack.org/#/c/76171/3 in02:32
derekhok, night, really going now02:32
*** derekh has quit IRC02:33
devanandalifeless: start, stop, you know, the basics02:35
devanandalifeless: console and volume aren't implemented yet. looks like rebuild isn't there yet either02:36
killer_princeslagle: thanks for reviewing https://review.openstack.org/#/c/79289/2 and comments :)02:36
lifelessdevananda: so we need boot/rebuild with --preserve-ephemeral and then we should be able to put a CI job up fo rit02:38
killer_princeslagle: i am not sure if it can be done via deploy images as it needs support from images which are being deployed.. as in greub should be present in image.. but i will give it a try and will come back on it02:39
lifelessdevananda: (and I'll be thrilled at tht point :))02:40
killer_princeslagle: could you also review https://review.openstack.org/#/c/79286/ pls..02:40
StevenKlifeless: That needs fixing02:43
lifelessStevenK: that?02:43
StevenKlifeless: The toci change to populates nodes:[]02:43
lifelessStevenK: right02:43
*** killer_prince is now known as lazy_prince02:44
openstackgerritGregory Haynes proposed a change to openstack/diskimage-builder: Add apache2 mod_wsgi pkg map for suse  https://review.openstack.org/7728502:44
*** lazy_prince is now known as killer_prince02:45
StevenKlifeless: I made the toci change before we both took an axe to create-nodes and passed the JSON in.02:45
*** rbrady is now known as rbrady-afk02:46
lifelessSpamapS: what time is it for you atm ?02:50
StevenKAKA, yay, what the hell has DST done02:51
StevenKpython -c 'print open("/root/.ssh/test_env_rsa").read().replace("\n", "\\n")'02:52
StevenKWhy can't we use tr like normal people?02:53
lifelesssure, do so02:53
openstackgerrityangxurong proposed a change to openstack/tuskar: Pin iso8601 logging to WARN  https://review.openstack.org/7951602:55
openstackgerritlifeless proposed a change to openstack-infra/tripleo-ci: Run overcloud deploy tests against the seed.  https://review.openstack.org/7922002:56
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: User docs should tell the user to use ./  https://review.openstack.org/7951702:56
*** rpodolyaka has quit IRC02:58
* StevenK grumbles at jq not helpfully writing the data back02:59
tchaypoStevenK: win02:59
StevenKtchaypo: Do you know where your luggage is?03:01
StevenKlifeless: I'd like to test this toci change. Just git review it and wait for jenkins?03:06
*** vkozhukalov has quit IRC03:09
lifelessStevenK: which toci script are you changing?03:09
tchaypomy luggage is on my bed03:10
lifelessStevenK: test that by deploying a test environment (script fo rthat is in tie/elements/tripleo-cd/03:10
lifelessStevenK: and then using it03:10
lifelessStevenK: or you can run it by hand locally03:11
StevenKtchaypo: When did it get there?03:11
tchaypoabout an hour ago03:12
StevenKtchaypo: Did you get the TSA luckydip?03:13
StevenKlifeless: Just running ensure-test-env locally doesn't seem like it will work, TBH ...03:14
*** CaptTofu has quit IRC03:14
tchaypoI didn't get a card03:15
tchaypoand I haven't noticed anything missing03:15
*** sballe_ has joined #tripleo03:22
*** sballe has quit IRC03:24
*** tzumainn has quit IRC03:36
*** matsuhashi has quit IRC03:41
tchaypoI'd love a review/approval on https://review.openstack.org/#/c/77048/ (adding a note to devtest.sh to say we test on fedora 20 and ubuntu trusty) if anyone has time03:43
StevenKI thought we were running on Precise03:45
tchaypowe spin up precise nodes in the overclould (and maybe the undercloud?)03:50
tchaypobut devtest.sh is only tested on trusty afaik03:50
tchaypoand F20, but not precise03:51
tchaypoI believe (but haven't checked myself) that precise has an old version of libvirt that lacks features we use03:51
StevenKIt might be Saucy03:52
greghaynesIve also been told its OVS support is lacking03:52
greghayneslots of heresay about devtest and precise :p03:52
tchaypobut in any case, I've tried to make no claims about what *works*, only what *has CI testing*03:53
greghaynesugh, for some reason now I get No valid host was found in nova when making undercloud when I devtest03:54
StevenKtchaypo: Trusty isn't released yet, which is why I'm questioning03:55
giulivolifeless, this isn't working for me https://github.com/openstack/tripleo-image-elements/blob/master/elements/nova-baremetal/os-apply-config/mnt/state/var/lib/nova/virtual-power-key03:59
giulivothe keyfile is correctly populated but because of wrong permissions it can't be used, so virtual power fails03:59
giulivoever faced it? ideas where the chmod/chown should be placed?04:00
giulivolifeless, os-config-refresh configure.d ?04:01
*** ccrouch1 has quit IRC04:05
*** ccrouch has joined #tripleo04:05
*** victor_lowther_ has quit IRC04:05
greghayneseek, getting "unrecognized partition table type\nNo partitions found\nWarning: given size (21504) exceeds max allowable size (21503)\n\nsfdisk: bad input\n'" in nova-baremetal-deploy-helper04:05
*** ccrouch has quit IRC04:05
*** ccrouch has joined #tripleo04:05
lifelessthe slaves are precise04:06
lifelessthe vm host is saucy I believe04:06
*** victor_lowther_ has joined #tripleo04:06
lifelessthe build images are saucy (the default in dib)04:06
lifelessgiulivo: we use paramiko, the permissions on the key shouldn't matter04:07
* lifeless watches his disk subsystem max out04:10
lifelessraid rebuildreshapes, FTW04:10
*** CaptTofu has joined #tripleo04:10
StevenKlifeless: I always tend to get worried when I see my fileservers disk light constantly on04:13
StevenKUsually it's softraid going "Time for a resync!"04:13
*** CaptTofu has quit IRC04:15
*** killer_prince is now known as lazy_prince04:24
*** sdake has quit IRC04:27
lifelessjust kicked in the final 2 disks from my old server04:27
lifelessdm-0              0.00     0.00    0.40    2.40     1.60     7.20     6.29     6.76 1877.71  902.00 2040.33 357.14 100.0004:27
lifelessasdg               0.00 10777.00    0.00   94.20     0.00 43481.00   923.16     2.96   31.40    0.00   31.40   4.19  39.4404:28
lifelesssdh               0.00 10777.00    0.00   94.20     0.00 43481.00   923.16     3.37   35.78    0.00   35.78   4.37  41.2004:28
*** akuznetsov has joined #tripleo04:33
lifeless      [>....................]  reshape =  2.5% (75434912/2929556480) finish=1144.5min speed=41559K/sec04:34
* StevenK ponders the readable of "sed -e ':a;N;$!ba;s/\n/\\n/g' /root/.ssh/test_env_rsa" versus the python04:34
StevenKlifeless: Sllloow04:35
StevenKlifeless: My resyncs tend to hit 116000K/sec04:35
SpamapSlifeless: we are now UTC-7 FYI04:35
lifelessStevenK: are you running devtest at the same time ?04:35
lifelessSpamapS: what time is it for you now ?04:35
StevenKlifeless: Fair point :-)04:35
SpamapSlifeless: 21:3504:35
lifelessStevenK: also, raid5 or 6 ?04:36
StevenKNeither, 104:36
lifelessStevenK: ahahahaha no comparison then04:36
StevenKI lost a fair bit of data years using softraid 5 when the kernel refused to tell me which drive failed04:36
greghaynesPlayed a bit of raid5 roulette, eh?04:37
StevenKgreghaynes: I knew a drive had failed, since the resync stopped at 7 something %.04:37
StevenKThe kernel didn't log which drive it was, and SMART wasn't clever enough to tell me04:38
greghaynesYep, then you have a nonzero chance of yanking a random drive and it being a correct guess :p04:38
lifelessStevenK: yeah, but raid1 resync is linear reads looking for mismatch04:38
lifelessStevenK: this is raid6 reshape which is moving all the data down 30% on the drives04:39
StevenKlifeless: I ponder moving to RAID 5, since reshape is a thing now04:39
lifelessStevenK: so reading all previous members, writing to /all/ members04:39
giulivoStevenK, anyway, with raid1 you'll have 50% chances to pick the "non working" disk !04:39
StevenKI've had a few drive failures with RAID 1 and it's always told me04:40
StevenKI was using RAID 5 with like kernel 2.204:40
giulivowith raid5 you had what, 66%04:40
giulivoStevenK, only joking about it04:40
lifelessStevenK: buy 4 3TB WD REDs and raid6 em04:40
greghaynesdepends on num of drives04:40
*** matsuhashi has joined #tripleo04:40
giulivogreghaynes+1 :)04:40
lifelessStevenK: then even if you guess wrong you'll be ok :)04:40
StevenKlifeless: I have 2 2TB and 2 3TB04:41
StevenK4.5T on /srv04:41
greghaynesIve just been doing raid0 ssd's for devtest box...04:41
lifelessthese drives are reading 70MBps and writing 45MBps simultaneously04:41
lifelessI'm pretty happy with that04:41
greghaynesIs kind of nice actually04:41
lifelessfor all that they're spinning rust04:41
tchaypoUse zfs, problem solved04:42
StevenKtchaypo: No, then you have two problems04:42
lifelesstchaypo: which problem ?04:42
greghaynes"What could go wrong?"04:42
StevenKgreghaynes: What could *possibly* go wrong!04:42
StevenKJust like "How hard can it be"04:43
tchaypoStevenK's problems with the linux kernel04:43
StevenKtchaypo: zfs didn't exist then04:43
tchaypoeither he'll be using a BSD kernel or he'll be using the FUSE driver, but either way the linux kernel won't be making him sad04:43
StevenKLike I said, kernel *2.2*04:43
tchaypoand this is why last week was valuable04:44
lifelesstchaypo: because it let you get your trolling skills in order?04:44
SpamapSwtf do you all store on these disks?04:44
cody-somervillemy thoughts *exactly*04:44
tchaypoif we hadn't gone to the other side of the world to spend a week together I want have such an accurate mental rendering of StevenK's voice saying "Like I said, kernel *2.2*"04:44
lifelessSpamapS: family DVD and blueray collection04:44
SpamapSI have that too04:45
SpamapSits -> [amazon|netflix|googleplay]04:45
giulivoyou mean04:45
giulivosorry guys, see you enjoy the disks04:45
StevenKSpamapS: Move to .au or .nz, and then say that.04:45
SpamapSZero reason to "buy" content .. lots of reason to actually pay for it and enjoy it at a reasonable price. :)04:45
StevenKI dare you.04:45
lifelessSpamapS: not available, not available, stupidly expensive04:45
greghaynesFair point04:45
SpamapSSo the first two, you all should definitely move away from such a backward country...04:46
SpamapSand what is stupidly expensive?04:46
lifelessSpamapS: seriously, 30NZD for a recent release; buying the DVD / blueray is cheaper04:46
lifelessSpamapS: google play04:46
SpamapSbecause I just wait for movies to go on special and buy them for $8 or so.04:46
*** giulivo has quit IRC04:46
SpamapSI've found Play to be the more reasonable one actually.04:46
lifelessSpamapS: perhaps in the states04:46
SpamapSNetflix has almost nothing good except for kids shows. Amazon's prices are meh. Play's prices fluctuate madly.04:47
SpamapSlifeless: do you have robber barons there that are willing to pay these prices? It seems counter-intuitive to price oneself out of the market. :-/04:47
clarkbSpamapS: netflix and amazon have tons of good stuff04:47
clarkband I get amazon for freeish since I pay for prime for other reasons04:47
SpamapSclarkb: they occasionally have good stuff04:47
SpamapSand we took the plunge and got Amazon Fresh...04:47
greghaynesI do like now nowdays netflix works better than amazon for me in linux04:48
clarkbSpamapS: amazon fresh isn't subscription though04:48
clarkbyou just order and stuff arrives the next day04:48
SpamapSin fact I'm ordering ingredients to make goulash tomorrow right now from Fresh.04:48
StevenKtchaypo: So the RAID 5 disaster was ~ 2000, and zfs appeared in 200504:48
SpamapSclarkb: it most certainly _IS_ a subscription here in LA.04:48
lifelessSpamapS: indeed04:48
StevenKMmmm, dinner. I should think about that.04:48
lifelessSpamapS: google play is strictly more expensive than dvd or blueray here04:48
clarkbSpamapS: huh it isn't in seattle04:48
SpamapSclarkb: maybe it is cheaper in Seattle. In LA, it is an upgrade from Prime.04:48
clarkbSpamapS: you literally load up a cart and pick a delivery time04:48
*** akuznetsov has quit IRC04:49
StevenKTakes me a few days to remember that I need to plan meals after coming home from a sprint04:49
lifelessSpamapS: and since we pay for total traffic (vs bandwidth) play is less attractive to consumers except if you can cache it04:49
lifelessSpamapS: which you can't04:49
clarkbSpamapS: pro tip you can order a bunch of normal amazon stuff via amazon fresh whcih means scheduled next day delivery for all the things :)04:49
SpamapSclarkb: $299 for the year.04:49
SpamapSbut it includes Prime.04:49
SpamapSand they're _wicked_ fast.04:49
SpamapSclarkb: It enrages me a bit to hear that you got it for free.04:50
clarkbSpamapS: well I have to pay for delivery04:50
clarkbbut I am not required to pay a subscription04:50
*** akuznetsov has joined #tripleo04:50
StevenKSpamapS: So, you're pointing and laughing about movies. Can we do the same about say, health care? :-P04:51
SpamapSBut I did the math.. and I use it twice a week.. it saves me about an hour a week. $230 (prime already pays for itself other ways) is well worth 52 hours of my time and wife's time.04:51
SpamapSclarkb: OH.. delivery is free for me.04:51
SpamapSStevenK: let's not04:51
SpamapSI'm not laughing actually04:51
SpamapSI was seriously WTF'ing04:51
lifelessStevenK: thats cruel :P04:51
SpamapSjust weird to me to think about having any kind of server at home.04:51
clarkbSpamapS: ah so thats not too terrible04:51
SpamapSclarkb: they're A/B'ing us ;)04:52
lifelessSpamapS: its inconcievable to me not to04:52
* mordred has no servers at home04:52
* mordred has like 10 cloud accoutns04:52
StevenKmordred: Lies.04:52
lifelessmorganfainberg: other than the drizzle build farm :P04:52
lifelessmordred: ^04:53
mordredlifeless: that hasnt' been running since openstack started :)04:53
lifelesssee, openstack is too hard to install04:53
StevenKmordred: They may not be powered on, but they're still in your house.04:53
lifelessStevenK: technically they aren't servers if they aren't serving.04:53
lifelessStevenK: OTOH ipads and phones and laptops are servers much of the time :)04:54
lifelesshttps://jenkins07.openstack.org/job/check-tripleo-overcloud-precise/176/console :<04:55
SpamapSSo really what's going on is that you need somebody in .nz and/or .au to stand up something like M-GO so Google will have to actually compete.04:55
*** akuznetsov has quit IRC04:55
lifelessthat would help04:56
mordredStevenK: they are, in fact, Mac Pro Towers running that would run Windows and OSX if powered on - so I'd say they're barely computers04:56
lifelessbut honestly, my server is about 1/10th the cost of cloud storage04:56
lifelessand I get compute04:56
StevenKSpamapS: Bandwidth04:56
lifelessoffsite backup with a second instance at a friends04:57
mordredlifeless: my cloud accounts are all free04:57
lifelessmordred: for your personal stuff?04:57
StevenKDSL speeds in much of .au and .nz are still pathetic04:57
mordredmy cloud operations are uch cheaper than local things :)04:57
clarkblifeless: my personal cloud account is free04:57
clarkbI opened the cloud04:57
mordredlifeless: yeah man. I opened the cloud - rackspace likes to gift stuff04:57
lifelessmordred: huh; well for you special snowflakes04:57
lifelessbut for those of us with 7TB of data to backup04:58
StevenKBut then you have to *get* the data, which could take a day04:58
lifelessoh, and don't forget the steam liobrary05:00
lifelessthats a big chunk on its own05:00
lifeless7GB on S3 is 550USDpm05:00
StevenKI could store my data in Amazon for $210US/month, not counting the charges to actually push more stuff and download existing05:00
lifelesslocal storage pays for itself in < 1yr05:01
lifeless440 on RRS05:01
lifelesswith no data transfer05:01
StevenKlifeless: Are you counting bandwidth charges (yours, not Amazon or Rackspace) in your calculations?05:01
lifeless71USDpm i glacier05:01
lifelessStevenK: no, just base storage05:02
StevenKlifeless: And then 4 days notice to actually get anything out05:02
lifelessgo https://jenkins07.openstack.org/job/check-tripleo-overcloud-precise/176/console go05:06
SpamapSlifeless: you don't seem to be valuing your time very highly. :p05:09
tchaypopypi is really making me sad - I keep getting random errors.05:10
tchaypoI may have to stop thinking that greghaynes is over-the-top for running a local mirror05:10
SpamapStchaypo: -> pypi.openstack.org and/or local pypi-mirror run05:10
lifelessSpamapS: thats the save-1-hour-on-test-runs-test05:10
SpamapSI started a local mirror run earlier today because I want to start playing w/ wheels. :p05:10
lifelessSpamapS: pypi.o.o is terrible on the internet05:10
lifelessSpamapS: you need to stop thinking like an american :)05:11
SpamapSlifeless: it is wonderful through my squid though05:11
StevenKtchaypo: I have a local Ubuntu mirror, is that over-the-top ?05:11
lifelesstchaypo: bandersnatch can mirror all of pypi for you, minus a few bad things05:11
tchaypoi see there's a pypi-openstack element05:11
lifelesstchaypo: yeah, it hardcodes pypi.o.o in05:11
tchaypoI don't really want to mirror all of pypi05:12
SpamapStchaypo: As lifeless says, pypi.openstack.org will not necessarily improve things for you in .au and .nz ..05:12
StevenKtchaypo: It's only 70G05:12
SpamapSTHough just forcing http does improve things if you have a squid. :)05:12
tchaypojust the packages I use05:12
StevenKOh, that's right. You don't have real computers.05:12
SpamapSI'm running the OpenStack pypi-mirror .. not bandersnatch..05:12
tchaypoyeah, but if squid checks upstream for if-modified-since and gets a 502, you're out of luck05:12
tchaypo70G is ~3/16ths of my monthly quota.05:13
StevenKtchaypo: My monthly quota is 60G05:14
StevenKHowever, it's not quite that bad, since that's only peak (noon-midnight)05:14
StevenKOff peak is unmetered05:15
*** akuznetsov has joined #tripleo05:15
StevenKAh ha. There is a hotel on my amex05:20
StevenKIf that's made it across, I know what I'm doing for the rest of the day05:20
SpamapSStevenK: let me know if you can get eem working with Firefox 2705:21
SpamapSit has been reporting an OCSP cert problem for me the last 3 weeks05:21
SpamapS(and chromium has never worked)05:21
tchaypoI still have nothing.05:22
tchaypobut plan to file the expense claim just as soon as i get my laptop running properly-encrypted linux05:23
StevenKSpamapS: I can't see the toolbar05:23
StevenKBut random clicking around works05:23
*** akuznetsov has quit IRC05:24
StevenKSpamapS: The cert for the header was changed on the 3rd05:26
StevenKAnd EEM only has two transactions from the amex, so more waiting05:26
* StevenK looks for tchaypo's instructions05:26
*** akuznetsov has joined #tripleo05:28
tchaypobandersnatch is on the move... i'll let you know how long 70Gb takes to download once I found out05:30
*** akuznetsov has quit IRC05:30
SpamapSStevenK: if you pop up the javascript console you can click the javascript links that paint the toolbar and accept the security exceptions05:31
SpamapSlifeless: 2014-03-11 04:39:54.268 | Waiting for the overcloud stack to be ready05:31
SpamapS2014-03-11 05:21:58.994 | Build timed out (after 145 minutes). Marking the build as failed. :(05:31
StevenKSpamapS: I'm trying to get firefox to trust HP's CA05:31
SpamapSStevenK: you have HP's ca in /etc/openvpn05:31
SpamapSStevenK: trust your feelings.. and your corporate cert05:32
StevenKTrying to work out how to pour into Firefox05:32
mikaltchaypo: what pypi problem are you seeing? I'm hoping it mirrors mine.05:32
StevenKBut also trying to see details about it, which means I'm trying to swap back in how the hell to drive openssl05:33
SpamapS# du -hs /opt/stack/venvs/os-*-config/05:33
SpamapSOne of these things, is nooot like ottherrr05:33
tchaypoStevenK: once you find the cert file, look in settings for the button that lets you manage certificates and then import it05:34
SpamapSbabel.. 11M05:34
SpamapSwe should consider sharing things like babel among all virtualenvs05:34
tchaypomikal: i haven't pinned it down - the last one was a 502 trying to download netaddr, but earlier I think I got a 401 for something05:35
StevenKSpamapS: So, yes, EEM works for me with Firefox 2705:35
SpamapS152MB of duplicate babel/localedata on our controller image05:35
mikaltchaypo: I've been seeing "connection reset by peer" all day05:35
SpamapSStevenK: thx.. will try the windows solution (reboot!)05:35
mikaltchaypo: I think its the SYD CDN node being broken05:35
clarkbStevenK: I couldn't get it to work in sacuy firefox but chromium was fine05:36
mikaltchaypo: hard coding /etc/hosts to a node in the US seems to have helped05:36
tchaypomikal: my 502 (proxy error) must have come from squid - mayhap that represents an upstream reset-by-peer05:36
clarkbno idea why but gave up when I found the chromium workaround05:36
StevenKclarkb: Haha. And SpamapS says that chromium has never worked05:36
StevenKclarkb: Works fine for me in current saucy05:36
tchaypomikal: that would make sense. bandersnatch seems to be chugging away. Having a local mirror probably makes sense anyway.05:37
mikalTell me more about bandersnatch05:38
mikalI've never used it05:38
clarkbits does a proper mirror of pypi05:38
tchayponeither had I unto 5 minutes ago05:38
mordredright. but the problem is05:38
mikalAll of pypi?05:38
clarkbso you get external linkage breakage and no special wheels05:38
mordredit's a PROPER mirror05:38
mordredso it's poop05:38
StevenKmordred: How so?05:38
mordredbecause it doesn't spider external things05:38
SpamapSone man's poop, is another man's web app05:38
clarkbStevenK: or build platform specific wheels05:39
mordredsomeone should check out devpi and see if it's good now05:39
mordredlast time I checked it out it was almost good05:39
StevenKMind you, I haven't looked at a local pypi mirror. I was schooling people that local Ubuntu mirrors aren't 0.5T at LCA05:40
clarkbStevenK: the size isn't an issue05:40
clarkbStevenK: the problem is it misses important bits05:40
clarkbthe bits that fail most often05:40
mikalSo, we've been using the dodgy pypi mirror thingie from the tripleo people05:40
mikalBut it broke this morning when our connectivity to pypi.python.org became unreliable05:40
mikalAnd took turbo hipster with it05:41
clarkbmikal: did pypi.openstack.org not work?05:41
tchayposo I hear I shouldn't be bothering with bandersnatch...05:41
mikalclarkb: I'd have to double check with jhesketh, but we saw our nodes will poking upstream as well05:41
tchaypomikal: can you clarify what you mean by "dodgy pypi mirror thingie"? is that the pip-cache element?05:41
clarkbtchaypo: it will probably help the cross pacific problem05:41
mikalclarkb: probably because we test against stable releases as well05:41
clarkbtchaypo: but when external links disappear it won't05:41
clarkbmikal: upstream?05:42
mikaltchaypo: https://git.openstack.org/cgit/openstack-infra/pypi-mirror05:42
mikalclarkb: pypi.python.org05:42
mikaltchaypo: dodgy because nothing that poorly documented can be good05:42
tchaypoI don't think I have a cross-pacific problem - ping times to pypi.python.org are not consistent with trans-pacific traversal05:42
StevenKpypi is on a CDN05:43
mikaltchaypo: yeah05:43
mikaltchaypo: and I have lost faith in the local SYD CDN endpoint05:43
StevenKpypi.python.org is an alias for global.prod.fastly.net.05:43
clarkbmikal: it does fit a particular niche because we would like bandersnatch to not suck05:43
mikalclarkb: the mirror thing? I think I'm mostly tainted because lifeless insisted the documentation was in the tripleo image builder code, and I was jetlagged05:44
clarkbmikal: yeah05:44
mikalclarkb: some documentation wouldn't be terrible05:44
mikalclarkb: do you guys see unicode decode errors when you run it?05:45
mikalclarkb: that's the problem I currently have05:45
clarkbI don't think so05:45
mikalCommand python setup.py egg_info failed with error code 1 in /tmp/tmpmmbWtg/venv/build/sphinxcontrib-docbookrestapi05:45
clarkbits easy to check jenkins /me looks05:45
tchaypoclarkb: do we have a standard yaml config for the pypi-mirror?05:45
mikalTraceback (most recent call last):05:45
mikal  File "/tmp/tmpmmbWtg/venv/bin/pip", line 11, in <module>05:45
mikal    sys.exit(main())05:46
mikal  File "/tmp/tmpmmbWtg/venv/local/lib/python2.7/site-packages/pip/__init__.py", line 185, in main05:46
mikal    return command.main(cmd_args)05:46
mikal  File "/tmp/tmpmmbWtg/venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 161, in main05:46
mikal    text = '\n'.join(complete_log)05:46
mikalUnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 62: ordinal not in range(128)05:46
clarkbtchaypo: yes, ts in openstack-infra/config somewhere05:46
StevenKI think we're going to find the entire Internet in openstack-infra/config somewhere soon05:46
mikaltchaypo: I use http://paste.openstack.org/show/73105/05:47
*** akuznetsov has joined #tripleo05:47
StevenKSince everything else is in there05:47
clarkbmikal: looks happy https://jenkins.openstack.org/job/periodic-mirror-python27/215/consoleFull05:47
mikalclarkb: that makes me wonder what we've done wrong05:47
lifelessmordred: you can layer pypi-mirror for wheels on top of bandersnatch, and the external issue thing is slowly going away05:47
clarkbmikal: we use older pip though05:47
mikaltchaypo: ./modules/openstack_project/files/pypi-mirror.yaml seems like a winner05:47
mikalclarkb: what version of pip are you using?05:48
clarkblifeless: we also use the mirror for enforcement or something05:48
clarkbbut I suppose if bandersnatch was only behind the scnees that would work05:48
lifelessclarkb: refuted several times05:48
clarkblifeless: o?05:48
clarkbI tried refuting it many times and lost each time :)05:49
lifelessclarkb: check infra logs for phrases like 'and this is why the mirror isn't an enforcement for version constriants'05:49
mordredclarkb: I think we shoud stop using the mirror for enforcement05:49
mordredclarkb: which I believe was your original point05:49
clarkbmordred: it was/is05:49
mordredclarkb: I think now that we've got requirements repo doing that for us05:49
mordredthe mirror should be about solving network glitches05:49
mordredthe amount of fail that we deal with in the current design is immense05:50
tchaypohttps://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/pypi/README.md is most helpful also05:50
mordredlifeless: the mirror IS currently a second layer of enforcement - but it shoudl stop beign so05:50
lifelessSpamapS: so yeah, that build fail sucks05:50
openstackgerritA change was merged to openstack/tripleo-heat-templates: Add rabbit.cookie attribute  https://review.openstack.org/7023805:51
* tchaypo realises why he's tired - it's 21:51 Sunnyvale non-DST05:52
StevenKI got roughly 11 hours sleep, so I'm okayish05:52
StevenKI'm expecting to hit a wall at about 9:30pm05:52
lifelessSpamapS: 2014-03-11 00:13:41.770 3267 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip:
lifeless2014-03-11 00:13:41.829 3267 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip:
lifeless2014-03-11 00:14:11.021 3267 WARNING nova.network.neutronv2 [req-4ed301ee-6f85-467b-bdc6-247aaa634ee9 None None] Using neutron_admin_tenant_name for authentication is deprecated and will be removed in the next release.  Use neutron_admin_tenant_id instead.05:53
lifeless2014-03-11 00:14:11.214 3267 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip:
*** akuznetsov has quit IRC05:53
*** rpodolyaka has joined #tripleo05:54
lifelessjogo: so - this failed to get metadata for X05:55
lifelessjogo: you're saying that when there is an exception trace, we hide it by default ?05:55
tchaypoI hit the wall badly about 9 last night05:55
lifelessok, C-> bed then some r&r before my 2200 call with the UK05:56
lifelessI'm going to file a bug on this flaky test thing though05:56
lifelessits batshit05:56
lifelessDO RRIAGE05:56
*** CaptTofu has joined #tripleo05:58
*** CaptTofu has quit IRC06:02
lifelessStevenK: https://bugs.launchpad.net/tripleo/+bug/1290483 might be interesting to you once you finish the nodes arc.06:03
*** matsuhashi has quit IRC06:05
*** akuznetsov has joined #tripleo06:07
*** matsuhas_ has joined #tripleo06:08
*** akuznetsov has quit IRC06:08
*** akuznetsov has joined #tripleo06:09
*** akuznetsov has quit IRC06:10
greghaynesso, has anyone experienced nova bm barfing with  ERROR: sector 0 does not have an msdos signature for the undercloud iscsi?06:15
lifelesspretty sure sfdisk outputs that always06:17
lifelessgreghaynes: what log dod you see that in ?06:17
greghayneshrm. Heres /nova-baremetal-deploy-helper.log06:19
lifelessjogo: https://bugs.launchpad.net/tripleo/+bug/129070106:19
lifelessgreghaynes: ok, I may be wrong :)06:19
greghaynessyslog also shows iscsi detected connection error, not sure how relevant though06:19
greghaynesaaah, forgot link: Heres /nova-baremetal-deploy-helper.log https://gist.githubusercontent.com/anonymous/9480493/raw/467dea61459a80c516e7e3886652ec88215cd12d/gistfile1.txt06:21
lifelessgreghaynes: this is your problem06:22
lifelesssfdisk: bad input\n'06:22
lifelessgreghaynes: the rest is a distraction06:22
lifelessWarning: given size (21504) exceeds max allowable size (21503)06:22
lifelessmay be implicated06:22
lifelessgreghaynes: does your flavor allocate more disk space than you have ?06:23
greghaynesShouldnt, shows 21gb for flavor, 211gb avail on disk06:24
lifelessgreghaynes: paste nova flavor-list ? just for my satisfaction :)06:24
greghaynes21. useful units are useful06:25
lifelessgreghaynes: and the 211G is the qcow2 VM disk size, or free space on your host ?06:26
greghaynesfree space on host06:26
lifelessneed the VM block device size :)06:27
lifelessqemu-img info or some such06:27
*** vkozhukalov has joined #tripleo06:27
greghaynesvirtual size for qemu being created shows 4.3G06:28
lifeless20G >> 4.3G06:29
lifelessbut try06:30
lifelesssudo qemu-img info /var/lib/libvirt/images/baremetal_0.qcow206:30
lifelessfor instance06:30
greghaynesah, virtual size: 21G (22548578304 bytes)06:31
lifelessso 21 and 2106:31
lifelessshould fit06:31
lifelessyou might try, just for kicks, a 20G flavor definition instead06:31
greghaynesyes, although that size discrepency was pretty narrow06:31
lifelessone (unit)06:32
lifelessbut note that we consume one 'track' for partition table06:32
lifelessso its entirely possible....06:32
lifelessok, bbiaw06:33
greghaynesah, fun06:33
*** killer_prince has joined #tripleo06:40
*** rdopieralski has joined #tripleo06:40
*** rdopieralski has quit IRC06:41
*** rdopieralski has joined #tripleo06:41
*** Qiming has quit IRC06:44
openstackgerritA change was merged to openstack/tuskar-ui: Error out on too few nodes  https://review.openstack.org/7794206:59
*** rpodolyaka has quit IRC07:01
*** rpodolyaka has joined #tripleo07:01
*** rpodolyaka has quit IRC07:01
*** rpodolyaka has joined #tripleo07:01
*** rpodolyaka has quit IRC07:02
*** rpodolyaka has joined #tripleo07:07
*** rsacharya has joined #tripleo07:14
GheRiverogreghaynes: https://review.openstack.org/#/c/79344/07:16
GheRiveromorning TripleO07:16
*** rpodolyaka has quit IRC07:17
tchaypomorning GheRivero07:18
*** mrunge has joined #tripleo07:18
*** mrunge has quit IRC07:18
*** mrunge has joined #tripleo07:18
openstackgerritA change was merged to openstack/tuskar-ui: Fix task states of nova-baremetal nodes  https://review.openstack.org/7868807:18
*** ramishra has joined #tripleo07:28
openstackgerritRohan Kanade proposed a change to openstack/tuskar-ui: Fix task state of nova instance  https://review.openstack.org/7868507:39
*** CaptTofu has joined #tripleo07:46
*** vkozhukalov has quit IRC07:50
*** CaptTofu has quit IRC07:51
*** rcarrillocruz1 has joined #tripleo07:57
*** rcarrillocruz has quit IRC07:59
*** akrivoka has joined #tripleo08:00
*** ifarkas has quit IRC08:04
*** pblaho has joined #tripleo08:05
*** bauzas has joined #tripleo08:07
*** rcarrillocruz has joined #tripleo08:07
*** rcarrillocruz1 has quit IRC08:08
*** e0ne has joined #tripleo08:10
*** jprovazn has joined #tripleo08:16
*** ramishra has quit IRC08:16
*** ramishra has joined #tripleo08:16
*** ramishra has quit IRC08:21
*** jcoufal has joined #tripleo08:24
*** giulivo has joined #tripleo08:25
openstackgerritstephen-mulcahy proposed a change to openstack/tripleo-incubator: Add basic timing instrumentation to devtest.sh  https://review.openstack.org/7756408:25
*** morganfainberg is now known as morganfainberg_Z08:27
*** jistr has joined #tripleo08:33
*** ifarkas has joined #tripleo08:34
*** rlandy has joined #tripleo08:41
*** bogdando has quit IRC08:42
openstackgerritAna Krivokapic proposed a change to openstack/tuskar-ui: Show reason of failure in case of deployment error  https://review.openstack.org/7841208:43
*** rcarrillocruz1 has joined #tripleo08:46
*** rcarrillocruz has quit IRC08:46
*** andreaf has joined #tripleo08:48
*** petertoft has joined #tripleo08:50
*** adrian17od has joined #tripleo08:51
*** akrivoka has quit IRC08:55
*** akuznetsov has joined #tripleo08:58
*** jcoufal has quit IRC09:00
*** jcoufal_ has joined #tripleo09:00
*** derekh has joined #tripleo09:01
*** jcoufal_ has quit IRC09:02
*** rsacharya has quit IRC09:02
rlandySpamapS: ping09:03
lifelessgreghaynes: https://bugs.launchpad.net/tripleo/+bug/129039909:04
lifelessgreghaynes: same symptoms09:04
GheRiverolifelesss: https://review.openstack.org/#/c/79344/09:04
SpamapSrlandy: pong (but fading fast ;)09:05
lifelessGheRivero: check-tripleo-overcloud-precise FAILURE in 2h 25m 16s09:05
rlandySpamapS: I'll be quick :) ...09:05
*** pbelanyi has joined #tripleo09:05
rlandySpamapS: I posted a question yesterday (before I faded out) about heat stack registering that the overcloud had completed before a cinder node had finished building09:06
*** martyntaylor has joined #tripleo09:06
*** rsacharya has joined #tripleo09:06
*** killer_prince has quit IRC09:07
rlandySpamapS: what qualifies as a 'completed overcloud'?09:07
*** jrist has quit IRC09:07
openstackgerritJan Provaznik proposed a change to openstack/diskimage-builder: Adds "provides" support to element dependencies  https://review.openstack.org/7942109:07
*** matsuhas_ has quit IRC09:08
SpamapSrlandy: the status of the stack would be 'CREATE_COMPLETE'09:08
rlandySpamapS: http://paste.openstack.org/show/73083/09:08
*** hashar has joined #tripleo09:09
SpamapSrlandy: right and I asked you for 'heat event-list overcloud'09:09
rlandycopying ... sec09:09
*** matsuhashi has joined #tripleo09:09
*** martyntaylor has quit IRC09:10
rlandySpamapS: http://paste.openstack.org/show/73118/09:10
*** tomh_ has joined #tripleo09:10
*** akrivoka has joined #tripleo09:12
SpamapS| BlockStorage0                   | aeee778c-543c-4b95-8214-b11c3ff586c5 | state changed          | CREATE_COMPLETE    | 2014-03-10T22:09:15Z |09:12
SpamapSrlandy: that is likely a bug in Heat09:12
SpamapSand with that, I pass out09:13
rlandySpamapS: thanks - will log that09:13
*** jcoufal has joined #tripleo09:15
lifelessSpamapS: https://bugs.launchpad.net/tripleo/+bug/129075909:19
Ng+1 more logging09:19
lifelessnight all09:19
StevenKNg: Such log09:20
lifelessmuch debug09:21
*** jrist has joined #tripleo09:21
StevenKNg: Ursula is a bad person, and shared http://dogeweather.com/ on FB09:21
*** lucasagomes has joined #tripleo09:24
*** martyntaylor has joined #tripleo09:28
*** ramishra has joined #tripleo09:33
*** CaptTofu has joined #tripleo09:35
*** athomas has joined #tripleo09:35
*** ramishra has quit IRC09:37
*** CaptTofu has quit IRC09:40
*** jp_at_hp has joined #tripleo09:49
*** martyntaylor has quit IRC09:49
*** akuznetsov has quit IRC09:50
*** martyntaylor has joined #tripleo09:53
*** martyntaylor has quit IRC09:56
*** martyntaylor has joined #tripleo09:57
*** rsacharya has quit IRC10:05
*** martyntaylor has quit IRC10:05
*** saju_m has joined #tripleo10:05
*** killer_prince has joined #tripleo10:07
*** saju_m has quit IRC10:18
*** rcarrillocruz has joined #tripleo10:18
*** martyntaylor has joined #tripleo10:19
*** rcarrillocruz1 has quit IRC10:20
*** matsuhashi has quit IRC10:29
*** rcarrillocruz1 has joined #tripleo10:30
*** saju_m has joined #tripleo10:30
*** rcarrillocruz has quit IRC10:31
*** andreaf has quit IRC10:35
*** markmc has joined #tripleo10:48
openstackgerritA change was merged to openstack/tuskar-ui: Fix task state of nova instance  https://review.openstack.org/7868510:53
*** saju_m has quit IRC10:54
*** CaptTofu has joined #tripleo11:05
*** hashar has quit IRC11:17
*** rcarrillocruz has joined #tripleo11:17
*** hashar has joined #tripleo11:19
*** rcarrillocruz1 has quit IRC11:20
*** rcarrillocruz1 has joined #tripleo11:28
*** rcarrillocruz has quit IRC11:29
*** morazi has joined #tripleo11:30
*** rcarrillocruz has joined #tripleo11:32
*** andreaf has joined #tripleo11:35
*** rcarrillocruz1 has quit IRC11:35
*** rdopieralski has quit IRC11:37
*** jtomasek has quit IRC11:46
*** hashar has quit IRC11:57
*** jrist has quit IRC11:58
*** jtomasek has joined #tripleo12:00
*** weshay has joined #tripleo12:03
*** crimi has joined #tripleo12:03
*** killer_prince has quit IRC12:05
jp_at_hpis there a bug for a generic tripleo CI failure?  saying recheck no bug seems to be avoiding the fact that the ci wasn't working?  Or is that the implied meaning of no bug?12:08
Ngjp_at_hp: I would tend to think that any failure of the CI environment to self-assemble, is a bug somewhere. I'm not sure that we necessarily want a new bug in lp for each failure, but we should be checking over failures to see what was going wrong12:15
jp_at_hpNg: yes - that's what I would have thought.  It makes sense...12:18
jp_at_hpSo to that end - I have a test failing because of permissions writing into the seed  image it is trying to create.12:19
jp_at_hp2014-03-11 12:03:30.517 | cp: cannot create regular file `/tmp/image.KcBeq5Bk/mnt/etc/dib-git-manifest-seed': Permission denied12:19
slaglejp_at_hp: b/c you need to use sudo there12:20
jp_at_hpI see other writes to that dir happening, and my testing didn't have a problem :(  Although I was testing as root.  I guess from this I can assume that the upstream ci run is as a non-root user?  But with passwordless sudo?12:20
slaglejp_at_hp: was just looking at your review :). scripts under extra-data.d do no run as root in the chroot12:20
jp_at_hpSo is the bug in my change, or in the running of the extra-data.d scripts?12:21
slaglejp_at_hp: in your change.12:21
jp_at_hpWhat is the case for not running any of those scripts as root, when the sole purpose is to create the image as desired?12:21
slaglejust add a sudo before that cp at line 18712:21
jp_at_hpslagle: yes, I know that will fix it.  But what is the reason for not running the extra-data.d scripts as root?12:22
slaglejp_at_hp: some hook scripts run chrooted into the image as root, some don't. have a look at the README12:23
slaglejp_at_hp: it's so that you can manipulate the image to do things *before* chrooting into it12:23
*** rbrady-afk is now known as rbrady12:23
slaglelike copy stuff in from the build host, as you're doing here12:23
slagleoriginally, i *believe* dib had the sudo happening at a higher level, so that an element author didn't have to use sudo. but, that was changed12:24
jp_at_hpI'll change my patch, no probs, but getting insight into *why* thingsd are as they are would be super-helpful.12:24
jp_at_hpThis commit seems to be related: 357c2de681c698915ea062de72cd1863067c080b12:25
slaglewell, i would guess it's because dib tries to not use sudo/root where possible12:26
slagleso instead of running every extra-data.d with sudo by default, where it may not be needed, it's left up to the element author to determine when it's needed12:26
slagleand for the hook scripts that run chrooted into the image, you don't have to use sudo12:27
slaglei admit it's confusing, i often forget to use sudo in extra-data.d12:27
jp_at_hpslagle: I can appreciate that, just not sure it makes complete sense to me.  Any package management system operating outside of a users home directory requires execution as root.  It seems that requiring same for diskimage-builder is not inreasonable, and is just a further tightening of the run requirements, as was the case for the last commit affecting sudo-related things that I posted above.12:28
slaglejp_at_hp: sure, though, i would not like running dib as root. even though it uses sudo everywhere as-is12:29
jp_at_hpslagle: what - you don't trust it?12:30
jp_at_hpI suppose there is a danger of a riogue element affecting your local system, but if you need passwordless sudo anyways...12:30
slaglenot explicitly12:30
*** lucasagomes is now known as lucas-hungry12:31
*** jdob has joined #tripleo12:31
*** e0ne_ has joined #tripleo12:39
openstackgerritJon-Paul Sullivan proposed a change to openstack/diskimage-builder: Create a git manifest from source-repositories  https://review.openstack.org/7582912:39
*** e0ne has quit IRC12:41
*** hewbrocca has joined #tripleo12:44
*** nosnos has quit IRC12:44
*** CaptTofu has quit IRC12:44
openstackgerritA change was merged to openstack/diskimage-builder: Additional swift storage service mappings  https://review.openstack.org/7675312:48
giulivoguys I restarted my seed and didn't set on the eth1 interface, if I set it manually things get back working but I can't find the element setting it up ... that's because init-neutron-ovs isn't called again after initial configuration, right?12:51
openstackgerritA change was merged to openstack/diskimage-builder: Replace more then just "-" in REPONAME  https://review.openstack.org/7667912:51
*** rcarrillocruz1 has joined #tripleo12:53
*** rcarrillocruz has quit IRC12:54
*** lazy_prince is now known as killer_prince12:56
*** Qiming has joined #tripleo13:05
*** dprince has joined #tripleo13:05
*** Qiming has quit IRC13:06
*** CaptTofu has joined #tripleo13:07
*** lblanchard has joined #tripleo13:08
*** akuznetsov has joined #tripleo13:09
*** CaptTofu has quit IRC13:11
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: remove keystone-signing dir creation  https://review.openstack.org/7498913:13
jprovazngreghaynes: hi, are you around?13:13
openstackgerritJon-Paul Sullivan proposed a change to openstack/diskimage-builder: Fixup all occurrences of REPONAME for replacing '-'  https://review.openstack.org/7960213:14
*** hashar has joined #tripleo13:14
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Excplicitly install dhcp-agent  https://review.openstack.org/7499113:14
*** tzumainn has joined #tripleo13:17
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Restrict deletion of packaged files to Fedora like packages  https://review.openstack.org/7499013:17
*** julim has joined #tripleo13:18
*** jrist has joined #tripleo13:26
*** rdopieralski has joined #tripleo13:31
*** jprovazn has quit IRC13:34
*** jprovazn has joined #tripleo13:37
*** lucas-hungry is now known as lucasagomes13:38
*** bogdando has joined #tripleo13:42
*** ftcjeff has joined #tripleo13:48
*** spzala has joined #tripleo13:49
*** jcoufal has quit IRC13:50
*** crimi has quit IRC13:52
openstackgerritA change was merged to openstack/tuskar-ui: Show reason of failure in case of deployment error  https://review.openstack.org/7841213:53
*** Matt2 has joined #tripleo13:56
*** akuznetsov has quit IRC13:58
*** akuznetsov has joined #tripleo13:59
openstackgerritPetr Blaho proposed a change to openstack/tuskar: Updates CONTRIBUTING docs sort bug in fedora 20  https://review.openstack.org/7960914:08
*** pblaho has quit IRC14:08
*** ramishra has joined #tripleo14:24
*** matty_dubs|gone is now known as matty_dubs14:28
GheRiveroIs anyone using the pypi-mirror element? The released version is failing due to the external pypi references (fixed upstream)14:31
*** mrunge has quit IRC14:31
rbradybnemec: https://review.openstack.org/#/c/77283/14:34
bnemecrbrady: Thanks14:37
rbradybnemec: np14:38
slaglederekh: https://review.openstack.org/#/c/79344/14:39
derekhslagle: thanks14:39
*** jcoufal has joined #tripleo14:40
*** rpodolyaka has joined #tripleo14:40
*** jprovazn has quit IRC14:40
*** rcarrillocruz1 has quit IRC14:42
*** jprovazn has joined #tripleo14:43
*** rcarrillocruz has joined #tripleo14:43
*** jrist has quit IRC14:43
killer_princeplease review https://review.openstack.org/#/c/79286/14:44
openstackgerritA change was merged to openstack/tuskar: Pin iso8601 logging to WARN  https://review.openstack.org/7951614:45
*** blamar has joined #tripleo15:02
*** sparkycollier has joined #tripleo15:03
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: List filesystem and swift and known glance stores  https://review.openstack.org/7963115:06
*** jcoufal has quit IRC15:06
*** jcoufal has joined #tripleo15:07
derekh^^ probably the reason ci overcloud jobs are currently failing (because we don't have enough problems)15:07
*** rlandy has quit IRC15:07
*** fungi has quit IRC15:07
*** rlandy has joined #tripleo15:08
*** jistr has quit IRC15:11
*** jcoufal has quit IRC15:12
*** jcoufal has joined #tripleo15:13
*** rcarrillocruz1 has joined #tripleo15:19
jprovaznI'm getting "Image storage media is full: There is not enough disk space on the image storage media." when loading undercloud image into seed's glance - seed VM is created with incorrect storage (4G instead of 20G defined in NODE_DIST)15:20
jprovaznis this known issue ^?15:20
*** rcarrillocruz has quit IRC15:21
jdobjprovazn: i think I saw a similar problem when changing NODE_DIST, but I assumed it was something I screwed up15:22
jdober, DISK15:23
jdob(I was looking at your comment when I typed it and got mentally screwed up)15:23
jprovaznjdob: :) ah, sry15:23
jprovaznjdob: thanks, will give it one more try15:24
jdobnah, its my fault. need more coffee.15:24
*** jcoufal has quit IRC15:25
openstackgerritJon-Paul Sullivan proposed a change to openstack/diskimage-builder: Require that root runs image manipulation scripts  https://review.openstack.org/7963615:26
jprovazngreghaynes: hi, ping me when you are here, I'd like to discuss one percona issue15:27
openstackgerritJon-Paul Sullivan proposed a change to openstack/diskimage-builder: Require that root runs image manipulation scripts  https://review.openstack.org/7963615:27
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: List filesystem and swift and known glance stores  https://review.openstack.org/7963115:29
*** sparkycollier has quit IRC15:30
*** jistr has joined #tripleo15:35
*** jistr is now known as jistr|afk15:36
derekhdprince: backing up a copy of the seed host's /root so I can update some repo's to build a new testenv image15:39
derekhdprince: backup is in /var/lib/misc/backups/20140311_1537/15:39
*** rpodolyaka has quit IRC15:40
dprincederekh: okay. Is there something that needs rebasing?15:40
dprincederekh: that is fine for now... but I would rather keep us in one tree so as to avoid confusion.15:41
* dprince runs off for lunch15:41
derekhdprince: ok, I'll move into a subdir15:41
*** noslzzp has quit IRC15:42
*** jcoufal-mobile has joined #tripleo15:45
*** fungi has joined #tripleo15:46
*** noslzzp has joined #tripleo15:46
*** rdopieralski has quit IRC15:46
*** ramishra has quit IRC15:48
*** ramishra has joined #tripleo15:48
killer_princei am trying to put common code used to find appropriate kernel to be used for booting instances (in vm and baremetal element and disk-image-get-kernel) but i can see that some scripts are executed using chroot and some are executed without chroot. is there a place where i can put the code and will be accessible by all scripts..?15:53
*** bogdando has quit IRC15:55
*** adrian17od has left #tripleo15:57
*** ramishra has quit IRC15:57
*** rpodolyaka has joined #tripleo15:57
*** jprovazn has quit IRC15:58
*** mrunge has joined #tripleo15:59
*** jcoufal has joined #tripleo16:00
openstackgerritRyan Brady proposed a change to openstack/tripleo-heat-templates: Merge Block Storage into Overcloud  https://review.openstack.org/7965416:04
*** cwolferh_ has joined #tripleo16:04
*** hewbrocc` has joined #tripleo16:08
rpodolyakamorning tripleo16:09
*** hewbrocca has quit IRC16:09
*** bogdando has joined #tripleo16:09
*** ifarkas has quit IRC16:11
*** rpodolyaka has quit IRC16:15
*** vkozhukalov has joined #tripleo16:16
*** rpodolyaka has joined #tripleo16:17
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Install jq on Test Environment hosts  https://review.openstack.org/7966316:19
*** jrist has joined #tripleo16:19
*** matty_dubs is now known as matty_dubs|lunch16:22
*** cadenzajon has joined #tripleo16:24
*** rpodolyaka has quit IRC16:27
*** e0ne has joined #tripleo16:27
*** e0ne_ has quit IRC16:30
*** mandeep has joined #tripleo16:31
*** mandeep has left #tripleo16:31
*** mandeep has joined #tripleo16:33
*** jcoufal-mobile has quit IRC16:36
*** jistr|afk is now known as jistr16:36
*** jcoufal-mobile has joined #tripleo16:36
*** jcoufal-mobile has quit IRC16:37
*** rcarrillocruz has joined #tripleo16:38
*** rcarrillocruz1 has quit IRC16:38
*** rpodolyaka has joined #tripleo16:40
openstackgerritJonathan Brownell proposed a change to openstack/diskimage-builder: Fix "(None)" hostname; provide default to cloud-init  https://review.openstack.org/7730116:41
*** killer_prince is now known as lazy_prince16:42
*** CaptTofu has joined #tripleo16:43
*** CaptTofu has quit IRC16:43
*** mandeep_ has joined #tripleo16:44
*** CaptTofu has joined #tripleo16:45
*** mandeep has quit IRC16:45
*** jang1 has joined #tripleo16:47
*** mandeep_ has quit IRC16:50
SpamapSderekh: perhaps you know this.. have we considered building a local on-disk pypi mirror for the test envs?16:56
derekhSpamapS: for the repositories being tested ? yes https://review.openstack.org/#/c/71837/16:57
derekhSpamapS: I was hoping to get feedback on it befor proceeding too far down that path16:58
*** ifarkas has joined #tripleo16:58
*** rsacharya has joined #tripleo16:59
SpamapSderekh: so I have a once-per-day-updated mirror on my laptop now and it cut about 20 minutes off my usual run time, even though I have a warm squid and 100Mbit internet....17:00
*** mandeep has joined #tripleo17:00
SpamapSderekh: seems like we should just have one pypi-mirror that we tar up and push out onto the testenvs.17:00
derekhSpamapS: ahh ok, thats a different case, yes its on the list https://etherpad.openstack.org/p/tripleo-parallel-test17:01
SpamapSderekh: IMO, we shouldn't bother fixing pypi-openstack to work with pypi. Our goal should be to use --offline anyway.17:01
*** jcoufal has quit IRC17:04
derekhSpamapS: makes sense to me, although I think the tarball should be kept update as much as possible17:04
*** mandeep has quit IRC17:06
*** mandeep_ has joined #tripleo17:06
*** mandeep_ has quit IRC17:10
SpamapSderekh: yeah hourly would even make sense.17:10
*** e0ne has quit IRC17:11
*** mandeep has joined #tripleo17:11
jogolifeless:         if CONF.verbose or CONF.debug:17:11
jogo            extra['exc_info'] = (exc_type, value, tb)17:11
*** sdake has joined #tripleo17:11
*** d0ugal has quit IRC17:13
*** rpodolyaka has quit IRC17:16
*** mandeep has quit IRC17:20
jogobut that is not it we have a exception and error message that are identical17:22
*** rpodolyaka has joined #tripleo17:22
*** mandeep has joined #tripleo17:22
*** matty_dubs|lunch is now known as matty_dubs17:24
*** martyntaylor has quit IRC17:25
*** pbelanyi has quit IRC17:26
*** mandeep has quit IRC17:27
derekhci-overcloud is busted, I think will be fixed by this https://review.openstack.org/#/c/79631 but I can't know for sure because we no longer grab enough logs (will fix)17:28
derekhthe error that caused that jobs to fail was a PXE deploy error, probably fixed by this https://review.openstack.org/#/c/79502/17:29
*** mrunge has quit IRC17:29
*** mandeep has joined #tripleo17:29
derekhthats a lot of I thinks and probables but its all I got :-(17:30
*** marun has quit IRC17:30
*** marun has joined #tripleo17:31
*** markmc has quit IRC17:31
SpamapSwe should be watching for UpgradeImpact commits as a team17:34
openstackgerritJenkins proposed a change to openstack/os-collect-config: Updated from global requirements  https://review.openstack.org/7969017:35
SpamapSderekh: damnit17:37
*** rupsky has joined #tripleo17:38
*** bauzas has quit IRC17:38
*** rpodolyaka has quit IRC17:38
*** rcarrillocruz1 has joined #tripleo17:39
derekhdamnit indeed17:39
*** rcarrillocruz has quit IRC17:39
*** rcarrillocruz has joined #tripleo17:41
*** rupsky has quit IRC17:41
SpamapSderekh: I'll +A as soon as CI passes. Don't need two +2's17:41
*** rupsky has joined #tripleo17:41
*** SpamapS changes topic to "FIREDRILL: CI failing https://review.openstack.org/#/c/79631/ | tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt"17:42
derekhSpamapS: The ci on the swift change is going to fail, and will need to be restarted17:42
SpamapSderekh: right, but the glance fix will pass right?17:42
derekhSpamapS: opps, I meant glance, I took a look at the instance and the heat stack-create has already failed17:43
*** rcarrillocruz1 has quit IRC17:43
derekhSpamapS: because of a PXE deploy failure17:44
derekhSpamapS: I hav't restarted the job becasue I thought we shoudl wait to get the logs17:44
*** rsacharya has quit IRC17:44
*** athomas has quit IRC17:45
SpamapSderekh: ah gotchya17:45
SpamapSderekh: let's get a bug ref on it17:46
derekhk, willdo17:46
*** petertoft has quit IRC17:46
SpamapSderekh: https://bugs.launchpad.net/tripleo/+bug/129096917:46
SpamapSderekh: filed already17:46
SpamapSand submitting revert to glance17:46
derekhActually no point in waiting as we no longer get the undercloud logs anyways17:47
*** rollerj has joined #tripleo17:47
*** hashar has quit IRC17:48
*** rpodolyaka has joined #tripleo17:49
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: List filesystem and swift as known glance stores  https://review.openstack.org/7963117:49
*** rupsky has quit IRC17:49
*** mandeep has quit IRC17:50
SpamapSderekh: _closes_ bug17:50
SpamapSderekh: So the bot will close tripleo's task on the bug.17:51
openstackgerritDerek Higgins proposed a change to openstack/tripleo-image-elements: List filesystem and swift as known glance stores  https://review.openstack.org/7963117:51
derekhSpamapS: ahh I was trying to prevent it closing the glace part, didn't know it could be selective17:52
*** rupsky has joined #tripleo17:53
SpamapSderekh: yes it is entirely selective.. one of my favorite parts of our launchpad integration actually17:54
*** sgrasley1 has joined #tripleo17:57
derekhgotta run, fingers crossed that job passes....17:59
SpamapSderekh: ok, I'll babysit it17:59
derekhlifeless: looked at the new update to json for testenv-workers today, comments added https://review.openstack.org/#/c/79631/ testing a new testenv at the moment with that change18:00
*** derekh has quit IRC18:00
*** mrunge has joined #tripleo18:00
*** mandeep has joined #tripleo18:01
*** mandeep has quit IRC18:04
jdobmeeting today?18:06
slaglei assume there will be at the correct time :)18:07
slaglei was wonder myself for a moment :)18:07
greghaynesYep, DST messing up all the things18:07
jdobshit, ok, thanks18:07
*** bauzas has joined #tripleo18:14
*** andreaf has quit IRC18:17
SpamapSI kind of wish I could get Unity's clock to show me Local + UTC at all times18:19
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Neutron: don't hard code the venv dir  https://review.openstack.org/7662618:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Update elements to use os-svc-daemon -i.  https://review.openstack.org/7696318:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Update elements to use os-svc-install -i.  https://review.openstack.org/7696418:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Wire in OPENSTACK_VENV_DIR default if set  https://review.openstack.org/7696518:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Wire in _EXTRA_INSTALL_OPTS...  https://review.openstack.org/7696618:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Add a new common-venv element  https://review.openstack.org/7696718:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Glance/Cinder/Keystone/Swift: don't hard code venv  https://review.openstack.org/7666618:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Stop using the os-svc-install -n -c options.  https://review.openstack.org/7660418:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Drop the os-svc-install -n -c options.  https://review.openstack.org/7660518:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Link db sync utilities to /usr/local/bin...  https://review.openstack.org/7657918:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Heat/Horizon/Tempest: don't hard code venv dir  https://review.openstack.org/7688818:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Ceilo/Ironic/Tuskar: don't hard code venv dir  https://review.openstack.org/7688718:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Nova: don't hard code the venv dir  https://review.openstack.org/7661918:23
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Add -i options to os-svc-install/os-svc-daemon...  https://review.openstack.org/7695218:23
*** rpodolyaka has quit IRC18:24
SpamapSmuch commit18:26
SpamapSso patch stream18:26
SpamapSdprince: what's it like out there in the weeds? ;-)18:26
dprinceSpamapS: quite a nice day actually. I may have to leave and go for a ride18:26
*** rpodolyaka has joined #tripleo18:27
SpamapSdprince: You will have to recheck bug 1290969 all those changes unfortunately18:27
dprinceSpamapS: meh. I'll do it later18:28
*** rcarrillocruz1 has joined #tripleo18:28
dprinceSpamapS: thanks for the heads up though18:28
*** rcarrillocruz has quit IRC18:29
SpamapSdprince: yeah we're still about 1.5hrs away from the fix commit landing anyway :-P18:31
*** cwolferh has joined #tripleo18:32
*** cwolferh_ has quit IRC18:32
dprinceSpamapS: well. if you guys would just land all my code we could be (with a single venv) only an hour away :)18:32
dprinceSpamapS: maybe less than that?18:32
*** mrunge has quit IRC18:32
dprinceSpamapS: no rush though, I can wait on CI18:33
SpamapSdprince: I have an idea for a compromise on the single venv idea.18:34
SpamapSdprince: it involves using hard links though.. so might be scary for some. ;)18:35
dprinceSpamapS: Well... so long as it is optional we could support both. I like that the single venv behaves more like real packages anyway.18:36
dprinceSpamapS: in any case like I mentioned before 99 percent of my changes (everything except that last commit for the common-venv element) is just plain good coding practice i.e. DRY18:37
*** akrivoka has quit IRC18:37
SpamapSdprince: do you not see venvs as "real" ?18:39
SpamapSOr just, not real enough for redhat customers? ;-)18:40
dprinceSpamapS: I suppose I don't. :)18:40
SpamapSanyway, didn't mean to be confrontational.. :)18:41
dprinceSpamapS: They have some utility... but are quite hard to nail down for reproducability18:41
SpamapSdprince: so the single venv thing is an optimization that loses some features.. I'm thinking we can get most of that optimization without losing the isolation feature.18:41
dprinceSpamapS: never know what the cat is going to drag into one of them...18:41
SpamapSdprince: well you do know, but sometimes you don't realize that you authorized the cat to drag in crap. ;)18:42
*** jprovazn has joined #tripleo18:43
SpamapSdprince: my idea is simply to comb through all the requirements.txt's before pip installing them, and build a single venv that is a union of the common requirements, and then link (sym or hard, not sure) lib/python2.7/* from there, the same way system-site-packages symlinks, into the new venvs...18:43
SpamapSthat will get a huge space savings (babel == 11MB * 20 venvs on controller) and will avoid double-building things.18:44
dprinceSpamapS: I'm not against it but it is conceptually more complex than my solution (and could break things). I think the right solution is to make all these things options and then users can choose.18:44
SpamapSoptions just slow users down18:44
dprinceSpamapS: not sure how it could break things but it is something more to maintain and I could see us having to fix it18:45
SpamapSIf we really don't want the isolation feature anymore, lets just drop it.18:45
SpamapSI don't see any value in having an explosion of options.18:45
*** akuznetsov has quit IRC18:45
dprinceSpamapS: well, to be fair I didn't add any new options. Just a new element that sets a simple ENV variable to override the default venv dir.18:45
dprinceSpamapS: Although I suppose os-svc-daemon got an extra option. But then I removed two unused ones as well so I think we made out for the better18:46
SpamapSnet loss of code == win18:46
SpamapSdprince: I think rather than having an option for something you think sucks, and an option for something I think sucks, we should just pick something that sucks the least. :)18:46
SpamapSI'm really o-k with one venv, as long as we all agree per-project venv isolation is too expensive.18:47
*** jprovazn has quit IRC18:47
SpamapSBecause if it is too expensive in CI, it is too expensive in CD environments.18:48
dprinceSpamapS: I don't think your solution sucks. It is on the clever side though and perhaps could lead to unintended consequences if we mess w/ hard linked venvs too much after the fact18:48
SpamapSyeah I don't like clever things either18:48
*** bauzas has quit IRC18:48
SpamapSI'm hoping to distill it down to something simpler.18:48
SpamapSRealistically the mechanics are already in virtualenv.18:49
dprinceSpamapS: new topic. ensure-bridge. https://review.openstack.org/#/c/69918/. Based on your comment if I spin up an Ubuntu seed VM and test it I'll see your error?18:49
SpamapSI just need it to work like --system-site-packages, but linking to another virtualenv instead of the system's site packages.18:49
SpamapSdprince: yes I would expect that.18:50
SpamapSdprince: I did not dig into the actual files to see what the problem is specifically18:51
*** rpodolyaka has quit IRC18:51
*** rpodolyaka has joined #tripleo18:52
dprinceSpamapS: I'll give it a go. Haven't tried Ubuntu yet but Fedora works quite nicely w/ this patch18:52
SpamapSdprince: If you give me a simple test I'm happy to test on Ubuntu as wel18:54
SpamapSreminder: meeting in 6 minutes18:54
dprinceSpamapS: the test is does it work!18:55
dprinceSpamapS: I need to push another iteration anyway so let me see what I can do18:55
SpamapSdprince: ah, so in theory CI will provide enough information that we can debug?18:57
dprinceSpamapS: for this one, yes18:58
dprinceSpamapS: there are some finer points though. Previously after ensure bridge ran you would have (I think) a dhclient on both the physical NIC and the OVS bridge. Which isn't what you want. With this you should just end up w/ a dhclient on the bridge alone.18:59
dprinceSpamapS: at least that is what I see on Fedora18:59
SpamapSdprince: there are a few things that I feel like we should do that are CI specific.. like asserting that there is only a dhclient on the bridge.19:00
dprinceSpamapS: plus it is nice to test a reboot and see if it works after this refactoring (it should)19:00
*** lblanchard has quit IRC19:00
SpamapS-> meeting19:00
*** sgrasley1 has left #tripleo19:02
*** rcarrillocruz has joined #tripleo19:04
*** rcarrillocruz1 has quit IRC19:06
*** sballe_ has quit IRC19:08
*** petertoft has joined #tripleo19:11
*** lazy_prince has quit IRC19:14
*** ganetin has joined #tripleo19:15
*** lazy_prince has joined #tripleo19:23
*** lazy_prince is now known as killer_prince19:23
*** ganetin has quit IRC19:24
rbradylooking for reviews of cinder element backend changes: https://review.openstack.org/#/c/74563/19:24
*** pblaho has joined #tripleo19:26
*** SpamapS has quit IRC19:26
*** SpamapS_ has joined #tripleo19:29
*** rcarrillocruz1 has joined #tripleo19:30
*** SpamapS_ is now known as SpamapS19:30
*** jp_at_hp has quit IRC19:30
*** e0ne has joined #tripleo19:31
*** rcarrillocruz has quit IRC19:31
*** ifarkas has quit IRC19:33
* tchaypo grumbles about smartbuy giving a 40419:34
lifelessstupidbuy ?19:35
*** bauzas has joined #tripleo19:36
tchayposo I'd like to start poaching some bugs and smacking them19:42
*** voskmaxx has joined #tripleo19:43
tchaypobut before i start smacking bugs I've been trrying to get a stable test environment so that i don't waste time wondering if the breakage i see is related to the change I jus tmade or not19:43
tchaypobut on my devtest box under my desk, every run produces strange and unusual breakages.19:44
giulivotchaypo, maybe one of the breakages is that the undercloud doesn't boot as in the seed neutron complains about "no dhcp agents" ?19:45
SpamapStchaypo: are you cherry picking the already known needed fixes?19:45
tchaypoI may have seen that one last night19:45
*** dprince has quit IRC19:46
tchaypoSpamapS: we have known needed fixes?19:46
SpamapStchaypo: see topic19:46
*** voskmaxx has quit IRC19:46
SpamapStchaypo: any time there are known devtest breakers you can expect to see them there (if not, add them. ;)19:46
tchaypoone of the known needed fixes is that I need to buy myself some peripherals to make this hp laptop useable19:46
tchaypostarting with a microsoft keyboard19:47
*** e0ne has quit IRC19:47
*** morganfainberg_Z is now known as morganfainberg19:48
tchaypoah, you mean the FIREDRILL bit?19:49
*** rcarrillocruz has joined #tripleo19:49
tchaypoI've been ignoring that because i thought it was saying that was a critical bug that needed work done, and I wanted to get a comfortable working environment before I start looking at them.19:49
tchaypobut there's a bit of chicken-and-egg there19:49
*** rcarrillocruz1 has quit IRC19:50
SpamapStchaypo: that is a patch that, without, you will surely not finish devtest.19:51
tchaypoi believe i figured out yesterday that if I cherry-pick that into ~/.cache/tripleo/tripleo-image-elments, devtest will leave that repo alone when it's doing its update, right?19:52
* tchaypo feels like such a noob19:53
SpamapStchaypo: correct19:53
SpamapStchaypo: yes, you noob, you shoul understand all 17 ways we have to get a patch inserted into our 2 hour developer setup process. _SHEESH_19:54
*** petertoft has quit IRC19:54
tchaypoI alredy know about the DIB_REPO* variables thanks to writing the README.md on derekh's patch, so i tried to use them the other day to make it use a patch for DIB itself19:56
tchaypomuch facepalm ensued when I realised the problem with that19:57
*** d0ugal has joined #tripleo19:58
*** d0ugal has quit IRC19:58
*** d0ugal has joined #tripleo19:58
*** petertoft has joined #tripleo19:59
*** markmc has joined #tripleo19:59
tchaypomy other question is about what i need to do to progress https://review.openstack.org/#/c/77048/20:01
*** pblaho has quit IRC20:01
*** jistr has quit IRC20:02
tchaypofirstly - can i just ignore jenkins? the check that failed doesn't seem to be gating (if I understand the term correctly) so it's got a +1 from jenkins anyway. Sicne this is just a doc change, and given the state of CI, it feels like it's okay to ignore that and charge on20:02
tchaypobut secondly - I'm going to need a +2 in order to make it land20:03
devanandalifeless: woops, ignore the ping in -meeting. meant to ping you here and wave there, ...20:04
devanandalifeless: do you want to continue? or resume after meetings?20:04
SpamapShas anyone else seen the keystone auth fail thing?20:04
SpamapSwe have an out of date keystone-paste.ini which I think may be causing it20:05
bnemecSpamapS: I think I saw something similar yesterday, but I was messing around with some cherry-picked patches locally so I wasn't sure if it was my fault.20:05
bnemecAnd I'm not sure it was exactly the same failure.20:06
lifelessdevananda: after meetings20:06
lifelessslagle: SpamapS: thinking we should fast track https://review.openstack.org/#/c/79502/20:06
lifelesssince it may fix the couldn't-get-metadata-for-IP issue, and simultaneously unbreak overcloud jobs20:06
*** rpodolyaka has quit IRC20:07
lifelessit failed due to glance20:07
SpamapSlifeless: you got my +220:07
SpamapSlifeless: do you think it might be a reason for the glance job's failure?20:07
SpamapSbecause I see a commit that landed changing keystone-paste.ini ..20:07
SpamapSrelated to tokens/revocation..20:07
openstackgerritClint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: Use upstream/package keystone-paste.ini.  https://review.openstack.org/7975020:08
SpamapSthese paste.ini files are ridiculous20:08
SpamapSthey're effectively software20:08
lifelessthey shouldn't be in /etc then20:09
lifelessfile bugs20:09
openstackgerritA change was merged to openstack/tripleo-incubator: Destroy all testenv domains  https://review.openstack.org/7950220:09
SpamapSlifeless: would you think less of me if I said I'm a little tired of filing bugs like that, sort of like I eventually get tired of throwing pebbles back in the sea?20:10
lifelessSpamapS: no, I would sympthasise and tell you to flame the list20:10
SpamapSOk so I need to test 79631 + 79750 ...20:11
SpamapSif I make one dep on the other will CI test them together?20:11
SpamapSdur.. of course it will20:11
openstackgerritClint "SpamapS" Byrum proposed a change to openstack/tripleo-image-elements: List filesystem and swift as known glance stores  https://review.openstack.org/7963120:12
SpamapSok.. so that CI run should pass if the keystone thing is in fact what I think it is.20:12
*** vkozhukalov has quit IRC20:13
*** rcarrillocruz has quit IRC20:13
* bnemec subscribes to the review20:14
SpamapSOk, while it runs I think I need a little TF2 style carnage.20:15
*** rcarrillocruz has joined #tripleo20:16
*** hashar has joined #tripleo20:17
bnemecTF2 has much more fun criticals than we do. :-)20:17
*** andreaf has joined #tripleo20:20
tchaypoSo if you're looking for something fare more exciting than TF220:21
*** andreaf has quit IRC20:21
tchaypolifeless was talking to me last week about wanting to have a more robust way of bootstrapping things like galera20:22
tchayporather than nominating one node as the master and building it first, using a consensus protocal such as paxos to let the nodes decides on a master for themselves, and if that master dies during the build, just elect another one20:23
slagleSpamapS: were you saying you thought git format-patch -1 bddd2908b3f9758a8c6be9594599200101f5d7bb20:23
tchayposo this led to me looking at python paxos implementations, particularly concoord, and it made me sad20:23
slaglegah, copy/paste fail20:23
* slagle tries again20:23
slagleSpamapS: were you saying you thought https://review.openstack.org/#/c/79750/ was a fix for the "Authentication Required" bug I filed?20:24
tchaypoconcoord doesn't bother using paxos to elect a leader, it just uses the node with the lowest port number. pragmatically speaking that might work fine most of the time...20:24
tchaypowhen nodes go away, instead of saying "oh no, i don't have quorum any more", it just shrinks its idea of what the quorum is - so if there's segmentation, both sides will happily elect a leader and keep accepting proposals20:25
greghaynestchaypo: is that split brain documented somewhere?20:25
greghaynesseems like a big issue20:25
tchaypogreghaynes: no, it's something i noticed from debug logs and then confirmed by looking at the code20:26
tchaypoits idea of quorum is "number of currently live nodes /2 + 1"20:26
*** jprovazn has joined #tripleo20:26
greghaynesthats normal20:26
*** weshay has quit IRC20:26
*** rupsky has quit IRC20:27
* greghaynes tries not to get nerdsniped by things on the backburner20:27
tchaypoit's not what i expected20:27
*** rlandy has quit IRC20:27
lifelesstchaypo: this is very odd :)20:27
tchaypoit leads to the split-brain thing20:27
lifelesstchaypo: perhaps raft is better?20:27
lifelesstchaypo: either way we're going to be a stte machine on top of the substrate20:27
lifelesstchaypo: I think we probably want to write up that state machine into the etherpad20:28
tchaypobecause segmentation leads to both sides just discarding all the nodes that aren't talking to them any more and shrinking their quorum... when what I'd like is for the smaller half of the segmentation to realise they don't have a quorum and sit around waiting for a reconnect20:28
tchaypolifeless: I was hoping to avoid that, but I'm starting to think it's inevitable.20:29
greghaynesyea, that seems silly20:29
tchaypoWriting a consensus protocol on top of a consensus protocol..20:29
tchaypoI found the live chat and asked about smartbuy, they gave me an entry to put in /etc/hosts20:29
greghaynesoh yes20:29
tchaypoif only there was some way of distributing these hostname/ip mappings20:30
*** rupsky has joined #tripleo20:30
lifelesstchaypo: sbesp.houston.hp.com ? yeah, epic DNS fail20:30
tchaypoI swear I poked at py-raft last week, but since I don't remember anything I can only assume that it must have been on sunday20:31
*** rcarrillocruz has quit IRC20:31
tchaypogreghaynes: it sounds like you've got some prior experience with concoord?20:32
tchaypojust paxos in general?20:33
greghaynesI do have a couple papers in my backlog before I dive into that, though20:33
greghaynesnot much, just have had to interface with a couple services that do similar things before20:33
lifelesstchaypo: whats the etherpad again ?20:34
tchayposo if we're building state on top of an underlying consensus mechanism20:36
tchaypomaybe it doesn't matter if the underlying implementations' idea of quorum is a little strange20:36
tchaypoif heat is telling the nodes "There's six of you", that can be fed into our state machine20:37
tchaypoand we can do the quorum checking ourselves20:37
*** d0ugal_ has joined #tripleo20:38
lifelesstchaypo: ugh no lets avoid that sort of insanity :)20:39
*** d0ugal has quit IRC20:39
* SpamapS feels better now that he sploded stuff20:39
lifelesstchaypo: specifically, lets have a simple state machine that has amongst its possible transitions 'new master' for every input state20:40
*** julim has quit IRC20:40
SpamapSslagle: yes I think that may be it.20:40
SpamapSslagle: the undercloud keystone log has some refused API calls in it20:40
*** valerii has joined #tripleo20:43
*** d0ugal_ has quit IRC20:43
tchayposo my lateset devtest failed because...20:44
tchaypo"given size (21504) exceeds max allowable size"20:44
tchaypogreghaynes: didn't you have this happen yesterday?20:44
*** d0ugal_ has joined #tripleo20:45
SpamapStchaypo: more context?20:45
*** valerii has quit IRC20:47
slagletchaypo: https://review.openstack.org/#/c/79344/ should fix that20:49
slagleSpamapS: cool, thx20:49
SpamapSslagle: I'm running up a local overcloud from scratch here to confirm that20:50
lifelesstchaypo: see the bottom of the etherpad20:50
greghaynesI overclouded from scratch and that worked20:53
greghaynesearlier today20:53
greghayneswell, that + the glance fix20:54
*** jdob has quit IRC20:58
tchaypohttp://paste.openstack.org/show/73190/ has more context20:59
tchaypothanks slagle and greghaynes, that looks like it's it21:00
*** e0ne has joined #tripleo21:00
*** petertoft has quit IRC21:02
jprovazngreghaynes, hi, I tried the percona patch today, did you hit any memory issue? On my devtest setup (fedora 64bit) after a short time mysql allocates ~700M memory and any openstack command fails because of not enough memory21:03
greghayneseek, no I did not21:03
jprovazngreghaynes, it seems that some memory optimization in my.cnf would be required21:03
lifeless700M. yay.21:04
jprovazngreghaynes, will try look at this deeper tomorrow, but if you find anything, let me know please21:04
greghaynesok, building with that patch now and will mess with it21:04
jprovazngreghaynes, thanks21:04
tchaypolifeless: i think we also need to handle a couple of transitions - having a replica suddenly elected the new master, and having a master re-connect after a split and get demoted to a replica21:06
tchaypobut I guess the specifics of how to handle that will depend on what it is that we're deploying21:07
*** rpodolyaka has joined #tripleo21:09
lifelesstchaypo: so, walk through the state transition for that. What state are we in, what happens. Perhaps in the etherpad to capture it21:15
*** e0ne has quit IRC21:16
*** e0ne has joined #tripleo21:16
*** jprovazn has quit IRC21:20
*** StevenK_ has joined #tripleo21:25
*** noslzzp has quit IRC21:28
*** ftcjeff has quit IRC21:28
*** morazi has quit IRC21:28
*** tomh_ has quit IRC21:28
*** mkerrin has quit IRC21:28
*** sdake_ has quit IRC21:28
*** rwsu has quit IRC21:28
*** ohadlevy has quit IRC21:28
*** d0ugal_ has quit IRC21:28
*** cwolferh has quit IRC21:28
*** rollerj has quit IRC21:28
*** victor_lowther_ has quit IRC21:28
*** tserong has quit IRC21:28
*** dkehn_ has quit IRC21:28
*** dtantsur has quit IRC21:28
*** tteggel has quit IRC21:28
*** EntropyWorks has quit IRC21:28
*** rainya has quit IRC21:28
*** stevebaker has quit IRC21:28
*** funzo has quit IRC21:28
*** bauzas has quit IRC21:28
*** Matt2 has quit IRC21:28
*** giulivo has quit IRC21:28
*** Kim-Chi-San has quit IRC21:28
*** antonym has quit IRC21:28
*** slagle has quit IRC21:28
*** dguerri has quit IRC21:28
*** sdague has quit IRC21:28
*** smulcahy has quit IRC21:28
*** shortstop has quit IRC21:28
*** markmc has quit IRC21:28
*** CaptTofu has quit IRC21:28
*** spzala has quit IRC21:28
*** jtomasek has quit IRC21:28
*** cody-somerville has quit IRC21:28
*** Kiall has quit IRC21:28
*** lynxman has quit IRC21:28
*** vipul has quit IRC21:28
*** sbadia has quit IRC21:28
*** dkehn has quit IRC21:28
*** lsmola has quit IRC21:28
*** ewindisch has quit IRC21:28
*** Shrews has quit IRC21:28
*** clarkb has quit IRC21:28
*** kevinbenton has quit IRC21:28
*** rbrady has quit IRC21:28
*** e0ne has quit IRC21:28
*** rpodolyaka has quit IRC21:28
*** hashar has quit IRC21:28
*** marun has quit IRC21:28
*** jang1 has quit IRC21:28
*** blamar has quit IRC21:28
*** juice has quit IRC21:28
*** davidlenwell has quit IRC21:28
*** AaronGreen has quit IRC21:28
*** sseago has quit IRC21:28
*** Slower_ has quit IRC21:28
*** yamahata_ has quit IRC21:28
*** rha has quit IRC21:28
*** hewbrocc` has quit IRC21:28
*** sgrasley has quit IRC21:28
*** Hefeweizen has quit IRC21:28
*** jeblair has quit IRC21:28
*** openstackgerrit has quit IRC21:28
*** jpeeler has quit IRC21:28
*** sff has quit IRC21:28
*** SloppyJ_ has quit IRC21:28
*** michchap has quit IRC21:28
*** DuncanT- has quit IRC21:28
*** pleia2 has quit IRC21:28
*** shadower has quit IRC21:28
*** spiffxp_ has quit IRC21:28
*** jcooley has quit IRC21:28
*** jogo has quit IRC21:28
*** gilliard has quit IRC21:28
*** matty_dubs has quit IRC21:28
*** lifeless has quit IRC21:28
*** jesusaurus has quit IRC21:28
*** marios has quit IRC21:28
*** retr0h has quit IRC21:28
*** mordred has quit IRC21:28
*** Daviey has quit IRC21:28
*** lucasagomes has quit IRC21:28
*** greghaynes has quit IRC21:28
*** al has quit IRC21:28
*** boris-42 has quit IRC21:28
*** StevenK has quit IRC21:28
*** wendar has quit IRC21:29
*** adam_g has quit IRC21:29
*** mikal has quit IRC21:29
*** ChanServ has quit IRC21:29
*** morganfainberg has quit IRC21:29
*** cadenzajon has quit IRC21:29
*** Ng has quit IRC21:29
*** ekarlso has quit IRC21:29
*** zaro has quit IRC21:29
*** bnemec has quit IRC21:29
*** anteaya has quit IRC21:29
*** tchaypo has quit IRC21:29
*** jomara has quit IRC21:29
*** phschwartz has quit IRC21:29
*** shardy has quit IRC21:29
*** NobodyCam has quit IRC21:29
*** devananda has quit IRC21:29
*** SlickNik has quit IRC21:29
*** cwolferh has joined #tripleo21:41
*** al has joined #tripleo21:41
*** EntropyWorks has joined #tripleo21:41
*** wendar has joined #tripleo21:41
*** tteggel has joined #tripleo21:41
*** victor_lowther_ has joined #tripleo21:41
*** funzo has joined #tripleo21:41
*** dkehn_ has joined #tripleo21:41
*** tserong has joined #tripleo21:41
*** dtantsur has joined #tripleo21:41
*** adam_g has joined #tripleo21:41
*** stevebaker has joined #tripleo21:41
*** greghaynes has joined #tripleo21:41
*** lucasagomes has joined #tripleo21:41
*** derekh has joined #tripleo21:41
*** boris-42 has joined #tripleo21:41
*** mikal has joined #tripleo21:41
*** weshay has joined #tripleo21:41
*** rpodolyaka has joined #tripleo21:41
*** e0ne has joined #tripleo21:41
*** hashar has joined #tripleo21:41
*** markmc has joined #tripleo21:41
*** bauzas has joined #tripleo21:41
*** marun has joined #tripleo21:41
*** jang1 has joined #tripleo21:41
*** CaptTofu has joined #tripleo21:41
*** cadenzajon has joined #tripleo21:41
*** hewbrocc` has joined #tripleo21:41
*** blamar has joined #tripleo21:41
*** Matt2 has joined #tripleo21:41
*** spzala has joined #tripleo21:41
*** jtomasek has joined #tripleo21:41
*** giulivo has joined #tripleo21:41
*** Kim-Chi-San has joined #tripleo21:41
*** antonym has joined #tripleo21:41
*** retr0h has joined #tripleo21:41
*** morganfainberg has joined #tripleo21:41
*** sgrasley has joined #tripleo21:41
*** Hefeweizen has joined #tripleo21:41
*** cody-somerville has joined #tripleo21:41
*** dguerri has joined #tripleo21:41
*** juice has joined #tripleo21:41
*** davidlenwell has joined #tripleo21:41
*** AaronGreen has joined #tripleo21:41
*** sseago has joined #tripleo21:41
*** Slower_ has joined #tripleo21:41
*** jeblair has joined #tripleo21:41
*** dickson.freenode.net sets mode: +v davidlenwell21:41
*** slagle has joined #tripleo21:41
*** ekarlso has joined #tripleo21:41
*** openstackgerrit has joined #tripleo21:41
*** jpeeler has joined #tripleo21:41
*** sdague has joined #tripleo21:41
*** smulcahy has joined #tripleo21:41
*** shortstop has joined #tripleo21:41
*** lynxman has joined #tripleo21:41
*** vipul has joined #tripleo21:41
*** yamahata_ has joined #tripleo21:41
*** sbadia has joined #tripleo21:41
*** dkehn has joined #tripleo21:41
*** sff has joined #tripleo21:41
*** lsmola has joined #tripleo21:41
*** Ng has joined #tripleo21:41
*** SloppyJ_ has joined #tripleo21:41
*** ewindisch has joined #tripleo21:41
*** Shrews has joined #tripleo21:41
*** gilliard has joined #tripleo21:41
*** rha has joined #tripleo21:41
*** michchap has joined #tripleo21:41
*** clarkb has joined #tripleo21:41
*** kevinbenton has joined #tripleo21:41
*** DuncanT- has joined #tripleo21:41
*** rbrady has joined #tripleo21:41
*** zaro has joined #tripleo21:41
*** pleia2 has joined #tripleo21:41
*** bnemec has joined #tripleo21:41
*** shadower has joined #tripleo21:41
*** spiffxp_ has joined #tripleo21:41
*** jcooley has joined #tripleo21:41
*** jogo has joined #tripleo21:41
*** anteaya has joined #tripleo21:41
*** tchaypo has joined #tripleo21:41
*** jomara has joined #tripleo21:41
*** matty_dubs has joined #tripleo21:41
*** lifeless has joined #tripleo21:41
*** jesusaurus has joined #tripleo21:41
*** marios has joined #tripleo21:41
*** mordred has joined #tripleo21:41
*** Daviey has joined #tripleo21:41
*** phschwartz has joined #tripleo21:41
*** ChanServ has joined #tripleo21:41
*** shardy has joined #tripleo21:41
*** NobodyCam has joined #tripleo21:41
*** devananda has joined #tripleo21:41
*** dickson.freenode.net sets mode: +vvo Ng lifeless ChanServ21:41
*** SlickNik has joined #tripleo21:41
*** rainya has joined #tripleo21:41
tchaypolifeless: the machine; have it pxe-boot again and start the build fresh.21:41
*** kiall has joined #tripleo21:42
lifelesstchaypo: I believe I have handled that case in the state machine21:42
tchaypowhen you say "layering violation" i think you're talking about the fact that the system that formerly was just electing a master is suddenly reaching down a layer or two and telling heat to reset the host it's on?21:42
derekh+ os-svc-enable -n rsync21:43
derekhFailed to issue method call: No such file or directory21:43
*** rollerj has joined #tripleo21:44
tchaypojust before my session times out again - if i order an SSD from smartbuy, will it come with a caddy to put it in the upgrade bay? I can't find those listed seperately21:45
lifelesstchaypo: no idea :)21:45
*** sdake_ has joined #tripleo21:46
*** petertoft has joined #tripleo21:46
*** rwsu has joined #tripleo21:46
*** giulivo has quit IRC21:46
tchaypolifeless: then it seems like I don't understand what you've written. I think states are down the left, transitions across the top?21:46
lifelesstchaypo: yes, so consider what happens if a node being configured as master dies21:47
tchaypoand the entries in the cells indicate that you transition from the state on the left via the transition at the top to the state in the cell?21:48
openstackgerritBen Nemec proposed a change to openstack/tuskar: Remove tools/flakes.py  https://review.openstack.org/7978321:48
lifelesscode to run on the master (can influence transitions) and slaves in each state is listed below21:48
tchaypogive me a second to draw this up a different way to make sure I have it in my head21:49
*** morazi has joined #tripleo21:49
*** petertoft has quit IRC21:50
*** e0ne has quit IRC21:57
*** markmc has quit IRC21:57
*** petertoft has joined #tripleo21:59
rbradylooking for reviews of https://review.openstack.org/#/c/74563/22:00
*** hashar has quit IRC22:01
tchaypolifeless: I've added a graphviz version of the state machine, which I prefer because it makes a pretty picture that my brain can understand22:03
*** cadenzajon has quit IRC22:03
derekhfrick, pretty much all of the ci jobs simultaneously failed about 30 minutes ago (network blip??)22:05
lifelesstchaypo: dote ?22:08
tchaypolifeless: well done, you've now conflated the concepts of "dot" and "doge" in my head.22:10
tchaypoVery digraph, much acyclic, wow22:10
*** StevenK_ is now known as StevenK22:10
SpamapS2014-03-11 21:33:28.560 | Updating cache of https://github.com/kanaka/noVNC.git in /home/jenkins/.cache/image-create/source-repositories/novnc_eac807d7dcef19baf4dd80e1f38fc67a7720c193 with ref master22:10
SpamapS2014-03-11 21:33:43.049 | error: The requested URL returned error: 403 while accessing https://github.com/kanaka/noVNC.git/info/refs22:11
tchaypolifeless: not sure if you mean "what is this dot thing, I have not heard of it" or if you mean "when you say graphviz, do you mean dot?"22:11
lifelesstchaypo: I don't have a dote command, and your prose says to use it22:11
tchaypoSpamapS: that's one i saw yesterday, and I can't figure out why. It's worked for me after that, and it works fine in my browser22:11
tchaypolifeless: well done me. fixed22:12
SpamapStchaypo: likely rate limiting or just intermittent github fail22:12
tchaypookay, my female cat seems to be attempting to mount my male cat. I don't know what to think.22:13
*** rupsky_ has joined #tripleo22:13
SpamapSmaybe we should d/l the tarball instead of git cloning it22:13
jang1as long as it's not "youtube needs to see this", tchaypo22:14
*** rupsky has quit IRC22:14
tchaypolifeless: *really* fixed, now that etherpad has decided to reconnect22:15
*** jang1 has quit IRC22:16
tchaypolifeless: what I think is missing from that state machine is some of the initialisation. I think we need to start in a "Waiting for quorum" state, which has an "election" transition leading to the "uninitialised" state (which has seperate code paths for master and replicas)22:18
*** matty_dubs is now known as matty_dubs|gone22:19
lifelesstchaypo: so I'm trying to focus on the programming model we need, not on the consensus tech22:19
lifelesstchaypo: for the avoidance of confusion, do you mean 'wait for consensus quorum' or 'wait for db/etc/thing quorum' ?22:20
tchaypoi mean "wait until we have >1/2 the nodes heat says are coming are alive". I think that whether that is the same thing as "consensus quorum" or whether it has to be implemented on a layer above that is an implementation details22:22
bnemecrbrady: Did your change to fix the duplicate RabbitPassword in the block-storage template get lost?  I'm still getting that error with your latest change and I don't see any other pending reviews.22:23
lifelesstchaypo: it doesn't, because you can't start running the replicated state machine until consensus is reached22:24
tchaypoyep; so we assume that the underlying consensus mechanism is handling that, and our state machine won't exist until that's true22:28
tchaypohence waiting for quorum, electing quorum leader etc is outside the scope of this state machine22:28
lifelessWe may well need to hack on that consensus plumbing22:28
lifelessbut I would like to be able to map what we come up with into e.g. taskflow22:28
lifelesswhich suggesets separating concerns carefully22:29
tchaypomakes sense22:29
tchaypoWhy is Quorum replicas going to master inited via new master?22:29
tchaypowouldn't that transition happen because the inited master has gone away, so the new master has to start its initialisation again? ie, shouldn't it be back to uninitialised?22:30
*** andreaf has joined #tripleo22:31
lifelessok so say we hve 3 nodes22:33
lifelessA B C22:33
lifelessA and B come online ok22:33
lifelessand we get a leader (A)22:34
lifelessB gets a full state vector sync22:34
lifelessA fails22:34
lifelesswe're now new leader (and thats B)22:34
lifelessA is down, B is up with a full replica (but isn't quorate), and C is still not up22:35
ccrouchSpamapS: from todays meeting, who has the ball around answering this question?22:39
ccrouch19:59:19 <SpamapS> - There is a question as to whether TripleO's gate jobs can make use of an incubated but not integrated project, like Ironic, that needs answering.22:39
*** openstackstatus has joined #tripleo22:44
SpamapSccrouch: good question we ended abruptly w/o actions22:46
*** petertoft has quit IRC22:49
*** CaptTofu has quit IRC22:51
lifelesstchaypo: so - I think we should let this mull and evolve and focus on things that affect more than initial deploy for now23:00
lifelesstchaypo: since its fairly crunch time23:00
openstackgerrittom-howley proposed a change to openstack/tripleo-incubator: Add standadlone script to build all images.  https://review.openstack.org/7746223:07
*** rpodolyaka has quit IRC23:09
openstackgerrittom-howley proposed a change to openstack/tripleo-incubator: Add standalone script to build all images.  https://review.openstack.org/7746223:09
*** rpodolyaka has joined #tripleo23:12
tchaypolifeless: agree23:14
tchaypoI'd say that in the scnario above we don't have a new leader. B can't know if it's really the only survivor, or if A and C are talking but segregated from it23:15
*** lucasagomes has quit IRC23:16
tchaypoIf C starts talking to B, *then* we have a new leader, and it makes sense to elect B rather than C (election protocol probably needs to weight "full state vector" highly)23:16
lifelesstchaypo: right, so the state machine pauses23:16
lifelesstchaypo: then if A comes back either A becomes leader, or B becomes leader.23:17
lifelesstchaypo: so there you want to make sure the thing under intiialisation hits quorum itself, and then move forward a state23:17
lifelesstchaypo: but if A doesn't come back and C comes back,  thats when B || C becomes leader.23:17
lifelesstchaypo: if B becomes the leader. we have one replica and need to wait for C (slave) to sync from B; if C becomes leader I'm proposing we drop back down to unitialised (so that we don't have to deal with a master that is not leader23:18
*** shakayumi has joined #tripleo23:19
lifeless(see line 103 of the etherpad)23:19
* tchaypo is certified ethically compliant and stuff23:20
*** andreaf has quit IRC23:21
openstackgerrittom-howley proposed a change to openstack/tripleo-incubator: Add standalone script to build all images.  https://review.openstack.org/7746223:27
SpamapSbtw we should not use 'slave' and 'master' .. it is insensitive. dom and sub have far less negative cultural context.23:29
tchaypoI swear I saw a van that said "S&M Cleaning" while I was in Sunnyvale23:30
SpamapSWe don't clean the house, BUT WE WILL MAKE SURE YOU DO23:30
tchaypo... come to think of it, around SF, that kind of niche is probably big enough to attract one or two businesses23:31
SpamapSwho you calling "niche" ;)23:31
tchaypoIn other news23:33
tchaypoI justt discovered https://code.google.com/p/canviz/23:33
tchaypoand now I'm imagining a graphviz-pastebin23:33
tchaypopaste your text, get a url to a rendered graph23:33
SpamapStchaypo: lovely23:35
lifelesstchaypo: like http://g.jk.gs/ ?23:36
tchaypoalmost exactly like that23:38
SpamapS2014-03-11 23:35:22.803 | Waiting for the overcloud stack to be ready23:38
tchaypoexcept that rather than a static .png I'd like the graph-source to be an etherpad.23:38
tchaypowhen i get to that bit, i tend to start opening a second window and watching logs and checking heat/nova output by hand23:38
tchayposince the scripts don't seem to notice failures23:39
tchaypohttps://review.openstack.org/#/c/79344/ seems to be breaking lots of things, mayhap it should be listed as FIREDRILL as well?23:41
SpamapStchaypo: yes, we need CI to do that23:41
*** rupsky_ has quit IRC23:41
SpamapStchaypo: what? More fail? I can't take more fail.23:42
tchayponeither can I, but it's been 5 hours since I started work so I'm stepping away. Going to go on a bit of an officeworks binge - need to get myself a new keyboard, a laptop stand, and maybe some kind of desk organizer thing23:43
tchaypoopenstack-infra mailing list has someone asking about kanaka/noVNC23:45
tchaypowhich mordred has responded to already, very good23:48
*** dividehex has joined #tripleo23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!