Friday, 2016-02-26

« Thursday, 2016-02-25 Index Saturday, 2016-02-27 »
*** chlong_ has quit IRC00:03
openstackgerritMerged openstack/sahara: Making health verification periodics distributed  https://review.openstack.org/28468800:49
openstackgerritMerged openstack/sahara: implement sending health notifications  https://review.openstack.org/28119401:23
*** chlong_ has joined #openstack-sahara01:49
*** links has joined #openstack-sahara03:26
*** Poornima has joined #openstack-sahara04:26
openstackgerritJaxon Wang proposed openstack/sahara: Refine the code for CDH PluginUtils class  https://review.openstack.org/25081005:08
openstackgerritJaxon Wang proposed openstack/sahara: CDH plugin config helper refactoring  https://review.openstack.org/25582505:46
openstackgerritJaxon Wang proposed openstack/sahara: Update CDH user doc for CDH 5.5.0  https://review.openstack.org/28167005:46
openstackgerritJaxon Wang proposed openstack/sahara: CDH plugin edp engine code refactoring  https://review.openstack.org/25730905:46
openstackgerritJaxon Wang proposed openstack/sahara: Add CDH 5.5 support  https://review.openstack.org/27996405:46
openstackgerritJaxon Wang proposed openstack/sahara: Update CDH user doc for CDH 5.5.0  https://review.openstack.org/28167006:01
openstackgerritJaxon Wang proposed openstack/sahara: CDH plugin edp engine code refactoring  https://review.openstack.org/25730906:01
openstackgerritJaxon Wang proposed openstack/sahara: Add CDH 5.5 support  https://review.openstack.org/27996406:01
*** sgotliv has joined #openstack-sahara06:14
*** links has quit IRC06:26
*** sgotliv has quit IRC06:33
*** sgotliv has joined #openstack-sahara06:33
*** nkrinner has joined #openstack-sahara06:37
openstackgerritJaxon Wang proposed openstack/sahara-specs: Add new spec for NFS-as-a-data-source blueprint  https://review.openstack.org/21083907:03
*** rcernin has joined #openstack-sahara07:05
*** sgotliv has quit IRC07:10
openstackgerritJaxon Wang proposed openstack/sahara-specs: Add new spec for NFS-as-a-data-source blueprint  https://review.openstack.org/21083907:16
*** Haomeng has quit IRC07:19
*** Haomeng has joined #openstack-sahara07:20
*** chlong_ has quit IRC07:26
*** vgridnev has joined #openstack-sahara08:01
*** vgridnev has quit IRC08:05
*** witlessb has joined #openstack-sahara08:21
openstackgerritNikita Konovalov proposed openstack/sahara: HA for NameNode and ResourceManager in HDP 2.2  https://review.openstack.org/19755108:28
openstackgerritLuigi Toscano proposed openstack/sahara: Use the integrated tempest.lib module  https://review.openstack.org/28515908:39
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Add check of scaling for CDH and Ambari  https://review.openstack.org/27467508:47
*** vgridnev has joined #openstack-sahara08:56
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Add pylint to tox  https://review.openstack.org/28517309:10
*** esikachev has joined #openstack-sahara09:12
*** rcernin has quit IRC09:14
*** tellesnobrega is now known as tellesnobrega_af09:20
*** esikachev has quit IRC09:24
*** rcernin has joined #openstack-sahara09:28
openstackgerritLuigi Toscano proposed openstack/sahara-ci-config: Temporary install tempest-lib in tox venv  https://review.openstack.org/28518609:35
openstackgerritMerged openstack/sahara: Updated from global requirements  https://review.openstack.org/28506909:55
*** _degorenko|afk is now known as degorenko10:16
openstackgerritLuigi Toscano proposed openstack/sahara: Use the integrated tempest.lib module  https://review.openstack.org/28515910:25
*** vgridnev has quit IRC10:27
*** vgridnev has joined #openstack-sahara10:31
openstackgerritMerged openstack/sahara-dashboard: Changing name of "Data image registry"  https://review.openstack.org/28325910:33
openstackgerritMerged openstack/sahara-dashboard: Use oslo timeutils for durations  https://review.openstack.org/28469710:36
openstackgerritMerged openstack/sahara-dashboard: Job duration column added  https://review.openstack.org/27398110:36
*** tosky has joined #openstack-sahara10:43
toskysorry for the mess with the reviews, just trying to find the most reliable way to adapt to the changes in tempest/tempest-lib10:43
*** sgotliv has joined #openstack-sahara10:49
*** sgotliv has quit IRC11:08
*** Poornima has quit IRC11:25
*** vgridnev has quit IRC11:44
*** vgridnev has joined #openstack-sahara11:51
*** krotscheck_dcm is now known as krotscheck12:13
openstackgerritNikita Konovalov proposed openstack/sahara-dashboard: Uptime column for clusters  https://review.openstack.org/28526212:13
*** raildo-afk is now known as raildo12:32
openstackgerritMerged openstack/python-saharaclient: Updated from global requirements  https://review.openstack.org/28506712:53
*** itisha has joined #openstack-sahara13:15
openstackgerritAndrey Pavlov proposed openstack/python-saharaclient: Adding "health verification --show" CLI call  https://review.openstack.org/28529513:22
*** AndreyPavlov has joined #openstack-sahara13:22
*** _crobertsrh is now known as crobertsrh13:25
*** Akanksha08 has quit IRC13:42
openstackgerritChad Roberts proposed openstack/sahara: Updating dashboard user guide post-reorg  https://review.openstack.org/28488613:46
*** nkrinner has quit IRC14:26
*** tmckay has joined #openstack-sahara14:33
rickflaremorning Sahara studs!!!!!14:38
rickflareI tried several times last night to deploy my HDP cluster and I keep getting the following error from heat Error Authentication Failed  Heat_Include_Password=114:40
vgridnevhey, rickflare14:40
vgridnevwhat version of sahara you are using?14:40
rickflaremorning vgridnev how are you?14:40
rickflarelet me check14:40
rickflareI am running liberty14:40
vgridnevAh, got it14:41
vgridnevI'm nice14:41
tmckayrickflare, hmm, could it be the role?14:41
rickflareopenstack-sahara-common-3.0.0-5.cc218ddgit.el7.noarch14:41
tmckayI have seen problems when the user I am using to launch the sahara cluster does not have "heat_owner"14:41
vgridnevso, it looks like heat stack was not moved to CREATE_COMPLETE in one hour14:41
tmckayI forget exactly how to spell heat owner14:41
rickflarerole?14:42
rickflaretmckay14:42
rickflareI am using the admin account under the admin project14:42
rickflareif that helps14:42
* tmckay checks something on his liberty packstack install14:42
vgridnevrickflare, could you please execute heat stack-list --show-hidden?14:43
rickflarehttp://pastebin.com/jxt7dvEA14:44
rickflarenot sure why my hdp cluster does not show up here14:44
toskyrickflare: openstack user role list -> does admin have heat_stack_owner?14:44
tmckayrickflare, just a guess based on errors I've seen -- if you source the admin rc file and do "keystone user-role-list" does it say the admin user has "heat_stack_owner" and "heat_stack_user" ?14:44
tmckayor what tosky said :)14:45
toskytmckay: future-proof change: "keystone" CLI client is deprecated :)14:45
tmckaywell, just heat_stack_owner for me ...14:45
tmckaytosky, yes, yes, you're right :)14:45
rickflarehttp://pastebin.com/L1M2yUDQ14:45
tmckayI don't like change ;-) wrong business14:46
tmckayok, that looks alright14:46
rickflareI dont know what to do14:50
toskyrickflare: can you please post more from your heat logs (for api and engine), paying attention to sanitize them from confidential details? Especially when you start the cluster14:50
rickflarenot sure what to grap tosky14:51
toskyrickflare: stop heat, clean heat logs (/var/log/heat/heat-*.log), restart heat, run a cluster, wait for failure, get the heat logs14:53
*** vgridnev has quit IRC14:54
elmikotmckay: lol, don't like change, works on cutting edge software ;)14:54
toskyreading around, that error could be just a red herring, coming from the timeout, and the real issue could happen before14:54
rickflarewhoa14:55
rickflarejust got some strange errors14:55
elmikoyou mean, more strange errors ;)14:55
rickflarethis is on the engine restart14:55
rickflarehttp://pastebin.com/p8ZQ5FP614:55
rickflareelmiko!14:55
rickflareits Friday14:55
rickflareWHOOOOOOOOOO!14:55
elmiko\o/14:55
elmikowow, zaqar, mistral, *and* designate. going for a full openstack buffet eh?14:56
toskyI wonder why the import error in heat - are there references in the configuration? Maybe you installed them, then removed but the entrypoint stayed14:57
*** vgridnev has joined #openstack-sahara14:57
rickflarethose python errors14:59
rickflarethough?14:59
*** vgridnev has quit IRC14:59
elmikothose are stevedore errors though, so it might be something related to the dynamic loading of those plugins15:00
elmikolike, some config file is calling for their inclusion15:00
*** vgridnev has joined #openstack-sahara15:00
rickflareso the package python-zaqarclient is not installed on my system15:01
rickflarei just installed it15:01
toskydid you install it at some point in the past, through package or pip?15:02
*** vgridnev has quit IRC15:04
rickflarewhat python rpm provides mistralclient.api15:05
rickflaretosky nope15:05
elmikopresumably python-mistralclient, but i'm not sure if rdo has packaged that one. tosky you know?15:05
rickflareits not in yum15:06
rickflareand heat is bitching about it15:06
elmikoseems weird for rdo to configure these extra packages15:06
toskynot here http://cbs.centos.org/repos/cloud7-openstack-liberty-release/x86_64/os/Packages/15:06
elmikoi'm not even sure if any of those(zaqar,mistral,designate) have been included for GA yet15:06
tmckayelmiko, btw, this one needs another +2, on the ff list https://review.openstack.org/#/c/284653/15:06
tmckaylooks like all of the health check stuff has passed except for "tempest tests", failing on multiple (probably broken)15:07
toskyrickflare: I wonder what happened then; how the heat initialization tries to find them15:07
elmikotmckay: yup, it's on my list just didn't get to it last night :/15:07
tmckayk, thanks :)15:07
toskytmckay: I have at least two patches to fix the client scenario tests15:07
elmikoi also want to hit the review you had up, sadly though, i need to go heads down at some point and do something *besides* reviews....15:08
toskybut the gates are horribly slow, so I don't have the results yet15:08
tmckayyay!  File "/usr/local/lib/python2.7/dist-packages/sahara/tests/tempest/scenario/data_processing/client_tests/test_job_binaries.py", line 16, in <module>     from tempest_lib.common.utils import data_utils ImportError: No module named tempest_lib.common.utils15:08
toskycheck my reviews, it's all explained there15:08
tmckaytosky :) will do.15:08
toskyone for sahara, one for sahara-ci-config15:08
rickflarerebuilding the cluster again15:13
rickflarethis should brighting everyones Friday! https://www.youtube.com/watch?v=xk8mm1Qmt-Y15:16
rickflareman HDP takes for ever to spawn15:20
toskyrickflare: do you see other exceptions? And can you run a simple cirros instance?15:22
rickflareyea15:22
rickflarei have stood up a spark cluster15:22
rickflareno prob15:22
*** vgridnev has joined #openstack-sahara15:24
*** vgridnev has quit IRC15:28
*** vgridnev has joined #openstack-sahara15:28
*** vgridnev has quit IRC15:31
rickflarethis is so strange15:33
rickflareif i boot the hdp image15:33
rickflareas a instance15:33
rickflareit boot just fin15:33
rickflarerun it though the cluster15:33
rickflareand its like it hangs15:33
elmikoweird15:35
rickflareoh shit15:35
rickflareDUHHHHHHHH15:35
rickflarei figured it out15:35
elmikopbkac?15:35
rickflarethis is so dumb on my part15:35
rickflarei some how selected my centos image15:35
rickflarelet me test this15:36
rickflarei know my ubuntu image works15:36
rickflarebut the centos one15:36
rickflarenot sure15:36
openstackgerritChad Roberts proposed openstack/sahara: Updating dashboard user guide post-reorg  https://review.openstack.org/28488615:37
rickflareso i am not sure the mirantis hdp images work15:41
toskyrickflare: but hdp 2.0.6 is centos only, do you talk about ubuntu image for another plugin?15:43
rickflareso I used ubuntu for spark15:44
rickflareand that image works fine15:44
rickflarethe image I downloaded from mirantis15:44
rickflareboots15:44
rickflarebut when it gets to the dhcp ip part15:44
rickflareit hands15:44
rickflareand just sits there15:44
toskyiirc it takes a while for resizing15:45
rickflareresizing the disk?15:45
rickflarebut why would it take 20times the time the ubuntu image does15:45
rickflarewell15:45
rickflareit is 1.6gb15:45
rickflarebut15:45
rickflarestill15:45
rickflareseems long15:45
rickflareto me15:46
toskyit should not take so long15:47
rickflaretrying again15:47
toskyif you did run the instance manually (outside the cluster) can you please paste the logs from the console?15:47
rickflarestrange15:48
rickflarethe instance finally booted15:48
rickflareim going to make my clust smaller15:48
rickflareall the instances are getting written to a raid 5 arrasy15:48
rickflaremaybe the large cluster i just overwhelming the IO15:49
rickflareand its hanging15:49
*** vgridnev has joined #openstack-sahara15:49
*** esikachev has joined #openstack-sahara15:50
*** vgridnev has quit IRC15:51
*** vgridnev has joined #openstack-sahara15:53
openstackgerritAndrey Pavlov proposed openstack/sahara: Adding cluster scaling and changing CLI calls in grenade  https://review.openstack.org/25042216:03
*** vgridnev has quit IRC16:10
*** logan- has quit IRC16:14
*** logan- has joined #openstack-sahara16:14
openstackgerritMichael Ionkin proposed openstack/sahara-dashboard: Added Base Image field on Node Group Template form  https://review.openstack.org/25953516:16
*** sgotliv has joined #openstack-sahara16:17
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Fix using proxy node for checks  https://review.openstack.org/27944716:17
*** rcernin has quit IRC16:31
rickflareyup16:31
rickflarethat was it16:31
rickflareit was timing out16:31
rickflarereduced the cluster size16:31
rickflareand seeming to be moving along now16:31
rickflarein the waiting stage16:31
openstackgerritNikita Konovalov proposed openstack/sahara-dashboard: Import reorg and cleanup  https://review.openstack.org/28541116:34
*** tellesnobrega_af is now known as tellesnobrega16:34
toskyrickflare: how big was the cluster?16:35
*** AndreyPavlov has quit IRC16:40
rickflare12 nodes16:41
rickflarex.large instances16:41
rickflareim now 816:41
openstackgerritMerged openstack/sahara-tests: Add more infomation when create cluster failed for scenario test  https://review.openstack.org/28109516:55
openstackgerritMerged openstack/sahara-tests: Use ostestr instead of the custom pretty_tox.sh  https://review.openstack.org/28483116:55
rickflaregeeze16:59
rickflarestill waiting16:59
toskyuhm16:59
*** tmckay is now known as _tmckay16:59
toskymany compute nodes/17:00
tosky?17:00
rickflare617:00
rickflare1 master17:00
*** AndreyPavlov has joined #openstack-sahara17:02
toskyjust to be clear, I was asking about the compute nodes from openstack point of view, not the workers in the hadoop world17:02
rickflareits one massive server17:03
rickflare64 cores17:03
rickflare15TB17:03
rickflare2TB ssd17:03
rickflare500GB ram17:03
rickflareI have no idea17:07
rickflarewhat its doing17:07
rickflareat this point17:07
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Add ability export results to file  https://review.openstack.org/28461117:08
*** ekarlso- has quit IRC17:20
*** krotscheck has quit IRC17:20
*** bapalm has quit IRC17:20
*** krotscheck has joined #openstack-sahara17:20
*** bapalm has joined #openstack-sahara17:21
*** esikachev has quit IRC17:24
*** esikachev has joined #openstack-sahara17:26
*** ekarlso- has joined #openstack-sahara17:27
openstackgerritMerged openstack/sahara-tests: Add pylint to tox  https://review.openstack.org/28517317:31
openstackgerritMerged openstack/sahara-tests: Fix scenario tests for correct output to swift  https://review.openstack.org/27662417:31
*** esikachev has quit IRC17:45
openstackgerritLuigi Toscano proposed openstack/python-saharaclient: Use ostestr instead of the custom pretty_tox.sh  https://review.openstack.org/28546717:54
openstackgerritLuigi Toscano proposed openstack/python-saharaclient: Use ostestr instead of the custom pretty_tox.sh  https://review.openstack.org/28546717:56
*** tosky has quit IRC17:59
*** degorenko is now known as _degorenko|afk18:01
*** sgotliv has quit IRC18:02
*** _tmckay is now known as tmckay18:13
rickflareits still waiting18:25
rickflarewtf18:25
tmckayrickflare, the sahara status "waiting" I believe is sahara waiting to be able to log in to each of the instances.  In the case of hdp, it might be trying to connect to the ambari server.18:44
tmckayrickflare, I'd look at the sahara-engine logs and try to see what sahara thinks it's doing18:45
tmckayone thing to check is whether or not you can ssh into each of the boxes from the controller using the key your provided18:45
tmckayelmiko, do you remember if "waiting" includes connecting to the ambari server? ^^18:46
elmikohmm18:46
elmikoit could18:46
rickflarei can ssh into each node18:46
elmikoi concur with tmckay's advice though, look at the sahara api controller logs to see where it thinks it is stuck at18:47
tmckayrickflare, okay, so somewhere in the sahara-engine log it should say something like "all instances available"18:47
elmikoyou should be able to get a good picture of what stage it is in18:47
tmckayyou could try grepping for "instances.*available", I think18:47
rickflaredoing a tiny cluster now18:48
rickflare4 nodes18:48
tmckayk. you have debug level logging in sahara?18:48
tmckaythat would be good18:48
rickflarenaw18:48
rickflareI dont think its turned on18:48
tmckaydoh! yeah, I would turn that on18:48
elmiko+118:48
tmckayespecially for this18:48
rickflarehow do I turn it on18:49
tmckayshould just be "debug = true" in sahara.conf, in the DEFAULT section18:49
tmckaythen systemctl to restart the sahara processes18:49
tmckayshould be logging to /var/log/sahara, I believe18:50
rickflareok18:50
rickflarewill do18:50
rickflaresoon as this finished18:50
tmckayack18:50
rickflarehttps://bugs.launchpad.net/sahara/+bug/154765318:51
openstackLaunchpad bug 1547653 in Sahara "Sahara Image Elements Fails to build Centos or Fedora images using Cloudera CDH 5.4" [High,Confirmed]18:51
rickflareanyone know what the status is with this?18:51
elmikoi have not looked into it18:52
crobertsrhrickflare:  doesn't seem to be assigned to anyone yet.18:52
crobertsrhNot my forte, otherwise I'd have grabbed it.18:52
elmikolikewise :/18:53
crobertsrhbrb...errand18:54
*** agireud has quit IRC18:55
*** agireud has joined #openstack-sahara18:57
tmckayrickflare, trying on my F21 box.  Jaxon (owner of this review) has been doing a lot of work with cdh lately -- might be the person to contact for ideas https://review.openstack.org/#/c/279964/19:04
rickflaretry19:05
rickflarehdp19:05
rickflarecdh worked fine19:05
tmckayoh, I mean the bug19:05
tmckaybug was cdh, wasn't it?19:05
rickflarecdh19:06
rickflareand19:06
rickflarehdp19:06
*** dencaval has joined #openstack-sahara19:13
*** Alex____ has joined #openstack-sahara19:29
*** Alex____ has quit IRC19:30
elmikotmckay: just workflow'd the spark 1.0.0 removal reviews19:34
tmckayelmiko, thanks!19:34
tmckayelmiko, sorting through details of proxy command and gateway host related to use_floating_ip logical tests :)19:35
elmikocool19:35
elmikoi'm almost done with reviews for the week though, i'm cooked and have a bunch of other stuff to get to19:35
tmckaylol, trying to keep my gray matter contained19:35
tmckayelmiko, ack, I hear you19:36
elmikoheh, wish i could do that... ;)19:36
openstackgerritMichael McCune proposed openstack/sahara: remove hdp from the default plugin list  https://review.openstack.org/28492219:40
openstackgerritMichael McCune proposed openstack/sahara: remove hdp from the default plugin list  https://review.openstack.org/28492219:41
tmckayelmiko, you're just going to rebase when the ambari change merges?19:48
elmikoi'm rebasing now19:49
rickflareman19:49
rickflarei think hdp19:49
elmikosigh... except it's fighting with me19:49
rickflareis a no go19:49
openstackgerritMichael McCune proposed openstack/sahara: remove hdp from the default plugin list  https://review.openstack.org/28492219:50
openstackgerritMichael McCune proposed openstack/sahara: enable ambari plugin by default  https://review.openstack.org/28471919:50
elmikosigh...19:51
tmckayrickflare, btw, I just successfully built  tox -e venv -- sahara-image-create -i centos -p cloudera -v 5.4 -j oracle-java on Fedora 2119:51
tmckayI'll add that note to the bug19:51
rickflarewait19:52
tmckaycrobertsrh, ^^, worked with sie master on F2119:52
rickflaredo centos7 on 2.7.1 vanilla19:52
tmckayk, in progress tox -e venv -- sahara-image-create -i centos7 -p vanilla -v 2.7.119:54
rickflareNOICE!19:54
tmckayI left off the -j oracle on this one19:54
tmckaywe'll see if it builds. sounds from crobertsrh comment that there might be a delta between F21 and F23, or if not then it's environmental19:56
tmckayor I suppose there could have been a recent sie commit19:56
tmckayhmm, no recent commits19:58
*** sgotliv has joined #openstack-sahara19:59
*** itisha has quit IRC20:09
crobertsrhNot sure if the f21 -> f23 difference is key or not.  Did you get it going?20:12
openstackgerritMerged openstack/sahara: Remove support for spark 1.0.0  https://review.openstack.org/28252820:14
*** sgotliv has quit IRC20:37
*** esikachev has joined #openstack-sahara20:42
crobertsrhelmiko, tmckay:  any chance you guys have seen this?  http://paste.openstack.org/show/488427/20:49
crobertsrhI get it when trying to run tools/get_auth_token.py (in a test script I have)20:49
tmckayhmm, haven't seen that before. first guess would be stale venv?20:51
rickflareso20:51
rickflareI notice20:51
rickflarethe correct key was aplied to my hdp instances20:51
crobertsrhtmckay: that was my guess, but fresh venv didn't fix it20:51
rickflarebut I can ssh in20:51
rickflarei get a permission denied20:51
tmckayk, centos7 vanilla 2.71 build failed, complaining that grub2 was already installed.20:53
rickflareyup20:53
rickflareI dont know what the hell is going on20:54
*** dave-mccowan has quit IRC20:56
tmckayrickflare, did you build the images with the debug flag, so you can log in as root?20:56
openstackgerritMerged openstack/sahara: Add default templates for spark plugin, version 1.6.0  https://review.openstack.org/28253021:02
rickflareno21:02
rickflarethese are direct from mirantis21:02
tmckayrickflare, okay. hmm, so how do you know for sure that the right key was applied?  (other silly possibility is using the wrong user to log in?)21:03
elmikocrobertsrh: not sure if i've seen that specific one, but for me it either mean old venv or old version of tox21:03
rickflarewell21:03
crobertsrhelmiko:  I have a bead on it21:04
rickflarei could be using the wrong user21:04
elmikocrobertsrh: what is it?21:04
crobertsrhLooks like keystoneclient.middleware is now keystonemiddleware21:04
rickflareits the centos image21:04
rickflareso i assumes21:04
crobertsrhfiling a bug/fix now21:04
rickflarecentos is the user21:04
elmikoah.. right21:04
elmikocrobertsrh: good spot21:04
tmckayrickflare, try cloud-user21:04
tmckayI think it changed somewhere along the line ...21:04
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Fix using proxy node for checks  https://review.openstack.org/27944721:05
rickflareoh geeze21:06
rickflarelmao21:06
rickflareFAIL21:06
tmckayhttps://stackops.zendesk.com/hc/en-us/articles/201923327-Centos-6-X-and-7-0-default-username-and-password21:06
rickflareim in21:06
tmckaycentos 6 is cloud-user, 7 is centos21:06
tmckayrickflare, not your fault :)21:06
*** esikachev has quit IRC21:07
rickflareok21:07
rickflareso im in the master node21:08
rickflareand its like21:08
rickflarenothing is going on21:08
tmckayis this during cluster launch?21:08
rickflareyea21:09
tmckayrickflare, hmm, just a guess -- what if you gave sahara the wrong user name also when you launched the image?  I wonder if it's trying to use that user as well for ssh commands21:10
tmckaylaunched the cluster, I mean21:10
rickflarefuck21:10
rickflareLMAO21:10
rickflareyup21:10
rickflaresorry for the cussing21:10
tmckaywell, you *could* hack it21:10
rickflareill rebuild21:10
rickflarethats it21:10
rickflareone sec21:10
tmckayrickflare, if you want to, and you generate your own images, you can throw -d on create and it will setup up root login through the console for you.21:12
tmckaythat can help sometimes21:12
rickflareman21:14
rickflarethis def21:14
rickflarewould have kept it from working21:14
tmckaygrowing pains21:14
openstackgerritMerged openstack/sahara-dashboard: Import reorg and cleanup  https://review.openstack.org/28541121:15
*** AndreyPavlov has quit IRC21:15
openstackgerritChad Roberts proposed openstack/sahara: Updating get_auth_token to use keystonemiddleware  https://review.openstack.org/28554321:15
rickflarespawning21:16
rickflareagain21:16
* tmckay hopes for the best ...21:17
rickflaredude21:18
rickflarethat was it21:18
rickflareway more cpu activity21:19
rickflarenow21:19
tmckayexcellent!21:19
tmckayif you tail the sahara log, and you have debug set, you should see it doing a bunch of stuff to the machine21:19
tmckayand it should come out of "waiting" and go to "configuring" and "starting"21:19
tmckayrickflare, yeah, why centos 6 used "cloud-user", I don't know21:20
tmckayit was the outlier21:20
rickflareso21:20
rickflarenow waiting on the disked to be filled21:20
rickflareso21:20
rickflarewith nova21:20
rickflareI just made a big raid 10 partitiion21:20
rickflarei wonder I should have just left them a bunch of small disks21:21
tmckayI don't know, now you're out of my domain :) I know little about openstack/hadoop volume optimization21:21
rickflarei know for hadoop this is not optimal21:22
rickflarei need to read more on swift21:22
*** crobertsrh is now known as _crobertsrh21:28
tmckayah, an oldie but a goodie  "An admin context does not possess the service catalog and therefore does  not allow neutron interaction (or interaction with nova etc).  So the  neutron info is recorded in the job execution to allow that repeated  task to execute successfully in a neutron environment."21:34
tmckaywent and tracked this down from 2013, wondering if it was necessary to keep extra['neutron'] in the job_execution21:35
tmckayhmm, I wonder if this situation has changed ...21:35
_crobertsrhprobably not changed is my best guess21:36
tmckay_crobertsrh, I'm trying to factor out use of use_floating_ips and ran into this quirky neutron info storage in the job manager, with logic duplicated in ssh_remote.py21:36
_crobertsrhAt least there was a comment hanging around to 'splain it21:37
tmckaywould be nice if it could disappear ... sounds like it's easy to kick the tires, though -- just erase and see if periodics fail :)21:37
tmckayyeah, comment should go in the code21:37
_crobertsrhoh, I was assuming the comment was in the code21:37
tmckayit's in an old review I had to find by successive git blame, git checkout commit-1, check review, repeat21:37
tmckayback to december 201321:38
_crobertsrhwow, there's no quit in tmckay on a Friday afternoon....blame or bust!21:38
tmckaylol, I should have been an archeologist21:39
_crobertsrhThat level of work gets you my nod for employee of the week.21:39
_crobertsrhit's not worth anything really, but well done.21:39
tmckay\o/21:40
tmckaythx :) I may add the comment as part of my refactor21:40
*** agireud has quit IRC21:41
_crobertsrh+a bunch  for commenting something like that21:43
*** jamielennox is now known as jamielennox|away21:43
*** agireud has joined #openstack-sahara21:43
rickflarestill building21:48
rickflareI made the cluster much much bigger21:48
rickflare30 nodes21:48
rickflareall x.large21:48
rickflare16GB of ram each21:48
tmckaythat's bigger than anything I've made ...21:50
_crobertsrhtmckay:  maybe we should get a new test machine so we can make such a monstrosity of a cluster....purely for testing purposes, mind you.21:51
* rickflare believes go hard or go home21:51
rickflareseriously21:51
rickflarewhen I get to TX21:51
rickflarewe have to get some drinks21:51
rickflarei owe you guys21:51
rickflareshould be a good time!21:51
* _crobertsrh looks forward to that.21:51
rickflareif you cant already tell21:51
rickflareim a fun time21:51
rickflareLOL21:51
tmckaycrobertsrh is also fun21:52
rickflareI was even more fun when I was younger21:52
tmckayme, not so much21:52
* rickflare was the life of the party21:52
rickflaretmckay you are fun bro21:52
_crobertsrhwhaaa??  tmckay is plenty of fun.21:52
rickflarei can tell21:52
tmckaywonder if misc will be there .. now there's some fun21:52
_crobertsrhI still owe him a beating at billiards from the Atlanta openstack summit.21:52
_crobertsrhpro tip rickflare....do NOT play pool against tmckay.21:52
tmckayoh, yeah, I've got to practice21:53
elmikolol21:53
tmckaymuscle memory from when I was 11, that's all it is21:53
_crobertsrhelmiko witnessed the greatness of tmckay21:53
elmikoyup21:53
* _crobertsrh felt the greatness21:53
elmikohaha, let's not go too far ;)21:53
rickflarelisten dont challange me to a drinking contest21:53
elmikooh, i won't. but _crobertsrh might ;)21:53
tmckayI think I must have played a lot when I was a kid, and just don't remember how much21:53
* rickflare has blown past 20 shots of tequila21:54
_crobertsrhHeh.  It's important for us to lay out what NOT to do when we meet.21:54
elmikohahaha21:54
tmckaywe're supposed to get work done, remember21:54
tmckay20 shots of tequila != good design decisions21:55
rickflarelol21:55
rickflaretruth21:55
* rickflare sets a 3 shot max21:55
rickflarelol21:55
_crobertsrh"Yeah, let's rewrite sahara in pascal"21:55
tmckaypascal, haskell, anything that rhymes21:55
rickflarehaha21:55
rickflareor in Ada21:55
rickflarewoot woot21:55
tmckayI did Ada for 13 years21:56
rickflarei did it for 621:56
rickflarehorrible times21:56
rickflarein my life21:56
* rickflare was a vms guy as well21:56
tmckayoh, yeah, vms21:57
tmckaybeen there21:57
_crobertsrhok, I'm actually bailing now.  Getting thirsty.  Have a good weekend everyone.21:58
rickflareok have a good one crobertsrh21:59
tmckaybye, I need to head out too in few minutes22:01
elmikohey, haskell is actually pretty cool22:01
elmikolater _crobertsrh22:01
rickflarewooooo22:02
rickflareits working22:02
rickflarenaw22:04
rickflarespoke too soon22:04
rickflareyum is running now though22:05
tmckaythat's good. post in here later if it succeeds, I'll peek occasionally22:05
*** tmckay is now known as _tmckay22:06
openstackgerritMerged openstack/sahara: Await start datanodes in Spark plugin  https://review.openstack.org/27910522:09
rickflaretime out error22:12
rickflarehow do I extend the timeout22:12
elmikowhich timeout?  there are several values you can control in the conf file22:22
*** witlessb has quit IRC22:43
« Thursday, 2016-02-25 Index Saturday, 2016-02-27 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!