Monday, 2015-08-03

« Sunday, 2015-08-02 Index Tuesday, 2015-08-04 »
*** chlong has joined #openstack-sahara00:03
*** saneax has quit IRC00:04
*** jamielennox is now known as jamielennox|away01:08
openstackgerritli,chen proposed openstack/sahara: Rename oozie_job_id  https://review.openstack.org/20593001:08
openstackgerritli,chen proposed openstack/sahara: Add CLUSTER_STATUS  https://review.openstack.org/19991601:14
openstackgerritli,chen proposed openstack/sahara: Remove cluster status change in HDP plugin  https://review.openstack.org/20345701:14
*** coolsvap|away is now known as coolsvap03:40
*** hdd has joined #openstack-sahara03:40
*** hdd has quit IRC03:46
*** saneax has joined #openstack-sahara04:19
*** Poornima has joined #openstack-sahara04:37
*** jamielennox|away is now known as jamielennox04:38
*** hdd has joined #openstack-sahara05:20
*** hdd has quit IRC05:27
*** ig0r_ has joined #openstack-sahara05:33
*** ig0r_ has quit IRC05:53
*** ig0r_ has joined #openstack-sahara05:56
*** pcaruana has quit IRC06:09
*** saneax has quit IRC06:58
*** witlessb has joined #openstack-sahara07:06
*** sgotliv has quit IRC07:09
*** saneax has joined #openstack-sahara07:17
*** pino|work has joined #openstack-sahara07:33
*** chlong has quit IRC07:38
*** pcaruana has joined #openstack-sahara07:40
openstackgerritEvgeny Sikachev proposed openstack/sahara-image-elements: Fix version of Centos in Mapr plugin  https://review.openstack.org/20634407:51
*** jamielennox is now known as jamielennox|away07:57
openstackgerritEvgeny Sikachev proposed openstack/sahara: [DO NOT MERGE] Workaround for mapr  https://review.openstack.org/20454808:22
openstackgerritEvgeny Sikachev proposed openstack/sahara: Implement ability of creating flavor for scenario tests  https://review.openstack.org/20356708:24
*** sgotliv has joined #openstack-sahara08:50
*** Nikolay_St has joined #openstack-sahara09:02
openstackgerritAndrey Pavlov proposed openstack/sahara-specs: Addding spec for Objects update support in Sahara API  https://review.openstack.org/20837809:04
*** tosky has joined #openstack-sahara09:04
openstackgerritArtem Osadchiy proposed openstack/sahara: Added support of Hive 1.0 to MapR plugin  https://review.openstack.org/20840009:50
*** sgotliv has quit IRC09:58
*** witlessb_ has joined #openstack-sahara10:02
*** sanjayu_ has joined #openstack-sahara10:07
*** witlessb has quit IRC10:08
*** sanjayu has quit IRC10:08
*** logan2 has quit IRC10:08
*** witlessb_ is now known as witlessb10:08
*** logan2 has joined #openstack-sahara10:08
openstackgerritAndrey Pavlov proposed openstack/sahara: Adding clusters_update api call  https://review.openstack.org/19502410:28
openstackgerritArtem Osadchiy proposed openstack/sahara: Added support of Mahout 0.10 to MapR plugin  https://review.openstack.org/20841410:47
openstackgerritArtem Osadchiy proposed openstack/sahara: Added support of HBase 0.98.12 to MapR plugin  https://review.openstack.org/20841810:54
*** tmckay has joined #openstack-sahara10:55
*** degorenko has joined #openstack-sahara10:56
*** pcaruana has quit IRC10:57
openstackgerritArtem Osadchiy proposed openstack/sahara: Added support of Drill 1.1 to MapR plugin  https://review.openstack.org/20842011:01
*** coolsvap is now known as coolsvap|away11:06
openstackgerritArtem Osadchiy proposed openstack/sahara: Added support of Mahout 0.10 to MapR plugin  https://review.openstack.org/20841411:09
*** tmckay has quit IRC11:11
*** tmckay has joined #openstack-sahara11:11
openstackgerritArtem Osadchiy proposed openstack/sahara: Added support of HBase 0.98.12 to MapR plugin  https://review.openstack.org/20841811:13
*** sgotliv has joined #openstack-sahara11:14
*** pcaruana has joined #openstack-sahara11:14
openstackgerritVitaly Gridnev proposed openstack/sahara: Small refactoring for vanilla 2  https://review.openstack.org/20842411:15
openstackgerritArtem Osadchiy proposed openstack/sahara: Added support of Drill 1.1 to MapR plugin  https://review.openstack.org/20842011:18
openstackgerritArtem Osadchiy proposed openstack/sahara: Added support of Hadoop 2.7.0 to MapR plugin  https://review.openstack.org/20844311:40
*** hdd has joined #openstack-sahara11:55
*** htruta_ has joined #openstack-sahara11:58
*** coolsvap|away is now known as coolsvap11:58
openstackgerritAndrey Pavlov proposed openstack/sahara: Adding clusters_update api call  https://review.openstack.org/19502412:04
openstackgerritAndrey Pavlov proposed openstack/sahara: Adding job_update api call  https://review.openstack.org/19792412:05
openstackgerritArtem Osadchiy proposed openstack/sahara: Added support for MapR v5.0.0  https://review.openstack.org/20845412:09
*** raildo has joined #openstack-sahara12:18
*** chlong has joined #openstack-sahara12:35
*** hdd has quit IRC12:35
openstackgerritAndrey Pavlov proposed openstack/sahara: Adding job_update api call  https://review.openstack.org/19792412:37
*** Poornima has quit IRC12:39
openstackgerritAndrey Pavlov proposed openstack/sahara: Adding job_execution_update api call  https://review.openstack.org/19791412:55
*** tmckay has quit IRC13:03
*** esikachev has joined #openstack-sahara13:03
*** esikachev has quit IRC13:11
*** esikachev has joined #openstack-sahara13:11
openstackgerritVitaly Gridnev proposed openstack/sahara: Make starting scripts module for vanilla 2 plugin  https://review.openstack.org/20847413:20
*** ig0r_ has quit IRC13:26
*** ig0r__ has joined #openstack-sahara13:33
*** egafford has joined #openstack-sahara13:36
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org has been restarted in an attempt to improve performance.13:41
*** hdd has joined #openstack-sahara13:42
openstackgerritVitaly Gridnev proposed openstack/sahara: Drop support of deprecated 2.4.1 Vanilla plugin  https://review.openstack.org/20601113:43
openstackgerritVitaly Gridnev proposed openstack/sahara: Make starting scripts module for vanilla 2 plugin  https://review.openstack.org/20847413:44
*** ig0r__ has quit IRC13:44
openstackgerritVitaly Gridnev proposed openstack/sahara: Make starting scripts module for vanilla 2 plugin  https://review.openstack.org/20847413:46
*** h00327910__ has quit IRC13:48
*** esikachev has quit IRC13:48
*** Nikolay_St has quit IRC13:49
openstackgerritAndrey Pavlov proposed openstack/sahara-specs: Addding spec for Objects update support in Sahara API  https://review.openstack.org/20837813:54
*** esikachev has joined #openstack-sahara13:56
*** tmckay has joined #openstack-sahara13:57
openstackgerritAndrey Pavlov proposed openstack/sahara: Formatting and mounting methods changed for ironic  https://review.openstack.org/20048314:00
*** saneax has quit IRC14:01
openstackgerritVitaly Gridnev proposed openstack/sahara: Make starting scripts module for vanilla 2 plugin  https://review.openstack.org/20847414:06
*** ig0r__ has joined #openstack-sahara14:14
openstackgerritAndrey Pavlov proposed openstack/sahara: Adding job_binary_internal_update api call  https://review.openstack.org/20849614:18
*** elmiko_ has joined #openstack-sahara14:32
*** elmiko_ is now known as __elmiko14:32
*** __elmiko is now known as _el_miko14:32
*** _el_miko has left #openstack-sahara14:33
openstackgerritVitaly Gridnev proposed openstack/sahara: Update vanilla plugin to the latest version  https://review.openstack.org/20851014:33
openstackgerritMerged openstack/sahara: Made 'files' dict as member field of ClusterStack  https://review.openstack.org/20327114:42
openstackgerritChad Roberts proposed openstack/sahara: Support manila shares as binary store  https://review.openstack.org/20469014:48
*** barra204 has joined #openstack-sahara14:57
openstackgerritEvgeny Sikachev proposed openstack/sahara-specs: Add cluster verification checks  https://review.openstack.org/19657615:04
*** ig0r__ has quit IRC15:09
*** barra204 has quit IRC15:09
*** pcaruana has quit IRC15:13
*** shakamunyi has joined #openstack-sahara15:14
*** shakamunyi is now known as barra20415:16
*** esikachev has quit IRC15:20
*** barra204 has quit IRC15:26
*** saneax has joined #openstack-sahara15:34
tmckaySergeyLukjanov, others, okay the hdp 2.0.6 gate test is becoming a real issue.  Can we make it non-voting, and get a volunteer to find the reason for the failure in the meantime?15:39
tmckayI would chase hdp on my own devstack but I don't have time at the moment15:39
tmckaybut, it's slowing down merges quite a bit15:39
*** barra204 has joined #openstack-sahara15:40
openstackgerritTrevor McKay proposed stackforge/sahara-ci-config: Temporarily make  gate-sahara-nova-heat-hdp_2.0.6-c6.6 non-voting  https://review.openstack.org/20853815:51
tmckaythere, taking the bull by the horns.  Please comment if I did this incorrectly, or if you disagree ^^15:52
*** openstackgerrit_ has joined #openstack-sahara15:53
openstackgerritTrevor McKay proposed openstack/sahara: Remove test for job type in get_data_sources  https://review.openstack.org/20854216:00
*** ig0r_ has joined #openstack-sahara16:08
*** melaks has joined #openstack-sahara16:09
*** sgotliv has quit IRC16:40
*** tosky has quit IRC16:42
*** tosky has joined #openstack-sahara16:47
*** ig0r_ has quit IRC16:49
*** ig0r_ has joined #openstack-sahara16:54
*** ig0r_ has quit IRC16:55
*** coolsvap is now known as coolsvap|away16:58
openstackgerritMerged stackforge/sahara-ci-config: Temporarily make  gate-sahara-nova-heat-hdp_2.0.6-c6.6 non-voting  https://review.openstack.org/20853817:03
*** ig0r_ has joined #openstack-sahara17:04
*** pcaruana has joined #openstack-sahara17:06
openstackgerritTelles Mota Vidal Nóbrega proposed openstack/sahara: Increase internal_ip and management_ip column size  https://review.openstack.org/20306717:15
*** ig0r_ has quit IRC17:25
*** melaks has quit IRC17:30
*** chlong has quit IRC17:31
*** melaks has joined #openstack-sahara17:31
*** ig0r_ has joined #openstack-sahara17:33
*** Nikolay_St has joined #openstack-sahara17:39
*** ig0r_ has quit IRC17:44
*** tosky has quit IRC17:54
*** pino|work has quit IRC17:54
*** openstackgerrit_ has quit IRC18:07
crobertsrhIs the ci stuff totally hosed at the moment?18:29
*** melaks has quit IRC18:31
tmckayuh oh18:31
tmckayMaybe my merge broke it?18:32
*** htruta_ has quit IRC18:32
tmckaydegorenko, ^^18:32
crobertsrh novaclient.exceptions.BadRequest: Keypair data is invalid: failed to generate fingerprint18:32
tmckayoh, that seems different.  elmiko ^^18:32
crobertsrhSeems to be same message for all the tests18:33
crobertsrh(at least the 3 I've tracked down so far)18:33
elmikoweird...18:33
tmckayhmm, create_keypair in the scenario tests failing18:35
tmckayseems like18:35
tmckayI don't think it was me then :)18:35
elmikoyea, sounds like a stack or config issue18:36
*** htruta_ has joined #openstack-sahara18:44
*** shakamunyi has joined #openstack-sahara18:46
*** esikachev has joined #openstack-sahara19:05
tmckayelmiko, crobertsrh, looks like that issue has to come from nova/nova/crypto.py, generate_fingerprint (based on latest nova source)19:07
*** sgotliv has joined #openstack-sahara19:07
tmckaythat is the only place that "failed to generate fingerprint" shows up19:07
crobertsrhoh, did we pick up (or fail to pick up) a new lib somewhere?19:07
tmckaydon't know.  I wonder if we ping on #openstack-nova.  Maybe search launchpad first19:08
*** vgridnev has joined #openstack-sahara19:08
tmckayvgridnev, hi!  We were just discussing nova create_keypair failures in Sahara CI19:09
tmckaylooks like all scenario tests are breaking19:09
vgridnevyeah, I see19:09
tmckayI thought maybe my change to make hdp non-voting was an issue, but I can't see how it is related19:09
*** esikachev has quit IRC19:09
tmckaybut it was the latest change to the CI19:09
vgridnevSeems that CI failures started at about one day ago19:10
vgridnevAslo, CI lab was reinstalled today19:11
tmckayah, okay.  Looks like the error is from nova/nova/crypto.py, generate_fingerprint19:11
crobertsrhmakes sense that the new fingerprints for the new boxes might not match19:12
tmckaycould even be different serialization of the key, maybe19:12
tmckaytrouble is we don't have the real exception19:12
tmckayhmm, I wonder if there is a way to tweak nova on the CI, and get more information in the reason field for what is going in19:13
tmckay"on19:13
crobertsrhThere probably is, but it's probably painful for us to try :)19:14
tmckayvgridnev, ^^19:14
elmikohmm19:14
tmckayit's not a very long routine19:14
elmikodid something in nova.crypto change recently?19:14
elmikoor did the ci lab reinstall come with new versions of openssl or something?19:14
elmiko(or more likely, old versions of openssl)19:15
tmckayelmiko, https://review.openstack.org/#/c/197748/  ??19:16
tmckayjust merged a few weeks ago19:16
vgridnevtmckay, hm, intresting19:17
elmikoyou think we would have seen it sooner19:17
tmckayvgridnev, not sure if there is anything here ^^ but it touches nova crypto and it's recent19:17
tmckayoh, you saw :)19:17
tmckayssh_encrypt_text() changed19:18
elmikocertinaly sounds plausible19:18
tmckayso, maybe something in the Sahara env does not match the nova instance?19:18
tmckaylike, maybe we're passing ascii instead of unicode, or unicode instead of ascii?19:20
elmikoi dunno19:22
elmikothis is a 400 coming back from the nova api server19:23
elmikolike, novaclient POSTs to create a key, and something about the request is most likely not working19:23
tmckaysomething weird with my repo, I don't seem to be seeing the same code19:24
tmckayoh, duh, there were multiple changes19:26
tmckayelmiko, so, to answer your question, yes, a whole bunch of stuff has changed in nova crypto lately :)19:26
vgridnevwhich log of nova is most useful?19:27
vgridnevI devstack screen19:27
vgridnev*In19:27
tmckayI don't know, maybe the nova-api log?19:27
elmikoi'm looking at http://paste.openstack.org/show/406785/ from a failed test19:27
elmikoyea, nova-api is probably a good one. see if we can line up the keypair request to a bad call19:28
tmckayright, what I was doing was grepping for "failed to generate fingerprint" in nova code19:28
tmckayonly place is crypto.generate_finger_print19:28
tmckaybut the real exception is suppressed19:29
tmckaywe could hack nova and return the traceback as the reason string :)19:29
tmckayor just dump it with traceback.print_exc() but I don't know where it would go19:30
elmikothat would be one way, i think it might help to trace back from the 400 returned on the call. find out what endpoint is being hit, then start to drill down19:32
elmikoi'm guessing this is the call, http://developer.openstack.org/api-ref-compute-v2.1.html#keypairs-v2.119:32
openstackgerritVitaly Gridnev proposed openstack/sahara: Small refactoring for vanilla 2  https://review.openstack.org/20842419:35
elmikotmckay: looks like crypto@L15119:37
elmikoassuming that's the call, but i can't imagine what we are passing that could create that19:38
*** sgotliv has quit IRC19:40
elmikotmckay: also, is it possible that our novaclient is out of date?19:40
elmikoand, has anyone started asking in -nova?19:41
tmckayelmiko, yeah, line 151 for sure.  I haven't ask in -nova19:45
elmikoi'm guessing this is probably going to be an issue on our end, looking at sahara.utils.crypto19:45
elmikowhich is generating the keypair info that makes the call blow up19:45
tmckayyeah, hmm, I wonder if the nova version is available in the CI output.  Seems to be I've seen info like that before19:45
tmckayack19:46
elmikoyea, would be nice to see the nova logs, but i think we'd need to know what it expects19:46
elmikoi mean, to me, a returned 400 means that we sent the wrong thing to the server19:47
elmikowhich is what the novaclient is telling us19:47
tmckayelmiko, well, I have devstack running.  I could restack it, then try to do the same thing the scenario tests are doing.19:49
tmckayor, is it easy to run the scenario tests on a local stack? I've never tried19:49
*** raildo has quit IRC19:49
tmckaysome tox command, maybe19:49
elmikotmckay: you could try writing a small test to do the same thing as sahara.test.scenario.base@L56119:50
tmckayyeah19:50
elmikoshould be easy, just use the sahara crypto to generate a keypair, then try to create it with nova client19:51
tmckayk, I'll give it a shot. First, restack :)19:51
elmikoand make sure to set RECLONE=yes i suppose19:53
vgridnevseems that there are nothing intresting in nova-api-logs19:54
tmckayelmiko, one better than reclone, rm -rf /opt/stack ;-)19:54
elmikovgridnev: too bad =(19:55
elmikotmckay: nice19:55
vgridnevhttp://paste.openstack.org/show/406791/19:55
vgridnevactually I generated public key using https://github.com/openstack/sahara/blob/master/sahara/utils/crypto.py#L3219:55
elmikosame as the test, not sure why it doesn't like that data19:56
elmikois it possible the trailing newline is screwing things up?19:57
tmckayah, nice. but now if it is reproducible, we can step through the nova code19:57
elmikoGenerated by Sahara\\n"19:57
tmckayhmm, maybe19:57
tmckayelmiko, hah! reproducible on by devstack with openstack client and key from Sahara20:09
tmckayelmiko, now to jam pdb into nova ....20:09
tmckayI love this stuff20:09
tmckaywait, did you guys find it already? :)20:09
elmikotmckay: i'm going the other way, generating a keypair and calling nova client20:10
elmikobut the nova-api server is being wonky to me20:10
tmckayanyone know how to just restart nova in a devstack, without running the whole thing again?20:11
elmikohmm, not off-hand20:11
elmikoi think you can restart the httpd service and this will cause a restart20:12
elmikowell, of the client code at least20:12
elmikoer the api-server that is20:12
crobertsrhtmckay:  you can stop/restart things from the rejoin-stack.sh window-o-screens20:13
crobertsrhfind the service you want, then usually ctrl-c20:13
crobertsrhup arrow and enter to restart is usually also fine20:13
tmckaycrobertsrh, screen noob.  How do I select one?20:14
crobertsrhctrl-a then "20:14
crobertsrhthat brings up a screen where you can use up and down arrows to select20:14
crobertsrhenter to select20:14
crobertsrhctrl-a, then d  to exit the screen session20:15
tmckaynice, thanks!20:15
crobertsrhnp20:15
tmckayTraceback (most recent call last):20:18
tmckay  File "/opt/stack/nova/nova/crypto.py", line 140, in generate_fingerprint20:18
tmckay    pub_bytes, backends.default_backend())20:18
tmckay  File "/usr/lib/python2.7/site-packages/cryptography/hazmat/primitives/serialization.py", line 40, in load_ssh_public_key20:18
tmckay    'Key is not in the proper format or contains extra data.')20:18
tmckayValueError: Key is not in the proper format or contains extra data.20:18
tmckayelmiko, maybe you're right about the trailing newline, I'll try without it20:19
elmikomakes sense20:19
elmikoi seem to have lost all endpoints in my catalog, wtf20:21
tmckayawesome, inserted pdb into nova, stepping through the code using rejoin.sh.  thanks crobertsrh!20:24
tmckayso far, don't think it's the newline.  Removed with the debugger, same error20:25
crobertsrhglad I could help :)20:25
elmikooh wel20:25
elmikoi'm gonna jump off this fire brigade then20:25
tmckayelmiko, hah!20:26
tmckayit's the "Generated by Sahara"20:27
tmckaykey is split on spaces20:27
tmckayit sees this as a key with 5 components instead of 320:27
tmckaylet me verify ....20:27
elmikolol20:28
elmikocould be a nova bug then, i think you are allowed to have spaces in that end part20:28
tmckayhot darn20:29
elmikomight be worth mentioning in -nova20:29
tmckayvgridnev, ^^20:29
tmckay"Generated by Sahara" is the culprit20:29
elmikofunny20:29
elmikogood catch tmckay20:29
tmckayquestion is, do we fix our public key really fast so our tests work?20:29
tmckaythen ping nova?20:30
elmikohmm20:30
elmikoprobably20:30
elmikoman, my stack is fubar. openstack endpoint list shows nothing, wtf20:32
elmikoand novaclient keeps giving 40420:33
*** ig0r_ has joined #openstack-sahara20:36
vgridnevlet's fix ci first, then ping nova team20:37
vgridnevgenerated_by_sahara is good solution, I think20:38
elmiko+120:40
elmikoassuming there are no other spaces causing issues20:41
elmikotmckay: are you making a patch for the sahara stuff?20:49
openstackgerritTrevor McKay proposed openstack/sahara: Remove spaces from Sahara key comment  https://review.openstack.org/20864420:49
tmckayyep :)20:49
tmckayprobably filing a nova bug too20:50
elmikohehe nice20:50
tmckaystill unclear whether spaces in the key comment should be allowed20:50
elmikoyea, there is probably an rfc about that20:50
*** ig0r__ has joined #openstack-sahara20:50
*** ig0r_ has quit IRC20:50
openstackgerritOpenStack Proposal Bot proposed openstack/sahara: Updated from global requirements  https://review.openstack.org/20685421:03
openstackgerritTrevor McKay proposed openstack/sahara: Remove spaces from Sahara key comment  https://review.openstack.org/20864421:03
*** crobertsrh is now known as _crobertsrh21:04
elmikotmckay: i'm trying to track down an answer about the comment, it might be that spaces are not allowed21:05
elmikoby default that field is initialized to user@host21:05
tmckayelmiko, dims on -nova found a statement from the rfc that says that the rest of the line should be ignored21:06
tmckayelmiko, ie, at the end of the key text everything else should be tossed before it's parsed21:06
tmckayso nothing is illegal21:06
elmikoah, very good21:06
tmckayelmiko, sims like the split that happens prior to the split in the serializer changed21:07
tmckay"seems"21:07
tmckayso the comment is never supposed to get that far21:07
tmckayelmiko, (04:54:56 PM) dims: tmckay: "The optional comment field continues to the end of the line, and is not used." from sshd man page21:07
elmikoyea, sounds like a bug21:08
elmikotmckay: sounds like we've covered all bases21:09
openstackgerritTrevor McKay proposed openstack/sahara: Remove spaces from Sahara key comment  https://review.openstack.org/20864421:09
tmckayfixed up my comment, just because21:10
tmckayno more changes from me, so the CI has a chance to run :)21:10
tmckayelmiko, crobertsrh, that was some awesome debugging.  Thank you for the suggestions, I just put the legos together21:11
tmckayand vgridnev, for the suggestion to use the key generator method from Sahara utils21:11
elmikotmckay: there might be an upstream crypto bug you can file as well, just tracking it down now21:12
tmckayokey doke, I have to run.  I'll check in on the CI later.  If something goes wrong, feel free to take over my commit\21:13
elmikotmckay: relavant rfc section https://tools.ietf.org/html/rfc4716#section-3.3.221:13
tmckayyeah, nothing about spaces in there21:14
*** tmckay has quit IRC21:15
*** ig0r__ has quit IRC21:21
vgridnevelmiko, ci passed21:52
vgridnevmaybe merge?21:52
vgridnevalazarev, ^^ could you please review change https://review.openstack.org/#/c/208644/ and merge that to fix ci?21:53
alazarevvgridnev, done21:56
vgridnevthanks21:56
elmikocool, bugs filed upstream too and nova has a fix as well lol22:00
*** hdd has quit IRC22:03
*** egafford has quit IRC22:22
*** witlessb has quit IRC22:22
*** witlessb has joined #openstack-sahara22:23
*** witlessb has quit IRC22:23
*** shakamunyi has quit IRC22:51
*** egafford has joined #openstack-sahara23:13
« Sunday, 2015-08-02 Index Tuesday, 2015-08-04 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!