Friday, 2017-09-01

« Thursday, 2017-08-31 Index Saturday, 2017-09-02 »
*** shuyingya has joined #openstack-sahara00:10
*** shuyingya has quit IRC00:45
*** shuyingya has joined #openstack-sahara01:17
*** shuyingya has quit IRC01:18
*** caowei has quit IRC01:45
*** links has joined #openstack-sahara03:32
*** esikachev has joined #openstack-sahara04:58
*** hoonetorg has quit IRC05:05
*** esikachev has quit IRC05:07
*** caowei has joined #openstack-sahara05:08
*** esikachev has joined #openstack-sahara05:09
*** ssmith has joined #openstack-sahara05:16
*** hoonetorg has joined #openstack-sahara05:22
*** esikachev has quit IRC05:25
*** esikachev has joined #openstack-sahara05:33
*** links has quit IRC05:50
*** links has joined #openstack-sahara06:01
*** abalutoiu has joined #openstack-sahara06:03
*** pgadiya has joined #openstack-sahara06:15
*** pcaruana has joined #openstack-sahara06:17
*** rcernin has joined #openstack-sahara06:33
*** zemuvier has joined #openstack-sahara06:33
*** pgadiya has quit IRC06:42
*** tesseract has joined #openstack-sahara07:12
*** shuyingya has joined #openstack-sahara07:16
*** abalutoiu has quit IRC07:54
*** abalutoiu has joined #openstack-sahara08:00
*** ssmith has quit IRC08:10
*** anshul has joined #openstack-sahara08:14
rcerninping sahara, do we have any diagrams on sahara that would show us where the cluster copies the .authorized_keys from master to slaves ? In which phase of the cluster deployment is that?08:41
*** tianqing has joined #openstack-sahara08:46
tianqinghello...08:46
tianqingi found there only 2.2.1 for ambari in http://sahara-files.mirantis.com/images/upstream/newton/08:47
tianqingbut the dashboard can only choose 2.3 or 2.4 when registrying image...08:48
tianqingwhat's happend?08:48
tianqingwhat's happened?08:48
remix_tjrcernin: i suppose is with cloud-init09:05
remix_tjtianqing: try http://sahara-files.mirantis.com/images/upstream/ocata/09:06
tianqingall right...09:06
tianqingdo we want to support tez?09:07
remix_tjtez?09:08
remix_tjapache tez? never heard before09:09
tianqingoh, tez is hadoop 2.0, and built on yarn....09:29
tianqingyes,apache tez...09:29
*** tianqing has quit IRC09:55
*** caowei has quit IRC10:12
*** caowei has joined #openstack-sahara10:12
*** caowei has quit IRC10:17
*** tosky has joined #openstack-sahara10:55
*** esikachev has quit IRC11:29
shuyingyatellesnobrega, as you said, the bug can't reproduce in the latest master branch. But I am sure it affects stable/ocata at least, maybe minor version in pike. and there is still some thing we can improve in such situation.11:37
*** abalutoiu has quit IRC11:52
*** tellesnobrega has quit IRC11:56
*** esikachev has joined #openstack-sahara12:16
*** shuyingya has quit IRC12:28
*** abalutoiu has joined #openstack-sahara12:46
openstackgerritOpenStack Proposal Bot proposed openstack/python-saharaclient master: Updated from global requirements  https://review.openstack.org/50003212:47
openstackgerritOpenStack Proposal Bot proposed openstack/sahara master: Updated from global requirements  https://review.openstack.org/50003812:48
*** shuyingya has joined #openstack-sahara12:51
*** tellesnobrega has joined #openstack-sahara12:54
iwonkatellesnobrega: i looked at the docs12:54
iwonkaand apparently it's auto generated for openstack CLI12:54
toskyyep12:54
iwonkaso probably one small commit to sahara is enough12:55
toskythere is magic™12:55
iwonka:D12:55
*** shuyingya has quit IRC12:55
iwonkai like this magic tosky12:55
toskyiwonka: for python-saharaclient, please wait for my reorganization patch: https://review.openstack.org/#/c/498580/12:55
iwonkatosky: i think i don't have to add anything to python-saharaclient12:56
toskyif you need to touch anything in doc12:56
toskyright, because it's not a new command12:56
iwonkabtw, tellesnobrega, tosky could you please explain me code review for this: https://review.openstack.org/#/c/485210/ ?12:56
toskyoh, right12:57
iwonkatosky: i added new commands, like four12:57
toskyso, if I read this correctly, the API defined in saharaclient/api/node_group_templates.py returns12:57
toskyreturn self._get('/node-group-templates/%s/export' % ng_template_id)12:57
toskywhat is the format of ng_template_id? The usual UUID, right?12:57
iwonkai think sooo12:58
toskyso what we were asking is to use something looking like an UUID in the test12:58
tosky'id' may be confusing12:59
toskythat's it12:59
iwonkaso i should just rename it?12:59
toskybut12:59
toskyon the other side12:59
toskyyou were not the first one doing it, now that I check better the other tests12:59
toskyall the other tests follow the same pattern, using 'id'13:00
toskyok, ok, so maybe better to think about changing this later13:00
toskyadded a comment, let's wait for the others13:02
iwonkasure13:02
toskydid you have some time to take a look at tempest? (probably not, but I should try )13:04
openstackgerritIwona Kotlarska proposed openstack/sahara master: Add docs about template portability.  https://review.openstack.org/50004413:04
iwonkanot really13:04
tellesnobregatosky, do you know when the keys are exchanged between instances during cluster creation?13:06
tellesnobregais that done on heat side?13:06
toskytellesnobrega: the ssh keys? No idea - but do each machine can access to the other through ssh? I thought that only the controller or the proxy accessed all instances13:07
tellesnobregayes13:07
tellesnobregathey can13:07
tellesnobregaiirc13:08
tellesnobregalet me check13:08
tellesnobregaeita13:08
openstackgerritIwona Kotlarska proposed openstack/sahara master: Add docs about template portability.  https://review.openstack.org/50004413:11
*** esikachev has quit IRC13:22
*** esikachev has joined #openstack-sahara13:23
*** dave-mccowan has joined #openstack-sahara13:32
*** links has quit IRC13:44
*** zemuvier has quit IRC13:47
tellesnobregatosky, the instances can access each other14:03
tellesnobregabut, what I see is, they all have the same id_rsa14:04
tellesnobregaso probably it is not copied from master to slave, it is generated by sahara and sent to all instances and added the same authorized_key on the .ssh/authorized_keys14:04
*** jeremyfreudberg has joined #openstack-sahara14:05
toskytellesnobrega: so basically each instance reuses the basic configuration required by the controller to talk with the instances14:06
tellesnobregait seems so14:06
toskyso it's easy, it's most likely done by heat by requesting a nova instance with a certain key14:06
jeremyfreudbergthe eavesdrop logs haven't quite caught up to this conversation - what's the topic here?14:08
jeremyfreudberg(currently reading over the rest of what i missed)14:08
jeremyfreudberg^ tosky/tellesnobrega14:10
tellesnobregajust wondering at what point configuration happens that allows instances to ssh to each other14:11
tellesnobregafirst question was at what point key is copied from master to slaves14:11
jeremyfreudbergssh between each other? something that vitaly worked on, i remember reading the patch14:11
jeremyfreudbergi can probably find it14:11
tellesnobregabut from what I saw the key is not copied from master to slaves14:11
tellesnobregathey all have the same id_rsa14:12
jeremyfreudbergi'll have to look closely then14:12
tellesnobregaand the generated by sahara pub key on the authorized_keys14:12
tellesnobregaso, probably this is created by heat and set up on all instances during creation14:12
jeremyfreudbergwell i have to check that too, one key (sahara key) is set by heat, and the other key (user key) is in the first-boot script, i can never remember which is which14:13
jeremyfreudbergregardless, let me poke a round14:13
jeremyfreudberg*around14:13
tellesnobregasure14:14
toskyjeremyfreudberg: and when you can, we re-reviewed https://review.openstack.org/#/c/485210/14:15
toskyoh, in case you missed it: (cheap) Data Science books by O'Reilly through Humble Bundle https://www.humblebundle.com/books/data-science-books14:15
toskyDRM-free14:15
jeremyfreudbergtosky, noted14:18
jeremyfreudbergtellesnobrega, this is the patch i was thinking of https://review.openstack.org/#/c/250830/14:18
*** tellesnobrega has quit IRC14:20
*** tellesnobrega has joined #openstack-sahara14:38
*** esikachev has quit IRC14:38
jeremyfreudbergtellesnobrega, i think you were offline when i linked to this https://review.openstack.org/#/c/250830/ -- i was something about something for ssh specific to vanilla14:45
tellesnobregathanks jeremyfreudberg will look14:46
tellesnobregathat is only for vanilla14:47
jeremyfreudbergindeed it is14:47
tellesnobregai tried a spark cluster and there is access between them without that14:47
jeremyfreudbergin fact only for hadoop user, not even ubuntu user14:47
tellesnobregaok14:47
tellesnobregaso this is for allowing hadoop commands14:48
tellesnobregaconfiguring hdfs probably14:48
tellesnobreganot exactly what I was looking for14:48
tellesnobregaprobably my first assumption was right, they all have the same private key and its pub is added to authorized14:48
tellesnobregaso they can ssh between each other14:48
toskyuhm, but then the public key is added when you request a public key14:49
toskybut the private key?14:50
tellesnobregachecking that14:52
tellesnobregabut probably is done on heat14:53
tellesnobregatosky, https://github.com/openstack/sahara/blob/master/sahara/service/engine.py#L197-L21814:54
tellesnobregahttps://github.com/openstack/sahara/blob/master/sahara/service/heat/heat_engine.py#L249-L25914:55
jeremyfreudbergright, but that's only adding the public key to authorized_keys14:55
tellesnobregayes14:55
tellesnobregathat the second part14:55
tellesnobregathe private generation I didn't find it yet14:56
jeremyfreudbergi feel like i've seen the private key file (that matches up with Generated-By-Sahara pubkey) hanging around the image before, but maybe i'm misremembering14:56
jeremyfreudberghttps://github.com/openstack/sahara/blob/master/sahara/plugins/spark/plugin.py#L28814:59
*** tellesnobrega has quit IRC15:00
*** anshul has quit IRC15:05
*** links has joined #openstack-sahara15:18
*** rcernin has quit IRC15:29
*** pcaruana has quit IRC15:30
*** abalutoiu_ has joined #openstack-sahara15:39
*** abalutoiu__ has joined #openstack-sahara15:40
*** abalutoiu has quit IRC15:42
*** abalutoiu_ has quit IRC15:44
*** abalutoiu__ has quit IRC15:45
*** esikachev has joined #openstack-sahara15:50
*** dave-mccowan has quit IRC16:20
*** esikachev has quit IRC16:30
*** links has quit IRC16:31
*** jeremyfreudberg has quit IRC16:33
openstackgerritMerged openstack/python-saharaclient master: Add export of node group templates  https://review.openstack.org/48521016:37
openstackgerritMerged openstack/sahara master: Updated from global requirements  https://review.openstack.org/50003816:37
openstackgerritMerged openstack/python-saharaclient master: Updated from global requirements  https://review.openstack.org/50003216:38
*** tellesnobrega has joined #openstack-sahara16:40
*** tellesnobrega has quit IRC17:14
*** tellesnobrega has joined #openstack-sahara17:14
*** esikachev has joined #openstack-sahara17:16
*** tosky has quit IRC17:17
*** esikachev has quit IRC17:20
*** esikachev has joined #openstack-sahara17:23
*** esikachev has quit IRC17:27
*** esikachev has joined #openstack-sahara17:30
*** esikachev has quit IRC17:35
*** esikachev has joined #openstack-sahara17:37
*** jeremyfreudberg has joined #openstack-sahara17:39
tellesnobregajeremyfreudberg, did you find where the private key is created?17:41
*** esikachev has quit IRC17:42
*** esikachev has joined #openstack-sahara17:42
jeremyfreudbergi didn't quite finish looking into it yet, but it seems like it is plugin-specific17:43
jeremyfreudbergand in fact limited to vanilla,spark,mapr17:43
jeremyfreudbergfor spark: https://github.com/openstack/sahara/blob/master/sahara/plugins/spark/plugin.py#L28817:43
jeremyfreudbergmapr https://github.com/openstack/sahara/blob/master/sahara/plugins/mapr/util/general.py#L17417:45
jeremyfreudberginterestingly the thing that we needed it for in mapr was removed in this commit https://github.com/openstack/sahara/commit/36640a8e19d423d0fbc740d3a5a5ccabe0800035#diff-f87878e92b3ead2040c2e4c7293532f2L16817:46
jeremyfreudbergso actually mapr plugin has the bits already there to accomplish it, it's just not being used17:47
*** esikachev has quit IRC17:50
tellesnobregaiirc correctly I think we can ssh from instances on storm as well17:59
*** tosky has joined #openstack-sahara18:01
tellesnobregahttps://github.com/openstack/sahara/blob/master/sahara/utils/crypto.py#L32-L6018:01
jeremyfreudbergright, but does that actually end up stored in the instances .ssh folder18:01
tellesnobregajeremyfreudberg, tosky https://github.com/openstack/sahara/blob/master/sahara/conductor/manager.py#L18818:02
jeremyfreudbergmy tentative answer to my own question is: only if the plugin asks to do that18:02
toskytellesnobrega: that's the generation18:03
tellesnobregayeah18:03
jeremyfreudbergtellesnobrega, since you mentioned storm I was poking around in the storm plugin code, in this function https://github.com/openstack/sahara/blob/master/sahara/plugins/storm/config_helper.py#L43 shouldn't we have checks for storm 1.1.0 as well?18:04
tellesnobregawe would have to check each plugin to see if this is possible on all of them18:04
tellesnobregajeremyfreudberg, yes, need to check if it is seeds or host on 1.1,018:05
tellesnobregaand update the check18:05
openstackgerritMerged openstack/sahara-dashboard master: Add export of node group templates  https://review.openstack.org/48521518:09
*** esikachev has joined #openstack-sahara18:35
openstackgerritMerged openstack/python-saharaclient master: Add import of node group templates  https://review.openstack.org/48757518:37
*** esikachev has quit IRC18:40
*** esikachev has joined #openstack-sahara18:42
*** esikachev has quit IRC18:48
*** abalutoiu__ has joined #openstack-sahara20:13
*** jeremyfreudberg has quit IRC20:20
*** rcernin has joined #openstack-sahara20:31
openstackgerritIwona Kotlarska proposed openstack/sahara master: Add docs about template portability.  https://review.openstack.org/50004420:45
*** rcernin has quit IRC20:55
openstackgerritTelles Mota Vidal Nóbrega proposed openstack/sahara-dashboard master: Fix issue creating NGT using dashboard  https://review.openstack.org/50019320:59
openstackgerritIwona Kotlarska proposed openstack/python-saharaclient master: Add export of node group templates to CLI  https://review.openstack.org/48672120:59
openstackgerritIwona Kotlarska proposed openstack/python-saharaclient master: Add export of cluster templates  https://review.openstack.org/49852021:02
iwonkatellesnobrega: i have gate-tempest-dsvm-sahara-ubuntu-xenialFAILURE in one commit21:04
iwonkarecheck?21:04
toskyiwonka: which job?21:08
toskyaka: can you please give me the link to the failure?21:08
iwonkahttps://review.openstack.org/#/c/500044/21:09
*** esikachev has joined #openstack-sahara21:26
*** esikachev has quit IRC21:31
tellesnobregatosky, do you know how to setup devstack with for plugin installation?21:40
toskytellesnobrega: the usage if enable_plugin?21:40
tellesnobregafor example i can enable_plugin sahara-dashboard git://git.openstack.org/openstack/sahara-dashboard stale/newton for branches21:40
tellesnobregaI forgot the real part of the question21:40
*** abalutoiu__ has quit IRC21:41
tellesnobregahow to set up with a review for installation21:41
toskyehm, I didn't get the question21:41
tellesnobregayeah, it was very malformed21:41
tellesnobregalet me try again21:41
tellesnobregaI'm trying to test if my patch works. For that I want to force devstack plugin to use the patch I sent21:42
tellesnobregado you know to do that?21:42
toskyoh, uh, in the jobs or manually?21:42
toskybecause in the jobs, I think that there is some magic that makes sure that the code is downloaded with the reviews21:43
tellesnobregamanually21:43
tellesnobregasomething that I can add on local.conf that downloads that review for that plugin21:43
toskycan't you specify a local directory as the last parameter of enable_plugin?21:44
tellesnobreganot sure21:44
tellesnobregatrying with review number after the enable plutin21:47
tellesnobregaplugin21:47
toskyif it accepts a local directory, or a local git checkout, where the patch is applied, it may work21:47
tellesnobregaI will check21:47
*** tesseract has quit IRC22:09
iwonkawhat's wrong with cli patches? both fail on the same test and i can't get to any kind of resonable logs22:19
tellesnobreganot sure iwonka22:21
tellesnobregawe have to check what is going on, probably infra issues22:21
iwonkahow to check it?22:22
tellesnobregayou would have to go deeper on logs22:24
tellesnobregaand check other patches as well on zuul to see if failure is all around22:24
toskyiwonka: the other review passed after the recheck; do you have another failure somewhere else?22:31
iwonkatosky: https://review.openstack.org/#/c/486721/ and https://review.openstack.org/#/c/498520/ fail on the same gate and they are similar patches22:32
iwonkaso i'm wondering if it's problem with my patches or something external22:33
tellesnobregaiwonka, just recheck and see how it goes22:33
toskyone moment :)22:33
iwonkai did a recheck and the failure was later22:34
toskyhttp://logs.openstack.org/21/486721/8/check/gate-sahara-cli-ubuntu-xenial-nv/ae28e21/logs/devstacklog.txt.gz#_2017-09-01_21_15_44_07822:34
toskyit's something in the infrastructure22:34
toskyblindly rechecking can make that pass, but it would be better to check if there is a reported bug for this22:34
tellesnobregasure22:36
toskybut no recent changes in devstack22:37
toskyor devstack-gate22:37
tellesnobregaI have to drop now22:39
tellesnobregahave a great weekend tosky and iwonka22:40
tellesnobregadon't lose your minds on this22:40
tellesnobregait will get fixed soon22:40
tellesnobregayou both should be enjoying your fridays at this point :)22:40
iwonkayou're probably right, have a nice weekend :)22:40
toskyiwonka: ok, let's try one recheck, but if it fails, let's investigate better on Monday22:40
toskyo/22:40
iwonkatosky: ok22:41
*** tosky has quit IRC22:59
*** esikachev has joined #openstack-sahara23:09
*** esikachev has quit IRC23:26
« Thursday, 2017-08-31 Index Saturday, 2017-09-02 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!