| *** elynn has joined #senlin | 00:00 | |
| *** eldon_ has quit IRC | 00:19 | |
| *** elynn has quit IRC | 00:42 | |
| openstackgerrit | Merged openstack/senlin: Initialize docker driver https://review.openstack.org/322692 | 00:44 |
|---|---|---|
| *** yanyanhu has joined #senlin | 01:53 | |
| *** elynn has joined #senlin | 01:55 | |
| *** elynn has quit IRC | 02:01 | |
| *** elynn has joined #senlin | 02:01 | |
| elynn | morning senlin | 02:12 |
| *** zzxwill has joined #senlin | 02:16 | |
| *** yuanying has quit IRC | 02:51 | |
| *** eldon_ has joined #senlin | 02:53 | |
| *** yuanying has joined #senlin | 02:54 | |
| eldon_ | may I ask some question about rpc call? | 02:55 |
| eldon_ | we have 3 servers installed senlin. | 02:55 |
| eldon_ | each one has its own api+engine. | 02:55 |
| yanyanhu | hi, eldon_ | 02:55 |
| eldon_ | so we have 3 apis and 3 engiens. | 02:56 |
| eldon_ | now, when we stop one of the three engine. | 02:56 |
| eldon_ | it will have 3 apis and 2 engines. | 02:56 |
| eldon_ | and when I use senlin node-list to show nodes. | 02:57 |
| eldon_ | it will fail one of the three times. | 02:57 |
| yanyanhu | so you configure loadbalancer for senlin api? | 02:58 |
| eldon_ | any one had idea about it? | 02:58 |
| eldon_ | yes. | 02:58 |
| eldon_ | I have lb for api use ha-proxy | 02:58 |
| eldon_ | rabbitmq for rpc | 02:58 |
| yanyanhu | eldon_, you can configure more than one engine worker for each senlin service | 02:59 |
| eldon_ | I don't knoe it's rabbit's problem or senlin's | 02:59 |
| yanyanhu | but for the case you described, we don't have good support now | 02:59 |
| eldon_ | yes, I think that's the point | 02:59 |
| eldon_ | but how? | 02:59 |
| yanyanhu | eldon_, it's senlin's problem now | 03:00 |
| yanyanhu | since senlin-api are not aware of engine failure | 03:00 |
| yanyanhu | and thus will always try to send rpc calls to engine | 03:00 |
| eldon_ | can senlin-api find other engine not in it's server automatically? | 03:00 |
| yanyanhu | no, that is infeasible | 03:01 |
| eldon_ | or do you know how nova make it work? | 03:01 |
| yanyanhu | nova can't handle that case as well as I know | 03:01 |
| yanyanhu | I mean nova-api in one deployment can't talk with nova-compute in another deployment | 03:02 |
| eldon_ | really? | 03:02 |
| yanyanhu | but you can configure multiple engines at backend, and if api can be aware of engine failure, it can stop sending rpc request to failed engine | 03:02 |
| yanyanhu | yes, I guess so. Since the rpc msg is passed through queue | 03:03 |
| eldon_ | yes. | 03:03 |
| yanyanhu | and the queue server and topic are fixed in configuration | 03:03 |
| eldon_ | so, I think you mean nova has its backend and nova-api has function to process engine failure? | 03:04 |
| eldon_ | senlin-api doesn't have this function yet. | 03:04 |
| yanyanhu | yes, I guess so | 03:04 |
| eldon_ | ok, thank you:) | 03:05 |
| yanyanhu | eldon_, I was not sure about it:) | 03:06 |
| yanyanhu | I guess nova api will stop creating VM in a compute node after that compute node crashed | 03:06 |
| yanyanhu | what you said is a really important issue | 03:06 |
| eldon_ | yes. nova knows which nova-cpt is down. | 03:06 |
| yanyanhu | eldon_, so the same problem won't happen in nova :) | 03:07 |
| yanyanhu | unless no any nova-compute nova is available in all nova deployment | 03:07 |
| eldon_ | I thought it was rabbitmq's job, now you remind me about nova-api's function. | 03:08 |
| yanyanhu | for senlin, it does happen since senlin-api doesn't know senlin-engine failure | 03:08 |
| yanyanhu | eldon_, yes :) | 03:08 |
| yanyanhu | nova supports that I thiknk | 03:08 |
| eldon_ | So i should review some code about senlin-api and nova-api | 03:08 |
| yanyanhu | I think this is an important feature we want to support | 03:08 |
| eldon_ | yes. | 03:08 |
| yanyanhu | I believe there has been some basic support for this feature. Engine failure detection is now supported | 03:09 |
| yanyanhu | just senlin-api doesn't know the result | 03:09 |
| yanyanhu | will try to figure out how to implement it | 03:09 |
| yanyanhu | after that, senlin-api won't send rpc calls to those failed engine | 03:10 |
| eldon_ | yes, that's interesting :) | 03:10 |
| yanyanhu | will file a bp for it and think how to support this feature :) | 03:11 |
| eldon_ | ok | 03:13 |
| yanyanhu | thanks, eldon_ :) | 03:13 |
| yanyanhu | oh, BTW, do you guys have any new progress for senlin rally test | 03:14 |
| yanyanhu | or any new requirement? | 03:14 |
| eldon_ | we are using our rally tasks doing some tests. | 03:15 |
| yanyanhu | more plugins for senlin has been added | 03:15 |
| yanyanhu | http://git.openstack.org/cgit/openstack/senlin/tree/rally-jobs | 03:15 |
| yanyanhu | and also test example | 03:15 |
| eldon_ | We haven't use your tasks as we uses codes from 2015-12-16 | 03:15 |
| eldon_ | ok~ | 03:16 |
| yanyanhu | I see. but the I think the plugin works as well for that version :) | 03:16 |
| yanyanhu | since the interface of senlinclient didn't change I think | 03:16 |
| yanyanhu | you can have a try and just ping me if you meet any problem :) | 03:16 |
| eldon_ | ok, we will add them in our docker image~ | 03:17 |
| yanyanhu | ok | 03:18 |
| *** eldon_ has quit IRC | 03:19 | |
| *** eldon_ has joined #senlin | 03:37 | |
| eldon_ | I think we should add an interface. | 03:37 |
| eldon_ | like senlin service-list | 03:37 |
| yanyanhu | eldon_, we have one :) | 03:46 |
| yanyanhu | oh, let me check | 03:47 |
| yanyanhu | I know we have DB support for it | 03:47 |
| yanyanhu | let me check whether we have user interface for it | 03:47 |
| yanyanhu | elynn, around? | 03:47 |
| elynn | yes | 03:48 |
| yanyanhu | hi, do we have user interface for checking senlin-engine service status? | 03:48 |
| elynn | senlin-manage service list | 03:48 |
| yanyanhu | nice :) | 03:48 |
| yanyanhu | hi, eldon_ you can use that cmd to check engine service status | 03:48 |
| yanyanhu | if using latest senlin code | 03:49 |
| eldon_ | that is dynamic? | 03:49 |
| *** zzxwill has quit IRC | 03:49 | |
| yanyanhu | I think so | 03:49 |
| eldon_ | if we have this interface, it will not hard to solve that problem I said early. | 03:50 |
| yanyanhu | eldon_, sure | 03:50 |
| yanyanhu | technically, there is no critical obstacle | 03:50 |
| yanyanhu | just need to figure out what is the best way to support it | 03:50 |
| *** eldon_ has quit IRC | 03:54 | |
| *** yuanying has quit IRC | 04:02 | |
| *** yuanying has joined #senlin | 04:03 | |
| *** elynn has quit IRC | 04:34 | |
| *** elynn has joined #senlin | 04:53 | |
| *** elynn has quit IRC | 04:58 | |
| *** elynn has joined #senlin | 04:58 | |
| openstackgerrit | Yanyan Hu proposed openstack/senlin: Releasenote for tempest API test https://review.openstack.org/331939 | 05:05 |
| *** zzxwill has joined #senlin | 05:07 | |
| openstackgerrit | xu-haiwei proposed openstack/senlin: Add Container create/delete driver https://review.openstack.org/331947 | 05:35 |
| *** eldon_ has joined #senlin | 05:36 | |
| openstackgerrit | Yanyan Hu proposed openstack/senlin: Clean action records after deleting cluster/node https://review.openstack.org/329338 | 05:40 |
| openstackgerrit | Merged openstack/senlin-dashboard: modify help_text message https://review.openstack.org/329560 | 05:57 |
| openstackgerrit | Merged openstack/senlin-dashboard: Delete duplicate period https://review.openstack.org/329738 | 05:57 |
| *** eldon_ has quit IRC | 06:05 | |
| openstackgerrit | xu-haiwei proposed openstack/senlin: Add Container create/delete driver https://review.openstack.org/331947 | 06:21 |
| *** eldon_ has joined #senlin | 06:25 | |
| *** eldon_ has quit IRC | 06:31 | |
| openstackgerrit | Yanyan Hu proposed openstack/senlin: Fix oslo.config.opts in setup.cfg https://review.openstack.org/331978 | 07:09 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/senlin-dashboard: Imported Translations from Zanata https://review.openstack.org/331991 | 07:31 |
| yanyanhu | hi, elynn , around? | 07:40 |
| elynn | Hi | 07:40 |
| elynn | hi yanyanhu ? | 07:41 |
| yanyanhu | hi, I found senlin-manage service list doesn't work in my local env | 07:41 |
| yanyanhu | I got the following error: NoSuchOptError: no such option in group DEFAULT: periodic_interval | 07:41 |
| elynn | is it broken? | 07:41 |
| yanyanhu | can you have a try in your env? | 07:41 |
| yanyanhu | guess so | 07:41 |
| elynn | let me | 07:41 |
| yanyanhu | looks like CONF is not loaded correctly | 07:42 |
| yanyanhu | thanks | 07:42 |
| elynn | oh... | 07:42 |
| elynn | let me find out why it doesn't work | 07:42 |
| yanyanhu | ok | 07:43 |
| yanyanhu | I guess we may need to add "cfg.CONF.import_opt('periodic_interval', 'senlin.common.config')" before refering to CONF.periodic_interval | 07:43 |
| yanyanhu | but I don't know why... | 07:43 |
| elynn | Let's have try | 07:43 |
| yanyanhu | ok | 07:43 |
| elynn | You are right! | 07:45 |
| elynn | Not sure why it's needed now... | 07:45 |
| yanyanhu | yes... | 07:45 |
| yanyanhu | it worked before I think | 07:46 |
| yanyanhu | I guess something changed in oslo.config causes this problem | 07:47 |
| yanyanhu | let me try to downgrade oslo.config version to make a try | 07:48 |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Fix NoSuchOptError when using senlin-manage https://review.openstack.org/331998 | 07:50 |
| yanyanhu | didn't work as well after down grading to 3.8.0... | 07:50 |
| yanyanhu | hi, elynn , I think you can just that line before L57, since only format_service needs it | 07:51 |
| yanyanhu | db_sync doesn't need that option | 07:52 |
| elynn | Maybe some codes change in senlin make this potential error appear | 07:52 |
| yanyanhu | yes, that's possible as well | 07:52 |
| elynn | yanyanhu, changing :) | 07:52 |
| yanyanhu | very weird... | 07:52 |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Fix NoSuchOptError when using senlin-manage service list https://review.openstack.org/331998 | 07:55 |
| yanyanhu | hi, elynn, you just updated commit msg in patchset2... | 08:01 |
| elynn | oh... | 08:01 |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Fix NoSuchOptError when using senlin-manage service list https://review.openstack.org/331998 | 08:05 |
| yanyanhu | elynn, hmm, I noticed either import_opt or register_opt must be invoked before options in CONF can be visible | 08:05 |
| yanyanhu | once an option is registered or imported, all other options become available | 08:06 |
| elynn | that's odd | 08:06 |
| yanyanhu | I guess the reason manage.py worked before is some embedded importing includes the invoking of register_opt or import_opt | 08:06 |
| yanyanhu | yes | 08:06 |
| yanyanhu | unreasonable... | 08:06 |
| elynn | I guess so | 08:06 |
| yanyanhu | or oslo.conf is designed like that and we didn't noticed that before :) | 08:07 |
| elynn | some other project | 08:07 |
| elynn | they don't directly import oslo.config | 08:07 |
| elynn | They use a wrapper to register their configs | 08:07 |
| yanyanhu | you mean? | 08:08 |
| yanyanhu | any example? | 08:08 |
| elynn | https://github.com/openstack/keystone/blob/master/keystone/common/config.py#L1098 | 08:08 |
| *** eldon_ has joined #senlin | 08:08 | |
| elynn | So at other places, keystone directly use CONF in common/config.py. | 08:09 |
| elynn | a bit difference from us. | 08:09 |
| elynn | But we are the same as heat. | 08:10 |
| yanyanhu | so all other modules will refer to this 'CONF'? | 08:11 |
| yanyanhu | rather than create a global CONF | 08:11 |
| eldon_ | can we deploy senlin-api and senlin-engine in different hosts? | 08:11 |
| yanyanhu | a new CONF based on global configuration | 08:11 |
| yanyanhu | eldon_, we can I think | 08:11 |
| elynn | If I remember correctly | 08:11 |
| yanyanhu | eldon_, if the message queue is configured correctly | 08:12 |
| yanyanhu | let me check it | 08:12 |
| eldon_ | how it works? | 08:12 |
| eldon_ | how can i configure it ? | 08:12 |
| elynn | haven't go through these codes recent release. | 08:12 |
| eldon_ | thanks | 08:12 |
| yanyanhu | eldon_, will confirm it and give you reply :) | 08:12 |
| yanyanhu | elynn, me neither... | 08:12 |
| yanyanhu | hi, eldon_ , you just need to configure "rabbit_hosts" correctly in you env to ensure both senlin-api and senlin-engine can reach it | 08:14 |
| eldon_ | i see~ | 08:14 |
| yanyanhu | and also rabbit_userid and pass | 08:14 |
| eldon_ | and also this means, one api refers to one engine or multi apis refers to one engine | 08:17 |
| eldon_ | but can't one api refers to multi engines | 08:17 |
| eldon_ | def __init__(self): | 08:17 |
| eldon_ | 29 self._client = messaging.get_rpc_client( | 08:17 |
| eldon_ | 30 topic=consts.ENGINE_TOPIC, | 08:17 |
| eldon_ | 31 server=cfg.CONF.host, | 08:17 |
| eldon_ | 32 version=self.BASE_RPC_API_VERSION) | 08:17 |
| eldon_ | server is fixed. | 08:18 |
| eldon_ | rpc_opts = [ | 08:18 |
| eldon_ | 94 cfg.StrOpt('host', | 08:18 |
| eldon_ | 95 default=socket.gethostname(), | 08:18 |
| eldon_ | 96 help=_('Name of the engine node. This can be an opaque ' | 08:18 |
| eldon_ | 97 'identifier. It is not necessarily a hostname, FQDN, ' | 08:18 |
| eldon_ | 98 'or IP address.'))] | 08:18 |
| yanyanhu | hi, eldon_, multiple engines will listen to the same server.topic I think | 08:21 |
| yanyanhu | which is the place API submit rpc call request | 08:21 |
| yanyanhu | s/which/where | 08:21 |
| yanyanhu | only one engine can get the chance to receive that request and handle it | 08:22 |
| eldon_ | yanyanhu, which website can I paste code? | 08:23 |
| yanyanhu | hmm, if so, the conclusion about engine failure detection in senlin-api we made this morning might be incorrect | 08:23 |
| yanyanhu | paste.openstack.org | 08:23 |
| yanyanhu | you can paste it here | 08:24 |
| eldon_ | http://paste.openstack.org/show/520886/ | 08:26 |
| eldon_ | I compared senlin with heat | 08:27 |
| eldon_ | the init is different | 08:27 |
| eldon_ | we fixed host in senlin rpc client, but heat doesn't fix it. | 08:28 |
| yanyanhu | you mean? | 08:28 |
| eldon_ | I guess, if we fix the host, it will not call function from that host. | 08:30 |
| eldon_ | the structure is host/topic/function. | 08:31 |
| yanyanhu | hi, eldon_ ,that option defines the host of rabbit_mq | 08:31 |
| yanyanhu | but not the host of senlin engine | 08:31 |
| yanyanhu | you can find an option named exactly as "host" in senlin.conf | 08:32 |
| eldon_ | and if host is not defined, it will call function from any hosts where topic is same | 08:32 |
| yanyanhu | that option defines the 'host' of senlin engine | 08:32 |
| eldon_ | yes ,i get it. | 08:32 |
| yanyanhu | actually that is just a symbol, it doesn't have to be a ip address | 08:32 |
| yanyanhu | eldon_, yes | 08:32 |
| yanyanhu | you're right | 08:32 |
| eldon_ | http://paste.openstack.org/show/520891/ | 08:33 |
| yanyanhu | actually naming it to "server" could make it easier to understand :) | 08:33 |
| eldon_ | server=cfg.CONF.host | 08:34 |
| eldon_ | so here in rpc/client.py, it is the engine host | 08:34 |
| yanyanhu | yes, the engine host whose default value is node1 | 08:34 |
| yanyanhu | I think | 08:34 |
| eldon_ | we fix it to localhost, if we didn't configure host in senlin.conf | 08:34 |
| eldon_ | that is not appropriate, right? | 08:35 |
| yanyanhu | you mean localhost? | 08:35 |
| yanyanhu | localhost is ok | 08:35 |
| *** eldon_ has quit IRC | 08:35 | |
| yanyanhu | it doesn't matter | 08:35 |
| *** eldon_ has joined #senlin | 08:36 | |
| yanyanhu | hi, eldon_, localhost is ok | 08:36 |
| yanyanhu | you can even name it to 'abc' | 08:36 |
| eldon_ | but if we only deploy api in host1, and not deploy engine in host1, it will not work. | 08:37 |
| eldon_ | so, I suggest not pass server to get_rpc_client function | 08:37 |
| eldon_ | just like heat. | 08:38 |
| yanyanhu | hmm, are the rabbit_hosts option in api node and engine node the same? | 08:38 |
| eldon_ | yes, same. | 08:38 |
| yanyanhu | eldon_, if so, we can't differentiate multiple senlin engines if they belongs to different deployments | 08:39 |
| yanyanhu | rpc server of multiple senlin engines | 08:39 |
| yanyanhu | all of them will have the same topic | 08:39 |
| yanyanhu | without unique server being specified | 08:39 |
| eldon_ | ok, maybe I should try first to test something:) | 08:40 |
| yanyanhu | let me double check the code of oslo.messging | 08:40 |
| yanyanhu | eldon_, yes, you can make a test to observe the behavior of engine and api :) | 08:41 |
| eldon_ | ok, thanks a lot | 08:41 |
| yanyanhu | no problem | 08:42 |
| yanyanhu | BTW, eldon_ , you can try to enable multiple engine worker to see whether it can meet your requirement for handling engine failure when enabling API loadbalancing | 08:45 |
| yanyanhu | if an engine dies completely and doesn't receive any rpc calls, I think it won't cause rpc request hanging | 08:46 |
| eldon_ | now, it returns time out error. | 08:50 |
| yanyanhu | so there are multiple senlin-apis being loadbalanced | 08:51 |
| yanyanhu | and each of them has multiple senlin-engine at backend? | 08:51 |
| yanyanhu | oh, you are doing the rabitmq test | 08:51 |
| eldon_ | I guess it's caused by the api.py:get_rpc_client function I said early. | 08:51 |
| eldon_ | yes. | 08:52 |
| yanyanhu | hmm, let me have a try | 08:52 |
| yanyanhu | need to setup senlin in the second host, need several minutes | 08:56 |
| yanyanhu | hi, eldon_ is there any error log reported during engine and api started up | 09:00 |
| eldon_ | no, they can start up successfully. | 09:00 |
| yanyanhu | ok, that's means rabbit connection is correct | 09:01 |
| yanyanhu | I installed senlin-api and engine in the same host, but configured rabbit_host to another host | 09:04 |
| yanyanhu | it works well | 09:04 |
| yanyanhu | the installation is slow... for poor network | 09:04 |
| *** elynn has quit IRC | 09:05 | |
| yanyanhu | hi, eldon_ so I guess deploying senlin-api and senlin-engine into different hosts will work as well | 09:07 |
| *** eldon_ has quit IRC | 09:07 | |
| *** openstackgerrit has quit IRC | 09:18 | |
| *** openstackgerrit has joined #senlin | 09:19 | |
| *** ChanServ sets mode: +v openstackgerrit | 09:19 | |
| *** yanyanhu has quit IRC | 10:30 | |
| *** eldon_ has joined #senlin | 12:02 | |
| *** zzxwill_ has joined #senlin | 12:22 | |
| *** zzxwill has quit IRC | 12:23 | |
| *** yanyanhu has joined #senlin | 12:53 | |
| yanyanhu | hi, eldon_, around? | 12:57 |
| yanyanhu | for coming meeting, I added a topic about deploying senlin-api and senlin-engine in different hosts | 12:58 |
| yanyanhu | hope can make a quick discussion about it with you | 12:58 |
| yanyanhu | hi, guys, meeting will start in 2 minutes | 12:59 |
| *** elynn has joined #senlin | 12:59 | |
| yanyanhu | in meeting channel, Qiming can not join it for family reason, so I will hold this meeting on behalf | 12:59 |
| yanyanhu | hi, elynn, meeting will start in openstack-meeting channel | 13:00 |
| elynn | will join | 13:00 |
| yanyanhu | :) | 13:00 |
| elynn | Just in time :) | 13:01 |
| yanyanhu | yep | 13:02 |
| yanyanhu | hi, lixinhui , around? | 13:08 |
| *** xuhaiwei has quit IRC | 13:13 | |
| eldon_ | yes | 13:13 |
| elynn | eldon_, we are holding senlin meeting at channel #openstack-meeting | 13:14 |
| elynn | welcome to join | 13:14 |
| eldon_ | ok | 13:14 |
| elynn | you can share us some work you do recently. | 13:14 |
| eldon_ | I am very glad to have this chance. | 13:15 |
| yanyanhu | hi, eldon_ , plz join openstack-meeting channel | 13:17 |
| yanyanhu | we can make a discussion about deploying senlin-engine and api individually there | 13:17 |
| *** lixinhui_ has joined #senlin | 13:21 | |
| *** eldon_ has left #senlin | 13:24 | |
| *** eldon_ has joined #senlin | 13:24 | |
| *** lixinhui_ has quit IRC | 13:41 | |
| openstackgerrit | Merged openstack/senlin: Fix NoSuchOptError when using senlin-manage service list https://review.openstack.org/331998 | 13:42 |
| *** openstackgerrit has quit IRC | 13:48 | |
| *** openstackgerrit has joined #senlin | 13:49 | |
| *** ChanServ sets mode: +v openstackgerrit | 13:49 | |
| *** yanyanhu has quit IRC | 14:05 | |
| *** eldon_ has quit IRC | 14:08 | |
| *** zzxwill_ has quit IRC | 14:35 | |
| *** zzxwill_ has joined #senlin | 14:36 | |
| *** elynn has quit IRC | 14:37 | |
| -openstackstatus- NOTICE: Gerrit is being restarted now to apply an emergency security-related configuration change | 16:06 | |
| *** zzxwill_ has quit IRC | 16:14 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-senlinclient: Updated from global requirements https://review.openstack.org/332379 | 18:05 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/senlin: Updated from global requirements https://review.openstack.org/332390 | 18:06 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/senlin-dashboard: Updated from global requirements https://review.openstack.org/332391 | 18:06 |
| *** Drago has joined #senlin | 18:25 | |
| *** Drago has quit IRC | 18:26 | |
| *** Drago has joined #senlin | 18:26 | |
| *** jdandrea_ is now known as jdandrea | 18:27 | |
| *** Drago has quit IRC | 23:23 | |
| *** Drago has joined #senlin | 23:23 | |
| *** Drago has quit IRC | 23:28 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!