Friday, 2016-03-11

*** zzxwill has quit IRC00:00
*** sridhar_ram1 has joined #senlin00:01
*** sridhar_ram has quit IRC00:03
*** zzxwill has joined #senlin00:06
*** Qiming has quit IRC00:08
*** zzxwill has quit IRC00:20
*** openstackgerrit_ has quit IRC00:23
*** zzxwill has joined #senlin00:24
*** openstackgerrit_ has joined #senlin00:25
*** zzxwill has quit IRC00:41
*** zzxwill has joined #senlin00:42
*** openstackgerrit_ has quit IRC00:48
*** openstackgerrit_ has joined #senlin00:50
*** zzxwill has quit IRC00:52
*** sridhar_ram1 has quit IRC00:57
*** zzxwill has joined #senlin00:59
*** sridhar_ram has joined #senlin01:01
*** Qiming has joined #senlin01:07
*** zzxwill has quit IRC01:13
*** zzxwill has joined #senlin01:18
*** zzxwill has quit IRC01:33
*** zzxwill has joined #senlin01:34
*** sridhar_ram has quit IRC01:35
*** zzxwill has quit IRC01:49
*** zzxwill has joined #senlin01:49
*** Yanyanhu has joined #senlin01:58
*** zzxwill has quit IRC02:00
*** zzxwill has joined #senlin02:12
*** elynn has joined #senlin02:19
*** zzxwill has quit IRC02:24
*** elynn has quit IRC02:25
*** elynn has joined #senlin02:25
*** zzxwill has joined #senlin02:30
Qiminghi, guys02:39
Qimingduring a discussion with some early users, I was answering their questions regarding how authentication is done, how multi-region deployment is performed02:40
Qimingso I was revisiting the api middleware source to make sure I'm talking accurately02:41
Qiminghowever, I am not feeling comfortable when I saw the trust middleware talks directly to db02:41
Yanyanhuyou mean the part about keystone middleware?02:41
Qimingthat is the question I want to raise02:42
Qimingshould we eliminate direct DB interactions from the API layer02:42
Qimingthat will make the architecture much cleaner02:42
YanyanhuQiming, this is what we wanted. but didn't find a better way to address problem before02:42
QimingI want to introduce some "internal" RPC interfaces for trust retrieval and creation02:42
Qimingin the api pipeline, trust comes after the context middleware02:43
Qimingso it is possible to decouple it from db02:43
Yanyanhuhmm, this is feasible. But on potential problem is the API request handling progress could be blocked if engine is busy02:44
QimingI can look into that if there is no objections on this02:44
Qimingif the engine is busy, you cannot make any progress anyway02:44
Qimingdon't believe that is a valid concern02:44
YanyanhuI mean the pgress inside middleware02:44
Yanyanhuyes, but that's a little bit different02:44
Qiminghave to leave for a moment02:45
Yanyanhuanyway, not a serious problem02:45
Yanyanhuok, ttyl02:45
lixinhuiYanyanhu02:50
lixinhuihave you ever meeting this problem02:50
lixinhuioctavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying02:50
Yanyanhuno...02:51
lixinhuiOnce you said some config need to pay attention02:51
Yanyanhuhi, lixinhui, octavia is another lb service in openstack?02:51
lixinhuiit is the provider of lb02:51
lixinhuiin haproxy driv er02:51
lixinhuiI wonder what it is02:52
Yanyanhuoh, I see02:52
Yanyanhuabout the configure, I mean the driver type and I can't recall whether there is some authentication related options02:52
Yanyanhuby the driver, it's actually the service provider02:53
lixinhuiDifficult here is hard to find real error02:56
lixinhuibehind the PENDING-to....02:56
Yanyanhuyes...02:58
YanyanhuI remember I used to check haproxy related process or something to ensure it works well02:58
Yanyanhubut can't recall the detail how to check it...02:58
lixinhuiokay03:00
lixinhuiso you do not use  octavia03:00
Qimingsounds like a networking problem?03:03
Yanyanhulixinhui, yes, I didn't use it before03:03
Qimingafter your ha_proxy machine is up03:03
Qimingyou can try connect to it manually03:03
Qimingsometimes, it takes time to have the controller realize there are new destination reachable03:04
Qimingsometimes, the security group thing jumps into the way when you try connecting to a VM03:04
*** zzxwill has quit IRC03:12
*** zzxwill has joined #senlin03:16
*** elynn_ has joined #senlin03:26
*** elynn has quit IRC03:30
*** gmann has left #senlin03:48
*** elynn_ has quit IRC04:01
*** shu-mutou-AFK is now known as shu-mutou04:02
*** zzxwill has quit IRC04:11
*** zzxwill has joined #senlin04:12
*** zzxwill has quit IRC04:33
*** zzxwill has joined #senlin04:51
*** elynn_ has joined #senlin04:56
*** elynn_ has quit IRC05:00
*** elynn_ has joined #senlin05:01
*** zzxwill has quit IRC05:14
*** zzxwill has joined #senlin05:18
*** sridhar_ram has joined #senlin05:23
*** zzxwill has quit IRC05:27
*** zzxwill has joined #senlin05:30
*** jdandrea has quit IRC05:33
*** heyongli has joined #senlin05:39
*** jdandrea has joined #senlin05:40
*** heyongli has quit IRC05:41
*** heyongli has joined #senlin05:41
*** zzxwill has quit IRC05:43
*** zzxwill has joined #senlin05:44
*** heyongli has quit IRC05:46
*** heyongli has joined #senlin05:46
*** jdandrea_ has joined #senlin05:49
*** jdandrea has quit IRC05:50
*** zzxwill has quit IRC05:54
*** zzxwill has joined #senlin05:55
*** heyongli has quit IRC05:56
*** heyongli has joined #senlin05:57
Yanyanhuhi, Qiming, I'm considering retore the obsoleted BP of access permission control in Senlin. Since this is not a work that will be done in this cycle, I will mark it as under discussion.06:02
Qimingokay06:04
QimingYanyanhu, http://www.gossamer-threads.com/lists/openstack/dev/52819?page=last06:06
*** heyongli has quit IRC06:06
*** heyongli has joined #senlin06:07
Yanyanhuwill read it carefully :)06:07
Qimingwhat we are proposing is actually called "DYNAMIC" RBAC in keystone terms06:07
Yanyanhuyes06:08
Qimingspes from ayoung is here: https://review.openstack.org/#/c/279379/1/specs/backlog/dynamic-policy.rst06:08
Yanyanhuthanks a lot for this clue06:10
*** heyongli has quit IRC06:17
*** heyongli has joined #senlin06:17
*** openstackgerrit_ has quit IRC06:17
*** openstackgerrit_ has joined #senlin06:18
*** zzxwill has quit IRC06:25
*** heyongli has quit IRC06:27
*** zzxwill has joined #senlin06:28
lixinhuiQiming, it should irrelevant to secuity group06:33
*** zzxwill has quit IRC06:33
lixinhuiand the loadbalancer instance hide behind and nova can not list it06:33
*** zzxwill has joined #senlin06:33
lixinhuithe port id seems right06:33
lixinhuibut the error means the ha_procy driver can not connect to the built instance06:34
lixinhuiI will search more to see what is the process06:34
lixinhuibehind06:34
*** shu-mutou is now known as shu-mutou-AFK06:46
*** sridhar_ram has quit IRC06:56
*** zzxwill has quit IRC06:58
*** zzxwill has joined #senlin07:07
*** elynn_ has quit IRC07:09
Qimingwhat is the built instance referring to?07:10
Qimingif you cannot list it, try the admin account?07:11
lixinhuiadmin is the same07:14
*** elynn has joined #senlin07:14
lixinhuithe built instance is the loadbaancer imstance in lbaas v207:14
Qimingthat instance is a VM07:16
Qimingmaybe it is in a special project07:16
Qiminglixinhui, this line: http://git.openstack.org/cgit/openstack/octavia/tree/etc/octavia.conf#n14107:18
Qimingit means the amphora instance may need a security group, but by default it is empty07:18
Qimingthat is something you may want to verify07:18
Qiminganother possibility, it takes too long to wait for the instance to become reachable07:19
lixinhuiit will us ethe default07:19
lixinhuiok07:20
Qimingokay, you have configured your default security group?07:20
lixinhuiyes07:20
Qiminggreat07:20
lixinhuithe port it used to ccreate the instance is right one07:20
Qimingthe new "default" security group is not blocking any port range, right?07:20
lixinhuien07:20
Qimingokay, that almost eliminates the possibility of secgroup07:21
lixinhuiall the samples I searched about lbaas v2 is using octavia07:22
lixinhuiso I am seaeching how to check correctness of ha_proxy process07:22
Qimingyou got to log into that instance07:22
lixinhuiokay07:22
lixinhuilet me try to find some clue to do it07:23
Qimingone way to try07:23
Qimingexport OS_USERNAME=admin07:23
Qimingthen 'nova list --all-tenants 1'07:23
QimingI didn't try it before07:24
lixinhuiok07:24
lixinhuiwill try it07:24
Qimingbut I really believe the nova instance is created somewhere07:24
lixinhuien07:24
*** gongysh has joined #senlin07:26
Qiminggongysh, hi07:30
*** zzxwill has quit IRC07:31
gongyshQiming,  hi07:32
Qiminghi, we have encountered some LBaaS and octavia problem07:33
Qimingwhere I believe you are THE expert, :)07:33
*** zzxwill has joined #senlin07:33
gongyshQiming,  no, I have not play with it yet. beyond mime.07:33
Qimingokay, give me a name07:34
QimingI'll let you go07:34
*** zzxwill has quit IRC07:38
Qiming???07:38
Qiminggongysh, ???07:38
* gongysh looking ...07:39
Qimingamong these names:07:39
QimingAdam Harwellflux.adam@gmail.com07:39
QimingBertrand Lallaubertrand.lallau@gmail.com07:39
QimingBrandon Loganbrandon.logan@rackspace.com07:39
QimingDoug Wiegleydougwig@parkside.io07:39
QimingGerman Eichbergergerman.eichberger@hp.com07:39
QimingMichael Johnsonjohnsomor@gmail.com07:39
QimingStephen Balukoffstephen@balukoff.com07:39
gongyshhttps://wiki.openstack.org/wiki/Octavia07:40
gongyshQiming, IRC: #openstack-lbaas07:41
Qimingokay, :)07:41
Qimingyou are of no value now07:41
* Qiming pulls the trigger ...07:41
gongyshQiming,  shot dead.07:43
QimingLOL07:43
*** zzxwill has joined #senlin07:45
Yanyanhuhi, Qiming, just quickly went through the spec from adam young07:52
Yanyanhuit's very helpful for controlling http resource access based on role check07:52
Qimingyep07:53
Qimingit looks like one07:53
Yanyanhubut it's not satisfied enough to meet our requirement07:53
Qimingbut as ayoung mentioned07:53
Yanyanhue.g. control the access to each single entity07:53
Qimingapi level or pre-api level access control is still limited07:53
Yanyanhuyes07:53
Qimingit is not touching the resources inside database07:54
Yanyanhuit's understandable07:54
Yanyanhuright07:54
Qimingyes07:54
Qimingso, with that in minde07:54
Yanyanhuso I guess maybe we can have both of them07:54
Qimings/minde/mind07:54
Yanyanhudefinitely07:54
Qimingwe can work on the permission thing07:54
Yanyanhuright07:54
Yanyanhuthis is what I'm thinking now07:54
Qimingcool07:54
Yanyanhuwill try to mix them07:54
Yanyanhuleverage what keystone can provide us as much as possible07:55
Qiminghowever, I'm not so sure if a per-resource access control is needed07:55
Yanyanhuand implement the function it doesn't provision in Senlin07:55
Yanyanhuwill leave for a while07:55
Yanyanhuttyl07:55
Qimingwe may need 'chown', 'chmod' calls07:55
Yanyanhuyea, will further think about it07:55
Yanyanhuyes07:56
Yanyanhuthis is what we want07:56
Yanyanhuleave now07:56
Yanyanhugo back later07:56
*** zzxwill has quit IRC08:02
*** zzxwill has joined #senlin08:03
Qimingso ...08:05
Qimingthis is one of the quiet weeks where we are not supposed to add new features08:05
Qiminguntil newton development opens08:06
QimingI'd like to give testing a higher priority08:06
Qiminge.g. tempest, rally, functiona, stress ... etc08:06
Qimingnone of those will break the existing code08:06
Qiming(hopefully) :)08:06
YanyanhuQiming, yes, actually I'm now planning to add some functional test for failure cases which can be implemented inside existing framework08:16
Yanyanhue.g. creating cluster with invalid profile-id08:16
Yanyanhusomething like this08:16
Yanyanhufor more complicated cases, or API consistency test, may need new design08:17
Qimingwe can start shifting to tempest I think08:18
Yanyanhuyou mean shift functional test to tempest? Or using tempest for API test08:19
YanyanhuI recalled maintaining functional test inside each project individually is recommened?08:19
Qimingat least, api surface test should be done via tempest plugin08:20
Qimingyes, no conflict there08:20
Qimingthe code should live in senlin08:21
YanyanhuI see08:21
Qimingwe got some pretty good guidance the other day from a tempest expert08:21
Qimingwe can check how it works by looking into congress08:21
Yanyanhuso the difference is where we define the job, devstack gate or tempest?08:21
Qimingfor example08:21
Qimingit can be a tox env specified in tox.ini08:22
Yanyanhuok08:23
*** zzxwill has quit IRC08:25
*** zzxwill has joined #senlin08:29
*** zzxwill has quit IRC08:37
openstackgerritQiming Teng proposed openstack/senlin: Add engine service RPC api for credentials  https://review.openstack.org/29159908:40
*** zzxwill has joined #senlin08:40
openstackgerritQiming Teng proposed openstack/senlin: Add engine service RPC api for credentials  https://review.openstack.org/29159908:45
*** gongysh has quit IRC08:46
*** zzxwill has quit IRC08:47
*** zzxwill has joined #senlin08:50
*** gongysh has joined #senlin08:51
lixinhuiQiming08:56
lixinhuihttp://paste.openstack.org/show/490105/08:56
lixinhuithat is the loadbalancer instance08:56
lixinhuibut can not ssh it08:56
lixinhuibecause of lack of router08:56
Qimingokay, checking08:58
Qimingyou cannot even ssh to it via 10.0.0.4?08:59
Qimingit has a security group named lb-mgmt-sec-grp09:00
Qimingand the management network is 192.168.0.409:01
Qimingyou will need a octavia_ssh_key to ssh, after switching to project eb59c8ab580b40c586c5bda06f51c8f809:01
Qimingif the ssh is still rejected, you can treat it as a normal SSH problem09:02
openstackgerritQiming Teng proposed openstack/senlin: RPC support for credential operations  https://review.openstack.org/29161409:04
lixinhuino reponse too by 10.0.0409:05
Qimingcan you ping it?09:06
lixinhuiI can not09:07
lixinhuibut sure for management net09:07
lixinhuiIP09:07
lixinhuiand I can find the key under /etc/octavia/.ssh/octavia_ssh_key09:08
*** elynn_ has joined #senlin09:09
*** elynn has quit IRC09:10
Qiming"but sure for management net" ... what does this mean?09:11
Qimingyou can reach it from management network?09:13
lixinhuiI can ping through the management net09:13
Qimingthen you should log in via management net09:13
lixinhuibut ssh faild although I can find the key09:13
lixinhuissh: connect to host 192.168.0.4 port 22: No route to host09:14
Yanyanhusecurity group problem?09:14
Qimingyou can ping it09:14
lixinhuihave done this09:14
lixinhuineutron security-group-rule-create f29c45ff-dfd3-44a3-a1a6-b7716eff9041 --protocol tcp --port-range-min 22 --port-range-max 2209:14
lixinhuiping workds09:15
lixinhuiworks09:15
lixinhuif29c45ff-dfd3-44a3-a1a6-b7716eff9041 is the management net09:15
Qimingssh is giving an inaccurate error message ...09:15
Yanyanhudid you apply this security group when booting up VM?09:15
Qimingwhat are the first few characters of the octavia_ssh_key file?09:17
lixinhuiPermission denied (publickey).09:19
lixinhuihttp://paste.openstack.org/show/490109/09:19
*** zzxwill has quit IRC09:20
*** zzxwill has joined #senlin09:21
Qimingokay, that is a private key09:22
Qimingssh -i /etc/octavia/.ssh/octavia_ssh_key 192.168.0.4 doesn't work?09:23
lixinhuien, same error09:23
Qimingit didn't ask for assword?09:24
lixinhuino...09:24
Qimingokay, the image has disabled password authentication09:25
Qimingdo you know the OS installed in the image?09:25
lixinhuibu default it should be ubuntu09:27
lixinhuihttp://docs.openstack.org/developer/octavia/specs/version0.5/base-image.html09:27
Qimingssh -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@192.168.0.4  ?09:32
QimingI think we are creating a lot of problems because the installation and configuration of octavia wasn't complete09:33
Qimingwhen I tried read the source code of rest_api_driver.py09:33
QimingI found that octavia is actually doing a REST call09:34
Qimingwith a server certificate09:34
Qimingwhich should be configured in haproxy_amphora section, named 'server_ca'09:34
Qimingthat is the only thing required for the REST request09:35
lixinhuiys09:36
lixinhuiit is set by default value09:36
Qiminghttp://docs.openstack.org/developer/octavia/design/version0.5/component-design.html#some-notes-on-controller-amphorae-communications09:36
lixinhuiand to avoid the mismatch problem09:37
Qiminghow is the server_ca generated?09:37
Qimingany script doing that?09:37
lixinhuiI ensure generate the octavia.conf very time now09:37
Qimingbut generating octavia.conf doesn't mean the generation of a server_ca09:38
lixinhuiopt/stack/octavia/devstack/plugin.sh09:39
Qimingokay, I see09:39
*** elynn_ has quit IRC09:41
lixinhuiit is indeed generated with the conf together09:41
Qimingokay, so it is generating certificates09:42
*** Yanyanhu has quit IRC09:42
lixinhuien09:42
Qimingif I were you I will add a LOG.error('%s' % reqargs) at here: http://git.openstack.org/cgit/openstack/octavia/tree/octavia/amphorae/drivers/haproxy/rest_api_driver.py#n24109:44
Qimingor, at line 242, try catch the exception and print it out09:45
QimingI really hate this design, which is making the most fragile link very difficult to debug09:46
lixinhuiokay09:46
Qimingafter you have modified the code09:47
*** elynn_ has joined #senlin09:47
Qimingyou don't have to reinstall devstack, alright?09:47
Qimingjust kill the octavia service and restart it09:47
lixinhuiI see09:47
Qimingalso, you may want to try your luck in the #openstack-lbaas channel09:49
QimingI think such a problem must be among the top 5 in their FAQ list09:49
lixinhuiactually09:49
lixinhuithere have some one report this bug09:49
lixinhuibut no further action there09:50
*** elynn_ has quit IRC09:52
lixinhuithanks09:55
lixinhuiI leave some message at the channel hope someone could help09:55
lixinhuithe error content is09:55
lixinhui2016-03-11 17:47:52.794 3785 ERROR octavia.amphorae.drivers.haproxy.rest_api_driver [-] {'url': 'https://192.168.0.5:9443/0.5/plug/vip/10.0.0.5', 'verify': '/etc/octavia/certs/ca_01.pem', 'json': {'subnet_cidr': u'10.0.0.0/24', 'gateway': u'10.0.0.1', 'mac_address': u'fa:16:3e:94:b0:2c'}, 'timeout': (10.0, 60.0), 'headers': {'User-Agent': 'Octavia HaProxy Rest Client/0.5 (https://wiki.openstack.org/wiki/Octavia)'}}09:55
*** zzxwill has quit IRC09:58
Qimingokay09:59
Qimingthe above log shows you that the rest api is invoked from the 10 network, not the 192.168 network10:00
*** zzxwill has joined #senlin10:01
Qimingsorry, it is about getting data from 192.168.0.510:01
Qimingand the port is now 944310:01
Qimingis that port opened?10:02
* Qiming is feeling very very very hungry ... 10:03
lixinhuiyes10:08
*** elynn_ has joined #senlin10:11
*** Qiming has quit IRC10:11
lixinhui~$ netstat -nap|grep 944310:14
lixinhui(No info could be read for "-p": geteuid()=1000 but you should be root.)10:14
lixinhuitcp        0      0 0.0.0.0:9443            0.0.0.0:*               LISTEN      -10:14
*** elynn_ has quit IRC10:20
*** gongysh has quit IRC10:31
*** zzxwill has quit IRC10:35
*** zzxwill has joined #senlin10:38
*** zzxwill has quit IRC10:40
*** zzxwill has joined #senlin10:50
*** zzxwill has quit IRC10:54
-openstackstatus- NOTICE: Gerrit is going to be restarted due to bad performance10:55
*** ChanServ changes topic to "Gerrit is going to be restarted due to bad performance"10:55
*** ChanServ changes topic to "IRCLog: http://eavesdrop.openstack.org/irclogs/%23senlin/ | Bugs: bugs.launchpad.net/senlin | Review: https://review.openstack.org/#/q/project:openstack/senlin,n,z"11:01
-openstackstatus- NOTICE: Gerrit has been restarted successfully11:01
*** Qiming has joined #senlin11:01
*** zhenguo_ has quit IRC12:10
*** zzxwill has joined #senlin12:37
*** zzxwill has quit IRC13:41
*** zzxwill has joined #senlin13:44
*** Qiming has quit IRC13:54
*** Qiming has joined #senlin13:54
*** lixinhui_ has joined #senlin14:16
*** lixinhui_ has quit IRC14:17
*** zzxwill has quit IRC14:55
*** zzxwill has joined #senlin14:57
*** Qiming has quit IRC14:58
*** Qiming has joined #senlin14:58
*** zzxwill has quit IRC14:59
*** zzxwill has joined #senlin15:57
*** Qiming has quit IRC16:11
*** zzxwill has quit IRC16:39
*** zzxwill has joined #senlin16:42
*** zzxwill has quit IRC16:46
*** zzxwill has joined #senlin16:57
*** zzxwill has quit IRC17:23
*** zzxwill has joined #senlin17:31
*** zzxwill has quit IRC17:45
*** zzxwill has joined #senlin17:58
*** zzxwill has quit IRC18:04
*** zzxwill has joined #senlin18:17
*** zzxwill_ has joined #senlin18:31
*** zzxwill has quit IRC18:31
*** zzxwill_ has quit IRC18:38
*** sridhar_ram has joined #senlin18:58
*** sridhar_ram1 has joined #senlin19:01
*** sridhar_ram has quit IRC19:03
*** sridhar_ram1 is now known as sridhar_ram19:49

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!