13:00:07 #startmeeting senlin 13:00:08 Meeting started Tue Mar 29 13:00:07 2016 UTC and is due to finish in 60 minutes. The chair is Qiming. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:00:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:00:11 The meeting name has been set to 'senlin' 13:00:35 evening/morning 13:00:43 Hi 13:00:50 HI 13:01:20 Hi 13:01:28 Hello Qiming. 13:01:54 just posted an agenda for meeting today 13:01:57 #link https://wiki.openstack.org/wiki/Meetings/SenlinAgenda 13:02:14 please feel free to add items if you have some 13:02:21 \o 13:02:37 #topic austin summit planning 13:03:24 I think we are in good shape regarding the deep dive session and the autoscaling session 13:03:45 anything you want to discuss regarding these two talks? 13:04:23 The demo is still under refinement 13:04:41 we have just got some preliminary session allocation info 13:04:45 thanks for the help of elynn today on security group 13:04:51 Just begin to think of it 13:05:02 so far we don't have conflicts between the talk sessions and the design summit 13:05:31 will discuss with you later if I need help. 13:05:46 elynn, consider backport that to stable/mitaka? 13:06:03 ok, keep in mind 13:06:19 Is it merged? 13:06:24 not yet 13:07:01 speaking of container cluster one, let's see what we can get as a poc in this week 13:07:28 haiwei, we will need to sync on that more frequently 13:07:57 Qiming, I started to implement it in senlin, tried to get container list 13:08:07 it is almost there 13:08:17 okay 13:08:36 #link design summit sessions 13:08:44 sorry 13:08:54 #link https://etherpad.openstack.org/p/newton-senlin-sessions 13:09:24 we have a S7 added though we only have 5 working session + 1 fishbowl session applied 13:10:06 adopting existing nova server sounds an interesting scenario 13:10:33 but you know, senlin needs a profile for any node created (or adopted) 13:11:11 yes 13:11:28 even if we want to adopt an existing server, we may still have to do two steps: 13:12:07 Should we add the 'semi-autoscaling' stuff to this? 13:12:11 openstack cluster profile create --from --type os.nova.server 13:13:14 openstack node create --spec --from 13:13:37 cschulz, you mean design summit session? 13:13:52 Yes, or is it too early? 13:14:18 cschulz, no, we had a plan for design summit sessions 13:14:31 we are yet to finalize it 13:14:55 it sounds to me that S3 is not so interesting now 13:15:34 we have already started tempest and rally investigation, tempest framework is getting merged, not a lot to discuss 13:15:49 since we already start working on that 13:15:56 Yanyan has started contributing to rally about support to Senlin 13:16:15 so how about we replace S3 with S7? 13:16:32 I agree 13:16:35 no problem 13:16:40 the session number is limited? 13:16:52 it's ok I think 13:17:11 yes, haiwei, we only have 5 + 1 + 1 13:17:25 ok 13:17:32 5 work sessions, + 1 fishbowl, +1 developer meetup 13:18:02 then we can make S3 cover both your item and cschulz's proposal 13:18:34 sounds good 13:18:37 haiwei, I'm still leaving you as the driver 13:18:54 ok 13:21:08 let's move on 13:21:23 #topic high-priority bugs before final release 13:21:45 the security group one is a good candidate 13:22:07 are you aware of any high priority bugs we should merge into stable branch? 13:22:24 it is being pushed out of door next week 13:23:16 btw, Liuqing, just proposed a release of senlin-dashboard 0.2.0 https://review.openstack.org/#/c/298710/ 13:24:18 emm... no one is yelling, great, let's continue work on newton items 13:24:29 #topic newton work items 13:24:42 #link https://etherpad.openstack.org/p/senlin-newton-workitems 13:25:27 testing side, no news from china mobile yet, but they have some guys evaluating multi-region deployment, that I know 13:25:46 Qiming 13:25:59 I think it would be better if we can schedule 13:26:05 some meeting with them here 13:26:09 yes? 13:26:12 about oslo and 13:26:20 scalability things 13:26:26 okay, will do 13:27:01 remember they once showed some parameters there 13:27:03 #action Qiming to sync with china mobile guys on oslo.messaging and scalability lessons learnt 13:27:06 when mid-cycle 13:27:24 will schedule a call with them soon 13:27:33 great! 13:28:02 tempest testing, where are we, elynn? 13:28:24 I saw some patches coming in 13:28:41 yes 13:28:52 anything you want to discuss, or it is all under your good control? 13:29:05 An initial patch is submitted and I will rebase it based on your comments. 13:29:20 It's good for now. 13:29:41 okay, please consider adding support to some failure cases, such as node creation failures, driver failures 13:29:48 Just need some time to move all tests to use tempest plugin. 13:29:51 Sorry, 'china mobile'? To deploy clusters in their cloud by senlin? 13:29:54 It's trivial. 13:30:54 great. what we need is not just tests against successes, but also those against failures 13:31:28 as a user, I really hope the service never fails, and if it fails, it fails in a predictable way 13:32:09 zzxwill, that is true 13:33:12 lixinhui_, do we have some data about stress testing senlin? 13:33:22 Okay, so let me think of it. Haven't figure out how to implement them in tempest, I will keep on digging. 13:33:39 Yes, Qiming 13:33:45 with simulated drvier 13:34:05 Thx. 13:34:20 one engine runing with 8 cpu and 16mem can support 1000 scale node creation without any problem 13:34:44 With real driver, we meet some image authentication failure 13:34:45 does multi-engine setup help in scaling? 13:35:04 so it is broken, but not at senlin side 13:35:25 not in senlin side 13:35:46 okay, hope we can learn something from china mobile guys 13:35:52 with two engine, 2500 creation has no problem 13:35:58 but will do more tests 13:36:07 cool. 13:36:12 to find your question 13:36:30 find answer/evidence to your question 13:36:37 okay 13:37:03 Yes, it would nice if we can have more exchange with china mobile guys 13:37:06 health mgmt side, do we have lb based health poller? 13:37:12 yes 13:37:17 I implemented one 13:37:32 and adopt it into the auto-scaling demo 13:37:44 okay, would be curious to see how it looks like, :) 13:38:11 will discuss more details with you tomorrow 13:38:19 okay 13:38:24 nowdays, I polling neutron for node status 13:38:29 member status 13:39:02 the status is changed by health monitor of lbaas 13:39:09 regarding HA scenario, I have pasted some references to the etherpad 13:39:11 #link https://etherpad.openstack.org/p/senlin-ha-recover 13:39:33 yes 13:39:45 I read them and treasure them very much 13:40:14 it is very complicated and a huge problem domain 13:40:35 I think we should try these works in permitted scope 13:40:43 need to collect a lot inputs to form a global picture, then find something we want to focus on 13:41:00 exactly 13:41:00 yes 13:41:17 there are interests in auto-evacuation 13:41:29 but ... that is only one of the scenarios 13:41:45 I think this is another area where we could use Zaqar to get information into Senlin 13:41:58 we need to find out how to amend our health policy to get fencing done 13:42:18 Okay 13:42:20 A cluster in trouble could send a notification in via a Zaqar queue 13:42:33 yes, cschulz, we will need many ears and eyes 13:43:02 right, we have some related todo items documented in source tree 13:43:26 we need to yield notifications when something note worthy happens 13:43:50 we also need to generate some meters for ceilometer if there are such use cases 13:44:30 lixinhui_, regarding the full end-to-end story documentation 13:44:34 I would think that a cluster size meter would be very useful 13:44:53 I'm thinking of sharing your autoscaling template into heat-templates project some day 13:45:10 i.e. raise an alarm when a cluster grows beyond some alarm specified limit. 13:45:29 YeYes, Qiming 13:45:37 we send out an event after cluster size change 13:45:57 leave the evaluation and alarming logic to telemetry/metering/alarming service, :) 13:46:43 what do you mean by generate some meters for ceilometer if there are such use cases 13:47:09 i think we will leverage heat for alarm creation 13:47:14 service like monasca/ceilometer would collect runtime statistics from every service 13:47:42 okay 13:47:56 it is up to senlin to generate whatever such measurements 13:48:04 I see 13:48:17 thanks for explanation 13:48:33 api-microversioning 13:48:38 it is almost done 13:49:14 still poking around and see if there things missed, will mark the bp completed if nothing else left 13:49:49 * Qiming reading semi-autoscaling notes 13:50:36 so, if we are leveraging zaqar for messaging/notifications 13:50:40 I'm not sure about the notification part. We might be able to do that via ceilometer per our earlier discussion 13:51:01 we can incorporate the incoming messages via receivers, just a new sub-type of it 13:51:18 Yes, I've started working on a Zaqar client but could use some guidance. 13:51:27 notification (outgoing) side, I'd like that to be a deployer configured option 13:51:40 it can be a list of drivers 13:51:53 cschulz: Qiming u guys working on a zaqar client for what? 13:51:57 * flaper87 sneaks in 13:52:05 There's a python client 13:52:15 * flaper87 admits he didn't read the backlog 13:52:19 let the deployers decide if notifications going to oslo or zaqar or both 13:52:36 flaper87 for Senlin incoming and outgoing messages. 13:52:50 flaper87, we are looking to zaqar to trigger autoscaling from user land 13:53:00 ah, sounds great! 13:53:02 and also some other possibilities 13:53:19 Qiming OK. 13:53:26 well, happy to help and provide guidance if needed 13:53:42 Also, #openstack-zaqar is a pretty cool channel 13:53:43 auto-scaling triggered from infrastructure layer is a toy, if not a mistake 13:53:44 :D 13:53:52 flaper87, Thanks. I'll contact you. 13:53:57 flaper87, cookies there? 13:54:12 :) 13:54:26 Qiming always feel hungry 13:54:41 flaper87, will reference your blog post in a coming talk, :) 13:54:42 flaper87, what time zone you in? 13:54:46 Qiming: yup but we're fans of gummybears 13:54:59 anyway, that's pretty much what we have on agenda 13:55:03 Qiming: a-ha! :D Which one? Let me know if you need help 13:55:05 #topic open discussion 13:55:11 cschulz: oh, that's a very good question :P 13:55:16 I don't even know 13:55:19 flaper87, the ptl one 13:55:34 It's easier to ping me than keeping track of my TZs whereabouts 13:56:16 flaper87 OK 13:56:46 that said, I'm supposed to be in Central Europe TZ 13:56:55 (should be summer time now so CEST) 13:56:57 fei long will continue his ptl job next cycle? 13:57:05 which I'm not right onw 13:57:08 Qiming: yes 13:57:18 Any particular IRC channel to use? 13:57:25 #openstack-zaqar 13:57:35 OK thanks 13:59:12 okay, guys, thanks for joining, talk to you next week 13:59:21 Bye 13:59:24 #endmeeting