13:00:07 <Qiming> #startmeeting senlin
13:00:08 <openstack> Meeting started Tue Mar 29 13:00:07 2016 UTC and is due to finish in 60 minutes.  The chair is Qiming. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:11 <openstack> The meeting name has been set to 'senlin'
13:00:35 <Qiming> evening/morning
13:00:43 <cschulz> Hi
13:00:50 <lixinhui_> HI
13:01:20 <elynn> Hi
13:01:28 <zzxwill> Hello Qiming.
13:01:54 <Qiming> just posted an agenda for meeting today
13:01:57 <Qiming> #link https://wiki.openstack.org/wiki/Meetings/SenlinAgenda
13:02:14 <Qiming> please feel free to add items if you have some
13:02:21 <Liuqing> \o
13:02:37 <Qiming> #topic austin summit planning
13:03:24 <Qiming> I think we are in good shape regarding the deep dive session and the autoscaling session
13:03:45 <Qiming> anything you want to discuss regarding these two talks?
13:04:23 <lixinhui_> The demo is still under refinement
13:04:41 <Qiming> we have just got some preliminary session allocation info
13:04:45 <lixinhui_> thanks for the help of elynn today on security group
13:04:51 <elynn> Just begin to think of it
13:05:02 <Qiming> so far we don't have conflicts between the talk sessions and the design summit
13:05:31 <elynn> will discuss with you later if I need help.
13:05:46 <Qiming> elynn, consider backport that to stable/mitaka?
13:06:03 <elynn> ok, keep in mind
13:06:19 <elynn> Is it merged?
13:06:24 <Qiming> not yet
13:07:01 <Qiming> speaking of container cluster one, let's see what we can get as a poc in this week
13:07:28 <Qiming> haiwei, we will need to sync on that more frequently
13:07:57 <haiwei> Qiming, I started to implement it in senlin, tried to get container list
13:08:07 <haiwei> it is almost there
13:08:17 <Qiming> okay
13:08:36 <Qiming> #link design summit sessions
13:08:44 <Qiming> sorry
13:08:54 <Qiming> #link https://etherpad.openstack.org/p/newton-senlin-sessions
13:09:24 <Qiming> we have a S7 added though we only have 5 working session + 1 fishbowl session applied
13:10:06 <Qiming> adopting existing nova server sounds an interesting scenario
13:10:33 <Qiming> but you know, senlin needs a profile for any node created (or adopted)
13:11:11 <haiwei> yes
13:11:28 <Qiming> even if we want to adopt an existing server, we may still have to do two steps:
13:12:07 <cschulz> Should we add the 'semi-autoscaling' stuff to this?
13:12:11 <Qiming> openstack cluster profile create --from <server_id> --type os.nova.server
13:13:14 <Qiming> openstack node create --spec <spec_file> --from <server_id>
13:13:37 <Qiming> cschulz, you mean design summit session?
13:13:52 <cschulz> Yes, or is it too early?
13:14:18 <Qiming> cschulz, no, we had a plan for design summit sessions
13:14:31 <Qiming> we are yet to finalize it
13:14:55 <Qiming> it sounds to me that S3 is not so interesting now
13:15:34 <Qiming> we have already started tempest and rally investigation, tempest framework is getting merged, not a lot to discuss
13:15:49 <lixinhui_> since we already start working on that
13:15:56 <Qiming> Yanyan has started contributing to rally about support to Senlin
13:16:15 <Qiming> so how about we replace S3 with S7?
13:16:32 <lixinhui_> I agree
13:16:35 <elynn> no problem
13:16:40 <haiwei> the session number is limited?
13:16:52 <haiwei> it's ok I think
13:17:11 <Qiming> yes, haiwei, we only have 5 + 1 + 1
13:17:25 <haiwei> ok
13:17:32 <Qiming> 5 work sessions, + 1 fishbowl, +1 developer meetup
13:18:02 <Qiming> then we can make S3 cover both your item and cschulz's proposal
13:18:34 <haiwei> sounds good
13:18:37 <Qiming> haiwei, I'm still leaving you as the driver
13:18:54 <haiwei> ok
13:21:08 <Qiming> let's move on
13:21:23 <Qiming> #topic high-priority bugs before final release
13:21:45 <Qiming> the security group one is a good candidate
13:22:07 <Qiming> are you aware of any high priority bugs we should merge into stable branch?
13:22:24 <Qiming> it is being pushed out of door next week
13:23:16 <Qiming> btw, Liuqing, just proposed a release of senlin-dashboard 0.2.0 https://review.openstack.org/#/c/298710/
13:24:18 <Qiming> emm... no one is yelling, great, let's continue work on newton items
13:24:29 <Qiming> #topic newton work items
13:24:42 <Qiming> #link https://etherpad.openstack.org/p/senlin-newton-workitems
13:25:27 <Qiming> testing side, no news from china mobile yet, but they have some guys evaluating multi-region deployment, that I know
13:25:46 <lixinhui_> Qiming
13:25:59 <lixinhui_> I think it would be better if we can schedule
13:26:05 <lixinhui_> some meeting with them here
13:26:09 <Qiming> yes?
13:26:12 <lixinhui_> about oslo and
13:26:20 <lixinhui_> scalability things
13:26:26 <Qiming> okay, will do
13:27:01 <lixinhui_> remember they once showed some parameters there
13:27:03 <Qiming> #action Qiming to sync with china mobile guys on oslo.messaging and scalability lessons learnt
13:27:06 <lixinhui_> when mid-cycle
13:27:24 <Qiming> will schedule a call with them soon
13:27:33 <lixinhui_> great!
13:28:02 <Qiming> tempest testing, where are we, elynn?
13:28:24 <Qiming> I saw some patches coming in
13:28:41 <elynn> yes
13:28:52 <Qiming> anything you want to discuss, or it is all under your good control?
13:29:05 <elynn> An initial patch is submitted and I will rebase it based on your comments.
13:29:20 <elynn> It's good for now.
13:29:41 <Qiming> okay, please consider adding support to some failure cases, such as node creation failures, driver failures
13:29:48 <elynn> Just need some time to move all tests to use tempest plugin.
13:29:51 <zzxwill> Sorry, 'china mobile'? To deploy clusters in their cloud by senlin?
13:29:54 <elynn> It's trivial.
13:30:54 <Qiming> great. what we need is not just tests against successes, but also those against failures
13:31:28 <Qiming> as a user, I really hope the service never fails, and if it fails, it fails in a predictable way
13:32:09 <Qiming> zzxwill, that is true
13:33:12 <Qiming> lixinhui_, do we have some data about stress testing senlin?
13:33:22 <elynn> Okay, so let me think of it. Haven't figure out how to implement them in tempest, I will keep on digging.
13:33:39 <lixinhui_> Yes, Qiming
13:33:45 <lixinhui_> with simulated drvier
13:34:05 <zzxwill> Thx.
13:34:20 <lixinhui_> one engine runing with 8 cpu and 16mem can support 1000 scale node creation without any problem
13:34:44 <lixinhui_> With real driver, we meet some image authentication failure
13:34:45 <Qiming> does multi-engine setup help in scaling?
13:35:04 <Qiming> so it is broken, but not at senlin side
13:35:25 <lixinhui_> not in senlin side
13:35:46 <Qiming> okay, hope we can learn something from china mobile guys
13:35:52 <lixinhui_> with two engine, 2500 creation has no problem
13:35:58 <lixinhui_> but will do more tests
13:36:07 <Qiming> cool.
13:36:12 <lixinhui_> to find your question
13:36:30 <lixinhui_> find answer/evidence to your question
13:36:37 <Qiming> okay
13:37:03 <lixinhui_> Yes, it would nice if we can have more exchange with china mobile guys
13:37:06 <Qiming> health mgmt side, do we have lb based health poller?
13:37:12 <lixinhui_> yes
13:37:17 <lixinhui_> I implemented one
13:37:32 <lixinhui_> and adopt it into the auto-scaling demo
13:37:44 <Qiming> okay, would be curious to see how it looks like, :)
13:38:11 <lixinhui_> will discuss more details with you tomorrow
13:38:19 <Qiming> okay
13:38:24 <lixinhui_> nowdays, I polling neutron for node status
13:38:29 <lixinhui_> member status
13:39:02 <lixinhui_> the status is changed by health monitor of lbaas
13:39:09 <Qiming> regarding HA scenario, I have pasted some references to the etherpad
13:39:11 <Qiming> #link https://etherpad.openstack.org/p/senlin-ha-recover
13:39:33 <lixinhui_> yes
13:39:45 <lixinhui_> I read them and treasure them very much
13:40:14 <Qiming> it is very complicated and a huge problem domain
13:40:35 <lixinhui_> I think we should try these works in permitted scope
13:40:43 <Qiming> need to collect a lot inputs to form a global picture, then find something we want to focus on
13:41:00 <Qiming> exactly
13:41:00 <lixinhui_> yes
13:41:17 <Qiming> there are interests in auto-evacuation
13:41:29 <Qiming> but ... that is only one of the scenarios
13:41:45 <cschulz> I think this is another area where we could use Zaqar to get information into Senlin
13:41:58 <Qiming> we need to find out how to amend our health policy to get fencing done
13:42:18 <lixinhui_> Okay
13:42:20 <cschulz> A cluster in trouble could send a notification in via a Zaqar queue
13:42:33 <Qiming> yes, cschulz, we will need many ears and eyes
13:43:02 <Qiming> right, we have some related todo items documented in source tree
13:43:26 <Qiming> we need to yield notifications when something note worthy happens
13:43:50 <Qiming> we also need to generate some meters for ceilometer if there are such use cases
13:44:30 <Qiming> lixinhui_, regarding the full end-to-end story documentation
13:44:34 <cschulz> I would think that a cluster size meter would be very useful
13:44:53 <Qiming> I'm thinking of sharing your autoscaling template into heat-templates project some day
13:45:10 <cschulz> i.e. raise an alarm when a cluster grows beyond some alarm specified limit.
13:45:29 <lixinhui_> YeYes, Qiming
13:45:37 <Qiming> we send out an event after cluster size change
13:45:57 <Qiming> leave the evaluation and alarming logic to telemetry/metering/alarming service, :)
13:46:43 <lixinhui_> what do you mean by generate some meters for ceilometer if there are such use cases
13:47:09 <lixinhui_> i think we will leverage heat for alarm creation
13:47:14 <Qiming> service like monasca/ceilometer would collect runtime statistics from every service
13:47:42 <lixinhui_> okay
13:47:56 <Qiming> it is up to senlin to generate whatever such measurements
13:48:04 <lixinhui_> I see
13:48:17 <lixinhui_> thanks for explanation
13:48:33 <Qiming> api-microversioning
13:48:38 <Qiming> it is almost done
13:49:14 <Qiming> still poking around and see if there things missed, will mark the bp completed if nothing else left
13:49:49 * Qiming reading semi-autoscaling notes
13:50:36 <Qiming> so, if we are leveraging zaqar for messaging/notifications
13:50:40 <cschulz> I'm not sure about the notification part.  We might be able to do that via ceilometer per our earlier discussion
13:51:01 <Qiming> we can incorporate the incoming messages via receivers, just a new sub-type of it
13:51:18 <cschulz> Yes, I've started working on a Zaqar client but could use some guidance.
13:51:27 <Qiming> notification (outgoing) side, I'd like that to be a deployer configured option
13:51:40 <Qiming> it can be a list of drivers
13:51:53 <flaper87> cschulz: Qiming u guys working on a zaqar client for what?
13:51:57 * flaper87 sneaks in
13:52:05 <flaper87> There's a python client
13:52:15 * flaper87 admits he didn't read the backlog
13:52:19 <Qiming> let the deployers decide if notifications going to oslo or zaqar or both
13:52:36 <cschulz> flaper87 for Senlin incoming and outgoing messages.
13:52:50 <Qiming> flaper87, we are looking to zaqar to trigger autoscaling from user land
13:53:00 <flaper87> ah, sounds great!
13:53:02 <Qiming> and also some other possibilities
13:53:19 <cschulz> Qiming OK.
13:53:26 <flaper87> well, happy to help and provide guidance if needed
13:53:42 <flaper87> Also, #openstack-zaqar is a pretty cool channel
13:53:43 <Qiming> auto-scaling triggered from infrastructure layer is a toy, if not a mistake
13:53:44 <flaper87> :D
13:53:52 <cschulz> flaper87,  Thanks.  I'll contact you.
13:53:57 <Qiming> flaper87, cookies there?
13:54:12 <lixinhui_> :)
13:54:26 <lixinhui_> Qiming always feel hungry
13:54:41 <Qiming> flaper87, will reference your blog post in a coming talk, :)
13:54:42 <cschulz> flaper87, what time zone you in?
13:54:46 <flaper87> Qiming: yup but we're fans of gummybears
13:54:59 <Qiming> anyway, that's pretty much what we have on agenda
13:55:03 <flaper87> Qiming: a-ha! :D Which one? Let me know if you need help
13:55:05 <Qiming> #topic open discussion
13:55:11 <flaper87> cschulz: oh, that's a very good question :P
13:55:16 <flaper87> I don't even know
13:55:19 <Qiming> flaper87, the ptl one
13:55:34 <flaper87> It's easier to ping me than keeping track of my TZs whereabouts
13:56:16 <cschulz> flaper87 OK
13:56:46 <flaper87> that said, I'm supposed to be in Central Europe TZ
13:56:55 <flaper87> (should be summer time now so CEST)
13:56:57 <Qiming> fei long will continue his ptl job next cycle?
13:57:05 <flaper87> which I'm not right onw
13:57:08 <flaper87> Qiming: yes
13:57:18 <cschulz> Any particular IRC channel to use?
13:57:25 <flaper87> #openstack-zaqar
13:57:35 <cschulz> OK thanks
13:59:12 <Qiming> okay, guys, thanks for joining, talk to you next week
13:59:21 <cschulz> Bye
13:59:24 <Qiming> #endmeeting