16:01:31 <kozhukalov> #startmeeting Fuel
16:01:33 <openstack> Meeting started Thu Jan 22 16:01:31 2015 UTC and is due to finish in 60 minutes.  The chair is kozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:37 <openstack> The meeting name has been set to 'fuel'
16:01:41 <kozhukalov> #chair kozhukalov
16:01:42 <openstack> Current chairs: kozhukalov
16:01:46 <Guest12691> Hi
16:01:50 <kozhukalov> hi everyone
16:01:51 <mkwiek> hello
16:01:52 <ikalnitsky> o/
16:01:53 <dshulyak> hi
16:01:56 <evgeniyl___> hi
16:01:57 <seeg> \o
16:01:57 <kozhukalov> agenda as usual
16:02:00 <IvanKliuk> hi
16:02:03 <agordeev> hi
16:02:06 <kozhukalov> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:02:07 <prmtl> o/
16:02:14 <akasatkin> hi
16:02:26 <kozhukalov> #topic granular deployment (dshulyak)
16:02:30 <Guest12691> Guest12691 is docaedo, nick issues this morning :(
16:02:53 <dshulyak> as you know we merged mvp that is required for library modularization,
16:02:53 <dshulyak> and that process is already started, i know that Alex D is able to provide more
16:02:53 <dshulyak> details
16:03:08 <kozhukalov> Guest12691: ok, got that
16:03:19 <dshulyak> next step is to provide api for partial deployment, all patches for this
16:03:19 <dshulyak> stuff is on review already, and we are currently testing it
16:03:40 <dshulyak> also we have all necessery stuff to easily plug tasks for pre/post stages, this
16:03:40 <dshulyak> patches also on review, and ofcourse there is ongoing process of moving pre/post tasks from astute to library
16:04:23 <dshulyak> well, and warpc is working on plugable reboot task, but this is another story
16:04:34 <dshulyak> any questions guys?
16:04:49 <kozhukalov> dshulyak: great to read that
16:05:26 <dshulyak> it looks i started to early)
16:05:42 <kozhukalov> dshulyak:  you started in time
16:06:00 <seeg> do we really need tasks like apt-get update, yum update, yum clean, etc? can't one just say 'puppet install latest package of this' and update will be made automatically?
16:06:00 <kozhukalov> everyone is reading what you just write
16:06:04 <mihgen> hi guys, I'm a bit late
16:06:17 <barthalion> dshulyak: we're late, not the other way
16:06:34 <mihgen> did we start meeting or not?
16:06:45 <kozhukalov> mihgen: we did
16:07:06 <kozhukalov> mihgen: meeting is going on
16:07:30 <kozhukalov> topic is granular deployment
16:07:47 <kozhukalov> ok, looks like we in time with this feature by 6.1
16:07:49 <angdraug> so what about seeg's question?
16:07:51 <dshulyak> seeg: what if package will be required in pre deployment?
16:08:11 <dshulyak> so basically your sugestion is to do same stuff implicitly by puppet
16:08:47 <dshulyak> also we may want to know is repo valid right after  we uploaded it
16:08:48 <kozhukalov> dshulyak: are we planning to have network configuration as a separate stage?
16:09:08 <dshulyak> kozhukalov: we already have it as separate task
16:09:34 <dshulyak> kozhukalov: https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/tasks.yaml#L1
16:09:35 <kozhukalov> and even more, i'm interested in this feature in the context of IBP
16:09:58 <kozhukalov> currently we use cloud-init as our initial configuration tool
16:10:19 <kozhukalov> can we somehow use puppet instead of cloud-init
16:10:44 <kozhukalov> and is it possible to create a separate granular stage for that
16:11:08 <kozhukalov> so that it is not connected to any task which we currently track?
16:11:17 <evgeniyl___> kozhukalov: it looks like another task in pre_deployment stage
16:12:02 <dshulyak> kozhukalov: evgeniyl___: i would say that we can refactor provisioning to use task api, or even merge provisioning into pre_deployment stage
16:12:09 <kozhukalov> evgeniyl___: yes but i wonder if it is possible to deal with that pre-deployment stage in terms of granular deployment
16:12:52 <kozhukalov> for example we put astute.yaml into /etc/astute.yaml during provisioning and before first reboot
16:13:24 <kozhukalov> dshulyak: ok, let's move this discussion to ML
16:13:38 <kozhukalov> that would be great if it was possible
16:13:42 <kozhukalov> ok moving on
16:13:52 <dshulyak> i dont see any problems, we need to decide how to do it properly and this is it
16:14:00 <kozhukalov> #topic 200 nodes and image-based provisioning (vkozhukalov)
16:14:16 <kozhukalov> ok, just to make sure everyone is aware
16:14:44 <kozhukalov> it is an official decision to make IBP production ready by 6.1
16:15:03 <kozhukalov> and it is to be a default provisioning option
16:15:18 <xarses> \o/
16:15:19 <xarses> yay
16:15:29 <kozhukalov> we have tried it on 100 nodes scale lab and it works pretty well
16:15:46 <kozhukalov> it looks like 200 nodes is not a problem
16:16:11 <kozhukalov> besides IBP gonna solve some important issues
16:16:33 <kozhukalov> so it sounds rational to have it as our default provision method
16:16:55 <kozhukalov> is anyone from scale team around?
16:17:07 <dshulyak> kozhukalov: how long does it take to provision 200 nodes on scale lab?
16:17:38 <kozhukalov> provisioning itself take about 2 minutes on 100 nodes lab
16:17:58 <kozhukalov> + reboot takes around 3-4 minues
16:18:14 <angdraug> compared to 20 minutes for traditional provisioning?
16:18:41 <kozhukalov> we also have some suboptimal code when we reboot nodes
16:19:03 <kozhukalov> we reboot them one by one, one per second
16:19:20 <kozhukalov> so when it is just 5 nodes, it is not a big deal
16:19:41 <kozhukalov> but when you have 200 nodes it gonna take 200 seconds
16:20:02 <kozhukalov> so it is just the place where we can improve our system
16:20:18 <angdraug> what about not rebooting them at all?
16:20:32 <kozhukalov> angdraug: actually traditional provisioning works pretty well too
16:20:59 <kozhukalov> it takes around 11 minutes (including 2 reboots)
16:21:18 <angdraug> so overall it's 6 minutes vs 11?
16:21:29 <kozhukalov> and it looks like 200 nodes for traditional provisioning is also not a problem
16:21:45 <kozhukalov> angdraug: yes, exactly
16:22:17 <agordeev> kozhukalov: looks like you are going to cover IBP status. I have something to add. or should i do it separetely?
16:22:18 <mattymo> we increased a lot of timeouts in traditional installers back in 6.0, so that it didn't flip out so much. But with suboptimal networking, it could hang for some time in the beginning
16:22:39 <kozhukalov> angdraug: about not rebooting, it really sounds interesting, it was xarses's idea, but i have not tried it yet
16:23:24 <xarses> pivot root would be interesting
16:23:45 <kozhukalov> agordeev: ok, i just would like all fuelers be aware that IBP in our scope for 6.1 and a default one
16:23:53 <xarses> i thought we had cobbler only booting nodes in chunks of 20 so that it's not overloaded
16:24:13 <xarses> that was something that the scale lab came up with to get to 100 nodes
16:24:41 <kozhukalov> xarses: according to what i've seen in logs it is not true
16:24:49 <kozhukalov> nodes are rebooted one by one
16:25:00 <kozhukalov> ok, moving on
16:25:10 <xarses> hmm ok
16:25:25 <kozhukalov> let me change the order of our topics
16:25:47 <kozhukalov> #topic image based provisioning (agordeev)
16:25:49 <mattymo> xarses, 1 at a time because of timeouts and failures to download preseed file. We hacked debian-installer to retry 10 times insteadof just 1
16:25:56 <mattymo> sorry for interrupting
16:25:56 <kozhukalov> agordeev: please go ahead
16:26:14 <agordeev> 2 new blueprint were approved and targeted to 6.1 as high
16:26:16 <agordeev> https://blueprints.launchpad.net/fuel/+spec/build-ubuntu-images-on-masternode
16:26:18 <agordeev> https://blueprints.launchpad.net/fuel/+spec/fuel-agent-improved
16:26:20 <agordeev> the next step is to prerare the specs for it
16:26:22 <agordeev> good news: IBP had been switched to automatic build tests. So new bugs will possibly appear more frequent.
16:26:24 <agordeev> regarding bugs: still few on review, most of high were merged to master and waiting to be backported to 6.0.1
16:26:39 <kozhukalov> agordeev: we are looking forward for specs
16:27:25 <kozhukalov> and next week we are planning to give a talk about IBP mostly for scale team
16:27:48 <kozhukalov> agordeev: are you done?
16:28:03 <agordeev> kozhukalov: yes, i'm done, have nothing to add
16:28:24 <mihgen> folks did we discuss that we want ibp be default and run against 200 nodes
16:28:35 <kozhukalov> ok, if there are no any other q, let's move on
16:28:35 <angdraug> I don't like when we put words like "improve" in blueprint names
16:28:40 <kozhukalov> mihgen: yes
16:28:47 <mihgen> good
16:29:02 <agordeev> angdraug: what is the better word for that?
16:29:13 <angdraug> be more specific
16:29:17 <kozhukalov> angdraug: it will be highly defined in the spec what it means "improve"
16:29:41 <angdraug> if you don't yet know what you're going to improve, why create BP at all?
16:30:00 <angdraug> if it's just bugfixing, shouldn't be a BP
16:30:20 <kozhukalov> angdraug: the word improve is used because we have 2 particular improvements
16:30:33 <angdraug> should be 2 separate BP's
16:30:48 <angdraug> we've had that with pacemaker-improvement BP before
16:30:58 <kozhukalov> angdraug: we want it to be able to reconnect when disconnected and we want it to compare checksums of images
16:31:21 <angdraug> so, ibp-reconnect and ibp-image-checksums
16:31:26 <xarses> angdraug: +1
16:31:39 <xarses> openended BP's are never finished
16:31:47 <kozhukalov> maybe we'll split, but for me it looks ok, because planned improvements are pretty small
16:32:16 <angdraug> please split, you already plan to have separate specs for those two, no?
16:32:16 <kozhukalov> ok, let's make three then
16:32:22 <kozhukalov> agordeev: will you?
16:32:40 <agordeev> yes, i'll do separete BP's. thanks for suggestions
16:32:54 <angdraug> thanks
16:33:02 <kozhukalov> #action agordeev splits IBP improvement BP into two separate BPs
16:33:09 <kozhukalov> moving on
16:33:21 <kozhukalov> #topic Empty role (evgeniyl)
16:33:36 <evgeniyl___> We’ve tested and merged the feature, also we’ve fixed a bug with progress bar which was related to Granular deployment.
16:33:44 <evgeniyl___> QA team started to work on tests for this role.
16:33:59 <evgeniyl___> Also Meg is going to update the docs, I provided all required information.
16:34:20 <xarses> it's base-os role, not empty role
16:34:58 <evgeniyl___> Now, user can assign Operating System role to node, and no additional configuration will be performed
16:35:23 <evgeniyl___> xarses: yes, we had different name for this role
16:35:43 <evgeniyl___> That is all.
16:35:49 <evgeniyl___> Any questions?
16:36:56 <kozhukalov> looks like, no one has
16:37:07 <kozhukalov> evgeniyl___: thanx
16:37:28 <kozhukalov> great to read that we have progress here
16:37:33 <kozhukalov> moving on
16:37:47 <kozhukalov> #topic downloadable ubuntu release (ikalnitsky)
16:37:58 <ikalnitsky> We had a hot discussion about implementation details this week.
16:37:59 <ikalnitsky> You can see them in the spec:
16:37:59 <ikalnitsky> #link https://review.openstack.org/#/c/147838/1/
16:37:59 <ikalnitsky> Well, we all agreed to use Nailgun for uploading iso and creating tasks, while Astute would be used for repo extraction.
16:37:59 <ikalnitsky> Also, we need to research Nginx capabilities for uploading files. It looks like we can do not block Nailgun worker until file uploaded.
16:38:00 <ikalnitsky> The most important questions about design are resolved, so I think we can start implementation since tomorrow.
16:38:00 <ikalnitsky> Questions?
16:38:12 <kozhukalov> you are really good at typing )
16:38:19 <ikalnitsky> yes, i'm ))
16:38:44 <xarses> tanks for pre-typing ikalnitsky
16:38:50 <barthalion> there is no support for uploading in upstream nginx iirc
16:38:55 <xarses> it helps move the meeting along
16:39:10 <barthalion> so we will probably end up with external module, unless there is some black magician around
16:39:19 <ikalnitsky> barthalion: what is nginx iirc?
16:39:25 <xarses> barthalion: my understanding is that ngnix has problems with large files, and we will need to switch to a fork
16:39:34 <xarses> rmoe knows about it well
16:39:34 <angdraug> ikalnitsky: iirc = if i remember correctly
16:39:36 <barthalion> ikalnitsky: iirc == if I remember correctly
16:39:47 <ikalnitsky> thank you guys)
16:40:16 <xarses> ikalnitsky: talk with rmoe regarding ngnix uploads
16:40:36 <ikalnitsky> xarses: ok
16:41:15 <kozhukalov> ok, guys how do you think is it ok, when user waits for 1 minute to upload an iso , but then it is not the end, it is just means additional task is started.
16:41:18 <xarses> #link https://dmsimard.com/2014/06/21/a-use-case-of-tengine-a-drop-in-replacement-and-fork-of-nginx/
16:41:21 <xarses> ikalnitsky: ^
16:41:37 <ikalnitsky> xarses: thanks
16:41:46 <xarses> kozhukalov: we should have feedback that its working
16:41:47 <ikalnitsky> kozhukalov: good question, actually
16:41:48 <kozhukalov> we had a discussion about UX in spec but maybe there are other opinions
16:42:12 <ikalnitsky> should fuel cli wait until uploading task and repo extraction get ready?
16:42:25 <xarses> maybe use notification area in UI
16:42:39 <xarses> ikalnitsky: yes, and no
16:42:58 <ikalnitsky> xarses: notification is good, but it's about UI. and we're interested in fuelclie
16:43:16 <evgeniyl___> ikalnitsky: if request uploads something, there is no way not to block request
16:43:21 <xarses> correct, they should get task id back
16:43:35 <xarses> and if they want to wait, they should attach to monitor the task
16:43:42 <xarses> so you get both
16:43:42 <barthalion> "With nginx, the upload took 1 minute 13 seconds. With Tengine, the upload took 41 seconds." sounds like microoptimization
16:44:00 <ikalnitsky> xarses: when we upload iso there's no way to do this way. it will be performed by pure http.
16:44:04 <evgeniyl___> xarses: how are you going to return task id, if you upload the file?
16:44:41 <evgeniyl___> xarses: you perform post request with a huge body, and you should complete uploading to get something from the server
16:45:00 <kozhukalov> please guys have a look at the spec and give your opinion, we have some arguments "for" and "against"
16:45:19 <xarses> correct, i was talking about while astute is running in the background
16:45:54 <kozhukalov> ok, let's move on and leave this discussion for the spec and ML
16:45:55 <xarses> for the upload I'm not sure if it matters if you have progress or not, openstack clients dont
16:46:24 <kozhukalov> #topic Multi-HV support (adanin)
16:46:37 <adanin> We are going to implement two hypervisors in one environment. They are KVM (or QEMU) and VMware vCenter.
16:46:37 <adanin> Both of them will have their own Cinder backend and will be placed into different Availability Zones.
16:46:39 <adanin> It'll be done to avoid attaching volumes with unsupported type to a hypervisor.
16:46:40 <adanin> A new role "cinder-vmdk" will be added. New vCenter-specific OSTF tests will be added. A new UI tab with vCenter settings will be added.
16:46:42 <adanin> We still not decided which network backend will be used: nova-network or Neutron with non-upstream ML2 DVS plugin.
16:46:43 <adanin> Links to blueprints:
16:46:45 <adanin> #link https://blueprints.launchpad.net/fuel/+spec/vmware-ui-settings
16:46:45 <adanin> #link https://blueprints.launchpad.net/fuel/+spec/cinder-vmdk-role
16:46:46 <adanin> #link https://blueprints.launchpad.net/fuel/+spec/vmware-dual-hypervisor
16:47:02 <adanin> Questions?
16:47:10 <angdraug> why new cinder-vmdk role?
16:47:28 <angdraug> can it need a dedicated node?
16:47:59 <xarses> adanin: cant we run multiple hypervisors if we just move the compute tasks around and get multi-hv for all types?
16:48:09 <xarses> also, cinder-vmdk role?
16:48:17 <adanin> To allow a user place cinder-volume service to a node he wants.
16:48:20 <xarses> run it on the controllers like ceph
16:48:33 <xarses> that would be a task descision, not role
16:49:00 <xarses> I already had to unbreak a cinder-vmdk deployment because it's not automatic
16:49:19 <xarses> #link https://bugs.launchpad.net/fuel/+bug/1410517
16:49:45 <adanin> xarses: cinder-volume in this case is just a proxy. I don’t think it’s a good idea to put it on Controller node. It will increase IO for the node.
16:49:56 <angdraug> what about multiple cinder backends?
16:49:58 <angdraug> #link https://blueprints.launchpad.net/fuel/+spec/fuel-cinder-multi-backend
16:50:20 <angdraug> adanin: is cinder-volume in the data path?
16:50:24 <adanin> with new role a user will be able to place this service to whatever node he wants.
16:50:31 <xarses> again, cinder-vmkd volume service should be a task, not a role
16:50:48 <adanin> multi-backend is not applicable in that case.
16:51:35 <adanin> xarses: and how a user choose a particular node to run this task?
16:52:01 <angdraug> choose a backend
16:52:31 <xarses> granular deployments is getting us there
16:53:11 <angdraug> once again, why does it have to be a particular node, is it in the data path?
16:53:17 <angdraug> and what about HA for that service?
16:53:27 <adanin> angdraug: xarses: a user will have two Cinder backends simultaneously - one for KVM and another for vCenter. There is no way to use VMDK for KVM and vise versa.
16:54:27 <xarses> adanin: correct
16:55:00 <adanin> angdraug: HA - assign cinder-vmdk role to two or more nodes. Each node will have identical settings for cinder-volume service.
16:55:01 <xarses> adanin: please lets review this offline, probably as seperate meeting
16:55:05 <kozhukalov> 5 minutes
16:55:08 <adanin> Like we do it for Ceph now.
16:55:32 <adanin> I’m going to advertice it in ML.
16:55:49 <kozhukalov> adanin: thanx
16:56:30 <kozhukalov> we have nothing more to discuss
16:56:38 <angdraug> open discussion?
16:56:44 <kozhukalov> 3 minutes
16:56:52 <kozhukalov> does it make sense?
16:57:00 <kozhukalov> any announcements?
16:57:26 <kozhukalov> ok, closing then
16:57:32 <kozhukalov> thanx everyone
16:57:37 <kozhukalov> great meeting
16:57:40 <adanin> thank you guys
16:57:51 <kozhukalov> #endmeeting