16:01:13 <kozhukalov> #startmeeting Fuel
16:01:13 <openstack> Meeting started Thu Mar  5 16:01:13 2015 UTC and is due to finish in 60 minutes.  The chair is kozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:17 <openstack> The meeting name has been set to 'fuel'
16:01:20 <kozhukalov> #chair kozhukalov
16:01:20 <openstack> Current chairs: kozhukalov
16:01:24 <akislitsky> hi
16:01:29 <mkwiek> hi
16:01:36 <xarses> o/
16:01:40 <salmon_> hi
16:01:44 <kozhukalov> We've just started
16:01:58 <kozhukalov> Today we have just open discussion
16:02:38 <bookwar> actually there is a lot going on with our merges, but i guess everyone has been just to tired to be ready for status report
16:02:48 <kozhukalov> my suggestion is to discuss what we've managed to merge
16:02:55 <sbog> hi all
16:03:00 <salmon_> and what we didnt :)
16:03:25 <xarses> kozhukalov: sounds good
16:03:49 <agordeev> hi
16:04:23 <kozhukalov> ok, i have some info about ubuntu external repos
16:04:32 <kozhukalov> 14.04 is merged
16:05:12 <kozhukalov> so, we are rebasing our feature and try to rub bvt
16:05:59 <kozhukalov> ikalnitsky: around?
16:06:06 <ikalnitsky> yep
16:06:46 <kozhukalov> afaik you've just started  another attempt to build iso
16:06:57 <ikalnitsky> yes, i did
16:07:00 <ikalnitsky> and it's failed
16:07:14 <ikalnitsky> is there any devops?
16:07:20 <ikalnitsky> bookwa, around?
16:07:26 <ikalnitsky> bookwar ^^
16:07:33 <teran_> ikalnitsky: yup
16:07:52 <kozhukalov> rebase related issues?
16:07:53 <ikalnitsky> teran_: do you know something about failed custom iso?
16:08:44 <teran_> ikalnitsky: 6.1 ?
16:08:45 <ikalnitsky> as far as i see, jenkins jon failed to fetch debian installer
16:09:01 <ikalnitsky> yes, custom 6.1 iso
16:09:09 <ikalnitsky> build no 525
16:09:17 <kozhukalov> some detailed info about external ubuntu repos feature is available here
16:09:21 <kozhukalov> #link https://etherpad.openstack.org/p/consume-external-ubuntu
16:09:38 <ikalnitsky> it looks like it's the same issue tht we have with trusty yesterday
16:09:57 <bookwar> ikalnitsky: there were issues with custom builds, but we fixed them
16:10:23 <bookwar> ikalnitsky: how about we move discussion of you particular case into to #fuel-dev?
16:10:40 <ikalnitsky> ok
16:10:52 <bookwar> general CI status: custom iso should work, fuel ci should work
16:11:16 <xarses> kozhukalov: have we been able to do anything about mirror building or mirror validation yet?
16:11:53 <kozhukalov> ikalnitsky: if you build iso with this patch https://review.openstack.org/#/c/161640/3 it is not going to work because it needs one extra patch set here https://review.openstack.org/#/c/161208/4
16:12:45 <ikalnitsky> kozhukalov: ok, i'll keep in mind
16:12:47 <kozhukalov> xarses: what exactly do you mean? split mirrors?
16:12:57 <ikalnitsky> though i've tried to build without them
16:13:05 <xarses> for external ubuntu repos
16:13:29 <kozhukalov> xarses: if so, it is better to ask vitaly parakhin
16:13:52 <kozhukalov> xarses: he is working on building separate ubuntu repos
16:14:55 <kozhukalov> xarses: our work is mostly to make fuel understand several repos with pinning
16:15:38 <angdraug> looks like vitaly is not here
16:15:39 <kozhukalov> xarses: but afaik Vitaly promised to finish his patch today
16:15:58 <brain461> I'm here
16:15:59 <kozhukalov> ikalnitsky: do you know anything about this?
16:16:17 <brain461> and patch is ready for review, as well as prepared Ubuntu mirror for 6.1
16:16:26 <xarses> yes, but I mean, we need to be able to on the fuel node build local mirror of the packages needed by the install
16:16:28 <brain461> #link https://review.openstack.org/#/c/161222/
16:16:38 <kozhukalov> brain461: GREAT
16:16:44 <brain461> we ship MOS packages via iso
16:17:37 <brain461> regarding the caching solution for master node - PL team is investigating it, afair
16:17:55 <xarses> yes, but what happens when i want to / need to / build a local mirror of the operating system packages?
16:17:56 <kozhukalov> xarses: afaik we are not going to build mirror on the fuel node
16:18:12 <xarses> kozhukalov: we MUST have a solution for this
16:18:20 <xarses> otherwise we are back to fuel 3.0.1
16:18:37 <kozhukalov> xarses: user is going to set several repos via UI and then we will use these repos/mirrors
16:19:02 <brain461> we want to preprare script that will create a local copy for upstream mirror. full upstream or only pkgs required by fuel - it's still to be considered
16:19:22 <xarses> yes, and if they provide their own mirror, we must be able to validate that it has all the packages
16:19:40 <xarses> we can't just start and expect all the packages to be around
16:20:24 <brain461> yes, we should validate downloaded packages in some way, at least by checksum
16:22:14 <kozhukalov> xarses: the flow is supposed to be like this: we test our deployment with official ubuntu mirror, and if it works, then for a user it is just recommended to use full mirror (internet/local)
16:22:46 <angdraug> kozhukalov: should be the other way around
16:23:06 <angdraug> we include the repo download and validation script in the test scenario
16:23:27 <kozhukalov> xarses: we have a chance to check if our deployment works with updated mirror around one week before it become official
16:23:58 <kozhukalov> canonical make kind of update announcements
16:24:36 <xarses> kozhukalov: packages change and disappear from external repos all the time. mirrors could be un-available. Hey we even have it listed that we won't use the mirror list
16:24:46 <xarses> instead we pick a single site
16:24:54 <kozhukalov> if user defines a mirror which is not compatible with our deployment it is all up to a user
16:25:10 <xarses> no
16:25:16 <xarses> we have to give them the tools to check it
16:25:34 <xarses> we know what we need
16:25:39 <xarses> not the user
16:25:51 <xarses> we have these tools already
16:25:57 <kozhukalov> how is this tool going to look like?
16:25:58 <xarses> in fuel-main
16:26:02 <xarses> crap for now
16:26:09 <xarses> but they must exist
16:26:24 <xarses> and must be documented
16:26:45 <kozhukalov> xarses: should it be a button on ui?
16:27:04 <xarses> kozhukalov: it should have been. its too late for that now
16:27:05 <kozhukalov> kinda 'validate my repos'
16:27:21 <angdraug> you can at least make it a pre-deployment check
16:27:54 <agordeev> kozhukalov: i think i know about what xarses is talking. Do you remember the fact that fuel-main builds its own repos with all needed packages somehow?
16:28:06 <vkramskikh> we had a plan to implement this button on UI, did the plans change?
16:28:28 <angdraug> can you still do it in time for feature freeze?
16:28:39 <kozhukalov> agordeev: yes, and we going to get rid of this scheme at all
16:29:29 <xarses> kozhukalov: agordeev, yes worse case we can make the fuel-main make mirror scripts work inside the fuel master node
16:29:34 <xarses> at least it's something
16:29:36 <angdraug> kozhukalov: looks like you've missed last week of discussions in the separate-mos-from-linux spec
16:30:12 <angdraug> #link https://review.openstack.org/#/c/148279/33..35/specs/6.1/separate-mos-from-linux.rst
16:30:29 <kozhukalov> angdraug: yes, absolutely
16:30:59 <angdraug> to sum up, we agreed that local repo creation and validation has to remain in scope
16:31:16 <angdraug> and xarses is arguing for reusing the code from fuel-main to save some implementation time
16:32:11 <kozhukalov> i am sorry i did not take part in those discussions, but my opinion is the opposite
16:32:47 <angdraug> opposite to which part?
16:33:02 <angdraug> keeping this part of the feature in scope, or reusing fuel-main code?
16:33:34 <kozhukalov> i am sure we need to stop rebuilding mirrors and continue to call them ubuntu and centos
16:34:36 <kozhukalov> if we build mirror with our own versions of packages we need to start calling it mos without any references
16:34:42 <angdraug> if we cleanly separate our packages from what we take verbatim from ubuntu, why is that still a problem?
16:35:29 <kozhukalov> if we separate, then it is ok, ubuntu is the copy of upstream nothing else
16:35:40 <kozhukalov> exact copy of upstream
16:35:41 <xarses> kozhukalov: my issue is not with shipping an iso with these packages
16:36:01 <xarses> kozhukalov: my issue is that the user MUST have tools to build local mirrors
16:36:14 <xarses> the user must have tools to validate their mirrors
16:36:19 <agordeev> xarses: to build or just to validate ?
16:36:24 <angdraug> both
16:36:32 <kozhukalov> i mean user defines a set of repos: 1) copy of upstream 2) mos packages (not basic linux)
16:36:54 <xarses> agordeev: in a perfect world, both. I could be convinced to live with one of the two
16:36:55 <angdraug> yes, and we have to give them the tools to build and validate (1)
16:37:11 <kozhukalov> we need to validate mos part far before we deliver it to a user
16:37:18 <angdraug> I can't be convinced to live with just one of the two :)
16:37:36 <angdraug> of course
16:37:59 <angdraug> which is why I also argued on that review I linked that we must include mos packages on the iso and not force user to download them from us
16:38:00 <kozhukalov> 1) is EXACT copy of upstream 2) is EXACT copy of mos
16:38:08 <kozhukalov> nothing to be build
16:38:22 <kozhukalov> we need to build them not on master node
16:38:32 <kozhukalov> we need to build them on our ci
16:38:34 <angdraug> 1) is a STRICT SUBSET of upstream
16:38:37 <xarses> kozhukalov: how do you know that 1) will work for fuel /mos
16:38:43 <kozhukalov> and then deliver it as is
16:39:05 <xarses> kozhukalov: how do you know that 1) has all the packages we expect
16:39:15 <kozhukalov> 1) NOT SUBSET
16:39:23 <kozhukalov> 1) MIRROR
16:39:44 <kozhukalov> it is a full mirror
16:39:47 <angdraug> mirror is going to be 30GB worth of packages even if you don't take multiverse
16:39:50 <xarses> you expect some one to have a full mirror of ubuntu lying around?
16:39:54 <xarses> isn't that 30 gb
16:40:04 <kozhukalov> we don't need to put it on the master node
16:40:04 <xarses> =)
16:40:08 <xarses> no we dont
16:40:12 <kozhukalov> there are to possibilites
16:40:13 <angdraug> downloading 3GB iso sucks badly enough
16:40:19 <xarses> IF we moved the make mirror script to the fuel master
16:40:28 <kozhukalov> first is the mirror available via internet
16:40:43 <kozhukalov> second is a customer has it is own mirror
16:40:45 <xarses> it would pull all of the packages we need
16:41:02 <angdraug> mirror on the internet is a poor man's option that should never be offered as the first choice
16:41:08 <xarses> kozhukalov: no, customers only keep partial mirrors
16:41:20 <kozhukalov> if customer's local mirror is broken then it is all up to a customer
16:41:21 <angdraug> fuel is already too biased towards people setting up poc's in virtualbox
16:41:26 <xarses> and the from the internet isn't a good choice, it breaks all the time
16:41:28 <angdraug> lets not make it even worse
16:42:01 <xarses> it sounds like we need to have a voice call on this
16:42:09 <kozhukalov> mirror can not be a subset
16:42:21 <angdraug> as I said, we already had a week's worth of discussion across 3 spec reviews
16:42:24 <kozhukalov> mirror is what you get when you make a copy
16:42:31 <xarses> kozhukalov: can you invite the relevant feature leads to discuss this.
16:42:40 <brain461> why 30gb, btw? we need only one distro (trusty), not the whole ubuntu mirror - it should take ~5gb even with multiverse
16:43:09 <kozhukalov> xarses: yes, voice call is what we actually need )
16:44:01 <angdraug> https://help.ubuntu.com/community/Debmirror#Start_The_Mirror_Build_Process
16:44:09 <angdraug> trusty = 60G
16:44:45 <xarses> sounds like smaller if we dropped 32bit, but ya... way too big
16:44:58 <xarses> no one who maintains their own repo keeps a full mirror
16:45:07 <xarses> they use it as a gate for package control
16:45:19 <kozhukalov> what we need to provide to a customer is the script which make a mirror (exact copy of upstream) if a customer does not want to use internet
16:45:36 <kozhukalov> or maybe caching proxy is also not a bad option
16:45:39 <angdraug> kozhukalov: you're going in circles
16:45:50 <angdraug> we had extensive discussions about exactly what you're talking about
16:46:16 <angdraug> it's counterproductive to reiterate them all here
16:46:18 <xarses> we have already beat caching proxy's to death, they fall through too much
16:46:32 <kozhukalov> ok
16:46:50 <zigo_> If we offer the possibility to download from internet, the possibility to also use an official Ubuntu CD should be added too, IMO.
16:46:52 <angdraug> please have a discussion with feature leads in MSK to catch up, and then lets have a voice call
16:47:34 <kozhukalov> angdraug: sure
16:47:54 <kozhukalov> do, we have anything else to discuss?
16:47:58 <angdraug> yes
16:48:07 <angdraug> I have a question about *-updates release series in LP
16:48:24 <zigo_> kozhukalov: Why would we need anything not in Ubuntu main?
16:48:31 <angdraug> looks like I'm in the same situation as you: I must have missed that discussion
16:48:33 <kozhukalov> zigo_ : it was one of the options, but it is terrible from user experience point of view
16:48:38 <zigo_> Canonical has all of OpenStack in main, AFAIK.
16:48:47 <angdraug> no
16:48:56 <angdraug> we have a bunch of packages not from main
16:49:01 <zigo_> angdraug: Which part isn't?
16:49:08 <salmon_> meetings without agenda sucks ;)
16:49:09 <kozhukalov> zigo_: because we have lots of patches
16:49:15 <zigo_> (please don't mention the python-xstatic... ;)
16:49:18 <angdraug> salmon_: +100500
16:49:44 <angdraug> can we please shelve the ubuntu discussion and move on to my question?
16:49:48 <kozhukalov> salmon_: you are welcome to add topics into agenda
16:50:19 <kozhukalov> angdraug: which question?
16:50:37 <angdraug> the question is why do we need separate *-updates release series for every release?
16:50:56 <angdraug> that adds an order of magnitude of complexity to our bug triage and release management
16:51:21 <angdraug> and I don't see any benefits whatsoever
16:51:50 <kozhukalov> it is better to ask aglarendil
16:51:55 <angdraug> if a patch is not suitable for 6.0-updates, it shouldn't be in 6.0.x release series, either
16:52:00 <angdraug> aglarendil: ^^^
16:52:51 <zigo_> What you wrote makes sense dmitry.
16:52:56 <bookwar> andreaf: the plan is that -updates branch is for updates which can be applied randomply
16:53:01 <bookwar> angdraug: ^^
16:53:04 <bookwar> sorry :)
16:53:17 <angdraug> define randomly
16:53:31 <angdraug> and why shouldn't the same requirement apply to everything in 6.0.x?
16:53:48 <bookwar> stable/6.0 is for updates which are released, like 6.0.1, and you can do update from 6.0 to 6.0.1
16:54:01 <bookwar> 6.0-updates is for individual updates
16:54:35 <angdraug> why should 6.0.1 be more than the sum of individual updates?
16:54:36 <bookwar> update for a package which is delivered as a new package or even via some random bash script
16:54:56 <angdraug> any random bash script can be delivered as a pacakge
16:55:05 <bookwar> individual updates are exceptional and require additional checks
16:56:03 <bookwar> if this bash script does something on your compute nodes you deliver it in a package and then you add 3-pages long instruction how to run it, 6.0-updates branch is for these very custom exceptional cases
16:56:05 <angdraug> my point is, we should do the same checks for everything in 6.0.1, and maybe reduce the number of fixes we backport to only those that can pass those checks
16:56:24 <xarses> angdraug: +1
16:56:54 <xarses> If we bothered to backport it, the asumption is people want/need to consume it
16:56:55 <bookwar> we don't check that fix which comes to 6.0.1 HEAD can be cherry-picked on top of 6.0 without other changes
16:57:20 <xarses> bookwar: we are building packages from these, not patches
16:57:28 <angdraug> at least I hope we do
16:57:38 <xarses> the package should allways be rebuilt from HEAD
16:57:48 <kozhukalov> 3 minutes
16:57:58 <bookwar> so sometimes people need to apply those very special changes independently of our release cycle, as we don't have 6.0.1 release yet, and they don't want to move to 6.0.1 at all, but they want this particular fix
16:58:13 <angdraug> so they should take a package from 6.0.1 package repo
16:58:29 <angdraug> along with all its dependencies
16:58:35 <bookwar> actually i am not defending this position, i am explaining it, and as i understand discussion is in progress here, and this is considered to be 'temporary'
16:58:39 <xarses> then we build a package for it and push it to stable
16:58:59 <angdraug> I understand, and thanks for explaining it to us
16:59:15 <xarses> I don't understand how we should be excluding any change from this process
16:59:18 <bookwar> angdraug: they take a package but our QA team doesn't check that this package can be taken alone
16:59:31 <angdraug> what you're telling us confirms my worry that it will significantly increase our complexity
16:59:52 <angdraug> so, QA burden is also increased
17:00:06 <bookwar> yes, it will
17:00:08 <xarses> time
17:00:10 <kozhukalov> ok, guys, let's move our discussion to another channel
17:00:18 <kozhukalov> thanx everyone
17:00:21 <kozhukalov> ending
17:00:27 <kozhukalov> #endmeeting