16:01:01 <xarses> #startmeeting fuel
16:01:01 <xarses> #chair xarses
16:01:02 <xarses> Todays Agenda:
16:01:02 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:01:02 <xarses> Who's here?
16:01:02 <openstack> Meeting started Thu Aug  4 16:01:01 2016 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:05 <openstack> The meeting name has been set to 'fuel'
16:01:06 <dpyzhov> hi
16:01:06 <openstack> Current chairs: xarses
16:01:07 <vkmc> o/ thanks
16:01:10 <kozhukalov> hello
16:01:12 <maximov> hi
16:01:12 <agordeev> o/
16:01:13 <romcheg> o/
16:01:14 <gkibardin> Hi
16:01:16 <warpc> Hi!
16:01:32 <vkramskikh> hi
16:01:38 <ikutukov> Hi!
16:01:58 <omolchanov_> hi
16:02:23 <xarses> #topic CSV and YAML output from handlers in nailgun (https://blueprints.launchpad.net/fuel/+spec/ui-deployment-history) (dguryanov)
16:02:41 <ashtokolov> o/
16:03:10 <kozhukalov> oops, looks like dguryanov2 is afk
16:03:17 <dguryanov2> Hi!
16:03:20 <xarses> ok
16:03:37 <xarses> #topic Removing old fuel CLI (romcheg)
16:03:45 <romcheg> Hi everyone!
16:03:48 <dguryanov2> About CSV format!
16:04:02 <xarses> dguryanov2: we'll come back
16:04:02 <romcheg> I let's allow dguryanov2 to comment on his topic
16:04:09 <kozhukalov> dguryanov2: let's postpone your topic
16:04:09 <xarses> erm hang on then
16:04:17 <xarses> #topic CSV and YAML output from handlers in nailgun (https://blueprints.launchpad.net/fuel/+spec/ui-deployment-history) (dguryanov)
16:04:22 <xarses> ok, go ahead
16:04:27 <dguryanov2> Thanks!
16:04:47 <dguryanov2> I have 2 patches, one of them splits decorator @content
16:05:36 <dguryanov2> Into the one, which handles http errors, second does validation and the last one converts return value from hanlders to string
16:05:51 <kozhukalov> could you please share links to the patches?
16:05:54 <dguryanov2> And second patch adds support of CSV format
16:05:57 <dguryanov2> Sure
16:06:09 <evgenyl> Hi!
16:06:11 <dguryanov2> https://review.openstack.org/346957 and https://review.openstack.org/348419
16:07:05 <dguryanov2> I've also added ability to export data in YAML, if client provides header Accept: application/x-yaml
16:07:05 <maximov> so looks like splitting decorator is refactoring
16:07:29 <maximov> and it isn;t directly related to CSV export
16:07:43 <kozhukalov> remember, we are now under xenial code freeze (no mergers until freeze is lifted)
16:08:19 <dguryanov2> Yes, it's sort of refactoring, but it's much erasier to implement CSV with this patch
16:08:33 <xarses> so what do you need from the group?
16:08:47 <kozhukalov> refactoring is welcome, of course if you have enough time
16:09:06 <ikutukov> Yes, without this refactoring we have all output logic melted in the @content decorator and it's hard to aply only part of the @content logic.
16:09:50 <dguryanov2> kozhukalov: The patch is ready for review
16:09:55 <maximov> ikutukov: dguryanov2 please just make sure we don't introduce any regressions, please build custom bvt and run some tests
16:10:04 <kozhukalov> second patch failed to pass pep8
16:10:24 <dguryanov2> kozhukalov: I'll fix it today
16:10:28 <ikutukov> We have a capacity reports handler with existing and very custom CSV output implementation. Does anyone know do we have to support this function or not?
16:10:39 <ikutukov> Who use this?
16:11:17 <kozhukalov> maximov: no need to build custom bvt, we run deployment tests on every nailgun patch
16:11:18 <xarses> I was wondering why we support multiple output's in api too, I figured the client would convert it
16:12:23 <xarses> dguryanov2: what is the ask from the group here?
16:12:31 <vkramskikh> xarses: fuel ui is another client, i don't consider a good idea to implement convertion to yaml on the client side
16:13:28 <dguryanov2> xarses: just wanted to discuss implemented approach and ask to review :)
16:13:29 <romcheg> my 50 cent is that we are about to introduce a layer violation
16:13:31 <ikutukov> in this case we have to re-implement serialisation on each client. Moreover json->yaml->json is possible to automatically without any logic, but in many cases it is not possible to convert json(tree structure) to CSV automatically having usable output.
16:14:05 <ikutukov> Layer of client or Nailgun? )
16:14:45 <romcheg> API is responsible for taking data from users and returning a result
16:15:03 <romcheg> why should it care about how user needs to convert that data?
16:15:34 <kozhukalov> mvc
16:15:58 <romcheg> I remember we had a lot of problems supporting this feature in Neutron
16:16:02 <ikutukov> because many products providing API do, we are slightly boosting integration ability with relatively no cost.
16:16:06 <kozhukalov> why do you think it is a layer violation&
16:16:10 <kozhukalov> why do you think it is a layer violation?
16:16:45 <kozhukalov> moving on?
16:16:49 <xarses> we need to move along
16:16:58 <maximov> ikutukov: regarding capacity report, I don't know actually, maybe we need to consider removing this capacity report feature if nobody uses it. I don't remember any feedback about it. can you send a link to this handler, just want to check?
16:17:02 <xarses> we can discuss more in #fuel or on the ML
16:17:08 <romcheg> kk
16:17:16 <maximov> ok, let's move on
16:17:17 <ikutukov> ... or moving to extension/plugin
16:17:30 <xarses> #topic Removing old fuel CLI (romcheg)
16:17:31 <romcheg> lets shift that to #fuel or to the ml
16:17:58 <romcheg> So I've just realized I cleared my buffer so I have to type that again :)
16:18:12 <ikutukov> https://github.com/openstack/fuel-web/blob/acbb7370052ffb4526429b4255f26a9c13d4e7c7/nailgun/nailgun/api/v1/handlers/capacity.py#L105
16:18:32 <romcheg> I would like to announce we have finally found resources to consolidate features from the old and the new CLIs
16:18:43 <xarses> \o/
16:19:08 <kozhukalov> that is awesome
16:19:10 <xarses> so what are the plans there?
16:19:19 <romcheg> As the result will have 'fuel' command that will start the new cliff based application which is now available as fuel2
16:19:51 <romcheg> The first step is to implement several commands in fuel2 that are still missing
16:20:18 <romcheg> That is mostly done. The patches that are to merge are available under the deprecate_old_cli topic in Gerrit
16:20:52 <romcheg> I work closely with the QA team to resolve any dependencies and remove the old CLI on the next week
16:20:57 <xarses> #link https://review.openstack.org/#/q/topic:deprecate_old_cli,n,z
16:21:04 <romcheg> xarses: thanks!
16:21:07 <ashtokolov> romcheg: great, awesome news! Are we going to land it in 9.1?
16:21:15 <romcheg> ashtokolov: yes we are
16:21:24 <ashtokolov> thanks!
16:21:30 <maximov> romcheg: some of our users tends to use fuel client for automation purposes and their automation scripts depend on fuel cli and even depend on output of particular commands
16:21:41 <romcheg> #note If someone has their scripts that use the old fuel command, it's time to update them
16:22:02 <romcheg> maximov: old fuel cli is still installable from PyPi
16:22:05 <ashtokolov> maximov: it's a wrong approach to use CLI for it, they should use API
16:22:10 <kozhukalov> first we need to land it onto master and then onto mitaka
16:22:37 <maximov> did we explicitly mention it in spec that old cli will be deprecated in newton ?
16:22:47 <romcheg> there is no spec for this
16:22:52 <kozhukalov> ashtokolov: no, you are wrong
16:23:04 <maximov> ashtokolov: we didn't document our API unfortinatly
16:23:19 <kozhukalov> client's intent is exactly to provide python binding to API
16:23:19 <xarses> maximov: probably not
16:23:21 <romcheg> I mean, there is a spec, but it is old and it says that the old CLI is going to be removed soon
16:23:24 <maximov> romcheg: we need to create spec which reflects all these details
16:24:09 <maximov> romcheg: ok, but if we are ready to get rid of old cli then let's have this spec "deprecation of old fuel cli"
16:24:12 <ashtokolov> kozhukalov: the CLI output can be changed, but API should be backward compatible
16:24:53 <kozhukalov> yes, i mean CLI is optional, CLI itself uses python binding to API
16:24:54 <xarses> ok, romcheg do you have good idea of the next steps?
16:25:08 <romcheg> maximov: ok, probably it's a good idea, at least we will be able to create a table with old and new commands
16:25:21 <kozhukalov> yes, users should use python not shell commands if they do automation
16:25:24 <maximov> romcheg: yes.
16:25:32 <maximov> ashtokolov: our API is not documented so it is hard to use it
16:25:44 <kozhukalov> but not REST directly
16:25:56 <romcheg> I will comment on API vs shell later
16:25:59 <romcheg> re: steps
16:26:07 <ashtokolov> maximov: that's the problem, but not the CLI backward compatibility
16:26:15 <romcheg> Wi are trying to merge remaining patches ASAP
16:26:51 <romcheg> then I help the QA team to update fuel-qa and then we get rid of the old CLI
16:26:53 <xarses> guys, let's not worry about CLI / python bindings / API users will use what they do
16:26:54 <romcheg> in master
16:27:14 <xarses> romcheg: we probably have to keep it for newton if we didn't mark it as deprecated
16:27:15 <kozhukalov> romcheg only after xenial code freeze is lifted
16:27:37 <romcheg> kozhukalov: is that code freeze relevant to python-fuelclient?
16:27:43 <maximov> kozhukalov: we are very close to lift freeze, I guess it will happen tomorrow
16:27:55 <romcheg> re: API
16:28:44 <kozhukalov> to avoid any ambigoities, yes
16:28:53 <romcheg> the new fuel client also provides convenient API wrapper that allows to use python-fuelclient as a library, supports parallel connections and does more useful stuff as for a library
16:29:01 <romcheg> kozhukalov: makes sense
16:29:57 <romcheg> It ever supports API versioning (which is pretty useless now) :)
16:30:11 <xarses> ok, moving on?
16:30:16 <romcheg> +1
16:30:23 <xarses> #topic Discuss http://lists.openstack.org/pipermail/openstack-dev/2016-June/097558.html [openstack-dev] [Fuel] It is impossible to queue UpdateDnsmasqTask (gkibardin)
16:30:25 <maximov> romcheg: thanks for update
16:30:36 <gkibardin> 
16:30:36 <gkibardin> I sent this http://lists.openstack.org/pipermail/openstack-dev/2016-June/097558.html some time ago but almost nobody has shown any particular interest so I've decided to bump it here. I need more input to make a decision, probably more ways to solve the problem. The option 1 is considered a mitigation not a fix. The option 4, as it turned out, doesn't work.
16:31:07 <gkibardin> so any input is welcome
16:32:17 <maximov> gkibardin: can you provide example (scenario) when user can run into this issue?
16:32:31 <xarses> delete two clusters near the same time?
16:32:39 <gkibardin> exactly
16:33:04 <maximov> maybe it is easier just to prohibit deletion of two cluster in parallel
16:33:20 <maximov> it isn't very frequent operations
16:33:28 <gkibardin> with an error "come again later"?
16:33:55 <gkibardin> I agree this doesn't happen frequently
16:33:55 <kozhukalov> maximov: that would be weird from api point of view
16:34:09 <gkibardin> however, we may face other problems
16:34:12 <maximov> from UI we can just show error message instantly
16:34:27 <gkibardin> with master node tasks from different env clashing
16:35:01 <gkibardin> solution 1 suggest to put the second env to an error state
16:35:08 <gkibardin> so that it could be retried later
16:35:28 <xarses> shouldn't we just prevent the master node from running more than one task at a time
16:35:37 <kozhukalov> maybe to update dnsmasq via dbus
16:35:59 <kozhukalov> that would be much more reliable (no need to restart)
16:36:09 <gkibardin> yes, it is option 2 - queue things in Nailgun
16:36:50 <maximov> gkibardin: what if I cancel operation
16:36:58 <ikutukov> what queue?!
16:37:30 <gkibardin> maximov: you mean in case it clashes with another one?
16:38:02 <gkibardin> ikutukov: we need to either queue clashing operation or report an error
16:38:09 <maximov> if you queue something (some task) and then you cancel deployment you should remove this task from queue
16:38:16 <xarses> how does the flow work now? isn't the task sent to amqp and picked up by astute anyway? and astute runs in on the master node?
16:38:18 <gkibardin> sure
16:38:45 <gkibardin> xarces: yes
16:38:51 <gkibardin> but Nailgun prevent this
16:39:12 <xarses> why? won't astute just queue it because the master executor is busy?
16:39:18 <gkibardin> two UpdateDnsmask tasks to run at the same time
16:39:34 <gkibardin> astute doesn't queue it, it executes them in parallel
16:40:01 <gkibardin> and this may cause problems
16:40:10 <xarses> hmm, seems that it shouldn't
16:40:10 <maximov> what we do in UpdateDnsmask ?
16:40:26 <maximov> exactly
16:40:36 <xarses> update the dhcp pool data in /etc/cobbler/dnsmasq.template
16:40:49 <gkibardin> yes, using puppet
16:41:02 <gkibardin> and uploading a network list before that
16:41:03 <xarses> based on cfg for multiple fwadmin networks
16:42:23 <xarses> it seems like we should fix astute to be globally sensative to number of parallel execution on master node in some cases
16:42:43 <gkibardin> this is option 3
16:42:46 <xarses> and nailgun should tell astute that this task can only run one at a time
16:43:19 <gkibardin> the only downside - it is relatively hard to implement
16:43:33 <gkibardin> astute spawn several workers
16:43:43 <gkibardin> they know nothing about each other
16:43:51 <kozhukalov> looks like we need synchronization primitive for syncing tasks for different clusters
16:43:54 <xarses> but we can give it this context in the deployment graph
16:44:04 <xarses> why cant we do that here?
16:44:26 <kozhukalov> like yaql
16:44:50 <kozhukalov> yes, i mean in graph
16:45:08 <xarses> well, we could just send a magic cluster ID for stuff that isn't really bound to a cluster so it can sort them with out too much change
16:45:12 <gkibardin> yes, we can mark it with some tag in a graph so that asutute queue all tasks with the same tag
16:46:02 <gkibardin> yes, its also an option ashtokolov proposed to use cluster_id 0
16:46:55 <maximov> gkibardin: ok, looks reasonable. what are drawbacks?
16:47:28 <gkibardin> astute gets more complicated, its workers must synchornize somehow
16:47:49 <gkibardin> but it is feasible
16:48:02 <ashtokolov> gkibardin: we often use cluster_id=0 in tests, so we should use it  carrefully
16:48:34 <kozhukalov> let's use id=-1 then
16:48:37 <maximov> ashtokolov: well, we can use -1
16:48:38 <gkibardin> ashtokolov: I would also think about avoiding using cluster id
16:48:45 <gkibardin> )))
16:49:07 <ashtokolov> +1 to -1 ;)
16:49:21 <gkibardin> but anyway, these are details, I like that we almost agreed on choosing at least a direction
16:49:55 <xarses> gkibardin: so lets look more at trying to fix astute, it seems like the correct place
16:50:10 <ikutukov> don't shift the range of identifiers, please)
16:50:22 <xarses> moving on?
16:50:33 <gkibardin> ikutukov: I'll try
16:50:38 <gkibardin> yes, thanks everybody!
16:50:53 <xarses> #topic http://bit.ly/1RD6JLR need review
16:51:05 <xarses> dpyzhov: is this part of your topic?
16:51:12 <dpyzhov> nope
16:51:19 <kozhukalov> it is mine
16:51:21 <xarses> k
16:51:41 <kozhukalov> it is just a link to review requests that need attention
16:51:50 <kozhukalov> please help to review
16:51:54 <kozhukalov> thanks
16:52:02 <xarses> ya, there are a lot open =)
16:52:12 <xarses> or =/ evne
16:52:16 <xarses> even
16:52:24 <xarses> #topic Bugs triage for 9.1 https://etherpad.openstack.org/p/fuel-9-1-bugs-triage (dpyzhov)
16:52:53 <dpyzhov> Guys, I'm trying to setup a flow for proposing bugs for stable/mitaka
16:53:21 <dpyzhov> As I see we have only 7 minutes left so there is no sense to review particular bugs
16:53:34 <dpyzhov> let's talk about approach
16:54:15 <dpyzhov> I guess I can mark all product bugs with special tags and send an e-mail for discussion
16:54:22 <xarses> so what are we looking to do here?
16:54:36 <dpyzhov> and if someone have disagreements then we can find a consensus )
16:55:20 <dpyzhov> What I want is to limit scope for stable/mitaka
16:55:21 <xarses> and how much does it change if we decouple the fuel-library 9.1 release and deploy it with current master
16:56:55 <dpyzhov> xarses: for what reason?
16:57:14 <kozhukalov> because we can?
16:57:20 <xarses> because with https://review.openstack.org/#/c/346143 we can
16:57:22 <xarses> and should
16:57:49 <xarses> we make many releases of fuel from master
16:58:14 <xarses> and 9.x library release can consume what it needs
16:58:16 <dpyzhov> I'm trying to understand how it relates to bugfix scope
16:58:31 <xarses> you don't backport non library fixes
16:58:42 <xarses> you take them from master
16:58:54 <ikutukov> But we will have chaos in tests.
16:59:26 <xarses> why? we'd test both the mitaka library and newton
16:59:34 <dpyzhov> xarses: so you propose to not backport any patches except library and rely on this feature?
16:59:55 <xarses> dpyzhov: in the long term yes, it's too late for 9.1
17:00:02 * hyakuhei_ waves
17:00:13 <xarses> thanks
17:00:13 <browne> o/
17:00:23 <xarses> we are over time, we can discuss more on #fuel
17:00:24 <elmiko> hi
17:00:35 <xarses> #endmeeting