08:00:16 <d0ugal> #startmeeting mistral
08:00:17 <openstack> Meeting started Fri Jul 13 08:00:16 2018 UTC and is due to finish in 60 minutes.  The chair is d0ugal. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
08:00:21 <openstack> The meeting name has been set to 'mistral'
08:00:36 <rakhmerov> d0ugal: that's why the current behaviour should be the default one
08:00:38 <d0ugal> Morning everyone! It is the Friday office hour slot!
08:00:42 <rakhmerov> hi!
08:00:46 <d0ugal> Etherpad: https://etherpad.openstack.org/p/mistral-office-hours
08:00:53 <d0ugal> ^ apetrich, bobh, mcdoker181818
08:01:03 <d0ugal> rakhmerov: indeed, changing the default is bad.
08:01:18 <d0ugal> rakhmerov: but couldn't the workflow DSL have something to control if it is merging or replacing?
08:01:28 <rakhmerov> d0ugal: yeah, but maybe just having a reserved env variable is enough
08:01:45 <rakhmerov> d0ugal: aaah, changing DSL again :)
08:02:00 <rakhmerov> my guts are against it..
08:02:04 <d0ugal> lol
08:02:14 <rakhmerov> :)
08:02:27 <d0ugal> I guess we need somebody to come up with a proposal and then we can discuss it there
08:02:33 <rakhmerov> yep
08:02:43 <rakhmerov> Vitalii I guess )
08:02:47 <d0ugal> Indeed.
08:02:53 <rakhmerov> btw, I have something to discuss
08:02:55 <rakhmerov> https://blueprints.launchpad.net/mistral/+spec/mistral-http-action-binary-data
08:02:59 <rakhmerov> just created
08:03:27 <rakhmerov> my colleagues would like to have this ability in some form
08:03:32 <d0ugal> Interesting
08:03:41 <rakhmerov> but I have doubts about how to implement it properly
08:03:42 <rakhmerov> yeah
08:04:06 <rakhmerov> working with a file system doesn't look right to me
08:04:15 <rakhmerov> so base64 encoding maybe
08:04:25 <d0ugal> yeah, I would prefer base64
08:04:36 <d0ugal> but I guess binary data can sometimes be large? that might be bad for the database?
08:05:25 <rakhmerov> d0ugal: of course..
08:05:26 <rakhmerov> yes
08:05:35 <d0ugal> Using Mistral with another service in the case might be best? Something like Swift could work
08:05:48 <rakhmerov> may be setting some limit would be reasonable
08:06:00 <rakhmerov> or something like this, yes..
08:06:11 <rakhmerov> something that's specifically designed for that
08:06:15 <rakhmerov> for storing large data
08:06:29 <d0ugal> Is this something you hope to do for rocky?
08:06:33 <rakhmerov> no
08:06:41 <rakhmerov> I don't think it's feasible
08:06:54 <rakhmerov> given all other things that i'm hoping to achieve
08:07:04 <rakhmerov> next cycle
08:07:40 <d0ugal> :)
08:08:18 <d0ugal> but that is good, then we have time to think about it more
08:08:26 <rakhmerov> yeah
08:08:41 <rakhmerov> using an external storage seems the best solution for me BUT
08:08:48 <rakhmerov> we need to have that external storage :)
08:09:04 <rakhmerov> lots of people (including us) don't care about OpenStack that much
08:09:04 <d0ugal> Indeed
08:09:06 <rakhmerov> :)
08:09:09 <d0ugal> I guess it could be pluggable
08:09:13 <rakhmerov> yes
08:09:17 <rakhmerov> through some interface
08:09:31 <d0ugal> Yeah, and I guess the interface could be very simple
08:09:39 <rakhmerov> otherwise, our DB would be abused
08:09:45 <d0ugal> (so easy to implement on other backends)
08:09:50 <rakhmerov> yep
08:10:04 <d0ugal> The Mistral DB could even be a backend, but with with warnings and limitations
08:10:25 <rakhmerov> so it could look like just an additional option in std.http to store content into that storage
08:10:41 <rakhmerov> and then if it's needed we could have a YAQL/JINJA function to extract it
08:10:53 <rakhmerov> yeah, agree with your latest thought
08:10:54 <rakhmerov> yes
08:10:57 <d0ugal> Sounds good
08:11:15 <rakhmerov> architecturally it would be more proper solution
08:11:53 <rakhmerov> I think I'll copy our conversation to the BP description
08:11:56 <rakhmerov> do you mind?
08:12:07 <d0ugal> rakhmerov: sure, go for it
08:12:09 <rakhmerov> ok
08:12:14 <d0ugal> it is in the meeting log anyway :)
08:14:07 <rakhmerov> other than that I don't seem to have anything else important
08:14:27 <pgaxatte> hello
08:14:28 <d0ugal> I remember a bug/blueprint (but can't find it now) about passing files into workflows. I wonder if we could use the same storage mechanism to solve that
08:14:30 <d0ugal> pgaxatte: hey
08:14:51 <d0ugal> Found it. https://bugs.launchpad.net/mistral/+bug/1602599
08:14:51 <openstack> Launchpad bug 1602599 in Mistral "Accept multipart file upload as input to Mistral workflows" [Low,Triaged]
08:15:51 <rakhmerov> d0ugal: yeah-yeah, I think it's all different sides of the same problem
08:16:27 <rakhmerov> the bottom line is that we need some kind of storage that better works with large data and binary files
08:16:31 <d0ugal> Yup
08:16:58 <d0ugal> and then we could reduce some of our db col sizes :)
08:17:03 <rakhmerov> yes
08:17:19 <rakhmerov> we've also considered using Glare for some related things too
08:17:28 <d0ugal> oh yes, I remember that.
08:17:39 <d0ugal> I have not heard about Glare in some time...
08:17:46 <rakhmerov> but, as it often happens, we didn't move too far with that
08:17:51 <d0ugal> :)
08:17:58 <pgaxatte> for the storage part, could it be done with swift if you have it available on your cloud
08:18:00 <pgaxatte> ?
08:18:05 <rakhmerov> d0ugal: yeah, because Mike is now your colleague :)
08:18:27 <rakhmerov> pgaxatte: Swift, yes, don't see why not
08:19:07 <pgaxatte> rakhmerov: like a swift container as destination of a std.http action
08:19:16 <d0ugal> oh, I didn't realise he was the main contributor
08:19:18 <pgaxatte> as destination or source
08:19:23 <rakhmerov> pgaxatte: we already have swift integration now (swift actions) but the tricky part is that this integration needs to be deeper
08:19:28 <rakhmerov> imagine std.http action
08:19:46 <rakhmerov> it needs to be smart enough to understand to store its result into Swift
08:19:55 <rakhmerov> w/o even using a swift action implicitly
08:20:19 <rakhmerov> otherwise, we again face the problem of keeping the result temporarily in the context
08:20:24 <pgaxatte> yes that's tricky :)
08:20:26 <rakhmerov> which is relational DB
08:20:28 <rakhmerov> yep
08:20:31 <d0ugal> A very simple option would be to add specialised options std.http_swift for example
08:20:39 <rakhmerov> so, seems like just having swift actions doesn't help much
08:20:51 <pgaxatte> yes d0ugal that's what I was thinking
08:21:33 <d0ugal> but I would prefer a more general solution I think
08:26:09 <pgaxatte> d0ugal, rakhmerov: I have a question on another subject: is it possible to use two different executors that would have different network access and different security levels and specify in a single workflow that an action should be run on the more secure executor
08:26:19 <pgaxatte> like a tag system
08:26:37 <pgaxatte> no tag = this action can run anywhere
08:26:54 <pgaxatte> specific tag = run on specific executor
08:27:36 <d0ugal> pgaxatte: have you seen the target task attribute?
08:27:50 <d0ugal> I have never actually used this, but it might do some of what you want
08:27:54 <d0ugal> It isn't exactly the same...
08:27:58 <d0ugal> https://docs.openstack.org/mistral/latest/user/wf_lang_v2.html#common-task-attributes
08:28:09 <d0ugal> "target - String parameter. It defines an executor to which task action should be sent to. Target here physically means a name of executors group but task will be run only on one of them. Optional."
08:28:54 <d0ugal> I can't find any other docs on it
08:29:43 <pgaxatte> d0ugal: yeah that could be what we're looking for
08:30:03 <d0ugal> pgaxatte: https://github.com/openstack/mistral/blob/1e577ae73925f7703a5d4e4452855208d0788679/mistral/tests/unit/engine/test_environment.py#L53
08:30:12 <d0ugal> It is used in the tests here :)
08:30:15 <pgaxatte> but how do you group the executors? I don't see the option in conf
08:30:24 <d0ugal> I was wondering that too!
08:31:12 <d0ugal> I'm sure rakhmerov knows, but lets see if I can find out before he responds :)
08:31:36 <pgaxatte> :D
08:34:04 <rakhmerov> sorry
08:34:09 <rakhmerov> I was out for a few mins..
08:34:37 <rakhmerov> reading..
08:36:00 <rakhmerov> pgaxatte: I think it's a problem of our doc again
08:36:30 <pgaxatte> rakhmerov: yeah this info is hard to find
08:36:35 <rakhmerov> there needs to be the "host" property in the executor config
08:36:43 <rakhmerov> if I'm not mistaken..
08:36:56 <rakhmerov> but looks like it's not documented, gosh
08:37:31 <d0ugal> https://github.com/openstack/mistral/blob/master/mistral/config.py#L147-L153
08:37:35 <pgaxatte> it's in the config template at least :D
08:37:47 <pgaxatte> # Name of the executor node. This can be an opaque identifier. It is
08:37:47 <pgaxatte> # not necessarily a hostname, FQDN, or IP address. (unknown value)
08:37:47 <pgaxatte> #host = 0.0.0.0
08:37:53 <d0ugal> :)
08:37:56 <rakhmerov> yes, this is confusing
08:38:14 <rakhmerov> the thing is that several executors can have the same value here
08:38:22 <rakhmerov> like "executor_group_1"
08:38:42 <rakhmerov> and then if the "target" is specified, the action will be sent to one of such executors
08:38:49 <pgaxatte> maybe mentionning that this is related to the target option is a good start ^^
08:38:56 <rakhmerov> it's just a matter of building a topic for rabbit
08:38:56 <d0ugal> and if they have the same value, how are executions chosen? round robin?
08:39:03 <rakhmerov> pgaxatte: yep
08:39:11 <rakhmerov> d0ugal: yeah, pretty much
08:39:33 <rakhmerov> round robin by default, yes, it actually fully depends on the message broker
08:39:54 <rakhmerov> because on the client side (mistral engine), it's simply a non-empty topic for the messaging system
08:40:03 <rakhmerov> that's it
08:40:21 <rakhmerov> then MQ decides what receiver is chosen
08:40:55 <pgaxatte> rakhmerov: that's awesome because it is as simple as I was hoping :D
08:41:03 <rakhmerov> yeah :)
08:41:17 <rakhmerov> so, I have to admin that the doc is fairly bad for this
08:41:22 <rakhmerov> including the comment in the config
08:41:29 <rakhmerov> worth filing a bug
08:41:44 <rakhmerov> .. to admit..
08:42:00 <rakhmerov> let me do it
08:43:44 <d0ugal> Thanks :)
08:44:06 <pgaxatte> rakhmerov: are there any blueprints to reorganize the whole documentation?
08:45:05 <d0ugal> pgaxatte: there is at least one :)
08:45:28 <rakhmerov> https://bugs.launchpad.net/mistral/+bug/1781556
08:45:28 <openstack> Launchpad bug 1781556 in Mistral "Improve the doc for "target" task parameter" [Undecided,New]
08:46:25 <rakhmerov> pgaxatte: yeah, we keep improving the doc step by step but we've also discussed that we need to reorganize it
08:46:55 <rakhmerov> d0ugal: not sure where this BP is.. I remember someone filed it recently (a month ago may be)
08:47:12 <pgaxatte> I like how the heat documentation is structured: operating, using, developing
08:47:23 <d0ugal> pgaxatte: yeah, we have discussed something like that several times.
08:47:52 <d0ugal> The problem is finding somebody that knows Mistral well enough and has the time to write lots of documentation
08:47:58 <d0ugal> so we end up stuck every time
08:48:53 <pgaxatte> d0ugal: yeah that's hard to find
08:48:56 <rakhmerov> there it is: https://blueprints.launchpad.net/mistral/+spec/mistral-new-docs
08:49:06 <rakhmerov> there's even a spec for this
08:49:11 <rakhmerov> see a link in the BP
08:49:23 <d0ugal> Yup, the spec needs lots of discussion however
08:49:32 <rakhmerov> yes
08:50:17 <rakhmerov> I know I could do that well but I have no idea when I'll be able to have enough focus on that
08:50:45 <rakhmerov> this work requires a lot of focus during a month-two
09:02:04 <d0ugal> #endmeeting