15:02:51 <d0ugal> #startmeeting mistral
15:02:53 <openstack> Meeting started Mon Oct  9 15:02:51 2017 UTC and is due to finish in 60 minutes.  The chair is d0ugal. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:53 <apetrich> is it now that we start highfiving ?
15:02:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:55 <apetrich> o/
15:02:57 <openstack> The meeting name has been set to 'mistral'
15:03:01 <d0ugal> \o
15:03:13 <toure> o/
15:03:20 <bobh> o/
15:03:29 <d0ugal> Hey all
15:03:34 <d0ugal> Who else is here for the mistral meeting?
15:03:39 <d0ugal> I believe rakhmerov will join us soon
15:04:36 <rakhmerov> hi, just 2 mins please
15:05:10 <rakhmerov> can you maybe give updates in the mean time?
15:05:35 <rakhmerov> what was done, what you're currently doing, plans
15:05:58 <d0ugal> #chair rakhmerov
15:05:59 <openstack> Current chairs: d0ugal rakhmerov
15:06:27 <d0ugal> Sure.
15:06:38 <d0ugal> Please share your statuses.
15:06:44 <d0ugal> I have been mostly trying to move forward with https://bugs.launchpad.net/mistral/+bug/1718353
15:06:46 <openstack> Launchpad bug 1718353 in Mistral "The "context" parameter of Action.run() isn't filled properly for asynchronous actions" [High,In progress] - Assigned to Dougal Matthews (d0ugal)
15:06:57 <d0ugal> Which is a bug related to a limitation in the design of mistral-lib
15:07:08 <d0ugal> apetrich, toure, bobh - do you have any statuses to share?
15:07:11 <bobh> Working on spec - https://review.openstack.org/#/c/510248/
15:07:18 <apetrich> so dynamic actions names landed
15:07:23 <bobh> also a couple of bugs
15:07:27 * toure typing status
15:07:29 <d0ugal> apetrich: \o/
15:07:32 <thrash> o/
15:07:44 <bobh> and proposal to extract expressions module needs discussion
15:07:44 <d0ugal> thrash: we are just sharing statuses if you have anything
15:07:50 <rbrady> o/
15:07:50 <d0ugal> thrash: Renat will be joining us shortly.
15:07:55 <thrash> d0ugal: ack.
15:08:02 <thrash> d0ugal: just zuulv3 stuff.
15:08:12 <d0ugal> thrash: :)
15:08:23 <thrash> d0ugal: which is not so pressing atm considering the rollback
15:08:37 <d0ugal> bobh: interesting, thanks - I'll have to read that spec. I have not looked at the specs for a while.
15:08:44 <apetrich> sometime next week I will take time to do some work on https://blueprints.launchpad.net/mistral/+spec/mistral-workflow-executions-yaql-function
15:09:25 <toure> status: working on unit test for https://review.openstack.org/#/c/506652/ and https://review.openstack.org/#/c/506653/, still making forward progress on https://review.openstack.org/#/c/455447/
15:09:48 <rakhmerov> my status: I keep fixing bugs so that Mistral could work with multiple engine reliably
15:10:02 <rakhmerov> and struggle with CI
15:10:11 <rakhmerov> reviewed a number of patches too
15:10:29 <rakhmerov> thrash: yeah, thanks for doing this
15:10:31 <rakhmerov> appreciate
15:10:59 <rakhmerov> seems like zuul v3 jobs are mostly running fine for us, right
15:11:01 <bobh> I also need to check with Winson on the status of https://review.openstack.org/#/c/455083/
15:11:32 <d0ugal> bobh: that would be good, it was very active for a while and then stalled.
15:11:53 <bobh> d0ugal: we discussed with him at the PTG and agreed on a plan
15:11:57 <rakhmerov> thrash: do we still need to merge https://review.openstack.org/#/c/509184/ ?
15:12:14 <rakhmerov> I remember that similar patches to other repos were reverted by the infra team
15:12:38 <rakhmerov> bobh: good luck :) I mean Winson
15:12:47 <bobh> rakhmerov: :-)
15:12:52 <rakhmerov> he doesn't talk much to us, but I can try to help with that )
15:12:53 <thrash> rakhmerov: yes... I had feedback from the infra team.
15:13:24 <thrash> rakhmerov: but I need to see what the problem with the failed job is.
15:13:41 <rakhmerov> bobh: yeah, he usually does what he's supposed to, he's just a little disconnected from the community
15:13:41 <thrash> but it looks like it's failing under zv2 and zv3
15:13:50 <rakhmerov> thrash: ok
15:14:02 <thrash> rakhmerov: W-1 for now
15:14:09 <rakhmerov> ack
15:15:06 <rakhmerov> d0ugal: is there anything that you think is ready for review related to action context?
15:15:27 <rakhmerov> did the release of mistral-lib come out?
15:15:40 <d0ugal> rakhmerov: yes, it did - 0.3.0 is released
15:15:44 <rakhmerov> ok
15:15:48 <d0ugal> rakhmerov: The patches need work, I have some updates locally I am testing
15:16:03 <rakhmerov> d0ugal: ok, ping us once you need something
15:16:05 <d0ugal> rakhmerov: but I was going to try and ping you tomorrow morning (my time) with some questions
15:16:16 <rakhmerov> sure, let's do it
15:16:27 <d0ugal> thanks.
15:16:30 <rakhmerov> bobh, d0ugal: so let's discuss expressions? )
15:16:38 <d0ugal> Sure
15:16:42 <bobh> rakhmerov: great
15:17:09 <bobh> I had thought of making it a separate library but wasn't sure if that wasn't too big a step
15:17:16 <bobh> but maybe the right step
15:17:31 <apetrich> what is the issue?
15:17:49 <bobh> apetrich: I'd like to be able to use the expressions logic in other applications
15:17:49 <d0ugal> First we should give some context to everyone else.
15:18:05 <d0ugal> Yeah, that is the context basically :)
15:18:23 <rakhmerov> honestly, I'm now kind of confused. I really see that this code would be useful for other projects and sub projects if we moved it to a lib. But I also agree with d0ugal that once we do that we'll have to be maintaining it, documenting and worry about backwards compatibility etc. etc
15:18:25 <d0ugal> the logic for this is mostly under mistral.expressions
15:18:29 <bobh> https://blueprints.launchpad.net/mistral/+spec/move-expressions-to-mistral-lib
15:19:02 <bobh> It is mostly decoupled from mistral itself other than the task, tasks and expressions functions
15:19:35 <rakhmerov> well, btw making a separate lib just for expressions would still mean that we'll have to maintain it etc. )
15:19:43 <rakhmerov> it doesn't matter where it is
15:19:50 <d0ugal> bobh: I think it would be good to clarify exactly what you need - do you also want support for the custom yaql/jinja functions etc.?
15:20:03 <apetrich> and possibly add in the future other expressions that might not be used in mistral
15:20:05 <apetrich> ?
15:20:11 <rakhmerov> yep, good question
15:20:18 <bobh> d0ugal: yes - the custom functions is useful too
15:20:28 <bobh> I agree this is a rabbit hole
15:22:01 <d0ugal> hmm, it is tricky.
15:22:06 <rakhmerov> bobh: so, if you'd like to use the same stuff in your project?
15:22:18 <d0ugal> I wonder if there is a clear enough scope for a oslo.<thing> project.
15:22:19 <rakhmerov> ..remove "if"..
15:22:39 <rakhmerov> d0ugal: ooh, that may be a good idea
15:22:47 <bobh> Yes - I have a templating scheme that will benefit from the ability to process expressions
15:23:02 <rakhmerov> however, we'd still need to maintain it unless we convince someone else to
15:23:19 <d0ugal> rakhmerov: sure, I am less worried about maintaining it really
15:23:20 <rakhmerov> bobh: ok
15:23:43 <bobh> Since I'm the one raising the issue I feel obligated to help with maintaining it
15:23:52 <rakhmerov> bobh: and you want to use YAQL? I'm asking because lots of people just use pure Jinja 2
15:23:53 <bobh> If I just copied the code into my project I'd have to maintain it anyway
15:23:54 <d0ugal> rakhmerov: I am worried about having to fix bugs in $project and then releasing it before mistral can use it
15:24:03 <rakhmerov> which is exactly for this purpose, templating
15:24:23 <rakhmerov> d0ugal: true
15:24:42 <rakhmerov> as long as it's small it's fine but if it grows it may become a problem
15:25:11 <bobh> rakhmerov: I haven't considered just using Jinja - it's a possibility, until there is something that needs the power of YAQL that can't be done in Jinja
15:25:27 <rakhmerov> that's why we need to review it very very thoroughly before moving something to a lib and move only time and battle proven stuff
15:25:41 <rakhmerov> bobh:
15:25:43 <rakhmerov> ok
15:25:44 <bobh> I can take that path for now (Jinja only) and see if we run into any issues
15:26:02 <d0ugal> Maybe you could write a custom jinja function that calls YAQL, then you can have both :)
15:26:15 <bobh> d0ugal: that may be the solution :-)
15:26:16 <rakhmerov> bobh: so, just one more clarification
15:26:56 <rakhmerov> bobh: I guess the main thing that's missing for you in Jinja maybe recursive evaluation over arbitrary structures?
15:27:20 <bobh> rakhmerov: yes, that's probably the biggest difference
15:27:25 <rakhmerov> ok
15:27:26 <rakhmerov> I see
15:27:37 <rakhmerov> yeah, that's something that we could really share
15:27:48 <rakhmerov> and btw, the code is not so big that does that
15:27:51 <bobh> rakhmerov: we use Jinja in most cases because it's faster, but it's not uncommon to find a case where we need YAQL
15:28:31 <rakhmerov> bobh: yeah, and, in fact, they are much different
15:28:36 <rakhmerov> in their nature
15:28:56 <rakhmerov> strictly speaking, Jinja is not primarily an expression language
15:29:00 <rakhmerov> it's a templating lang
15:29:04 <d0ugal> Indeed, the way we use it in Mistral is strange :)
15:29:11 <rakhmerov> whereas YAQL is a pure expression and query languate
15:29:20 <rakhmerov> d0ugal: yes
15:29:37 <bobh> rakhmerov: Yes, I just wish YAQL was as fast as Jinja
15:29:43 <rakhmerov> some of our contributors and users wanted it very much so we decided to include it
15:30:04 <rakhmerov> bobh: did you make any comparisons?
15:30:31 <rakhmerov> I did about 6 months ago and yes, on big structures Jinja was ~3 times faster
15:30:32 <bobh> rakhmerov: we had a bunch of workflows that were heavily using YAQL and took about 5-6 minutes to run
15:30:40 <rakhmerov> ok
15:30:42 <bobh> we converted to Jinja and they run in 1-2 minutes
15:30:50 <rakhmerov> ooh, ok
15:31:14 <bobh> and that's leaving the complex stuff in YAQL so it's a mix - probably 90% Jinja/10% YAQL
15:31:33 <rakhmerov> I know that some performance improvements for YAQL were made a few months ago but I didn't retest after that
15:31:53 <rakhmerov> bobh: haha, I see )
15:32:15 <rakhmerov> bobh: are you using Mistral Pike?
15:32:19 <bobh> yes
15:32:22 <rakhmerov> or an older version?
15:32:25 <rakhmerov> ok
15:32:32 <bobh> I haven't retested on Pike - we were on Ocata at the time I thinkg
15:32:41 <rakhmerov> then all YAQL improvements should be there..
15:32:56 <rakhmerov> ooh, then I'd recommend to retest
15:33:11 <rakhmerov> you may see better results
15:33:49 <rakhmerov> I actually had a chat with the YAQL author in Atlanta and he explained me why it was slower
15:33:50 <bobh> ok, I'll give it a try
15:34:04 <rakhmerov> and soon after that he made necessary changes to speed it up
15:34:09 <rakhmerov> ok
15:35:06 <bobh> ok, so maybe I should just go with YAQL and skip Jinja :-)
15:35:08 <rakhmerov> d0ugal, bobh: so I'm just thinking what we decided )
15:35:20 <rakhmerov> bobh: if it works for you
15:35:40 <bobh> rakhmerov: leave it for now and revisit if needed?
15:36:02 <rakhmerov> d0ugal: so if we make sure that this expressions code is stable enough would it be ok to move it to mistral-lib?
15:36:14 <rakhmerov> so that we know that we won't need to maintain it much
15:36:20 <d0ugal> rakhmerov: I still think it is a strange place for it to live
15:36:22 <rakhmerov> or it's not your biggest concern?
15:36:49 <d0ugal> I liked mistral-lib being small and focused, we could tell people that they just needed to use that to extend mistral
15:36:52 <rakhmerov> strange why? Can you just state it clearly again please?
15:37:06 <d0ugal> but then we need to tell them this other bit in there is for writing other random projects
15:37:24 <rakhmerov> hm... yes
15:37:35 <rakhmerov> hah..
15:37:45 <d0ugal> and where in mistral-lib should it go?
15:37:55 <bobh> could extend mistral with other expression handlers?
15:38:19 <rakhmerov> d0ugal, bobh: now to me it feels a lot like we'd just need to create a totally new open source project and publish it on pypi
15:38:29 <d0ugal> bobh: I guess, but I think that is just inventing a use-case :)
15:38:30 <rakhmerov> and include into global requirements
15:38:41 <d0ugal> If you really want to put it in mistral-lib, I wont block it
15:38:54 <d0ugal> but just prepare for me to moan each time we change it and have to make a new release for mistral to use it :P
15:38:55 <rakhmerov> bobh: yes, expression evaluators are now pluggable
15:39:09 <rakhmerov> we can write something else and easily plug it
15:39:19 <bobh> rakhmerov: I agree a separate lib would be nice, minus the overhead, and I'd be willing to take it on if it makes sense to do it that way
15:40:05 <rakhmerov> well, since we're in the OpenStack community then the logical idea would be oslo umbrella, of course
15:40:10 <bobh> it could also go in oslo, either as oslo.expressions or just expressions (or whatever) if they will take it
15:40:20 <bobh> great minds type alike
15:40:21 <rakhmerov> like oslo.utils or something like this
15:40:29 <rakhmerov> :)
15:40:32 <d0ugal> We should raise this idea with the oslo team and see what they think
15:40:40 <rakhmerov> yes, exactly
15:40:47 <d0ugal> I am not familiar with how oslo projects are created/managed
15:40:57 <rakhmerov> bobh: how about writing an email to ML and ask oslo team directly?
15:41:03 <bobh> sounds like a good plan
15:41:07 <rakhmerov> ok
15:41:11 <bobh> I'll send it out today
15:41:49 <rakhmerov> this is my #1 candidate for now: https://github.com/openstack/oslo.utils
15:42:07 <rakhmerov> if they agree that it's a good place then we won't even need to create any new projects
15:42:17 <rakhmerov> we can just create a new package within this one
15:42:31 <rakhmerov> or a module
15:42:33 <bobh> rakhmerov: that would be ideal
15:42:36 <rakhmerov> expression_utils.py
15:42:50 <d0ugal> rakhmerov: lots of people use oslo.utils, I wonder if they will want to add jinja2 and yaql as requirements.
15:43:01 <rakhmerov> no, probably a package because we have several things
15:43:03 <rakhmerov> but, anyway..
15:43:04 <d0ugal> hah
15:43:06 <d0ugal> https://github.com/openstack/oslo.utils/blob/master/requirements.txt#L5-L8
15:43:44 <rakhmerov> ooh, damn it..
15:44:10 <rakhmerov> haaah.. what Doug wrote though is a pure practical thing to me
15:44:53 <d0ugal> yeah
15:45:00 <d0ugal> anyway, I guess an email will clarify this.
15:45:09 <d0ugal> Then we can take it from there.
15:45:26 <bobh> sounds good - thanks for the discussion - it's a big help
15:45:27 <rakhmerov> ok
15:45:28 <rakhmerov> yes
15:45:41 <rakhmerov> yeah, it's better to ask them directly and hear their feedback
15:45:58 <rakhmerov> having it will allow us to proceed at least'
15:46:25 <rakhmerov> so do we need anything else?
15:46:50 <d0ugal> Nothing from me.
15:46:53 <rakhmerov> #action bobh: write an email to ML about adding expression utils to oslo
15:47:51 <bobh> rakhmerov: I'd like to get feedback on https://review.openstack.org/#/c/510248/ - specifically the name of the new join action
15:47:59 <rakhmerov> I don't have much more for now, I'm focused on performance and stability now. Again realized that a number of places must be seriously changed to improve overall performance
15:48:20 <rakhmerov> bobh: yes, I saw it but didn't open yet
15:48:25 <bobh> rakhmerov: np
15:48:33 <rakhmerov> #action rakhmerov: review https://review.openstack.org/#/c/510248/
15:48:38 <bobh> rakhmerov: no rush
15:48:39 <rakhmerov> will do
15:49:50 <rakhmerov> ok
15:50:19 <rakhmerov> so are we done then?
15:50:21 <rakhmerov> d0ugal: ?
15:50:33 <d0ugal> rakhmerov: yup, I'm done :)
15:50:44 <rakhmerov> ok
15:50:49 <rakhmerov> thanks to all
15:50:56 <d0ugal> Thanks!
15:50:58 <rakhmerov> have a good week
15:51:05 <bobh> thanks!
15:51:09 <toure> thanks
15:51:12 <rbrady> thanks
15:51:33 <rakhmerov> #endmeeting