Tuesday, 2016-10-18

*** bobh has joined #openstack-mistral00:00
*** bobh has quit IRC00:05
*** bobh has joined #openstack-mistral00:33
*** jamielennox is now known as jamielennox|away01:01
*** jamielennox|away is now known as jamielennox01:08
*** evgenyl has quit IRC01:51
*** evgenyl has joined #openstack-mistral01:52
*** bobh has quit IRC02:38
openstackgerritLingxian Kong proposed openstack/mistral-specs: Move spec files to correct location  https://review.openstack.org/38776603:00
*** jamielennox is now known as jamielennox|away03:01
openstackgerritLingxian Kong proposed openstack/mistral-specs: Move spec files to correct location  https://review.openstack.org/38776603:15
*** jamielennox|away is now known as jamielennox03:33
*** rbrady is now known as rbrady-afk03:35
kongd0ugal: hi, i remember you added the tripleo-ci, could you please take a look at why jenkins failed for this patch in stable/newton: https://review.openstack.org/#/c/387757/?03:45
kongi'm not familiar with how tripleo works with mistral03:46
rakhmerovd0ugal: hi, sorry, missed your last message yesterday04:02
rakhmerovping me anytime when you're ready04:03
*** dkushwaha has joined #openstack-mistral05:08
*** jaosorior has joined #openstack-mistral05:18
*** chlong has quit IRC05:47
*** chlong has joined #openstack-mistral06:01
*** shardy has joined #openstack-mistral07:08
*** shardy has quit IRC07:22
*** jpich has joined #openstack-mistral07:39
d0ugalrakhmerov: Thanks, today is a particularly busy one for me, so I might try tomorrow :)07:52
d0ugalrakhmerov: but the tl;dr is that we have lots of repetition in our workflows. How would you feel about the idea of workflow inheritance?07:52
d0ugal:)07:52
rakhmerovok07:52
d0ugalI guess it could be a bit like ad-hoc actions, with a base defined etc.07:53
d0ugalFor example, we have this in a large number of places: https://github.com/openstack/tripleo-common/blob/master/workbooks/plan_management.yaml#L110-L12107:53
d0ugalThat we could probably reduce by using a ad-hoc workflow, but there are others too07:54
d0ugalbrb07:55
d0ugalrakhmerov: ^07:55
rakhmerovhm.. it generally feels good to me, need to discuss details07:56
rakhmerovreading07:56
rakhmerovyes, that makes sense to me07:56
*** igormarnat has quit IRC07:56
rakhmerovmy only concern is that we should not overcomplicate the language07:56
*** akuznetsova has quit IRC07:56
rakhmerovpeople like the fact that it's pretty simple now07:57
d0ugalrakhmerov: Yeah, I would consider it a more advanced feature07:59
d0ugalbut I think if you have a large number of workflows it should make it easier to maintain them08:00
rakhmerovyeah08:00
rakhmerovok08:00
rakhmerovcan you include it into our summit topics pls with some basic description?08:01
*** akuznetsova has joined #openstack-mistral08:01
d0ugalrakhmerov: Sure, I'll try and start a draft spec to describe it a bit more08:02
rakhmerovok08:03
*** igormarnat has joined #openstack-mistral08:03
rakhmerovgenerally, I think it's a good idea08:03
d0ugalGreat, I wanted to check this before I spent time writing a spec :)08:04
*** openstackgerrit has quit IRC08:04
*** openstackgerrit has joined #openstack-mistral08:05
*** flwang1 has joined #openstack-mistral08:09
flwang1rakhmerov: ping08:10
rakhmerovflwang1: hi08:10
flwang1need your help08:10
rakhmerovyes08:10
flwang1see bug https://bugs.launchpad.net/mistral/+bug/163409008:10
openstackLaunchpad bug 1634090 in Mistral "Failed to create keystone client" [Undecided,New] - Assigned to Fei Long Wang (flwang)08:10
flwang1and i also test a very simple workflow, which just do nova.servers_list08:10
rakhmerovok, I see08:12
rakhmerovcan you show me your Mistral config?08:12
rakhmerovspecifically how you configure auth08:12
flwang1in mistral executor log, i also got http://paste.openstack.org/show/586127/08:12
flwang1sure, wait a sec08:12
rakhmerovyeah, the error in the executor is a consequence08:13
flwang1http://paste.openstack.org/show/586128/08:13
rakhmerovlooks fine to me08:16
rakhmerovwhat env vars do you source on the client side?08:16
rakhmerovI mean, with what vars do you initialize Mistral client?08:17
flwang1wait a sec08:18
flwang1http://paste.openstack.org/show/586129/08:19
flwang1i installed mistral by devstack08:22
rakhmerovexport OS_PASSWORD=passw0rdshow/586128/08:22
rakhmerovis this ok?08:23
flwang1why do i need a different password?08:23
flwang1i don't think it's related08:23
rakhmerovI mean it looks strange08:23
rakhmerovbut ok08:23
therveI don't think mistral is equipped to handle trust tokens08:24
therveIt doesn't keystone sessions or keystoneauth to do the job08:24
flwang1rakhmerov: seems the password is changed by paste, my original password is passw0rd08:24
rakhmerovflwang1: ok08:24
rakhmerovflwang1: this is what I use on the client usually: http://paste.openstack.org/show/586130/08:25
rakhmerovit works08:25
flwang1rakhmerov: hmm... i don't think it's related to the rc file, tbh08:26
rakhmerovnot sure if it's the reason but, for example, I use OS_AUTH_URI, not OS_AUTH_URL08:26
flwang1since i can user it for all the other services08:27
rakhmerovok08:27
flwang1to be more sepcifci08:27
flwang1we're using a trust token to access mistral08:27
flwang1from zaqar08:27
rakhmerovyes08:27
flwang1and i can see the context is this08:28
flwang1MistralContext {u'project_name': None, u'user_id': u'af6f84aaf58442cb85fc34e9ba6d58a4', u'roles': [u''], u'auth_uri': u'http://127.0.0.1:5000/v3', u'auth_cacert': None, u'auth_token': u'e86a0754f9e04fc79ca58b6d833a6f8a', u'expires_at': u'2016-10-18T08:11:25.000000Z', u'is_trust_scoped': False, u'project_id': None, u'user_name': u'admin', u'target_service_catalog': None}08:28
flwang1is_trust_scoped is False08:28
flwang1so https://github.com/openstack/mistral/blob/master/mistral/utils/openstack/keystone.py#L125 line is false, can then go to line 13608:29
therveflwang1, is_trust_scoped is an internal flag to know if mistral issued the trust, I think08:29
rakhmerovhm..08:30
rakhmerovthis is, btw, relatively a new change08:30
flwang1therve: yep, but it's causing mistral goes to line 13608:30
flwang1technically, it should go to line 12608:30
therveI don't think so, because it's using service credentials there08:30
rakhmerovso, just to clarify: you pass a token to Mistral obtained from a trust, right?08:31
flwang1rakhmerov: yep08:32
flwang1and in my latest tests, even i use a normal token, i still failed to do any openstack service actions08:32
flwang1i hope it's my env issue, but it's a new devstack env08:32
flwang1before re-installed this env, i got the same error before08:33
rakhmerovflwang1: ok, let me try to get the author of these changes involved08:35
flwang1therve: probably you're right, since there is no trust id in ctx08:35
rakhmerovtherve: my understanding is that "is_trust_scoped" is to determine the type of token08:35
rakhmerovwhether it was obtained from a trust or not08:36
flwang1therve: rakhmerov: is it related the project_id is None?08:36
therverakhmerov, Right, but only if the trust was created by mistral08:36
therveIt's not meant to handle trust passed from the outside, AFAICT08:36
flwang1therve: +108:37
rakhmerovhm.. maybe08:37
rakhmerovflwang1: yeah, I think project_id must not be None08:37
rakhmerovtry to pass it too08:37
flwang1it's based on my latest simple mistral test08:38
flwang1let me re run my script to see the context08:39
flwang1rakhmerov: if you have a local env, could you pls try run a simple workflow to trigger a heat or even nova action?08:42
therveflwang1, http://paste.openstack.org/show/586134/ works locally for me08:44
flwang1 MistralContext {u'project_name': u'admin', u'user_id': u'af6f84aaf58442cb85fc34e9ba6d58a4', u'roles': [u'admin'], u'auth_uri': u'http://127.0.0.1:5000/v3', u'auth_cacert': None, u'auth_token': u'ac172d0eeb154ec3ab3416fecafcb202', u'expires_at': u'2016-10-18T09:43:14.000000Z', u'is_trust_scoped': False, u'project_id': u'13401d12e2b44e9893c744868687d5b2', u'user_name': u'admin', u'target_service_catalog': None}08:44
therveIt works around the issue, but it may be good enough for now08:44
flwang1therve: awesome, let me try08:46
flwang1therve: except mistral executor, do i need to restart any other processes?08:49
rakhmerovI'd restart all of them08:52
therveflwang1, You only need to restart api actually08:53
flwang1i got same error after only restarted executor, now i'm going to restart api as well08:54
rakhmerovtherve: do you think it's only a workaround?08:59
flwang1therve: it works :)08:59
rakhmerov:)08:59
rakhmerovgood08:59
therverakhmerov, Yeah because if the catalog is not there we're still not going to be able to fetch it08:59
flwang1rakhmerov: therve: do you mind me propose it as the initial patch?08:59
rakhmerovtherve: did you check if we have a bug filed for that?08:59
therverakhmerov, No09:00
rakhmerovflwang1: yeah, that's what I thought too09:00
therveflwang1, No09:00
flwang1therve: oh, do you have any better idea?09:00
flwang1i'm happy to follow up09:00
flwang1after summit :D09:00
therveflwang1, I mean I don't mind, no09:00
rakhmerovat least let's file a bug pls09:00
therveWell the bug is already filled, no?09:01
rakhmerovtherve: can you file a bug please?09:01
flwang1rakhmerov: i have filed a bug09:01
rakhmerovtherve: I don't know, honestly09:01
flwang1https://bugs.launchpad.net/mistral/+bug/163409009:01
openstackLaunchpad bug 1634090 in Mistral "Failed to create keystone client" [Undecided,New] - Assigned to Fei Long Wang (flwang)09:01
rakhmerovooh, shoot....09:01
rakhmerovsorry, seems like my head is somewhere else09:01
rakhmerovok, yes09:01
flwang1cool, i will do it and add you guys as reviewer, then we can shape it there09:02
flwang1thank you so much for your help, you make my day09:02
flwang1i can finally trigger a heat action in mistral09:02
rakhmerovflwang1: thanks to therve09:02
rakhmerovflwang1: I'll make the slides either today or tomorrow09:03
rakhmerovand get into the demo you prepared09:03
flwang1rakhmerov: awesome, thanks09:04
rakhmerovnp, thanks for delays, hot time09:04
*** openstackgerrit has quit IRC09:04
flwang1therve: any idea for this http://paste.openstack.org/show/586141/09:04
rakhmerov...sorry for delayes..09:04
*** akovi has joined #openstack-mistral09:04
flwang1i can call mark unhealthy, but failed when calling stack update09:04
flwang1rakhmerov: no worries09:04
*** openstackgerrit has joined #openstack-mistral09:04
rakhmerovakovi: hi Andras09:04
therveflwang1, Did you pass --existing?09:04
rakhmerovwe just discussed a bug we have with trust tokens09:05
flwang1therve:     stacks_update: {action: heat.stacks_update stack_id=<% $.root_stack_id %> existing=True}09:05
rakhmerovyou can read our conversation about and look at the workaround proposed by therve09:05
flwang1rakhmerov: is it correct ? if i want to pass the existing param09:06
rakhmerovakovi: I thought you could probably propose a non-workaround solution for this09:06
therveflwang1, It's calling PUT, so it didn't take into account existing09:07
akoviI don't know if I can help but I'll try my best09:07
rakhmerovflwang1: honestly, don't know, you need to look at how stacks_update works09:07
rakhmerovakovi: ok, thanks09:07
rakhmerovakovi: the bug is here: https://bugs.launchpad.net/mistral/+bug/163409009:08
openstackLaunchpad bug 1634090 in Mistral "Failed to create keystone client" [High,New] - Assigned to Fei Long Wang (flwang)09:08
flwang1therve: https://github.com/openstack/python-heatclient/blob/master/heatclient/v1/stacks.py#L17909:09
flwang1rakhmerov: for a param like above, how should i pass the existing?09:09
akovican you please clarify what is the heat service version?09:10
rakhmerovflwang1: it looks OK to me09:11
flwang1rakhmerov: ok, thanks09:12
rakhmerovakovi: as far as I understand, flwang1 uses devstack (latest or close to latest)09:13
akoviwe have some heat tests in the pipeline that do not fail, right?09:14
akoviwe may need to adjust those too after this bug is solved09:14
flwang1rakhmerov: pls see line https://github.com/openstack/python-heatclient/blob/master/heatclient/v1/stacks.py#L17609:14
flwang1after stack_id, it's a kwargs09:14
flwang1so if i want to pass 'existing'09:15
flwang1should i write the action like this stacks_update:  { action: heat.stacks_update stack_id=<% $.root_stack_id %> kwargs={existing:True}} ?09:15
flwang1instead of     stacks_update: {action: heat.stacks_update stack_id=<% $.root_stack_id %> existing=True}09:15
*** shardy has joined #openstack-mistral09:16
*** dkushwaha has quit IRC09:17
openstackgerritFei Long Wang proposed openstack/mistral: Get service catalog from token info  https://review.openstack.org/38788309:28
rakhmerovakovi: I think this is not specifically related to Heat, it's a major issue when Mistral needs to use a token made of trust which is just given to Mistral (Mistral itself didn't create that trust)09:29
flwang1rakhmerov: can you answer my above question?09:33
flwang1should i write the action like this stacks_update:  { action: heat.stacks_update stack_id=<% $.root_stack_id %> kwargs={existing:True}} ?09:33
flwang1instead of     stacks_update: {action: heat.stacks_update stack_id=<% $.root_stack_id %> existing=True}09:33
rakhmerovflwang1: give me a minute..09:33
flwang1rakhmerov: thanks09:33
akovirakhmerov: we haven't played around with trusts yet09:36
rakhmerovflwang1: try existing=true09:38
rakhmerovakovi: ok09:38
rakhmerovflwang1: generally, just existing=true should work because params are passed as kwargs to methods09:39
rakhmerovthe only problem is that Mistral syntax is slightly different when we use it inside a Heat template (very inconvenient) and I don't know all these differences09:40
rakhmerovmostly I use Mistral language directly09:40
rakhmerovso it might make a difference09:40
flwang1rakhmerov: ok, so the   case matters here?09:40
flwang1True vs. true09:40
rakhmerovwe would like to change that in the near perspective so that we could use same syntax in both cases09:41
rakhmerovflwang1: it's just a YAML's true09:41
rakhmerovyes09:41
*** jaosorior has quit IRC09:42
*** jaosorior has joined #openstack-mistral09:42
flwang1rakhmerov: good, it works now. thank you very much09:47
rakhmerovnp09:47
*** akovi has quit IRC09:55
*** jtomasek is now known as jtomasek|biab10:40
*** chlong has quit IRC10:46
*** dprince has joined #openstack-mistral11:14
*** dprince has quit IRC11:14
*** shardy is now known as shardy_lunch11:19
openstackgerritpawnesh kumar proposed openstack/python-mistralclient: Fix warning when running tox -e docs  https://review.openstack.org/38795711:27
openstackgerritpawnesh kumar proposed openstack/python-mistralclient: Fix warning when running tox -e docs  https://review.openstack.org/38795711:30
openstackgerritpawnesh kumar proposed openstack/python-mistralclient: Fix the warning while executing "tox -e pylint".  https://review.openstack.org/38796311:38
*** rbrady-afk is now known as rbrady11:51
*** jtomasek|biab is now known as jtomasek11:54
*** dprince has joined #openstack-mistral11:55
*** janki has joined #openstack-mistral12:17
*** shardy_lunch is now known as shardy12:21
*** Qiming_ has joined #openstack-mistral12:56
*** Qiming_ has quit IRC12:56
*** jaosorior is now known as jaosorior_mtg13:04
*** jtomasek is now known as jtomasek|afk13:33
*** bobh has joined #openstack-mistral13:34
*** bobh has quit IRC13:36
*** bobh has joined #openstack-mistral13:36
*** bobh has quit IRC13:36
*** jaosorior_mtg is now known as jaosorior13:46
*** jaosorior has quit IRC14:16
d0ugalrakhmerov: Are you around?14:24
rakhmerovd0ugal: here14:34
rakhmerovd0ugal: btw, I replied to you in ML14:34
rakhmerovd0ugal: let's try to find a better slot14:35
d0ugalrakhmerov: oh, thanks - I'll take a look at that again.14:35
rakhmerovd0ugal: excuse me if I was a little bit offensive14:35
rakhmerovd0ugal: do you have something to discuss? Some questions?14:36
d0ugalrakhmerov: haha, no problem - I was probably a bit terse in my reply - sorry.14:37
*** bobh has joined #openstack-mistral14:37
d0ugalrakhmerov: Yeah, we have a small usability issue I wanted to discuss.14:37
*** vishwanathj has joined #openstack-mistral14:37
d0ugalrakhmerov: We use this pattern: https://github.com/openstack/tripleo-common/blob/master/workbooks/plan_management.yaml#L137-L14314:37
rakhmerovsure14:37
d0ugalrakhmerov: Basically, if there isn't an error we continue. If there is an error we create it.14:38
d0ugalrakhmerov: however, that means we end up with tracebacks in our log, and people debugging often find them and think it is a problem14:38
d0ugalrakhmerov: so, I wonder if there is a way we could have tasks that we expect to fail, but not look scary in the logs :)14:38
rakhmerov:))14:39
rakhmerovwhere exactly do you create an error?14:39
d0ugalrakhmerov: so the error is in the executor log14:40
rakhmerovyes, I mean what's the reason for this?14:40
rakhmerovwhat line in this file should I look at?14:40
d0ugalrakhmerov: okay, I think I explained it badly.14:40
rakhmerovnp14:41
d0ugalrakhmerov: Our task uses the standard openstack actions - we do that to check if a swift container and mistral environment exist14:41
d0ugalrakhmerov: because if they do, we want to stop the workflow, since it will try to recreate them otherwise14:41
*** bobh has quit IRC14:41
d0ugalrakhmerov: so, we use the on-error and on-success to control the workflow.14:42
rakhmerovok14:42
rakhmerovI see14:42
d0ugalrakhmerov: so the first time the container doesn't exist, but the workflow creates it.14:42
d0ugalThen the log contains this: http://paste.openstack.org/show/586207/14:42
rakhmerovwait a sec14:42
d0ugalok :)14:42
rakhmerovjust want to clarify14:42
rakhmerovmistral.environments_get name=<% $.container %>14:42
rakhmerovthis action leads to 'on-error' to trigger if env in Mistral doesn't exist?14:43
rakhmerovright?14:43
d0ugalrakhmerov: correct14:44
rakhmerovyeah, I see, same for swifth14:44
d0ugalrakhmerov: and we want that to happen sometimes, so it is fine14:44
rakhmerovgot it14:44
d0ugalbut it just means the logs look scary14:44
rakhmerovyeah, I see the problem now14:44
d0ugalIt's only a small problem, but everyone debugging issues asks me "is this the problem?"14:44
d0ugalWe could solve it by writing a custom action, but I don't want to do that :)14:45
d0ugalI wonder if there could be a flag or a way to suppress errors or specific errors.14:45
rakhmerovyeah, the obvious solution is just to write a wrapper around these actions14:45
rakhmerovit doesn't require any changes in Mistral code base14:45
d0ugalTrue14:45
d0ugalI guess that is the easy option14:45
rakhmerovno, nothing like this flag actually exists14:46
rakhmerovunfortunately14:46
rakhmerovhm... let me think14:46
rakhmerovthe misunderstanding comes from the fact that we have two types of errors14:47
rakhmerov1) Errors showing that something is wrong with Mistral itself14:47
rakhmerov2) Errors coming from actions14:47
rakhmerovin case #1 it's totally ok to see these errors in log14:48
rakhmerovin case #2 it's probably ok too, BUT.. They could be probably displayed in a different way14:48
d0ugalYeah, it is a tricky problem I think14:49
rakhmerovso that we clearly see that it's an error not related to Mistral setup14:49
d0ugalI don't know if Mistral should be changed for this, I am just starting to think about it14:49
ddejarakhmerov: maybe in case #2 they should be seen as debug log, not error?14:49
d0ugalddeja: but sometimes errors from actions are real errors14:49
d0ugallol, if that makes sense14:49
rakhmerovddeja: that's not a bad idea I think14:50
ddejad0ugal: yes, but they are not related with how mistral operates14:50
d0ugalddeja: True14:50
d0ugalddeja: We already solve that by using multiple log files14:50
ddejaI see14:50
rakhmerovit's not bad in a sense that we should be using a different logging for it14:50
d0ugalwhen you last looked it all went into one, now we have an executor, engine and api14:50
ddejayes14:51
d0ugalexample: http://logs.openstack.org/44/375544/15/check/gate-tripleo-ci-centos-7-undercloud/9b50ad5/logs/var/log/mistral/14:51
d0ugaland that is actually a good example14:51
d0ugalexecutor has two errors I expect, that are fine. The third one is the real issue.14:51
rakhmerovaha14:51
rakhmerovd0ugal: another topic for the design summit? :)14:52
d0ugalYeah, I'll add it :)14:52
rakhmerovok14:52
rakhmerovI don't believe we will solve it this week14:52
d0ugalhaha, no14:52
d0ugalI don't have any urgency here, it would just make my life easier :-D14:52
rakhmerovat the summit I'll tell more thoughts on debugging and usability that are related to this too14:52
rakhmerovit's a part of a bigger picture14:53
rakhmerovyeah, me too14:53
rakhmerovwe have lots of ideas14:53
d0ugalGreat, thanks for the chat about it14:53
rakhmerovin the last month or two, I discussed a lot of them with my colleagues who use Mistral every day14:53
rakhmerovfor example, we have an idea to add a function on API that allows to easily find all objects in ERROR state14:54
rakhmerovfor example, your WF failed14:54
rakhmerovand you don't know how to investigate it14:54
d0ugalthat would be very useful14:55
rakhmerovyou do something like: mistral find-errors <execution_id>14:55
d0ugalYeah, so Heat has a similar command14:55
rakhmerovand it gives all actions and tasks in ERROR in a hierarchical form14:55
d0ugal`openstack stack failures list $STACKNAME`14:55
rakhmerovyes14:55
rakhmerovbtw, one of my colleagues is now working on a YAQL function to do that14:56
rakhmerovas far as I know, close to get it done14:56
rakhmerovbut that's for a slightly different purpose14:57
rakhmerovto be able to see something in a WF itself14:57
*** vishwanathj has quit IRC14:57
rakhmerovddeja: I know you're not going to the summit but I'd like to ask you to add topics that are important for you too14:58
ddejarakhmerov: yeah, sure14:58
rakhmerovinto the etherpad I created14:58
rakhmerovd0ugal, ddeja: this time we have only two work sessions but we will have a contributors meetup on Friday. Hopefully, we can continue there14:59
d0ugalrakhmerov: yeah, that would be good. I need to check what time I leave on Friday, I might not be able to stay for it all14:59
rakhmerovyeah, that's what I'm afraid of :)15:00
d0ugalrakhmerov: it would be good to meetup before the mistral sessions if you have time.15:00
rakhmerovpeople usually leave on Friday15:00
rakhmerovor simply too lazy :)15:00
d0ugallol15:00
rakhmerovd0ugal: absolutely, sir15:00
d0ugalI don't know if you are involved in any other projects?15:00
rakhmerovI would be glad to meet before our official sessions15:01
d0ugalI will be attending some other stuff too15:01
rakhmerovme too, but I'm sure we'll have time15:01
rakhmerovmy list of mandatory sessions/talks is not that huge15:01
d0ugalI'll probably be at Mistral/TripleO/Zaqar sessions15:02
rakhmerovI'll be there probably too15:02
rakhmerovI mean TripleO and Zaqar :)15:02
d0ugaloh, cool!15:03
d0ugalrakhmerov: I'm sure the mistral work sessions were not on friday last time I looked15:12
d0ugalhmm15:12
d0ugalThat causes more conflicts for me I think, damn15:12
*** vishwanathj has joined #openstack-mistral15:12
rakhmerovd0ugal: they are not15:13
rakhmerovwait a sec...15:13
rakhmerovyou scare me15:13
rakhmerovooh, summit schedule web site is down..15:14
d0ugalrakhmerov: http://i.imgur.com/eOUEx05.png15:14
d0ugalI think they were on thursday when I last looked.15:15
rakhmerovooh, yeah15:15
rakhmerovI forgot myself15:15
rakhmerovis it too bad for you?15:15
d0ugalrakhmerov: It's not good :(15:16
d0ugalI'll have to see what I can do15:16
d0ugalbecause I guess we are stuck with it now15:16
d0ugalI had planned everything based on the previous version15:16
rakhmerovf...k15:16
rakhmerovwhat do you mean by previous version?15:16
rakhmerovthere was another version of schedule?15:16
d0ugalrakhmerov: Yeah, I guess I looked when they were still updating it15:17
d0ugalso it is probably my fault15:17
rakhmerovno worries15:18
rakhmerovplease see if you can do something15:18
rakhmerovI would love to have you on those meetings15:18
rakhmerovif you can't be there then we need to meet separately15:18
rakhmerovin any case15:18
*** vinod_ has joined #openstack-mistral15:24
d0ugalrakhmerov: Yup, we should do that anyway15:25
d0ugalrakhmerov: I am looking into it now.15:25
rakhmerovok15:25
vinod_@rakhmerov Hi15:26
vinod_last week I was here asking about Mistral's ability to scale ... I was told you had done some tests on the same ...15:26
vinod_so if you have any notes to share, config options to tweak, gotchas to avoid kind of stuff that would be very helpful15:27
rakhmerovvinod_: give a min, I'll be with you..15:31
vinod_@rakhmerov Sure. Thanks. Shall wait15:34
*** bobh has joined #openstack-mistral15:38
rakhmerovvinod_: sorry, soon :)15:40
*** bobh has quit IRC15:42
rakhmerovvinod_: ok, here15:44
rakhmerovso, the biggest limitation that exists now as far as scaling is that you can't use multiple engines if you run workflows that have "join" tasks15:44
rakhmerovthat's gonna be fixed I believe in about a month15:44
rakhmerovexecutors and apis are totally ok to scale as you need15:45
rakhmerovall components are stateless so scaling is pretty easy15:46
rakhmerovwhat's missing though now monitoring and auto-scaling15:46
rakhmerovif some component dies then Mistral can't rebalance itself and, for example, spin up fresh components15:47
rakhmerovthat's all ahead on a Roadmap15:47
rakhmerovbut honestly, I don't believe it'll be done in Ocata cycle15:47
rakhmerovit's a far future (> 6 months)15:47
rakhmerovas far as config tweaks it's pretty simple: if you use different hosts for different components your config will be probably the same except maybe IPs if you configure them explicitly15:48
rakhmerovlike IP for a web server15:49
rakhmerovotherwise you might want to split your logs15:49
rakhmerovin this case you need to direct logs into different files and you'll have to use either different config files or pass log config properties in your mistral startup commands15:50
rakhmerovas far as numbers, it's kind of hard to provide something certain because it makes sense to talk only about workflows of a certain configuration15:51
vinod_yes, without a benchmark and standard environment numbers wont mean much indeed15:51
rakhmerovfor example, even if we have 2 workflows with 100 tasks their runtime maybe very different depending on their topology15:51
rakhmerovbut15:52
vinod_can u explain what u mean by 'join' tasks?15:52
rakhmerovI can still provide some info15:52
vinod_or point me to some url which explains it?15:52
rakhmerovfor example, I've been experimenting with parallel tasks (simple WF where we just have a number parallel not related tasks) and in the summer such a WF with 500 parallel tasks took minutes to complete on my laptop15:53
rakhmerovnow it takes seconds15:53
rakhmerovsome other types of workflows too15:53
vinod_ok15:54
rakhmerovwhich means that in the last few months we improved performance by ~ 2 orders of magnitude15:54
rakhmerovas far as how, that's a long different story15:54
vinod_great improvement indeed! cheers to your team :-)15:54
rakhmerovtrying hard15:54
rakhmerovok, now on your last question15:54
rakhmerov'join' task is a task that 1) merges multiple parallel routes in a WF 2) works as a distributed mutex and syncs all the incoming routes15:55
rakhmerov#2 means that a 'join' task will only start when all incoming routes will reach it15:56
rakhmerovdocs: http://docs.openstack.org/developer/mistral/dsl/dsl_v2.html#join15:57
vinod_ok, got that ... we might be having only a few such scenarios ...15:57
vinod_At peak loads, Im expecting to have about 50 mistral template executions a minute ... how many tasks that would translate into isnt something I cannot guess right now15:57
rakhmerovthe implementation of 'join' is pretty tricky and it causes a lot of pain with scaling15:57
rakhmerovvinod_: a number of executions (workflows) don't matter too much, yes15:58
rakhmerovmore importantly, how many tasks that will be and what their topologies (configurations) are15:59
rakhmerovwith joins, w/o joins, etc.15:59
rakhmerovparallelism15:59
rakhmerovif you don't have many joins then the Mistral overhead will be pretty small16:00
rakhmerovnew tasks quickly gets scheduled with just a few DB queries required for that16:00
vinod_ok ... would you believe that the code is optimized to use all the cores available? Im just wondering if additional cores can get me better throughput16:03
vinod_im a noob to python and not so comfortable reading a lot of code yet, otherwise I would dig into the codebase than ask this question16:04
rakhmerovhm.. good question16:04
rakhmerovI see, np16:04
rakhmerovso16:04
rakhmerovMistral API component definitely yes16:05
rakhmerovit can use multiple cores16:05
rakhmerovbecause it runs separate processes underneath the surface16:05
rakhmerovMistral Engine and Mistral Executor themselves no16:06
rakhmerovbut you can run many of them on same host16:06
rakhmerovscale them kind of locally16:06
rakhmerovin this case they will be using multiple cores as just being separate processes16:06
rakhmerovvinod_: excuse me, I have to leave now16:08
vinod_i c, so even if I have the API server on a stronger vm with more cores, I will still be constrained by the engine's and executor's throughput ... but atleast I can keep the users happy with a quick response on the UI that way16:08
rakhmerovif you have any other questions, you're very welcome16:08
vinod_thanks a lot @rakhmerov!16:08
rakhmerovwelcome16:08
vinod_we are planning to use Mistral and wanted to know it better before we decide ..ur inputs give me more confidence :-)16:09
vinod_im not sure what time zone you are in. Sorry if its too late in the night already for u16:11
vinod_wish u a pleasant day ahead!16:11
*** vinod_ has quit IRC16:12
*** clenimar has joined #openstack-mistral16:13
*** jtomasek|afk is now known as jtomasek16:22
d0ugalrakhmerov: BTW, I should be fine for both work sessions. Just. :)16:33
*** jpich has quit IRC16:44
*** janki has quit IRC16:55
*** Kiall_ has quit IRC17:33
*** pkoniszewski has quit IRC17:33
*** jistr has quit IRC17:33
*** enykeev_ has quit IRC17:33
*** cargonza has quit IRC17:33
*** EmilienM has quit IRC17:33
*** Kiall has joined #openstack-mistral17:33
*** pkoniszewski has joined #openstack-mistral17:33
*** enykeev has joined #openstack-mistral17:34
*** jistr has joined #openstack-mistral17:34
*** EmilienM has joined #openstack-mistral17:36
*** bobh has joined #openstack-mistral17:39
*** cargonza has joined #openstack-mistral17:44
*** bobh has quit IRC17:44
*** dprince has quit IRC17:52
*** dprince has joined #openstack-mistral18:07
*** bobh has joined #openstack-mistral18:40
*** bobh has quit IRC18:45
*** bobh has joined #openstack-mistral19:57
*** flwang1 has quit IRC19:59
*** flwang1 has joined #openstack-mistral21:01
*** dprince has quit IRC21:28
*** flwang1 has quit IRC21:52
*** flwang1 has joined #openstack-mistral21:52
*** bobh has quit IRC22:15
*** bobh has joined #openstack-mistral23:00
kongrakhmerov, ddeja, please review https://review.openstack.org/#/c/387757, which i think is critical for stable/newton branch23:01
*** bobh has quit IRC23:19
*** bobh has joined #openstack-mistral23:33
*** bobh has quit IRC23:39
*** vishwanathj has quit IRC23:53

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!