17:30:35 <aspiers> #startmeeting self-healing
17:30:36 <openstack> Meeting started Wed Aug 14 17:30:35 2019 UTC and is due to finish in 60 minutes.  The chair is aspiers. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:30:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:30:39 <openstack> The meeting name has been set to 'self_healing'
17:30:43 <aspiers> #topic automated testing
17:30:52 <aspiers> ihti: anything you want to bring up?
17:31:12 <aspiers> for the minutes ...
17:31:34 <aspiers> #link https://review.opendev.org/#/c/675138/ adds a list of testing frameworks to the SIG repo
17:31:35 <ihti> not much from my side yet, playing around with Rally on a dev setup
17:31:38 <aspiers> cool
17:31:48 <aspiers> the matrix clearly shows an awful lot of overlap
17:32:05 <aspiers> it occurred to me that we should try to get all the maintainers of the various tools to review that matrix
17:32:20 <ihti> yes, it have, rally covering most of the features
17:32:27 <aspiers> as they may be able to offer extra info
17:33:00 <ihti> yes, that would be very helpful with and would make the whole list more accurate
17:33:13 <aspiers> #link https://logs.opendev.org/38/675138/4/check/openstack-tox-docs/344de18/html/testing/tools-list.html Zuul build of the new doc
17:33:41 <aspiers> Yardstick + Doctor also seems to cover everything
17:34:26 <ihti> +NFV bench
17:34:30 <aspiers> ihti: Oh I just noticed, Eris is not in there yet
17:34:37 <aspiers> I guess that's intentional?
17:34:43 <ihti> yes
17:34:58 <ihti> as it will be worked on so i thought not include it yet?
17:35:04 <ihti> not sure
17:35:19 <aspiers> I think it would be better to include it, but add a new column "maturity"
17:35:43 <aspiers> e.g. unfinished/alpha/beta/immature/mature
17:36:04 <ihti> that's good idea.
17:36:08 <aspiers> also maybe another column for "OpenStack-specific"
17:36:16 <aspiers> since tools like Jepsen are agnostic
17:37:15 <aspiers> and there is no column for performance testing, unless that is under "Scalability testing"?
17:37:24 <aspiers> but I think they are different things
17:37:34 <aspiers> so separate columns would be useful
17:37:37 <ihti> 'Openstack-specific' one I added but then removed it afterwards, as I jepsen was the only one falling in that category
17:37:47 <aspiers> Hmm
17:38:40 <aspiers> OK, so maybe change the Jepsen cell to read "Jepsen (not specific to OpenStack)"
17:39:16 <aspiers> Another suggestion: add a separate column for reliability testing
17:39:38 <ihti> For performance I thought it overlaps with 'data plane testing'? I could be off
17:39:38 <aspiers> since the ability to inject disruptions / failures is separate from the ability to test resilience against them
17:39:58 <aspiers> I think we need performance testing for control plane operations oto
17:39:59 <aspiers> too
17:40:01 <ihti> okay I will mention that for jepsen
17:40:22 <aspiers> so no, I would consider performance testing somewhat separate to data plane
17:40:40 <aspiers> control plane operations can be slow
17:40:41 <ihti> alright
17:40:44 <aspiers> especially things like failover
17:41:02 <aspiers> Sorry, I am just thinking of this new feedback now :)
17:41:36 <ihti> no worries, I appreciate it :)
17:41:43 <aspiers> Another column we need is around the level of community activity
17:41:49 <aspiers> Some of these tools are dead
17:41:54 <aspiers> That is an important datapoint
17:41:56 <ihti> alright, yes that's true
17:42:04 <aspiers> Other tools are not dead, but only have a few contributors
17:42:07 <ihti> not update for a couple of years
17:42:25 <aspiers> so a column "size of active community" would help
17:42:40 <aspiers> but maybe it's not just size, but also level of activity
17:43:12 <aspiers> e.g. 2 contributors could do total of 500 commits in a month, or 20 contributors could do total of 10 commits in a month ...
17:43:39 <aspiers> These are important considerations when designing a full solution
17:45:13 <ihti> that makes sense and can be easily gathered by having a look at repos
17:45:48 <ekcs> yea maybe instead of putting up a static “grade” we can just link to the repo activities page.
17:47:03 <aspiers> well it's useful to be able to compare them all at a glance
17:47:09 <aspiers> that's kind of the point of this matrix
17:47:28 <aspiers> any cell could change, but that's OK :)
17:47:29 <ekcs> yea that makes sense too.
17:48:13 <ihti> I can add both a link and value
17:48:17 <aspiers> perfect!
17:48:18 <ihti> to each sell
17:48:21 <ihti> *cell
17:48:28 <aspiers> that would be great
17:48:39 <aspiers> sorry, so it seems I asked for like 59 new columns :)
17:49:14 <ihti> I think its good and makes the decision making process much easier
17:49:21 <aspiers> yes
17:49:52 <aspiers> maybe once you've updated the matrix we can try to contact the communities of the various tools for feedback
17:50:03 <aspiers> some of them are already cc'd on the review
17:50:16 <aspiers> but we'll need to find others probably
17:50:46 <aspiers> ihti, ekcs: OK anything else on this topic?
17:50:51 <ihti> cool
17:50:55 <ekcs> great
17:50:55 <ihti> no
17:51:05 <ihti> no topic from my side
17:51:11 <aspiers> so we can move on, or end the meeting
17:51:23 <aspiers> ekcs: anything to discuss about Shanghai at this point, or anything else?
17:51:38 <aspiers> ihti: thanks a LOT for all your great work so far!
17:51:45 <aspiers> it's hugely appreciated
17:51:50 <ekcs> not really. my focus now is just to find time to complete this task finally: https://storyboard.openstack.org/#!/story/2005641
17:52:02 <ihti> thanks for the reviews and feedback :)
17:52:09 <ekcs> thank you ihti !
17:52:20 <aspiers> ekcs: oh nice, that would be great!!
17:52:39 <aspiers> ihti: BTW soon we should email openstack-discuss announcing this new doc
17:52:46 <aspiers> maybe we can announce after merging
17:53:30 <aspiers> OK, sounds like we are done for today
17:53:35 <ihti> yes, after reviews by the tool devs is done, it would be good idea to spread it out
17:53:41 <aspiers> yup
17:53:56 <aspiers> it will help restart the discussion around how to fix this
17:54:28 <aspiers> ihti: don't hesitate to ask me if you have any questions or need support on anything
17:55:09 <aspiers> see you both online, bye for now! o/
17:55:09 <ihti> aspiers, sure i will, thanks very much till now :)
17:55:21 <aspiers> :)
17:55:24 <ekcs> bye!
17:55:31 <aspiers> #endmeeting