14:02:03 <andreykurilin> #startmeeting Rally
14:02:04 <openstack> Meeting started Mon Feb 22 14:02:03 2016 UTC and is due to finish in 60 minutes.  The chair is andreykurilin. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:02:07 <openstack> The meeting name has been set to 'rally'
14:02:30 <andreykurilin> #link https://wiki.openstack.org/wiki/Meetings/Rally#Agenda
14:02:35 <ikhudoshyn> o/
14:02:36 <amaretskiy> o/
14:03:01 <andreykurilin> \o
14:03:42 <andreykurilin> boris-42 will join us a little bit later, so let's start without him :)
14:04:41 <andreykurilin> stpierre, rvasilets : ping
14:04:47 <stpierre> i'm here
14:05:01 <andreykurilin> nice
14:05:47 <andreykurilin> stpierre:  if you do not mind we will skip your first item from agenda for 10-20 minutes, so boris-42 will be able to take part on discussion
14:05:58 <stpierre> sure, i was going to suggest that myself :)
14:06:01 <andreykurilin> :)
14:06:11 <andreykurilin> #topic [andreykurilin] plugins based on entry-points
14:06:43 <andreykurilin> I have an idea how to move our plugins to separate repository
14:06:49 <andreykurilin> with own requirements list
14:07:01 <andreykurilin> proof-of-concept https://review.openstack.org/#/c/282381/
14:07:10 <andreykurilin> It bases on entry-points
14:07:29 <andreykurilin> Such way is used by novaclient's extensions
14:07:45 <andreykurilin> It doesn't require a lot of changes in Rally-side
14:08:00 <andreykurilin> just something like https://review.openstack.org/#/c/282381/5/rally/common/plugin/discover.py
14:08:16 <andreykurilin> folks, what do you think about such type of plugins?
14:09:08 <stpierre> would we need one git repo per set of plugins? (e.g., the openstack plugins would be in one repo, the common plugins in another, etc.?)
14:09:36 <boris-42> stpierre: andreykurilin ok I am ehere
14:09:38 <andreykurilin> I suppose it will be one repo for all openstack plugins
14:09:40 <stpierre> i'm not quite familiar enough with the entry_points magic in setup.cfg -- would we only get to specify one "opts" entry point per repo?
14:10:02 <amaretskiy> andreykurilin: I like the idea very much, but I haven't review this patch yet :(
14:10:38 <andreykurilin> stpierre: plugin package should contains only 2-3 lines - https://github.com/andreykurilin/rally_vm_plugins/blob/master/setup.cfg#L26-L28
14:10:41 <boris-42> stpierre: so basically the idea is to make it simple to keep plugins with some specific requriemnents in separated repo
14:10:47 <stpierre> so we'd need a 'rally-plugins-openstack' repo and a 'rally-plugins-common' repo and, whenever someone starts using rally for something else, a 'rally-plugins-foobaz' repo?
14:11:00 <boris-42> stpierre: so users can do pip istanll rally (and it won't need bunch of system packages)
14:11:07 <stpierre> yeah, i understand the goal
14:11:11 <boris-42> stpierre: after that he can run pip isntall rally-openstack-plugins
14:11:15 <boris-42> stpierre: and magic everything works
14:11:21 <andreykurilin> stpierre: I suppose we will keep comman stuff at the main repo
14:11:21 <boris-42> stpierre: he has plugins for openstack
14:11:28 <boris-42> andreykurilin: yep
14:11:30 <andreykurilin> *common
14:11:47 <boris-42> andreykurilin: core stuff like runners, slas, some context, maybe generalized cleanup mechanims should stay in repo
14:11:52 <stpierre> so would we need a 'rally-plugins-openstack' repo and a 'rally-plugins-common' repo and, whenever someone starts using rally for something else, a 'rally-plugins-foobaz' repo?
14:12:03 <boris-42> stpierre: nope
14:12:16 <stpierre> okay, so we could specify multiple separate entry points for plugins and opts in the setup.cfg?
14:12:16 <boris-42> stpierre: everything that is common goes into rally
14:12:18 <andreykurilin> rally-plugins-commin will be part of rally repo
14:12:30 <stpierre> so would we need rally-plugins-openstack and rally-plugins-foobaz and so on?
14:12:33 <stpierre> all in separate repos?
14:12:34 <boris-42> stpierre: so andreykurilin knows how not to do that
14:12:50 <boris-42> stpierre: yep everything should go in seprated repo
14:12:54 <stpierre> and, similarly: would we only get *one* opts module per repo? the openstack plugins are *big*, with lots of options, and that's a big ugly hard-to-maintain module full of options.
14:13:18 <boris-42> stpierre: so as far as I understand we don't need to touch setup.cfg
14:13:24 <boris-42> andreykurilin: ^ am I right?
14:13:30 <stpierre> i'm not explaining myself well
14:13:37 <andreykurilin> stpierre: we need to touch setup.cfg in plugin repo
14:13:47 <stpierre> https://github.com/andreykurilin/rally_vm_plugins/blob/master/setup.cfg#L26-L28 lists a single 'opts' entry point with all of the options for the vm plugins
14:13:52 <stpierre> that's okay, because there aren't many
14:13:59 <stpierre> but there are *tons* of options for all of the openstack plugins
14:14:08 <stpierre> will we need to squash them all into a single module?
14:14:14 <stpierre> or can we specify multiple opts entry points?
14:14:16 <boris-42> stpierre: so we need to get rid of that stuff
14:14:19 <stpierre> there's a code maintainability issue here
14:14:21 <andreykurilin> stpierre: no
14:14:22 <boris-42> stpierre: in that way
14:14:23 <stpierre> ok
14:14:40 <boris-42> andreykurilin: stpierre as far as I understand each repo has own conf file?
14:14:49 <andreykurilin> I suppose single entry point for opt is required only for sample config generation
14:14:56 <andreykurilin> boris-42: no
14:15:45 <stpierre> this looks and sounds promising, but i'd like to see a much bigger POC. keeping it small shows off the easy parts, but it could be hiding issues that will crop up once we try to cram all bazillionty openstack plugins into it
14:16:18 <boris-42> stpierre: yep yep
14:16:23 <boris-42> stpierre: we are not hurry up with this
14:16:51 <boris-42> stpierre: we are working very slowly =) we moved all plugins under common base and split framework and plugins
14:16:54 <andreykurilin> stpierre: I can try to move all openstack-plugins to separate repo(without tests :) )
14:17:03 <boris-42> stpierre: now we have 2 more problems how to make autodiscovering of plugins
14:17:15 <boris-42> stpierre: and what to do with conf options
14:17:40 <boris-42> stpierre: as well there are few more changes in our framework that I would like to do before split
14:17:50 <boris-42> stpierre: it's rally.task.types and rally.task.validation
14:17:50 <andreykurilin> boris-42: we can make config file per plugin-package, but it looks like not a good idea
14:18:11 <boris-42> andreykurilin: keeping conf options iniside rally is even worse idea
14:18:15 <boris-42> andreykurilin: any other ideas?
14:18:18 <andreykurilin> ok
14:18:46 <boris-42> andreykurilin: I mean we will keep 200 options that are not related to rally code at all and that are not used iniside it
14:18:49 <andreykurilin> so we can have a separate config files. It sounds like easy-stuff
14:19:03 <andreykurilin> *have separate
14:19:48 <andreykurilin> I'll modify my change to support multiconfigs
14:19:59 <andreykurilin> any other questions?
14:20:02 <boris-42> andreykurilin stpierre or we can try to remove oslo.config from plugins
14:20:17 <andreykurilin> hm...
14:20:18 <andreykurilin> no)
14:20:45 <boris-42> andreykurilin: ok then we have only one choice to split conf files
14:20:56 <andreykurilin> boris-42: I suppose we need to use the same base libraries across all plugins
14:21:40 <boris-42> andreykurilin: I was thinking about removing configuration at all in that way
14:21:45 <andreykurilin> :)
14:21:57 <boris-42> andreykurilin: okay we can move to the next topic
14:22:02 <andreykurilin> nice
14:22:13 <andreykurilin> [stpierre] 0.3.0 release was buggy; need more transparency, possibly branching/change freeze to protect soon-to-be-released code
14:22:20 <andreykurilin> #topic [stpierre] 0.3.0 release was buggy; need more transparency, possibly branching/change freeze to protect soon-to-be-released code
14:22:24 <boris-42> stpierre: ^ yep ti was very bad release
14:22:28 <stpierre> yeah
14:22:34 <stpierre> and i have a few questions about the release process
14:22:38 <stpierre> 1. how is the release process initiated and communicated? i had no idea we were close to 0.3.0, and i would have reviewed things differently if i had (e.g., "this looks great, we'll merge it after the release")
14:22:44 <stpierre> 2. why was there no official change freeze to prep and stabilize the release?
14:22:49 <stpierre> 3. do we need a git branching strategy to protect releases? e.g., a few weeks before a release we fork the 0.3.0-prerelease branch and put it into change freeze. that way we can continue to work on master, but we don't merge a bunch of half-tested stuff right before release
14:22:56 <stpierre> </wall-o-text>
14:23:01 <andreykurilin> a lot of questions:)
14:23:24 <andreykurilin> what about small release-cycles?
14:23:30 <boris-42> so
14:23:47 <boris-42> 1. I am trying to cut releases asap so people will get new functionallity sunner
14:23:52 <boris-42> sooner
14:24:32 <boris-42> 2. You should always put +2 on patches if you are ready limitedly to use it in your prod
14:24:50 <boris-42> 2. I don't like feature freezes (it's better to have better CI)
14:24:59 <boris-42> and not merge bad code in master
14:25:09 <stpierre> obviously that didn't work this time :)
14:25:25 <boris-42> stpierre: so something changed
14:25:37 <boris-42> stpierre: usually I was more involved in Rally, however I had some interal work to do
14:25:43 <andreykurilin> "pre-release branches" sounds good
14:25:53 <stpierre> but rally can't depend on you -- that's an antipattern no matter your release schedule
14:25:58 <boris-42> stpierre: plus this thing was not covered by our gates wich is bad
14:26:00 <amaretskiy> release-candidate
14:26:05 <stpierre> yeah, several things weren't
14:26:13 <boris-42> stpierre: yep but second release was ok =)
14:26:31 <andreykurilin> boris-42: noone cares about bug-fixes now :( so pre-release branch with one week ff is not a bad idea. Even more - I like it:)
14:26:31 <stpierre> so if we're dedicated to the idea of small, frequent releases (which is good IMO), how can we gain more confidence in our testing so we don't repeat the 0.3.0 fiasco?
14:26:32 <boris-42> stpierre: so that thing was not tested in gates and it is still not tested
14:27:04 <ikhudoshyn> i support boris-42 in that we'd better have a better CI than a defined release process
14:27:08 <boris-42> stpierre: that is old missing in gates stuff
14:27:10 <andreykurilin> +1 to stpierre
14:27:16 <stpierre> i'm aware of at least four bugs covering at least three bits of functionality, so it's not just one thing
14:27:30 <boris-42> stpierre: it was all related to the same problem
14:27:34 <stpierre> no, it wasn't
14:27:42 <stpierre> lemme find one of the other bugs
14:27:59 <boris-42> stpierre: in any case I don't like Feature freeze it is really useless
14:28:06 <boris-42> andreykurilin: ^
14:28:12 <stpierre> that's fine, but then my last question stands
14:28:16 <boris-42> what we should care is work more on CI/tests
14:28:26 <boris-42> and not merge bad code=(
14:28:28 <andreykurilin> boris-42: https://bugs.launchpad.net/rally the number of opened bugs is not reduced for a long time
14:28:32 <rvasilets> <ikhudoshyn> i support boris-42 in that we'd better have a better CI than a defined release proces +1
14:28:47 <stpierre> here's one of the other bugs that got into 0.3.0: https://review.openstack.org/#/c/281437/
14:28:53 <stpierre> unrelated to the keystone v3 stuff
14:29:17 <stpierre> and even within the keystone authn changes, that broke several different use cases, any of which *could* break independently of the others, and none of which were tested in gates
14:29:27 <andreykurilin> +1 for better CI, but it can't fix everything in short time.
14:29:29 <stpierre> so a single release revealed a bunch of gaps in our coverage
14:29:31 <ikhudoshyn> in fact it would be great to have a dedicated QA activities, yet I'm not sure that there is a good sample ofsuch in any OpenSource project
14:30:08 <stpierre> and just saying "we need better CI" i think glosses over the scope of the issue here -- we're depending on our CI absolutely, but it clearly misses enough that we should be very concerned about our quality
14:30:20 <breton> what got broken in keystone?
14:30:27 <boris-42> stpierre: +1
14:30:34 <boris-42> breton: no worries it's rally=)
14:30:48 <andreykurilin> breton: wrong usga of keystoneclient
14:30:52 <andreykurilin> *usage
14:31:01 <boris-42> stpierre: so about that issue with kesytone testing
14:31:08 <boris-42> stpierre: we have long long running work on that
14:31:22 <boris-42> stpierre: however it didn't finished
14:31:56 <stpierre> let me ask this a different way: what is our plan to improve release quality? where is it documented? what is its timeline? and how will it be measured?
14:32:42 <andreykurilin> +1 for page with release scheduling
14:33:19 <andreykurilin> +1 for short releases(2-4 weeks), so users will be able to get latest features quicker
14:33:26 <boris-42> stpierre: so until we improve gates and cover all the ways of atunetication of keystone we will face this issue
14:33:36 <andreykurilin> +1 for 2-5 days ff, before release
14:33:44 <boris-42> stpierre: nobody is able to test 300 plugins in 10 different envs
14:33:48 <boris-42> stpierre: by hands
14:34:02 <boris-42> stpierre: so my proposal is to restore redixin patch
14:34:18 <stpierre> which patch?
14:34:24 <boris-42> stpierre: that allows us to test this (like run many tasks in single dsvm job) with different rally deployment
14:34:32 <boris-42> stpierre: let me find it
14:34:36 <boris-42> stpierre: it's very old
14:34:42 <stpierre> that seems like a specific solution to a general problem
14:34:51 <boris-42> stpierre: https://review.openstack.org/#/c/163785/
14:35:11 <andreykurilin> #link https://review.openstack.org/#/c/163785/
14:35:13 <boris-42> stpierre: so we have a lot of places where we don't test everything and we can
14:35:29 <boris-42> stpierre: especially with new functioanllity like multi API version supports
14:35:56 <boris-42> stpierre: however i don't believe in feature freeze in our case, and I do believe that we need to sit down and start fixing bugs, until we get <30 open bugs
14:36:10 <stpierre> so i guess i'm surmising that our plan to improve release quality is to just continue trying to increase CI coverage, but there's no documentation, timeline, or measureables?
14:36:10 <boris-42> stpierre: plus yep we can make release model more transparent
14:36:28 <boris-42> stpierre: documentation and timeline should be done
14:36:35 <boris-42> stpierre: I was trying few time to introcude this
14:36:37 <andreykurilin> boris-42: but we need to start fixing bugs sometimes...
14:36:51 <boris-42> andreykurilin: nope we don't need to start
14:36:56 <boris-42> andreykurilin: we need to fix them and always
14:37:08 <boris-42> andreykurilin: no matter is it begging of release or end of release or something else
14:37:24 <andreykurilin> but most of developers just introduce new features and new bugs:)
14:37:26 <boris-42> stpierre: so lets' just set releases each 3 weeks
14:37:37 <boris-42> stpierre: and we will merge them no matter what
14:37:45 <boris-42> cut them*
14:37:51 <stpierre> sounds good. having a known schedule will be helpful
14:38:05 <boris-42> stpierre: okay and it will be quite simple and not depend on feature set that we have
14:38:26 <boris-42> stpierre: because I failed to do that when we have short term releases
14:39:04 <boris-42> stpierre: I am probably going to update this page https://docs.google.com/spreadsheets/d/16DXpfbqvlzMFaqaXAcJsBzzpowb_XpymaK2aFY2gA2g/edit#gid=1993147046
14:41:29 <andreykurilin> let's move to the next topic?
14:41:40 <boris-42> andreykurilin: aure
14:42:01 <andreykurilin> #topic [andreykurilin] Using *.openstack.org URLs - https://review.openstack.org/#/c/282646
14:42:36 <andreykurilin> AJaeger proposed a change for changing links for docs(from readthedocs to docs.openstack.org) and for git repo
14:42:46 <andreykurilin> #link http://docs.openstack.org/developer/rally/
14:42:55 <andreykurilin> #link https://git.openstack.org/cgit/openstack/rally
14:43:16 <boris-42> so I like more read the docs theme lol
14:43:23 <andreykurilin> yeah...
14:43:38 <andreykurilin> but docs.openstack.org can be configured too
14:44:10 <andreykurilin> also, each new merged patch updates it
14:44:31 <amaretskiy> docs looks good for me but github UI is better than git.penstack.org
14:44:43 <andreykurilin> while readthedocs should be updated manually(or we should write own job for it)
14:45:20 <rvasilets> I don't like this change at all
14:45:22 <boris-42> andreykurilin: lol
14:45:25 <andreykurilin> amaretskiy: but github is just a mirror for git.openstack.org. Also, if infra-scripts will be broken, github will be not updated
14:45:44 <rvasilets> github and readthedocs is more cute
14:46:19 <amaretskiy> we can use git.openstack.org and additional link to github beside
14:46:24 <ikhudoshyn> yep, and they are both more trueЪ
14:46:26 <andreykurilin> what about proposing both openstack git and github
14:46:32 <andreykurilin> :)
14:46:44 <amaretskiy> andreykurilin: +1
14:46:55 <ikhudoshyn> andreykurilin: proposing?
14:47:40 <andreykurilin> ikhudoshyn: write something like "Git repo: git.openstack.org or it's github mirror: github"
14:47:43 <andreykurilin> in the docs
14:47:58 <ikhudoshyn> ic
14:48:49 <ikhudoshyn> as for github vs git.os it's not a big deal for me -- we could give a link to the latter but use the former
14:49:03 <andreykurilin> ok
14:49:22 <boris-42> offtopic stpierre: I updated release page
14:49:30 <andreykurilin> it looks like there is no disagreements. what about docs.openstack.org ?)
14:49:31 <boris-42> stpierre: so now it's clear when to expect new release
14:49:32 <ikhudoshyn> but it's more about politics, I think
14:49:39 <stpierre> boris-42: thanks
14:49:49 <boris-42> stpierre: however we need to fix CI =)
14:49:59 <ikhudoshyn> if we're to split from OS, why we'd use these resources)
14:50:01 <boris-42> stpierre: othewise just clear release schedule won't help us
14:50:40 <ikhudoshyn> as for docs -- we're kinda have to eat our own dog food
14:50:58 <andreykurilin> :)
14:51:04 <andreykurilin> let's move to the last topic
14:51:22 <andreykurilin> stpierre: your turn, again:)
14:51:30 <andreykurilin> #topic [stpierre] keystone v3 support
14:51:31 <ikhudoshyn> if we move (officially) to docs.os we'd make sure our docs look pretty there with that theme (which I agree, is uglier than readthedocs)
14:51:41 <boris-42> uh that one is pain
14:51:56 <andreykurilin> ikhudishyn: agreed
14:51:56 <boris-42> stpierre: so did you see document with rally brainstorm
14:52:00 <stpierre> yes
14:52:19 <boris-42> stpierre: so the idea is that services utils refactoring should help us a lot with that
14:52:22 <stpierre> the lack of keystone v3 is really starting to hurt us. one of the patches i linked is just the second attempt at doing keystone auth properly; the other is much bigger. just want some eyes on both
14:52:25 <boris-42> stpierre: especially with atomic actions
14:53:12 <boris-42> stpierre: what patches ?
14:53:23 <stpierre> that's great, but the services refactoring spec still hasn't been updated in six months, and we can't realistically wait for it to get updated, iterated on, approved, and then work started. i think julian's approach represents a good forward-looking middle ground that doesn't abuse wrappers too much, and it's something we can work on merging soon
14:53:27 <andreykurilin> #link https://review.openstack.org/#/c/282918
14:53:34 <andreykurilin> #link https://review.openstack.org/#/c/277225
14:53:36 <stpierre> thanks
14:53:37 <andreykurilin> boris-42 ^
14:53:58 <andreykurilin> so, imo, we need to be careful with them:)
14:54:02 <boris-42> stpierre: so basically I am going to update it very soon=)
14:54:05 <stpierre> the second one there has already had several people look at it (thanks!)
14:54:14 <boris-42> stpierre: maybe even tomorrow
14:54:18 <boris-42> stpierre: today*
14:54:23 <stpierre> boris-42: that's the third time you've said that in the past six weeks :P
14:54:25 <boris-42> stpierre: after next my meeting
14:54:33 <boris-42> stpierre: so but we did brainstorm
14:54:41 <boris-42> stpierre: and now it's really clear what to do
14:54:44 <stpierre> but updated != merged, and i suspect it'll take a while before that happens
14:54:48 <boris-42> stpierre: I just need to copy paste that text in spec
14:54:50 <andreykurilin> stpierre: :D
14:55:01 <boris-42> stpierre: so if you are ok with that approach
14:55:14 <boris-42> stpierre: that is described in doc then it will be merged soon
14:55:58 <boris-42> stpierre: so I will upload and wait for your vote
14:56:05 <boris-42> stpierre: I will upload it in 2 hours from now
14:56:18 <stpierre> i still don't think that's really reasonable. i'm not optimistic that "merged soon" will happen any time in the next week or two, and there's still a bunch of refactoring that we'll need to depend on before we can completely refactor julian's patch
14:56:44 <stpierre> at times rally errs too much on the side of "do it right" over "do it", and this is becoming a pain point for us :/
14:57:09 <andreykurilin> +1 to stpierre
14:57:18 <andreykurilin> fully agreed
14:57:21 <boris-42> stpierre: andreykurilin so guys you won't stable releases or features?)
14:57:26 <boris-42> want*
14:57:49 <boris-42> even if we are trying to merge everything slow and doing it right we are getting bugs and broken releases
14:58:01 <andreykurilin> boris-42: you said that we will have a good CI in near future. lol
14:58:04 <boris-42> I am personally ok with manual testing Julien patch
14:58:20 <boris-42> and checking that it work in background working on CI
14:58:22 <boris-42> stpierre: ^
14:58:31 <stpierre> ok, that's great
14:58:39 <stpierre> i fully acknowledge that we need to refactor
14:58:45 <stpierre> i just don't think we should block other work on refactoring
14:59:59 <boris-42> stpierre: let's get merged today/tomorrow spec
15:00:10 <boris-42> stpierre: that will be good balance between do it and do it right
15:00:14 <andreykurilin> boris-42: previously, we did stable work quicker...
15:00:18 <andreykurilin> ;(
15:01:24 <boris-42> andreykurilin: so previously I was younger lol
15:01:28 <andreykurilin> :D
15:01:39 <andreykurilin> you need to increase a number in you nickname
15:01:41 <andreykurilin> lol
15:01:57 <andreykurilin> folks, we don't have more time for a meeting today
15:02:05 <andreykurilin> thanks everyone
15:02:16 <andreykurilin> #endmeeting