16:00:07 #startmeeting oslo 16:00:08 Meeting started Mon Dec 8 16:00:07 2014 UTC and is due to finish in 60 minutes. The chair is dhellmann. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:11 The meeting name has been set to 'oslo' 16:00:13 o/ 16:00:21 our agenda: 16:00:22 o/ 16:00:23 #link https://wiki.openstack.org/wiki/Meetings/Oslo 16:00:33 o/ 16:00:57 o/ 16:00:58 o/ 16:01:37 jd__, dims, flaper87: ping? 16:01:47 bonjour ! 16:01:50 dhellmann: pong 16:01:52 IIRC, bnemec is out this week 16:01:58 o/ 16:01:58 o/ 16:01:59 y 16:02:11 hi 16:02:19 let's get started while people trickle in 16:02:23 #topic Review action items from previous meeting 16:02:34 * dhellmann prepares paste-bomb 16:02:41 We released a bunch of libraries last week: 16:02:41 #info dhellmann to release cliff DONE 16:02:41 #info bnemec to release oslo.concurrency DONE 16:02:41 #info dims to release oslo.utils DONE 16:02:41 #info dhellmann to release oslo.config DONE 16:02:42 #info sileht to release oslo.messaging DONE 16:02:43 #info viktors to release oslo.db DONE 16:02:44 #info dims to release oslo.1i8n DONE 16:02:45 #info jd__ to release oslo.middleware DONE 16:02:47 #info dhellmann to release oslo.rootwrap (or talk to ttx about doing it) DONE 16:02:48 #info bnemec to release oslo.serialization DONE 16:02:49 #info dims to release oslo.vmware DONE 16:02:50 #info dhellmann to release oslosphinx DONE 16:02:51 #info dhellmann to release oslotest DONE 16:02:58 we also approved 3 more specs: 16:02:59 #info dhellmann approve spec https://review.openstack.org/#/c/127532/ DONE 16:03:00 #info dhellmann approve spec https://review.openstack.org/#/c/131666/ DONE 16:03:01 #info dhellmann approve spec https://review.openstack.org/#/c/124847/ DONE 16:03:05 we have a few items on the list I'm not sure we did: 16:03:14 #info sileht talk to devstack team about updating to not install oslo from source in stable branches 16:03:21 oh, actually that one is done, it just wasn't marked that way 16:03:28 dhellmann, this is done since a while 16:03:41 we do have some stable tests that install from the stable branches of oslo libs 16:04:05 there was an email about changing our stable branch handling policy, as part of that work 16:04:11 #info harlowja to release taskflow 16:04:19 harlowja_at_home: ping? 16:04:24 dhellmann, yo yo 16:04:32 where do we stand on a taskflow release? 16:04:56 still got a few reviews that need to get merged before i think thats ready; will poke some people today to see if i can get them to do that 16:05:22 harlowja_at_home: ok, we have a lot of unreleased changes, and the longer that list gets the more nervous I get 16:05:34 ya, i know 16:05:37 remember, the projects are not running their tests against our master source trees any more 16:05:41 only we run those tests 16:06:08 two more that I need to do this week 16:06:09 #action dhellmann update release instructions after https://review.openstack.org/#/c/136401/ is merged 16:06:09 #action dhellmann make sure instructions for creating a new library are up to date with stable requirements jobs 16:06:34 I spent time with the infra team to add a guide for creating new projects to their manual that is replacing their wiki pages 16:06:49 next up is to modify our directions to point to them at the appropriate place 16:06:55 (s) 16:07:10 I think that's it from last week; did I miss anything? 16:07:48 moving on then 16:07:49 #topic backwards compatibility concerns 16:07:55 We had a couple of issues with libraries last week breaking projects. 16:08:02 First, we had an import issue in oslo.concurrency, which turned out to need a fix in hacking so we didn't break the pep8 rules in other projects. 16:08:16 I'm going to be testing my namespace package changes under pep8 in a few of the projects before releasing them (the ones up for review have been tested). 16:08:25 Second, we removed a config option from oslo.messaging that we thought only we were using. 16:08:32 This was pretty easy to fix, but I want to stress that we need to be more careful in the future. 16:08:41 As we have more and more libraries, we will be working on code that can break the other projects more directly and immediately. 16:08:43 y that one was weird 16:08:49 We cannot assume that no one uses a feature without checking *every* repository. 16:09:01 We have to be extremely cautious about removing things as well (the uuidutils functions come to mind) 16:09:10 We need to follow a deprecation period, and publicize changes with our liaisons, who will help us make sure nothing breaks downstream. 16:09:51 It's great when we can remove dead code, and clean up APIs, so we should keep doing that. But we need to take care so we don't block other developers in the process. 16:10:11 This is also why we need to be doing frequent releases -- so that changes that break unit tests are caught early while they are still fresh in our minds. 16:10:22 questions or comments? 16:10:25 dhellmann: did we make any headway on updating global requirements? (when and if we can bump up versions)? 16:10:48 dims: we're going to bump versions when the libraries include features that applications need, not just when we create new releases 16:11:14 dhellmann: who gets to decide that? repository cores or us? 16:11:20 dhellmann, Maybe just that removing dead code should be low priority, if it doesn't by anything else 16:11:23 we have to ensure that the allowed ranges for the previous stable branch and master overlap, to allow rolling updates 16:11:31 s/by/buy 16:11:44 I think projects will bump on demand 16:11:49 dims: well, we need to make the case as new libs are released and features are adopted/required 16:12:13 dhellmann: k thanks 16:12:16 dims: if an old version of a library does work with master of all consumers, then the min can stay lower than latest 16:12:45 I know that's a bit frustrating, but we have to balance our needs with everyone else. 16:13:15 dhellmann: i hear you. the blanket -2's got to me :) 16:13:19 therve: yes, I don't know that we're prioritizing removing dead code specifically 16:13:39 Cool 16:13:39 dims: yeah, there was a bit of a "let's not change anything until we know what's going on" reaction to the fact that we had to uncap the stable branches last week, too 16:13:49 we had a confluence of several sorts of failures 16:13:55 dhellmann: understood 16:14:00 just venting :) 16:14:03 I think that will loosen up again now 16:14:06 totally understand 16:14:35 ok, I think we've said all that needs to be said there -- let me know if you have questions 16:14:42 #topic Red flags for/from liaisons 16:14:43 I have two items to mention this week, before we get reports: 16:14:51 We had a couple of tests in nova that were mocking parts of oslo.concurrency instead of using fixtures. 16:14:58 I don't know if those were actually implementation details, but regardless I would like for liaisons to scan the unit tests in your projects quickly for any that are mocking anything in an oslo library. 16:15:06 We want to make sure we have fixtures to to cover those cases, so if you find something that is being mocked we should look at it to see if that's better than using a fixture. 16:15:35 dhellmann: an example of bad nova test maybe? 16:15:35 let's use an etherpad to make a list: 16:15:37 #link https://etherpad.openstack.org/p/oslo-mocks-in-project-unit-tests 16:15:43 ihrachyshka: sure, let me find that review 16:16:02 #link https://review.openstack.org/#/c/138463/ 16:16:21 dhellmann, Heat does stuff on oslo.config for sure. 16:16:49 I think that's a public function, so I think that would normally be legitimate, but the lock stuff is very tricky to mock properly 16:17:03 I need to admit that neutron test suite is huge, so it will take time 16:17:15 I don't remember if the failure was a change to that function's API or what. Like I said, this made me think of it, but may not be the best example. 16:17:23 yes, this is going to be a long term project 16:17:30 therve: that's my second point :-) 16:17:37 The other thing that came up was the use of configuration options in unit tests. 16:17:49 This was what broke most of the projects when we released oslo.messaging with the option removed. 16:17:54 Strictly speaking, the config options are not part of the "API" of the library, and shouldn't be used in tests if we can avoid it. 16:18:01 We won't be able to avoid it, for now, but that's another thing to look for in the scan. 16:18:07 We want to create fixtures or other APIs for setting up configuration objects instead of using them directly. 16:18:15 This will take more thought and time, but we need your help to collect the data about what sort of config options are used now. 16:18:27 it seems like nothing is testing with the changes in olso libs? why wasn't it caught in the check tests? 16:18:34 the idea is to create APIs we can support for unit testing users of the libraries 16:18:53 bknudson: we do not run the unit or pep8 tests of other projects against the sources of the libraries 16:18:56 I guess that would require running nova unit tests with the oslo change. 16:19:14 we have never had test configurations like that, and decided not to add them because of the number of test resources that would be required 16:19:32 dhellmann: so basically looking for things like set_override/__getattr__ for oslo stuff? 16:19:38 this wasn't a problem when we had the code in the incubator, because when a sync was done the unit tests ran 16:19:50 ihrachyshka: yes 16:20:16 dhellmann: hm, how come that pep8 validation for oslo libraries influence consumers? 16:20:20 ihrachyshka: I don't yet have a good answer to what we will do to remove them, but knowing what sorts of cases we have will help us figure out what to change 16:20:57 ihrachyshka: that was a fun one. When I moved a module from oslo.concurrency to oslo_concurrency, I created some import hacks in oslo.concurrency so you could import the same names. However, hacking did not recognize that the things being hacked were actually modules. 16:21:15 So the pep8 rule for "you must only import modules" failed 16:21:26 aha, 'roger that' on both points 16:21:40 #link https://review.openstack.org/#/c/138462/ 16:22:19 I will send an email to the ML documenting this request for review 16:22:23 #action dhellmann send email asking liaisons to prepare list of potential fixtures to be added to libraries and config options used by tests 16:22:38 are there any other questions now, so I can include more detail in that message? 16:23:10 if not, let's hear from the liaisons about any issues in the projects 16:23:42 dhellmann: the only thing for neutron is oslo.concurrency 16:23:52 and it has a bunch of stuff to push/fix before migration 16:24:03 so you're still working on adoption there? 16:24:23 first, oslo-incubator cleanup: https://review.openstack.org/122796 16:24:31 Nothing to complain about in Heat :). Still a bit of catching up to do. 16:24:58 second, there is a bug in oslo.concurrency that blocks neutron way of setting $lock_path: https://bugs.launchpad.net/oslo.config/+bug/1399897 16:25:00 Launchpad bug 1399897 in oslo.config "Config filter used for lockutils makes it impossible to use string substitution" [Undecided,New] 16:25:09 ihrachyshka: I'll add that to our review priority list at the end of the meeting 16:25:35 dims knows the details on the bug, but basically, oslo.config's ConfigFilter does not play nice with substitutions 16:25:49 and in neutron, we set lock_path = $state_path/lock, and it blows up 16:26:10 where does state_path come from? 16:26:27 dhellmann: from global cfg.CONF object (so not available in lock_path extraction context) 16:26:28 i've updated the bug with a test case dhellmann 16:26:42 ihrachyshka: is state_path defined by neutron or another oslo lib? 16:26:44 since lock_path is defined on ConfigFilter and not cfg.CONF 16:26:57 dhellmann: state_path?.. I think neutron, but let me check 16:27:09 yeah, right, in neutron/common/config.py 16:27:11 dhellmann: its easy to recreate 16:27:28 http://paste.openstack.org/show/147086/ 16:27:45 dhellmann: my idea is to drop ConfigFilter usage from lockutils, and release a new version 16:27:55 this will require another release and another version bump though 16:28:18 ihrachyshka: yeah, we *want* the options defined by the library to be hidden 16:28:29 other than that, there are no more libraries for neutron to consume, so eagerly waiting for more to come (nah, joking) 16:28:39 we may need to modify the filter to allow it to see options in the master namespace 16:29:08 I thought the idea was to prevent anything else from seeing into the options used by the lib, but not necessarily the other way around 16:29:31 ok, we'll talk about that more after the meeting 16:29:38 sounds good 16:29:39 are there any other liaisons with reports? 16:30:32 ok, moving on 16:30:34 #topic Moving the hacking project to QA program 16:30:34 #link https://review.openstack.org/#/c/138499/ 16:30:34 #link http://lists.openstack.org/pipermail/openstack-dev/2014-December/052025.html 16:30:51 I know keystone code did some overriding (mock) of some time functions, but it seemed like those were designed to be overridden 16:30:53 last week when we had to scramble to find someone who could release the hacking project, I started wondering why it was part of Oslo and not QA 16:31:23 bknudson: yes, those were, but we should still list them so (a) we don't remove those features and (b) in case we can come up with a better API 16:31:33 seems to make sense to me; joe gordon is the main guys on it i think 16:32:04 yeah, jogo is the maintainer for hacking. Several of us have been reviewers, but it feels like a QA tool. 16:32:40 seems ok to me 16:32:42 I talked to jogo and mtreinish about moving it, and they agreed. All of the oslo-core reviewers are invited to stay on as reviewers after the move, but I think the plan it to remove the oslo-core group 16:33:26 are there any objections? 16:33:41 +1 16:34:03 more project changes: 16:34:07 #topic tooz adoption under way 16:34:07 the tooz repository was renamed, so we need to add it to our list of projects to review now 16:34:08 can I get a volunteer to update the dashboard builder file and review links in the wiki? 16:34:12 * dhellmann looks at jd__ 16:34:29 dhellmann: sure 16:34:43 I don't know what the dashboard builder file though 16:35:05 #action jd__ to update gerrit dashboard config for oslo and review links in the oslo wiki to add tooz and remove hacking 16:35:07 only other question for that adoption is the 1 requirement of tooz that isn't in global requirements, i'm not sure if it will cause an issue (will it be gating against the requirements repo?) 16:35:12 jd__: just a sec 16:35:55 that being pymemcache (the better memcache library) 16:36:01 harlowja_at_home: good question, let's see if it explodes? 16:36:10 jd__: http://git.openstack.org/cgit/stackforge/gerrit-dash-creator 16:36:26 jd__: there are a couple of oslo-related files in http://git.openstack.org/cgit/stackforge/gerrit-dash-creator/tree/dashboards 16:36:37 jd__, ok dokie, guess that will start the fun around why 2 libraries for memcache then :-/ 16:36:45 dhellmann: ok 16:36:54 harlowja_at_home: yes, as an oslo library it will need to be gating against the global requirements 16:36:55 harlowja_at_home: because the first one is utter crap? 16:37:15 jd__, the choir agrees ;) 16:37:25 sold! :) 16:37:28 if we're going to have tooz and oslo.cache providing abstractions for memcache, we can update both of them to use the same lib and stop using the old one 16:37:37 dhellmann: +1 16:37:45 however, this is another example of a thing where we need to be careful about rolling out the change 16:37:58 dogpile will be using the pymemcache one (oslo.cache -> dogpile someday?) 16:38:19 harlowja_at_home: yeah, oslo.cache is just a front-end for dogpile that knows about oslo.config 16:38:37 if dogpile is using pymemcache that's great news indeed 16:38:55 jd__: we should put together a list of any projects using the old library so we can make sure it will be possible to migrate them 16:39:14 s/we/you :-) 16:39:20 I can volunteer 16:39:24 k 16:39:33 I'll do a list for next week 16:39:40 ok, good 16:39:58 https://bitbucket.org/zzzeek/dogpile.cache/src/345d07d5b740c4aca3f16ce34e6d9316fcf39b55/dogpile/cache/backends/memcached.py?at=master#cl-243 :( 16:40:01 jd__, ^ 16:40:04 #action jd__ prepare a list of projects using memcache libraries directly that will need to be migrated if we recommend a new memcache lib 16:40:25 once we have the list, we can figure out how much publicity and documentation needs to be prepared before any changes happen 16:40:41 harlowja_at_home: ah, some work to do then :) 16:41:11 in the mean time, everyone please take a look at the tooz docs and code so you can become familiar enough to help with reviews 16:41:23 if the code sucks, don't blame me! 16:41:24 haha 16:41:37 jd__: is the blueprint up to date with the list of remaining tasks to be done before we can mark it "implemented"? 16:42:14 dhellmann: I think we're good! 16:42:19 jd__: ok, thanks 16:42:23 #topic Sprint retrospective 16:42:23 #link https://etherpad.openstack.org/p/oslo-kilo-sprint 16:42:24 #link http://lists.openstack.org/pipermail/openstack-dev/2014-December/052316.html 16:42:32 thank you all for your help with the sprint last week 16:42:41 we made good progress on a lot of our smaller libraries 16:42:54 we do still have quite a backlog on the big 3: oslo.db, oslo.messaging, and taskflow 16:43:15 As I said in my email, I think we should set up 1-day sprints to focus on each of those libraries 16:43:24 +1 to that; i think its a great idea 16:43:25 probably because they're not critical :-) 16:43:52 :D 16:44:02 I'm going to leave that up to the lib maintainers to organize, in part because I think we'll need a lot of help understanding which reviews are important and having intros to the code, etc. 16:44:19 We should coordinate them here in this meeting, so we don't schedule them too close together 16:44:43 and we don't need to be any more formal than the sprint we had, so the actual planning is just publicity and doing that preparation work 16:44:52 who would like to go first? :-) 16:45:03 i can :-P 16:45:06 * dhellmann looks at viktors, sileht, and harlowja_at_home 16:45:11 harlowja_at_home: ok, sounds good 16:45:37 do you want to work out a date to announce at next week's meeting? 16:45:52 sure, although everyone probably gonna be on vacation soon, but sure 16:46:13 all that xmas stuff everywhere 16:46:21 yeah, consider that as well as the timezone spread of the team (i.e., not monday or friday) 16:46:50 yes, so we should probably move quickly or wait until january 16:46:54 agreed 16:47:14 and you don't have to wait to announce until next week, but you should probably give us enough notice to make sure we can spend the day on it 16:47:21 i'm fine with something next like wednesday 16:47:28 *not this wed (to early notice probably) 16:47:29 we might have to do a bit of date negotiation, so let's do that on the ML 16:47:32 kk 16:47:34 right, that makes sense 16:47:52 anyway, i'll send something out 16:47:52 #action harlowja_at_home to plan a date for a taskflow review sprint 16:47:57 cool, thanks 16:48:12 #topic Ongoing work & Review priorities 16:48:12 #link https://launchpad.net/oslo/+milestone/next-kilo 16:48:12 #link https://launchpad.net/oslo/+milestone/kilo-1 16:48:12 Is anyone blocked on work we prioritized for this release? 16:48:38 #info dhellmann needs reviews on the namespace packaging changes 16:48:38 #link https://review.openstack.org/#/q/status:open+branch:master+topic:bp/drop-namespace-packages,n,z 16:48:56 #info oslo-incubator cleanup for oslo.concurrency is blocking neutron adoption 16:48:56 #link https://review.openstack.org/122796 16:49:03 others? 16:49:27 there are a few specs that could use reviews 16:50:00 at this point I'm ignoring specs with unanswered comments, but there were several today that had been updated and might still make it in this cycle 16:51:18 that's everything I had for this week 16:51:20 #topic open discussion 16:51:36 is there anything else you'd like to discuss this week? 16:51:39 so I have enginefacade up and working, some more changes to be made 16:51:51 give it a looksee at https://review.openstack.org/#/c/138215/ 16:52:20 also it looks like we are dumping SQLA < 0.9.7 ! woop 16:52:30 so im going to yank a bunch of cruft out of oslo.db 16:52:35 dhellmann: oslo.context? 16:52:37 zzzeek: prepared to kill thousands of lines? 16:52:46 ihrachyshka: couple hundred in compat/ 16:53:01 nah, not worth it 16:53:02 dims: where does that stand? I don't see any open reviews 16:53:29 zzzeek: bear in mind the stuff I mentioned earlier about removing things 16:53:30 dhellmann: last we talked, there was additional work from spec required 16:54:03 dhellmann: absoutely, I’d just be removing things that are specifically SQLA 0.8 isms b.c. when requirements are bumped we can be sure everyone is on newer SQLAlchemy right ? 16:54:30 zzzeek: we need to be VERY careful with those sorts of changes; consider the rolling update case 16:55:06 dhellmann: OK if oslo.db has SQLAlchemy 0.9.7 in requirements, how would any project run oslo.db w/ that requirements file and not have SQLA >= 0.9.7 in place ? 16:55:20 dims: there's no list of work items on https://blueprints.launchpad.net/oslo-incubator/+spec/graduate-oslo-context so I don't remember what those things were. Let's work that out today. 16:55:23 it's not updated in stable/ requirements 16:55:45 dhellmann: ack 16:56:04 zzzeek: did all juno projects adopt oslo.db? 16:56:06 bknudson: if we cap versions in stable branches, then we're fine 16:56:27 capping versions in stable branches didn't work. 16:56:38 I think you could cap stable/icehouse but not stable/juno. 16:56:43 zzzeek: and if not, can those projects use sqla 0.9.7 with the incubator version of the db code 16:56:52 dhellmann: i dont know. but also, i dont know why that would be significant. if a version of oslo.db has a requirements file in it, the library must still maintain code that is specific to library versions that are specifically barred in its own requirements.txt file ? 16:56:58 dhellmann: whats the purpose of a requirements.txt then ? 16:57:14 ihrachyshka: any caps in the stable branches need to allow one project on a box to be updated while another is not (glance but not nova, for example). That means the version ranges have to overlap between juno and kilo 16:57:29 dhellmann: we’re talking about code that specificallty starts with: “if sqla_version < 0.9:” 16:57:39 zzzeek: well, it's very possible we can't update that requirement all in one go 16:57:50 zzzeek: see my comment to ihrachyshka ^^ 16:58:02 dhellmann: OK that’s somethign else, I’m not touching any code until the requirements.txt is updated 16:58:02 start on a server with everything installed 16:58:15 icehouse: SQLAlchemy>=0.7.8,!=0.9.5,<=0.9.99 ; juno: SQLAlchemy>=0.8.4,<=0.8.99,>=0.9.7,<=0.9.99 ; master: SQLAlchemy>=0.9.7,<=0.9.99 16:58:31 bknudson: cool, thanks 16:58:57 based on that, it looks like we believe icehouse, juno, and master would all run with 0.9.7 16:59:09 * dhellmann wonders if that's actually true 16:59:32 so we already have much in oslo.db that is not compatible with SQLA 0.7.8, 0.7.9. oslo.db’s own requriements file starts at 0.8.4 16:59:35 there's no test that tries the min version supported... kind of a gap imo. 16:59:46 SQLAlchemy>=0.8.4,<=0.8.99,>=0.9.7,<=0.9.99 (so weird, haha) 16:59:48 bknudson: yeah 17:00:20 zzzeek: it sounds like it's safe to remove the 0.7 stuff from oslo.db 17:00:31 harlowja_at_home: I wonder what it even does? 17:00:36 dhellmann: im quite confused about this 17:00:39 I'm less certain of the other parts. That's why I'm asking you to think about it more than just "but our requirements file says" 17:00:52 bknudson, pip does some weird stuff, so who knows 17:00:58 dhellmann: if i have a line of code in oslo.db taht says: “if sqlalchemy_version < 0.8.4” 17:00:59 ok, we're out of time here, let's talk about it more in the oslo room 17:01:01 OK 17:01:10 thanks everyone 17:01:13 #endmeeting