19:01:38 #startmeeting swift 19:01:38 rock and roll 19:01:38 no meet bot? 19:01:38 who's here? 19:01:38 o/ 19:01:38 o/ 19:01:38 hello 19:01:38 o/ 19:01:38 hi 19:01:39 not sure if we'll get a meetbot or not 19:01:39 Meeting started Wed Apr 30 19:01:38 2014 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:40 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:43 The meeting name has been set to 'swift' 19:01:46 ah ok then 19:01:54 o/ 19:02:08 welcome meetbot 19:02:19 #link https://wiki.openstack.org/wiki/Meetings/Swift 19:02:23 agenda for this week 19:02:48 I've been catching up after being out last week (and a little bit of news stuff this morning...) 19:03:00 so let's see where we are 19:03:37 looks like there are been some patches landing (yay) 19:03:48 the correct version is making it's way through zuul now 19:04:12 what happened is that the bot had a bug, so the milestone-proposed branch never got merged 19:04:20 whoops 19:04:32 then it got fixed, but seemingly fell through the cracks with the gerrit upgrade and heartbleed stuff 19:04:44 so it was only just this morning retriggered 19:05:18 so once that merges and we rebase we should be good? 19:05:30 so that means the window of inaccurate version numbers was a little longer, but ya--should be fine after that lands 19:05:48 speaking fo patches that have landed recently.... 19:05:55 #topic in process func tests 19:06:01 portante: you should be ahppy about that :-) 19:06:02 yes 19:06:06 we all should be 19:06:10 :-) 19:06:14 of course 19:06:15 that was not a fun patch to work on, for sure 19:06:33 and it looks like you're working with -infra to get that added to the auto test suite 19:06:43 but I am hoping that it will help us keep the tree more stable as the big sp patches go through 19:06:46 yes 19:06:52 +1 19:07:05 I am hoping that we can all agree to make sure that new code added is covered by functional tests 19:07:10 portante: are both the devstack func tests and the in-process func tests needed? should we prefer one over the other? 19:07:22 way cool portante, I have yet to do anything policy specific with it but its on my todo list 19:07:25 my understanding is the main difference is the coverage report for in-process 19:07:38 notmyname: the devstack one would test the other openstack components 19:07:45 if we don't break them or they don't break us 19:07:52 I thought in-process didn't cover something that regular did. I func-tested against Keystone at some point. 19:08:18 notmyname: too much, lemme sum up, the six fingered man ... 19:08:31 that's right. the in-process won't do keystone/ceilometer/glance stuff. but isn't that covered by the tempest suite? 19:08:52 notmyname: yes, one main difference is that in-process allows us to track coverage of the main code base under functional test 19:09:17 one *could* that with a contrived SAIO installation and running all the servers by hand, but that is more difficult 19:10:05 the second, is that it allows folks to run functional tests quickly without an SAIO, so that there should be little excuse for not running functional tests before a commit is made 19:10:26 portante: `tox -e func` right? 19:10:29 yes 19:10:31 notmyname: the tempest suite is not as comprehensive as the swift functional tests 19:10:41 chmouel: right 19:11:13 I'm not opposed to running both if they test different things, but I don't want to add multiple runs of the same effective test if it's not adding any value 19:11:18 I'd like to see folks look for gaps in coverage during reviews and help out by adding tests that increase the coverage 19:11:30 that's the outstanding question I have about adding it to jenkins 19:12:05 notmyname: I see the in-process tests in the gate as a way to make sure a commit does keep them from working 19:12:29 Now if we started tracking coverage of the project ... 19:12:57 I prefer to make sure a commit did not keep from working. 19:13:00 there are jobs that run coverage specifically after a patch lands, and that would be need to get that information public and easikly viewable 19:13:13 yup 19:13:41 * notmyname imagines a graph on his desktop of historic swift code test coverage 19:13:49 like the gate success rate one I have now 19:13:49 would be nice 19:13:53 ;) 19:13:59 hopefully that will look better 19:14:01 ;) 19:14:18 heh 19:14:26 notmyname: for commit in swift - nosetests -c >> log.txt 19:14:29 so are folks on board with reviewing code changes to catch coverage of new changes by both unit and functional tests? 19:14:39 portante: what do you need from the rest of this? 19:14:45 that ^ 19:14:48 ;) 19:14:49 *rest of us on this 19:15:56 portante: do you mean checking functional test coverage for new patches in addition to unit test coverage? and adding more coverage to functests? 19:16:02 I'd like us all to be comfortable with running the functional tests that way, write new functional tests to add coverage, and when reviewing check for coverage, so that we are less likely to cause the kinds of problems we have had over the past six - 8 months 19:16:08 yes 19:16:17 agreed 19:16:18 ya, I'm totally on board witht hat 19:16:30 does anybody have any concerns with that? 19:16:59 we just need to put our money where our mouths are (or keyboards are I should say) 19:17:07 I think we'll need to update some docs and simply remind each other for a while until it becomes natural 19:17:19 * peluse_ says not it on docs 19:17:29 certainly, and I'd be happy to work on docs as folks find places that it would benefit 19:17:56 portante: ya, I was thinking more with the SAIO docs to start with. or wiki pages for new contributors. etc 19:18:07 ie stuff actual contributors would look at, not deployers 19:18:23 sure, we can talk about that offline then 19:18:28 ya. 19:18:41 two things I can think of that may help it set in with everyone 19:19:00 portante: I'll test your SAIO docs no problem, just had my share of writting in englush for a while 19:19:06 see what I mean? 19:19:09 ;) 19:19:15 portante: are func test coverage results being created now regardless of if you are running func tests in-process or not? 19:19:27 allow ./.functests to run in both modes (I think you might already have this). and track coverage (people optimize to change published numbers, whatever they are) 19:19:40 it is allowed 19:19:55 I run ./.functests and then SWIFT_TEST_IN_PROCESS=1 ./.functests 19:20:11 ah, ok 19:20:13 tdasilva: I think you have to specifically ask for coverage 19:20:14 I hoped that it would increase confidence in PBE 19:20:45 ..and policies and for sure EC when we get there 19:20:54 I use "tox -e pep8,py27,func" before a commit from my laptop, and then in my SAIO use .functests to get the SAIO func test runs 19:21:28 zaitcev: yes, that is the goal, but we might have to raise the coverage on master before a given patch can be shown as "good" 19:21:37 nice 19:21:49 I am hoping that the core dev team will help point those kinds of changes necessary 19:22:11 portante: just so I'm clear (and others too I think) what specifically if anything do we have to do to get coverage to be generated running the in process func tests? 19:22:12 I'd like to have a way to automatically publish the coverage of master today 19:22:18 who's point on making sure this new functionality is written down and findable for people (not that you have to do all the work, just to be the person to coordinate that) 19:22:31 is that me? portante? zaitcev, you interested? 19:22:33 notmyname: I'll do that 19:22:37 portante: ok, cool. thanks 19:22:41 please back me up and check me on it 19:22:52 I'd be happy to 19:22:56 portante: i think there was a gate sometime ago allowing to do that on other openstack project you may want to ping infra about it 19:23:15 chmouel: yes, I'll do that, thanks 19:23:35 one can always run a bot listening to gerrit stream and run the coverage and post back the results 19:23:55 peluse_: the .unittests file shows you how to get coverage with unit tests 19:23:56 it should not be too hard (TM) 19:24:11 portante: K, thanks 19:24:11 chmouel: agreed 19:24:25 peluse_: it would be a nice option to add for .functests 19:24:56 portante: yeah, agree. thought it ws there as part of what you checked in but am clear now that its a separate thing 19:24:58 portante: +1 19:25:08 peluse_: me too 19:25:12 glad for the clarification 19:25:14 you get it with tox 19:25:19 anything else on this subject? 19:25:21 so a tox -e func will get you coverage 19:25:26 ah ok 19:25:29 got it 19:25:31 thanks 19:25:41 (well now I get it) 19:26:30 #topic storage policy update 19:26:47 I haven't sat down with clayg yet, but I see that there has been some good progress 19:26:54 peluse_: can you give a summary? 19:27:16 peluse: I don't have any 'TODO' coding items. I have a few still in need of more core reviewers and have spent time going through clayg's stuff but most of this past week has either been non-Swift for me or EC 19:27:28 good progress on EC design work though :) 19:27:40 so is clayg here? 19:27:42 ok 19:27:50 hmm, does not seem so 19:27:54 no, clayg got puled into a customer issue 19:27:57 okay 19:27:59 also, docs seemed to have slowed down on comments and had at least one 'test' of the SAIO w/policies from someone out there so they're looking good 19:28:14 yeah, clayg would have to talk to his stuff. He's been very busy on it though as you mentioned 19:28:27 AFAIK, he's got a really good reconciler and is getting the patches restaged to propose to master 19:28:34 so he is working on preparing a patch series to master, right? 19:28:35 but I don't know the details on that 19:28:44 thus my recent plea for more reviews on my outstanding patches so we can focus just on the reconclier 19:28:47 yes, that's my understanding 19:29:18 wrt staged patches to master, the plan was to get the reconsilcer set done first and get the rest of mine merged 19:29:19 and are we going to finalize the reconcilier into feature/ec and then propose everyting to master? Or do that in parallel? 19:29:25 ;) 19:29:27 thanks 19:29:30 but he may have mad additional progress that I'm not aware of 19:29:32 peluse_: I think the first 19:29:38 love answers before questions asked ... 19:29:44 :-) 19:29:48 So, what does prevent us from starting pulling SP 19:29:52 so key asks now: review my outstanding stuff (only a few things) and start looking at clay's long series of aptches :) 19:30:05 if not " any 'TODO' coding items" 19:30:16 zaitcev: the idea is that all needs to be on the ec branch first before we can stage a meaningful cohesive set of patches to intro to master 19:30:45 zaitcev: what I meant is that I don't have any more code to write 19:30:50 so we'd want to finalize the docs on feature/ec first as well, then? 19:30:58 yes sir 19:31:03 portante: yes, and I think peluse_ has good ones there already 19:31:04 great 19:31:07 or at least a very good start 19:31:17 that should be the first patch to look at, actually 19:31:22 agreed 19:31:22 for anyone reviewing SP patches 19:31:32 and first in the chain of stuff to land, IMO 19:31:32 they've had several reviews, may need some RST specific formatting but I think content is good (minus reconslicer) 19:31:37 and to make sure it has good functional test coverage. :) 19:31:44 "reconslicer" I like that 19:31:45 indeed 19:31:51 yeah, I was close 19:32:00 recon-somthing-or-another 19:32:35 btw, before SP lands, master has 54% total coverage from just functional tests 19:32:43 good to know 19:33:07 portante: I suspect that much of the gaps are in middleware 19:33:24 how 'bout I prepare a report of sorts on that then 19:33:31 that would be great 19:33:36 +1 19:33:38 maybe it will help us with SP get that shaped up 19:33:52 I think so 19:33:59 I'll run on the S code this week for sure and see what % we're at 19:34:08 S=SP 19:34:17 S=SP=storage policies 19:34:21 ;-) 19:34:27 I would prefer a movement on SP in master in small steps. 19:34:44 "Reviewing" it in EC branch is a lot of work. 19:34:56 zaitcev: do you mean by movement, that the individual patches are small? 19:35:13 applied to master that is? 19:35:14 Off-loading the logical structuring to Paul makes it easier 19:35:34 Well, not small by line count. Small by impact. 19:35:54 zaitcev: I think what we're planning will make sense. Series of patches that flow logically that don't erquire any diffs with the ec branch to understand them 19:36:00 the plan is to propose digestible pieces to master 19:36:17 bring it on 19:36:24 digestible so we don't have a movement :) 19:36:30 But eventually I'll want to trade. 19:36:55 zaitcev: oh yeah 19:37:47 good on S=SP=sotrage policies for this week? 19:37:53 for the update, that is 19:37:59 yup 19:38:20 I'll be finding time to wrap my head around it later this week 19:38:23 reminder for everyone: notmyname and I have a talk at the summit on policies 19:38:32 bah 19:38:34 ya, that's the other kind of thing that I gotta do :-) 19:38:40 just start posting patches 19:38:44 4 summit talks and a day of swift sessions :-) 19:39:06 heh, good luck notmyname 19:39:09 and policies is the first day jsut after the keynotes 19:39:14 zaitcev: `git checkout feature/ec && git branch -m master` ;-) 19:39:39 notmyname: you can't be serious 19:39:45 zaitcev: of course not :-) 19:40:00 zaitcev: you did say to just propose patches... 19:40:19 heh.. hey SP realted, I'd like to start giving a brief update on EC weekly as well 19:40:24 related 19:40:53 peluse_: I think that would be nice to do after the summit. ie next week is last-presummit, then summit, then let's do that 19:41:05 any questions or concerns on the new gerrit version? 19:41:16 cool, I'd just like to see more folks inovled up front on EC than were on policies is all 19:42:56 I think the -infra team is working on putting together some useful dashboards for the new gerrit instance, similar to some of the bookmarks we have now (eg stuff needing one more +2, stuff that hasn't been reviewed at all, etc) 19:43:01 so that should be good 19:43:13 notmyname: great 19:43:29 ok, summit stuff then 19:43:40 oh, I should have been using the #topic 19:43:45 #topic summit prep 19:43:59 I pushed a schedule for the 8 swift sessions 19:44:39 my idea was to choose things (and group things) that will be good for discussion at the summit. remember its facilitaded discussion, not presentations 19:45:21 where did you push it to? :) 19:45:29 I'm looking forward to the sessions on some of the community management stuff. ("swift growing pains"). stuff about making sure we do reviews in a timely manner and encourage new devs and improvign docs 19:45:37 notmyname: but we'll have a project to show pictures if needed though right? Will be hard to talk EC without a few pics 19:45:39 but we also have the swift project pod 19:45:44 peluse_: ya, I think so 19:45:51 peluse_: ie we always have in the past 19:46:22 the "project pod" is a table for us. we've got the corner of a fairly large room, so plenty of space. and I'm bringing a projector for it 19:46:52 peluse_: or you can run to nearest Kinkos/FedEx and make a few copies 19:46:59 unscheduled time there (but we could keep a list on the table or something), but it should be a good place to start and continue discussions 19:47:12 notmyname: which days do we have the pod? 19:47:25 acoles: all week AFAIK (tuesday through Friday) 19:47:33 notmyname: great 19:47:36 that is, the entire "Design Summit" time 19:47:54 I think these pods may be in lieu of a dev lounge. not sure about that 19:48:15 also, there are some interesting cross-project sessions earlier in the week 19:48:28 full schedule is: 19:48:30 #link http://junodesignsummit.sched.org 19:50:16 any questions or things I can do for you about hte summit? 19:51:02 #topic open discussion 19:51:08 anything else? 19:51:17 dfg: here? 19:52:05 gholt: dfg: I'm a little curious about your "turn of fsync" experiment 19:52:27 off? 19:52:38 yes, that sounds very interesting 19:52:56 is there a commit somewhere documenting it? 19:53:22 https://github.com/rackerlabs/swift/tree/rackspace_production 19:53:33 in lieu of that, physcx has been posting the buffering patch 19:53:59 very interesting 19:54:03 oh yeah. anything to discuss on that here? or keep in in -swift? 19:54:30 just to make folks aware of what he observed 19:54:59 I think it's great. I want to see us all keep looking at performance and efficiency in swift 19:55:16 that the pipeline through proxy server to disk relies on buffers filling up, otherwise PUT writes to disk are small and everything starts going compute bound 19:56:06 anything else this week? 19:56:56 thanks for coming! 19:56:58 #endmeeting