19:03:11 <mtaylor> #startmeeting
19:03:12 <openstack> Meeting started Tue Sep 20 19:03:11 2011 UTC.  The chair is mtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:03:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:04:05 <jeblair> o/
19:05:57 <carlp> o/
19:06:58 <vishy> o?
19:06:58 <soren> o/
19:07:18 <soren> o⪐
19:07:30 <medberry> o no
19:07:32 <mtaylor> I like o???
19:07:55 <mtaylor> #topic Actions from last week
19:08:04 <mtaylor> #link http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-09-13-19.03.html
19:08:39 <mtaylor> still nothing on those, sorry - been focused on getting diablo out the door
19:08:55 <mtaylor> #action mtaylor Add how to contribute section to ci.openstack.org
19:09:02 <mtaylor> #action mtaylor add packaging docs to ci.openstack.org
19:09:08 <mtaylor> #action mtaylor infrastructure.openstack.org web config
19:09:15 <mtaylor> #action mtaylor will work with carlp on setting up netstack CI hardware
19:09:32 <mtaylor> #topic open discussion
19:09:55 <mtaylor> soren, vishy and I (mostly soren and vishy) were talking earlier about pulling libvirt 0.9.5 into the openstack ppas
19:10:15 <mtaylor> I've just about got a package ready there - but I'd LOVE it if you'd look at it soren
19:10:20 <vishy> yehaw 9.5!
19:10:23 <mtaylor> because there are a crap-ton of moving parts in there
19:10:33 <carlp> I got a session approved for planning the CI for NetStack, I would love it if some or all of you could attend.
19:10:47 <mtaylor> carlp: oh, I'll be there
19:11:27 <carlp> mtaylor: Awesome.  I figured getting everyone in the same room for an hour may help things along :)
19:11:40 <mtaylor> yes. I think you are 100% correct on that
19:12:39 <mtaylor> soren, ttx: also - need to chat at some point (again, sorry I'm annoying) about version numbering. folks want to upload things to PyPI at a frequency greater than full releases, and the current versioning scheme and the mechanism that producees it are - well - problematic for PyPI
19:13:36 <mtaylor> I don't have a specific fix in mind - but I know the two of you have spent a LOT of time working out how that version sequencing works
19:13:55 <ttx> mtaylor: yes, it's about the only thing that works from trunk to release PPAs
19:14:14 <ttx> so that's the reason why it looks like it does -- not sure you can come up with an alternative that would work
19:14:19 <ttx> but you can try :)
19:14:27 <mtaylor> ttx: yup. heard and understood! :)
19:15:13 <mtaylor> ttx: perhaps sitting down with a whiteboard and a LOT of beer might be the right choice there
19:15:23 <ttx> If the solution is to drop the capability to upgrade packages seamlessly whatever their origin, I think I'd not agree
19:15:32 <mtaylor> nope. I do not want that to be the solution
19:15:43 <mtaylor> either
19:16:00 <ttx> I have ignorance of PyPI... hmm... specificities.
19:16:16 <mtaylor> but I do think that we need to be able to produce tarball artifacts that work outside of the context of debian versioning
19:16:16 <ttx> so I miss part of the puzzle
19:16:33 <mtaylor> ttx: the main thing is that nobody other than debian understands ~ version specifiers
19:16:50 <ttx> the reason why stuff is versioned 2012.1~ is so that you don't do a version bump as the very last thing before a release
19:16:59 <mtaylor> so when we depend on them, the version sequencing for our tarball releases becomes broken
19:17:30 <ttx> it could be called 2012.1~prerelease~blabla so that people understand ~
19:17:37 <mtaylor> I totally undestand the reasoning there - I'm just saying that the mechanism isn't supported by all of the places where people would like for us to upload release artifacts (such as pypi)
19:18:06 <mtaylor> it's that pypi and pip do not understand that 20112.1 > 2012.1~prerelease~blabla
19:18:12 <mtaylor> afaik
19:18:14 <medberry> nod.
19:18:44 <ttx> mtaylor: we /could/ rename tarballs before uploading to ubuntu... the trick is that 2012.1~ enables us to have a painless release process
19:18:53 <medberry> (that's pretty critical for ensuring clean debian-style upgrades. PyPI probably isn't worried about such OS-ish things.)
19:18:58 <ttx> sice you don't regenerate anything one time at the very end
19:18:59 <mtaylor> so, I think that where we are in trouble is that we're doing a GREAT job of getting things set for debian-based distro releases, but we sort of skip over the step of making sure our tarball sequencing is sane
19:19:40 <ttx> mtaylor: the alternative is some fugly stuff like odd/even schemes + a version bump at the very end of the process that breaks stuff at the wors tmoment
19:19:51 <ttx> and usually when everyone is sleeping and I have to release before 9am.
19:20:11 <ttx> I've been there before.
19:20:31 <mtaylor> well... as I think you know - I believe that any system in which a version bump breaks anything is a HUGE bug ... but I totally hear that pain and it's possible we have not fixed that in our codebase at the moment
19:20:59 <mtaylor> but like I said- I don't have a solution - I just have a new use case/problem :)
19:21:21 <ttx> mtaylor: not meant as a criticism of your work, but almost every piece of CI I have touched so far failed the first time I used it -- I'd rather that first time not be the release day.
19:21:48 <ttx> that's release management common sense
19:22:07 <ttx> does PyPI understand some alpha/beta/rc marker ?
19:22:09 <soren> mtaylor: It's not just the chance of actual breakage.
19:22:40 <soren> mtaylor: There are so many things that are massively more convenient by bumping early rather than late.
19:22:41 <ttx> mtaylor: PyPI is just not meant to support beta versions, AFAICT
19:24:48 <soren> mtaylor: A good example is documentation. The job that publishes documentation will simply publish the docs from trunk, saying "this is the docs for the thing that will eventually become version X", and it can publish this at docs.openstack.org/X/ with a "work in progress" disclaimer.
19:24:57 <ttx> so trying to (ab)use it to deliver frequent milestones sounds... difficult
19:25:35 <jeblair> mtaylor just told me he lost his network connection
19:25:41 <soren> As one of the very last things before release, we set "Final = True" to remove the disclaimer.
19:26:31 <soren> And whatever we push as the final version, automatically has its docs published in the right place. As the very next thing, we bump the version in the code to reflect the next planned version.
19:26:39 <mtaylor> sorry - network droped me - I'm back
19:26:46 <soren> mtaylor: Do you have scrollback?
19:26:47 <ttx> mtaylor: did you see the backlog ?
19:26:52 <mtaylor> yes. I have - reading
19:26:55 <soren> Ok.
19:27:21 <mtaylor> ok. yes. I agree with all of the words that you're saying.
19:27:40 <soren> The alternative involves either maintaining the upcoming version number somewhere outside the repository and pull from there, but keeping things in sync is just easier if there's nothing to keep in sync, because everything reads from the same place: the code.
19:28:08 <soren> Either that or having some sort of logic somewhere that's supposed to guess the next version.
19:28:30 <soren> ...but then we have to disable that when we're pushing the final revision (because that's actually the final revision, not the first revision of the next version).
19:28:35 <soren> It's mad.
19:29:08 <soren> It's funny, really, because you were the one who bumped the version early to begin with. I went "wait, what?", but later realised this is the only sensible way to do things.
19:29:32 <mtaylor> ok. let's be clear ... I'm not criticising anyone's design or decisions - and yes indeed I did that
19:29:52 <mtaylor> all I'm saying is - there is a desire from people to have things hit pypi more frequently than every six months
19:30:05 <mtaylor> and the current system, for good or bad, does not support this as a goal
19:30:24 <mtaylor> which means that as a goal should be re-assessed, or other elements of the system design should
19:30:37 <mtaylor> which I believe is ripe for a conversation - probably one in person, I'm guessing
19:30:53 <ttx> mtaylor: sure, I'll wait for a clear alternative
19:31:16 <soren> Where can I read about the versioning limitations that pypi imposes?
19:31:25 <ttx> just saying that we are not using ~ just because we can... We use the right solution, and ~ just enabls us to do it conveniently.
19:31:32 <soren> I've not completely understood how our scheme is problematic for pypi.
19:31:49 <mtaylor> soren: well, they are normal versioning limitations. nothing other than debian understands ~ as a version modifier
19:32:00 <jeblair> http://epydoc.sourceforge.net/stdlib/distutils.version.StrictVersion-class.html
19:32:00 <mtaylor> so pypi does not grok that 2012.1~a < 2012.1
19:32:08 <mtaylor> thanks :)
19:32:09 <jeblair> is that documentation operative?
19:32:20 <jeblair> i'm not positive, i've just been googling around during this conversation...
19:32:22 <mtaylor> I believe so
19:32:24 <ttx> mtaylor: yes, most others accept some 2012.1-alpha1 form though
19:32:34 <mtaylor> they do
19:32:41 <ttx> does PyPI support it ?
19:33:05 <mtaylor> yes. check the link jeblair just posted. 0.5a1
19:33:05 <ttx> because then it's just a matter of renaming before using in Debian, that's not the first upstream that needs to d othat
19:33:22 <vishy> it looks like 5a sorts before 5
19:33:31 <vishy> so can't we just replace the ~ with an a
19:33:32 <ttx> way simpler than dropping the whole "name the version by the next release, not the previous one" discussion
19:33:36 <vishy> and were golden?
19:33:50 <mtaylor> right. I thnk that's all I'm getting at - tarballs/pypi need to be produced by those rules
19:34:31 <mtaylor> potentially - we still hit the problem of our tarballs not being actually produced via version in setup.py but by renaming - which will cause some problems for the python setup.py sdist upload step
19:34:31 <mtaylor> BUT
19:34:42 <ttx> Yes, I respect that ~ is a debianism. I just want to make sure we don't throw the baby (pre-release versioning) with the bath water (~)
19:34:44 <mtaylor> we can start from that versioning as a step one and see what we can come up with
19:34:50 <mtaylor> ttx: oh god no
19:34:55 <mtaylor> sorry if I was implying that
19:35:25 <jeblair> hopefully there is a pipy compatible way of doing the same kind of versioning
19:36:10 <mtaylor> for the record:
19:36:25 <mtaylor> #link http://epydoc.sourceforge.net/stdlib/distutils.version.StrictVersion-class.html
19:36:44 <jeblair> #link http://www.python.org/dev/peps/pep-0386/
19:36:50 <jeblair> should probably keep up with that too
19:37:36 <mtaylor> agree
19:38:01 <soren> Just read through the distutils code.
19:38:06 <jeblair> there also seems to be some history and narrative in there (which i haven't read yet)
19:38:07 <soren> 'a' and 'b' are magic.
19:38:30 <soren> They're the only characters we can use for this. Anything else is rejected.
19:38:41 <ttx> also you need a number
19:38:44 <ttx> "a1"
19:38:46 <jeblair> interesting.  no gammas or c's.  :)
19:38:53 <soren> Right.
19:39:01 <ttx> or e's for essex
19:39:09 <soren> :(
19:39:44 <ttx> so you could replace ~ by a1 but that would look fun. 2012.1a1~e2
19:40:17 <mtaylor> well - why e2? why not just 2012.1a20110944.1234 ?
19:40:17 <ttx> you would probably drop the idea of naming the milestones e's, to just call them alphas
19:40:32 <ttx> which sounds a bit scarier
19:40:47 <soren> mtaylor: That's invalid as well.
19:40:57 <carlp> or just go the Flickr route and say "loves you" :)
19:40:59 <soren> mtaylor: No dots allowed after the 'a' or 'b'.
19:41:03 <mtaylor> 2012.1a201109441234 ?
19:41:10 <ttx> mtaylor: because the milestones are "released" as nova-2011.3~d4.tar.gz currently
19:41:17 <soren> The reason is that we..
19:41:20 <soren> right, what ttx said.
19:41:23 <mtaylor> correct
19:41:37 <ttx> how would you release them under a new scheme ?
19:41:45 <soren> When we switched to having to branches, we had to be able to tell them apart.
19:42:08 <soren> *two* branches.
19:42:16 <mtaylor> I don't know - I need to think about it some... as I said earlier - I do not have this solved, merely wanted to bring up that it was something we needed to think about
19:42:20 <soren> Not "to branches". That's nonsense.
19:42:40 <mtaylor> hehe. if that's the only nonsense anyone says in this channel today then I think we're doing well :)
19:43:04 <ttx> mtaylor: but yes, we should discuss that
19:43:21 <ttx> the current scheme was defined in Brussles with lots of aspirin
19:43:36 <ttx> and lots of graphs
19:43:37 <mtaylor> #agreed we should discuss versioning and how it relates to tarballs and PyPI
19:43:53 <soren> hm..
19:43:58 <ttx> I still have the drawings if you want to stress-test your solution
19:44:06 <soren> WEll, how's this:
19:44:45 <soren> On PyPI we either only ever publish stuff from trunk or the milestone branch.
19:45:02 <mtaylor> yes. I agree with that
19:45:06 <soren> Or, we create two separate.. err.. I don't know the terminology.. "projects"?
19:45:12 <heckj> +1
19:45:13 <mtaylor> oh - wait. no. actually
19:45:15 <soren> One for trunk, one for milestone.
19:45:22 <heckj> (+1 for the first suggestion)
19:45:29 <mtaylor> this becomes a bit of a shitshow as well
19:45:37 <ttx> mtaylor: if you like to be hurt: http://ubuntuone.com/p/vjd/
19:45:56 <jeblair> #link http://tarekziade.wordpress.com/2009/07/03/dropping-pep-386-versions-comparison/
19:46:02 <mtaylor> because of pip-requires
19:46:03 <mtaylor> BUT
19:46:20 <mtaylor> if we only released milestones and releases to pypi (probably frequent enough?)
19:46:58 <mtaylor> then we could release 2012.1~d4 as 2012.1b4 on pypi, no?
19:47:39 <soren> Yes.
19:47:42 <mtaylor> so that pypi just always gets the latest milestone or release (or in the case of swift, every release)
19:47:53 <soren> I think that makes sense.
19:47:57 <mtaylor> and then we don't have to worry about 2012.1~d4~2340239742.234
19:48:00 <mtaylor> sweet
19:48:17 <soren> If anyone cares enough, we can add a nova-i-like-pain project with stuff from trunk.
19:48:21 <mtaylor> yes
19:48:26 <soren> But meh.
19:48:40 <mtaylor> but I think if they want that, then grabbing source and doing python setup.py develop is probably a better choice
19:48:54 * ttx takes a quick break before ppb meeting
19:48:59 <soren> ttx: Enjoy.
19:49:08 <mtaylor> I would like to also do that. anybody got anything else for in here?
19:49:38 <jeblair> #link http://tarekziade.wordpress.com/2010/02/10/pep-345-and-386-accepted-summary-of-changes/
19:49:47 <heckj> mtaylor: I do the setup.py develop all the time - might be good to just suggest it as a standard pattern somewhere
19:49:52 <mtaylor> #action mtaylor will propose something concrete based on milestone/releases going to PyPI using pep386 versioning
19:49:54 <mtaylor> heckj: ++
19:51:03 <mtaylor> great. I need a sandwich
19:51:05 <mtaylor> #endmeeting