19:03:46 <jeblair> #startmeeting infra
19:03:47 <openstack> Meeting started Tue Dec 23 19:03:46 2014 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:03:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:03:51 <openstack> The meeting name has been set to 'infra'
19:03:55 <jeblair> #topic Meeting schedule
19:04:39 <jeblair> so.... normally we follow the lead of the other general project meetings (eg, tc and cross-project) and cancel on those dates...
19:04:57 <jeblair> but that kind of snuck up on me and i forgot to bring it up last meeting
19:05:11 <fungi> heh
19:05:20 <anteaya> I second the motion ot cancel next weeks meeting
19:05:34 <jeblair> so to avoid having two meetings where we know lots of people won't be here....
19:05:40 <fungi> is tc meeting also cancelled today then? i saw the announcement for cross-project
19:06:07 <jeblair> i'd suggest we cancel next weeks, and be sort of informal about this one
19:06:14 <anteaya> yes it went to the tc list
19:06:15 <nibalizer> jeblair: wfm
19:06:20 <pleia2> wfm
19:06:23 <fungi> i expect to be around next tuesday, but will likely be holding down the fort in irc so i'm fine with no meeting
19:06:34 <jeblair> #agreed cancel dec 30 infra meeting
19:06:38 <anteaya> http://lists.openstack.org/pipermail/openstack-tc/2014-December/000908.html
19:06:41 <jeblair> #topic Open Discussion
19:06:45 <clarkb> sounds good
19:06:58 <fungi> heh
19:07:07 <anteaya> ha ha ha
19:07:19 <pleia2> I did add an agenda item this week about where we land 3rd party CI documentation
19:07:23 <clarkb> my only thing is I will be rebuilding slave images shortly so that we fix this pbr freeze bug
19:07:23 <SergeyLukjanov> fwiw in Russia Jan 1-11 are holidays, so, /me will be mostly unavailable
19:07:25 <fungi> how 'bout that pep 440?
19:07:27 <anteaya> I would like to hear from jhesketh as he did get up for this
19:07:34 <anteaya> fungi: ha ha ha
19:07:44 <pleia2> jaypipes has volunteered to write the documentation in .rst format, but he needs to know where to land it
19:07:45 <clarkb> fungi: I think you deserve a few days off now :)
19:08:10 <fungi> clarkb: maybe i'll take thursday off
19:08:13 <jhesketh> anteaya: for what sorry?
19:08:14 <anteaya> clarkb: ++
19:08:20 <anteaya> jhesketh: swift logs
19:08:25 <anteaya> jhesketh: are logs logging?
19:08:37 <zaro> o/
19:08:43 <anteaya> SergeyLukjanov: thanks for letting us know
19:08:43 <jeblair> fungi: how about it! :)
19:09:03 <jhesketh> Ah, seems like we're discussing 3rd party docs first
19:09:09 <jeblair> fungi: pep440/pip/pbr/etc is a good thing to talk about, actually...
19:09:18 <jeblair> let's talk about that for a minute
19:09:29 <jeblair> because it's something some of us may need to continue dealing with over the holidays
19:09:34 <fungi> it's mostly (entirely? knock on wood) resolved at this point
19:10:05 <jeblair> clarkb: is the pbr freeze thing causing actual errors, or just not working as intended?
19:10:05 <clarkb> I am personally interested in a recap of yesterday as I was mostly caught up until then
19:10:11 <fungi> i think the only lingering issues are we haven't built new nodepool images without the setuptools <8 cap in install_puppet.sh
19:10:13 <clarkb> jeblair: its actually causing errors
19:10:20 <anteaya> I was here and would appreciate a recap
19:10:25 <clarkb> jeblair: jobs are failing due to the check >0 tests run check
19:10:33 <fungi> ahh, right, pbr freeze got implemented in the slave scripts incorrectly
19:10:43 <fungi> i approved clarkb's fix just now
19:11:06 <fungi> i'm guessing the change adding it ended up merging after yesterday's images built
19:11:17 <fungi> so we didn't see the issue until today after we got new images
19:11:36 <fungi> mostly pip 6.0 release was uneventful
19:11:37 <anteaya> yes I do believe I approved it yesterday midday sometime
19:12:08 <fungi> it pointed out one pep 440 issue we missed in devstack, which required an icehouse->juno->master series to fix
19:12:12 <jeblair> so since we're going to be in and out and flying and driving around, etc, perhaps if anything further comes up, we should make an etherpad and stick the link in the channel topic so we can keep up to date on what's broken and needs reviewing, etc.
19:12:34 <fungi> but aside from that it was mostly finding regressions in pip 6.0 and pointing them out so that they would get fixed in 6.0.1
19:12:41 <fungi> which they did
19:12:44 <jhesketh> +1
19:13:32 <fungi> oh, other lingering item is that some projects may not have yet merged their reqs sync patches to fix the setuptools entry
19:13:48 <fungi> one of the tripleo jobs was struggling with ceilometer this morning for that reason
19:14:06 <clarkb> s/setuptools/sqlalchemy/?
19:14:06 <fungi> s/setuptools/sqlalchemy/
19:14:12 <fungi> yeah
19:15:18 <fungi> so anyway, i think things are likely to be quiet on the pep 440 front from this point forward
19:15:31 <fungi> pip 6 was the big unknown until yesterday
19:15:44 <fungi> and all the pbr fixes outstanding have merged in the past few hours
19:15:48 <clarkb> fungi: did pip 6 have the single regression?
19:16:05 <fungi> clarkb: there were a couple regressions with it which bit us
19:16:06 <clarkb> whcih was don't copy VCS dirs because speed
19:16:12 <fungi> clarkb: yeah, that was one
19:16:39 <fungi> another was failing to make scripts executable when building wheels
19:17:27 <fungi> we had quite a few jobs failing on "permission denied" in various places because of that
19:17:41 <fungi> oh, also we had to work around the fact that pip uses ~/.cache now
19:18:01 <fungi> so sudo pip whatever was making a root-owned .cache in the calling user's homedir
19:18:11 <fungi> sudo -H pip is preferred to avoid that
19:19:02 <fungi> anyway, it was way fewer issues than we had to work around for setuptools 8
19:20:13 <fungi> also, i'd like to see us finish whatever we need to be able to release pbr master so we can leave the feature/0.10 branch behind us
19:21:12 <fungi> there are a few random improvements hanging out in review for pbr master that have been there a while (since before setuptools 8)
19:21:20 <clarkb> fungi: its the semver series which was mostly lifeless
19:21:32 <clarkb> so we probably need someone to adopt that code if we want it to move forward in the near future
19:21:48 <jeblair> clarkb: has lifeless abandoned that?
19:22:08 <fungi> i think he just has other stuff going on, so temporarily abandoned maybe
19:22:22 <fungi> i know he wants to finish it when he gets time
19:22:34 <clarkb> jeblair: no he is just afk
19:22:37 <fungi> it's basically usable now, but there may be one or two reported bugs to triage
19:22:51 <fungi> related to the semver-esque implementation details
19:24:10 <fungi> anyway, that's all the updates for that topic i think
19:24:47 <fungi> takeaway is that we're fragile around python packaging, and i'm not sure there's a good answer other than try to help the python packaging devs with quick feedback
19:25:35 <fungi> dstufft has been awesomely responsive, and has given us a heads up on releases too
19:25:40 <zaro> would anyone like to continue to explore idea of upgrading gerrit?
19:26:04 <fungi> zaro: you were going to look into upgrading review-dev next, right?
19:26:10 <anteaya> dstufft has been fantastic
19:26:53 <clarkb> fungi: greghaynes and I were talking about it over brewing beer yesterday and we could run a periodic job that runs devstack/tempest with latest pip master
19:26:56 <anteaya> zaro: and based on your email from last night it sounds like the only way to utilize the its-storyboard plugin you are working on upstream is to upgrade
19:27:10 <zaro> anteaya: yes, that is correct.
19:27:19 <clarkb> fungi: maybe dstufft would "subscribe" to that?
19:27:32 <zaro> fungi: i kinda wanted to get an idea which version we were gonna target before testing.
19:27:52 <anteaya> zaro: I thought we decided 2.9 and 2.10 isn't released
19:27:56 <zaro> i think there was the idea of going to 2.10 ?
19:27:57 <jeblair> clarkb: yeah, that would be a good step.  though even better would be if someone released a github or bitbucket trigger for zuul so we could trigger and report on pull requests
19:28:06 <anteaya> zaro: 2.10 is an option?
19:28:25 <fungi> clarkb: perhaps. assuming that they merge pull requests fairly continuously rather than right before releasing
19:28:47 <zaro> 2.10 is what google uses, jeblair mentioned that. don't remember if he meant we should do same
19:28:49 <fungi> and yeah, zuul trigger is the answer there
19:29:18 <jeblair> zaro: i think we got as far as thinking that we should at least work on 2.9
19:29:37 <jeblair> zaro: i think we may want to defer the conversation about whether to track upstream in a CD fashion until more people are around
19:29:38 <zaro> also jeblair wanted to review the ssh problem
19:29:47 <clarkb> fungi: yup long term I think we want a proper trigger but we should get a lot of value out of "pip stopped working today"
19:30:02 <clarkb> fungi: espeically if someone like dstufft is able to watch that
19:30:12 <jhesketh> jeblair: I have a bunch of zuul refactoring that needs reviewing but will hopefully make a github trigger easier
19:30:22 <jeblair> yeah
19:30:26 <zaro> jeblair: did you get a chance to read about the stream events issue?
19:30:32 <jeblair> btw, does anyone want to review jhesketh's pep8 change to zuul?
19:31:01 <jeblair> i kind of feel like if people think it's important, they ought to review it.  i do not think it is important, and yet i reviewed the first patchset and left 40 (not exaggerating) comments
19:31:04 <zaro> i believe they have fixed in 2.9.3
19:31:41 <jhesketh> https://review.openstack.org/#/c/115187/
19:34:18 <jeblair> zaro: not yet
19:35:02 <zaro> jhesketh: isn't there a hacking guideline for that change?
19:35:11 <clarkb> ya if we go to latest stable gerrit that would be good
19:35:23 <clarkb> then we can worry about CD gerrit when not time sunk into holidays
19:35:35 <jhesketh> zaro: not sure?
19:35:39 <anteaya> latest stable +1
19:35:51 <clarkb> jeblair: for zuul you mean that if other reviewers are happy to review and approve the tox changes yo uare on board?
19:35:53 <jeblair> zaro: i have not had a chance to read up on that, but i think prepping for 2.9 doesn't have to wait
19:35:59 <clarkb> jeblair: or you reviewed it and feel even more strongly it shouldn't merge?
19:36:00 <jhesketh> zaro: that's just to re-enable the broken pep8 job
19:36:01 <zaro> jeblair: here's a similar change https://review.openstack.org/#/c/133465
19:36:13 <zaro> opps jhesketh ^
19:36:41 <jeblair> zaro: yeah, i'm pretty sure that jjb change durned of pep8 checks
19:36:45 <jeblair> grr
19:36:47 <jeblair> "turned off"
19:37:03 <jeblair> or rather, would turn off, since it hasn't merged
19:37:31 <clarkb> I do have to say that hacking is one of the most frustrating things ever
19:37:44 <clarkb> and am a huge vote in favor of ignoring H* in infra projects
19:38:13 <jeblair> also, what the heck?
19:38:25 <jeblair> i mean there is a comment that says "do not do what that jjb change does"
19:38:32 <jeblair> and that jjb change does it and removes the comment
19:38:33 <jhesketh> zaro: if you could leave a comment that'd be great
19:38:53 <zaro> ok.  i'll start testing review-dev.o.o with Gerrit 2.9.3.  I remember talking to someone in channel that got their system working with 2.9
19:38:54 <mtreinish> fwiw all of those broken hacking import jobs which can't work are being removed in the next release
19:39:05 <mtreinish> s/jobs/checks/
19:39:06 <jeblair> zaro: i have left a -2 on https://review.openstack.org/#/c/133465/
19:39:23 <clarkb> mtreinish: its not just import ordering though its docstrings must end with punctuation. commit title must end in period. and so on
19:39:57 <mtreinish> sigh, yeah a lot of those are annoying. Yell at jogo
19:40:02 <fungi> period in docstring titles, and periodless in commit message titles
19:40:04 <mtreinish> I was just looking at the review link
19:40:15 <jeblair> clarkb: and what i meant about zuul and pep8 is that i have already spent more time than i would like reviewing whitespace changes.  in doing so i found that the change was _much_ larger than needed.  i think it would be great if someone else who is actually interested in whitespace enforcement review that change and ensure it is minimal.
19:40:42 <clarkb> jeblair: gotcha
19:41:07 <clarkb> I can give it a go. In general normal pep8/pyflakes doesn't bother me
19:42:53 <jeblair> clarkb: most of it doesn't bother me either.  it's going _beyond_ pep8 that bothers me.
19:43:04 <clarkb> ya E125 and friends
19:44:01 <jeblair> zaro: so yeah, we don't use hacking at all in infra projects.
19:44:20 <jeblair> zaro: there was an attempt to use it to check one thing in zuul.  it massively backfired and reminded us why we don't use hacking in infra projects
19:44:35 <jeblair> zaro: that's why we're having this conversation about getting zuul back into compliance with pep8
19:45:03 <jeblair> so i thought it was a bad idea before, and now i think it's a really, really, really bad idea.
19:45:07 <zaro> why is it still on our wiki?
19:45:14 <jeblair> zaro: what?
19:45:21 <zaro> hacking guidlines
19:45:25 <jeblair> zaro: link?
19:45:52 <zaro> http://docs.openstack.org/developer/hacking/
19:46:32 <jeblair> zaro: that's for the openstack project itself, not for openstack-infra
19:46:46 <jeblair> zaro: there are quite a lot of guidelines for openstack that we do not adhere to
19:46:57 <zaro> ahh, gotcha.
19:47:20 <jeblair> notably, we work on code that isn't in python, have different CLA requirements, etc.
19:48:06 <zaro> should probably include this disclaimer in the openstack-infra manual?
19:48:13 <jeblair> with respect to hacking and the like, what works for 2000 developers working together isn't necessarily desirable for 10
19:48:32 <jeblair> zaro: maybe in the ci docs, but the infra-manual is for openstack developers, so that might be confusing
19:48:40 <fungi> s/isn't necessarily desirable/is sometimes actively harmful/
19:48:57 <jeblair> zaro: though tbh, we should minimize that kind of content in the manual anyway
19:49:49 <pleia2> can we talk about docs for a few minutes?
19:49:50 <jeblair> zaro: it should focus more how to use the tools to get things done
19:50:27 <jeblair> pleia2: are we excluding folks who might want to participate?
19:50:34 <fungi> pleia2: ahh, yes, the raging debate about where third-party testing documentation lives long-term
19:50:39 <pleia2> fungi: that's the one
19:50:52 <jhesketh> I have to leave a few minutes early, sorry guys. Re swift logs it is working for infra jobs and I have fixed in review to make it work for devstack gate and improve the log index from feedback. (I have plenty of open changes if anybody feels like reviews)
19:50:53 <pleia2> jeblair: I don't think so
19:51:10 <anteaya> jhesketh: thank you
19:51:24 <jhesketh> s/fixed/fixes
19:51:26 <jeblair> jhesketh: thanks!
19:51:48 <jhesketh> Okay catch you later :-)
19:51:50 <pleia2> I think it's important, it's really difficult to support these folks via jaypipes' blog posts and I think it would bring value to our project to have it documented, since it supports CI as an open source project too
19:51:52 <jhesketh> o/
19:51:56 <anteaya> jhesketh: enjoy holidays
19:52:10 <jeblair> pleia2: http://ci.openstack.org/third_party.html
19:52:29 <fungi> pleia2: so is the question just ci.openstack.org vs docs.openstack.org/infra/manual vs something else new?
19:52:37 <jeblair> so there's that...
19:52:38 <pleia2> fungi: that's it
19:53:10 <fungi> the infra manual, as i see it, is about teaching people how to interact with the infrastructure we're running
19:53:24 <fungi> whereas third-party testing is only tangentially that
19:53:35 * pleia2 nods
19:53:44 <jeblair> fungi: right.  infra manual is "the manual for users of the infrastructure"
19:53:53 <pleia2> we also have http://ci.openstack.org/running-your-own.html which has a fair amount of overlap
19:53:58 <fungi> (the consuming from our gerrit event stream and automated commenting in gerrit is the extent of their interaction with our infrastructure)
19:54:18 <pleia2> so what I think makes the most sense is bringing jaypipes' blog posts in to http://ci.openstack.org/third_party.html and updating /running-your-own.html where there is overlap
19:54:27 <jeblair> pleia2: i think that asselin's proposal to make a puppet module for third-party folks that lives in infra is really useful
19:54:41 <jeblair> pleia2: and i think maybe the best way forward is to proceed with that, and then document that thing appropriately
19:54:43 <pleia2> jeblair: yeah, that will help a lot
19:54:55 <pleia2> ok
19:55:04 <asselin> still not completely clear where to put the docs though
19:55:14 <anteaya> third_party.rst
19:55:25 <pleia2> yeah, in system-config
19:55:32 <fungi> running-your-own was i believe initially intended as a walkthrough for someone who wanted to set up a copy of most of our infrastructure to test it out and develop improvements for it, so slightly different use case from third-party testing
19:55:46 <jeblair> yep, that way the docs live with the code
19:55:58 <pleia2> fungi: yeah, it's what lifeless wrote when he went through it, but there are similarities
19:56:22 <jeblair> and running-your-own is already bitrotting because we don't "run our own" or test things in the way it descibes, and no one is keeping it up to date
19:56:25 <fungi> agreed, there is common content in several documents which we might be able to link up/de-dup more effectively
19:56:37 <jeblair> fungi: ++ lots of good stuff to pull from
19:57:00 <asselin> anteaya, third_party.rst == http://docs.openstack.org/infra/manual/third-party.html?
19:57:01 <fungi> i think running-your-own and the readme in devstack-gate suffer for similar reasons
19:57:05 <jeblair> so that's why i'm hoping we end up with a real puppet module for some of this, and real testing of it, and docs that go along with that, so we can actually keep everything up to date and working :)
19:57:13 <pleia2> asselin: no, http://ci.openstack.org/third_party.html
19:57:19 <fungi> they're not written for the people who are managing the codebase, so we aren't the ones who notice when they get stale
19:57:23 <anteaya> asselin: no, we just outlined that third party docs do not belong in the infra manual
19:57:39 <pleia2> asselin: these are the docs generated from our system-config doc repo
19:57:40 <anteaya> asselin: as pleia2 said, the third_party.rst file in system-config
19:58:05 <asselin> ok I'm +1 with third_party.rst
19:58:12 <pleia2> jeblair: ok, thanks, we have a plan then
19:58:13 <jeblair> not to say that no third-party stuff belongs in infra-manual, just not a complete set of documentation about how to run a system
19:58:28 * jeblair is all out of negatives
19:58:31 <asselin> we should probably remove this empty link then: http://docs.openstack.org/infra/manual/third-party.html
19:58:59 * clarkb hands jeblair a never
19:59:37 <jeblair> clarkb: i'm going to hold on to that...
19:59:42 <fungi> i have a nor you can borrow, like new condition
19:59:53 <jeblair> asselin: probably so
20:01:16 <jeblair> well, thanks everyone, and see you next year!  enjoy the holidays!
20:01:18 <fungi> i guess we're at time, if we want to free up the channel for the dearth of meetings scheduled the rest of the day
20:01:27 <jeblair> #endmeeting