19:02:48 <fungi> #startmeeting infra
19:02:49 <openstack> Meeting started Tue Feb  4 19:02:48 2014 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:52 <openstack> The meeting name has been set to 'infra'
19:03:07 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:15 <SergeyLukjanov> o/
19:03:19 <fungi> #topic Actions from last meeting
19:03:51 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-01-28-19.01.html
19:03:52 <SpamapS> lifeless: sure..
19:04:22 <fungi> mrmartin work with smarcet to get through infra processes
19:04:26 <fungi> that's been happening
19:04:35 <fungi> #link https://review.openstack.org/#/c/69636/4
19:04:49 <fungi> mordred find a way to run manage-projects automagically without puppet
19:05:02 <fungi> updates on that? anything newer than in the bug?
19:05:10 <jeblair> fungi: can you _very_ quickly update me on why that's necessary?
19:05:12 <jeblair> is it in the bug?
19:05:18 <anteaya> he has been sick, I have been talking with UtahDave about salting it
19:05:28 <anteaya> then UtahDave has been sick
19:05:31 <anteaya> no code yet
19:05:48 <fungi> jeblair: seems the issue we're encountering only arises when manage-projects is called from puppet agent
19:05:59 <clarkb> we have a 100% success rate when run by hand
19:06:00 <jeblair> fungi: is it more complicated than the 10min exec timeout?
19:06:21 <fungi> jeblair: mordred had suggested punting and working around the puppet call with an additional layer
19:06:30 <clarkb> jeblair: yes, it actually does create the projects and some of the groups but not all of the groups
19:06:41 <clarkb> which indicates the timeout isn't being hit, but weirdness is happening
19:06:52 <zaro> o/
19:07:10 <jeblair> clarkb: does it log?
19:07:52 <clarkb> jeblair: I think so but not with the -v flag when run by puppet iirc
19:08:41 <fungi> one of my suggestions in the bug was to either figure out why the exec stdout/stderr capture in puppet isn't working as advertised, or to add a python logging config for manage-projects, but i haven't foung time to look into either of those
19:08:51 <jeblair> okay, well, this idea that we can't figure out what this simple program is doing is way too voodoo for me....
19:09:11 <clarkb> jeblair: fungi: puppet execs are weird and voodoo like
19:09:15 <clarkb> they are documented as such
19:09:24 <jeblair> i would like manage-projects to get proper logging and log what it's doing so we can see what's actually happening
19:09:25 <clarkb> and the recommendation from puppetlabs is to not use them
19:09:31 <clarkb> but I am all for better logging
19:09:45 <jeblair> clarkb: do they fail at calling fork/exec?
19:09:50 <clarkb> jeblair: yes
19:10:08 <clarkb> well they blame ruby, but yes
19:10:10 <jeblair> clarkb: how does anything in puppet work?
19:10:31 <jeblair> i mean we exec stuff _all the time_
19:10:42 <jeblair> are they not closing fd's properly or something?
19:10:49 <clarkb> no it has to do with environment stuff
19:11:47 <jeblair> okay, so basically we're not anywhere near being able to automatically create new projects
19:12:30 <jeblair> should we resume the moratorium on approving new projects?
19:12:45 <fungi> right. mordred has resorted to manually running manage-projects after new project configs merge
19:13:13 <fungi> and we merged a puppet config change to disable the exec temporarily
19:13:18 <jeblair> ok
19:13:31 <fungi> we should capture more of this into the bug
19:13:35 <fungi> #link https://launchpad.net/bugs/1242569
19:13:36 <jeblair> so mordred is in charge of semi-manual new project creation until that bug is fixed
19:14:15 <fungi> #action mordred continue looking into bug 1242569
19:14:21 <jeblair> if that's the case, that works for me
19:15:04 <fungi> i do agree that "let's just not use puppet" seems like a bit of a kluge
19:15:28 <fungi> but maybe not, without a more detailed proposal it's hard to say
19:15:31 <jeblair> yes.  i think we need proper logging regardless of how it's invoked.
19:15:35 <anteaya> sorry, I guess I mis-understood then, I thought that was where we were last week
19:15:52 <jeblair> i'm just plain embarrassed that we're like "it's broken and we don't know why because we don't know what the computer is doing".
19:16:22 <fungi> i concur
19:16:44 <jeblair> i've dragged this on enough though; please continue
19:16:53 <fungi> we have an agenda item for this, so we can pick it up again in a bit if needed
19:17:06 <fungi> clarkb upgrade jenkins.o.o and jenkins01 to 1.543 and upgrade zmq plugin and scp plugin everywhere
19:17:12 <fungi> that all happened, correct?
19:17:23 <clarkb> that is all done but the zmq bit ebcause there is an outstanding review to fix the master name reportig
19:17:33 <clarkb> so the action can be updated to:
19:17:48 <clarkb> #action clarkb upgrade zmq plugin everywhere to report master name in events
19:17:58 <fungi> oh, i missed that the master name patch was buggy. have a link to the fix?
19:18:16 <clarkb> #link https://review.openstack.org/#/c/69763/
19:18:42 <fungi> fungi move graphite whisper files to faster volume
19:18:56 <fungi> jeblair actually moved these to a cinder volume
19:19:14 <jeblair> fungi: oh do we want to move that from spinning to ssd?
19:19:41 <fungi> we should keep tabs on performance and see whether it improves before pvmoving it to an ssd volume
19:19:51 <fungi> but should be trivial now that it's lvm
19:20:05 <jeblair> fungi: ok.  also, that host would probably benefit from a move to performance flavor
19:20:12 <jeblair> which should also be easier now :)
19:20:38 <fungi> #action fungi see if graphite whisper files need to be moved to a faster pv or faster vm
19:20:59 <fungi> (since i fell down on getting it taken care of so far, it's only fair)
19:21:15 <fungi> also didn't get around to...
19:21:18 <fungi> #action fungi prune obsolete whisper files automatically on graphite server
19:21:28 <jeblair> that's hard because some metrics don't report very often
19:21:41 <fungi> is once a month sane there?
19:21:46 <fungi> once a quarter?
19:21:58 <fungi> how often is not very often?
19:21:59 <jeblair> fungi: i'm thinking some release jobs only happen twice a year
19:22:05 <fungi> ahh, yep
19:22:09 <fungi> ick
19:22:25 <jeblair> so the easy answer is prune after 1 year; more frequently becomes tricky.
19:22:30 <fungi> i can start by seeing if we have any whisper files untouched for a year
19:22:44 <fungi> if we've had a graphite server that long
19:22:58 <fungi> anteaya assist mordred in automating manage-projects again
19:22:59 <jeblair> could probably do like 8 months or so
19:23:14 <clarkb> we can probalby also manually prune using a list generated by find
19:23:21 <clarkb> a lot of things will be obviously obsolete
19:23:32 <fungi> i guess that ties back in with the earlier bug 1242569 discussion
19:23:35 <jeblair> clarkb: yeah, with some thinking we could identify metrics that zuul/nodepool don't use anymore.
19:24:08 <fungi> anteaya: were you still working with mordred on bug 1242569?
19:24:26 <anteaya> I had it on my radar
19:24:37 <anteaya> as of yet, no progress
19:24:46 <anteaya> he was sick
19:24:48 <fungi> should we keep that as an action item for next week?
19:24:55 <anteaya> may as well
19:25:03 <fungi> #action anteaya assist mordred in automating manage-projects again
19:25:08 <anteaya> thanks
19:25:17 <fungi> push for improved logging first to see whether it's necessary
19:25:37 <fungi> i know this hasn't happened yet...
19:25:38 <anteaya> k
19:25:41 <fungi> #action mordred to lift virtualenv 1.10.1 pin when we're ready to babysit it
19:25:52 <anteaya> yes, that is my name on the logging bug
19:26:02 <fungi> zaro to point zuul-dev at gerrit-dev
19:26:08 <fungi> zaro: that happened right?
19:26:09 <zaro> done
19:26:13 <fungi> awesome!
19:26:32 <dguitarbite> hey guys, I have something to discuss if possible
19:26:55 <fungi> dguitarbite: is it on https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting ?
19:27:05 <fungi> if not, i can try to work it in
19:27:31 <fungi> zaro to review jeepyb integration with new gerrit and update gerritlib for gerrit 2.8
19:27:39 <fungi> i know i saw some work on that as well
19:27:42 <zaro> done.
19:27:46 <fungi> that's continuing, right?
19:27:50 <fungi> oh, even better
19:27:53 <zaro> it's done.
19:27:57 <fungi> great
19:28:12 <zaro> fixes have been pushed to review.oo
19:28:45 <zaro> #linkhttps://review.openstack.org/#/c/69800
19:28:56 <zaro> fix gerrit hooks - #link https://review.openstack.org/#/c/69791
19:29:08 <zaro> fix replicate command - #link  https://review.openstack.org/#/c/69768
19:30:46 <dguitarbite> fungi: thanks
19:30:46 <zaro> ok. nothing more on that testing.
19:31:36 <zaro> ohh just wanted to mention that tested the following scripts with gerrit 2.8: expire_old_reviews.py, manage_projects.py, update_blueprint.py, update_bug.py, welcome_message.py, openstackwatch.py
19:32:02 <fungi> anything major worth reporting on that real quick, or just want to plug the etherpad issues list?
19:32:02 <fungi> #link https://etherpad.openstack.org/p/gerrit-2.8-upgrade
19:32:02 <fungi> okay, so we've got half the meeting slot left. i'll try to blow through topics which we covered in the action items
19:32:02 <fungi> also, my irc client's lag timer keeps creeping up, no idea whether freenode is still under a ddos, but i'll add clarkb as an optional chair in case i vanish
19:32:25 <fungi> #chair clarkb
19:32:26 <openstack> Current chairs: clarkb fungi
19:32:44 <fungi> #topic Trove testing (mordred, hub_cap, SlickNik)
19:33:05 <fungi> looks like only mordred is in channel
19:33:18 <fungi> we can come back to this one at the end of they show up
19:33:26 <clarkb> that was the case last week. ++ to shifting
19:33:32 <pleia2> and several weeks
19:33:41 <fungi> wow meetbot is sluggish
19:33:43 <pleia2> is there a bug to track it? maybe we pull from agenda for now
19:33:54 <SlickNik> hello
19:34:04 <fungi> howdy SlickNik
19:34:24 <fungi> any quick updates on "Trove testing"
19:34:28 <fungi> ?
19:35:06 <SlickNik> I have a patch out to tempest (basic trove flavor API tests) waiting on reviews.
19:35:10 <SlickNik> one sec
19:35:24 <SlickNik> #link https://review.openstack.org/#/c/69501/
19:35:38 <SlickNik> I'm working on adding more tests
19:36:02 <SlickNik> Also the gate job has been set up and is running on jenkins.
19:36:08 <SlickNik> It's non-voting as of now.
19:36:33 <fungi> okay, great
19:36:40 <fungi> that's some awesome progress
19:36:50 <jeblair> yaaay!
19:37:17 <fungi> since we use trove for some of our infrastrucutre (and soon hopefully much more) i'm glad it's being tested ;)
19:37:41 <fungi> moving on quickly...
19:37:45 <fungi> #topic Tripleo testing (lifeless, pleia2, fungi)
19:38:16 <fungi> we've had some setbacks with nodepool bugs/tripleo-ci outages but all in all things are progressing
19:38:46 <pleia2> yeah, continuing to work on getting fedora in the gate too (helps tripleo and some other things we want to do)
19:39:29 <fungi> anything in particular we should be aware of beyond that for now?
19:39:42 <pleia2> I don't think so, gritty details otherwise were covered in tripleo meeting
19:40:59 <fungi> as an aside, i've got puppet agent temporarily disabled on nodepool.openstack.org until the tripleo-ci provider is back up and running, which i gather will probably be some time later today. for now it's been deleted out of nodepool.yaml since it was bringing new bugs to the surface we haven't had time to dig into yet
19:41:10 <clarkb> I do think we may need to consider reverting some of those nodepool changes though. Should debug the slow deletes first, but performance is worse than before
19:41:17 <pleia2> thanks for staying on top of that, fungi
19:41:26 <fungi> which is why all the tripleo jobs are in a pending state in zuul's status
19:42:26 <fungi> #topic Requested StackForge project rename (fungi, clarkb, zhiwei)
19:42:52 <fungi> no update on this... it's still pending hearing back from zhiwei but we'll probably also be moving some oslo libs out of stackforge to openstack
19:43:00 <fungi> so we can try to time those together
19:43:11 <clarkb> and savanna if they choose a new name
19:43:11 <jeblair> oh neato
19:43:12 <fungi> also maybe savanna renames if they work out what their new name is
19:43:18 <fungi> what clarkb said
19:43:26 <dhellmann> this item is on the agenda twice, because I didn't notice that until the meeting started
19:43:38 <fungi> dhellmann: no worries
19:43:38 <SergeyLukjanov> fungi, yup, we're searching new name atm
19:43:57 <dhellmann> and we're up to 4 libraries moving into oslo, not just the 3 mentioned on the agenda :-)
19:43:59 <fungi> we'll hit it at that point on the agenda
19:44:05 <fungi> or now is fine
19:44:08 <clarkb> SergeyLukjanov: `cat /dev/urandom | tr -dc '[:alpha:]' | head -c6`
19:44:20 <fungi> dhellmann: and you have a change up for that already too, right?
19:44:20 <SergeyLukjanov> fungi, my estimate is 2-3 weeks
19:44:22 <dhellmann> fungi: either works, I don't have much to say unless there are questions
19:44:33 <dhellmann> fungi: yes
19:44:35 <dhellmann> #link https://review.openstack.org/#/c/70435/
19:44:35 <SergeyLukjanov> clarkb, :)
19:44:46 <annegentle> savanna is definitely out though?
19:44:50 <fungi> #topic Oslo adopting some stackforge libs (dhellmann)
19:44:57 <fungi> since we seem to be doing that
19:45:06 <fungi> cliff, stevedore, taskflow and...
19:45:10 <dhellmann> pycadf
19:45:18 <fungi> excellent
19:45:39 <fungi> dhellmann: is that an asap move, or do you need some lead time/.
19:45:41 <fungi> ?
19:45:49 <dhellmann> fungi: we're ready when you all are
19:46:07 <dhellmann> there's no particular hurry, but sooner is better, so I think I'd rather not wait for savanna to pick a name if that's the only thing holding us up
19:46:57 <SergeyLukjanov> dhellmann, agreed, it could take up to a month for savanna
19:46:57 <fungi> i think i'm free this weekend, but would appreciate at least one other infra core on hand in case we get into the weeds
19:47:20 <clarkb> fungi: I am maybe free this weekend
19:47:37 <clarkb> no plans yet but the planning of possibilities has started...
19:47:52 <dhellmann> I have a family thing this weekend, or I'd hang out with you guys -- I'm not sure I'll be much more than moral support, though :-)
19:48:32 <fungi> dhellmann: well, we'll have .gitreview file patches which you'll want someone to merge
19:48:44 <jeblair> we could do it friday afternoon...
19:48:52 <fungi> i'm cool with that
19:49:02 <dhellmann> friday works for me
19:49:16 <fungi> activity seems to be mostly dead on friday afternoons north-american time
19:49:29 <jeblair> cool, and it's just a few mins of downtime
19:49:43 <fungi> yep
19:49:59 <fungi> what do we think... 20:00 utc friday then?
19:50:19 <fungi> that's noon pst i think. go later?
19:51:10 <dhellmann> I'm in EST, so that time is great for me, but later works too
19:51:22 <jeblair> that time or later wfm; your pick
19:51:26 <fungi> yeah, i'm in est as well
19:51:57 <fungi> okay, let's do 20:00 utc. good compromise without pushing pst friday lunch too far out
19:52:35 <dhellmann> I'll be there
19:52:44 <jeblair> ++
19:52:45 <fungi> #agreed will rename new oslo projects on friday 2014-02-07 at 20:00 utc
19:52:47 <clarkb> sounds good
19:53:12 <fungi> pulling some topics out of order for the last few minutes
19:53:21 <fungi> #topic Request for a new branch of openstack/operations-guide for ongoing O'Reilly work (annegentle)
19:53:31 <fungi> annegentle: i can do this--just get up with me after the meeting
19:53:58 <fungi> #topic Storyboard update (jeblair)
19:54:14 <jeblair> oh neat!
19:54:28 <jeblair> so we had a really good meeting; see the etherpad for what we captured
19:54:40 <jeblair> should have _basic_ functionality real soon
19:54:42 <jeblair> i would have said by wednesday
19:54:47 <jeblair> but krotscheck_sick is sick
19:54:56 <jeblair> so it could be later than that
19:55:10 <zaro> said he was going on vacation too right?
19:55:22 <jeblair> hopefully with the basic arch in place, it'll be easier for others to pitch in and cargo-cult forms and stuff.  :)
19:55:23 <annegentle> fungi: sounds good
19:55:34 <jeblair> zaro: yes, he has vacation scheduled after his illness
19:55:53 <fungi> going on vacation sick is such a waste of a vacation. he should just work instead ;)
19:55:57 <jeblair> anyway, once that's done we can start using it to dogfood storyboard and for infra projects
19:56:13 <jeblair> it will lack _many_ features, but we can start adding them quickly
19:56:36 <jeblair> this is all under the "minimum viable product" on the etherpad
19:56:50 <fungi> the lack of features will drive us to add what we need, i suspect
19:57:20 <fungi> good for itch-scratching
19:57:25 <gothicmindfood> fungi: that's the plan. To psychologically compel contributions with super bare features. :)
19:57:47 <jeblair> gothicmindfood: have the etherpad link handy?
19:58:02 <ttx> https://etherpad.openstack.org/p/StoryboardMeetup
19:58:06 <jeblair> #link https://etherpad.openstack.org/p/StoryboardMeetup
19:58:10 <gothicmindfood> https://etherpad.openstack.org/p/StoryboardMeetup
19:58:18 <jeblair> [eot]
19:58:19 <gothicmindfood> ha. jeblair and ttx were quicker
19:58:28 <ttx> years of training
19:58:28 <gothicmindfood> we all owe each other cokes.
19:58:46 * fungi is so very excited
19:58:58 <jeblair> oh
19:59:10 <lifeless> ttx: sorry, I have to run a short errand mid-meeting (C->kindy) - bbs
19:59:10 <gothicmindfood> and fwiw - I am currently putting together an etherpad set of stories related exclusively to Storyboard authentication
19:59:22 <gothicmindfood> https://etherpad.openstack.org/p/StoryboardAuth
19:59:31 <jeblair> also, http://storyboard.openstack.org/#!/ is in puppet and should do things like upgrade (including db migrations) as needed automatically
19:59:31 <gothicmindfood> I even have nice diagrams in powerpoint form
19:59:44 <jeblair> #action jeblair get ssl cert for storyboard
19:59:54 <fungi> very productive
20:00:00 <fungi> sarob: on the moodle app integration for training manuals thing (dguitarbite seems to have vanished), we can talk over in #openstack-infra if you like
20:00:10 <fungi> we're at time
20:00:14 <fungi> #endmeeting