19:03:20 <fungi> #startmeeting infra
19:03:20 * Ng comes to order
19:03:20 <openstack> Meeting started Tue Apr  7 19:03:20 2015 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:03:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:03:24 <openstack> The meeting name has been set to 'infra'
19:03:34 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:48 <fungi> #topic Announcements
19:03:55 <SpamapS> o/
19:03:57 <mmedvede> o/
19:04:12 <pcrews> o/
19:04:15 <fungi> um, we should probably start thinking about summit sessions. that's the only psa i'm aware of for now
19:04:20 <mordred> Duke won the NCAA Basketball Tournament last night
19:04:24 <fungi> heh
19:04:34 <fungi> #topic Actions from last meeting
19:04:36 <pleia2> did the ask.o.o migration happen yesterday? (I saw meeting logs, didn't read backscroll)
19:04:41 <fungi> yep
19:04:44 <pleia2> cool :)
19:04:47 <mrmartin> pleia2: yep
19:04:48 <pleia2> congrats to all
19:04:50 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-03-31-19.00.html
19:04:51 <pleia2> mrmartin: nice work!
19:05:07 <fungi> jeblair document openstack id deployment mechanism in http://ci.openstack.org/openstackid.html
19:05:11 <fungi> looks like he did that
19:05:18 <fungi> anybody have the url handy?
19:05:22 <asselin_> hi
19:05:44 <pleia2> https://review.openstack.org/#/c/170974/
19:06:09 <fungi> #link https://review.openstack.org/170974
19:06:13 <fungi> thanks pleia2
19:06:30 <fungi> zaro to continue utf8 testing
19:06:41 <fungi> i believe there's a script you hacked together
19:07:07 <fungi> #link https://github.com/zaro0508/gerrit-db-migration
19:07:16 <fungi> feedback on that would be useful
19:07:49 <fungi> zaro test gerrit 2.10
19:07:57 <zaro> i believe i am done with db migration testing.
19:08:10 <zaro> #link https://github.com/zaro0508/gerrit-db-migration
19:08:14 <fungi> okay, so 2.10 looks like it probably works?
19:08:28 <zaro> no i haven't tested 2.10
19:08:38 <zaro> i have only tested that utf8
19:08:43 <fungi> oh, you're still talking about the first item
19:08:51 <zaro> that script is for utf8 migration
19:08:54 <zaro> db only
19:09:07 <fungi> right, i had moved on to the next action item
19:09:30 <fungi> okay, so the end result of that script was that utf8 was working under gerrit 2.8 after it ran?
19:09:46 <zaro> not exactly.
19:10:16 <zaro> I didn't check that gerrit works with the data after migrating it.  i only check that the data looked good using db queries
19:10:38 <zaro> to validate that gerrit will also require the git data.
19:11:19 <zaro> that gerrit/that gerrit will start and run
19:11:19 <clarkb> which we have right?
19:11:45 <fungi> right, got it. so you were mainly just testing that the utf8 transformation won't likely be a problem with our production data. did you similarly test that it did the right thing to a gerrit 2.8 (for example review-dev prior to upgrade)?
19:11:46 <zaro> i would need the prod site info
19:12:23 <zaro> yes, i tested that scenario on review-dev end to end and it does seem to work.
19:12:55 <fungi> okay, great! so we should try it again production the next time we schedule an outage window to do other things
19:13:14 <fungi> #action zaro test gerrit 2.10
19:13:15 <zaro> yep. i pretty confident that it should work.
19:13:18 <fungi> i guess that's still pending
19:13:31 <zaro> yes, still 2.10 testing is still pending
19:14:12 <zaro> wondering if i should push that script to review.o.o instead of on github?
19:14:30 <fungi> probably not unless we think it's something we're going to run more than once in the future
19:14:33 <fungi> anteaya cause a problem statement and requirements regarding election tooling to be sent to the infra list so various solutions can be discussed
19:14:34 <zaro> just for review, then abandone afterwards
19:14:46 <fungi> i guess this was lingering from a previous meeting agenda
19:14:57 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2015-April/002622.html
19:15:02 <fungi> seems to have happened
19:15:04 <clarkb> that was sent
19:15:11 <fungi> zaro: yeah, if you want, should be fine
19:15:14 <clarkb> mtreinish responded with some feedback
19:15:35 <mtreinish> clarkb: I did?
19:15:39 <fungi> continued discussion can happen on the ml in that case
19:15:41 <mtreinish> oh the election stuff thread
19:15:46 <pleia2> yeah, would be good to respond to that with rationale as to why ML is not sufficient
19:15:53 <tchaypo> Some discussion also happened on the ether pad
19:15:55 <pleia2> (or maybe it is!)
19:16:17 <clarkb> yup I think we can push it to the ml, curious to see what the election officials have to say about ml
19:16:30 <fungi> #topic Priority Specs
19:16:46 <fungi> this topic is conspicuously blank in the meeting agenda
19:17:18 <fungi> looks like we skipped it last week too
19:17:25 <mordred> fungi: maybe we're done wtih all of them
19:17:28 <clarkb> we have nodepool-dib, swift logs, zanata, and ask migration andprobably some others I forget
19:17:37 <asselin_> downstream-puppet
19:17:39 <fungi> right, but those are in the next topic
19:17:42 <mrmartin> fungi: we have the backup left only for askbot
19:17:52 <fungi> so moving on
19:17:54 <fungi> #topic Priority Efforts
19:18:12 <fungi> let's see if we can get through these quickly
19:18:17 <fungi> #topic Priority Efforts (Swift logs)
19:18:18 <clarkb> oh specs vs efforts, my reading comprehension is poor
19:18:22 <mordred> clarkb: me too
19:18:27 <mordred> clarkb: I had the exact same thought
19:18:30 <fungi> any updates for swift logs?
19:18:34 <clarkb> yes
19:18:38 * mordred dances?
19:18:48 <clarkb> the devstack help footers are now uploaded to swift and rendered nicely
19:18:53 <mordred> woot
19:19:00 <clarkb> you can see the devstack-gate specific dg tempest jobs for examples
19:19:08 <fungi> excellent
19:19:17 <clarkb> but we have a bug in hosting non log file data via swift
19:19:29 <clarkb> #link https://review.openstack.org/#/c/107267/
19:19:49 <clarkb> that is the change that will begin to fix this issue, once we have that I think we are ready to move everything to swift
19:20:01 <fungi> ahh, yep, saw that change proposed
19:20:03 <clarkb> though we can move jobs to swift now that don't have non log data (PNGs, javascript, etc)
19:20:23 <clarkb> I can propose some more changes to migrate jobs for which it should be safe
19:20:31 <fungi> awesome
19:20:48 <fungi> is that a firm action item, or just a conjecture?
19:21:05 <clarkb> might as well make it an action item
19:21:15 <clarkb> #action clarkb migrate more jobs to swift log hosting
19:21:15 <fungi> #action clarkb propose some more changes to migrate jobs for which it should be safe
19:21:19 <fungi> oops
19:21:21 <fungi> #undo
19:21:22 <openstack> Removing item from minutes: <ircmeeting.items.Action object at 0x921a3d0>
19:21:24 <clarkb> double tap
19:21:40 <fungi> that it for swift logs since last week?
19:21:48 <clarkb> I think so
19:22:04 <fungi> #topic Priority Efforts (Nodepool DIB)
19:22:28 <fungi> i saw there was some good discussion in mordred's planning etherpad for the shade stuff
19:22:39 <greghaynes> So good news is we have an ubuntu image that works in both hp and rax now1
19:22:41 <mordred> we also landed most of the shade things
19:22:43 <mordred> and greghaynes did that
19:22:45 <tchaypo> Link?
19:22:47 <fungi> #link https://etherpad.openstack.org/p/dib-nodepool-merges
19:23:04 <greghaynes> https://review.openstack.org/#/c/167003/
19:23:08 <greghaynes> bad news is my wifi is horrible
19:23:47 <mordred> it's getting close to time to start landing "scary" nodepool changes, such as the ones that actually start consuming shade - I've been holding off because I'd like to be around to watch in case something goes south
19:24:15 <mordred> this week will likely be bad for that because pycon - although, maybe it'll be good because jeblair and I will be in the same place at pycon
19:24:45 <mordred> so maybe we can find a bar and land something scary
19:24:57 <fungi> i'm finally getting some review movement on the bindep changes (thanks everyone!) and have a nodepool config change up to add "centos-6" nodes. the open dib-nodepool topic changes are getting longer than one page in gerrit now
19:25:00 <mordred> greghaynes: have you tested it with centos or fedora yet?
19:25:03 <fungi> #link https://review.openstack.org/#/q/status:open+topic:dib-nodepool,n,z
19:25:14 <greghaynes> mordred: there are some bugs with dub relating to that ATM, im currently working through them
19:25:33 <clarkb> yes, I have recently been spending a good chunk of my time only reviewing priority effort related changes
19:25:44 <clarkb> so make sure your changes have the topic set and I am more than likely to review them eventually
19:25:53 * mordred needs to go review clarkb's changes
19:25:56 <clarkb> (re getting review traction)
19:26:31 <fungi> anything else urgent on nodepool dib?
19:26:57 <fungi> #topic Priority Efforts (Migration to Zanata)
19:27:03 <cinerama> hi there
19:27:05 <fungi> take it away pleia2 and cinerama
19:27:18 <pleia2> I've been making my way through cinerama's patches this morning, merged a few
19:27:22 <cinerama> so i made some good progress last week on the various patches in flight
19:27:46 <cinerama> we have a couple of things we ought to address with the zanata devs
19:28:04 <cinerama> next steps will be to get a server up and running and start testing stuff
19:28:19 <cinerama> i think that's it from me
19:28:24 <pleia2> see from line 125 in the etherpad for some plans: https://etherpad.openstack.org/p/zanata-install
19:28:30 <clarkb> I am more than happy to help with the jenkins job script stuff when you are ready for that
19:28:34 <pleia2> I think we're on track to get the dev server up and running
19:28:42 <fungi> #link https://etherpad.openstack.org/p/zanata-install
19:28:45 <fungi> #link https://review.openstack.org/#/q/status:open+topic:zanata,n,z
19:28:46 <clarkb> but the puppet module is coming together nicely big +2 to spinning up a server
19:28:48 <pleia2> I do need translate-dev.o.o removed from DNS when someone has a chance so I can create the new record when I bring up the new server
19:29:00 <fungi> i can do that right after the meeting
19:29:04 <pleia2> thanks fungi
19:29:04 <mordred> ++
19:29:10 <fungi> or maybe during if chairing gets slow
19:29:15 <fungi> (talk more, people!)
19:29:23 <pleia2> that's it from me, we're doing well :)
19:29:25 <mordred> Duke won the NCAA Basketball Tournament last night
19:29:37 <mordred> fungi: (that's all I've got, really)
19:30:13 <fungi> huh. who knew people in north carolina played basketball?
19:30:22 <fungi> #topic Priority Efforts (Downstream Puppet)
19:30:37 <yolanda> so i have several patches that landed, some others awaiting reviews
19:30:39 <fungi> any important news here?
19:30:51 <yolanda> all are under the downstream-puppet topic
19:31:03 <yolanda> i created one to use puppet-puppet to setup puppet
19:31:11 <nibalizer> heh
19:31:13 <clarkb> I have been trying to review those today. The one for cgit apache fork configs I need to test then approve if testing is happy
19:31:21 <yolanda> guess who can say more puppet on a sentence :)
19:31:21 <fungi> #link https://review.openstack.org/#/q/status:open+topic:downstream-puppet,n,z
19:32:01 <asselin_> common log server ones have been through a few reviews. should be stable now & ready to go
19:32:06 <yolanda> if changes for puppet-pip could land as well, will be helpful for us downstream
19:32:28 <fungi> great, thanks for working on that!
19:32:34 <fungi> anything else there?
19:32:50 <yolanda> not from my side
19:32:54 <nibalizer> not for me
19:32:59 <mordred> 19:31:03        yolanda | i created one to use puppet-puppet to setup puppet
19:32:59 <fungi> #topic Priority Efforts (Askbot migration)
19:33:02 <mordred> best sentence ever
19:33:10 <mrmartin> ok, so we are waiting for the backup patches to land
19:33:21 <yolanda> :)
19:33:23 <mrmartin> and have a minor issue with smartcn jar, that's all
19:33:25 <fungi> next meeting in here is the tc meeting
19:33:39 <fungi> #link https://review.openstack.org/168117
19:33:42 <mrmartin> yeah, I expect some decision, so we can move forward and finish the backup
19:33:46 <fungi> that should likely land during
19:34:01 <fungi> as soon as it does, we can approve
19:34:08 <fungi> #link https://review.openstack.org/167631
19:34:09 <mrmartin> ok, so we had a good progress
19:34:28 <clarkb> fungi: mrmartin I feel like I am missing a bunch of context here
19:34:29 <mrmartin> and I not received with any feedbacks affecting the ask.o.o site
19:34:35 <fungi> and yeah, we're running in production on an infra-managed server for ask.openstack.org now
19:34:44 <clarkb> the backup stuff I am familiar with
19:34:45 <mrmartin> clarkb: what you missing?
19:35:08 <clarkb> what does the tc meeting have to do with ask.o.o and what is the smartcn jar problem?
19:35:12 <fungi> i'll go ahead and raise the ttls for it back up to an hour shortly when i take out the other record pleia2 was asking about earlier
19:35:33 <fungi> clarkb: tc meeting is when the governance change to add that backup puppet module repo gets approved
19:35:38 <mrmartin> clarkb: we need to get an approval for pgsql_backup puppet repo from tc
19:35:40 <clarkb> oh I see
19:35:50 <fungi> clarkb: at which point we can approve the project-config change to add it
19:35:53 <mrmartin> because it is running on pgsql, not mysql
19:36:12 <clarkb> maybe we should just stackforge our puppet modules
19:36:25 <mrmartin> a smartcn issue a small bug in the puppet that affects a restarted instance I guess, I'm working on that but it is not an upfront bug
19:36:27 <mordred> or maybe we should fix the tc process for this some how
19:36:43 <clarkb> mrmartin: thanks
19:36:48 <mordred> but that's out of scope for this meeting
19:37:08 <mrmartin> we expect some upgrades in the near future for askbot, so puppetizing of the ask-staging.o.o will be the next task
19:37:17 <fungi> but anyway, i think that's it. we're basically wrapped up. eventually i'll go ahead and delete the old server once we're sure it's got nothing on it we still need
19:37:31 <mrmartin> and some discussion happens about release process of theme and askbot.
19:37:35 <mordred> yay!
19:37:55 <clarkb> awesome, nice work everyone
19:37:59 <mrmartin> thnx
19:38:12 <fungi> #topic Priority Efforts (Upgrading Gerrit)
19:38:21 <fungi> #note Gerrit 2.9 upgrade Saturday May 9, 2015
19:38:30 <fungi> good reminder
19:38:45 <fungi> probably nothing new to discuss for this other than what we covered in the action items?
19:39:06 <zaro> testing possibility to go directly to 2.10 underway
19:39:08 <clarkb> maybe note that we might upgrade to 2.10
19:39:16 <fungi> ahh, yep
19:39:34 <fungi> #note might be an upgrade to 2.10 depending on the results of testing between now and then
19:39:49 <fungi> #topic Priority Efforts (Docs publishing)
19:40:11 <fungi> still in a holding pattern while the logs on swift comes to full production i expect?
19:40:25 <clarkb> I think so, particularly that bug with non log data
19:40:32 <clarkb> but once that is sorted we should be ready to start work on this
19:41:09 <fungi> #topic Puppet Testing
19:41:16 <fungi> something about rspec and...
19:41:22 <fungi> #link https://review.openstack.org/164908
19:41:38 <fungi> nibalizer: jesusaurus: looks like you reviewed that
19:41:42 <clarkb> I think we covered this pretty well last meeting
19:41:55 <fungi> okay, so stale topic. i'll clear it off the agenda after the meeting
19:41:58 <clarkb> and asselin_ has made some initial progress on the outcome of that discussion. Basically do functional testing per module
19:42:27 <asselin_> #link https://review.openstack.org/#/c/169117/
19:43:32 <fungi> #topic Zuul layout split (tristanc)
19:43:35 <fungi> #link https://review.openstack.org/152290
19:43:37 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/zuul_split.html
19:44:07 <fungi> looks like we discussed this last meeting too. any updates or should i clear it from the agenda?
19:44:20 <fungi> seems to need reviews at the moment
19:44:33 <clarkb> I think jhesketh recently pushed an set of updates to the connections changes that jeblair wants to get in first
19:44:43 <clarkb> so probably just need to review those and get them in so this effort can proceed
19:44:57 <fungi> okay, good to know. have a link to those?
19:45:24 <clarkb> #link https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul+branch:master+topic:connections,n,z
19:45:40 <fungi> thanks!
19:45:41 <fungi> #topic Upcoming Project Renames
19:45:51 <fungi> looks like we have a handful requested
19:46:10 <fungi> murano projects are wanting to move into the openstack namespace since we had to skip them last maintenance
19:46:19 <fungi> #link https://review.openstack.org/167934
19:46:25 <clarkb> did they correct the confusion we ran into?
19:46:41 <fungi> yeah, the change has been updated, but needs more eyeballs
19:47:10 <fungi> congress projects similarly, since they got approved in last week's tc meeting
19:47:10 <fungi> #link https://review.openstack.org/169530
19:47:44 <fungi> also stackforge/tuskar-ui-extras wants to be retired to stackforge-attic
19:47:49 <fungi> #link https://review.openstack.org/169715
19:48:15 <fungi> do we want to pick a date/time to do these and also try the utf-8 conversion for review.openstack.org again?
19:48:31 <fungi> probably not this weekend what with so many at pycon all week
19:49:12 <fungi> i'm tempted to defer it a week and revisit in next week's meeting, but input from the group would be useful there
19:49:40 <mordred> yeah - let's defer
19:49:51 <clarkb> +1
19:49:55 <mordred> also - since this is being slightly more frequent what with big tent ...
19:50:14 <mordred> what do people think about making an ansible playbook we can reuse with a list of from-to names?
19:50:21 <mordred> that does all the things in order
19:50:43 <mordred> as in - if I write one, would people be opposed to it?
19:50:44 <clarkb> mordred: it should work for everything but github
19:50:44 <fungi> sounds awesome. i have a script i use right now to generate all the sql queries and filesystem changes
19:50:52 <mordred> clarkb: no reason it wouldnt' work for github too ...
19:50:55 <tchaypo> that sounds like it would make the task really boring
19:51:00 <tchaypo> I’m in favor of tasks that are really boring
19:51:01 <clarkb> mordred: yes big reason, githubs api is terrible and never works right
19:51:03 <mordred> tchaypo: ++
19:51:07 <mordred> clarkb: fair
19:51:08 <fungi> tchaypo: trust me, it's already crazy boring
19:51:08 <clarkb> mordred: it breaks at a really high rate
19:51:27 <tchaypo> boring+automated then
19:51:35 * mordred wants more boring things
19:51:46 <fungi> more lazy is more win
19:51:49 <tchaypo> boring tasks where I can sip coffee and watch a progress bar instead of trying to stay awake long enough to type out the whole command
19:52:05 <fungi> okay, so motion carries. we'll revisit next tuesday
19:52:06 <clarkb> mordred: as long as we can do it safely with github its fine, like maybe we do github move first and only move the rest of github succeeds or something
19:52:12 <zaro> do we want to schedule gerrit db migration with this?
19:52:30 <fungi> zaro: preferably do them in the same outage, yes
19:52:53 <mordred> clarkb: I'll see what I can come up with
19:52:56 <zaro> ok. might want to expand the outage window then.
19:53:00 <fungi> and perhaps also enact jeblair's change to log ssh usernames in the task list
19:53:09 <clarkb> also we could possibly reduce the need for renames
19:53:10 <fungi> yeah, we'd do a longer window if so
19:53:17 <clarkb> because in many cases renames are completely preventable
19:54:20 <clarkb> s/preventable/avoidable/ ?
19:54:34 <fungi> clarkb: as in projects created in the wrong namespace to start with or all these silly -attic moves?
19:54:44 <clarkb> fungi: both and trademark issues
19:55:01 <fungi> if we could start over, i'd do it with no git namespaces
19:55:10 <fungi> but getting there now would be painful
19:55:46 <fungi> anyway, we have a few minutes for open discussion, which this is sort of turning into anyway, so...
19:55:49 <fungi> #topic Open discussion
19:56:00 <clarkb> jenkins ssh-slaves plugin is broken
19:56:17 <fungi> we don't use that do we?
19:56:21 <clarkb> I haven't had time yet to put on my java hat (also motivation is hard to find) to work on fixing it
19:56:22 <pleia2> I'm conference+travel again Mon-Thursday of next week, speaking at POSSCON in South Carolina about how we do this open source, cross-company systems administration thing: http://posscon.org/talks/open-source-systems-administration/
19:56:47 <pleia2> I should be checking in though
19:56:49 <fungi> pleia2: ooh, where in sc? i might have suggestions for beer
19:56:55 <pleia2> fungi: columbia
19:56:58 <clarkb> so if anyone does want to wear a java hat I can point them to the bug which has the info that I have been able to collect and for everyone else keep an eye out for sluggish jenkins masters
19:57:19 <fungi> hrm, other than some poorly-air-conditioned night clubs, my familiarity with columbia is minimal
19:57:39 <sdague> fungi / clarkb if someone can get the CORS bits sorted out for the limitted status url in zuul, and put jquery visibility on the gerrit machine, the fixes for zuul status inline are here - https://review.openstack.org/#/c/170935/
19:57:40 <fungi> i think there's a halfway decent brewery i've had stuff from there though
19:57:42 <pleia2> that seems to be the consensus re: columbia ;)
19:58:01 <sdague> but the rest of the infrastructure to make it work are out of my scope for the near future
19:59:41 <zaro> would like anybody with mysql gumption to review my gerrit db migration script. https://github.com/zaro0508/gerrit-db-migration
19:59:41 <fungi> pleia2: hrm, yeah no luck jogging my memory on the brewery with web searches, though they seem to have a smattering of brewpubs since the last time i was there
20:00:00 <fungi> okay, we're at time
20:00:04 <fungi> thanks all!
20:00:05 <pleia2> thanks fungi
20:00:09 <fungi> #endmeeting