19:04:09 <jeblair> #startmeeting infra 19:04:10 <openstack> Meeting started Tue Sep 9 19:04:09 2014 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:04:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:04:14 <openstack> The meeting name has been set to 'infra' 19:04:21 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting 19:04:23 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-09-02-19.01.html 19:04:24 <fungi> (and now for something completely different) 19:04:28 <SergeyLukjanov> o/ 19:04:31 <jeblair> #topic Priority Specs (jeblair) 19:04:34 <krtaylor> o/ 19:04:42 <pleia2> o/ 19:04:52 <jeblair> hey, so we like totally merged those specs 19:05:09 <jesusaurus> yay 19:05:13 <jeblair> #link http://specs.openstack.org/openstack-infra/infra-specs/ 19:05:13 <fungi> showed them what's for 19:05:16 <nibalizer> yay! 19:05:17 <clarkb> o/ 19:05:21 <pleia2> yay 19:05:26 <nibalizer> also i have a todo to update my license 19:05:30 <nibalizer> because i screwed that up 19:05:36 <mordred> o/ 19:06:00 <fungi> nibalizer: you're no longer licensed? 19:06:10 <jeblair> i think the thing to do now with this time is to start tracking implementation of those and other important things we're working on 19:06:12 <nibalizer> fungi: i need to be re-registered in oregon 19:06:23 <anteaya> an unlicensed nibalizer 19:06:24 <jeblair> (we have a few things in progress that predate specs but probably would have been) 19:06:38 <nibalizer> a review to split out a module: https://review.openstack.org/#/c/119543/ 19:06:44 <jeblair> i'm thinking specifically about puppet 3 migration, config repo split, nodepool dib, logs in swift, docs publishing 19:07:12 <fungi> ...finishing moving jobs to trusty 19:07:33 <jeblair> fungi: right, the 34 work is in service of that, right? 19:07:40 <fungi> yeah 19:07:53 <fungi> or did you mean specifically things that are already specs? 19:08:07 <jeblair> fungi: nope, including something big that's pre-spec like that is good 19:08:30 <jeblair> fungi: i want to make sure we make forward progress on our priorities, and that we're good at communicating those 19:08:34 <fungi> right, that. i just reread your list ;) 19:08:36 <clarkb> ++ 19:08:38 <jeblair> and i think moving to trusty is one of our priorities 19:09:05 <jeblair> so maybe next week we should expect to have agenda items for each of those 19:09:17 <fungi> sounds like a good idea 19:09:31 <jeblair> we'll need to coordinate with jhesketh on logs/swift, or potentially have a meeting annex later in the day 19:09:37 <anteaya> sounds like a candidate for an action item 19:09:51 * nibalizer can keep updates for puppet3 migration as that goes forward 19:10:55 <jeblair> since that's already on todays agenda, and we have a looming deadline, let's go ahead and switch to that 19:11:03 <jeblair> #topic Puppet 3 Migration 19:11:18 * mordred welcomes his puppet3 overlords 19:11:19 * anteaya is curious aboaut the looming deadline 19:11:26 <nibalizer> so sept 30 is the day puppet 2.7 goes EOL 19:11:33 <anteaya> that would be it 19:11:34 <mordred> oh crikey 19:11:39 <nibalizer> jeblair: has stated that we will not run puppet if its eol 19:11:47 <mordred> ++ 19:12:00 <nibalizer> there is a patchset chain that paves the way for us to build a p3 master 19:12:06 <jeblair> #link https://review.openstack.org/#/c/117604/ 19:12:08 <anteaya> that is exactly 3 weeks from today 19:12:19 <nibalizer> after that rooters can run nodes against the p3 master with --noop and inspect the breakage 19:12:37 <nibalizer> it may be possible to move many nodes over immediately, others may require some refactors 19:12:40 <clarkb> looks like the beginning of that stack has merged 19:12:56 <clarkb> nibalizer: did the gerrit change merge? 19:13:00 <clarkb> have a link to that one handy? 19:13:13 <nibalizer> also there is a tool in puppet that lets us generate catalogs, so we could generate 2.7 and 3 catalogs and diff them 19:13:14 <jesusaurus> we could also create puppet3 nodes to run the apply test on to inspect breakage 19:13:45 <nibalizer> and wikipedia built a tool to operationalize that as a websevice, i have no experience with either 19:13:56 <nibalizer> jesusaurus: i created https://review.openstack.org/#/c/107250/ but got stuck on 'how do i jjb' 19:14:01 <anteaya> any current insurmountable obstacles? 19:14:14 <nibalizer> clarkb: the gerrit change did not merge i think, but it isn't required, those are only deprecation warnings 19:14:18 * jesusaurus looks at 107250 19:14:23 <clarkb> nibalizer: oh right 19:14:41 <nibalizer> clarkb: https://review.openstack.org/#/c/107243/ 19:15:25 <clarkb> sounds like reviewing the remainder of https://review.openstack.org/#/c/117604/ then getting 107250 going are the next steps? 19:15:37 <nibalizer> at any rate i think getting the chain that ends at https://review.openstack.org/#/c/117604/ to a place where it can land, then landing it, then building the p3 master should be our top priorities 19:16:01 <jeblair> nibalizer: sounds good; maybe we can get that merged this afternoon and i can spin up a p3 master again 19:16:13 <nibalizer> jeblair: that would be great 19:16:29 <anteaya> nibalizer: given we have 3 weeks to finish this, where do you want to be by next week? 19:16:53 <nibalizer> anteaya: with a p3 master up, and rooters checking nodes with noop 19:16:58 <jeblair> ++ 19:17:05 <anteaya> kk 19:17:16 * fungi volunteers to help test 19:17:22 <nibalizer> if we dont have a p3 master by next meeting, thats an oshitsignal 19:17:28 <jeblair> indeed 19:17:43 <fungi> the sooner we get the easy bits out of the way, the more time we have left to fix the harder stuff 19:17:56 <nibalizer> exactly 19:18:06 <clarkb> sounds good to me 19:18:13 <nibalizer> i dont want to burn meeting time theorizing what the hard bugs would be, but yea 19:18:18 <jeblair> if we have to prioritize nodes, we can do so for the gerrit/jenkins/zuul conglomerate 19:18:25 <nibalizer> ya 19:18:30 <jeblair> those are the ones we want to keep updating with the puppetmaster 19:18:41 <jeblair> if paste loses its connection to puppet27 master, we'll live :) 19:18:43 <nibalizer> the storyboard node, the puppetdb node those machines could have puppet just disabled on them 19:18:55 <anteaya> does the -dev ml need a heads up or just announce once it is completed? 19:19:15 <jeblair> anteaya: this should be invisible to developers 19:19:19 <anteaya> very good 19:19:25 <fungi> i don't think the -dev subscribers are likely to particularly care unless we break something 19:19:53 <clarkb> ya this should be a backend chagne no one ever knows we made 19:19:56 <jeblair> anything else on puppet3? 19:20:04 <clarkb> unless they are reconsuming our puppet so maybe a note to the infra list 19:20:17 <anteaya> yeah, ci might care 19:20:31 <nibalizer> nothing else on p3 19:20:39 <jeblair> #topic Manila project renaming (fungi, bswartz) 19:21:05 <jeblair> fungi: istr you communicated with someone about this? 19:21:14 <fungi> i talked to vponomaryov1 and he said it would be fine for us to just pick a date/time and then give the manila devs a heads up on the -dev ml 19:21:32 <fungi> and bswartz later agreed when he popped in to ask an unrelated question 19:21:58 <fungi> so... do we want to rename this weekend? hold off until there are other things we need to add to the pile? 19:22:11 <fungi> do it on a friday now that ff is behind us? 19:22:21 <clarkb> this weekend is hard for me to do. but I won't stop you guys from doing it 19:22:24 <fungi> though the gate still is a bit submerged 19:22:36 <mordred> this weekend is also tough for me 19:22:49 <fungi> hopefully by later this week it will have lightened up considerably 19:22:51 <jeblair> so friday or kick it down the road 19:22:55 <mordred> although I could do sunday before noon central time 19:23:08 <fungi> i'm honestly fine either way, just want to make sure it doesn't fall through the cracks indefinitely 19:23:28 <fungi> sounds like maybe we should just revisit next week 19:23:37 <anteaya> I'm for later, since p3 testing this week sounds like a priorty 19:23:43 <anteaya> and might leak into friday 19:24:19 <jeblair> okay, let's kick it down the road and see what sept 19-21 ends up looking like 19:24:28 <fungi> yeah, table it for now 19:24:39 <jeblair> or shelve it 19:24:41 <jeblair> #topic Fedora/Centos testing updates (ianw 09-09-2014) 19:24:53 <ianw> hey 19:25:04 <ianw> firstly nodepool 19:25:30 <ianw> wondering if any thoughts on https://review.openstack.org/#/c/118939/ (Ignore min-ready when at capacity) 19:26:26 <jeblair> ianw: cool, we should look at that since it may help with operational problems; sorry i haven't had a chance 19:26:39 <ianw> ok 19:26:41 <jeblair> ianw: we did reduce min-ready to minimize that problem, but i'd like to put it back 19:26:59 <jeblair> i think we may need to do a nodepool restart soon to help mordred with the dib work 19:27:11 <mordred> jeblair: dib work schmib work 19:27:19 <jeblair> so maybe we can try to push on that review and maybe get it in too 19:27:29 <ianw> on the bare-f20 nodes, last week was happy to wait for postgresql puppet release to enable these nodes, but that didn't happen 19:27:49 <ianw> https://review.openstack.org/117397 and dependent 19:27:56 <clarkb> ianw: ya we should probably just go ahead with the git installed version 19:28:26 <nibalizer> so my concern with that is that it will break the puppetdb server 19:28:35 <nibalizer> but that wouldn't actually break anything else i think 19:28:47 <clarkb> nibalizer: because puppetdb uses postgres? 19:28:50 <clarkb> arg 19:29:03 <nibalizer> im +1 on the change 19:29:06 <nibalizer> like lets do it 19:29:18 <nibalizer> if it breaks, chances are good it would have broken on the stable release anyways 19:29:28 <clarkb> nibalizer: what do we need to do to fix puppetdb? and will it be puppetdb the service or just puppeting of puppetdb 19:29:30 <jeblair> nibalizer: you don't think it will break the postgres testing on the gate workers? 19:29:42 <nibalizer> jeblair: ianw said he tested that 19:29:51 <jeblair> rock on 19:30:02 <nibalizer> also why did they take the travis badge off the postgres module? shame? 19:30:19 <mordred> nibalizer: maybe travis ran out of nodes 19:30:38 <nibalizer> but the little travis.png isn't on their readme 19:30:49 * anteaya is amazed to see jeblair type rock on 19:30:49 <fungi> mordred: you said we have more than twice the test capacity of travis at this point? 19:31:04 <nibalizer> hax: https://travis-ci.org/puppetlabs/puppetlabs-postgresql 19:31:24 <nibalizer> okay so yea tests are green 19:31:24 <clarkb> nibalizer: is it puppetdb the service that will break or puppeting of puppetdb server? 19:31:25 * fungi keeps meaning to find out where you got the capacity info for travis, but can defer that discussion to not-meeting 19:31:34 <clarkb> nibalizer: because puppetdb the service breaking completely hoses all of puppet 19:32:02 <nibalizer> clarkb: i can run the beaker tests locally for the puppet module 19:32:12 <nibalizer> that will give us "high confidence" that the git version is not busted 19:32:22 <clarkb> I thought it aws busted? 19:32:24 <nibalizer> since thats essentially what puppetlabs does to vet a release 19:32:25 <clarkb> I am confused 19:32:31 <nibalizer> clarkb: i see green in the travis 19:32:38 <clarkb> basically what I want to prevent is that our puppetdb will break 19:32:41 <clarkb> bceause that breaks everything 19:32:42 <nibalizer> clarkb: no i getcha 19:32:44 <ianw> clarkb: busted for fedora 20 19:33:11 <clarkb> nibalizer: ok. earlier you said puppetdb would break 19:33:20 <nibalizer> clarkb: it might 19:33:37 <nibalizer> idunno, i cant answer for all potentialities 19:33:58 <clarkb> thats fine but earlier you made it sound more definitel 19:34:09 <nibalizer> i was just calling out it as a dependency 19:34:32 <jeblair> okay, so it sounds like we're ready to try to collapse that wave function! 19:35:01 <ianw> so what was the conclusion? 19:35:07 <fungi> give it a shot 19:35:08 <nibalizer> i think we should merge it 19:35:31 <nibalizer> clarkb: if postgres really does crap itself we can switch puppetdb to use the in memory database until we fix postgres 19:35:39 <nibalizer> but i dont think that will happen 19:35:41 <clarkb> nibalizer: ok. 19:35:51 <clarkb> ianw: nibalizer: is that git sha1 the one we want to use still? 19:36:20 <clarkb> I assume that is the one that was tested so probably 19:36:48 <ianw> clarkb: that is the one i tested and got a good run with on f20 ... 19:36:54 <clarkb> great 19:37:08 <clarkb> last question. :) should we remove the regular module install from install_modules.sh? 19:37:16 <clarkb> I think its ok as is but will end up doing more work 19:37:27 <nibalizer> i think we want to use a1edd262d84f97a33ff64b8c542cffc5fa6d98c1 19:37:51 <nibalizer> oh i hadn't thought that ianw had done testing 19:38:16 <clarkb> ianw: see inline comment on the change otherwise I agree lets do this 19:38:33 <jeblair> seems like we should get rid of the regular module? 19:38:55 <clarkb> ya I think it is most correct to do so 19:39:07 <nibalizer> +1 to deleting the regular module 19:39:44 <nibalizer> ianw: how hard was testing the new postgres module with the rax node you used? 19:40:34 <ianw> nibalizer: not super hard 19:40:56 <nibalizer> okay, can you test on the tip of master then? it looks like hunner fixed the tests, since they are passing and the patch is called 'patch specs' 19:41:51 <ianw> nibalizer: i can, but i wouldn't like to be having this same discussion next week... 19:42:21 <ianw> ok, review updated with that removal 19:42:26 <nibalizer> that makes sense 19:42:32 <nibalizer> consensu is important 19:42:50 <jeblair> so maybe lets go ahead and merge with the tested change, and maybe ianw can also test master if he has a chance 19:42:57 <mordred> ++ 19:43:01 <jeblair> that will hopefully help us avoid breakage as they get closer to release 19:43:20 <ianw> i can do that 19:43:33 <nibalizer> yea lets merge ianw's change 19:43:55 <ianw> i've been keeping the centos7 images up-to-date https://review.openstack.org/#/c/118246/ (Update to newer Centos7 images) 19:44:01 <ianw> in disk-image-builder 19:44:23 <ianw> reviews are a little slow 19:44:50 <ianw> last time i added -core to it, and got back responses like "don't add core to changes" 19:45:21 <ianw> why this matters for infra is if d-i-b is a key part of nodepool, getting changes through in a timely manner is important 19:45:39 <sdake_> ianw that always annoyed me- reason being it means the change effects all of core and should be voted on by the entire cor eteam 19:45:45 <mordred> nod 19:45:53 <sdake_> that was "core" concept of it but no talways how people use it 19:45:54 <jeblair> ianw: indeed, using dib was supposed to free us from having to wait for operators to upgrade 19:46:40 <clarkb> it is always interesting to see how groups use errit differently 19:47:08 <ianw> sdake_: so my opinion is a bit different, if you're part of -core then i feel like you should be interested in all changes. but yeah, different groups, etc 19:47:20 <clarkb> ianw: ++ 19:47:47 <fungi> i honestly mostly don't notice if authors add me as a reviewer on changes (at least insofar as it would affect the urgency of review) 19:48:04 <jeblair> fungi: indeed, gertty lacks that 'feature' :) 19:48:36 <fungi> ;) 19:48:41 <ianw> well either way, timely sync between d-i-b and infra seems like it's going to be of increasing importance 19:48:46 <clarkb> ianw: indeed 19:48:48 <jeblair> ianw: thanks for mentioning it though; we'll want to keep an eye on that 19:49:42 <wenlock> cd 19:49:46 <wenlock> mt 19:50:01 <ianw> alright, well not much else for me, thanks 19:50:14 <jeblair> ianw: thanks! 19:50:14 <jeblair> #topic Open discussion 19:50:21 <pleia2> yolanda and I are giving some OpenStack CI talks at Fossetcon late this week (Friday and Saturday), if we could get our changes slides in so they're published by then it would be very helpful, particularly for yolanda's since no version is up on the publications site yet (my sysadmin ones are close enough) https://review.openstack.org/#/q/status:open+project:openstack-infra/publications,n,z 19:50:32 <pleia2> also, this means I'm gone tomorrow - next tuesday, will be back on Wednesday 19:50:51 <clarkb> pleia2: have fun 19:51:00 <pleia2> thanks :) forecast is hot and muggy, woo florida 19:51:08 <anteaya> enjoy yourself 19:51:27 <anteaya> I'll be away tomorrow and intermittent on Thursday, personal errands 19:51:28 * jeblair subscribes to publications. oops. 19:51:46 * mordred mocks jeblair 19:52:13 * jeblair wonders if mordred's mocking covers all interfaces 19:52:23 * mordred segfaults 19:52:42 <jesusaurus> lol 19:52:55 <nibalizer> pleia2: gl! 19:53:08 <clarkb> if all goesaccording to plan I will be working from a new place in oregon next week 19:53:14 <pleia2> nibalizer: thanks, I'll bug you when I'm back to review some puppetconf slides :) 19:53:15 <mordred> ++ 19:53:18 <clarkb> so I may be working off of a phone tether for a bit 19:53:21 <anteaya> clarkb: happy transfer 19:53:26 <pleia2> clarkb: yay, congrats 19:53:29 <mordred> clarkb: just wander over to puppetlabs and steal wifi from them 19:53:44 <anteaya> that leaves zaro alone in an office, yeah? 19:53:48 <nibalizer> pleia2: are you speaking at puppetconf!? puppetconf is gonna be siiiiiick 19:53:49 <clarkb> with krotscheck 19:53:55 <pleia2> nibalizer: I am! 19:54:03 <krotscheck> Wait, wha? 19:54:07 <mordred> anteaya: nah, jesusaurus and krotscheck are there ... unless jesusaurus is back to portland too... 19:54:12 <clarkb> and i have to run now to dlea with apartment stuff back shortly 19:54:24 <jesusaurus> mordred: not yet. but im aiming at heading that way in november 19:54:24 <krotscheck> Oh. Yes. I’ll be in the office. 19:54:27 <anteaya> ah cool, glad zaro still has company 19:54:31 <anteaya> krotscheck: great 19:54:45 <mordred> krotscheck: soon it will just be you and zaro up in there 19:55:04 <anteaya> in an office with tables where the pencils roll off 19:55:10 <nibalizer> but PDX will be full of awesome people 19:55:21 <mordred> nibalizer: _full_ is a strong word 19:55:31 <mordred> nibalizer: there will be a cadre of awesome people and then a bunch of hipsters ... 19:55:47 <krotscheck> mordred: s/awesome people/hipsters/ 19:55:56 <mordred> :) 19:56:30 <jeblair> is discussion of hipsters the natural terminus for all our conversations? 19:56:37 <jeblair> think about that over the next week 19:56:38 <anteaya> +1 19:56:40 <jeblair> thanks everyone! 19:56:43 <jeblair> #endmeeting