19:03:22 #startmeeting infra 19:03:23 Meeting started Tue Feb 28 19:03:22 2017 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:03:24 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:03:26 The meeting name has been set to 'infra' 19:03:31 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:03:41 #topic Announcements 19:03:47 clarkb: want to #info the zanata maintenance here? 19:04:14 sure 19:04:16 we can cover general discussion about it later 19:04:38 #info upgrading translate.openstack.org to Xenial + java 8 + zanata 3.9.6 Wednesday March 1 at 2300 UTC 19:04:51 thanks! 19:04:53 this adds a bunch of features that the translators have been asking for 19:04:58 as always, feel free to hit me up with announcements you want included in future meetings 19:05:10 #topic Actions from last meeting 19:05:25 #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-02-14-19.03.html Minutes from last meeting 19:05:33 "1. (none)" 19:05:40 and done 19:05:47 #topic Specs approval 19:06:13 we don't seem to have anything new up this week, though there are a few hanging out there that could use some eyeballs 19:07:11 #link https://review.openstack.org/434951 Stackalytics Persistent Cache 19:07:42 that one in particular could stand to get some attention since it is basically the last bit standing between us and being able to move to running stackalytics.openstack.org as production 19:08:09 so anybody with an interest in stackalytics, please weigh in 19:08:15 fungi: isn't that something better for the stackalytics devs to weigh in on? 19:08:24 I guess I don't really feel like I'd be a great reviewer 19:08:50 well, we need to review it from the standpoint of whether this is a fit for the current automation and configuration management issues we have with the service 19:09:05 I can take a look, only because I helped get stackalytics.o.o online. But would agree with clarkb , we should ping core reviewers on stackalytics 19:09:24 but yes, the implementation on the stackalytics side would still need buy-off from their maintainers 19:10:08 mrmartin: ^ reminder that the stackalytics devs should also take a look at that spec 19:10:37 #topic Priority Efforts 19:10:57 nothing called out specifically here for this week, but we got a lot tackled at the ptg last week 19:11:19 #topic Apache workers on static.o.o (ianw) 19:11:47 hi, this has caused several problems for me lately 19:12:11 looks like the first change linked there has already merged 19:12:18 people reporting job failures, and on monday at least two reports for static.o.o and docs.o.o of intermittent connections 19:12:45 did i link https://review.openstack.org/#/c/426639/ ? 19:12:57 #link https://review.openstack.org/426639 Switch static.openstack.org to worker MPM 19:12:58 I gave a +2 a while back, thought we were ready to move it into production 19:13:09 didn't +3 as couldn't monitor 19:13:25 yeah, i just didn't want to go fiddling with it without consensus 19:13:33 oh, right, it didn't get workflow +1 yet, i misread 19:14:10 +2. is mpm-event in xenial? 19:14:21 #link http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-02-26.log.html#t2017-02-26T23:57:30 discussion of job failures and user downtime 19:14:24 jeblair: its the default in xenial iirc 19:14:39 so this might be short lived if we organize our xenial virtual sprint 19:14:43 i believe that the xenial version has, however, put in fixes for this particular issue 19:15:41 ++ 19:15:57 alright, if no major issues then i might babysit it this afternoon when it's quiet. in theory, at least, it should just be an apache restart 19:16:22 yeah, i can't personally participate in the xenial upgrades sprint until at least week-after-next but hope we can do that rsn 19:16:38 ianw: sounds fine to me. go for it 19:17:24 anything else we need to cover on this topic? 19:17:35 nope /EOT 19:18:06 #topic migrate github/bagpipe-bgp into openstack/networking-bagpipe (pabelanger) 19:18:28 so, clarkb and I talked a little about this topic this morning. I think we have it under control now. 19:18:39 #link https://review.openstack.org/438573 migrate github/bagpipe-bgp into openstack/networking-bagpipe 19:18:45 I wasn't sure if we had an automated process for re-import or it was manual 19:18:50 looks like manual wins out 19:19:04 so, we just need to schedule this with project owner I think 19:19:08 right, there's no automation since we expect it to be a rare occurrence 19:19:17 great, thats all I had 19:19:36 excellently brief topic! 19:19:49 #topic translate.openstack.org upgrade (clarkb) 19:20:00 ohai 19:20:18 juts wanted to let people know that things are going well on this and plan to run through it tomorrow 19:20:24 #link https://etherpad.openstack.org/p/translate.o.o-upgrade translate server upgrade maintenance plan 19:20:34 translate-dev has been ahppy on new code once I igured out how jboss and zanata are different 19:20:41 ianychoi has tested translate-dev and is happy with it 19:20:48 so now we are ready to do production 19:21:00 is translate-dev back to authenticating against openstackid-dev.o.o now? 19:21:09 fungi: there is a change up for that but I don't think it has merged 19:21:12 let me find it really quick 19:21:34 nevermind, i just tested and it's not 19:21:35 https://review.openstack.org/#/c/419667/ 19:21:59 please read through the etherpad steps and let me know if you think anything is missing 19:22:07 #link https://review.openstack.org/419667 Switch to openstackid-dev for translate-dev 19:22:12 we are doing it at an apac friendly time so that ianychoi and others can help test once done 19:22:13 i'll go ahead and approve that one now 19:23:00 I can be on standby if needed 19:23:45 23:00 utc wednesday. i _may_ be around or might show up partway into the maintenance 19:24:04 hard to know what dinner plans will be with family here 19:24:36 but i'll review the changes you've got linked in the pad at least 19:24:42 fungi: I don't mind doing it 19:24:48 thanks pabelanger! 19:24:58 thanks 19:25:11 #link https://review.openstack.org/438738 Add translate01 to cacti 19:25:36 #link https://review.openstack.org/438737 Update db creds for translate01.o.o 19:25:52 note ^ is WIP because I don't want new server talking to db until we are ready to switch 19:26:06 yep, i assumed so 19:27:04 looks like it hasn't actually saved a db dump yet 19:27:39 i guess that merged more recently than utc midnight 19:28:37 nothing in the etherpad is jumping out at me as overlooked or concerning 19:28:49 oh it should've done a db dump since I manually ran one 19:29:20 fungi: I ran both the mysqldump command and bup command from cron (in that order) to jump start the backup process 19:29:25 the /var/backups/mysql_backups dir is empty except for what looks like a logrotated empty file 19:29:36 on translate01 19:29:37 fungi: on translate.openstack.org? 19:29:42 ya translate01 does not have backups yet 19:30:04 translate.o.o had never had backups so I wanted those in place first, then will transition backups to 01 when we switch 19:30:30 got it. i saw it had the dump job in its crontab, but i guess it doesn't have access to the production db yet anyway 19:30:37 correct 19:31:59 you're going to use zuul enqueue-ref to test/prime teh periodic translation jobs? 19:32:19 ya 19:32:35 well not enqueue-ref since they re periodic iirc 19:32:37 enqueue? 19:32:49 enqueue needs a change id 19:32:54 and patchset 19:33:07 enqueue-ref can be provided a branch tip or whatever 19:33:11 ah ok 19:33:24 though i guess the ref itself is irrelevant in that case 19:33:52 right just need to say run nova's periodic translation jobs early 19:33:53 i haven't personally tried manually enqueuing a periodic job in zuul. you might find you have to compose a trigger-job.py invocation for each one instead 19:34:12 but hopefully the zuul cli works in this case 19:34:13 otherwise they run around 0600 UTC which is quite a bit later 19:36:28 I will sort out what the command to enqueue periodic jobs is 19:37:24 might save you some time later 19:37:36 okay, anything else on this? if not, open discussion ensues 19:38:06 I don't have anything else. ianychoi has done a bunch of user side testing and I think I have the system side sorted so expect it to be mostly happyness (now I have jinxed it) 19:38:53 well, here's hoping that if nothing else, we only need to upgrade to zanata 3.9.6 once 19:39:19 #topic Open discussion 19:39:37 for open discussion more and more groups seem interested in upgrading our nodepool builders 19:39:41 we might want to schedule that oto 19:40:35 in service of assembling a ptg summary, i've seeded an etherpad from the zuul v3 accomplishments mentioned in yesterday's zuul meeting 19:40:38 #link https://etherpad.openstack.org/p/infra-ptg-pike-recap brainstorming pad for ptg recap 19:41:19 people who worked on other things during the ptg (storyboard, translations tooling, job failure diagnosis, et cetera) please add some bullets! 19:41:19 clarkb: agreed, should be something we can roll into production 19:41:37 looking at the puppetry for builders it apperas that we should already work with systemd except for possibly needing to do the systemctl command to reload units (in this case our sys v scripts) 19:41:42 We'd need xenial for epensuse images too 19:41:47 Hi, Mirantis currently hosts Murano's CI environment. They've expressed interest in moving it to me. Would it be possible to have infra do this? 19:42:32 felipemonteiro: does it do anything special or have any special requirements which they were unable to upstream previously? 19:42:47 clarkb: can you elaborate on 'upgrading our nodepool builders'? 19:43:24 jeblair: ianw/dib have wanted us to upgrade to xenial from trusty to be closer to how dib is running CI. And SUSE is interested in building suse images but that requires zypper install that works which isn't available on trusty but is on xenial 19:43:26 adding xenial-based builders 19:43:27 fungi: I doubt it. Serg Melikyan told me it's "1 medium sized VM with Jenkins and 1 hardware node with 96 RAM". 19:43:38 jeblair: so adding xenial based builders, then deleting the trusty ones 19:44:06 gotcha 19:44:13 felipemonteiro: the hardware node with 96 ram would be the thing that might make running the jobs in infra tricky 19:44:21 felipemonteiro: is that node just being used as a source of vms? 19:44:36 (I believe I remember Serg was doing things with libvirt builders) 19:44:41 clarkb, fungi, pabelanger: sounds good to me. we wanted to only change one variable at a time with nodepool. i think we're good to change the next now. ;) 19:45:01 felipemonteiro: you probably want to look at https://docs.openstack.org/infra/manual/testing.html to confirm there are no known show-stoppers for you 19:45:31 felipemonteiro: mordred fungi I think what we'd want to do is port the jobs over into our infra and not host a separate jenkins 19:45:36 #link https://docs.openstack.org/infra/manual/testing.html Test Environment documentation 19:45:37 clarkb: yes 19:45:37 jeblair: ya, I'm not rushing nodepool-builder swap. Maybe in a few weeks :) 19:45:44 clarkb, felipemonteiro: yes. that is 19:46:18 i'm happy to take as an action item getting a xenial builder into some pre-production state 19:46:30 i would like that for dib reasons 19:46:32 clarkb: also s/not host a separate jenkins/not host a jenkins at all/ 19:46:47 ianw: cool, I think that would be a good first step to just make sure that systemd and friends work due to funnyness around how puppet deals with that 19:47:04 #action ianw launch a "beta" nodepool builder on xenial 19:47:15 ianw: I half expect our post puppet reboot to fix that for our first install, but in general case it may not work since you need a systemctl load-units or whatever to pick up sysv scripts 19:48:50 if that's the only systemd issue, i think we call ourselves lucky :) 19:48:53 mordred: Thanks. I'll check with Mirantis to see if these requirements are good enough for murano-ci. 19:49:06 felipemonteiro: so I think you want to read over the document that fungi linked to understand the limitations involved, then if you can work with those start porting your jobs into our CI by adding new jobs like everyone else 19:49:14 felipemonteiro: awesome. 19:49:52 clarkb: You mentioned how you don't want to host a separate Jenkins...does this mean you want everything infra if I understood correctly? 19:50:05 in infra 19:50:09 felipemonteiro: yes 19:50:16 felipemonteiro: right we would not take over the setup by running a new jenkins for you. Instead you would need to run your jobs as first party jobs in infra 19:50:38 felipemonteiro: either all of the jobs should just be ported into our project-config repo and run as normal jobs, or none of it should be run in infra 19:51:11 I had a question regarding setting up CI gates for a new(ish) project 19:51:21 if the jobs are already written in jenkins-job-builder, then porting them over into project-config should be _fairly_ easy 19:52:19 blancos: feel free to ask, or hop over to #openstack-infra after the meeting and ask there 19:52:50 mordred: I see. I'll try to get more information regarding whether it's just the ci jobs that are hosted by Mirantis. I have no objection with moving everything to infra. 19:53:15 ++ 19:53:18 felipemonteiro: cool. if it's possible, it certainly seems like a good direction! 19:54:19 clarkb: I guess the question is really just procedural; i.e., what's needed from me? 19:54:51 blancos: you'll need to propose changes to openstack-infra/project-config that adds jobs in jenkins/jobs and then tells zuul to run them via zuul/layout.yaml 19:55:34 #link https://docs.openstack.org/infra/manual/creators.html#add-basic-jenkins-jobs Project Creator’s Guide: Add Basic Jenkins Jobs 19:55:52 blancos: ^ not sure if you've been following that guide 19:55:55 fungi: Thank you :) 19:56:07 http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/ 19:56:17 http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/ 19:57:27 ayoung: those are fun 19:57:42 sorry wrong room 19:57:55 ayoung: no problem. thought maybe you were reporting a problem/need wrt those 19:58:14 nah, more as a solution to other problems fungi 19:58:34 okay, open discussion seems to be winding down. i'll go ahead and polish up the ptg recap this evening. don't forget to add your highlights to the recap etherpad if you have any 19:58:47 thanks everyone! 19:58:49 I'm going to enter a cave this afternoon to do track chair duties for summit 19:59:00 I will try to watch irc for fires but really need to get ^ done before dealdine 19:59:15 i have family in town still so am mostly not around 19:59:41 and that basically brings us to time for the tc meeting, up now! 19:59:44 #endmeeting