19:02:04 #startmeeting infra 19:02:04 Meeting started Tue Sep 23 19:02:04 2014 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:04 o/ 19:02:05 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:07 The meeting name has been set to 'infra' 19:02:09 #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:09 o/ 19:02:16 Morning 19:02:18 #link last meeting http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-09-16-19.01.html 19:02:21 Hi all 19:02:28 #topic Actions from last meeting 19:02:31 o/ 19:02:31 jhesketh rework 109485 to impact only infra jobs 19:02:39 hey everybody 19:02:56 oh let's just jump to swift logs 19:02:59 #topic swift logs 19:03:23 jhesketh: still working on 109485? 19:03:50 #link https://review.openstack.org/#/c/109485/ 19:03:53 I put it up as a new review, let me dig up the link sorry 19:03:56 #link https://etherpad.openstack.org/p/swift_logs_next_steps 19:04:01 jhesketh: 122154? 19:04:55 clarkb: yep, that's it 19:05:01 #link reworked 109485 is 122154 https://review.openstack.org/#/c/122154 19:05:21 updated etherpad 19:05:49 okay, so that's something we should review soon to unblock this 19:06:24 looks straighforward too. I will review post meeting 19:06:42 * SergeyLukjanov lurking 19:06:54 anything else on swift-logs? 19:07:28 122159 is also related but not blocking 19:07:50 #link https://review.openstack.org/#/c/122159 19:07:57 jhesketh: can you add that to the etherpad too? 19:08:00 Otherwise I think we just want to review how switching over infra logs goes 19:08:20 sounds good 19:08:39 anteaya: yep, I will later :-) 19:08:43 jhesketh: thanks 19:08:59 #topic Config repo split 19:09:08 so first part of this is this spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet-modules.html 19:09:18 nibalizer: did you start a storyboard story for that? 19:09:38 he's puppetconfing today, not sure if he's here (I found corner to do meeting :)) 19:09:48 i don't see one 19:10:05 this is pretty much blocked on someone filing a story, and a task for each project so that people can actually get started on it 19:10:10 any volunteers to do that? 19:10:23 i didn't do that, sorry 19:10:42 ill do it now 19:11:06 #action nibalizer file a story with a task for each project for http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet-modules.html 19:11:22 the second part of this is http://specs.openstack.org/openstack-infra/infra-specs/specs/config-repo-split.html 19:11:41 i believe we are actually ready to go on that 19:11:48 i have a series of puppet changes up 19:12:01 and anteaya has prepared a strawman project-config repo 19:12:05 I think I have the repos in the two stages they need to be in 19:12:24 #link https://review.openstack.org/#/q/topic:project-config,n,z 19:12:30 one after the git filter branch and one after the filter branched has been reorganized 19:12:35 assuming changes look good to everyone, do we want to pick a day this week to freeze and do it? 19:12:38 so that is just needing review? 19:12:39 #link https://github.com/anteaya/project-config 19:12:47 #link https://github.com/anteaya/reorganized-project-config-02 19:12:49 clarkb: I believe yes 19:13:04 jeblair: so sequence is land new anteaya repo, then land config changes, then land change to config to remove stuff? 19:13:12 and please review the two git repos as well, so I can fix anything that is incorrect 19:13:23 mordred: with a freeze in the middle 19:13:25 yah 19:13:41 freeze as step #0 i think 19:13:41 wrap that whole thing in "obtain write-lock" 19:13:43 i think anteaya is anticipating that we would do the reorganization after the project-config import 19:14:00 that is what I have planned 19:14:09 since the reorg commit will be big 19:14:17 and would prefer to have it done via gerrit 19:14:21 rather than locally 19:14:22 gotcha 19:14:25 open to opinons 19:14:34 mordred: so that's: land anteaya new repo, land project-config reorg, land puppet changes, land change to config to remove stuff 19:14:56 i think that will work fine 19:15:00 anteaya: ^ 19:15:04 wfm 19:15:07 I haven't done the config change to remove stuff, but I can 19:15:20 and that is also a config rename at the same time, is it not? 19:15:44 config -> system-config 19:15:46 we need to create the gate jobs for the project-config repo; that's not done yet, 19:15:52 ah 19:16:00 that plan sounds good to me 19:16:00 anteaya: depending on your preferences, you may just want to script up the steps to do the delete-stuff change. otherwise it's going to conflict heavily between now and when we freeze 19:16:14 though it might be easier to add the jobs after the repo exists 19:16:24 I can work on the commands to do the delete stuff change 19:16:30 repo first, then jobs, i think 19:16:35 and put up a repo for review on github 19:16:43 anteaya: we do want to do a config -> system-config rename, but it doesn't have to happen at the same time; we probably need a few more puppet changes for that. 19:16:43 then offer the patch in the freeze 19:17:00 so, as far as scheduling goes... 19:17:06 okay I can do a config delete stuff patch then 19:17:14 i'm going to be away for 1.5 weeks starting saturday 19:17:21 yeah, i think we're better off planning for a breather between the split and the rename just to shake out oddness 19:17:41 kk 19:17:53 how about I do up an etherpad of steps 19:18:03 based on what we have identified 19:18:09 and then we can go from there 19:18:10 which means we (a) rush it in this week; (b) you do it without me; (c) we do it after i get back (> oct 8) 19:18:23 which do you prefer? 19:18:33 I don't feel rushed if we do it this week 19:18:40 just wednesday is bad for me 19:18:43 this week seems reasonable to get the split knocked out, and as long as everyone else isn't also taking vacation i expect we can handle issues which crop up once the split bakes in production while jeblair enjoys a much-deserved vacation 19:19:02 I concur with fungi 19:19:03 I'm booked today and tomorrow, but should be available to be helpful thursday and friday 19:19:14 as well as next week 19:19:17 fungi: ya 19:19:18 thursday or friday is good for me too 19:19:19 thursday seems like a good day 19:19:30 why don't we try for thursday then? 19:19:35 I'm out thursday for a holiday, but you can probably do without me :) 19:19:40 i'm open every day this week, and around for the forseeable future until the summit 19:19:45 it will take me about 45 minutes after the freeze to do the filter branch 19:19:52 kk 19:20:02 so the freeze should be at least 2 hours 19:20:03 pleia2: thursday's a holiday? 19:20:07 I think we can be frozen for the whole day if need be 19:20:14 rosh hashana 19:20:17 eating and temple 19:20:17 fungi: rosh hashanah 19:20:26 aha! yes, i totally forgot that was this week 19:20:53 so 1900 utc on thursday? 19:21:03 * jhesketh will be around to help where he can 19:21:04 how about we freeze starting at 00:01 utc thursday 19:21:11 and I will have an etherpad to track set up 19:21:11 (which is wed evening for most of us) 19:21:18 jeblair: oh I'm fine with that 19:21:19 sounds fine by me 19:21:29 jeblair: sounds good 19:21:32 a day-long freeze for config changes should be bearable to the project 19:21:42 if it's not, the project can learn patience 19:21:42 then we start the work when people wake up the next "day" 19:21:47 heh 19:21:48 clarkb: yep 19:22:42 i guess we'll remind jhesketh, SergeyLukjanov and me not to approve any config changes starting at 0 hours thursday utc 19:23:04 should remind all of us :) 19:23:05 Noted 19:23:26 indeed 19:23:36 when should we target unfreezing? 19:23:45 "when it's done?" 19:23:52 heh, works for me :) 19:23:53 I'm for that 19:24:09 done and reasonably seeming to be not-broken 19:24:22 yes to the non-brokenness 19:24:31 #agreed freeze project config changes 00:01 utc thursday sept 25 19:24:31 ++ 19:24:46 party goes until question marks 19:24:51 #agreed cutover to project-config repo thursday morning us-time 19:24:54 all invited to attend 19:25:16 anteaya: that means you can do your work wed night or thurs morning, whichever works better 19:25:42 I was just thinking I can have the new repo and repo regorg done my wed night 19:25:54 #action jeblair send project-config announcement to -dev list 19:26:03 then if jhesketh wants some fun he can do the jobs patches during the night, his day 19:26:20 and the great merging can happen thursday daytime north america time 19:26:25 jhesketh: does that work for you? 19:26:52 anteaya: yep, I can help review if you remind me :-) 19:26:59 jhesketh: thanks 19:27:12 do you want to merge in some config changes before Thursday? 19:27:14 anything else on this one? 19:27:32 AJaeger_: good point, we should try to clear out as much of those as possible today and tomorrow 19:27:33 I mean: Should we clean the queue as much as possible - or not? 19:28:19 we probably should, yes 19:28:19 and perhaps ask in your announcement mail to not submit new changes... 19:28:40 or "don't expect them to be merged until..." 19:28:51 fungi: can you be around wednesday night to merge the repo with manage projects? then we can offer patches to it like the reorg patch 19:28:54 eh, just warn that new changes proposed may need reworking or abandoning and reproposing to a different project after the split 19:29:01 fungi: yeah 19:29:31 anteaya, fungi: i think we can do that thurs morning 19:29:32 since folks won't read it anyway and it will give us something to point to when we tell them that afterward 19:29:35 jeblair: okay 19:29:54 ;) 19:29:54 anteaya: jeblair: yeah i think we do that with the other changes in sequence. no need to split them up overnight 19:30:01 fungi: kk 19:30:10 in fact, you doing the prop wed night; and fungi merging it thurs morning may work out really well timing wise 19:30:16 s/prop/prep/ 19:30:20 okay 19:30:23 if we're impatient, there are ways to speed up the patch taking effect when we're working on it 19:30:28 ya 19:30:35 no, just wanting to be efficient is all 19:30:39 not impatient 19:31:03 #topic Nodepool DIB 19:31:09 yay! 19:31:16 i meant if we get impatient because it's blocking us merging other changes depending on that existing 19:31:19 so this is sorta happening :) 19:31:27 wip 19:31:29 jeblair++ 19:31:33 fungi: ah yes 19:31:46 I am hoping that by this afternoon we will have restarted nodepool and have our first image built 19:31:47 we ran into an error in production, fixed it and some other things after local testing, and i think we're about ready to try in prod again 19:32:14 i still haven't built an image locally, and i'm not sure why; i suspect it may be a disk space issue, but the error output is not helpful :( 19:32:30 i can help/do a nodepool restart after this meeting, since i'll mostly just be lurking the tc/project meetings at that point 19:32:33 at any rate; i don't think it's going to kill production 19:32:42 jeblair: I'd love to learn more about what broke for you 19:32:48 me too 19:33:02 fungi: ya me too. watiing on the logging change to merge though 19:33:23 2014-09-23 18:31:51,010 INFO nodepool.image.build.devstack-trusty-dib: umount2: Invalid argument 19:33:23 2014-09-23 18:31:51,093 INFO nodepool.image.build.devstack-trusty-dib: umount: /tmp/image.lyVl2cy4: not mounted 19:33:32 mordred: that's the end of the dib log for me. :/ 19:33:45 jeblair: that looks like the exit cleanup, real issue will be before that 19:34:02 (seen that a fair bit :) 19:34:51 ianw: ok; i don't see anything error-like immediately before it 19:35:06 yah - it's like devstack - the errors at the end are not the real errors 19:35:15 last thing it does is 2014-09-23 18:31:45,092 INFO nodepool.image.build.devstack-trusty-dib: Caching cirros-0.3.0-x86_64-disk.vhd.tgz file from https://github.com/downloads/citrix-openstack/warehouse/cirros-0.3.0-x86_64-disk.vhd.tgz in /home/nodepool/.cache/image-create/source-repositories/cirros_0_3_0_x86_64_disk_vhd_tgz_c610756fad56eab78721f8601c631e88396b6a31 19:35:26 it doesn't say it errored, but it doesn't say it completed either 19:35:32 or at least they weren't real errors until someone added errorexit 19:35:54 so maybe dib needs to run under errorexit like devstack does ;) 19:35:55 best hypothesis: it ran out of space on that but didn't happen to mention it. :/ 19:36:06 jeblair: i've found important stuff is sometimes missing, have a change out to enable better tracing : https://review.openstack.org/119023 ... getting it merged is a pain though 19:36:42 oh no gertty crashed! 19:36:54 * jeblair files bug 19:37:02 SpamapS, lifeless: ^^ 19:37:05 anyway, i guess we'll try it in production soon 19:37:10 ++ 19:37:21 if it is a disk issue we should be fine on the current nodepool server 19:37:25 o/ 19:37:27 if it isn't then we keep debugging 19:37:36 dib should be set -e 19:37:45 Like, thats a bug, I'd happily triage that as High priority 19:38:19 #topic Jobs on trusty 19:38:33 er, do we have any current work going on with this? 19:38:42 jeblair: pypy switched 19:38:46 yeah, it's slowed a bit but progressing 19:38:56 we did move all the pypy jobs to trusty last week, yes 19:39:01 py34 is still in progress. the upstream bug for the py34 gc bug has an assignee now 19:39:18 any reviews need attention? 19:39:21 the main blocker bug for 3.4 is now picked up by barry so presumably new package coming to trusty soon 19:39:30 cool 19:39:37 #link https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1367907 19:39:38 Launchpad bug 1367907 in python3.4 "Segfault in gc with cyclic trash" [High,In progress] 19:39:43 all of the outstanding reviews at this point are in non-infra projects 19:40:03 #link https://etherpad.openstack.org/p/py34-transition 19:40:05 is up to date 19:40:24 #topic StoryBoard Migration (krotscheck) 19:41:05 krotscheck: is the migration script ready for use? 19:41:52 let's assume so. :) 19:42:06 do we want to move all the infra projects at once? 19:42:17 I think so 19:42:19 yah 19:42:19 i lean toward yes 19:42:28 if we are going to deal with the pain might as well jump into the deep end 19:42:39 more opportunities to get through issues too 19:42:43 i will go mad switching back and forth constantly otherwise 19:42:43 jeblair: I believe krotscheck said that it quite successfully loads his local instance with tons of data 19:42:43 and we close all the bug-trackers on launchpad except for openstack-ci, which we will leave open for e-r tracking only 19:43:02 jeblair: we could make an elastic-recheck tracker 19:43:08 and close openstack-ci 19:43:22 i'm guessing we move existing openstack-ci bugs to openstack-infra/config (soon to be system-config) 19:43:29 ++ 19:43:36 i agree 19:43:46 mordred: it's not for bugs in elastic-recheck, it's for "infra bugs" that elastic-recheck sees 19:43:46 Sorry 19:43:51 Was talking with zaneb 19:43:52 zaro 19:44:06 SO the migration script landed with https://review.openstack.org/#/c/122047/ 19:44:17 I’ve been able to load zuul and storyboard with it. 19:44:39 isn't storyboard already in storyborad? 19:44:51 There’s a couple of misfiled bugs on launchpad. 19:44:54 (ooh, there he is again -- that story borat) 19:45:06 jeblair: AAH. gotcha 19:45:09 does the migration script preserve bug number to identical story number or is there some indirect mapping? 19:45:10 krotscheck: oh, what project are they filed against? 19:45:26 jeblair: openstack-infra/storyboard 19:45:36 krotscheck: on launchpad 19:46:01 fungi: It does not maintain numbers, but it does keep track of an internal cache so duplicates aren’t imported on a failure. 19:46:09 so - two thoughts ... 19:46:21 krotscheck: (i'm confused because i think you said you moved storyboard bugs over, but i don't think there should be storyboard bugs in launchpad) 19:46:27 a) maintaining numbers would be nice ... however b) if we do that, we'll need to manage auto-increment-index 19:46:38 just curious how we look up the new story for a known lp bug number 19:46:43 mordred: i kind of think maintaining numbers is critical for openstack 19:46:43 storyborat - I see an April's fools in the making 19:46:51 maybe not 19:46:55 jeblair: I do too 19:47:05 because there are going to be a ton of patches up that reference the old number 19:47:05 okay, so what's the story there? 19:47:14 jeblair: Lemme go find those 19:47:15 and feel free to just rtfm me to where this is doc'd 19:47:17 I have a thought in my head of how we can do it 19:47:24 that I can write up and propose 19:47:33 maybe use a field to map storyboard # to launchpad #? 19:47:35 foreign keys! foreign keys! 19:47:42 mordred: okay, maybe we should make sure we know what the whole story is there before we actually import infra 19:47:44 does an autoincrementing index not jump values already present? 19:47:57 because we're maybe about to do something irreversible 19:48:05 clarkb: it does - we'll just need to alter the table and set the next value for auto-increment after we do the import 19:48:15 there's the additional complication of two imports 19:48:31 i think i already have patches that reference storyboard #s 19:48:39 because we don't want to autoincrement in a space that has numbers we're going to pull 19:48:41 we don't want storyboard's new very-high autoincrement (because of the infra import) to start using story numbers that exist in lp and might even be an openstack bug later 19:48:48 right 19:48:50 do we have a bot that stores quotes? " because we're maybe about to do something irreversible" deserves to be passed to history 19:48:50 mordred: ++ 19:48:54 which is why we set the index 19:49:02 mordred: That… worries me, because if we import one project whose last bug is ##, and autoincrement to ##+1, but that’s owned by a different project which is imported later(tm).... 19:49:02 reed: we should have one 19:49:07 right 19:49:11 mordred: gotcha 19:49:12 this is the thing I keep saying 19:49:18 we need to set the autoincrement index 19:49:22 for all of the above reasons 19:49:32 we can do this in one of two ways ... 19:50:05 I remember this discussion in Brussles being ‘well, by the time our normal storyboard autoindex hits the first bug filed for openstack, we’ll be old men. 19:50:12 have the imported bugs keep numbers and manage new bugs with letter-number combo? :) 19:50:13 we can set it back to what it was before the import - or we can just bump it up to well above the current max 19:50:25 s/(men)/wo\/($1) 19:50:27 I think I would vote for setting it low 19:50:34 mordred: yeah, that sounds like it will work 19:50:43 I'd liek to write this up in a sane way and send it out 19:50:44 (i've never done that, but i take your word it can be done in mysql) 19:50:47 and not try to explain it all here 19:50:52 mordred: sounds good 19:51:04 Righto 19:51:11 I think I know what you mean. 19:51:20 #action mordred to write up autoincrement plan for infra bug import 19:51:52 anything else on this? 19:52:06 Which project do you want to move first? I want to do test runs on a local instance. 19:52:22 krotscheck: i think we want to move them all at (approximately) the same time 19:52:24 krotscheck: I'd try test runs on openstack-ci - since that's got the most stuf 19:52:30 we should probably rustle up a list of what "all" means :) 19:52:35 jeblair: ++ 19:52:50 indeed. 19:53:09 anyone want to take a stab at that? 19:53:15 I can handle that. 19:53:34 #action krotscheck make list of infra projects in launchpad for krotscheck to use in testing 19:53:44 i don't think it's actually that many 19:53:56 Right, I’ll ping in the infra channel to get people’s opinions 19:54:02 ++ thanks 19:54:19 end of topic? 19:54:26 it is for me. 19:54:29 #topic Publish devstack.org content under infra (anteaya) 19:54:57 anteaya: ? 19:55:00 so at the beginning of Juno we had a meeting agenda item about this and agreed this was something we wanted to do 19:55:18 then we got bogged down in the devstack.org > foundation domain name change 19:55:21 oh hey the foundation owns the domain now :) 19:55:28 which jbryce has confirmed has happend 19:55:31 yes they do 19:55:34 so we are off again 19:55:40 what do we want to do? 19:55:53 did we decide on a home for the content to live? 19:56:00 if memory serves, we wer at the point of figuring out servers and redirects 19:56:07 we did not that I recall 19:56:10 if not, we should ask dtroyer where it should go 19:56:11 but we wanted to 19:56:19 eg, in devstack itself, or in a new "devstack-org" repo 19:56:28 oh the code is in devstac 19:56:48 anteaya: yeah, in the gh-pages branch 19:56:53 dtroyer made the source code change in the spring, it lives in devstack/docs 19:56:58 oh ok 19:56:59 cool 19:57:10 now where do we want to serve it up? 19:57:32 then yeah, we just need some jobs to publish it to, probably at this point, a vhost on static.o.o ? 19:57:44 #link http://git.openstack.org/cgit/openstack-dev/devstack/tree/docs/source 19:57:54 (eventually, probably a vhost on publish.o.o, but that's part of the docs publishing spec and work hasn't started on that yet) 19:58:33 and fungi I told jbryce you would let him know when we are ready for a domain name record change 19:58:55 is it not managed in the usual way? if so, we should be able to change it. if not, i'm not interested in doing this. :) 19:59:15 anteaya: yeah, he'll presumably need to repoint the authoritative dns to rackspace's nameservers once we set up a zone for it 19:59:16 oh 19:59:27 jeblair: ah sorry, I don't know 19:59:29 it's still pointed at different authoritative servers 19:59:33 assuming it isn't there already 19:59:53 I don't know what the usual way is 20:00:01 and jbryce may or may not either 20:00:08 so yeah, let's say the first step is getting it attached to the openstack account in rackspace cloud so we can manage dns for it 20:00:11 yeah, it's being served by not-rackspace dns servers at the moment 20:00:21 fungi: can you do that then? 20:00:23 jeblair: sounds good 20:00:38 sure, #action me bob 20:00:49 ha ha ha 20:00:50 #action fungi get devstack.org served by openstack rackspace dns account 20:00:55 i'l need to figure out what sequence rackspace wants it to happen in 20:01:01 thanks everyone, we're at time! 20:01:04 #endmeeting