19:02:01 #startmeeting infra 19:02:02 Meeting started Tue May 20 19:02:01 2014 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:06 The meeting name has been set to 'infra' 19:02:09 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:16 #link http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-05-06-19.01.txt 19:02:24 #topic Actions from last meeting 19:02:39 clarkb: did you have beer? 19:02:46 * clarkb fix tox over beer 19:02:52 o/ 19:02:54 o/ 19:03:05 jeblair: well tox is fixed 19:03:09 yay 19:03:11 close enough 19:03:12 indeed. it was tox-fixing-beering-maness 19:03:13 beer? 19:03:14 madness 19:03:18 o/ 19:03:20 clarkb: that implies beer, right? 19:03:31 so we're waiting on a tox release, then we can unpin, right? 19:03:45 jeblair: correct 19:03:58 the fix is unreleased but I installed from source and tested with zuul's tox.ini 19:04:02 and it was happy there 19:04:10 cool 19:04:21 #topic manage-projects status (mordred) 19:04:36 so i would have assumed this topic was stale; but fungi found a borked project today 19:04:53 yah. I saw that. I'm sad 19:05:01 sdague thinks there may still be a race around gerrit creating the acl file 19:05:08 unfortunately syslog rotated into oblivion long enough ago that i can't see if it logged a reason 19:05:48 maybe we figure out an acl retry 19:05:50 ? 19:05:57 we have a lot of new projects coming up. can someone keep an eye on the logs for them? 19:06:00 that was the leading suggestion i thinkl 19:06:14 ++ 19:06:16 jeblair: yeah, i'll try to keep tabs on anything which might touch an acl 19:06:30 how close is sdague improved error handling patch to merging? 19:06:33 but the more eyes the better 19:07:01 https://review.openstack.org/#/c/94196/ 19:07:19 that might improve logged info 19:07:29 the sooner someone can spot a failed project creation symptom, the better chance we have of spotting the cause 19:08:42 well, since my comments were ignored, not immediately 19:09:25 jeblair: I thought your comments were handled 19:09:27 * anteaya looks at the patch again 19:09:36 it now returns true or false in that one place and the logic was flipped 19:09:39 my _inline_ comment was handled, not my cover letter 19:10:33 anyway... 19:10:46 we want more data though we think we know what the problem is 19:10:50 so sdague is not here right now, maybe we can pick this up when he returns 19:11:23 anyone want to go ahead and fix the problem? 19:11:43 I'm not sure I understand the problem otherwise I would volunteer 19:11:44 i can work up a patch 19:11:48 cool 19:12:16 #action fungi add a wait for acl file creation on gerrit project initialization 19:12:31 #topic OpenStack specific data in openstack_project module bug 1319746 (pleia2) 19:12:33 Launchpad bug 1319746 in openstack-ci "review.pp manifest contains specific site data for SSLCertificate attributes" [Undecided,In progress] https://launchpad.net/bugs/1319746 19:12:39 #info i'd like to raise that openstack_project manifests contain quite a bit of site specific data, e.g. hostnames. I noticed this while working on a gerrit-launchpad integration issue, had to tweak quite a lot of things and go back and forth between several layers of manifests to get a sane 'review' server in a VM. IMHO, the manifests need cleanup to put as much site specific stuff as possible in site.pp and leave opensta 19:13:04 rcarrillocruz: i think you hit the irc 512 byte limit at "leave opensta"... 19:13:18 we've had some discussion about it in the bug too, so it's worth a quick look if you haven't read it already 19:13:46 #info IMHO, the manifests need cleanup to put as much site specific stuff as possible in site.pp and leave openstack_project/manifests/* stuff as generic as possible. I volunteer for this task, happy to help... 19:13:49 so I am a bit resistant to that specifically because openstack_project is intended on being site specific 19:14:06 site.pp doesn't scale. That said a lot of the specific things we do can be generalized instead 19:14:13 though we have had a not-fully-consistent trend to put hostnames specifically in site.pp to make it easier for dev/testing our exact setup 19:14:29 then we should change the docs, since we say that openstack_project/modules/* shouldn't contain site specific stuff 19:14:37 we say that? 19:14:38 i think that openstack_project is exactly where the site-specific bits belong 19:14:44 http://ci.openstack.org/sysadmin.html 19:15:11 "no site-specific configuration such as hostnames or credentials should be included in these files" 19:15:19 ya I don't know that I compeltely agree with that 19:15:21 and "This is what lets you easily test an OpenStack project manifest on your own server." 19:15:36 #link http://ci.openstack.org/sysadmin.html#making-a-change-in-puppet 19:15:46 I would love to see things like hostnames at least be moved to vars so they can be replaced using standard puppet methods. It would allow me to stop maintaining patches to config for our internal env and just set the vars. 19:15:47 I agree with the intent which is to make things easily testable but that should be achieved by pushing things further down in the stack 19:16:27 ++ 19:16:39 anyways no need to derail the meeting. This is definitely a good subject to become more consistent on 19:16:47 concur that we still have too much useful logic living in modules/openstack_project which should get moved into the separate modules 19:17:04 that may be, but we're not there yet, and our current/old plan, flawed as it is, was at least a workable plan... 19:17:08 as optional behaviors or whatever 19:17:43 i think rcarrillocruz correctly identifies some things out of alignment with that, and we can probably accept changes to clean that up a bit 19:18:09 especially if they make it easier to further refactor in the future (and don't interfere with it) 19:18:30 some of that may also dovetail into AaronGr's work 19:19:05 and jesusaurus's 19:19:09 fungi: which work? is it a blueprint or something linkable? 19:19:29 yeah AaronGr has commented on the associated review to that bug: https://review.openstack.org/#/c/93687/ 19:20:37 i think the upshot is that openstack_project holds a lot of "site specific" stuff in the since that it only makes sense if you are developing the openstack project... 19:21:12 but "site specific" in terms of the exact hostnames, keys, etc, should be in the next layer up so that we can actually test things on something other than the prod servers 19:21:41 at least, until we have a better plan 19:21:46 yup 19:21:50 which i hope will end up in the specs repo soon :) 19:22:41 yeah, sounds like we have two different definitions of "site specific", one applies to site.pp and the other to the openstack_project module 19:22:46 rcarrillocruz: looks like they might have gotten auto-abandoned 19:22:48 we should better define that 19:22:49 #link https://review.openstack.org/#/q/status:abandoned+owner:%22Aaron+Greengrass+%253Caaron%2540greenbtn.com%253E%22,n,z 19:22:50 rcarrillocruz: best thing might be to search for patches authored by AaronGr and jesusaurus 19:22:57 oh there you go 19:23:03 ah ok 19:23:15 jesusaurus: ++ 19:23:33 the way i see openstack_project and the way I think it should be is that it contains only layout stuff that specific to openstack project 19:23:33 a lot of the massive refactoring was put on hold while we got puppetboard up and running 19:23:40 anything that are paths, hostnames, etc 19:23:41 but should be open season now 19:24:10 should either be resolved by facter , passed on via hiera or at the last resort have a sane default in the parameterized class in question 19:24:27 rcarrillocruz: ++ 19:24:46 i think it would also be beneficial to have some initial docs on how to setup a hiera for local development 19:24:59 rcarrillocruz: so i think it's okay to go ahead and fix straightforward deviations like the one that triggered this 19:25:07 k 19:25:32 rcarrillocruz: since jesusaurus and AaronGr have ideas on refactoring, you might want to work with them on that; hopefully we can do so over a spec soon 19:25:44 rcarrillocruz: i was thinking that instead of setting up a dev hiera, it would make sense for the hiera calls to have sane dev default values 19:25:47 (i mean, for larger refactoring) 19:25:58 i'm happy to help, so i will look into it and open bugs on this topic, will chat with AaronGr and jesusaurus about it 19:26:13 rcarrillocruz: stories 19:26:23 since the infra-specs repo uses storyboard 19:26:43 storyboard.openstack.org 19:26:44 anteaya: that's going to be a little tricky since config doesn't use storyboard yet; we might be able to flip it though. we should take a look. 19:26:45 anteaya: can you link me on that? not familiar on this stories workflow, i assume is some agile methodology stuff you use 19:26:51 gotcha 19:26:52 thanks 19:27:12 rcarrillocruz: we are just starting, as jeblair says there maybe some gotchas 19:27:25 this will be a guniea pig situation 19:27:36 yup. we're making this up as we go along. :) 19:27:43 #topic Addition of elastic-recheck link 93610 & 93608 (pleia2) 19:28:02 this should be quick, is it ok to have both rechecks commands on status.o.o for now so it's discoverable until we rm Rechecks? 19:28:20 it's been 8 months or so since the page went live and it's still not easy to find 19:28:32 s/commands/links 19:28:46 jogo: sdague: ^ ? 19:28:55 i've resisted this in the past because i think we should only have one link 19:29:01 (noting that sdague may not be around just yet) 19:29:07 i think it makes sense for status.o.o to have links to both rechecks and elastic-recheck 19:29:25 i disagree -- the rechecks page needs to die. 19:29:43 i think we should just have it link to elastic-recheck and at the moment the content on the new page is superior enough that it ought to be safe to do so 19:29:56 yes elastic-recheck is where the love is right now 19:30:04 fungi: i agree but sdague does not, he still wants the old page 19:30:27 so here's the thing -- all the docs about what to do when your patch fails say to use the rechecks page and say nothing about e-r 19:30:31 do we have a roadmap of what's still needed? 19:30:37 can recheck be a link on the elastic-recheck page? 19:30:46 which makes the rechecks page the current process 19:30:51 oh 19:31:07 i don't want that to be the case, but as long as it is, it's the thing that should be on the navigation bar 19:31:13 what is involved in changing the docs to point to elastic-recheck? 19:31:24 do we know? 19:31:32 anteaya: changes submitted in gerrit, and wiki edits 19:31:33 anteaya: yes, there was a summit session about it 19:31:44 sorry 19:31:51 it was an infra session. was i the only one of us there? 19:31:54 should be pretty easy if it's just a link and some text 19:32:00 https://etherpad.openstack.org/p/juno-summit-elastic-recheck for reference 19:32:08 i was there... just trying to dig up the etherpad now to referesh my memory 19:32:13 ahh, thanks 19:32:19 #link https://etherpad.openstack.org/p/juno-summit-elastic-recheck 19:32:46 "sdague: Analysis of existing data for usefuleness of recheck bug # comments / filing "recheck bugs"" is the relevant thing 19:33:02 if sdague comes back and says it's useless, we just switch everything over 19:33:17 ok 19:33:20 so line 26 in the etherpad 19:33:22 if he says it's important, then we need to do more work 19:33:35 yup, so the stuff under the "Replace existing /rechecks/ page" bullet item essentially 19:33:38 but in the meantime while we do more work, I kind of want to see ER in the header along with the other 19:33:59 i really hope he says it's useless, and we can just drop it, and even drop the "recheck bug ###" syntax. 19:34:08 pleia2: feels like that isn't a decision we can make right now 19:34:08 pleia2: ++ 19:34:20 jeblair: agreed 19:34:24 anteaya: yes, this is after we hear from sdague :) 19:34:31 pleia2: kk 19:34:46 pleia2: i'm really opposed to exposing the complexity of two systems to all of our developers. i just want one or the other, and a plan to get there. 19:35:01 the lack of a navigation link is a constant reminder that the new system is "experimental and not in production". 19:35:16 i agree with the one-link sentiment 19:35:21 alright 19:36:31 #topic Single url for Third Party information (anteaya) 19:36:49 so in the summit session on third party testing 19:36:54 #link https://etherpad.openstack.org/p/juno-infra-improving-3rd-party-testing 19:37:08 we made it a requirement for ci accounts to have a wikipage 19:37:24 it would be nice if we could decide where these links could be indexed 19:37:42 and also various programs have documentation around third party testing 19:37:48 So I had some chats with folks about 3rd Party CI, is there any objecetion to running an environment for reasons other than testing out a product? We'd like to run tests against an IPv6-enabled environment 19:37:52 it would be great if we had one url for both 19:38:16 aveiga: we are addressing a single url for third party right now 19:38:21 sorry 19:38:31 it is okay, open discussion is coming up 19:38:41 anteaya, both 3rd party service account wiki pages and ? 19:38:53 anteaya: i think we should have a common prefix in the wiki; so wiki/ThirdPartySystems/NameOfSystem 19:39:00 ++ 19:39:02 I like that 19:39:03 ++ 19:39:08 i think you can easily get lists of things like that 19:39:11 ++ 19:39:15 then the requirements we formalized should go in infra docs so they re reviewed 19:39:26 okay 19:39:29 yep, that all sounds great 19:39:39 lgtm 19:39:41 do we want a single url for indexing all the wikipages? 19:39:54 including the acl changes we discussed (which means researching what projects different systems are voting on changes for as well) 19:40:01 folks in the third party meeting yesterday indicated that would be favourable to them 19:40:13 anteaya: the prefix page should solve that 19:40:27 yup so wiki/ThirdPartySystems/ would be the index 19:40:34 anteaya: i'm pretty sure there's a mediawiki macro you can add to the parent page to list all child pages 19:40:38 oh yes, I see it now, thanks 19:40:42 Agreed on prefix page asindex 19:40:47 cool, i will look at that 19:40:50 anteaya: do you want to mock that up, maybe create a template page for people to copy? 19:41:00 anteaya: then we can take a look at that before we send it out? 19:41:01 might need to bug Ryan_Lane 19:41:02 It's real easy. I could help. 19:41:18 jeblair: I like it, I will mock it up on an etherpad 19:41:25 ++ 19:41:30 anteaya: oh i think you should just use the wiki 19:41:30 rockyg: thanks that would be great 19:41:37 in fact, it's possible to create a mediawiki template (not just a template page) which can get included into a new page as a directive to prepopulate it 19:41:38 (it is a wiki after all) 19:41:38 oh I can do that too 19:41:41 +1 wiki 19:41:57 fungi: cool, I will look into that, I didn't know that 19:41:59 used to do that all the time at $oldjob 19:42:08 cool, I will bug you 19:42:15 I think I have enough to make me happy 19:42:17 thanks 19:42:32 i'll have to look the syntax back up, but it's great when you've got specific information you need people to put in new pages 19:42:39 * anteaya nods 19:42:46 sounds like ti will fit the bill here 19:42:53 i think we should get the wiki page/templates in order, update our docs with our new reqs and link in any project specific pages from the overview wiki page. then when all of that is in order, send an announcement to the 3rd party folks asking them to update to the new requirements. 19:43:06 I like it 19:43:07 (i think that will be the most orderly way to handle this). 19:43:21 I will try to get our stuff drafted up this week for review 19:43:28 sounds good to me 19:43:32 anteaya: thanks 19:43:49 #action anteaya work on third party wiki pages and infra docs 19:44:14 #topic infra-specs repo 19:44:34 I'm abusing my position as meeting chair to insert a last minute topic; but it's important ;) 19:45:02 we're adoping a specs model as many of the other projects are 19:45:10 #link http://git.openstack.org/cgit/openstack-infra/infra-specs/ 19:45:11 there's a change here with the initial template: https://review.openstack.org/#/c/94440/ 19:45:31 give that a once over and see if there's anything missing... 19:45:56 we have a lot of unique concerns that other projects don't have (we run servers!) 19:45:56 will do 19:46:04 what exactly is the specs model? 19:46:10 and we will be the first ones trying to use this with storyboard 19:46:14 jeblair: was 94440 cookie-cuttered from the specs template as a starting point? 19:46:17 jesusaurus: great question! 19:46:19 (mostly just curious) 19:46:22 fungi: yes, but heavily altered 19:46:29 cool 19:46:47 jesusaurus: basically, it's early design review of proposed enhancements 19:47:11 jesusaurus: so instead of showing up with a bunch of code and then having people say "why didn't you ...?" 19:47:39 jeblair: how specs relates to blueprints? 19:47:40 which we already do a lot of in various ad-hoc manners (lp bug comments, irc logs, et cetera), so this actually should be easier overall for keeping track of what we decide 19:47:42 jesusaurus: we get everyone on the same page by writing up a detailed proposal before starting major work 19:47:45 they seem similar to me 19:48:10 rcarrillocruz: other openstack projects are using blueprints to track the implementation progress of specs 19:48:15 rcarrillocruz: we'll be using storyboard to do that 19:48:27 rcarrillocruz: blueprints are generally fairly light on detail, and expect you to link to somewhere you've published the actual detail 19:48:42 and these would be that "somewhere" 19:48:55 gotcha, thanks for the clarification 19:49:07 rcarrillocruz: (the intent is that the rest of openstack will use storyboard when it's more robust, we're trying to use it early to work out the bugs) 19:49:08 jeblair: cool. this sounds similar to the DESIGN file i try to put at the top level of my personal projects 19:49:30 jesusaurus: quite likely so 19:49:56 though these are targeted at specific feature enhancements, not overall project philosophy 19:50:02 the hope is that if we're doing this right, it shouldn't be any more work than we currently put into discussing designs (potentially even less work) 19:50:28 but better aggregated into a workflow we already use for similar tasks 19:50:38 and easier to publish consistently too 19:50:47 http://git.openstack.org/cgit/openstack/qa-specs/tree/ 19:50:47 http://git.openstack.org/cgit/openstack/nova-specs/tree/ 19:50:48 those are the earliest adopters, so you can see some real examples there 19:51:09 ooh, we should add documentation impact... 19:51:25 fungi: ++ 19:51:55 you also want to talk about specs naming conventions 19:52:20 uh oh better shovel my blueprints from lp/tempest into qa-specs 19:52:32 the more you know 19:52:41 i'd like to see a security impact too (for similar reasons we have it in specs for other projects) 19:52:56 ++ 19:52:57 still hunting for where that gets stipulated 19:53:15 it isn't in that patchset 19:53:32 we can move that to review of the initial template :) 19:53:43 clarkb: agreed 19:54:01 (and to answer my question, i guess it would go into template.rst) 19:54:46 Correct. What about testing dependencies? 19:54:53 Hey, aveiga and I were hoping to get a chance to talk ipv6 - know that things are a bit hectic and bust 19:54:54 *busy 19:55:09 did we lose jeblair? 19:55:11 but we sort of got ping-ponged over to this meeting from yesterday's third party 19:55:13 Course, I'll just leave a comment on the spec for the spec.... 19:55:29 ugh 19:55:30 #topic project renames 19:55:30 anyone want to schedule a gerrit downtime to deal with the accumulated backlog of renames? 19:55:30 maybe this friday? 19:55:46 poor jeblair 19:56:07 that was from 5 mins ago :( 19:56:08 I will be around friday and can do downtime. Weekend is a holiday weekend though so bad for me 19:56:15 i could do it friday afternoon edt, but have an errand to knock out in the morning 19:56:27 fungi: cool we can do it then 19:56:34 sounds good 19:56:38 I'm not much help during project renames but I will be around Friday aft 19:56:43 and yes, having people over this weekend for cookout stuff, but could probably work around that if needed 19:56:52 i'll put together an email then 19:56:56 #topic open discussion 19:57:11 aveiga: sc68cal whats up 19:57:25 OK so quickly 19:57:33 aveiga: I don't think there is any specific requirement that 3rd party testing is only used for products BUT... 19:57:34 are we going to disable drafts? 19:57:45 zaro: yes, I will draft an email today 19:57:51 I believe we'd love to test ipv6 in the main gate (although I may be wrong) 19:57:55 sc68cal: so the general idea is that only things that physically _can't_ run in our upstream testing should be tested in a third party system. 19:58:01 and we can do more testing of it. disabling them does not require a downtime so we can do it whenever 19:58:05 mordred: we'd like to make sure the tests don't break gate first 19:58:06 aveiga: as far as ipv6-specific testing, is there anything you need tested which we should be testing upstream? do you have some details on what you're wanting to do? 19:58:09 hence 3rd party 19:58:11 We want to start doing a spike to help get more v6 tests in Tempest on the neutron side, as well as have a comcast Ci system that has our network layout 19:58:13 sc68cal, mordred: and yeah, sounds like something we should test upstream. 19:58:26 aveiga: we have a great way of bootstrapping new tests 19:58:31 offering it up since we have a prod setup, plus a lab 19:58:34 aveiga: http://ci.openstack.org/test-infra-requirements.html#test-run-styles 19:58:41 since parts of Neutron are dependent on the physical network layout 19:59:13 we need to point out that current IPv6 patches won't work with an l3 setup though 19:59:21 they need a provider net 19:59:29 until the patches land to support l3 agent 19:59:29 sc68cal: well, for our tests neutron is running within a vm along with the rest of devstack, so we can build out the virtual network devstack sits on however is needed to exercise that 19:59:49 fungi: true - but we have a deployment model where the network is not virtual 20:00:00 uses a hardware switch + provider network api ext 20:00:10 So that's the use case we'd use 3rd party ci to cover 20:00:16 sc68cal: right, so a dependency on some specific hardware would make third-party testing a necessity 20:00:29 since what you're testing is how openstack works with that hardware 20:00:38 however we should be testing ipv6 upstream anyways 20:00:39 this is why we'd like to run it as 3rd party until all patches land 20:00:44 sc68cal: however, i think we should really focus on upstream testing here -- we should be gating on this sort of thing 20:00:49 we are at time 20:00:50 ++ 20:00:57 jeblair: agree - we are attempting to make 99% upstream 20:00:59 can we move to the -infra channel and continue? 20:01:06 and our special 1% in 3rd party 20:01:06 aveiga: sc68cal: we can move to #opensatck-infra 20:01:08 jeblair: clarkb agreed, but until the RA daemon patches land you need something to initiate addressing 20:01:10 ok 20:01:12 k 20:01:19 er, #openstack-infra 20:01:21 thanks! 20:01:24 #endmeeting