18:01:21 #startmeeting Openstack Operators 18:01:22 Meeting started Wed Dec 17 18:01:21 2014 UTC and is due to finish in 60 minutes. The chair is mfisch. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:23 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:01:25 The meeting name has been set to 'openstack_operators' 18:01:44 anyone here for the ops meeting? 18:01:46 whoa, help included 18:01:47 o/ 18:01:49 I suspect it may be quiet 18:01:52 o/ 18:01:57 o/ 18:01:59 Here, while in a work stand up. 18:02:06 here 18:02:07 welcome 18:02:08 here 18:02:17 here 18:02:27 the agenda is here: https://etherpad.openstack.org/p/openstack-operators-meeting-171214-agenda 18:02:44 I'd actually like to cover the mid-cycle meetup first 18:03:08 Unless Tom is here, the info I have from last week is that Tom working to schedule it now 18:03:15 cool 18:03:17 Does anyone have anything more firm on a date and time? 18:03:35 (that info is from Dec 9) 18:03:45 is there a location? 18:03:49 not that I've heard 18:04:03 I will email Tom and poke him again 18:04:24 ask him to post on operators list if they still need a venue. if htat’s needed, i may see if we can host it in phoenix 18:04:37 i never went to a mid-cycle, is it better than summit? as an op 18:04:46 I'd offer in New Zealand but it's a bit far 18:04:57 wfm but probably not my travel dept 18:05:06 #action mfisch to email Tom Fifeld (sp) and figure out when the ops midcycle meet up is 18:05:14 I'll just CC the list 18:05:20 cool 18:05:41 #topic sample config files 18:06:02 did anyone participate in the discussion yesterday that happened in the cross-project meeting? 18:06:19 i did 18:06:20 * harlowja a little 18:06:24 I also did 18:06:28 kris was sorta there 18:06:46 so I believe that the outcome was that devs know its a problem for us and are actively investigating/proposing solutions 18:06:51 but that no final agreement was reached 18:07:10 I watched it, but didn't catch everything. 18:07:18 #link http://eavesdrop.openstack.org/meetings/crossproject/2014/crossproject.2014-12-16-21.01.log.txt 18:07:23 yeah i think the outcome is that someone was going to reply to the operators thread with some potential solutions. to get a feeling for what people prefer 18:07:26 there was desire to define a way to easily generate the docs, that's common across all the projects 18:07:36 I believe fungi owns the action 18:07:41 right 18:08:04 and that he's going to start the converstion on the ops list as well, I know many of us do not subscribe to dev 18:08:04 yep, i will 18:08:19 full plate, but hopefully later today 18:08:30 I have to say personally that this issue has come a long way since the bug I commented on wrt this was summarily closed 18:08:58 anything else on this mdorman or klindgren ? 18:09:10 (i'd suggest reading the minutes for the first half of that link) 18:09:13 don’ think so. we’re happy with the progress as well 18:09:26 at least the discussion is happening 18:09:27 in fairness, bugs opened without actionable solutions are often summarily closed. that doesn't mean they're non-negotiable and can't be reopened 18:09:28 i'd hopefully be able to propose another idea, but i'll wait for the ML proposal thread ;) 18:09:51 the only other hting - is I would like to not use tox do to do that generation 18:09:51 I'm okay with many options, I dont usually want to make pain for the devs 18:10:05 but if tox is jsut calling a script - I can call the script as well 18:10:13 pain for devs is ok, lol 18:10:24 tox is doing things inside a venv which helps keep the systme from getting dirty 18:10:27 whats the reason for no tox? I know I have mine 18:10:40 seems like ideally the sample config files would both get generated the same way, and done centrally so everyone doesn't have to do it themselves. 18:10:47 mfisch: tox keeps things clean in venv 18:10:54 because tox install all the requirements for the service into a venv 18:11:04 i get the impression the main use case for not doing it under tox is so that the version you generate locally on your server only has the relevant options based on the versions of libraries you have installed 18:11:11 and the versions tox installs via pip probably wont match what is installed on the system 18:11:39 fungi, exactly 18:11:42 however, if we ship sample configurations from upstream, our ci will likely generate them under tox just because that's simple. we can still make sure tox isn't required 18:11:42 I dont quite understand the scope of the libraries problem, how many config options come from libs? 18:11:44 fungi: for such use cases, if you would override requirements.txt with your own pinned versions, that would be great 18:11:46 in that way, tox is independant of the distro - isn't that a good thing? 18:12:07 mfisch: oslo.db, oslo.messaging?, keystonemiddleware 18:12:39 thx 18:12:40 if there is concern about generated docs not matching system, that's definitely an argument against upstream providing pre-generated docs 18:12:46 xavpaice: independent of the distro is a good thing if your services you're configuring are also independent of the distro 18:13:04 jlk: there's also value for me in seeing changes over time 18:13:09 xavpaice: looks to not be a good thing if your distro has older versions of libs than the ones found on pip, you won't have a sample config file fitting *your* environment 18:13:31 I would say you *might* not 18:13:45 I dont have a handle on how bad it would/could be... 18:13:49 I thought requirements.txt can be set to specific versions - in which case we set a min supported version there 18:13:54 can tox be taught when making the venv to include system libraries when possible? 18:13:58 klindgren: is your use case more package building? thats a bit different from mine 18:14:00 doesn't help if newer versions change things though 18:14:04 right, the point being that if you regenerate samples yourself then you have the option to generate them in the environment you're configuring, rather than using samples generated in an abatract/artificial environment disconnected from your deployment 18:14:06 so sounds like there are two issues, first being generating reference level configs and config docs for specific releaes (plus probably master), and the other being able to easily generate sample config files for specific environments 18:14:17 agreed 18:14:20 I think we have 2 or 3 use cases: those running against distro lib versions, ones running against trunk, ones running against pinned versions of libs 18:14:36 an idea; why haven't people thought that configs shouldn't change between major versions of packages? 18:14:58 harlowja: when should they change then? 18:15:07 *sorry in-between major versions 18:15:10 harlowja: wouldn't that effectively prevent /any/ changes from config files happening? 18:15:11 ah 18:15:13 they can change in the next major versions 18:15:15 the third thing which was brought up was an ability to see how the samples change over time, though i'm not sure that's 100% tractable since it's nonlinear/branching depending on library releases not tied to the application being configured 18:15:17 don't think we started the meeting btw 18:15:23 retr0h: Error: Can't start another meeting, one is in progress. Use #endmeeting first. 18:15:26 I did it 18:15:30 or maybe we did :) 18:15:31 retr0h: I started it 18:15:40 mfisch: awesome - ignore me 18:15:44 jlk example, configs don't change for oslo.messaging in oslo.messaging 1.0 -> 1.999 (for example) 18:15:46 it would be good to see changes in the configs, for when we upgrade packages 18:15:58 yes 18:15:58 just as u wouldn't expect an API to change, configs are somewhat like an API 18:16:05 +1 18:16:22 harlowja: I can buy that, if we're talking about "not changing in stable/juno" 18:16:30 usually the release notes are pretty good in that respect 18:16:40 depends, oslo libraries don't really have stable/juno, but have version numbers 18:16:42 Should a git repository be the only place/reference for those sample config files? We have a matrix of possibilities. I don't see git answering this need if we have a plethora of combinaison. 18:16:44 harlowja: definitely a topic for the oslo team to spearhead. i'm not at all opposed to option stability 18:16:46 but for versions that are essentially "next release" that could be different, or the other thing will happen, the version nubmers will just change more frequently. 18:16:56 to clarify, they shouldn't change in a non-backwards compat way. You can clearly introduce new options w/o breaking things in some cases. 18:16:58 I think lots of operators are not using stable branches 18:17:00 fungi it seems to be they would self-stabalize over time anyway 18:17:04 I'm OK with adding new items, just not changing/deprecating existing items except on a major version change 18:17:08 fungi and this whole thing might be mute... 18:17:13 mfisch: I'm using stable branches =) 18:17:19 harlowja: one can only hope, but we seem to grow wider faster than we grow higher 18:17:36 stop eating so much icecream! 18:17:36 lol 18:17:59 *wider faster joke... 18:18:12 okay so we should wrap this topic up. I think my advice would be to clearly state the use case you have in the TBD ML thread 18:18:13 har 18:18:21 probably makes sense to concentrate on something managable to start with, I agree with fungi that this could get out of control pretty easily. 18:18:24 I'm not sure they all got through yesterday it was pretty noisy 18:18:54 anything else on this one? 18:18:56 personally I'd be happy to just see reference level sample configs for each project release + master 18:19:06 dvorak: yeah, so my plan is to outline the discussion so far and request use cases to go along with any preferences people are indicating 18:19:12 wfm 18:19:14 someone else mentioned having config files for libs 18:19:33 klindgren: that's the main pain point of sample config files in tree 18:19:54 mgagne, this was more like oslo.db has its own ocnfig file 18:19:56 gah, sorry I'm late 18:20:10 so nova/neutron config doesn't change based upon included libs 18:20:27 up version of included libs* 18:20:29 klindgren: "the resulting content of a sample config file highly depends on the installed versions of those libraries." 18:20:30 right, i think if there are good software development reasons to switch that model then they will be likely to happen, but don't want to have the tail wagging the dog where configuration defines software rather than the other way around 18:21:11 would be nice to reduce some of the duplication between config files though 18:21:21 mgagne, right - for those options that come from librarys. 18:21:38 BUt no reason why nova couldn't keep up a nova.conf that has everything specific to nova in it 18:21:49 i bet the version change in oslo.db will only change the options related to db, so the rest of the config file is 'static' for a fixed version of nova/neutron 18:21:58 right 18:22:06 gfa ^^ exactly 18:22:11 a nova.conf that had all the nova specific stuff would help 18:22:23 so projects can ship all the config file but db, options 18:22:27 I mean it would solve most if not all of the things I've dug into in the past year 18:22:31 Isnt' that what the configuration reference is for? 18:22:49 again, that's software design issues, so out of scope for the sample config publishing discussion, but definitely something good to talk to the oslo devs about 18:22:53 true, if we could have an online tool to generate the config file of the project and then you could select the lib versions you have and then combine the resulting configs. 18:23:09 I don't look at sample configs, specificaly because they are bad, I rely on the reference on the docs site 18:23:10 that sounds.... fragile 18:23:18 eitherway - someone brought it up on the cross-project discussion 18:23:30 so I was jsut brining up here as well 18:23:38 bringing* 18:24:12 next topic? 18:24:16 or do we have more here 18:24:35 next 18:24:41 skipping around again 18:24:44 #topic mailing list 18:25:11 I've noticed since I joined the ML that we still get some traffic thats essentially people who've never used openstack before. 18:25:20 Does that belong on our ML and is it at a problem level? 18:25:34 I think we get CC'd a lot 18:25:49 personally i don’t think it’s an issue at this point 18:25:49 like, people write to openstack@lists, and openstack-operators, etc 18:25:53 yeah 18:25:55 ok 18:26:05 it has been quieter in terms of those messages, so moving on! 18:26:09 #topic packaging 18:26:12 I think the traffic level on the list is still pretty low, and if we need can can always gently direct people to the right ml 18:26:15 i just try to not respond to stuff that’s not appropriate for the list 18:26:44 re packaging: There was some ML Threads about packaging coming out of the paris discussions 18:26:55 I actually think it's a great forum, because we're the ones who may likely have the answer 18:27:41 alop: nod, I think mfisch is talking about the "my first openstack" questions that come along occassionally. 18:27:41 I'm not sure what the next steps are or if anyone is working on it 18:27:46 yes we may have the anwser, but IMO if you don'y know MTU you should not be doing openstack+gre 18:28:00 * mfisch reverts the topic 18:28:04 sorry 18:28:07 packaging 18:28:07 np 18:28:14 this is my first meeting 18:28:28 re packaging, I wasn't at Paris, but I'm keen to be involved 18:28:32 so, I missed the convo in Paris (stuck in the nova thing) 18:28:43 the tl;dr was that lots of us are packaging 18:28:45 it seems to me there's a bunch of people making packages 18:28:48 and we're all using different tools 18:29:11 and we want to make blessed generic packages? 18:29:15 we're using Jenkins debian glue I know klindgren uses anvil 18:29:23 + yahoo + cray 18:29:42 anvil sounds awesome except we dont use RHEL or its cousins 18:29:45 we use ubuntu's tools, but with our own repos (stable + selected patches) 18:30:02 mfisch ya; i'll avoid the diving deep into anvil, but its not designed just for RHEL :) 18:30:05 I feel that most distro tools feel like a huge Rube Goldberg machine. And I feel the need to simplify it a bit to fit my own particular needs 18:30:45 There's also the philosophy argument. A guy from Redhat and to some extent the debian guy on the ML (who's name I cannot remember) both were wondering why ops are doing packaging 18:31:04 meh 18:31:04 Thomas Goirand it is 18:31:10 The resistance is duplication of effort and for some folks support issues I guess 18:31:11 that's a good point though 18:31:19 because, both of those entities are slow to deliver changes upstream 18:31:25 absolutely 18:31:38 we package cos we don't want to wait for distro packages to arrive 18:31:42 true 18:31:44 That and I have my own patches that I need ot run with as well 18:31:47 also, I don't really want my openstack services to be directly tied to the version of my operating system, which is kind of the case today with ubuntu 18:31:47 to* 18:31:51 The distro tools also don't answer our need to fork a project and apply our own custom patches. 18:31:55 plus they drop support for stuff 18:32:01 like juno is RHEL7 only 18:32:05 even if the distro packages the day it lands in a backport you might have 2 weeks to get it there 18:32:25 iirc Ubuntu dropped havana support at 2013.2.3? 18:32:29 yup 18:32:29 mfisch: and you may have to take some other patch that you don't want. 18:32:34 yeah 18:32:40 Case in point, You find a bug, you report, you even submit a patch for it. The community may debate it for 6 months, meanwhile, your production is broken 18:32:42 nor do I have time to look at all the changes that come in the new package 18:32:50 lack of granulaity is a big issue, aside from speed and local patches 18:32:50 alop: thats absolutely happened here 18:32:55 here too 18:33:00 here too 18:33:02 that sounds impossible 18:33:03 haha 18:33:03 After upstream ignored it for 3 weeks it landed, then they ignored my backport for 4 more weeks. 18:33:18 i feel like we all generally understand the reasons for doing our own patching 18:33:18 backports are a *PAIN* 18:33:19 I suspect that's why we all do our own packages 18:33:24 agreed 18:33:25 xavpaice: yep 18:33:28 yeah 18:33:35 every place I've seen does their own fashion of packages 18:33:47 so the paris talk was basically is there a way we can share what we do better? 18:33:48 either actual OS packages, or python venvs, or containers using one or the other. 18:34:13 so the barrier to package tools being more generic for us is the differences between RHEL based distros and Debian based? 18:34:29 I don't mean to diminish the work of Thomas but he kind of missed the point about *our* needs as operators 18:34:47 mgagne: and was a bit rude 18:34:49 xavpaice i'll just throw this out there (https://review.openstack.org/#/c/87875/ was a start of debian buliding for anvil; its not impossible...) 18:35:05 mgagne: agreed, we're not competing with debian, we have different needs 18:35:20 mfisch: I'd think common tooling would be the biggest thing. We're all going to have local packages at different times 18:35:39 dvorak, mfisch: I can understand his frustration and need to vent but there was no opening about actually understanding our needs and answering it 18:35:44 harlowja: I'll have to read that in detail :) 18:35:48 even common for ubuntu would help, sounds like most RHEL derivatives just use anvil 18:35:57 xavpaice also https://github.com/stackforge/anvil/commit/d435f548276 which adds venv building to anvil 18:36:06 *or at least the start of that building... 18:36:21 we use jenkins debian glue, and it seems better than using the native tools it wraps, but it's still a pain in the butt 18:37:08 So broadly I'd like to propose that we resurrect this discussion on the mailing list with the goal of setting an agenda/goal/etc for the mid-cycle meetup 18:37:18 our changes to the ubuntu packages are so minor, that we can just use the source packages, and rebuild with our forked repo - easy for us but not particularly transportable 18:37:19 furthermore, we might have completely different philosophy around compat and versioning requirements. Installing in a venv is fine for some operators, it's not for Debian/Ubuntu/RHEL. 18:37:21 mfisch +1 18:37:41 xavpaice: that's basically what we're doing also, plus git-upstream 18:37:48 xavpaice: do you pull from upstream ubuntu for the debian/ folder when you do updates or just fork it once? 18:37:54 sounds like a good plan mfisch 18:37:55 xavpaice +1 18:37:58 regular plans 18:38:03 s/plans/pulls 18:38:44 I was playing with applying extra patches via quilt but it was hard to maintain 18:38:56 xavpaice: it's awful 18:38:57 i would like to share backports, i don't have many for icehouse but i had more in havana time 18:39:03 xavpaice: that's my current workflow 18:39:14 I really feel that any established environment is going to have a system already in place to manage patches 18:39:15 i would like to see a dump-your-backport repo 18:39:15 our devs in house use our copy of the upstream repo and add their own patches, but if quilt adds stuff they don't see it's hard to test properly 18:39:19 and convergence is going to be... hard 18:39:28 xavpaice: even with gbp-pq 18:39:46 RAX public cloud has one thing, rax private cloud has another, Blue Box has one (two), etc.. etc... 18:40:03 jlk: well, people come to it on their own terms, or stick with what they have. the problem right now is that there isn't anything but completely canned packages or roll your own. something in the middle might be nice 18:40:19 yeah, that's just hard to do without strong opinions 18:40:19 jlk: goes to show that nobody feels the existing solutions answer their needs 18:40:22 The goal of operators was not to bless a one true way of doing anything like packaging but instead to share 18:40:37 mgagne: many of the solutions are made behind closed doors to satisfy a product before being made public 18:40:45 take someone's elses work flow and fork it for your needs 18:40:51 yeah 18:40:57 jlk: hehe, devs/operators' life 18:41:32 for packaging that might mainly be documenting stuff in a wiki or blog post rather than writing a new tool. It might also be contributing your enhancements to whatever your upstream is 18:41:51 I agree it would be nice to point at $something as a framework for consuming upstream source, adding in downstream changes, and producing an artifact at the end. 18:42:12 queue fight over who's existing $thing becomes the reference. 18:42:19 agreed 18:42:26 jlk: true, those are the main challenging when starting packaging, there is no true reference/tutorial 18:42:29 a wiki with some discussion of options and how others have achieved their goal would be good though 18:42:40 superuser articles perhaps 18:42:41 leaves the reader to make their own decision, but with information 18:42:42 sounds like an action is brewing 18:42:47 jlk: I was just thinking that 18:42:49 on how each provider is managing their downstream changes and artifacts. 18:42:50 jlk: I'd be interested in just hearing what other people are doing at this point. I know anvil sounds like it works well for a lot of people, but I don't know the workflow, pain points, etc. 18:43:12 I want to hear from someone using giftwrap as well 18:43:18 the best part of the conference is going to a CI discussion and realizing hey we're not totally off base on the made up way we use 18:44:03 any preference for wiki vs superuser (blog) in this? 18:44:11 we could always start with blog and migrate the info or vice-versa 18:44:20 can anyone write a superuser article? 18:44:29 xavpaice: everyone should IMO 18:44:30 well they let me do it, so yeah 18:44:53 just wondering if there's a barrier, like being a foundation member, or a certain size of deployment, etc 18:44:57 xavpaice: as there won't be one tool to show/explain 18:45:10 I don't think there is a barrier 18:45:19 no barrier 18:45:28 excellent, suits me then 18:45:36 xavpaice: ping me and I'll get you a name where you can get more info 18:45:44 thanks 18:46:03 maybe I should just chat to tom? 18:46:07 #action everyone to document their packaging process for sharing purposes 18:46:20 xavpaice: allison@openstack.org 18:46:36 * xavpaice thanks mfisch 18:46:47 okay final topic 18:46:54 by fiat I have cancelled the meeting on Christmas day 18:47:01 +1 18:47:23 I wont be around much on new years eve either 18:47:29 so I'm voting to cancel that too 18:47:38 +1 18:47:40 i think that’s reasonable 18:47:59 see you guys on Jan 7 then 18:48:00 anything else? 18:48:22 can you make a note to announce the next meeting a few days ahead on the ML? 18:48:36 yes. Was yesterday's announce too late? 18:48:37 did we want to discuss the shared ops tools discussion on the ML? 18:48:45 mfisch: no that was fine 18:48:53 yes, lets do that xavpaice 18:48:57 xavpaice: can you kick off the convo? 18:49:18 the tl;dr was that we all have tools for things like monitoring, graphs, etc 18:49:31 there's a github for sharing some of these, but not much content 18:49:47 some discussion around stackforge and some around an official OpenStack project 18:49:54 xavpaice: yeah I'm not sure what the future of said repo is, I think that it might go away at some point 18:50:04 I'd be sad about that 18:50:08 I'd like to get Michael Chapman here to follow-up on that one 18:50:14 unless it moved to stackforge 18:50:36 might be a bit early for him, it's not even 8 here in NZ and we're two hours ahead 18:50:55 I think that stackforge is the theory, I wont say plan b/c I've not heard in awhile 18:51:20 okay folks, see you guys back in #openstack-operators 18:51:24 going once... 18:51:28 the concern about stackforge is that you have to do gating and what not 18:52:06 but yeah, can follow up on that discussion later. 18:52:21 cool 18:52:25 agreed 18:52:30 thanks all 18:52:32 #endmeeting