| *** rady has quit IRC | 00:06 | |
| *** lhcheng has quit IRC | 00:06 | |
| *** lhcheng has joined #openstack-operators | 00:07 | |
| *** mriedem is now known as mriedem_away | 00:10 | |
| *** zhangjn has joined #openstack-operators | 00:11 | |
| *** lhcheng has quit IRC | 00:11 | |
| *** lhcheng has joined #openstack-operators | 00:11 | |
| *** zhangjn has quit IRC | 00:15 | |
| *** jproulx has joined #openstack-operators | 00:18 | |
| *** lhcheng has quit IRC | 00:33 | |
| *** xavpaice has quit IRC | 00:34 | |
| *** lhcheng has joined #openstack-operators | 00:35 | |
| *** xavpaice has joined #openstack-operators | 00:38 | |
| *** Piet has joined #openstack-operators | 00:47 | |
| *** VW has joined #openstack-operators | 00:55 | |
| *** ahonda has joined #openstack-operators | 00:56 | |
| *** zhangjn has joined #openstack-operators | 00:56 | |
| *** zhangjn has quit IRC | 00:56 | |
| *** zhangjn has joined #openstack-operators | 00:57 | |
| *** VW has quit IRC | 00:59 | |
| *** zhangjn_ has joined #openstack-operators | 01:02 | |
| *** zhangjn has quit IRC | 01:04 | |
| *** lhcheng_ has joined #openstack-operators | 01:08 | |
| *** lhcheng has quit IRC | 01:08 | |
| *** lhcheng_ has quit IRC | 01:24 | |
| *** lhcheng has joined #openstack-operators | 01:26 | |
| *** vinsh has joined #openstack-operators | 01:32 | |
| *** dminer has joined #openstack-operators | 01:36 | |
| *** harshs has quit IRC | 01:39 | |
| *** smemon92 has joined #openstack-operators | 01:45 | |
| *** lhcheng has quit IRC | 01:55 | |
| *** harshs has joined #openstack-operators | 01:56 | |
| *** lhcheng_ has joined #openstack-operators | 01:56 | |
| *** harshs has quit IRC | 02:00 | |
| *** vinsh has quit IRC | 02:00 | |
| *** VW has joined #openstack-operators | 02:09 | |
| *** VW has quit IRC | 02:10 | |
| *** VW has joined #openstack-operators | 02:10 | |
| *** lhcheng_ has quit IRC | 02:17 | |
| *** lhcheng has joined #openstack-operators | 02:17 | |
| *** dminer has quit IRC | 02:21 | |
| *** lhcheng has quit IRC | 02:24 | |
| *** lhcheng has joined #openstack-operators | 02:25 | |
| *** VW has quit IRC | 02:34 | |
| *** VW has joined #openstack-operators | 02:38 | |
| *** VW has quit IRC | 02:38 | |
| *** VW has joined #openstack-operators | 02:39 | |
| *** vinsh has joined #openstack-operators | 02:41 | |
| *** harshs has joined #openstack-operators | 02:43 | |
| *** vinsh has quit IRC | 02:58 | |
| *** smemon92 has quit IRC | 03:00 | |
| *** harshs has quit IRC | 03:13 | |
| *** harshs has joined #openstack-operators | 03:23 | |
| *** lhcheng has quit IRC | 03:26 | |
| *** lhcheng_ has joined #openstack-operators | 03:26 | |
| *** lhcheng_ has quit IRC | 03:28 | |
| *** DarthVigil has quit IRC | 03:40 | |
| *** zhangjn_ has quit IRC | 03:55 | |
| *** zhangjn has joined #openstack-operators | 03:58 | |
| *** zhangjn has quit IRC | 03:59 | |
| *** zhangjn has joined #openstack-operators | 04:00 | |
| *** harshs has quit IRC | 04:06 | |
| *** zhangjn has quit IRC | 04:07 | |
| *** lhcheng has joined #openstack-operators | 04:21 | |
| *** zhangjn has joined #openstack-operators | 04:35 | |
| *** Marga_ has quit IRC | 04:38 | |
| *** lhcheng_ has joined #openstack-operators | 04:47 | |
| *** lhcheng has quit IRC | 04:50 | |
| *** britthouser has quit IRC | 04:58 | |
| *** britthouser has joined #openstack-operators | 05:07 | |
| *** harshs has joined #openstack-operators | 05:13 | |
| *** Piet has quit IRC | 05:15 | |
| *** Marga_ has joined #openstack-operators | 05:27 | |
| *** sanjayu has joined #openstack-operators | 05:46 | |
| *** pontusf has quit IRC | 05:50 | |
| *** pontusf has joined #openstack-operators | 05:50 | |
| *** zhangjn_ has joined #openstack-operators | 06:01 | |
| *** zhangjn has quit IRC | 06:04 | |
| *** zhangjn_ has quit IRC | 06:18 | |
| *** lhcheng has joined #openstack-operators | 06:19 | |
| *** lhcheng_ has quit IRC | 06:19 | |
| *** zhangjn has joined #openstack-operators | 06:26 | |
| *** zhangjn has quit IRC | 06:33 | |
| *** zhangjn has joined #openstack-operators | 06:35 | |
| *** zhangjn has quit IRC | 06:38 | |
| *** zhangjn has joined #openstack-operators | 06:44 | |
| *** zhangjn has quit IRC | 06:48 | |
| *** zhangjn has joined #openstack-operators | 07:18 | |
| *** harshs has quit IRC | 07:23 | |
| *** zhangjn has quit IRC | 07:24 | |
| *** Marga_ has quit IRC | 07:26 | |
| *** zhangjn has joined #openstack-operators | 07:26 | |
| *** miyagishi_t has joined #openstack-operators | 07:28 | |
| *** zerda has joined #openstack-operators | 07:46 | |
| *** zhangjn has quit IRC | 07:47 | |
| *** zhangjn has joined #openstack-operators | 07:53 | |
| *** zhangjn has quit IRC | 07:54 | |
| *** liverpooler has joined #openstack-operators | 07:56 | |
| *** zhangjn has joined #openstack-operators | 08:03 | |
| *** zigo has quit IRC | 08:16 | |
| *** bvandenh has joined #openstack-operators | 08:17 | |
| *** zigo has joined #openstack-operators | 08:19 | |
| *** berendt has joined #openstack-operators | 08:23 | |
| *** zhangjn has quit IRC | 08:27 | |
| *** zhangjn has joined #openstack-operators | 08:33 | |
| *** zhangjn has quit IRC | 08:36 | |
| *** zhangjn has joined #openstack-operators | 08:37 | |
| *** miyagishi_t has quit IRC | 08:49 | |
| *** VW has quit IRC | 08:49 | |
| *** bvandenh has quit IRC | 08:53 | |
| *** bvandenh has joined #openstack-operators | 08:55 | |
| *** matrohon has joined #openstack-operators | 09:00 | |
| *** cbrown2_ocf has joined #openstack-operators | 09:03 | |
| *** cbrown2_ocf has quit IRC | 09:07 | |
| *** cbrown2_ocf has joined #openstack-operators | 09:13 | |
| *** bvandenh has quit IRC | 09:16 | |
| *** openstack has joined #openstack-operators | 09:19 | |
| *** subscope has joined #openstack-operators | 09:22 | |
| *** lhcheng has quit IRC | 09:29 | |
| *** derekh has joined #openstack-operators | 10:02 | |
| *** electrofelix has joined #openstack-operators | 10:08 | |
| *** Marga_ has joined #openstack-operators | 10:15 | |
| *** belmoreira has joined #openstack-operators | 10:37 | |
| *** bradjones has joined #openstack-operators | 10:37 | |
| *** bradjones has quit IRC | 10:37 | |
| *** bradjones has joined #openstack-operators | 10:37 | |
| *** subscope has quit IRC | 10:51 | |
| *** subscope has joined #openstack-operators | 10:55 | |
| *** subscope has quit IRC | 11:03 | |
| *** maishsk has joined #openstack-operators | 11:05 | |
| *** zhangjn has quit IRC | 11:10 | |
| *** zhangjn has joined #openstack-operators | 11:36 | |
| *** zhangjn has quit IRC | 11:36 | |
| *** zhangjn has joined #openstack-operators | 11:36 | |
| *** bvandenh has joined #openstack-operators | 11:41 | |
| *** jaypipes has joined #openstack-operators | 11:41 | |
| *** weihan has joined #openstack-operators | 11:43 | |
| *** liverpooler has quit IRC | 11:47 | |
| *** maishsk has quit IRC | 12:01 | |
| *** liverpooler has joined #openstack-operators | 12:02 | |
| *** maishsk has joined #openstack-operators | 12:03 | |
| *** zerda has quit IRC | 12:06 | |
| *** liverpooler has quit IRC | 12:11 | |
| *** liverpooler has joined #openstack-operators | 12:11 | |
| *** bvandenh has quit IRC | 12:12 | |
| *** subscope has joined #openstack-operators | 12:14 | |
| *** bvandenh has joined #openstack-operators | 12:15 | |
| *** ig0r__ has joined #openstack-operators | 12:17 | |
| *** _nick has quit IRC | 12:41 | |
| *** _nick has joined #openstack-operators | 12:42 | |
| *** liverpoo1er has joined #openstack-operators | 12:51 | |
| *** sanjayu has quit IRC | 12:55 | |
| *** weihan has quit IRC | 12:57 | |
| *** subscope has quit IRC | 12:57 | |
| *** maishsk has quit IRC | 13:02 | |
| *** subscope has joined #openstack-operators | 13:08 | |
| *** maishsk has joined #openstack-operators | 13:08 | |
| *** VW has joined #openstack-operators | 13:44 | |
| *** VW has quit IRC | 13:45 | |
| *** ccarmack1 has left #openstack-operators | 13:45 | |
| *** regXboi has joined #openstack-operators | 13:53 | |
| *** liverpoo1er has quit IRC | 13:56 | |
| *** cbrown2_ocf has quit IRC | 14:03 | |
| *** cbrown2_ocf has joined #openstack-operators | 14:19 | |
| *** VW has joined #openstack-operators | 14:20 | |
| *** VW has quit IRC | 14:24 | |
| *** VW has joined #openstack-operators | 14:24 | |
| *** zhangjn has quit IRC | 14:30 | |
| *** mriedem_away is now known as mriedem | 14:35 | |
| *** subscope has quit IRC | 14:38 | |
| *** signed8bit has joined #openstack-operators | 14:44 | |
| *** ctina has joined #openstack-operators | 14:53 | |
| *** ctina_ has joined #openstack-operators | 14:54 | |
| *** ctina_ has quit IRC | 14:54 | |
| *** ctina has quit IRC | 14:54 | |
| *** liverpooler has quit IRC | 15:07 | |
| *** dminer has joined #openstack-operators | 15:10 | |
| *** cbrown2_ocf has quit IRC | 15:15 | |
| *** openstackgerrit has quit IRC | 15:17 | |
| *** openstackgerrit has joined #openstack-operators | 15:17 | |
| *** cbrown2_ocf has joined #openstack-operators | 15:27 | |
| *** csoukup has joined #openstack-operators | 15:30 | |
| *** berendt has quit IRC | 15:35 | |
| *** jmckind has joined #openstack-operators | 15:44 | |
| *** prometheanfire has joined #openstack-operators | 15:44 | |
| *** Brainspackle has quit IRC | 15:48 | |
| *** emagana has joined #openstack-operators | 15:49 | |
| *** maishsk has quit IRC | 15:56 | |
| *** maishsk has joined #openstack-operators | 16:07 | |
| *** krotscheck has joined #openstack-operators | 16:08 | |
| *** mdorman has joined #openstack-operators | 16:09 | |
| *** sanjayu has joined #openstack-operators | 16:11 | |
| *** SimonChung1 has quit IRC | 16:11 | |
| *** VW has quit IRC | 16:12 | |
| *** maishsk has quit IRC | 16:14 | |
| *** maishsk has joined #openstack-operators | 16:15 | |
| *** VW has joined #openstack-operators | 16:18 | |
| *** bvandenh has quit IRC | 16:21 | |
| *** vinsh has joined #openstack-operators | 16:26 | |
| *** VW has quit IRC | 16:30 | |
| *** VW has joined #openstack-operators | 16:30 | |
| *** Piet has joined #openstack-operators | 16:34 | |
| *** maishsk has quit IRC | 16:41 | |
| *** harshs has joined #openstack-operators | 16:46 | |
| *** gyee has joined #openstack-operators | 16:48 | |
| *** lhcheng has joined #openstack-operators | 16:58 | |
| *** jmckind is now known as jmckind_ | 17:00 | |
| *** jmckind_ is now known as jmckind | 17:00 | |
| *** Brainspackle has joined #openstack-operators | 17:02 | |
| *** matrohon has quit IRC | 17:05 | |
| *** krotscheck has quit IRC | 17:12 | |
| *** armax has joined #openstack-operators | 17:14 | |
| *** SimonChung has joined #openstack-operators | 17:16 | |
| *** jmckind is now known as jmckind_ | 17:18 | |
| *** VW has quit IRC | 17:19 | |
| *** jmckind_ is now known as jmckind | 17:21 | |
| *** krotscheck has joined #openstack-operators | 17:29 | |
| *** alop has joined #openstack-operators | 17:30 | |
| *** sanjayu has quit IRC | 17:44 | |
| *** regXboi has quit IRC | 17:47 | |
| *** derekh has quit IRC | 17:52 | |
| *** VW has joined #openstack-operators | 18:04 | |
| *** VW has quit IRC | 18:05 | |
| *** VW has joined #openstack-operators | 18:05 | |
| *** belmoreira has quit IRC | 18:08 | |
| klindgren | I keep reading that "upgrades" are terrible in the Ops mailing list. However, having done upgrades since havana. I dont recall them being terrible at all? I got lucky on some of the upgrades - Like I was already running ml2. But seems to me like people are blowing the upgrade issue *WAY* out of perspective | 18:10 |
|---|---|---|
| *** jmckind is now known as jmckind_ | 18:11 | |
| dmsimard | I think it's especially that there's a lot of work involved in upgrading properly and without impacting both end users and systems on large-scale production environments and the features in the newer version might not warrant them spend that time on that | 18:15 |
| dmsimard | that last message probably needed a couple commas or periods | 18:16 |
| klindgren | I can see the new features in new version may not provide anything that you care about. But from what I have seen our upgrades has been pretty much a zero data plane impact. Only the control plane takes a hit | 18:17 |
| dmsimard | klindgren: How did you upgrade from Icehouse to Juno ? It outright required an OS upgrade afaik (CentOS6 to CentOS7 and Ubuntu 12.04 to Ubuntu 14.04) | 18:19 |
| klindgren | We dont use distro packages | 18:19 |
| klindgren | so cent6 runs Juno code without any issues | 18:19 |
| klindgren | but our control plane we were able to one-by-one either upgrade or reload to cent7 | 18:20 |
| dmsimard | Bah, that's cheating :) | 18:20 |
| dmsimard | yeah, doing upgrades without impacting customers is possible, just a lot of work | 18:20 |
| dmsimard | live migrations around, re-installs (when using packages), etc. | 18:21 |
| klindgren | infact we have a hack in place to run kilo nova-compute and neutron-agent on Cent6 Hv's | 18:21 |
| klindgren | that we might continue running under libirty | 18:21 |
| *** stanchan has joined #openstack-operators | 18:21 | |
| klindgren | depends on when we can get our last 80 cent6 hv's upgraded to cent7 | 18:21 |
| klindgren | which I have shared the hack with the folks at Cern and they are using it/going to be using it for their kilo stuff | 18:22 |
| klindgren | since they have like 500 or 800 hv's to reload - I dont recal which | 18:22 |
| klindgren | iirc, cern also got CentOS to build juno packages for cent6 | 18:23 |
| dmsimard | So the lesson from the story is to roll your own packages ? | 18:23 |
| klindgren | That def helps :-) | 18:26 |
| klindgren | though - since its python - building of packages is not strictly required | 18:27 |
| mdorman | i think a lot of people would really like to see more the ‘yum update’ experience for updating. instead of this complicated orchestration of certain things that have to go in a certain order, upgrading schema, etc. etc. | 18:35 |
| mdorman | which, granted, for a complicated system like openstack, is kind of a pipe dream | 18:35 |
| *** elo has joined #openstack-operators | 18:37 | |
| *** prometheanfire has left #openstack-operators | 18:40 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:41 | |
| xavpaice | certainly when I did an upgrade using distro packages there was significant challenges in that we had to do the whole thing in one hit | 18:44 |
| xavpaice | the switch to our own packages made it easier | 18:44 |
| *** signed8bit_ZZZzz is now known as signed8bit | 18:44 | |
| xavpaice | if we had each service in its own container or VM that would have made the whole thing a bunch easier, and we may be happy to switch back to distro packages if we go down that route | 18:45 |
| *** harshs has quit IRC | 18:50 | |
| xavpaice | something I've learned so far is that the bugfixes in the newer versions are far more important than the new features - in fact so much so I'd rather be running a new version and collecting bug reports than looking at long fixed issues that haven't been backported | 18:51 |
| klindgren | +1 on bug fixes | 18:56 |
| klindgren | also easier if you have a bug and fix it locally - to get it upstreamed | 18:56 |
| klindgren | as the delta between what you are running and where master is - is usualy smaller/less painful | 18:56 |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:57 | |
| xavpaice | I guess the bug fixes we've had to carry locally have been the ones that are needed within hours because they're affecting a customer - so we can't afford to wait for the merge upstream | 18:59 |
| xavpaice | and adding those to a package is miles easier than patching code in place | 18:59 |
| xavpaice | for those making packages in venv's - have you had to add mysql-python to the requirements.txt? Not sure if that's just a by-product of my setup, or others have the same thing | 19:00 |
| xavpaice | also because I'm letting pbr do the version, and my tag is, e.g. 2015.1.2cat1, pbr gets quite upset | 19:01 |
| xavpaice | upset if it's 0.11.0 that is... | 19:01 |
| *** signed8bit_ZZZzz is now known as signed8bit | 19:03 | |
| klindgren | yea - PBR is terrible | 19:03 |
| klindgren | we just pass the ENV variable for PBR_VERSION when building packages | 19:04 |
| xavpaice | it's only the versioning I'm struggling with, and I'm sure it's just that I don't understand how to do it right | 19:04 |
| xavpaice | I can get the debs built OK, but when the code runs it spits errors about versions - guess I'll try that env var and see if it's better :) | 19:04 |
| xavpaice | I should read the pbr code and figure out how it works | 19:05 |
| klindgren | if you do 2015.1.2.1 | 19:05 |
| klindgren | it should give you a pbr version of 2015.1.2.dev1 | 19:06 |
| xavpaice | PBR_VERSION=2015.1.2.1 makes a package with 2015.1.2.dev1? | 19:06 |
| *** electrofelix has quit IRC | 19:18 | |
| *** rady has joined #openstack-operators | 19:19 | |
| *** signed8bit has quit IRC | 19:23 | |
| klindgren | no - if you use the PBR override it will be 2015.1.2.1 | 19:36 |
| klindgren | if not the semver stuff will set it to 2015.1.2.dev1 | 19:36 |
| klindgren | or -dev1 I forgot which | 19:36 |
| *** Piet has quit IRC | 19:40 | |
| *** VW has quit IRC | 19:44 | |
| *** VW has joined #openstack-operators | 19:46 | |
| *** VW has quit IRC | 19:46 | |
| *** VW has joined #openstack-operators | 19:47 | |
| *** berendt has joined #openstack-operators | 19:50 | |
| xavpaice | I'm happy with either - thanks for that! | 19:52 |
| klindgren | np - harlowja and I spent a *long* time fighting with PBR for naming of packages | 19:54 |
| *** jmckind_ is now known as jmckind | 19:57 | |
| *** cfarquhar has quit IRC | 20:01 | |
| *** matrohon has joined #openstack-operators | 20:01 | |
| mgagne | klindgren: live upgrade will make our life much harder since you won't be able to skip versions anymore AFAIK | 20:05 |
| mgagne | klindgren: the problem isn't the upgrade itself or database migration, it's all the required prep work which we can't afford every 6 months | 20:05 |
| *** cfarquhar has joined #openstack-operators | 20:07 | |
| *** cfarquhar has joined #openstack-operators | 20:07 | |
| *** kencjohnston has joined #openstack-operators | 20:08 | |
| *** harshs has joined #openstack-operators | 20:10 | |
| klindgren | yea for us it was 2-3 weeks of updating our patches to new version of code - removing patches that aren't needed, or re-writing them from scratch due to major arch changes | 20:10 |
| mgagne | klindgren: I like Tom Cameron's reply: http://lists.openstack.org/pipermail/openstack-operators/2015-November/008704.html | 20:12 |
| klindgren | I was just reading that | 20:12 |
| klindgren | Honestly - I wonder how many people responded - actually updated the details of their cloud | 20:13 |
| klindgren | I *ALMOST* didn't in the last user survey - which would have said we were on Icehouse | 20:13 |
| mgagne | klindgren: I think the openstack community has yet to fully understand why people aren't upgrading every 6 months and IMO are often mislead in their our hypothesis and assumptions made after misinterpreting survey or what not. | 20:13 |
| mgagne | klindgren: that's just an impression, not (yet) based on any empirical evidence | 20:15 |
| mgagne | reply by Clint Byrum on the list: "Upgrading across 2 years of development would possibly be a herculean effort that so far nobody has even tried to tackle without stepping through all intermediary releases." | 20:16 |
| mgagne | I have yet to reply to this comment but IMO, this reply wasn't made in good faith by taking an overly exaggerate example. We have plenty of examples of people skipping 1 or 2 releases. True it's not 2 years of releases but the idea of skipping versions in out there and for reasons. | 20:17 |
| xavpaice | can't do that now with Nova though | 20:19 |
| mgagne | can't anymore yes | 20:19 |
| mgagne | and it's frustrating because now, we will be stuck with older version for longer | 20:19 |
| xavpaice | up to that point, I gather it's fine, but I've never been confident to do it | 20:19 |
| xavpaice | mgagne: is this because of older Centos versions? | 20:20 |
| mgagne | we aren't gaining *anything* from the live migration thing. we don't have issue with database migration here. we have issue with the whole upgrade process. | 20:20 |
| mgagne | xavpaice: say that again? | 20:20 |
| xavpaice | the difficulty upgrading - are you stuck on centos 6 and having trouble with that? | 20:20 |
| mgagne | xavpaice: i'm not, we use ubuntu | 20:21 |
| mgagne | xavpaice: ... where the prep work is much bigger than the upgrade itself | 20:21 |
| xavpaice | cos of the distro packages and dependencies with versions? | 20:21 |
| mgagne | we do the operating system upgrade in between openstack upgrades | 20:21 |
| xavpaice | ah - you don't use the LTS release? | 20:22 |
| mgagne | we use LTS | 20:22 |
| mgagne | last time we upgraded was for precise -> trusty | 20:22 |
| xavpaice | ...? I'm missing something | 20:22 |
| mgagne | Icehouse is the only version where both LTS are supported | 20:22 |
| mgagne | xavpaice: no, I manage that part (packages) and it isn't that bad. Takes me ~2-3 weeks to repackage, update configs, test provisioning and upgrade. | 20:22 |
| xavpaice | we did the Trusty upgrade for Icehouse, but upgrading to Juno was easy once we got past the dependencies for packages | 20:23 |
| mgagne | xavpaice: now ask the business to prep itself (external systems depending/using openstack) | 20:23 |
| xavpaice | right, that's the same sort of pain we have too | 20:23 |
| mgagne | xavpaice: we have billing, custom interface, management interfaces to upgrade and we (the infra team) aren't the ones managing/dev those things | 20:23 |
| xavpaice | "hey, we have to just take down the entire region's network for a bit while we restart the routers" | 20:24 |
| mgagne | xavpaice: so the whole business have to go as fast as the openstack project itself and we aren't ready yet | 20:24 |
| xavpaice | same deal with a public cloud - customers are running things we have absolutely no control over | 20:24 |
| xavpaice | live migration has been mixed, sometimes it works fine and sometimes not | 20:25 |
| xavpaice | migrating routers between network nodes has been painful and not the most reliable - and sometimes, a 10 second network outage causes breakage | 20:25 |
| drwahl | ^^ a lot of that depends on the version of libvirt you are running | 20:25 |
| xavpaice | for sure | 20:25 |
| mgagne | yes | 20:25 |
| drwahl | actually, migrating routers has been amazingly smooth for us | 20:25 |
| xavpaice | migrationg across libvirt, qemu and kernel versions is.... risky... | 20:25 |
| drwahl | we RARELY lose even a single packet during migration | 20:25 |
| xavpaice | do you use VPNaaS? | 20:26 |
| drwahl | not yet. we're using astara for our routers | 20:26 |
| xavpaice | right - so there's the warning for you :) We use VPNaaS (ipsec), and the L3 agent with OVS etc for our networking | 20:26 |
| mgagne | So we can't slow down the openstack project itself. 6 months is fine. The problem is when you no longer have the ability to skip major version and save time/money. You end up geting pressured to upgrade faster by the community but yet, you can't afford it | 20:26 |
| xavpaice | 100% open source | 20:26 |
| xavpaice | mgagne: yep, feel your pain. We just did the Juno upgrade, then last week I hit Keystone with Kilo and am trying to get the next one done this week. | 20:27 |
| drwahl | astara has VPNaaS support in the works. i can't think of any reason why VPNaaS would cause issues during migration if we don't see any packets being dropped | 20:28 |
| drwahl | but there are always suprises :) | 20:28 |
| mgagne | "We assumed early on that the people deploying OpenStack would be more agile because of the ephemeral nature of cloud." | 20:28 |
| mgagne | on the list *now* | 20:28 |
| xavpaice | hmm, how did I not read about Astara before? | 20:30 |
| drwahl | it's a company we (dreamhost) spun out about a year ago | 20:30 |
| drwahl | we created the akanda router, then spun off the company and it's not become a "big tent" project | 20:30 |
| xavpaice | that makes it available to us - we're only allowed to use 100% open source software | 20:31 |
| mgagne | s/not/now/ ? | 20:31 |
| drwahl | it is 100% open source | 20:31 |
| xavpaice | :) | 20:31 |
| * xavpaice reads up on it | 20:31 | |
| drwahl | http://akanda.io/ | 20:31 |
| xavpaice | interestingly, we're talking about cumulus too | 20:32 |
| drwahl | cumulus has been pretty good for us | 20:33 |
| drwahl | there have been a few small hiccups, but overall, it's been really nice | 20:33 |
| drwahl | it's kind of weird to have your switch just be a linux box with a ton of ports on it | 20:33 |
| drwahl | but it's nice, 'cause then you can easily use tools like chef to configure them | 20:33 |
| wasmum | ok, wrapping my head around adding multiple networks to a single l3 agent. Have that working. The original provider net was a single net out. When I would assign a floating IP and ping it I would get the response from the ip I was pinging (floating ip). Now after I have added the additional networks, added the config in the plugin.ini and the ml2_config, add the bridges and port, add the neutron networks, assign a | 20:34 |
| wasmum | floating ip to a instance from one of the newly created/added nets, the reply from a ping comes from the internal IP of the instance | 20:34 |
| wasmum | I'm thinking, thats not how it's supposed to look? Did I miss something snat/nat related? | 20:34 |
| drwahl | xavpaice: if you care for a deep-dive into how we're using akanda, i'd be happy to share | 20:34 |
| *** Marga_ has quit IRC | 20:35 | |
| xavpaice | drwahl: thanks, that would be good | 20:35 |
| *** sanjayu has joined #openstack-operators | 20:36 | |
| clayton | xavpaice: yes, we have to add mysql-python also | 20:45 |
| clayton | it's not in requirements.txt, I assume because it's not actually required. you could be using db2 | 20:45 |
| clayton | the choice seems to be between having requirements.txt list every db backend people could be using, or me manually adding mysql-python, so I'm ok with the latter | 20:45 |
| xavpaice | fair call | 20:46 |
| xavpaice | just means a local patch is needed, but only because we're carrying venv built local packages so I guess that's the sort of thing I should do with quilt | 20:46 |
| clayton | I'm doing this for requiremens.txt - http://paste.openstack.org/show/478388/ | 20:46 |
| clayton | I don't use what's in the package directly | 20:47 |
| klindgren | I find that alot of the missing req's are defined in test-requirements.txt | 20:47 |
| clayton | xavpaice for heat, our requirements-source.txt looks like this - http://paste.openstack.org/show/478389/ | 20:48 |
| xavpaice | bingo - the mysql part is the same as ours | 20:49 |
| xavpaice | thanks for that | 20:49 |
| clayton | the update-requirements script I pasted will always produce a requirements.txt with everything frozen, and if you use branch names in the requirements-source.txt, it will freeze things as a commit hash | 20:49 |
| xavpaice | how is uwsgi working out for you with heat? | 20:49 |
| klindgren | eg: https://github.com/openstack/nova/blob/master/test-requirements.txt#L11 | 20:49 |
| clayton | haven't deployed it, but I think the dev work is mostly done | 20:49 |
| clayton | seems to be ok, I think we'll have to try it out and tweak settings | 20:49 |
| clayton | we're planning to use uwsgi directly in http mode, and front it with haproxy in http mode | 20:49 |
| xavpaice | PyMySQL =/= mysql-python though? | 20:49 |
| klindgren | yes | 20:50 |
| clayton | no, pymsql is the pure python implementation | 20:50 |
| xavpaice | that's what I was thinking - and people were saying it's really slow | 20:50 |
| clayton | that's the one the rackspace guy was saying was really slow in the liberty issues session | 20:50 |
| clayton | I was confused when he was saying that, since I didn't even know there were two different drivers. We've always used python-mysql, since that's what ubuntu packages | 20:51 |
| klindgren | ah - right - sorry - I keep getting mixed up on the ones that redhat renames and the ones that are different. | 20:51 |
| clayton | I'm looking into moving our nova control servers into docker now, to hopefully avoid having to upgrade to kilo.1 before we go to liberty | 20:51 |
| *** csoukup has quit IRC | 20:52 | |
| clayton | so hopefully we can upgrade *just* nova control services to latest stable/kilo | 20:52 |
| xavpaice | we were talking about containers for liberty, simply because we don't want to wait to finish the work on that before moving to kilo | 20:52 |
| clayton | xavpaice: we were planning to do keystone, heat, designate and glance before moving to liberty and the rest afterwards | 20:53 |
| *** kencjohnston has quit IRC | 20:53 | |
| clayton | we have two primary operational issues with docker right now 1) restarting docker engine restarts all services | 20:53 |
| clayton | and 2) we have an issue with aufs getting wedged and not letting us delete containers | 20:53 |
| xavpaice | those are both pretty significant | 20:53 |
| clayton | the latter I'm hoping to fix by upgrading the kernel and switching to overlayfs | 20:53 |
| *** kencjohnston has joined #openstack-operators | 20:54 | |
| clayton | the first one isn't too bad if ovs agent doesn't destroy everything when it restarts. | 20:54 |
| clayton | we restart services all the time anyway, so the latter is stupid, but managable | 20:54 |
| klindgren | we are planning on actually doing the reverse. Revamp our deployment process to include ci/cd and auto deployment then move to liberty | 20:54 |
| *** nikhil_k has quit IRC | 20:54 | |
| xavpaice | you're on kilo now though aren't you? | 20:54 |
| clayton | yes | 20:54 |
| xavpaice | yeah, so we're well behind you folk | 20:55 |
| clayton | not sure if you meant me or kris, but we're both on kilo :) | 20:55 |
| *** jmckind is now known as jmckind_ | 20:55 | |
| * xavpaice hangs head in shame at still being so far behind | 20:55 | |
| xavpaice | :) | 20:55 |
| clayton | we've spent a ridiculous amount of time on upgrading to juno and kilo | 20:56 |
| clayton | but I don't really see any other alternative | 20:56 |
| raginbajin | clayton: The whole upgrading for us problem is all around upgrading neutron and restarting the agent and it losing the flows. | 20:56 |
| raginbajin | We went to Juno in containers this last upgrade on the controller nodes. | 20:56 |
| raginbajin | and haven't actually had any problems with aufs or docker-engine, so we've been lucky on that side of things. | 20:56 |
| clayton | raginbajin yeah, we've managed to mitigate that to some extent, at least on control nodes. | 20:57 |
| xavpaice | that restarting issue is fixed in liberty iirc | 20:57 |
| clayton | on compute nodes it's fast enough to rebuild that's not that big an issue | 20:57 |
| raginbajin | Yeah it's still out an outage | 20:57 |
| clayton | depends on your tolerance for pain though :) | 20:57 |
| raginbajin | not mine, my users tolerance for pain | 20:57 |
| xavpaice | also depends on how many VMs on the hypervisor | 20:57 |
| raginbajin | and they don't like anything. | 20:57 |
| clayton | you should watch the talk we did for tokyo on upgrading to kilo ;) | 20:58 |
| raginbajin | I'll definitely check it out. We are just at Juno now and the thought of going to kilo/liberty sounds kinda daunting. One of the biggest issues we had was around deprecated settings that don't mean what they use to mean | 20:59 |
| clayton | raginbajin this part specifically: https://youtu.be/47YAH5Km9ho?t=4m37s :) | 20:59 |
| raginbajin | thanks! | 20:59 |
| clayton | you'll appreciate the 15 seconds there. | 20:59 |
| clayton | err, maybe 45 seconds, I linked to the wrong point :) | 20:59 |
| clayton | honestly, the OVS agent restart change is reason enough for us to upgrade to liberty | 21:00 |
| clayton | and upgrading to kilo was driven by the rabbit heartbeat chagnes | 21:00 |
| clayton | mfisch is upgrading rabbit in production tonight! | 21:01 |
| raginbajin | those are the two biggest things that are killing us I think as well. | 21:01 |
| *** berendt has quit IRC | 21:01 | |
| raginbajin | but juno was a little more rocky than we like so we are taking a breather to let the pitchforks and torches die down | 21:01 |
| clayton | juno was easy for us, compared to kilo :) | 21:02 |
| clayton | in fact, I'm pointing that out now, as I write my self-review ;) | 21:02 |
| *** sanjayu has quit IRC | 21:03 | |
| *** jmckind_ is now known as jmckind | 21:06 | |
| *** Piet has joined #openstack-operators | 21:08 | |
| *** kencjohnston has quit IRC | 21:09 | |
| *** VW has quit IRC | 21:11 | |
| *** jmckind is now known as jmckind_ | 21:13 | |
| *** VW has joined #openstack-operators | 21:13 | |
| *** VW has quit IRC | 21:14 | |
| *** VW has joined #openstack-operators | 21:14 | |
| klindgren | for us our upgrades were all easy | 21:15 |
| klindgren | but I wonder how much of that is due to our chosen network arch | 21:15 |
| klindgren | BTW we do upgrades at 9am ;-) | 21:16 |
| xavpaice | I do upgrades 'after hours' but in reality, a public cloud has no such thing and I might as well do them at 9am | 21:16 |
| xavpaice | yay for haproxy | 21:16 |
| *** subscope has joined #openstack-operators | 21:16 | |
| clayton | well, our customers are all in the US, so we definitely have off hours. | 21:21 |
| *** jmckind_ is now known as jmckind | 21:21 | |
| raginbajin | We've been doing them later on in the day say around 5pm-6pm that way it's not really late for us and we have some users we can use at guinea pigs if need be | 21:23 |
| raginbajin | klindgren: I definitely think your network arch makes it much easier thing to do. | 21:23 |
| clayton | I expect once we have HA routers in place and the OVS agent restart fix then the data plane impact should be almost nil | 21:23 |
| clayton | HA routers have been in the code base for like 2 releases now, so we're planning to try them once we're on liberty | 21:24 |
| raginbajin | That's been our thinking as well. | 21:28 |
| xavpaice | we were going to try that too, then apparently it's not great with VPNaaS | 21:31 |
| xavpaice | so we might need to wait till Liberty for that | 21:31 |
| clayton | it doesn't work with L2pop until liberty apparently | 21:32 |
| xavpaice | this is why we have to keep up with upgrades | 21:32 |
| clayton | yeap, exactly. | 21:33 |
| *** k_stev has joined #openstack-operators | 21:46 | |
| mfisch | are we discussing the off the rails thread on operators? | 21:48 |
| mfisch | clayton: I think we should first strive to get devs to backport a single release since they wont for the ovs fix... | 21:49 |
| *** jaypipes has quit IRC | 21:49 | |
| clayton | mfisch I think it stated that way. kris is amazed everyone doesn't find openstack upgrades to be trivial | 21:52 |
| mfisch | I was surpised that Matt said instances were the biggest issue | 21:53 |
| mfisch | other than virtual routers instances dont cause us any issues | 21:53 |
| clayton | yeah, that seems crazy to me. if you care about that, you do whatever you have to do to get live migration | 21:53 |
| mfisch | we dont l-m instances though, although we can | 21:53 |
| xavpaice | instances is only a problem when we want to do hypervisor maint, but that's not an openstack problem | 21:53 |
| clayton | sure, I think he meant for like operating system upgrades | 21:54 |
| xavpaice | klindgren: thanks for the tip re pbr, that worked perfectly for me | 21:57 |
| *** matrohon has quit IRC | 21:58 | |
| klindgren | xavpaice, cool | 21:58 |
| klindgren | clayton, mfisch its less that I think that upgrades are trivial. It's more that I think some people are making them out to be *way* harder than they actually are. Atleast in my exprience. | 21:59 |
| mfisch | once we dont have to move routers around like a shitty 3 card monty they will be way better | 21:59 |
| klindgren | provider networks - this time - for the win! | 22:00 |
| *** jmckind has quit IRC | 22:02 | |
| *** nikhil has joined #openstack-operators | 22:27 | |
| jlk | klindgren: they're hard if you're trying to preserve some level of uptime | 22:33 |
| jlk | or if you have busy clouds | 22:34 |
| jlk | they were way harder in previous releases | 22:34 |
| *** harshs has quit IRC | 22:39 | |
| *** rady has quit IRC | 22:41 | |
| klindgren | our upgrades have always been just about a 30 minute planned control plane outage - with 0 data-plane downtime. Aside from the rebasing patches, testing/planning phase. The upgrades themselves have gone smoothly without crazyness. But again, for me, we made some lucky design choices so I missed out on some of the upgrades other people had to go through. Such as the neutron OVS -> ML2+OVS migration. I was just point ou | 22:42 |
| klindgren | t that I seem to be reading people on the mailing list saying ZOMG upgrades it's the end-of-the-world. When my experience tells me for the most part its otherwise. Especially if you aren't running with local patches that need rebased. | 22:42 |
| *** harlowja_ has quit IRC | 22:43 | |
| *** harshs has joined #openstack-operators | 22:43 | |
| klindgren | Honestly for those running icehouse/juno just going to kilo for heartbeat support would most likely give them *more* uptime day-to-day | 22:43 |
| *** delattec has quit IRC | 22:50 | |
| *** albertom has quit IRC | 22:51 | |
| *** csoukup has joined #openstack-operators | 22:53 | |
| jlk | klindgren: yeah, for small clouds that works, small clouds that don't rely on vendor distros for the openstack software | 22:55 |
| clayton | klindgren it's not really meaningful that the actual upgrade itself is easy | 22:55 |
| clayton | I think most of us have automated it and agree that's the case | 22:55 |
| jlk | large clouds, 30 minutes is laughable. Mostly due to very lengthy database migrations | 22:56 |
| clayton | it's all the research and testing leading up to the upgrade that is the time sink | 22:56 |
| jlk | and some clouds can't take 30 minutes of API outage, so they're trying to optimize for 0 minutes of API outage | 22:56 |
| xavpaice | anyone having issues with configs changing with Juno -> kilo? I am finding that after doing keystone, all the other services when I upgrade them get WARNING keystonemiddleware.auth_token [-] Authorization failed for token | 22:56 |
| *** SimonChung has quit IRC | 22:57 | |
| *** SimonChung1 has joined #openstack-operators | 22:57 | |
| klindgren | pretty sure you have to redo the midleware paste config from juno-> kilo | 22:57 |
| *** SimonChung has joined #openstack-operators | 22:57 | |
| *** SimonChung1 has quit IRC | 22:58 | |
| xavpaice | I've been nabbing the paste.ini from the distro packages, so haven't even started to look there | 22:58 |
| *** SimonChung1 has joined #openstack-operators | 22:58 | |
| *** SimonChung2 has joined #openstack-operators | 22:58 | |
| *** SimonChung1 has quit IRC | 22:58 | |
| *** SimonChung has quit IRC | 22:58 | |
| *** albertom has joined #openstack-operators | 22:58 | |
| *** SimonChung has joined #openstack-operators | 22:59 | |
| *** SimonChung2 has quit IRC | 22:59 | |
| *** Marga_ has joined #openstack-operators | 23:00 | |
| *** Marga_ has quit IRC | 23:01 | |
| *** Marga_ has joined #openstack-operators | 23:01 | |
| xavpaice | wait, that's the per project api-paste.ini, not the one in /etc/keystone/keystone-paste.ini? | 23:02 |
| klindgren | per project-paste | 23:02 |
| klindgren | iirc some of the middleware was deprecated between juno -> kilo so if you haven't updated those since say icehouse - its got the wrong middleware code paths in them. | 23:03 |
| klindgren | though due to our app server layout we have to do all services at once on upgrade, due to package deps. Hello, moving to containers. | 23:04 |
| *** csoukup has quit IRC | 23:06 | |
| *** csoukup has joined #openstack-operators | 23:10 | |
| *** jmckind has joined #openstack-operators | 23:13 | |
| *** cdelatte has joined #openstack-operators | 23:21 | |
| *** subscope has quit IRC | 23:26 | |
| *** k_stev has quit IRC | 23:35 | |
| *** mdorman has quit IRC | 23:45 | |
| *** mriedem is now known as mriedem_away | 23:54 | |
| *** harshs has quit IRC | 23:54 | |
| *** dminer has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!