| *** clayton has joined #openstack-kolla | 00:00 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Allow an operator to run an action on all services https://review.openstack.org/325679 | 00:03 |
|---|---|---|
| *** tzn has quit IRC | 00:08 | |
| *** d_code has quit IRC | 00:24 | |
| *** zhiwei has joined #openstack-kolla | 00:24 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Add docs around bootstrapping and using the 'all' flag https://review.openstack.org/325681 | 00:25 |
| *** fragatin_ has joined #openstack-kolla | 00:30 | |
| *** SiRiuS has quit IRC | 00:31 | |
| *** fragatina has quit IRC | 00:32 | |
| *** fragatin_ has quit IRC | 00:35 | |
| *** salv-orlando has quit IRC | 00:54 | |
| *** salv-orlando has joined #openstack-kolla | 00:54 | |
| *** tzn has joined #openstack-kolla | 00:58 | |
| *** tzn has quit IRC | 01:04 | |
| *** dwalsh has joined #openstack-kolla | 01:25 | |
| *** d_code has joined #openstack-kolla | 01:35 | |
| *** dwalsh has quit IRC | 01:40 | |
| *** v1k0d3n has quit IRC | 01:53 | |
| *** sacharya has joined #openstack-kolla | 01:59 | |
| *** tzn has joined #openstack-kolla | 02:01 | |
| *** tzn has quit IRC | 02:05 | |
| *** sacharya has quit IRC | 02:19 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 02:24 | |
| *** klint has joined #openstack-kolla | 02:41 | |
| *** ozialien10 has quit IRC | 02:42 | |
| *** yuanying has quit IRC | 02:51 | |
| *** salv-orl_ has joined #openstack-kolla | 03:00 | |
| *** salv-orlando has quit IRC | 03:03 | |
| openstackgerrit | Merged openstack/kolla: Fix URL to Heka documentation in README file https://review.openstack.org/325595 | 03:14 |
| *** jrist has quit IRC | 03:16 | |
| *** sacharya has joined #openstack-kolla | 03:16 | |
| *** jrist has joined #openstack-kolla | 03:29 | |
| openstackgerrit | Md Nadeem proposed openstack/kolla-kubernetes: Heat services and pod https://review.openstack.org/316850 | 03:36 |
| *** d_code has quit IRC | 03:36 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove the deprecated kolla-build section https://review.openstack.org/325709 | 03:38 |
| *** dave-mccowan has quit IRC | 03:41 | |
| *** yuanying has joined #openstack-kolla | 03:46 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: [doc] change Ansible version to exactly 2.0.x in quickstart. https://review.openstack.org/322361 | 03:48 |
| *** v1k0d3n has joined #openstack-kolla | 03:48 | |
| *** tzn has joined #openstack-kolla | 03:49 | |
| *** tzn has quit IRC | 03:53 | |
| *** d_code has joined #openstack-kolla | 04:08 | |
| *** fragatina has joined #openstack-kolla | 04:08 | |
| *** fragatina has quit IRC | 04:09 | |
| *** fragatina has joined #openstack-kolla | 04:09 | |
| *** d_code has quit IRC | 04:19 | |
| *** d_code has joined #openstack-kolla | 04:25 | |
| Mech422 | https://coreos.com/blog/torus-distributed-storage-by-coreos.html | 04:56 |
| *** daneyon has joined #openstack-kolla | 05:06 | |
| *** v1k0d3n has quit IRC | 05:10 | |
| *** daneyon has quit IRC | 05:11 | |
| *** Mech422 has quit IRC | 05:19 | |
| *** Mech422 has joined #openstack-kolla | 05:21 | |
| *** tzn has joined #openstack-kolla | 05:37 | |
| *** tzn has quit IRC | 05:41 | |
| *** tfukushima has joined #openstack-kolla | 05:51 | |
| *** openstackgerrit has quit IRC | 06:02 | |
| *** openstackgerrit has joined #openstack-kolla | 06:03 | |
| Mech422 | Score! Looks like I fixed my deploy-ceph-by-partition problems :-) | 06:10 |
| *** Mr_Broke_ has joined #openstack-kolla | 06:15 | |
| *** Mr_Broken has quit IRC | 06:19 | |
| *** sacharya has quit IRC | 06:33 | |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add Kuryr ansible role https://review.openstack.org/298894 | 06:37 |
| *** v1k0d3n has joined #openstack-kolla | 06:45 | |
| coolsvap | mandre, ping | 06:47 |
| mandre | coolsvap: hi | 06:47 |
| coolsvap | mandre, i am trying to setup the vagrant dev environment, but the Mounting NFS shared folders.. seems to have stuck for a long time, any pointers? | 06:48 |
| Mech422 | Anyone noticed elasticsearch bombing deploy when 'enable_central_logging' is 'no' ? | 06:49 |
| *** godleon has joined #openstack-kolla | 06:49 | |
| mandre | coolsvap: not really, maybe have a look at the logs of your nfs server | 06:50 |
| mandre | coolsvap: can you mount the shares locally? | 06:51 |
| coolsvap | mandre, checking that | 06:51 |
| *** daneyon has joined #openstack-kolla | 06:55 | |
| *** v1k0d3n has quit IRC | 06:55 | |
| *** cu5 has joined #openstack-kolla | 06:55 | |
| *** Serlex has joined #openstack-kolla | 06:56 | |
| *** daneyon has quit IRC | 06:59 | |
| *** salv-orl_ has quit IRC | 07:03 | |
| *** salv-orlando has joined #openstack-kolla | 07:03 | |
| coolsvap | mandre, this seems to have been working after i added nfs to firewall service lists | 07:13 |
| coolsvap | will let you the result | 07:13 |
| * coolsvap brb lunch | 07:13 | |
| mandre | coolsvap: makes sense | 07:13 |
| *** matrohon has joined #openstack-kolla | 07:17 | |
| *** athomas has joined #openstack-kolla | 07:21 | |
| *** tzn has joined #openstack-kolla | 07:25 | |
| *** tzn has quit IRC | 07:29 | |
| *** sacharya has joined #openstack-kolla | 07:33 | |
| *** callahanca has quit IRC | 07:35 | |
| *** sacharya has quit IRC | 07:38 | |
| *** mikelk has joined #openstack-kolla | 07:39 | |
| *** shardy has joined #openstack-kolla | 07:39 | |
| *** v1k0d3n has joined #openstack-kolla | 07:52 | |
| *** v1k0d3n has quit IRC | 07:56 | |
| *** chopmann has joined #openstack-kolla | 08:04 | |
| *** chopmann is now known as chopmann_ | 08:04 | |
| *** shardy has quit IRC | 08:20 | |
| *** shardy has joined #openstack-kolla | 08:23 | |
| *** SiRiuS has joined #openstack-kolla | 08:31 | |
| *** chopmann_ has left #openstack-kolla | 08:40 | |
| *** tzn has joined #openstack-kolla | 08:40 | |
| *** mgoddard has joined #openstack-kolla | 08:50 | |
| *** v1k0d3n has joined #openstack-kolla | 08:53 | |
| *** papacz has joined #openstack-kolla | 08:53 | |
| *** v1k0d3n has quit IRC | 08:57 | |
| *** mgoddard has quit IRC | 09:01 | |
| *** dcwangmit01_ has quit IRC | 09:10 | |
| *** tfukushima has quit IRC | 09:11 | |
| *** dcwangmit01 has joined #openstack-kolla | 09:13 | |
| *** sacharya has joined #openstack-kolla | 09:19 | |
| *** sacharya has quit IRC | 09:23 | |
| *** tfukushima has joined #openstack-kolla | 09:26 | |
| *** tzn has quit IRC | 09:28 | |
| *** daneyon has joined #openstack-kolla | 09:37 | |
| *** daneyon has quit IRC | 09:41 | |
| *** v1k0d3n has joined #openstack-kolla | 09:54 | |
| *** v1k0d3n has quit IRC | 09:58 | |
| *** athomas has quit IRC | 10:05 | |
| *** pbourke has quit IRC | 10:11 | |
| *** pbourke has joined #openstack-kolla | 10:12 | |
| *** athomas has joined #openstack-kolla | 10:19 | |
| *** salv-orlando has quit IRC | 10:22 | |
| *** salv-orlando has joined #openstack-kolla | 10:23 | |
| *** tfukushima has quit IRC | 10:33 | |
| *** godleon has quit IRC | 11:09 | |
| *** sacharya has joined #openstack-kolla | 11:20 | |
| *** sacharya has quit IRC | 11:25 | |
| *** daneyon has joined #openstack-kolla | 11:25 | |
| *** rhallisey has joined #openstack-kolla | 11:26 | |
| *** mliima has joined #openstack-kolla | 11:28 | |
| *** daneyon has quit IRC | 11:30 | |
| *** mgoddard has joined #openstack-kolla | 11:32 | |
| *** salv-orlando has quit IRC | 11:35 | |
| mliima | morning | 11:35 |
| *** salv-orlando has joined #openstack-kolla | 11:35 | |
| *** rmart04 has joined #openstack-kolla | 11:37 | |
| *** salv-orl_ has joined #openstack-kolla | 11:39 | |
| *** coolsvap_ has joined #openstack-kolla | 11:39 | |
| *** eaguilar has joined #openstack-kolla | 11:42 | |
| *** salv-orlando has quit IRC | 11:42 | |
| *** dave-mccowan has joined #openstack-kolla | 11:46 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Cleanup help string of install_type parameter https://review.openstack.org/325600 | 11:48 |
| *** Jeffrey4l_ has quit IRC | 11:51 | |
| *** Liuqing has joined #openstack-kolla | 11:54 | |
| openstackgerrit | Merged openstack/kolla: Make container dind unpin old docker relase https://review.openstack.org/297434 | 11:55 |
| *** salv-orl_ has quit IRC | 11:58 | |
| *** salv-orlando has joined #openstack-kolla | 11:58 | |
| *** JoseMello has joined #openstack-kolla | 12:00 | |
| *** tzn has joined #openstack-kolla | 12:00 | |
| *** tzn has quit IRC | 12:05 | |
| *** Mech422 has quit IRC | 12:07 | |
| *** coolsvap_ has quit IRC | 12:09 | |
| *** salv-orl_ has joined #openstack-kolla | 12:13 | |
| *** salv-orlando has quit IRC | 12:16 | |
| *** eaguilar_ has joined #openstack-kolla | 12:24 | |
| *** tzn has joined #openstack-kolla | 12:27 | |
| *** eaguilar has quit IRC | 12:28 | |
| *** chopmann has joined #openstack-kolla | 12:32 | |
| *** chopmann is now known as chopmann_ | 12:32 | |
| *** chopmann_ has quit IRC | 12:33 | |
| *** tzn has quit IRC | 12:34 | |
| *** chopmann_ has joined #openstack-kolla | 12:34 | |
| *** tzn has joined #openstack-kolla | 12:35 | |
| *** jtriley has joined #openstack-kolla | 12:48 | |
| *** chopmann_ has quit IRC | 12:58 | |
| *** v1k0d3n has joined #openstack-kolla | 12:59 | |
| *** klint has quit IRC | 12:59 | |
| *** ppowell has joined #openstack-kolla | 13:01 | |
| *** ayoung has joined #openstack-kolla | 13:07 | |
| openstackgerrit | Merged openstack/kolla: Cleanup help string of install_type parameter https://review.openstack.org/325600 | 13:08 |
| *** daneyon has joined #openstack-kolla | 13:14 | |
| *** absubram has joined #openstack-kolla | 13:17 | |
| *** daneyon has quit IRC | 13:18 | |
| *** Liuqing has quit IRC | 13:20 | |
| *** sacharya has joined #openstack-kolla | 13:20 | |
| *** alyson_ has joined #openstack-kolla | 13:25 | |
| *** sacharya has quit IRC | 13:25 | |
| *** cu5 has quit IRC | 13:35 | |
| *** cu5 has joined #openstack-kolla | 13:35 | |
| *** dwalsh has joined #openstack-kolla | 13:39 | |
| *** mbound has joined #openstack-kolla | 13:42 | |
| *** mgoddard_ has joined #openstack-kolla | 13:44 | |
| *** dmk0202 has joined #openstack-kolla | 13:44 | |
| *** mgoddard has quit IRC | 13:47 | |
| *** sdake has joined #openstack-kolla | 13:51 | |
| *** sdake_ has joined #openstack-kolla | 13:56 | |
| *** mgoddard_ has quit IRC | 13:58 | |
| *** inc0 has joined #openstack-kolla | 13:58 | |
| *** sdake has quit IRC | 13:58 | |
| *** mgoddard has joined #openstack-kolla | 13:58 | |
| sdake_ | morning | 13:59 |
| inc0 | hello everybody | 13:59 |
| inc0 | hey sdake_ did you get a chance to test out customizations | 13:59 |
| inc0 | ? | 13:59 |
| sdake_ | inc0 i have no begun work on our plan yet | 14:00 |
| sdake_ | no/not | 14:00 |
| inc0 | kk so let's unblock mechanism patch and get this merged plz | 14:00 |
| sdake_ | inc0 i wanted to try friday but gerrit was un vacation | 14:00 |
| inc0 | yeah | 14:01 |
| sdake_ | andi was on pto sat/sun :) | 14:01 |
| *** vhosakot has joined #openstack-kolla | 14:07 | |
| mag009_ | morning all | 14:08 |
| inc0 | hey mag009_ | 14:09 |
| vhosakot | morning! | 14:09 |
| mliima | morning all | 14:12 |
| *** mgoddard_ has joined #openstack-kolla | 14:12 | |
| *** mgoddard has quit IRC | 14:15 | |
| *** mbound has quit IRC | 14:16 | |
| *** Mr_Broke_ has quit IRC | 14:21 | |
| *** Mr_Broken has joined #openstack-kolla | 14:22 | |
| *** Mr_Broken has quit IRC | 14:26 | |
| *** vhosakot has quit IRC | 14:37 | |
| *** salv-orlando has joined #openstack-kolla | 14:38 | |
| *** vhosakot has joined #openstack-kolla | 14:39 | |
| *** cfarquhar has quit IRC | 14:39 | |
| *** salv-orl_ has quit IRC | 14:42 | |
| *** cfarquhar has joined #openstack-kolla | 14:42 | |
| *** cfarquhar has joined #openstack-kolla | 14:42 | |
| *** sdake_ has quit IRC | 14:45 | |
| mag009_ | inc0 I'm having issue with : ceph : Mounting Ceph OSD volumes | 14:57 |
| mag009_ | [Errno 13] Permission denied: '/var/lib/ceph'" | 14:57 |
| mag009_ | I think we need to add become: yes | 14:57 |
| *** mgoddard_ has quit IRC | 14:58 | |
| *** tzn has quit IRC | 14:58 | |
| *** tzn has joined #openstack-kolla | 14:59 | |
| inc0 | well, yeah | 14:59 |
| inc0 | or chmod /var/lib/ceph | 14:59 |
| mag009_ | i found it weird tho | 14:59 |
| mag009_ | i've added become=yes to my ansible run | 14:59 |
| mag009_ | btw I think I've found some syntax problem the issue I had last Friday.. | 15:00 |
| inc0 | mag009_, do tell please | 15:00 |
| mag009_ | I think it's cleaner with a become: yes less code to add | 15:00 |
| inc0 | I always use become.. | 15:01 |
| mag009_ | I'm re-deploying as we speak :) | 15:01 |
| inc0 | I like that about kolla "I'm re-deploying as we speak" "OK I'll ask you again in 5min" | 15:01 |
| mag009_ | there : roles/neutron/tasks/deploy.yml | 15:01 |
| mag009_ | the when statement | 15:02 |
| mag009_ | or should be aligned with : and there's a space | 15:02 |
| wirehead_ | Indeed. Kolla-Kubernetes is still in the very early days, but I'm far more comfortable dealing with it than I ever was with DevStack. :D | 15:02 |
| mag009_ | I think thats why it failed for me last friday | 15:02 |
| *** daneyon has joined #openstack-kolla | 15:02 | |
| *** mliima has quit IRC | 15:03 | |
| rhallisey | wirehead_, sbezverk_ inc0, I added a bunch of new stuff in the queue | 15:03 |
| inc0 | hmm, you sure, doesn't it render as single line | 15:03 |
| rhallisey | for both kolla & kolla-k8s | 15:03 |
| inc0 | kk rhallisey looking at it | 15:03 |
| rhallisey | inc0, thx | 15:03 |
| inc0 | I've seen stuff in kolla, didn't checn k8s today yet | 15:03 |
| inc0 | rhallisey, did anyone deploy multinode k8s already? | 15:04 |
| rhallisey | not that I know of | 15:04 |
| *** dmk0202 has quit IRC | 15:05 | |
| *** daneyon has quit IRC | 15:06 | |
| mag009_ | how do I push my stuff as early stage ? | 15:07 |
| wirehead_ | rhallisey, I've already +2'd some of 'em. | 15:07 |
| rhallisey | wirehead_, thanks! | 15:07 |
| wirehead_ | inc0, I'm still making sure that everything actually properly provisions and works before I go farther. | 15:09 |
| *** tzn has quit IRC | 15:09 | |
| inc0 | btw wirehead_ you have Polish heritage? | 15:10 |
| wirehead_ | Yes. On my dad's side. | 15:10 |
| inc0 | I figured;) You have unmistakably polish second name | 15:10 |
| *** zhiwei has quit IRC | 15:10 | |
| wirehead_ | Well, my wife took my last name and is not Caucasian, which has caused some very amusing situations in the past. | 15:11 |
| inc0 | well, I'll try multinode;) stuff happends when you move beyond one server | 15:11 |
| inc0 | haha, I can imagine | 15:11 |
| inc0 | especially that you don't have super easy name too | 15:11 |
| inc0 | for US people | 15:11 |
| wirehead_ | My great grandparents changed the pronunciation but not the spelling at Ellis Island. | 15:12 |
| wirehead_ | That didn't fix anything. :D | 15:12 |
| wirehead_ | inc0, it's going to be pretty brokey till https://review.openstack.org/#/c/325613/ is merged. | 15:12 |
| patchbot | wirehead_: patch 325613 - kolla-kubernetes - Convert MariaDB to work without HostNetwork=True | 15:12 |
| Lyncos | inc0 is using base image mendatory for 'new' modules ? | 15:13 |
| *** tzn has joined #openstack-kolla | 15:13 | |
| Lyncos | I would like to start form alpine | 15:13 |
| inc0 | Lyncos, nothing is mandatory if it makes sense, but would break convention | 15:14 |
| Lyncos | ok | 15:14 |
| inc0 | I'd rather explore adding alpine as yet another distro alltogether | 15:14 |
| Lyncos | ok | 15:14 |
| wirehead_ | For the most part, Kubernetes is magical and well-behaved single-node apps magically work multi-cloud style as well. | 15:14 |
| inc0 | more work but might prove more beneficial | 15:14 |
| Lyncos | Agreed | 15:14 |
| *** _tzn has joined #openstack-kolla | 15:14 | |
| inc0 | that being said, there are things that alpine doesn't have yet | 15:15 |
| *** tzn has quit IRC | 15:15 | |
| wirehead_ | err. multi-node. | 15:15 |
| inc0 | we had that discussion before | 15:15 |
| inc0 | librbd is a no-no for example | 15:15 |
| inc0 | same as galera | 15:15 |
| wirehead_ | Yeah, I was going to comment. I had some 'fun' with Alpine and quickly went back to a more standard distro. | 15:15 |
| inc0 | so I don't think it's possible to move full alpine yet | 15:15 |
| inc0 | however nice idea that it might sound... it will be ugly during actual dev | 15:16 |
| inc0 | so Lyncos if you want to publish somethign to kolla repo, start from base (or at least clean ubuntu/centos) image | 15:17 |
| Lyncos | ok I'll do that way | 15:17 |
| *** dgonzalez has quit IRC | 15:17 | |
| inc0 | and we can think how to optimize our images at large | 15:18 |
| inc0 | Lyncos, is there any particular reason you want Alpine besides cutting few hundreds megs out of images? | 15:19 |
| *** dgonzalez has joined #openstack-kolla | 15:19 | |
| Lyncos | not reallt | 15:19 |
| Lyncos | I'm trying to do the felix container.. and I had a working example with alpine ... was trying to be lazy | 15:19 |
| inc0 | well locally you can do whatever you like | 15:20 |
| inc0 | if you'd like to push it up to kolla, that will require going along our standards | 15:20 |
| inc0 | however it's easier to debug stuff if they're similar | 15:20 |
| inc0 | and trugth be told, you wont gain anyting | 15:20 |
| inc0 | really you' | 15:20 |
| inc0 | ll lose space | 15:20 |
| Lyncos | I'll do the standard way | 15:20 |
| inc0 | as base image has to be pulled anyway | 15:21 |
| Lyncos | not a big problem | 15:21 |
| inc0 | for other containers | 15:21 |
| inc0 | so unless we move full alpine, it wont optimize anything:) | 15:21 |
| *** mliima has joined #openstack-kolla | 15:21 | |
| *** sacharya has joined #openstack-kolla | 15:21 | |
| inc0 | rhallisey, ad https://review.openstack.org/#/c/325613/ - you need bp for that? That will be requirement for HA | 15:24 |
| patchbot | inc0: patch 325613 - kolla-kubernetes - Convert MariaDB to work without HostNetwork=True | 15:24 |
| rhallisey | inc0, I just wanted the other blueprint to be included too | 15:25 |
| inc0 | in general, for any service managed by k8s for HA, you'll need k8s networkign | 15:25 |
| inc0 | kk | 15:25 |
| rhallisey | removing net=host is a bp | 15:25 |
| inc0 | so now when I think about it, what will l3 agents do?\ | 15:25 |
| inc0 | You'll need net-host | 15:26 |
| *** sacharya has quit IRC | 15:26 | |
| inc0 | for them | 15:26 |
| inc0 | and keepalived | 15:26 |
| inc0 | (well routers ha have it embedded) | 15:26 |
| *** cu5 has quit IRC | 15:26 | |
| inc0 | let me think if that will break with k8s.. | 15:26 |
| inc0 | shouldn't but k8s won't really do anything for them | 15:27 |
| inc0 | so you want to keep vrrp for HA for this one imho | 15:28 |
| inc0 | or Pacemaker really as it will need fencing regardless of orchiestration | 15:28 |
| inc0 | uhh, this one will be harder | 15:29 |
| *** dwalsh has quit IRC | 15:35 | |
| *** mgoddard has joined #openstack-kolla | 15:35 | |
| sbezverk_ | inc0 trying to undersatnd why would you need keepalived and haproxy in k8s? I think all these is taken care of by k8s infra, no?? | 15:38 |
| inc0 | sbezverk_, so in 99% you're right | 15:38 |
| *** salv-orlando has quit IRC | 15:38 | |
| inc0 | and you shouldn't require haproxy at all | 15:39 |
| inc0 | but for neutron routers you need net-host and floating IP | 15:39 |
| inc0 | as gateway | 15:39 |
| *** salv-orlando has joined #openstack-kolla | 15:39 | |
| inc0 | I don't think you can solve network gateway with k8s | 15:39 |
| inc0 | and even if you could it'll never beat vrrp milisecond-scale speeds | 15:40 |
| inc0 | but having keepalived has it's cost | 15:40 |
| *** sacharya has joined #openstack-kolla | 15:41 | |
| inc0 | you need deterministic set of hosts | 15:41 |
| inc0 | hmm....or maybe not, let me see quickly our keepalived conf | 15:41 |
| inc0 | hmm, no you don't need, but you need priority per server | 15:42 |
| inc0 | so technically you could let keepalived float even | 15:42 |
| inc0 | but k8s would have to put it in lower priority than existing master | 15:43 |
| inc0 | in fact, that might be better HA than normal keepalived cluster | 15:43 |
| inc0 | now when I think about it | 15:43 |
| *** mikelk has quit IRC | 15:43 | |
| openstackgerrit | Marc-Andre Gatien proposed openstack/kolla: add become to mount osds getting a permission denied when it try to create /var/lib/ceph https://review.openstack.org/325999 | 15:43 |
| inc0 | so consider this, we have 3 hosts in keepalived cluster A is master (highest priority) B and X | 15:44 |
| inc0 | C* | 15:44 |
| inc0 | all of them with net-host | 15:44 |
| inc0 | if A dies, B will get floating IP | 15:44 |
| openstackgerrit | Marc-Andre Gatien proposed openstack/kolla: add become to mount osds getting a permission denied when it try to create /var/lib/ceph https://review.openstack.org/325999 | 15:44 |
| inc0 | so if k8s restarts A, you don't want needlessly to swap IP again back to A (on some other host)( | 15:44 |
| inc0 | so new A can stand up with lower priority than C | 15:45 |
| sbezverk_ | inc0 how do you make sure that neutron will go to the same node as keepalived? | 15:45 |
| inc0 | that's doable | 15:45 |
| inc0 | however I think neutron itself manages keepalived for rounters HA | 15:45 |
| inc0 | sbezverk_, you can ensure that all of pods containers are on same host | 15:45 |
| inc0 | in fact that's what pod mean | 15:45 |
| sbezverk_ | inc0 ok so you suggest keepalived container is launched under neutron pod? | 15:46 |
| inc0 | sbezverk_, not sure, so if we need to manage keepalived, it will be in pod with l3 agent | 15:47 |
| inc0 | however I think neutron deals with it on it's own | 15:47 |
| *** salv-orl_ has joined #openstack-kolla | 15:47 | |
| inc0 | you're from Cisco, you speak networking;) | 15:48 |
| sbezverk_ | inc0 sure ;-) but in this configuration your provider network can end up all over the place | 15:48 |
| inc0 | yeah, we need to figure this one out | 15:49 |
| rhallisey | https://github.com/kubernetes/contrib/tree/master/keepalived-vip | 15:49 |
| sbezverk_ | with neutron pod moving back and forth | 15:49 |
| sbezverk_ | I just do not see at least now how neutron can always figure out where physically provider network is | 15:49 |
| *** salv-orlando has quit IRC | 15:50 | |
| *** dwalsh has joined #openstack-kolla | 15:50 | |
| inc0 | rhallisey, nice, we might very well use this one | 15:51 |
| inc0 | sbezverk_, it won't - you need to force k8s to schedule neutron pods on hosts with provider network connectivity | 15:51 |
| sbezverk_ | inc0: rhallisey: ok nailing can work, would be interesting to hear k8s operators how they build cluster with regards to provider network | 15:53 |
| inc0 | yeah, it's good practice to limit exposure of nodes to provider network connectivity | 15:54 |
| sbezverk_ | inc0 exactly that is why I think they do not expose k8s compute nodes directly to provider network at all and use some sort of proxy | 15:55 |
| inc0 | so depends on configuration reallyh | 15:55 |
| inc0 | if you use DVR you need provider network for CNodes | 15:55 |
| inc0 | or network routable to provider network | 15:55 |
| inc0 | without DVR you can limit this to network nodes | 15:56 |
| sbezverk_ | inc0 absolutely but we are not talking about openstack typical topology here | 15:56 |
| *** sdake has joined #openstack-kolla | 15:56 | |
| sbezverk_ | I have never seen a real k8s production cluster | 15:56 |
| inc0 | without DVR you need provider network only on network nodes | 15:56 |
| inc0 | that's the reason we put haproxy on network nodes | 15:56 |
| sdake | morning take 2 | 15:56 |
| sdake | inc0 rhallisey i am submitting a general ops session on kolla | 15:57 |
| inc0 | hands-on labs? | 15:57 |
| sdake | i'll list both of yoou as co-pesenter | 15:57 |
| sdake | we are doing a separate lab | 15:58 |
| sdake | we got wait listed liast time | 15:58 |
| inc0 | sure why not | 15:58 |
| rhallisey | k | 15:58 |
| sbezverk_ | sdake I submitted cinder and iscsi session | 15:58 |
| mag009_ | inc0 I'm not using the precheck | 15:58 |
| sdake | re the lab, i expect allt he cores to help with that one if we get accepted | 15:58 |
| inc0 | btw are we going to have any presence in ops midcycle? | 15:58 |
| mag009_ | I'd rather add a tasks instead | 15:58 |
| mag009_ | in the run itself | 15:58 |
| inc0 | mag009_, well, you should;) | 15:58 |
| sdake | inc0 i wont be able to make it | 15:58 |
| inc0 | or in inventory | 15:58 |
| *** dmk0202 has joined #openstack-kolla | 15:58 | |
| inc0 | so I don't want become=true in playbook really | 15:58 |
| inc0 | I'd rather add precheck for this one | 15:59 |
| inc0 | and add become=true in inventory | 15:59 |
| inc0 | if you add become=true in playbook you'll prevent non-root users to deploy kolla | 15:59 |
| inc0 | even if they can prepare their env beforehand | 15:59 |
| inc0 | and we need this to be possible | 15:59 |
| mag009_ | but even in the precheck you'll need root to chmod+chown | 16:00 |
| inc0 | mag009_, precheck will not do chmod+chown, it will validate it | 16:01 |
| inc0 | we don't meddle with host | 16:01 |
| inc0 | so imagine this - I want kolla to run as non-root | 16:01 |
| inc0 | for security reasons | 16:01 |
| inc0 | but I can prepare my env beforehand | 16:01 |
| inc0 | so I manually chmod this dir | 16:01 |
| inc0 | then su to non-root user | 16:02 |
| mag009_ | ok but my question is when do you chown/create this /var/lib/ceph ? | 16:02 |
| mag009_ | you do it manually ? | 16:02 |
| wirehead_ | Gah, of course this discussion happens whilst I am biking to the train. | 16:02 |
| inc0 | manually | 16:02 |
| mag009_ | outside kolla | 16:02 |
| mag009_ | ? | 16:02 |
| inc0 | correct | 16:02 |
| inc0 | also in your case | 16:02 |
| inc0 | just put become=true in inventory | 16:02 |
| rhallisey | wirehead_, :) | 16:02 |
| inc0 | inline | 16:02 |
| inc0 | will work just fine | 16:02 |
| mag009_ | ok | 16:03 |
| mag009_ | i'll close that one then. | 16:03 |
| inc0 | mind adding a precheck tho? | 16:03 |
| inc0 | or want me to do it? | 16:03 |
| inc0 | because it's actually valid precheck to have | 16:03 |
| wirehead_ | sbezverk_ rhallisey inc0 : I think we’ll have succeeded if most of the services (c.f. Keystone, Heat, et al) are not HostNetwork but Neutron stil is. | 16:04 |
| mag009_ | i'll do it | 16:04 |
| inc0 | thank you | 16:04 |
| inc0 | you can use same patchset | 16:04 |
| mag009_ | I'll run a full run again I'm still having issue with the when statement of neutron | 16:04 |
| inc0 | interesting | 16:04 |
| inc0 | it was working fine for months;) | 16:04 |
| wirehead_ | I was doing some experimentation with Kubernetes and the limits of Services and HostNetwork on Friday. | 16:04 |
| inc0 | brb | 16:04 |
| sdake | mliima will you be at summit? | 16:05 |
| mliima | barcelona? | 16:05 |
| sdake | what does c.f. mean wirehead_ | 16:06 |
| *** _tzn has quit IRC | 16:06 | |
| *** mgoddard_ has joined #openstack-kolla | 16:06 | |
| *** dmk0202 has quit IRC | 16:06 | |
| *** mgoddard has quit IRC | 16:06 | |
| sdake | mliima ya thatss the location | 16:06 |
| wirehead_ | Oh, I typed ‘c.f.’ when I meant to type ‘e.g.’ | 16:06 |
| wirehead_ | c.f. means ‘compare’, veruss e.g. meaning ‘for example'. | 16:06 |
| sdake | got it, know hot ot use eg and ie correctly :) | 16:07 |
| mliima | not yet know sdake | 16:07 |
| *** dwalsh has quit IRC | 16:07 | |
| wirehead_ | But, yeah, I was going to poke you two, inc0 and rhallisey, about the scale thing. Operationally speaking, that’s going to be more complex than just a replica value template. | 16:09 |
| sdake | you mena scaling past 100-500 nodes or so? | 16:10 |
| mliima | go at the summit is a will that I have, but it's something i can't confirm today. | 16:10 |
| rhallisey | wirehead_, can you auto scale with horizon? | 16:10 |
| rhallisey | didn't coreos demonstrate that? | 16:10 |
| wirehead_ | AFAICT, we should be able to have a decent chunk of services autoscaled. | 16:10 |
| rhallisey | sdake, 1.2 can scale to 1000 nodes | 16:11 |
| rhallisey | wirehead_, I'd like to have scaling through the dashboard | 16:11 |
| rhallisey | that would be awesome | 16:11 |
| sdake | not ver ywell but yes i know its scale limits | 16:11 |
| sdake | what I was going to say is openstack has a limit of about 300 nodes | 16:11 |
| rhallisey | gotcha | 16:11 |
| wirehead_ | This should be slick as a greased lepper armadillo in practice, because if someone’s causing more Keystone traffic but less Horizon traffic, it’ll add more Keystone pods and reduce the Horizon pods. | 16:11 |
| sdake | to get past 300 nodes multiregion and cells are used | 16:12 |
| rhallisey | wirehead_, nice. That sounds like we need a BP for it | 16:12 |
| wirehead_ | Yeah. | 16:12 |
| rhallisey | can you describe what this looks like in a BP? | 16:12 |
| rhallisey | with relevant links | 16:12 |
| wirehead_ | Totally. | 16:12 |
| rhallisey | wirehead_, thanks :) | 16:13 |
| rhallisey | so your patch then, how about for now we just template? | 16:13 |
| rhallisey | then add this feature when it's relevant | 16:13 |
| sdake | note the default setup of kolla spins up each api service with a certain number of processes | 16:13 |
| wirehead_ | OK. | 16:13 |
| rhallisey | because adding it now may not make the best sense | 16:13 |
| sdake | jeffrey4l has a patch he is workign on or that has merged to hard limit this at 3 or so | 16:13 |
| sdake | right now it choses the number of cores on the machine as the default | 16:14 |
| sdake | it being all of openstack | 16:14 |
| *** callahanca has joined #openstack-kolla | 16:14 | |
| rhallisey | good to know | 16:14 |
| sdake | scaling api has detrimental affects on rabbitmq and mariadb | 16:14 |
| sdake | the more apis there are, the more sluggish rabbitmq and mariadb become | 16:15 |
| wirehead_ | Would we ever exist in a situation where Kolla supports service-specific rabbitmq and mariadb instances? | 16:15 |
| *** daneyon has joined #openstack-kolla | 16:15 | |
| sdake | wirehead_ that is called multiregion an multicell | 16:15 |
| rhallisey | maybe the one db per service approach could help | 16:15 |
| sdake | oh service sspecific | 16:16 |
| sdake | ya that would work for rabbitmq and maybe for mariadb | 16:16 |
| rhallisey | side car database containers | 16:16 |
| sdake | gah side car | 16:16 |
| sdake | don't use that word | 16:16 |
| rhallisey | ha | 16:16 |
| sdake | it reminds me of ruby on rails! | 16:16 |
| rhallisey | that's their word | 16:16 |
| rhallisey | symbiotic container | 16:17 |
| sdake | from what i know of openstack setup | 16:17 |
| rhallisey | that's harder to say :/ | 16:17 |
| wirehead_ | Just call it a facehugger container. | 16:17 |
| sdake | it should work to have each service in a seprate database | 16:17 |
| rhallisey | haha | 16:17 |
| sdake | those ruby people, always sused to tlak about "we will ljust solve it with a sidecar" | 16:18 |
| sdake | right before theyblew me off for 3 months | 16:18 |
| sdake | hapepned multiple times | 16:18 |
| *** Mech422 has joined #openstack-kolla | 16:18 | |
| Mech422 | Morning | 16:18 |
| sdake | hence heat was born :) | 16:18 |
| sdake | hey meh | 16:18 |
| sdake | hey Mech422 | 16:18 |
| rhallisey | maybe it can't be a side car though because that means there will be on db service per pod | 16:18 |
| wirehead_ | That was one thing I was wondering, w.r.t. sbezverk_ ’s BP for services, if we should create a service per database or otherwise make things more divisible. | 16:18 |
| wirehead_ | service per service-database. | 16:18 |
| rhallisey | it would have to be per service vs per pod | 16:18 |
| Mech422 | sdake: Morning :-) I think I fixed my ceph in partition issues... | 16:19 |
| wirehead_ | Yeah, sidecar doesn’t work well for DB. It might work for memcached. | 16:19 |
| sdake | Mech422 nice | 16:19 |
| Mech422 | sdake: It appears the root cause was stale partition info since kolla renames partitions and the kernel doesn't update /sys | 16:19 |
| wirehead_ | Depends on how sophisticated we want to get. memcached per sidecar works more towards the bigger case. | 16:19 |
| rhallisey | wirehead_, I'll check the kube objects and see what there is for this | 16:19 |
| wirehead_ | And will get in the way for the smaller case. | 16:20 |
| wirehead_ | We might use namespaces instead. | 16:20 |
| wirehead_ | Yeah, the more I think about it, that’s the correct way. | 16:20 |
| rhallisey | I suppose this could be a rc=1 | 16:20 |
| sdake | re services and adatabases erandomly sscheduled on hardware | 16:20 |
| Mech422 | sdake: patches are here: https://bugs.launchpad.net/kolla/+bug/1589309 but you probably won't like them as it requires running kolla-toolbox with root, so you can use sgdisk to get the 'real' partition names | 16:21 |
| openstack | Launchpad bug 1589309 in kolla "Problem Bootstrapping ceph from Partitions due to stale part. table" [Undecided,New] | 16:21 |
| wirehead_ | If you want to have a per-openstack-service mariadb, I think you’d want to keep the service names fixed but use a different namespace for each one. | 16:21 |
| sdake | operators at this point in time have a universal appeal to knowing "where" there controller applicaitons are running | 16:21 |
| rhallisey | sdake, why | 16:21 |
| *** vhosakot has quit IRC | 16:21 | |
| sdake | Mech422 we can run kolla-toolbox as sudo | 16:21 |
| rhallisey | the idea here would be you wouldn't know | 16:21 |
| *** ssurana has joined #openstack-kolla | 16:22 | |
| sdake | Mech422 do you ahve teh review available - i'll have a look and suggest how to fix it | 16:22 |
| *** ssurana has quit IRC | 16:22 | |
| Mech422 | sdake: Oh, I read in the docker man page sudo bad for docker? sudo would work fine for what I do... | 16:22 |
| sdake | rhallisey why is complex - and I'd be speculating | 16:22 |
| rhallisey | kk | 16:22 |
| sdake | but part if it is their training in the past | 16:22 |
| sdake | they have always done it that way in the past | 16:22 |
| sdake | but typicallly contorller nodes getspecial treatment in a rack setup | 16:23 |
| sdake | they uuallyl go at the TOR (called the top of rack) | 16:23 |
| sdake | they are marked specially so people dont muck with them | 16:23 |
| sdake | vs the compute nodes, which are more expendable ;) | 16:23 |
| wirehead_ | Also, debugging issues with config. It’s helpful to know if Kube is properly distributing pods. | 16:23 |
| *** vhosakot has joined #openstack-kolla | 16:24 | |
| wirehead_ | Understanding that I’ve never operated an OpenStack cluster in anger, I kinda want ot have the kolla-kubernetes command contain some useful tools like `kolla-kubernetes map` that would display an infra-level view of what’s running. | 16:24 |
| *** ssurana has joined #openstack-kolla | 16:25 | |
| sdake | wirehead_ my comments go beyond that | 16:26 |
| sdake | wirehead_ in a real scenario, i think ops wil want their controller workload running on specific gear | 16:26 |
| *** jmccarthy has quit IRC | 16:26 | |
| *** ssurana has quit IRC | 16:26 | |
| wirehead_ | Yeah, that’s why I’m kinda wondering if the operational case is going to have a kube cluster for just the controller workload and then compute nodes as a separate thing. | 16:27 |
| *** jmccarthy has joined #openstack-kolla | 16:27 | |
| *** ssurana has joined #openstack-kolla | 16:28 | |
| sdake | wirehead_ its hard to say - as nobody is really using k8s in production all that much | 16:28 |
| sdake | wirehead_ I dont think best practices have been discovered or defined | 16:28 |
| wirehead_ | Best Practices are often discovered the hard way. :P | 16:28 |
| wirehead_ | Gut feel says that the smallest of the servicible installs and the developers are going to want no differentiation, whereas the larger installs will want a constrained control plane. | 16:29 |
| rhallisey | wirehead_, ya we would steer controller services away from compute nodes | 16:29 |
| rhallisey | but what services on what controller nodes will be determined by kuber | 16:30 |
| wirehead_ | K8s itself has a flag for if you want it to schedule compute on the control plane or not. | 16:30 |
| rhallisey | I figured we would use labels in the pods | 16:30 |
| sdake | wirehead_ which flag yo utalking about | 16:31 |
| sdake | the best practice re control plane gear at the top of rack is driven by hurricanes storms etc | 16:32 |
| wirehead_ | rhallisey: http://kubernetes.io/docs/admin/multiple-schedulers/ | 16:32 |
| *** fragatina has quit IRC | 16:33 | |
| rhallisey | nice we can write out own scheduler :) | 16:33 |
| *** matrohon has quit IRC | 16:35 | |
| sdake | ya and the scheduler can schedule to specific gear | 16:35 |
| wirehead_ | --register-schedulable=false on the master node kubelet, sdake. More stuff that landed in 1.1 | 16:35 |
| sdake | your talking about the kubernetes control plane | 16:36 |
| wirehead_ | Yeah, | 16:36 |
| sdake | I am tlaking about the openstack control plane | 16:36 |
| wirehead_ | Yeah. So, in k8s, they realized that they needed to support both the case where control and compute were intermingled as well as the case where control and compute were segregated. I would suspect that we will end up in the same situation. | 16:36 |
| sdake | sound slike the way to handle control vs compute separation is via a pluggable scheduler and labeling | 16:37 |
| wirehead_ | Yah. | 16:37 |
| *** vhosakot has quit IRC | 16:38 | |
| wirehead_ | This actually wraps around to the goals that Craton has in mind. | 16:38 |
| *** vhosakot has joined #openstack-kolla | 16:38 | |
| inc0 | back | 16:44 |
| sdake | back on autoscaling, autosacaling is a cool idea | 16:45 |
| sdake | but if kubernetes thinks apis need to be scaled, it could lbe detrimental to the operation of the cloud | 16:45 |
| sdake | because rabbitmq and mariadb are not independently scalable | 16:45 |
| inc0 | so problem is as you said rabbitmq and maria isn't really scallable this way | 16:46 |
| inc0 | and they are performance bottleneck most of the time | 16:46 |
| inc0 | s | 16:46 |
| inc0 | at scale, rabbit dies first | 16:46 |
| sdake | so re various government requirements | 16:49 |
| sdake | I keep hearing about this password rotation thing and how its a blocker for kolla deployments | 16:49 |
| inc0 | that's keystone issue | 16:49 |
| sdake | we need to be able to change all passwords in the system not just keystone | 16:50 |
| inc0 | and maybe you'll need to change passwords.yml and call reconfigure every now and then | 16:50 |
| sdake | including database passwords | 16:50 |
| sdake | reconfigure doesn't do the job currently | 16:50 |
| inc0 | if recondigure doesn't change confis for mariadb, we need to fix it | 16:50 |
| inc0 | it's more important than only passwords rotation | 16:50 |
| inc0 | we need to be able to do everything | 16:50 |
| inc0 | btw. that remind me, non-ini config overrides | 16:50 |
| sdake | agree but incremental approach is better then fix everthing | 16:51 |
| inc0 | sdake, we just need to make reconfigure works for every service, that's it | 16:51 |
| sdake | and password rotations is going to end up blocking kolla deployments | 16:51 |
| *** Mech422 has quit IRC | 16:52 | |
| *** Mech422 has joined #openstack-kolla | 16:52 | |
| inc0 | sdake, so does reconfigure miss anything? | 16:52 |
| sdake | globals.ml ad passwords.yml... | 16:52 |
| inc0 | really? why | 16:53 |
| inc0 | ? | 16:53 |
| sdake | think about the external VIP example | 16:53 |
| sdake | (the one on the mailing list) | 16:53 |
| inc0 | I mean ansible re-creates all the configs | 16:53 |
| inc0 | hmm, let me check one thing | 16:53 |
| sdake | for a password rotation, first you must reconfigure the database, then all services that use the databases to use the new database passwords | 16:54 |
| sdake | reconfigure as is hass no dependenncy ordering | 16:54 |
| sdake | however for password rotation there is an ordering | 16:54 |
| sdake | same story for vip changing | 16:54 |
| inc0 | sdake, not true | 16:54 |
| inc0 | it does things in order in playbooks | 16:54 |
| inc0 | and mariadb is higher in order of playbooks than services | 16:55 |
| inc0 | that being said, it does require maintenance mode as it obviously can't be rolling change | 16:55 |
| sdake | reconfigure passswords = change all passwords estart meriadb with new password, change all services datbase access, reconfigure all services | 16:55 |
| sdake | change all services passwords | 16:55 |
| sdake | reconfigure all service password | 16:55 |
| *** fragatina has joined #openstack-kolla | 16:55 | |
| sdake | it not a simple reoconfigure | 16:55 |
| inc0 | it is exactly how it goes with deploy | 16:56 |
| mliima | clean all and deploy again? | 16:56 |
| inc0 | no, why? | 16:56 |
| sdake | right one mechanism for password rotation would be to stop the cluster then rotate teh passwords then star tthe cluster | 16:56 |
| inc0 | well for maria it will require custom SQL query we don't use now | 16:56 |
| sdake | but some password information itself is stored in the datbase, so that mechanism wont work | 16:57 |
| sdake | we definately dont watn to clean all :) | 16:57 |
| mliima | sure :) | 16:57 |
| inc0 | well, for keystone | 16:57 |
| inc0 | but we won't do keystone calls | 16:57 |
| inc0 | that's not our job to do | 16:57 |
| wirehead_ | sdake: In an ideal world, the passwords for mariadb would be continually rotated, a la Hashicorp Vault. | 16:57 |
| inc0 | we don't do keystone user-update | 16:57 |
| sdake | the functionality i think we want is replace old passwords.yml with a new one | 16:58 |
| sdake | and do it with minimal downtime | 16:58 |
| inc0 | no, that's not it | 16:58 |
| inc0 | so for changing mariadb pasword you need to call SQL | 16:58 |
| inc0 | for changing keystone passwords you need to call keystone | 16:58 |
| inc0 | and only after that update configs | 16:58 |
| inc0 | and we don't want to do either - calling sql or keystone | 16:59 |
| inc0 | that's operators job | 16:59 |
| sdake | we want kolla to be operable | 16:59 |
| sdake | if that means we automate rottation for them, thats whatit means ;) | 16:59 |
| inc0 | operable != replacing operator | 16:59 |
| sdake | password changes can be automated | 17:00 |
| sdake | and if they can be they should be | 17:00 |
| inc0 | we don't want to do too much because it will hurt our maintainability and provide serious security risk | 17:00 |
| inc0 | I disagree, I don't want kolla to call running keystone | 17:00 |
| sdake | i dont get either point | 17:00 |
| sdake | could you expand | 17:00 |
| inc0 | we mess sometghing there and we destroy running envi | 17:00 |
| sdake | kolla already calls keystone all the time | 17:00 |
| inc0 | no, it does not, just for bootstrap | 17:00 |
| sdake | operator messes something up there and they destroy running env | 17:00 |
| inc0 | but that's operator not kolla | 17:01 |
| inc0 | also what wirehead_ said, you want proper secret management anyway | 17:01 |
| sdake | secret management is different from rotation | 17:01 |
| sdake | orthogonal topics | 17:01 |
| inc0 | rotating passwords will require maintenance mode, something we don't do with kolla | 17:01 |
| sdake | a maintenance node? | 17:02 |
| inc0 | would need to support separate logic, also not doing it with kolla now | 17:02 |
| inc0 | yeah, you need to turn off everything prior to change | 17:02 |
| inc0 | I mean you can keep maria/rabbit running | 17:02 |
| sdake | you can do a rolling upgrade of a password change | 17:02 |
| inc0 | then turn on keystone and change password there | 17:02 |
| inc0 | then turn on services | 17:02 |
| inc0 | no, you can't | 17:02 |
| inc0 | think about it | 17:03 |
| inc0 | unless you have a period of time of both passwords being correct | 17:03 |
| inc0 | and that's not how auth works | 17:03 |
| sdake | well maybe its different then reconfigure | 17:03 |
| inc0 | it totally is if you want to do whole thing | 17:03 |
| sdake | but atleast in the us, we have this bunch of laws called sarbanes soxely | 17:04 |
| sdake | one of the requirements of SBO is password rotation every 3 months | 17:04 |
| inc0 | if you make change manual thing then reconfigure will distritbute configs | 17:04 |
| sdake | if your a public company, you are bound by SBO | 17:04 |
| inc0 | and that is what we do | 17:04 |
| inc0 | ok, still, not kolla job, operators | 17:04 |
| inc0 | and if that's the case, there are already tools in place | 17:04 |
| sdake | i disagree on whoms job it is | 17:05 |
| sdake | but it doesn't matter, because atm we have no technique documented on even how to do it | 17:05 |
| inc0 | and afaik it doesn't mean service passwords | 17:05 |
| inc0 | only user passwords | 17:05 |
| inc0 | and that's not something we deal with at all | 17:05 |
| sdake | user passwords are operator's problem ;) | 17:05 |
| inc0 | yup, and afaik these are only ones forced to be rotated | 17:05 |
| sdake | here is what i'm getting at | 17:06 |
| inc0 | I might be wrong | 17:06 |
| sdake | I dont have th efaintest idea how to rotate passwords today in kolla | 17:06 |
| sdake | inc0 definately not true, all secrets must be rotated not just user passwords | 17:06 |
| inc0 | but I've been in ops of financial company | 17:06 |
| inc0 | and we didn't rotate db passowrd for services every 3 months;) | 17:06 |
| inc0 | and we were PCI compliant | 17:06 |
| inc0 | sdake, it's exactly same procedure as ops does now | 17:07 |
| inc0 | you don't rotate passwords with kolla, you rotate them yourself and kolla only distributes congis | 17:07 |
| inc0 | configs | 17:07 |
| sdake | what i'm getting at is we should automate the rotation | 17:09 |
| inc0 | I disagree | 17:09 |
| sdake | at minimum we should document how to change passwords | 17:09 |
| sdake | because atm nobody has any idea how to rotate passwords | 17:09 |
| sdake | (in our community) | 17:09 |
| inc0 | so reason why I don't want to do it is because I don't want kolla to even know how to access db | 17:09 |
| inc0 | we're deployment tool not fleet management tool | 17:09 |
| inc0 | I have..operators have... | 17:10 |
| sdake | we are deployment management tool | 17:10 |
| inc0 | they do it in the very same way they did it till now... | 17:10 |
| *** rmart04 has quit IRC | 17:10 | |
| sdake | which is what? | 17:11 |
| sdake | i suspect most operators have automated rotation ;) | 17:11 |
| inc0 | ehh...login to mariadb, change passwords, make config update | 17:11 |
| sdake | suspect/know atleast sowme have | 17:11 |
| inc0 | use keystone client | 17:11 |
| rhallisey | sdake, deployment managment tool is debatable | 17:11 |
| inc0 | sdake, and it will work just fine | 17:12 |
| inc0 | only instead of their previous config distribution thing they will call reconfigure | 17:12 |
| inc0 | if they do it manually - fine, if they have tooling - fine | 17:12 |
| inc0 | still works | 17:12 |
| rhallisey | well I guess it depends on the exact definition of deployment management tool | 17:13 |
| rhallisey | manual vs automatic | 17:13 |
| inc0 | in any case, we have higher priorities now | 17:14 |
| inc0 | like merge config for non-ini | 17:14 |
| inc0 | without that reconfigure doesn't really work anywya | 17:14 |
| sdake | rhallisey here is our mission: | 17:17 |
| sdake | https://github.com/openstack/governance/blob/master/reference/projects.yaml#L2112 | 17:18 |
| sdake | notice the keyword "operating" | 17:18 |
| sdake | that implies we do more then deploy a cloud to day0 working function | 17:18 |
| sdake | so in my mind atleast, it isn't debateable ;) | 17:18 |
| inc0 | sdake, but you will never replace need for manual work for ops | 17:20 |
| sdake | i dont want to do that inc0 | 17:20 |
| inc0 | btw we need monitoring | 17:20 |
| inc0 | we need reconfigure to work 100% of times | 17:20 |
| inc0 | we need non-ini configs | 17:20 |
| sdake | i want to replace all the manual junk that requires manual intervention with the infrastructure | 17:20 |
| sdake | reconfgiure was eant to solve the /etc/kolla/config dir merging case | 17:22 |
| sdake | atleast one person expects it to handle /etc/kolla/globals.yml changes | 17:22 |
| sdake | we dont want operators logging in to nodes and having to bounce docker services | 17:23 |
| sdake | just to do a rotation for example | 17:23 |
| mliima | I think a good idea do something to automated db password rotation, but rotate db passowrd for services every 3 months, we can leave this manual. | 17:24 |
| inc0 | so they won't need to log in to nodes | 17:24 |
| inc0 | they need to change passwords.yml | 17:24 |
| inc0 | login to mysql | 17:24 |
| inc0 | login to keytstonme | 17:24 |
| inc0 | and use reconfigure | 17:24 |
| inc0 | no ssh involved at any point | 17:24 |
| *** dwalsh has joined #openstack-kolla | 17:25 | |
| sdake | thats good | 17:29 |
| *** diogogmt has joined #openstack-kolla | 17:29 | |
| sdake | can someone take on documenting how its done then? | 17:29 |
| wirehead_ | I think my operational tooling rule for anything built within the past 2 years is that you should be able to cycle all of your credentials with an automated step. Obviously, some of that is outside of the scope of Kolla, but it’s not all outside our scope. | 17:32 |
| *** ppowell has quit IRC | 17:33 | |
| sdake | inc0 you missed a good conversation I had with harlowja about godaddy equirements for kolla | 17:35 |
| *** mark-casey has joined #openstack-kolla | 17:35 | |
| sdake | the missing peices atm are multi-region support and multi-cell support | 17:36 |
| sdake | pbourke you around | 17:37 |
| inc0 | sdake, I'd love to read the log | 17:40 |
| inc0 | brb | 17:40 |
| sdake | let me see if i can find it | 17:40 |
| sdake | http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2016-06-03.log.html#t2016-06-03T23:53:11 | 17:41 |
| *** rahuls has joined #openstack-kolla | 17:42 | |
| *** daneyon_ has joined #openstack-kolla | 17:44 | |
| *** harlowja has joined #openstack-kolla | 17:48 | |
| *** sdake_ has joined #openstack-kolla | 17:49 | |
| *** sdake has quit IRC | 17:49 | |
| *** daneyon_ has quit IRC | 17:49 | |
| *** ppowell has joined #openstack-kolla | 17:51 | |
| *** rmart04 has joined #openstack-kolla | 17:53 | |
| *** sdake_ has quit IRC | 18:01 | |
| Mech422 | harlowja: your at GoDaddy? you in the PHX offices ? | 18:03 |
| harlowja | nah, sunnyvale offices | 18:03 |
| harlowja | but yes, to the overall question Mech422 ha | 18:03 |
| Mech422 | harlowja: cool - I'm at Limelight in PHX - we seem to trade a lot people back and forth :-) | 18:04 |
| harlowja | lol | 18:04 |
| Mech422 | harlowja: my buddy just left to go to your 'minime' (?) email product | 18:04 |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Replication controllers for Keystone, Memcached, RabbitMQ. https://review.openstack.org/324106 | 18:06 |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Convert MariaDB to work without HostNetwork=True https://review.openstack.org/325613 | 18:07 |
| *** ppowell has quit IRC | 18:08 | |
| *** athomas has quit IRC | 18:08 | |
| *** fragatina has quit IRC | 18:09 | |
| harlowja | coolio, i'm not sure what minime is, guess i should look into that, ha | 18:10 |
| *** mbound has joined #openstack-kolla | 18:11 | |
| *** fragatina has joined #openstack-kolla | 18:11 | |
| *** fragatina has quit IRC | 18:11 | |
| *** rmart04 has quit IRC | 18:12 | |
| inc0 | I'm back | 18:12 |
| inc0 | harlowja, hey, I meant to talk to you, sooo...I'm planning refactoring of build | 18:12 |
| inc0 | which basically you started | 18:12 |
| harlowja | inc0 cool | 18:12 |
| harlowja | what u gonna do? | 18:12 |
| *** rmart04 has joined #openstack-kolla | 18:12 | |
| inc0 | so I was thinking about decoupling dockerfile generation from build | 18:13 |
| inc0 | for start | 18:13 |
| harlowja | ya, the build task is to huge imho, lol | 18:13 |
| inc0 | break this up into multiple submodules | 18:13 |
| wirehead_ | To eat an elephant, you only need to carve off a slice, then repeat. | 18:13 |
| inc0 | revisit the idea of copying whole thing to /tmp | 18:13 |
| inc0 | lol | 18:13 |
| harlowja | other idea | 18:14 |
| harlowja | which also addresses your 'it looks like it froze' issue | 18:15 |
| wirehead_ | OMG, I wasted around 20-30 minutes with that. It looked like it froze, I canceled it, and then the docker dameon was wedged and thus it really did freeze. I had to restart the daemon to un-wedge it. | 18:16 |
| harlowja | arg, its not frozen, the log files :-P | 18:16 |
| harlowja | but depends on how much u guys want to do with curses | 18:16 |
| wirehead_ | It may just be a choice of curses or curses, tho. :D | 18:17 |
| inc0 | harlowja, I'm also lookng at logs from cells/multiregion | 18:17 |
| inc0 | Lyncos, around? | 18:17 |
| inc0 | you might need cells too | 18:18 |
| inc0 | right now it might require custom playbooks and heavy config lifting | 18:18 |
| inc0 | I never deployed multi-cell so I can't say for sur | 18:18 |
| inc0 | e | 18:18 |
| *** sdake has joined #openstack-kolla | 18:20 | |
| harlowja | so what i was thinking | 18:22 |
| harlowja | is to use http://urwid.org/ (which i've used before) | 18:22 |
| harlowja | split the screen into X segments (1 for each thread) | 18:22 |
| harlowja | then have say the logs go to each segment (as well as to files) | 18:22 |
| harlowja | so that way u could see what a thread is working on, the output its producing | 18:22 |
| harlowja | but not have it be all cluttered up | 18:23 |
| harlowja | i've done this before with https://github.com/harlowja/gerrit_view/#czuul | 18:23 |
| inc0 | for build? so I think we need stdout to be readable | 18:23 |
| harlowja | just think of each box there as a thread | 18:23 |
| *** sdake_ has joined #openstack-kolla | 18:23 | |
| harlowja | ya, this would make stdout be readable | 18:23 |
| harlowja | by having each box (see that link) be a thread + its logs | 18:23 |
| inc0 | otherwise our CI will be a bitch and I don't expect people to build at hand | 18:23 |
| inc0 | readable => easy for machine parsing | 18:24 |
| *** sdake has quit IRC | 18:24 | |
| harlowja | machine parsing can just read the individual log files that are created? | 18:24 |
| inc0 | I think people will build CI/CD around build pretty soon and will want something easy to parse rather than ncurses-like experience | 18:24 |
| harlowja | hmmm, odd | 18:25 |
| inc0 | but let's think about it for a moment | 18:25 |
| inc0 | right now it's bunch of stuff, true | 18:25 |
| harlowja | personally i like stdout being used for the meaningful messages, not being a dump for all the stuff | 18:25 |
| inc0 | well, we can do that with log level | 18:26 |
| harlowja | not really | 18:26 |
| inc0 | which you did partially already | 18:26 |
| inc0 | how about we'll create urwid-interface for build, but external to build itself? | 18:26 |
| inc0 | we could also put ansible into it | 18:27 |
| inc0 | I was actually thinking about ncurses kolla client') | 18:27 |
| inc0 | so build.py will throw bunch of stuff | 18:27 |
| inc0 | and will retain API | 18:27 |
| inc0 | we'll clean it up, but still | 18:27 |
| inc0 | and we'll create 2 things | 18:27 |
| inc0 | 1. urwid gui | 18:27 |
| inc0 | 2. jenkins configs to do it automatically | 18:28 |
| inc0 | how does that sound? | 18:28 |
| harlowja | hmmm | 18:28 |
| harlowja | so build.py would be like a low-level thing | 18:28 |
| harlowja | that u would build 'UI's on top of? | 18:28 |
| inc0 | and build.py will become a lib rather than self contained script | 18:28 |
| harlowja | ok, seems f air | 18:29 |
| inc0 | from kolla import build | 18:29 |
| harlowja | what would the build.py then output | 18:29 |
| inc0 | so we can still use python logger | 18:29 |
| inc0 | wanna go fancy, make your own handler | 18:29 |
| harlowja | would it expose notifications (a listener) | 18:30 |
| harlowja | well a logger is pretty low-level notifications, lol | 18:30 |
| inc0 | yeah | 18:30 |
| inc0 | and it's already in stdlib so all we need to do is LOG.exception("omg omg my build broke") | 18:30 |
| harlowja | i was thinking of something more structured | 18:31 |
| inc0 | like what? | 18:31 |
| harlowja | https://pypi.python.org/pypi/notifier | 18:31 |
| harlowja | have the build.py emit notifications | 18:31 |
| harlowja | then have a 'log_ui' that translates those things to log statements | 18:31 |
| harlowja | or that's an idea | 18:32 |
| inc0 | yeah, but that would require a running service | 18:32 |
| harlowja | no | 18:32 |
| harlowja | 'Simple in-memory pub-sub.' | 18:32 |
| harlowja | lol | 18:32 |
| inc0 | so build.py finishes and memory is freed | 18:32 |
| harlowja | its gonna be pretty hard to make a decent curses UI with a bunch of log statements | 18:32 |
| harlowja | :-P | 18:32 |
| inc0 | true enough | 18:32 |
| harlowja | (or in fact any UI that isn't just a raw dump of the logs, lol) | 18:32 |
| inc0 | raw dump of logs with grep will be improvement;) | 18:33 |
| harlowja | :-/ | 18:33 |
| inc0 | but I hear ya | 18:33 |
| harlowja | ya, that's why if u have at least notifications (like using the above library) u can at least structure them | 18:33 |
| harlowja | and have build.py emit those kind of things | 18:33 |
| inc0 | so what I don't want to end up with is running service with bunch of infrastructure that will have to run once and after that...just hang out there | 18:33 |
| harlowja | nah nah, the notifier lib is just in memory stuff | 18:34 |
| harlowja | not a full service | 18:34 |
| harlowja | it came from taskflow so that taskflow engine users can do things like | 18:34 |
| harlowja | engine.notifier.register("ON_START", my_Callback) | 18:34 |
| harlowja | and then when the internals of the engine emit a "ON_START" event, my callback would be triggered | 18:35 |
| harlowja | u just need to define what those events are | 18:36 |
| inc0 | yeah, I know, but well, I'm just thinking if we need it for simple thing that build should be;) | 18:36 |
| harlowja | on_build_start, on_build_progress, on_build_done | 18:36 |
| inc0 | and one atomic action | 18:36 |
| harlowja | build isn't so simple though | 18:36 |
| inc0 | no ha, if something breaks just re-run it and so on | 18:36 |
| harlowja | its about 3 or 4 stes :-P | 18:36 |
| harlowja | (steps) | 18:36 |
| harlowja | build_download_things, | 18:36 |
| harlowja | build_download_more_things | 18:37 |
| harlowja | build_actually_start | 18:37 |
| inc0 | first render templates | 18:37 |
| harlowja | ya, so there are a bunch of steps, if u can split them up into tiny pieces, then it imho becomes more possible to have a useful stdout (that isn't just a log dump) | 18:37 |
| inc0 | so let's do that | 18:38 |
| inc0 | let's start by splitting | 18:38 |
| inc0 | then go for import kolla.build | 18:38 |
| inc0 | then come back to drawing board and see where we ware | 18:38 |
| inc0 | are | 18:38 |
| harlowja | ok | 18:38 |
| inc0 | let me check if there is bp for that | 18:38 |
| *** rmart04 has quit IRC | 18:39 | |
| inc0 | sdake_, around? | 18:39 |
| sdake_ | shoot | 18:40 |
| inc0 | when we at it, can we take longer look at our blueprints? | 18:40 |
| *** cu5 has joined #openstack-kolla | 18:40 | |
| inc0 | so for example I can't see non-ini merge config, which is essential | 18:40 |
| inc0 | we need to add bp for refactioring of build | 18:40 |
| *** ppowell has joined #openstack-kolla | 18:41 | |
| sdake_ | anyone can add blueprints | 18:41 |
| inc0 | it's more about looking at what we have, add missing and reprioritize it all | 18:41 |
| inc0 | harlowja, https://blueprints.launchpad.net/kolla/+spec/build-refactor | 18:45 |
| inc0 | let's make a session on next meeting to brainstorm possible improvements | 18:46 |
| inc0 | I'll add this to agenda | 18:46 |
| harlowja | cool | 18:49 |
| inc0 | so I'll start breaking the code | 18:50 |
| inc0 | that didnt came out right. | 18:50 |
| inc0 | but is true nonetheless | 18:51 |
| harlowja | lol | 18:51 |
| harlowja | ok with me | 18:51 |
| sdake_ | lets wait on refactor | 18:52 |
| inc0 | on what sdake_ ? | 18:52 |
| sdake_ | until after our customization refactor is done | 18:52 |
| sdake_ | or atlaest the build.py part is done | 18:52 |
| inc0 | hold on | 18:52 |
| sdake_ | it may be that way already | 18:52 |
| inc0 | these are orthogonal | 18:53 |
| vhosakot | sdake_: what do you think about this ? :) --> http://paste.openstack.org/show/508436/ | 18:53 |
| sdake_ | they touch the same code base do they not? | 18:53 |
| inc0 | well, I can rebase my code to it | 18:53 |
| sdake_ | vhosakot hotness dude | 18:53 |
| inc0 | easily | 18:53 |
| sdake_ | inc0 cool that wfm then - i'm lazy :) | 18:53 |
| vhosakot | sdake_: will push the PS today.... magnum is working now | 18:53 |
| inc0 | but really we're waiting for you | 18:53 |
| inc0 | in my mind customization is already solved | 18:54 |
| vhosakot | sdake_: magnum needs dynamic variables in jinja2 file | 18:54 |
| vhosakot | I'll push the PS today | 18:54 |
| vhosakot | sorry I took more time as I had to understand magnum and debug all sorts of errors | 18:54 |
| sdake_ | i know we are waiting on my prototyping | 18:55 |
| sdake_ | working on scraping off a block to work on it | 18:55 |
| inc0 | don't block good work on that plz | 18:56 |
| inc0 | we need to refactor build.py | 18:56 |
| *** mbound has quit IRC | 18:57 | |
| *** sdake has joined #openstack-kolla | 18:58 | |
| *** sdake_ has quit IRC | 19:00 | |
| clayton | so we don't use kolla, but we do run openstack in docker and found an issue recently that I think might affect kolla also | 19:00 |
| inc0 | clayton, what's that? | 19:00 |
| sdake | clayton cool care to share | 19:00 |
| clayton | The issue we found is that nova-compute does a volume mount of /run, but doesn't mount /run/netns in shared mode | 19:00 |
| *** fragatina has joined #openstack-kolla | 19:00 | |
| clayton | if you run routers/dhcp/metadata on the same node as nova-compute, you'll end up accidently capturing the netns mounts | 19:01 |
| clayton | which will make network namespaces undeletable | 19:01 |
| clayton | until the nova-compute container is restarted | 19:01 |
| inc0 | hmm...interesting | 19:01 |
| inc0 | that will appear on DVRs right? | 19:01 |
| clayton | for us the fix was to mount /run/libvirt and /run/openvswitch into the nova-compute container instead of /run | 19:02 |
| clayton | yes, or if (like us) you run routers on compute nodes | 19:02 |
| inc0 | good catch, thanks | 19:02 |
| clayton | we also ran into an issue on trusty with the 3.13 kernel not properly sharing mounts even with the shared flag | 19:02 |
| inc0 | yeah, I believe we have note of it in our docs | 19:03 |
| clayton | it works fine while the container is running, but if you restart the container, it the existing namespace mounts aren't created in shared mode | 19:03 |
| inc0 | and a workaround | 19:03 |
| clayton | which makes it capture the mount. the issue doesn't appear to exist on the 4.4 kernel that xenial ships with | 19:03 |
| clayton | inc0: have a link for that? | 19:04 |
| clayton | I'm curious what other pitfall I'm going to hit next :) | 19:04 |
| inc0 | yeah hold on, not sure if that's fix for this problem | 19:04 |
| inc0 | well if you' | 19:05 |
| inc0 | you'd use kolla we could hit pitfalls together | 19:05 |
| inc0 | it's always happier to share a suffering | 19:05 |
| clayton | I went to the feedback sessions in austin :) | 19:05 |
| inc0 | so we run mount --make-shared /run | 19:06 |
| inc0 | however I don't think that's fix for issue at hand | 19:07 |
| clayton | I think that newer versions of iproute2 do that automatically | 19:07 |
| inc0 | we will be moving to 16.04 for O | 19:07 |
| inc0 | so not sure if we should even care about that really | 19:08 |
| *** callahanca has quit IRC | 19:08 | |
| clayton | well, shared not working properly on 3.13 is the thing that will at least get us on the 16.04 kernel. we have other work that has to be done before we can start moving systems to xenial proper | 19:08 |
| clayton | http://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/tree/ip/ipnetns.c#n639 | 19:09 |
| clayton | the iproute2 package from trusty has that patch | 19:10 |
| *** cu5 has quit IRC | 19:10 | |
| *** callahanca has joined #openstack-kolla | 19:13 | |
| *** mliima has quit IRC | 19:21 | |
| *** salv-orl_ has quit IRC | 19:24 | |
| *** salv-orlando has joined #openstack-kolla | 19:24 | |
| *** cu5 has joined #openstack-kolla | 19:25 | |
| *** Serlex has left #openstack-kolla | 19:25 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Make build.py importable lib https://review.openstack.org/326108 | 19:28 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Make build.py importable lib https://review.openstack.org/326108 | 19:30 |
| PyroMani | We've working on a new AZ for our own OS cloud at the office and we've started with 16.04 on the host machines. At this moment we run into issues around docker containers and we're questioning ourself if we should use Ubuntu 16.04 on the hosts or CentOS 7. | 19:31 |
| inc0 | yeah I keep hearing about issues around xenial | 19:31 |
| *** matrohon has joined #openstack-kolla | 19:32 | |
| inc0 | CentOS 7 is much more tested by us | 19:32 |
| inc0 | I probably should move my dev env to xenial to get this feeling;) | 19:32 |
| PyroMani | We already downgraded our docker version from the latest in the repo to 1.10.x to match the specs in kolla. | 19:32 |
| inc0 | PyroMani, 1.11 works well | 19:32 |
| inc0 | no need to downgrade | 19:32 |
| clayton | we | 19:32 |
| inc0 | we should upgrade docs. | 19:32 |
| inc0 | update* | 19:32 |
| inc0 | 1.10 is minial version tho | 19:32 |
| clayton | we've had issues with containerd timeouts on 1.11.1. not scheduled to be fixed until 1.12 | 19:32 |
| PyroMani | Exactly, same here | 19:33 |
| clayton | PyroMani: our problems were on trusty with the 3.13 kernel, fwiw | 19:33 |
| PyroMani | Currently we see this: Error response from daemon: Cannot start container ABCD.... : [10] System error: could not synchronise with container process | 19:33 |
| PyroMani | Error: failed to start containers: nova_api | 19:33 |
| clayton | so that's not xenial specific | 19:33 |
| PyroMani | Tomorrow we're going to investigate this message: https://github.com/openstack/kolla/commit/e31d85e71c292b3af5f9a99b913f1d85fc0bcbad | 19:34 |
| PyroMani | As the dockers refuse to start after a few restarts and we have to reboot the whole host | 19:35 |
| clayton | PyroMani: I've had some luck with manually removing all containers then restarting docker-engine | 19:35 |
| clayton | but clearly that's not a great solution | 19:35 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Replication controllers for Keystone, Memcached, RabbitMQ. https://review.openstack.org/324106 | 19:36 |
| PyroMani | clayton: I agree on that | 19:36 |
| PyroMani | As far as we can see the issues does not lie in the containers themselves but the docker controller on the hosts | 19:38 |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Adding documentation for labels https://review.openstack.org/324110 | 19:40 |
| *** mbound has joined #openstack-kolla | 19:40 | |
| Mech422 | PyroMani: how is Openstack AZ stuff working for you in general ? I've gotten the dreaded 'single pane of glass' mgmt. request... | 19:43 |
| Mech422 | PyroMani: not sure if we should bother with AZ, or go with independent clusters and some sort of api driven dashboard | 19:44 |
| *** mliima has joined #openstack-kolla | 19:45 | |
| *** godleon has joined #openstack-kolla | 19:46 | |
| PyroMani | Mech422: unfortunately, this will be our first own build AZ as the previous was build and maintained for us. Our vision on it is that we like the idea of a single pane in which the clusters reside (managed with AZ's). | 19:50 |
| PyroMani | Mech422: Also, if we want to perform migrations it's easier to do between AZ's (even live migrations should be possible). | 19:51 |
| PyroMani | Mech422: It has it's downsides as well as you introduce dependencies between the clusters | 19:51 |
| Mech422 | PyroMani: yeah - so far cross AZ migration is 'out of scope' - but I'm sure that will change the day the cluster goes live :-P | 19:52 |
| mag009_ | i'm having issue with the kolla-build from master repo | 19:52 |
| mag009_ | cinder-base fail | 19:52 |
| PyroMani | Yeah, probably within the first month ;) | 19:52 |
| Mech422 | PyroMani: my big concern atm is ceph - not sure I want to run a ceph 'cluster' with nodes in Ukraine, Chicago and phoenix | 19:53 |
| Mech422 | PyroMani: just seems like latency would be a killer | 19:53 |
| PyroMani | Mech422: we clearly choose to ditch ceph for our first own AZ | 19:53 |
| mag009_ | anyone got that : Untar re-exec error: exit status 1: output: archive/tar: invalid tar header | 19:53 |
| *** mliima has quit IRC | 19:54 | |
| PyroMani | Mech422: we had plenty of problems with the managed AZ. A single problem in ceph took down the whole AZ cripling numerous hosts | 19:54 |
| Mech422 | PyroMani: oh ouch :-( | 19:54 |
| *** openstackstatus has quit IRC | 19:55 | |
| PyroMani | Mech422: so for our first AZ we decided to strip quite a lot of the original and build new AZ with additional services / redundancy each time so we can manage certain issues better | 19:55 |
| PyroMani | Mech422: for instance, we can have an AZ with almost no redundancy and one with triple redundancy. Then it's up to the application we need to host wether we host it on the redundant platform or the non redundant | 19:57 |
| *** openstackstatus has joined #openstack-kolla | 19:57 | |
| *** ChanServ sets mode: +v openstackstatus | 19:57 | |
| Mech422 | PyroMani: makes sense - sort of a 'staged rollout' ... | 19:57 |
| PyroMani | Mech422: yeah, instead of a dive in the deep. | 19:57 |
| Mech422 | PyroMani: ukraine will probably be my 'test bed' - its got the oldest equipment and the worst connectivity... | 19:58 |
| Mech422 | PyroMani: if I can get stuff working there, it should work anywhere :-p | 19:58 |
| PyroMani | Mech422: this way you can get a feeling of the core system and its working. | 19:59 |
| PyroMani | Mech422: if the connectivity isn't great I advise against ceph even more :p | 19:59 |
| Mech422 | PyroMani: heh | 19:59 |
| PyroMani | Currently I'm working on a test suite for our own stack | 19:59 |
| PyroMani | (started today :P) | 20:00 |
| Mech422 | PyroMani: I've been working on 'pre-scripting' kolla, playbooks to bridge what our provisioning sets up, and what kolla expects | 20:00 |
| PyroMani | Cool :) Is there a great difference? | 20:02 |
| PyroMani | I'm used to work with Chef, so for kolla I had to read into Ansible :P | 20:02 |
| Mech422 | PyroMani: Umm - we use OVS for base network connectivity - thata a killer... | 20:03 |
| Mech422 | we just setup a pop in baghdad, no way I'm going there for hands on if I screw networking :-P | 20:03 |
| PyroMani | Mech422: Ah, wouldn't want to do that either xD | 20:03 |
| Mech422 | PyroMani: then just basic stuff like we partition/raid our drives, and kolla/ceph want to use full devices | 20:04 |
| Mech422 | PyroMani: updating kernels, installing repo's, etc | 20:04 |
| Mech422 | PyroMani: oh - I added a kolla 'checkout' play that does the git clone and automatically installs our patches | 20:05 |
| Mech422 | PyroMani: thats a small thing, but makes life 10x easier | 20:05 |
| PyroMani | Mech422: Haha, I can imagine :) | 20:06 |
| PyroMani | Would the pre scripting work on bare metal using dracs as well? :P | 20:06 |
| Mech422 | PyroMani: umm - dunno drac.. we provision images/std. configs via PXE | 20:07 |
| *** zhiwei has joined #openstack-kolla | 20:08 | |
| Mech422 | PyroMani: then my playbooks use the pre-configured root login | 20:08 |
| inc0 | so we had session about kolla+bifrost | 20:08 |
| inc0 | to address this | 20:08 |
| inc0 | on summit | 20:08 |
| Mech422 | inc0: ? | 20:09 |
| inc0 | bifrost - standalone ironic | 20:09 |
| inc0 | for bare metal deployment prior to kolla | 20:10 |
| inc0 | so idea is to deploy bare ubuntu/centos | 20:10 |
| Mech422 | inc0: yeah - thing is, I can't just change the company wide provisioning... | 20:10 |
| inc0 | prepare kolla-host scripts to configure host | 20:10 |
| mag009_ | hm | 20:10 |
| inc0 | yeah, and it's just on paper now too | 20:10 |
| mag009_ | kolla is deploying ironic too no ? | 20:10 |
| mag009_ | kolla can replace bitfrost | 20:10 |
| inc0 | not in standalone mode | 20:10 |
| mag009_ | right | 20:10 |
| Mech422 | inc0: oh - I got the ceph-on-partitions stuff working: https://bugs.launchpad.net/kolla/+bug/1589309 | 20:11 |
| openstack | Launchpad bug 1589309 in kolla "Problem Bootstrapping ceph from Partitions due to stale part. table" [Undecided,New] | 20:11 |
| inc0 | idea is that you install one node | 20:11 |
| inc0 | deploy bifrost and bootstrap all other nodes | 20:11 |
| inc0 | Mech422, cool! I'll read through that | 20:11 |
| inc0 | thanks | 20:11 |
| PyroMani | inc0: a bit like bootstack does? | 20:11 |
| Mech422 | inc0: hmm - i'll have to look at that - we have multiple types of system on the same dhcp/pxe domain - so I'm not sure we can use it - I can't hijack dhcp/pxe | 20:12 |
| *** zhiwei has quit IRC | 20:12 | |
| sdake | Mech422 our plan as outlined by sean-k-mooney : | 20:12 |
| sdake | we will use bifrost for bare metal bootstappign of openstack nodes | 20:12 |
| *** ppowell has quit IRC | 20:13 | |
| inc0 | PyroMani, I'm not familiar with bootstack | 20:13 |
| sdake | we will use ironic as container's implemented by jpeeler for bare metal as a service | 20:13 |
| Mech422 | sdake: cool :-) | 20:13 |
| Mech422 | I think ironic wants to control dhcp/pxe though ? | 20:13 |
| Mech422 | unless i'm mistaken ? | 20:14 |
| sdake | indeed it is mandatory it does so | 20:14 |
| inc0 | Mech422, yeah it has it's own pxc | 20:14 |
| inc0 | pxe | 20:14 |
| inc0 | server | 20:14 |
| sdake | the ccontainers jpeeler wrote are nearly functional | 20:14 |
| sdake | but not for standalone mdoe | 20:14 |
| inc0 | it uses it and ipmi-reboot machines | 20:14 |
| Mech422 | yeah - I need more of a 'ironic in ansible'... so I can just take the machine after provisioning and jigger it howerver needed | 20:14 |
| sdake | whereas bifrost is about standalone mode | 20:15 |
| sdake | Mech422 we need both - sean-k-mooney working on both | 20:15 |
| inc0 | Mech422, we call it kolla-host ;) something in the depths of "things to do" | 20:15 |
| Mech422 | ahh - very cool...I'll have to look at that | 20:15 |
| mag009_ | i've tested ironic and it's far from being perfect in standalone mode | 20:15 |
| cinerama | we're reviewing the larger patch for splitting out the service startups now and it should be able to land soon | 20:16 |
| mag009_ | it really depend on neutron to control dhcp | 20:16 |
| Mech422 | I'm currently re-provisioning my cluster with Foreman | 20:16 |
| sdake | cinerama cool that is fantastic news! | 20:16 |
| inc0 | Mech422, I use cobbler | 20:16 |
| cinerama | iirc it also needs to be rebased on the current master because we needed to update our boolean handling. | 20:16 |
| sdake | in my lab cobbler is used | 20:16 |
| sdake | but in my home lab i'm going to be using bifrost once it works :) | 20:17 |
| Mech422 | inc0: cobbler does seem more stable then foreman | 20:17 |
| Mech422 | but I'm not switching back now :-P | 20:17 |
| inc0 | *khe khe*crowbar | 20:18 |
| inc0 | Mech422, mind publishing a review with this partition stuff? | 20:19 |
| sdake | cinerama we have a 132 node lab | 20:19 |
| inc0 | let's discuss it there | 20:19 |
| sdake | cinerama we will have it for 6 weeks starting iirc 7/22 | 20:19 |
| inc0 | also if possible I'd love to squeeze in file-based osd too | 20:19 |
| sdake | cinerama it would be hepful to use bifrost on that lab gear during our work | 20:19 |
| inc0 | super useful for dev/ci stuff | 20:19 |
| inc0 | actually guys yeah, cinerama if you want to help us with bare metal part of testing that'd be great | 20:20 |
| Mech422 | inc0: have you used crowbar? that and razor looked interesting.... | 20:20 |
| cinerama | sdake, wow really? cool | 20:20 |
| inc0 | I used it with canonical maas for a moment, actually it did good job | 20:21 |
| inc0 | sdake, and make it 3 weeks instead of 6 | 20:21 |
| inc0 | we have only 3 weeks | 20:21 |
| Mech422 | inc0: the 'hands off' approach sounded awesome for production - but at home, I switch os's on the boxes too often | 20:21 |
| inc0 | so we really need to make most of it | 20:21 |
| sdake | oh y bad i thought we had 6 | 20:21 |
| Mech422 | the nice lil web gui makes that easy | 20:21 |
| inc0 | Mech422, so reason we use pyudev is because it's python library for ceph | 20:22 |
| inc0 | I mean we use it, because we don't liek screenscraping outputs | 20:22 |
| Mech422 | inc0: yeah - but the kernel doesn't update /sys even after running partprobe... | 20:23 |
| inc0 | problem is, I didn't find good alternative in python to list disks, partiitons and such | 20:23 |
| Mech422 | so poor pyudev gets screwed | 20:23 |
| Mech422 | inc0: yeah - I'm not sure there is anything - you really need to re-read the part manually as kernel doesnt | 20:23 |
| mag009_ | anyone use ironic in prod env not a lab? | 20:23 |
| wirehead_ | Rackspace’s OnMetal product is ironic. | 20:24 |
| Mech422 | inc0: re: code review - I have no idea what to do next? | 20:24 |
| Mech422 | I got as far as singing up for the openstack.org account | 20:25 |
| mag009_ | rackspace is the number 1 commiter on ironic | 20:25 |
| mag009_ | if imnot mistaken | 20:25 |
| Mech422 | err...signing up...I tried singing but got voted off the project | 20:25 |
| *** daneyon_ has joined #openstack-kolla | 20:27 | |
| *** alyson_ has quit IRC | 20:28 | |
| inc0 | hold on Mech422, let me find guide | 20:28 |
| sdake | harlowja around? | 20:28 |
| sdake | Mech422 got voted off which project? | 20:29 |
| inc0 | http://docs.openstack.org/infra/manual/developers.html | 20:29 |
| Mech422 | sdake: oh - I was joking about how bad my singing is :-) | 20:29 |
| sdake | oh\ | 20:30 |
| *** ayoung has quit IRC | 20:30 | |
| Mech422 | inc0: i was thinking of using Terraform for the bare-metal-to-kolla stuff | 20:31 |
| Mech422 | inc0: but the 'bare metal' support is pretty weak | 20:31 |
| inc0 | I've heard about that | 20:31 |
| inc0 | pretty new project isn;t it? | 20:31 |
| *** daneyon_ has quit IRC | 20:31 | |
| wirehead_ | It’s been around for at least a few years. | 20:32 |
| Mech422 | inc0: yeah - hashicorps answer to cloudformation/sparkle ? | 20:32 |
| *** dave-mccowan has quit IRC | 20:32 | |
| *** dave-mccowan has joined #openstack-kolla | 20:33 | |
| Mech422 | inc0: Oh - did you see: https://coreos.com/blog/torus-distributed-storage-by-coreos.html | 20:33 |
| inc0 | interesting | 20:38 |
| *** jtriley has quit IRC | 20:40 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Use the Kube endpoint to dictate state instead of etcd https://review.openstack.org/325503 | 20:40 |
| *** mbound has quit IRC | 20:41 | |
| sdake | 108f out today | 20:41 |
| * sdake groans | 20:41 | |
| sdake | atleast its not 114f like last friday | 20:42 |
| Mech422 | sdake: looks like the 'corporate' agreement for openstack is broken? https://secure.echosign.com/public/hostedForm?formid=56JUVGT95E78X5 | 20:42 |
| inc0 | Mech422, I wonder if they want to compete with ceph | 20:42 |
| inc0 | according to latest user survey ceph has ~60% of prod in openstack | 20:42 |
| Mech422 | sdake: heh - it was 120 here yesterday - PHX - where are you ? | 20:42 |
| Mech422 | inc0: the 'build object store on top' part sounds like it... | 20:42 |
| *** jabari has quit IRC | 20:43 | |
| *** jabari has joined #openstack-kolla | 20:43 | |
| Mech422 | inc0: problem is, its not compatible with VMs (at least not now). Ceph works for both docker and VM | 20:43 |
| wirehead_ | Heh, so 62 in San Francisco, 74 at my apartment. | 20:43 |
| wirehead_ | This is why I’m always wearing long-sleeved shirts. | 20:43 |
| Mech422 | wirehead_: oh god.. your in SFO...I'm sooo sorry. I can't believe the rents you guys pay :-P | 20:44 |
| wirehead_ | Well, I live 30 miles south of SF. | 20:44 |
| wirehead_ | The rents are slightly less bad. | 20:44 |
| sdake | my mortgage is 740/mo | 20:44 |
| sdake | ;-) | 20:44 |
| Mech422 | wirehead_: I lived in Cupertino, right behind 1 infinity loop :-) And sunnyvale, right behind a dumpster :-P | 20:45 |
| *** shardy has quit IRC | 20:45 | |
| inc0 | sdake, Mech422 do a PHX openstack user group;) | 20:45 |
| sdake | plus 1600 every 6 months for insurance and taxxes | 20:45 |
| Mech422 | sdake: Oh! you in phx too ? | 20:45 |
| sdake | yup | 20:45 |
| sdake | Mech422 didn't know you were | 20:45 |
| harlowja | sdake sorta, ha | 20:45 |
| *** mbound has joined #openstack-kolla | 20:46 | |
| sdake | harlowja quick quesiton about your setup and neutron | 20:46 |
| Mech422 | yeah - I'm out in mesa - sossaman and broadway | 20:46 |
| sdake | as in do you use neutron at gd | 20:46 |
| sdake | if not, what do you use | 20:46 |
| *** mbound has quit IRC | 20:46 | |
| harlowja | eck | 20:46 |
| inc0 | :D | 20:46 |
| sdake | and if so, how do yoou get it to scale to your node count | 20:46 |
| harlowja | we use it | 20:46 |
| *** mbound has joined #openstack-kolla | 20:46 | |
| harlowja | ya, let me see if u can get someone else to answer :-P | 20:46 |
| Mech422 | harlowja: have you guys played with Midonet ? | 20:46 |
| Mech422 | harlowja: I'm looking at trying to tie that into kolla eventually | 20:47 |
| clayton | sdake: they carry a bunch of local patches last I heard :) | 20:47 |
| sdake | harlowja we have 3 1k+ node setups that people want to do | 20:47 |
| inc0 | that's gonna be an interesting | 20:47 |
| * harlowja getting kris in here, ha | 20:47 | |
| clayton | harlowja: figured :) | 20:47 |
| harlowja | :-P | 20:47 |
| sdake | harlowja so I suspect since it seems a thing, we will be working on multiregion and cells | 20:47 |
| inc0 | Lyncos, mag009_ <- they're playing with Calico | 20:47 |
| sdake | it would be nice to scrape some tribal knowledge out of the actual deployments at this scale | 20:47 |
| harlowja | def | 20:47 |
| Mech422 | inc0: ohh.. wonder how that compares to midonet | 20:48 |
| harlowja | kris knows all | 20:48 |
| harlowja | lol | 20:48 |
| *** klindgren_ has joined #openstack-kolla | 20:48 | |
| klindgren_ | o/ | 20:48 |
| inc0 | however our main installment is neutron, and cells+neutron is.... | 20:48 |
| *** klindgren_ is now known as klindgren | 20:48 | |
| sdake | hey klindgren_ | 20:48 |
| Mech422 | inc0: nice thing about midonet was it was openstack/neutron + docker compatible | 20:48 |
| mag009_ | who asked for calico? | 20:48 |
| klindgren | harlowja, was saying you had some neutron questions or something | 20:48 |
| klindgren | ? | 20:48 |
| sdake | mag009_ only you thus far | 20:48 |
| harlowja | klindgren knows all | 20:48 |
| harlowja | he's the man | 20:48 |
| harlowja | lol | 20:48 |
| klindgren | well - I wouldn't go that far | 20:48 |
| sdake | mag009_ that said, if its integrateable, we hould integrate it into kolla | 20:48 |
| inc0 | mag009_, we're talking about networking on 1k+ installments | 20:49 |
| sdake | klindgren oh your the cat with all the infos about hwo to run openstack at scale :) | 20:49 |
| mag009_ | of course it is | 20:49 |
| mag009_ | :) | 20:49 |
| harlowja | klindgren actually sdake might have some cells questions also (so that kolla + cells works out) | 20:49 |
| harlowja | i'm just some newb | 20:49 |
| harlowja | lol | 20:49 |
| mag009_ | thats what we expect 2k+ servers | 20:49 |
| Mech422 | does anyone use the plumgrid stuff ? | 20:49 |
| inc0 | heard it's neat | 20:49 |
| klindgren | I dunno about at scale. How about at a size and grwoth rate that makes me concerned :-D | 20:50 |
| mag009_ | If I can get pass those bugs with ansible 2.1 | 20:50 |
| sdake | klindgren atm kolla has been tested at 64 nodes | 20:50 |
| mag009_ | i might be able to push my changes... | 20:50 |
| sdake | we had to tune 3 variabbles in nova to get it to work | 20:50 |
| inc0 | so, we're all here because we want kolla on 1k+, for that we need cells and other stuff | 20:50 |
| Mech422 | inc0: yeah - sounds pricey though - wondering if its worth it | 20:50 |
| *** dwalsh has quit IRC | 20:50 | |
| harlowja | Mech422 are u paul bourke ? | 20:50 |
| sdake | kolla works really well at this size | 20:50 |
| sdake | harlowja no that is pbourke | 20:50 |
| inc0 | but we also need to figure out netowkring | 20:50 |
| harlowja | kk | 20:50 |
| Mech422 | harlowja: no - I'm Steve...but I answer to hey you :-) | 20:51 |
| harlowja | hey u | 20:51 |
| harlowja | lol | 20:51 |
| sdake | klindgren the two answers I know about for handling scale are multiregion and cells | 20:51 |
| sdake | klindgren to do multiregion, basically you setup fernet in keystone, then run one dedicated keystone + one dedicated HA database for keystone | 20:51 |
| harlowja | sdake the other thought i was having with my manager, is to delay the nova-compute container (until further proven) | 20:51 |
| sdake | and all nodes use that keystone | 20:51 |
| harlowja | use kolla for all the other stuff though | 20:51 |
| sdake | harlowja ya i hear that one alot | 20:52 |
| klindgren | k | 20:52 |
| *** shardy has joined #openstack-kolla | 20:52 | |
| inc0 | harlowja, so kolla won't be the bottleneck here | 20:52 |
| harlowja | klindgren though may get a lab we can mess around with this more, right klindgren :-P | 20:52 |
| sdake | klindgren is that accurate (multiregion implementation) and if so, do you have configs you can provide us that deploy it that way? | 20:52 |
| inc0 | issue is, how to configure stuff so it will work | 20:52 |
| inc0 | it's about what we want at the end, kolla will get us there | 20:52 |
| klindgren | we dont do multregion | 20:52 |
| klindgren | we use cells | 20:52 |
| inc0 | klindgren, do you use neutron + cells tho? | 20:52 |
| clayton | we do multi-region, but not for scaling purposes | 20:52 |
| *** dcwangmit01 has quit IRC | 20:53 | |
| klindgren | each dc we deploy in is also 100% independent of another DC, from an openstack component perspective | 20:53 |
| sdake | klindgren how does neutron scale to 1k+ nodes with cells? | 20:53 |
| klindgren | they all tie back into the same replicated AD for auth and things | 20:53 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: The Keystone bootstrap job need to run a db sync https://review.openstack.org/325665 | 20:53 |
| *** dcwangmit01 has joined #openstack-kolla | 20:53 | |
| sdake | AD = microsofts active directory product? | 20:54 |
| klindgren | so we dont yet have a single place that is at 1k nodes. | 20:54 |
| klindgren | largest spot so far is ~750 nodes so far | 20:54 |
| inc0 | klindgren, so it's almost like multi region | 20:54 |
| Mech422 | klindgren: yeah - our stuff all ties back to ldap for auth - slooowwww :-/ | 20:54 |
| inc0 | klindgren, how many cells in this 750? | 20:55 |
| klindgren | but have ~700 nodes coming online in the next few quarters | 20:55 |
| klindgren | inc0 - 2 cells right now | 20:55 |
| inc0 | and single neutron across these 2? | 20:55 |
| klindgren | with a third coming online to handle ironic | 20:55 |
| sdake | so 350 nodes per cell | 20:55 |
| klindgren | yes | 20:55 |
| klindgren | single neutron | 20:55 |
| inc0 | hmm...ok that's cool info | 20:55 |
| klindgren | I should not that we do networking... differently :-) | 20:56 |
| inc0 | so neutron can handle 750 nodes, I don't know when it starts to break | 20:56 |
| sdake | klindgren could you expand - we are stuck on the networking scalability part | 20:56 |
| klindgren | we dont do private networks at all - everything is a provider network | 20:56 |
| inc0 | uhh... | 20:56 |
| inc0 | I see | 20:56 |
| harlowja | (aka, no sdn) | 20:56 |
| clayton | klindgren: are you guys doing any tenant networks? | 20:56 |
| clayton | nm :) | 20:56 |
| klindgren | we do do floating ip's, but we extended neutron to handle that via route injection into the network | 20:57 |
| klindgren | also our L2 networks don't extend past the TOR for the HV's | 20:57 |
| sdake | are tenant networking hard to do with neutron at scale? | 20:57 |
| klindgren | so depending on the DC we have a max of 44 or 96(? I think) servers per rack | 20:57 |
| klindgren | *shrug* - I dont think virtualized tunnels does anything but virtualzie all the problems with L2 and take a way a decade + of tooling.knowledge on how to trouble shoot the issue | 20:58 |
| harlowja | and make bling bling for someone | 20:58 |
| harlowja | lol | 20:58 |
| klindgren | internal we run a L3 folded-clos network. So we are bascially mapping VM's directly onto that network | 20:58 |
| klindgren | since 99.9% of our users give 0 shits about private networking. | 20:59 |
| sdake | sbezverk_ around? | 20:59 |
| inc0 | klindgren, uhh | 20:59 |
| klindgren | support for this is getting added into neutron under the routed network spec. AKA segmented network support. | 20:59 |
| inc0 | well, that is an open question then, what do we do with networking if cells land | 21:00 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Add the KOLLA_KUBERNETES flag to containers https://review.openstack.org/325675 | 21:00 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: The Keystone bootstrap job need to run a db sync https://review.openstack.org/325665 | 21:00 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Use the Kube endpoint to dictate state instead of etcd https://review.openstack.org/325503 | 21:00 |
| inc0 | also, I think we should have separate plays for 1k+ multicell and our normal | 21:00 |
| klindgren | but TL;DR - our use of neutron at current is bascially assign IP from network, and when someone does a floating ip call inject a route into the network to route traffic for that IP to the vm. | 21:00 |
| inc0 | since configs will differ tremendously | 21:00 |
| sdake | klindgren do you run the agents on only one node? | 21:01 |
| klindgren | so in mitaka - everyone has cells | 21:01 |
| inc0 | mag009_, do you do multi-cell? | 21:01 |
| klindgren | we run the L2-agent everywhere | 21:01 |
| klindgren | and L3-agent we didn;t use to run, but due to some code changes we run it on a some node to satisfy some stupid HA thing that was added | 21:02 |
| klindgren | we now run it on a single node thats not connected to anything and I dont care if it actually works or not | 21:02 |
| sdake | do you use ovs, or linuxbridge, or your own thing? | 21:02 |
| klindgren | as long as it connects to rabbitmq and says its "alive" | 21:02 |
| klindgren | ovs | 21:02 |
| sdake | klindgren all scale problems revolve aorund rabbitmq and mariadb | 21:02 |
| sdake | since they are a single non-horizontally scalable component | 21:03 |
| *** rhallisey has quit IRC | 21:03 | |
| klindgren | we would like to move to linuxbridge - because its impler and due to security group usage - currently traffic flows through both ovs and linuxbridge anyway | 21:03 |
| inc0 | rabbit usually dies first | 21:03 |
| klindgren | we haven't had any issues with the DB | 21:03 |
| inc0 | yeah with your rather simplistic neutron usage that makes perfect sense | 21:03 |
| klindgren | we have issues scaling conductors per cell | 21:03 |
| sdake | using mariadb 5? | 21:03 |
| klindgren | and rabbitmq - ish | 21:04 |
| sdake | care about ipv6? | 21:04 |
| klindgren | 99% sure its 5. We dont actuall manage those servers. Our mysql team does. | 21:04 |
| klindgren | in the future yes, currently - not yet | 21:04 |
| klindgren | re: ipv6 | 21:05 |
| sdake | right | 21:05 |
| sdake | re cells - can you provide configuration files | 21:05 |
| sdake | so we can automate the deployment thereof | 21:05 |
| *** dwalsh has joined #openstack-kolla | 21:05 | |
| inc0 | sdake, hold on to that thought, we need to ensure it works well with neutron | 21:06 |
| inc0 | and find out scale limit | 21:06 |
| klindgren | so the question is cellsv1 or cellsv2 | 21:06 |
| inc0 | it's pointless to implement cells if neutron dies | 21:06 |
| sdake | is that a question? :) | 21:06 |
| mag009_ | we are not using multi-cell | 21:06 |
| inc0 | klindgren, cellsv2 even exist in any shape or form? | 21:06 |
| sdake | mag009_ your using multiregion? | 21:06 |
| klindgren | because everyone has cellsv2 in mitaka | 21:06 |
| klindgren | but you can only have a single cell | 21:07 |
| inc0 | ahh, I need to update my nova | 21:07 |
| mag009_ | yes (well... we only have 1 for now) | 21:07 |
| inc0 | mag009_, and rabbitmq works?:O | 21:07 |
| mag009_ | we only have one L3 setup and it's here in Montreal | 21:07 |
| sdake | i have heard that cells + neutron as a scalability solution isn't ideal | 21:07 |
| inc0 | how did you get rabbitmq working for 2k+ computes? | 21:07 |
| sdake | so another scalability option is multiregion | 21:07 |
| mag009_ | ... oh no no | 21:07 |
| mag009_ | we are not in prod. | 21:07 |
| mag009_ | the goal is to get to 2k+ | 21:08 |
| mag009_ | lol | 21:08 |
| mag009_ | we are not even in prod yet | 21:08 |
| inc0 | ahh that explains things | 21:08 |
| Mech422 | mag009_: your using calico ? | 21:08 |
| mag009_ | yes | 21:08 |
| klindgren | SO we have some patches to make cells work in cellsv1 with close to on par feature parity to a non-cells setup. By we I mean a combo between nectar, godaddy cern. | 21:08 |
| Mech422 | mag009_: any thoughts re midonet vs calico ? both looked usable for me... | 21:08 |
| mag009_ | the design is not finale yet still a lot of work to do! | 21:08 |
| sdake | klindgren i have heard cells v2 is in a massive rewrite atm | 21:08 |
| klindgren | On cells v1, the nova->neutron thing that doesn't work out of the box is the vif plugging communication. | 21:08 |
| Mech422 | mag009_: midonet just seemed 'easier' with the kuyr and neutron plugins | 21:08 |
| sdake | to make it work - basesd upon that forked work you talked about above (the cern functionality) | 21:09 |
| klindgren | afaik cellsv1 and cellsv2 has always been a massive change. | 21:09 |
| klindgren | One passes message between rabbitmq layers | 21:09 |
| klindgren | and the other connects dirrectly to the cell database to do things | 21:10 |
| sdake | cells v1 is database as a messaging layer solution? | 21:10 |
| mag009_ | midonet and calico are not quite the same | 21:10 |
| sdake | DB for messaging is an anti-pattern fwiw :) | 21:10 |
| klindgren | cells v1 = api rests -> rmq message on api cell -> nova-cells -> correct child cell rabbitmq -> pickedup by nova-cells in child -> nova-stuffs. | 21:11 |
| wirehead_ | cells v1 is… um. | 21:11 |
| klindgren | then some up/down message passing for events and the like | 21:11 |
| mag009_ | calico is pure L3 implementation which depend on a bgp client in our case we chose quagga instead of bird | 21:11 |
| mag009_ | so we have zero L2 in our infra.. everything is L3 | 21:11 |
| mag009_ | its what we wanted | 21:11 |
| sdake | WTB sbezverk_ | 21:11 |
| mag009_ | we use cumulus for our TOR | 21:11 |
| sdake | klindgren cellsv2 then goes directly to db? | 21:12 |
| sdake | and there is a db per cell? | 21:12 |
| clayton | there is a db per cell | 21:12 |
| klindgren | plus a "cell0" | 21:13 |
| inc0 | sdake, do you remember this nova-api db? | 21:13 |
| Mech422 | mag009_: we running arrista TORs, don't think i need to worry about L2 internally... | 21:13 |
| sdake | inc0 yes i do | 21:13 |
| klindgren | cell0 = things that didn't get scheduled to a cell | 21:13 |
| inc0 | that was because of cells | 21:13 |
| Mech422 | mag009_: does calico let you offload the forwarding at the edges like midonet ? | 21:13 |
| inc0 | nova api is singular | 21:13 |
| inc0 | Mech422, my understanding is that with calico edge would be another AS | 21:14 |
| sdake | clayton and a rabbitmq per cell? | 21:14 |
| Mech422 | inc0: yeah - I mean does calico require dedicated 'network' nodes, or can the hosts/edge route things directly to the vm.. | 21:14 |
| clayton | shared rabbitmq I assume. I''ve only seen stuff about extra databases | 21:15 |
| Mech422 | inc0: without having to go thru a 'middleman' box | 21:15 |
| inc0 | clayton, I don't think that would make sense as rabbitmq is what dies first | 21:15 |
| klindgren | I need to look at v2 - but under v1 - yes each cell has a rabbitmq and all the nova-computes/conductors/metadata for that cell connect to the child cells rabbitmq | 21:15 |
| clayton | talk to the nova guys about that then :) | 21:15 |
| klindgren | onything on the nova side that connect to the api cells rabbitmq is nova-cells | 21:15 |
| inc0 | yeah, I'm not sure about how it works really;) | 21:16 |
| mag009_ | each compute route directly to the vm | 21:16 |
| mag009_ | the bgp client is installed on the compute | 21:16 |
| sdake | clayton i did a bit - to russell | 21:16 |
| klindgren | So we looked at calico. Bascially take provider networks. remove any layer anywhere. | 21:16 |
| sdake | clayton he basicaly said with cells you run rabbitmq and mariadb in each celel | 21:16 |
| klindgren | Then use bpg per host to advertise to the network when IP's should be routed to it. | 21:17 |
| klindgren | bascially their is no "translation" | 21:17 |
| Mech422 | mag009_: I looked at juniper contrail last year too - but at the time, it seemed much more difficult to get going | 21:17 |
| sdake | the cell uses rabbitmq nad mariadb as a central point which reaches capacity at the cell limits | 21:17 |
| klindgren | or edge layer or anything like that. | 21:17 |
| sdake | clayton did you say you were using cells? | 21:18 |
| Mech422 | klindgren: Did you happen to look at midonet ? | 21:18 |
| klindgren | no - some people wanted us to look at plumgrid internally | 21:18 |
| Mech422 | that sounds nice - but pricey ? | 21:18 |
| klindgren | so I sat through a couple presentations of that | 21:18 |
| sdake | is plumgrid and midonet a competitor to neutron? | 21:19 |
| inc0 | sdake, nope, drivers | 21:19 |
| sdake | cool | 21:19 |
| mag009_ | Mech422: I personally didn't look at midonet so I don't know | 21:19 |
| Mech422 | klindgren: I'm not a networking guy, so I'm sort of lost as to the diff. between calico and midonet... | 21:20 |
| sdake | i think an initial easy answer to scalability is multi-region | 21:20 |
| Mech422 | it seems like both let you route packets directly from your 'edge' to the VM without going thru a neutron 'router' ? | 21:20 |
| *** mark-casey has quit IRC | 21:21 | |
| Lyncos | midonet seems to use network Overlays to emulate L2 .. which calico dosen't doo | 21:21 |
| Mech422 | Lyncos: umm...sorry, my network fu is weak... is that good or bad ? :-P | 21:22 |
| inc0 | Mech422, you know BGP? | 21:22 |
| Mech422 | inc0: no - I know what its supposed to do, but not anything about how it works | 21:22 |
| sdake | klindgren so if kolla goes and develops cellsv2, that is going to be wholy unsuitable for godaddy then - because gd is on cells v1? | 21:22 |
| Lyncos | Mech422 Network overlays add up a bit of overhead .. there is a cost of 'emulating l2' in a L3 environment | 21:22 |
| inc0 | so basically with bgp you think of networking differentlyu | 21:22 |
| inc0 | you forget about L2 really | 21:22 |
| inc0 | it's protocol used in internet itself | 21:23 |
| Lyncos | Mech422: and having L2 over L3 add up a layer of conlexity | 21:23 |
| inc0 | if you want vm connecting to vm, both have ip's | 21:23 |
| Mech422 | right - border gateway protocol for exchanging routes ? like OSPF ? | 21:23 |
| inc0 | not exactly | 21:23 |
| Lyncos | We are using BGP | 21:23 |
| klindgren | sdake, roughly correct. | 21:23 |
| inc0 | bgp knows doesn't know full route | 21:24 |
| Lyncos | should work with OSPF also I think | 21:24 |
| inc0 | only next hop | 21:24 |
| Mech422 | inc0: ahh - thanks :-) | 21:24 |
| inc0 | it can have multiple next hops and weights them | 21:24 |
| klindgren | I think we could take that work are fairly trvially extended it to work in the cellsv1 use case | 21:24 |
| sdake | klindgren is there a blocker to migrating to cellsv2? | 21:25 |
| Mech422 | Lyncos: so both calico and midonet use BGP, and midonet has some 'shared state' in zookeeper for loadbalancing, firewalling... | 21:25 |
| Lyncos | Calico is using ETCD | 21:25 |
| Lyncos | for that matter | 21:25 |
| klindgren | sdake, yea - it doesn't support multiple cells yet :-) | 21:25 |
| Mech422 | Lyncos: is there a practical difference between them? or only if you need to scale to Ks of nodes ? | 21:25 |
| Lyncos | load balancing is made with pure ECMP | 21:25 |
| sdake | cellsv 2doesn't work with more then one cell? | 21:25 |
| sdake | klindgren you ahve got to be kidding | 21:25 |
| klindgren | sdake, nope currently only support 1 and only 1 cell | 21:25 |
| Lyncos | Mech422: Calcio scale very will thru PODS | 21:26 |
| Lyncos | I think with our current design we can have up to 20 Racks per pods | 21:26 |
| Lyncos | something like that | 21:26 |
| Lyncos | when you want to scale you must add up a pod for every ~20 racks (depending on your design) | 21:26 |
| inc0 | Lyncos, pod == AS? | 21:27 |
| Lyncos | inc0 no | 21:27 |
| *** vhosakot has quit IRC | 21:27 | |
| Lyncos | AS are disabled on our setup I think | 21:27 |
| inc0 | ok | 21:27 |
| Lyncos | we are using a modified version of quagga (I think) | 21:27 |
| Mech422 | Lyncos: Cool... Does calico support both neutron and 'native' docker networking ? I'd sort of like to use the same thing everywhere rather then a bunch of different stacks | 21:27 |
| Lyncos | Mech422: You can use Calico with Kubernete | 21:28 |
| Lyncos | but not sure how it integrate when you are using both | 21:28 |
| sdake | klindgren if cellsv2 oly works with 1 cell | 21:28 |
| sdake | why is it in the code base? | 21:28 |
| sdake | klindgren if you know | 21:28 |
| inc0 | sdake, it's wip | 21:28 |
| inc0 | it obviously have to work with more | 21:28 |
| sdake | klindgren not complainign at you, just tyring to understand what the purpose is | 21:28 |
| mag009_ | inc0: redeploying with brand new images latest version from master repo | 21:29 |
| klindgren | because next release they are hoping to have multiple cells support | 21:29 |
| mag009_ | better work this time | 21:29 |
| inc0 | mag009_, good luck;) | 21:29 |
| Lyncos | We don't need luck we are using Kolla | 21:29 |
| Lyncos | should just work :_) | 21:29 |
| sdake | mag009_ are you running into trouble | 21:29 |
| inc0 | hahahahahah that's a good one Lyncos | 21:29 |
| inc0 | it's a software | 21:29 |
| inc0 | software is broken | 21:29 |
| klindgren | bascially it was this. Cellsv1 is "terrible" and not supported. But people need this functionality. So lets do cells v2. First lets fixup some of the internals to better support cells. | 21:29 |
| sdake | atually kolla is quite stable considering how many sheeer lines of code it works with | 21:29 |
| mag009_ | sdake: yes with ansible 2.1 | 21:29 |
| klindgren | THen lets make it so where everyone can have and is using cells by default. Then lets make it so where people can have mroe than one cell | 21:30 |
| Mech422 | Lyncos: Hmm - can you use calico with just a stand-alone docker? I'm sure I'm gonna have devs doing one-offs | 21:30 |
| sdake | klindgren thanks that seems logical | 21:30 |
| sdake | so in mitaka, cells are default to on | 21:30 |
| Lyncos | Mech422: if you use docker straight with L3 .. I guess you'll have to setup your 'host' machine to advertise your routes | 21:30 |
| mag009_ | sdake: fail at : neutron/tasks/deploy.yml the when statement | 21:31 |
| mag009_ | for my 3 ceph servers for some reason.. | 21:31 |
| sdake | mag009_ haven't sseen that | 21:31 |
| Mech422 | Lyncos: I mean can I install calico on a machine with just docker and no kube and have it play nice with the other boxes ? | 21:31 |
| sdake | mag009_ master is not really the best thing to evaluate with imo :) | 21:31 |
| kklimonda | which is "recommended" install type for deployment, source or binary? | 21:31 |
| Lyncos | Mech422: yes | 21:31 |
| Lyncos | Mech422: but you have no 'automation' | 21:32 |
| Lyncos | but it works | 21:32 |
| sdake | kklimonda doesn't matter imo | 21:32 |
| Lyncos | we are doing it for baremetal machines | 21:32 |
| inc0 | kklimonda, I use source, causes least headache for ubutuntu at leasy | 21:32 |
| sdake | kklimonda both work - unless you want ubuntu - then i'd definately go with source | 21:32 |
| inc0 | for centos both work stable in general | 21:32 |
| mag009_ | sdake: sure agreed I can try with tag version to see but I'm pretty sure it will do the same i'll give it a shot next | 21:32 |
| inc0 | writing is hard:/ | 21:32 |
| Mech422 | Lyncos: ahh - cool..I'll have to read up on it more then :-) I'd like something that can work in neutron for VMs, with OpenStack docker driver and 'stand-alone' docker machines, if possible :-) | 21:32 |
| sdake | ya centos + stable/liberty or stable/mitaka are good with binary or source | 21:32 |
| kklimonda | ok, thanks :) | 21:33 |
| sdake | mag009_ note stable/mitaka and stable/liberty are on ansible 1.9.4 | 21:33 |
| Lyncos | Mech422: we have a different topology than the standard calico one | 21:33 |
| mag009_ | there's a tag 2.0.0 | 21:33 |
| sdake | mag009_ there should be a 2.0.1 tag | 21:33 |
| mag009_ | yes that one | 21:33 |
| sdake | it was just released today | 21:33 |
| Lyncos | Mech422: I'm trying to find back the info... I'm not the one that worked the most on the network part | 21:34 |
| mag009_ | I want to keep up with you guys.. thats why I'm using the master | 21:34 |
| Mech422 | Lyncos: Heh - you've given me plenty to think about - I'm gonna have to look at calico more... | 21:34 |
| Lyncos | Really I think calico fits our needs.. but there are some limitations | 21:35 |
| Lyncos | like | 21:35 |
| mag009_ | the key is also the quagga version of cumulus | 21:35 |
| Lyncos | No: Multicast, no layer2 (means keepalive won't work) stuff like that | 21:35 |
| mag009_ | which allowed us to automate our switches and compute | 21:35 |
| Lyncos | when you want exception you can install openvswitch and do vxlan | 21:35 |
| Mech422 | Lyncos: hmm - I think our stuff uses multi-cast...but I might be mistaken | 21:35 |
| Mech422 | Lyncos: Our first use case is qa environments that can create private networks mirroring productin | 21:36 |
| Mech422 | Lyncos: so lack of layer-2 might be a problem? | 21:36 |
| Lyncos | Yes | 21:36 |
| Lyncos | you need to think about it | 21:37 |
| *** dwalsh has quit IRC | 21:37 | |
| Mech422 | Lyncos: I need to beat our NetEng/NetArch guys up to help me, but getting zero traction with them | 21:37 |
| Lyncos | Also our topology dosen't account for loosing a top of rack.. that means your apps must be resilient | 21:37 |
| Mech422 | I don't think they like the whole 'software defined network' thing :-P | 21:37 |
| Lyncos | Mech422: welcome to the real world | 21:37 |
| Lyncos | They are Cisco centric type of network guys ? | 21:38 |
| Mech422 | Lyncos: Arrista/Juniper | 21:38 |
| Lyncos | Same thing | 21:38 |
| Lyncos | :-) | 21:38 |
| Lyncos | We are running cumulus linux switches by the way | 21:38 |
| Mech422 | Lyncos: I think I need L2 - we do a lot of 'failover' and 'autoscaling' services.. | 21:38 |
| Mech422 | Lyncos: (We're a CDN) | 21:39 |
| Lyncos | Mech422: then you can check maybe with openvswitch | 21:39 |
| Lyncos | maybe you can add an overlay | 21:39 |
| Mech422 | Lyncos: yeah - we use openvswitch and vxlans currently | 21:39 |
| Lyncos | should still work in L3 | 21:39 |
| klindgren | sdake, I need to look to see what the rabbitmq setup is for cells v2. But I think we would only have 2 changes to support cells v1 vs's cells v2. | 21:39 |
| Lyncos | Failover and HA is a bit different in L3.... but you can stil use L2 stuff if you go with an overlays like ovs and vxlans | 21:40 |
| Mech422 | Lyncos: I was looking for something tht could play nice with all 4 configs (OS+VM,OS+Docker, standalone docker, standalone VM) | 21:40 |
| *** absubram has quit IRC | 21:40 | |
| Mech422 | Lyncos: since I'm not a network guy and didn't want to have to figure out 4 solutions :-P | 21:40 |
| Lyncos | I know calico are working on a baremetal version of calico | 21:40 |
| Lyncos | Mech422: if you are a CDN ... I guess you don't need very low latency ? | 21:41 |
| klindgren | 1.) being to start the nova-cells service and either configure it to use the cells.json or the database config (currently we use the database config which tells cellsv1 nodes how to talk to each other. You list the api cell and its rabbitmq servers, and the child cells and their rabbitmq servers. | 21:41 |
| Lyncos | * Since you are | 21:41 |
| Lyncos | my english not very good :-) | 21:41 |
| Mech422 | Lyncos: actually - latency is killer for us - but we do production on BSD boxes... | 21:41 |
| *** vhosakot has joined #openstack-kolla | 21:41 | |
| Lyncos | BSD Box | 21:41 |
| Lyncos | damn | 21:41 |
| Mech422 | Lyncos: we actually have 2-3 fulltime FBSD kernel commiters on staff to tweak our network stacks | 21:41 |
| Lyncos | VXLAN will add up latency | 21:41 |
| Lyncos | freebsd is not as bad as Openbsd | 21:42 |
| Mech422 | Lyncos: yeah - this is just for QA though | 21:42 |
| Lyncos | ok | 21:42 |
| Lyncos | I'm just telling | 21:42 |
| Mech422 | so the vxlan overhead is ok | 21:42 |
| Mech422 | Lyncos: thanks :-) | 21:42 |
| klindgren | 2.) possibly use/deploy another rabbitmq at the top level cell | 21:42 |
| Mech422 | Lyncos: but it does sound like if I don't have some sort of L2 support, _someone_ is gonna whine | 21:42 |
| Lyncos | Mech422: Probably | 21:43 |
| Lyncos | Mech422: we are ready to live with this | 21:43 |
| Lyncos | Mech422: for us it is an enabler to be in public clouds if needed | 21:43 |
| Lyncos | so we make sure our stack only support L3 which is what is used on the internet :-) | 21:43 |
| Mech422 | Lyncos: makes sense :-) | 21:44 |
| klindgren | sdake, if you care we did a presentation/blog on how to move from non-cells to cellsv1 configuration: http://www.dorm.org/blog/converting-to-openstack-nova-cells-without-destroying-the-world/ | 21:44 |
| Mech422 | blah - I probably need IPV6 too | 21:44 |
| klindgren | https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/moving-to-nova-cells-without-destroying-the-world | 21:44 |
| * Mech422 tries to figure out which NetEng he can blackmail into helping with this | 21:44 | |
| Lyncos | Mech422: find one that know linux/unix | 21:45 |
| Lyncos | :-) | 21:45 |
| Lyncos | Others won't see the benefit of it | 21:45 |
| Mech422 | Lyncos: yeah.. like I said, our guys basically don't trust it :-P | 21:46 |
| Mech422 | Lyncos: then again, I wouldn't trust me to do NetEng either :-P | 21:46 |
| Lyncos | Mech422: We eneded up installin linux in the switches and manage them ourself | 21:46 |
| Mech422 | which means, they should really help, just to keep me from screwing it up :-P | 21:46 |
| Lyncos | because we only have one net guy that know what is SDN | 21:47 |
| *** salv-orl_ has joined #openstack-kolla | 21:47 | |
| Mech422 | Lyncos: Hmm - isn't that how linux got on servers to begin with? someone snuck it in to run apache/etc :-D | 21:47 |
| Lyncos | ;) | 21:47 |
| Lyncos | the cool thin is that we configure our switches with Ansible | 21:47 |
| Lyncos | :-) | 21:48 |
| Lyncos | inc0 that one was for you :-) | 21:48 |
| inc0 | yeah, I know | 21:48 |
| Mech422 | Lyncos: oh nice - our arrista switches supposedly have APIs to allow stuff like that... | 21:48 |
| inc0 | man I really like what you did in your network;) will need to reproduce it in some lab | 21:48 |
| Lyncos | yeah but it's not the same think | 21:48 |
| Mech422 | Lyncos: but I'm not allowed to touch them - my technotard field makes them fail just by walking by | 21:48 |
| Lyncos | inc0 we can go to intel lab whenever you want | 21:49 |
| inc0 | maybe even publish stuff so it can be uised with trunk kolla | 21:49 |
| Lyncos | our lab resume to my own PC | 21:49 |
| *** JoseMello has quit IRC | 21:49 | |
| *** salv-orlando has quit IRC | 21:49 | |
| Lyncos | When we go live with our stuff we want to expose what we did to the world | 21:49 |
| inc0 | cool | 21:50 |
| Lyncos | because I think what we are doing is really 2016 | 21:50 |
| Lyncos | We are coming from a legacy world | 21:50 |
| Lyncos | Mech422: these are the kind of switches we are using: http://www.qct.io/-c77c75c159 | 21:51 |
| Lyncos | Just do the maths you'll never look behind | 21:52 |
| Lyncos | it is like 200% less expensive than cisco ( and I say 200% just because there are cisco dude here) | 21:52 |
| *** sdake_ has joined #openstack-kolla | 21:52 | |
| inc0 | I had quantas as storage servers | 21:53 |
| inc0 | I like quanta | 21:53 |
| Mech422 | Lyncos: oh nice :-) | 21:53 |
| Lyncos | we bought Quanta because baremetal switches are all using the same chipset and were less expensive than Dell or HP | 21:53 |
| Mech422 | Lyncos: I'll have to remember these next time lab budget rolls around... | 21:53 |
| Lyncos | price for a switch 40gbit spine is ~8k$ | 21:54 |
| Lyncos | 32 ports I think | 21:54 |
| *** sdake has quit IRC | 21:54 | |
| inc0 | :O | 21:54 |
| Lyncos | ahh.. this is in Canadian $ | 21:54 |
| mag009_ | at least you have a budget ! | 21:54 |
| mag009_ | -_- | 21:54 |
| inc0 | I'll buy one for my home! | 21:54 |
| Lyncos | inc0: yeah.. it shows that you have a better pay at Intel than us | 21:55 |
| inc0 | so I can put my raspberry pi on 40gig | 21:55 |
| Lyncos | :-) | 21:55 |
| inc0 | but 8k per switch is nothing | 21:56 |
| Lyncos | I agree | 21:56 |
| Lyncos | Canadian Dollars | 21:56 |
| Lyncos | :-) | 21:56 |
| inc0 | they're not that off from US | 21:56 |
| inc0 | anyway | 21:56 |
| inc0 | I'm going off guys, have a good one | 21:56 |
| Lyncos | CAN$ is very low right now | 21:56 |
| Lyncos | see you later | 21:57 |
| *** SiRiuS has quit IRC | 21:58 | |
| Mech422 | inc0: real quick - did you want that ceph patch against master? | 21:58 |
| *** inc0 has quit IRC | 22:01 | |
| *** Lyncos has left #openstack-kolla | 22:03 | |
| *** ayoung has joined #openstack-kolla | 22:04 | |
| mag009_ | alright time to go home | 22:06 |
| mag009_ | see you tomororw | 22:06 |
| *** shardy has quit IRC | 22:08 | |
| *** mbound_ has joined #openstack-kolla | 22:11 | |
| *** mbound has quit IRC | 22:14 | |
| *** daneyon_ has joined #openstack-kolla | 22:15 | |
| *** sdake has joined #openstack-kolla | 22:15 | |
| *** sdake_ has quit IRC | 22:18 | |
| *** daneyon_ has quit IRC | 22:20 | |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Stop using a global logger for all the things https://review.openstack.org/321884 | 22:20 |
| *** mbound_ has quit IRC | 22:20 | |
| *** mbound has joined #openstack-kolla | 22:21 | |
| *** zhiwei has joined #openstack-kolla | 22:24 | |
| *** salv-orlando has joined #openstack-kolla | 22:24 | |
| *** salv-orl_ has quit IRC | 22:25 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Adding documentation for labels https://review.openstack.org/324110 | 22:27 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Allow an operator to run an action on all services https://review.openstack.org/325679 | 22:28 |
| *** zhiwei has quit IRC | 22:28 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Add docs around bootstrapping and using the 'all' flag https://review.openstack.org/325681 | 22:28 |
| *** sdake has quit IRC | 22:36 | |
| *** absubram has joined #openstack-kolla | 22:41 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Fix Magnum trustee issues https://review.openstack.org/326163 | 22:42 |
| *** cu5 has quit IRC | 22:50 | |
| *** cu5_ has joined #openstack-kolla | 22:52 | |
| *** cu5_ has quit IRC | 22:53 | |
| *** diogogmt has quit IRC | 22:55 | |
| *** sdake has joined #openstack-kolla | 23:00 | |
| *** harlowja has quit IRC | 23:01 | |
| *** sdake_ has joined #openstack-kolla | 23:03 | |
| *** sdake has quit IRC | 23:05 | |
| *** cu5 has joined #openstack-kolla | 23:12 | |
| *** godleon has quit IRC | 23:19 | |
| *** salv-orlando has quit IRC | 23:23 | |
| *** salv-orlando has joined #openstack-kolla | 23:24 | |
| *** vhosakot has quit IRC | 23:32 | |
| *** vhosakot has joined #openstack-kolla | 23:42 | |
| *** diogogmt has joined #openstack-kolla | 23:44 | |
| *** vhosakot has quit IRC | 23:46 | |
| *** matrohon has quit IRC | 23:50 | |
| *** fragatin_ has joined #openstack-kolla | 23:58 | |
| *** sacharya has quit IRC | 23:58 | |
| *** ssurana has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!