| kfox1111 | sbezverk_: sed 's/.*|//' /tmp/after.html > /tmp/after2.html | 00:00 |
|---|---|---|
| kfox1111 | gets rid of most of the timestamps. | 00:00 |
| sbezverk_ | thanks | 00:00 |
| kfox1111 | np. :) | 00:00 |
| kfox1111 | if you dont have a copy, I highly recommend this book: http://shop.oreilly.com/product/9780596528126.do | 00:01 |
| kfox1111 | one of the best ban per buck books I've ever bought. :) | 00:01 |
| kfox1111 | bang | 00:01 |
| sbezverk_ | kfox1111: will do. thanks for the link | 00:03 |
| kfox1111 | k8s containers are all the same image. | 00:03 |
| kfox1111 | calico versions changed a lot. | 00:06 |
| sbezverk_ | wow diff -y is really cool | 00:06 |
| kfox1111 | calico/node v1.0.1 vs 1.1.0 | 00:07 |
| * kfox1111 nods | 00:07 | |
| kfox1111 | diff is a very cool tool. | 00:08 |
| sbezverk_ | kfox1111: have you seen that bad log is missing network configuration data!!! | 00:09 |
| sbezverk_ | http://logs.openstack.org/43/449843/1/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-iscsi-nv/279f89e/console.html#_2017-03-24_23_43_53_640092 | 00:09 |
| sbezverk_ | http://logs.openstack.org/54/449254/3/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-iscsi-nv/dd31b46/console.html#_2017-03-23_21_54_12_063764 | 00:10 |
| *** Pavo has quit IRC | 00:10 | |
| kfox1111 | hmm... not sure what that is. infra's code. | 00:10 |
| *** tonanhngo has joined #openstack-kolla | 00:10 | |
| kfox1111 | I'm betting this is a new version of calico not working with the openstack infra iptables. | 00:10 |
| kfox1111 | or something like that. | 00:10 |
| sbezverk_ | could be | 00:11 |
| kfox1111 | gonna see if we can pin it back real quick. | 00:11 |
| jemcevoy | inc0: Thanks for the override hint... for the I have kolla-build running... I had to find a bug in base_yum_repo_keys in that same file: The GPG key should be https://www.percona.com/downloads/RPM-GPG-KEY-percona not a local file... https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L63 | 00:11 |
| kfox1111 | this looks bad | 00:13 |
| kfox1111 | https://github.com/projectcalico/canal/commit/b84aad6492906a247e730cbdb234a635d5584983 | 00:13 |
| kfox1111 | lets try: https://raw.githubusercontent.com/projectcalico/canal/7deb07cda04147ab49115f437151705c747d0374/k8s-install/kubeadm/canal.yaml | 00:14 |
| *** lucasxu has quit IRC | 00:14 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test gate https://review.openstack.org/449843 | 00:15 |
| sbezverk_ | kfox1111: yeah this version looks like memt for 1.6, since they decided to switch to etcd 3 | 00:17 |
| sbezverk_ | meant for 1.6 | 00:17 |
| kfox1111 | looks like they merged 1.6 back to 1.5 by mistake. | 00:17 |
| kfox1111 | if this passes, its likely. | 00:17 |
| kfox1111 | and we can raise an issue. | 00:18 |
| sbezverk_ | let's hope | 00:18 |
| kfox1111 | telnet://158.69.81.40:19885 | 00:18 |
| sbezverk_ | that was it :) | 00:23 |
| *** caoyuan has joined #openstack-kolla | 00:23 | |
| kfox1111 | yup. | 00:24 |
| sbezverk_ | great catch Kevin!!! | 00:24 |
| *** tonanhngo has quit IRC | 00:24 | |
| *** unicell has quit IRC | 00:25 | |
| kfox1111 | thanks. | 00:26 |
| kfox1111 | it was your idea though to compare gate runs. :) | 00:26 |
| sbezverk_ | hopefully sdake is still aroubd so we could fix the gate | 00:27 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Pin Back Canal https://review.openstack.org/449843 | 00:30 |
| kfox1111 | inc0: ping | 00:30 |
| *** unicell has joined #openstack-kolla | 00:30 | |
| sdake | sbezverk_ what do yo uneed | 00:35 |
| *** eaguilar has quit IRC | 00:37 | |
| kfox1111 | sdake: https://review.openstack.org/#/c/449843/ | 00:38 |
| *** caoyuan has quit IRC | 00:38 | |
| sdake | canal is deployed via calico right? | 00:38 |
| kfox1111 | calico is part of canal | 00:39 |
| kfox1111 | canal = calico + flannel | 00:39 |
| sbezverk_ | sdake: we would need to merge kfox1111 fix asap to restore gate | 00:40 |
| sbezverk_ | it has been tested and confimred as a good solution.. | 00:41 |
| sdake | waiting on gate | 00:41 |
| sdake | i was pretty sure osme part of canal or calico was to blame as I saw that earlier today | 00:41 |
| sdake | as using weave doesn't exhibit the problem of kube-dns not starting | 00:42 |
| sdake | kfox1111 nice job on finding a version that works :) | 00:42 |
| kfox1111 | yeah. I think they merged back a 1.6 canal to 1.5's yaml by mistake. | 00:42 |
| kfox1111 | sdake: sbezverk_ had the idea of comparing gate logs. | 00:42 |
| kfox1111 | it poped up pretty quickly from there. :) | 00:42 |
| sdake | yup wish i could have helped more - still learning my way around the system | 00:43 |
| sdake | anyway the patch is +2'ed waiting on gate for a workflow | 00:43 |
| kfox1111 | k. thx. | 00:43 |
| *** haplo37 has quit IRC | 00:44 | |
| sdake | what is the downside of using weave? | 00:46 |
| *** srwilkers has joined #openstack-kolla | 00:47 | |
| * kfox1111 shrugs | 00:47 | |
| kfox1111 | we've just tested more with canal. | 00:47 |
| kfox1111 | shouldn't matter in theory. | 00:48 |
| kfox1111 | though we have had more reports of brokenness with weave so far. | 00:48 |
| kfox1111 | its like rabbit vs qpid. both work fine. one's more tested. | 00:48 |
| sbezverk_ | sdake: as far as I heard 99% of kube community do not use it | 00:49 |
| sdake | i see | 00:50 |
| sdake | and the other 1% do not use calico? | 00:50 |
| *** caoyuan has joined #openstack-kolla | 00:50 | |
| openstackgerrit | Merged openstack/kolla master: Merged multiple ENV into a single layer https://review.openstack.org/449371 | 00:51 |
| *** haplo37 has joined #openstack-kolla | 00:53 | |
| kfox1111 | I have no staticstics. I just started using it as I had previously used flannel. | 00:53 |
| kfox1111 | and I like flannels simplicity. | 00:54 |
| srwilkers | flannels keep you nice and warm, and they're simple | 00:58 |
| srwilkers | obligatory /s | 00:58 |
| kfox1111 | :) | 00:58 |
| sdake | hey srwilkers | 00:59 |
| sdake | hey sbezverk_ - http://sprunge.us/GOQP | 00:59 |
| sdake | what do you make of that | 00:59 |
| sdake | seems jammed up in some way? | 00:59 |
| *** unicell has quit IRC | 00:59 | |
| srwilkers | hey sdake | 01:02 |
| srwilkers | hows it going sdake kfox1111 sbezverk_? | 01:02 |
| srwilkers | having a good friday night? | 01:03 |
| sdake | srwilkers waiting on dinner then unplugging | 01:03 |
| srwilkers | sdake, right on | 01:03 |
| sdake | srwilkers almost have docs written however: https://review.openstack.org/#/c/447356/ | 01:03 |
| sdake | had meeting today and about 10 people showed up from the community while we debugged the install | 01:04 |
| kfox1111 | srwilkers: pretty good. you? | 01:04 |
| sdake | everyone seems keen to understand *how* kolla-kubernetes installed | 01:04 |
| sdake | I must have heard "we dont want a black box" atleast 5+ times | 01:04 |
| sdake | or variations on that | 01:04 |
| kfox1111 | as an op, fully relate. :) | 01:04 |
| kfox1111 | black boxes are things that bite you eventually. | 01:05 |
| sdake | these are devs | 01:05 |
| sdake | i guess - not totall ysure | 01:05 |
| sdake | although I have got to the ponit of having a clean cloud.yaml | 01:07 |
| sdake | (which doesn't work ;) | 01:07 |
| sdake | nto sure if it is my CNI plugin or what | 01:07 |
| kfox1111 | try the canal from the review. | 01:07 |
| kfox1111 | much more tested. :) | 01:07 |
| sdake | i was just getting ready to blow away my deploy to do that | 01:07 |
| sbezverk_ | kfox1111: have you seen on the gate fix job, some workflows failed with really weird issue, which looks more like a connectivity issue on the host where the job was running | 01:09 |
| kfox1111 | sbezverk_: seen it very rarely. usually a hub isue. | 01:10 |
| kfox1111 | still need to get that container cache ps through. | 01:10 |
| sbezverk_ | yep | 01:11 |
| *** srwilkers has quit IRC | 01:17 | |
| sdake | docker 17.03 not supported with kubeadm | 01:17 |
| * sdake groans | 01:17 | |
| kfox1111 | heh. | 01:18 |
| kfox1111 | distro docker. | 01:18 |
| sdake | [kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters. | 01:20 |
| kfox1111 | yup. | 01:22 |
| kfox1111 | thats why i hand roll my prod systems. | 01:22 |
| kfox1111 | well. one of the reasons. | 01:23 |
| *** vhosakot has joined #openstack-kolla | 01:24 | |
| sdake | yup kubeadm definately doesn't work with latest docker | 01:26 |
| sdake | after kubeadm deploys you can upgrade to latest docker if you like | 01:26 |
| kfox1111 | would be surprised if kubernetes did too. | 01:26 |
| sdake | kubernetes indeed seemed to | 01:26 |
| kfox1111 | hmm.. k. | 01:26 |
| sdake | although maybe that is why my system wedged - and it wasn't weave | 01:26 |
| sdake | getting ready to test that now | 01:27 |
| *** kolla-slack has joined #openstack-kolla | 01:29 | |
| *** srwilkers has joined #openstack-kolla | 01:41 | |
| *** Pavo has joined #openstack-kolla | 01:43 | |
| *** jtriley has joined #openstack-kolla | 01:44 | |
| *** Pavo has quit IRC | 01:48 | |
| vhosakot | haha, still chatting k8s on Friday evening... moving my k8s cluster now, will start new kubeadm run now | 01:50 |
| sdake | now weave wedges with distro docker | 01:51 |
| sdake | although weave did not wedge with docker-engine 17.03 | 01:51 |
| sdake | the proper way to tidy up: [sdake@kolla ~]$ docker stop $@ `docker ps -a -q` | 01:57 |
| vhosakot | sdake: weave in halcyon was great for me... but, weave in kubeadm made many pods go to CrashloopBackoff... canal+kubeadm is good for me | 02:01 |
| *** tovin07 has joined #openstack-kolla | 02:02 | |
| sdake | vhosakot this works well: | 02:02 |
| sdake | 1255 url="https://raw.githubusercontent.com/projectcalico/canal/7deb07cda04147ab49115f437151705c747d0374" | 02:02 |
| sdake | 1256 url="$url/k8s-install/kubeadm/canal.yaml" | 02:02 |
| sdake | 1257 wget $url | 02:02 |
| sdake | 1258 kubectl apply -f canal.yaml | 02:02 |
| vhosakot | ah cool | 02:03 |
| openstackgerrit | jimmygc proposed openstack/kolla master: Add vSphere Support https://review.openstack.org/449395 | 02:03 |
| sdake | kfox1111 what is with the sed in that script there | 02:04 |
| kfox1111 | which sed? | 02:05 |
| sdake | my pods dont run | 02:05 |
| sdake | the one in https://review.openstack.org/449843 | 02:05 |
| sdake | my pods wer erunning with weave (busybo for eample) | 02:06 |
| kfox1111 | sed -i "s@192.168.0.0/16@172.16.130.0/22@" /tmp/canal.yaml | 02:06 |
| kfox1111 | sed -i "s@10.96.232.136@172.16.128.100@" /tmp/canal.yaml | 02:06 |
| kfox1111 | ? | 02:06 |
| sdake | yup | 02:06 |
| sdake | could you explain what that is doing at a high level | 02:07 |
| kfox1111 | so, k8s sdn has 2 sets of ip's. | 02:07 |
| kfox1111 | a clusterip range. k8s services get pulled from this range. | 02:08 |
| kfox1111 | and a container network. | 02:08 |
| kfox1111 | each physical machine carves off a chunk of the container network to hand out to its containers. | 02:08 |
| kfox1111 | canal.yaml has some hard coded ip ranges in it. | 02:08 |
| kfox1111 | those are used by the openstack providers in the gate. | 02:08 |
| kfox1111 | so breaks things. | 02:09 |
| kfox1111 | in a few different places, we replace the ip's with ones valid in the gate. | 02:09 |
| sdake | what is 172.16.130.0? | 02:09 |
| kfox1111 | setup_kubernetes.sh and setup_canal.sh has things. | 02:09 |
| kfox1111 | the container network space. | 02:10 |
| sdake | things like docs ;) | 02:10 |
| sdake | the default in canal.yaml is 192..168.0.0/16? | 02:10 |
| kfox1111 | its something k8s should doc. | 02:10 |
| kfox1111 | yeah. | 02:10 |
| sdake | christ thats my home network | 02:10 |
| kfox1111 | which is bad for most organizations. | 02:10 |
| kfox1111 | yeah. | 02:10 |
| sdake | what should it be? | 02:10 |
| kfox1111 | anythign that doesn't conflict with your org. :) | 02:11 |
| sdake | can it be a routed address? | 02:11 |
| kfox1111 | sdn's always the hardest part. | 02:11 |
| kfox1111 | shoudln't be. | 02:11 |
| kfox1111 | rfc1918 | 02:11 |
| *** fooliouno has quit IRC | 02:11 | |
| sdake | my home network is 192.168.7.0/8 | 02:11 |
| kfox1111 | just space from it your org doesn't use. | 02:11 |
| sdake | so I should sed that crap to 192.168.8.0/8? | 02:11 |
| kfox1111 | 8? you mean 24? | 02:12 |
| sdake | ya /24 | 02:12 |
| sdake | LONG DAY | 02:12 |
| kfox1111 | no worries. | 02:12 |
| sdake | does that have to match to anything setup in kubeadm --init? | 02:12 |
| sdake | 1215 sudo kubeadm init --pod-network-cidr=10.244.0.0/16 | 02:13 |
| sdake | this is what i'm running | 02:13 |
| kfox1111 | how many nodes do you need? | 02:13 |
| sdake | 1 at present | 02:13 |
| sdake | writing AIO docs to begin | 02:13 |
| kfox1111 | if you don't have any 10 space stuff, I'd just use that. | 02:13 |
| sdake | so that people can give kolla-kubernetes a spin | 02:13 |
| sdake | so 10.0.3.0/24? | 02:14 |
| kfox1111 | each node will try and pull a /24 | 02:14 |
| kfox1111 | by default. | 02:14 |
| sdake | groan | 02:14 |
| kfox1111 | notsure what will happen if you only have a /24 | 02:14 |
| sdake | does my pod network cidr need to match anything in canal setup? | 02:14 |
| kfox1111 | yeah. | 02:14 |
| kfox1111 | compare setup_kubernetes and setup_canal | 02:15 |
| kfox1111 | you should see things that match. | 02:15 |
| sdake | ok | 02:15 |
| vhosakot | sdake: are you writing AIO docs to to deploy kolla-k8s? using what? vagrant+halcyon or kubeadm? | 02:18 |
| sdake | kubeadm on baremetal | 02:18 |
| vhosakot | sdake: kubeadm on a AIO node, followed by "helm init", and the install kolla-k8s charts to install OpenStack on k8s in that AIO VM? | 02:19 |
| sdake | vhosakot https://review.openstack.org/#/c/447356/ | 02:19 |
| vhosakot | ah that link, I'm reviewing it.. cool | 02:19 |
| vhosakot | sdake: why just kargo, what if users have their base cluster using kops, bootkube, or scripts? I'll add this comment? | 02:22 |
| sdake | vhosakot - huh | 02:22 |
| sdake | vhosakot i am documetning a workflow iwth kubeadm | 02:23 |
| sdake | if someone else wants to document a deployment workflow with kargo more power to them | 02:23 |
| vhosakot | sdake: ah right right. got it | 02:23 |
| vhosakot | sdake: does kolla-k8s mandate the base (empty) cluster must be deployed using kubeadm? | 02:23 |
| sdake | nope | 02:23 |
| vhosakot | sdake: cool, that is good :) | 02:24 |
| *** shasha_t_ has joined #openstack-kolla | 02:24 | |
| sdake | although devs are struggling to bring up kolla | 02:24 |
| sdake | and operators as well | 02:24 |
| vhosakot | true | 02:24 |
| sdake | might as well help them | 02:24 |
| vhosakot | cool | 02:24 |
| sdake | relying on documentation in 20 places is not ideal | 02:24 |
| sdake | "go look at project x go look at project y" | 02:25 |
| sdake | who does that? | 02:25 |
| sdake | nobody, thats who | 02:25 |
| *** fooliouno has joined #openstack-kolla | 02:25 | |
| sdake | sup fooliouno | 02:25 |
| fooliouno | sdake: did you all have the second webex session this afternoon | 02:25 |
| sdake | indeed we did | 02:25 |
| sdake | we got further | 02:25 |
| fooliouno | shoot .. I missed it | 02:25 |
| fooliouno | where did you all stop | 02:25 |
| fooliouno | what step | 02:26 |
| sdake | i have a cloud yaml I think works | 02:26 |
| sdake | I wiped out my system to start fresh | 02:26 |
| sdake | that is where we stopped - networking wasbusted | 02:26 |
| sdake | although I am about to get past that problem I hope | 02:26 |
| fooliouno | great. We started with helm init. got an error and stopped for the day. | 02:27 |
| vhosakot | fooliouno: I have stopped for the day after "helm init" breaking many days... lol... re-deploying new cluster now :) | 02:28 |
| fooliouno | vhosakot: I'm glad you all are clearing the way for us :) | 02:29 |
| vhosakot | fooliouno: bahahahahaha... I'm clearing backwards :) | 02:30 |
| *** eanylin has quit IRC | 02:30 | |
| vhosakot | kubeadm is such a great tool.. just needs some fixes... at a minimum, if "helm init", service account, and dashboard work OTB, I think kubeadm will be adopted a lot... | 02:31 |
| vhosakot | plus, supporting multiple masters (like kargo) | 02:31 |
| vhosakot | adding new workers in kubeadm is so much easier than in kargo. | 02:31 |
| kfox1111 | its coming. | 02:31 |
| vhosakot | kfox1111: are you part of the kubeadm sig ? | 02:32 |
| kfox1111 | I've kept an eye on it. | 02:33 |
| vhosakot | cool, | 02:33 |
| kfox1111 | they are making it more modular. | 02:33 |
| kfox1111 | so can be reused in more deployment tools. | 02:34 |
| *** dave-mccowan has joined #openstack-kolla | 02:34 | |
| vhosakot | I really saw hands-on how just two commands "kubeadm init" and "kubeadm join" replace like 5-10 Ansible playbooks. | 02:34 |
| kfox1111 | yeah. its a really nice piece of code. | 02:35 |
| *** pramodrj07 has quit IRC | 02:35 | |
| kfox1111 | when it becomes more mdular, | 02:35 |
| kfox1111 | I want to try it tied in with helm. | 02:35 |
| kfox1111 | use it to do all the ca stuff, | 02:35 |
| kfox1111 | and deploy kube-* with helm. | 02:35 |
| *** schwicht has quit IRC | 02:36 | |
| fooliouno | kfox1111: which time zone are you in | 02:38 |
| fooliouno | ok .. folks. my weekly poker game is not happening today. So I'll be off doing my taxes. Good night. | 02:43 |
| *** fooliouno has quit IRC | 02:44 | |
| *** caoyuan has quit IRC | 02:46 | |
| *** manheim has joined #openstack-kolla | 02:50 | |
| sdake | kfox1111 need some help | 02:53 |
| kfox1111 | whats up? | 02:54 |
| sdake | [sdake@kolla ~]$ sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=192.168.1.0/30 | 02:54 |
| sdake | locks up | 02:54 |
| kfox1111 | fooliouno: pdt | 02:54 |
| kfox1111 | 30's awefully small. | 02:54 |
| sdake | 192.168.1.0AIO | 02:54 |
| sdake | AIO | 02:54 |
| sdake | whoudln't that do? | 02:54 |
| kfox1111 | its one per service, not one per node. | 02:55 |
| sdake | ok my home network is 192.168.7.0/24 | 02:55 |
| sdake | --service-cidr should beo nthat same network right? | 02:55 |
| kfox1111 | glance-api, nova-api, neutron-server, heat-api, etc. all take up 1 service ip. | 02:55 |
| kfox1111 | no. differnet network. | 02:55 |
| kfox1111 | non overlapping. | 02:56 |
| kfox1111 | its internal to k8s. | 02:56 |
| sdake | i see | 02:56 |
| sdake | so 10.1.0.0/16 will work for sservice-cidr? | 02:56 |
| kfox1111 | yeah. | 02:56 |
| kfox1111 | or even 10.1.0.0/24 | 02:56 |
| sdake | that won't overwrite the pod network cidr? | 02:57 |
| kfox1111 | non overlapping with the pod network cidr too. | 02:57 |
| sdake | ok | 02:57 |
| sdake | [sdake@kolla ~]$ sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/16 | 02:57 |
| sdake | 10.2.0.0/16 is used in my network iirc | 02:57 |
| kfox1111 | should be ok I think. | 02:57 |
| kfox1111 | though my brains pretty fried. :/ | 02:58 |
| kfox1111 | I think those are non overlapping though. | 02:58 |
| sdake | ok - i'll assume non-overlapping | 02:58 |
| sdake | ok that gets me to this point: kube-dns-2924299975-kbtz7 0/4 ContainerCreating 0 14s | 03:00 |
| sdake | which i suspect needs some customization on canal.yml | 03:00 |
| sdake | etcd_endpoints: "http://192.168.7.101:6666" | 03:00 |
| sdake | this should be what? | 03:00 |
| sdake | note i did look at setup_kubernetes.sh and setu_canal.sh however there is a lack of comments :) | 03:01 |
| sdake | looking at seds randomly isn't all that helpful | 03:01 |
| vhosakot | sdake: doesn't kubeadm's default service cidr block (10.96.0.0/12).. yes, it is private and sees no internet. | 03:02 |
| sdake | vhosakot i want to understnad what the kubeadm optoins do | 03:03 |
| sdake | and what canal.yaml requirees | 03:03 |
| vhosakot | sdake: ah ok ok | 03:03 |
| sdake | the docs for canal are not that helpful | 03:03 |
| sdake | in fact rwellum said not to read them today :) | 03:03 |
| sdake | kfox1111 ^^ | 03:04 |
| sdake | kfox1111 i understand this is canals' etcd endpoint | 03:04 |
| sdake | canal runs its own etcd | 03:04 |
| sdake | what i dont understand is what ip to give it | 03:05 |
| vhosakot | hahaha, yeah, many of those docs are not readable... | 03:05 |
| sdake | including kollas | 03:05 |
| sdake | a readme is not helpful at all though | 03:06 |
| vhosakot | agree | 03:06 |
| sdake | vhosakot do you knwo what this value should be? | 03:06 |
| vhosakot | the QSG works OTB in kolla | 03:06 |
| sdake | you mean kolla-ansible? | 03:06 |
| kfox1111 | sdake: it needs an ip from the service cidr block. | 03:06 |
| sdake | kfox1111 any ip? | 03:06 |
| kfox1111 | yeah. | 03:06 |
| sdake | sweet | 03:06 |
| vhosakot | sdake: yes, kolla-ansible's QSG | 03:06 |
| sdake | vhosakot kolla-ansible is mature | 03:07 |
| vhosakot | ^^^ true | 03:07 |
| sdake | clusterIP: 10.3.1.100 | 03:07 |
| vhosakot | sdake: value for what? kubeadm's --external-etcd-endpoints ? | 03:07 |
| sdake | kfox1111 this one is the same as etcd_endpoints? | 03:08 |
| kfox1111 | I thihnk so. | 03:08 |
| *** caoyuan has joined #openstack-kolla | 03:08 | |
| kfox1111 | I wrote that code a long time ago now though. | 03:08 |
| sdake | kfox1111 cool - any advice you can provide is helpful not going to roast you for not knowing how it works :) | 03:09 |
| vhosakot | sdake: I think it is needed to point kubeadm to a non-default etcd endpoint if the user does not want the etcd cluster that kubeadm install OTB. | 03:09 |
| vhosakot | I'm installing now | 03:09 |
| vhosakot | sdake: I did not pass any arg to "kubeadm init" at all.. the default args worked beautifully for me | 03:09 |
| kfox1111 | canal has its own etcd. | 03:10 |
| kfox1111 | vhosakot: depends on if your home network/corperate network has overlapping ip's with the random one it contains in its yaml file. | 03:10 |
| sdake | vhosakot can you ru nthis command then: kubectl run -i -t $(uuidgen) --image=busybox --restart=Never | 03:10 |
| sdake | kfox1111 yup i know canal has its own etcd | 03:10 |
| vhosakot | sdake: yes, that command worked for me.. now no cluster here.. tore down.. I ran that command successfully and then also ran "nslookup kubernetes" to validate DNS in busybox pod. | 03:11 |
| sdake | kfox1111 what about this: # Writes basic flannel configuration to etcd. | 03:12 |
| sdake | - name: configure-flannel | 03:12 |
| sdake | image: quay.io/coreos/etcd:v3.0.9 | 03:12 |
| sdake | command: | 03:12 |
| sdake | - "etcdctl" | 03:12 |
| sdake | - "set" | 03:12 |
| sdake | - "/coreos.com/network/config" | 03:12 |
| sdake | - '{ "Network": "10.0.4.0/16", "Backend": {"Type": "vxlan"} }' | 03:12 |
| kfox1111 | thats the pod network. | 03:12 |
| sdake | specifically the last line | 03:12 |
| sdake | ok thanks | 03:12 |
| sdake | [sdake@kolla ~]$ kubectl apply -f canal2.yaml | 03:13 |
| sdake | here goes nothing | 03:13 |
| sdake | ok everyting in running state now | 03:15 |
| sdake | (kubernetes and canal specifically) | 03:15 |
| sdake | kfox1111 today on the webex we had, eveyrone was in general agreement it took them 1+ week to get kubernetes deployed | 03:16 |
| kfox1111 | for dev or test? | 03:16 |
| kfox1111 | or prod | 03:16 |
| sdake | for anything | 03:16 |
| sdake | to eval | 03:16 |
| kfox1111 | minikube. | 03:16 |
| kfox1111 | its like 10 min. | 03:17 |
| sdake | i get that | 03:17 |
| sdake | people want multinode | 03:17 |
| sdake | minikube do multinode? | 03:17 |
| vhosakot | minikube fakes using localkube :) | 03:17 |
| kfox1111 | people don't need minikube to kick the tires. | 03:17 |
| vhosakot | worse than devstack AIO IMO | 03:17 |
| kfox1111 | to many people wnat to run before walking. ;) | 03:17 |
| sdake | i'd rather crawl first | 03:17 |
| kfox1111 | yeah. and htats minikube. :) | 03:17 |
| vhosakot | yes, minikube is great to learn k8s concepts | 03:18 |
| kfox1111 | once you really understand minkube, then kubeadm next. | 03:18 |
| kfox1111 | then after that, then I did manual k8s deployment, to fully understand it. | 03:18 |
| sdake | [sdake@kolla ~]$ kubectl taint nodes --all dedicated- | 03:18 |
| vhosakot | kfox1111: how manually.. install service-after-service like k8s api-server, controller-manager, etcd, scheduler, etc | 03:19 |
| vhosakot | ? | 03:19 |
| kfox1111 | yeah. using kubelet and static pods to deploy those things. | 03:19 |
| kfox1111 | the tricky bit is setting up a ca. | 03:19 |
| kfox1111 | that and picking a right set of subnets for sdn. | 03:19 |
| kfox1111 | the rest isn't too bad. | 03:20 |
| sdake | most people give a project 30mins to 1 hour of eval time befor emoving on | 03:21 |
| kfox1111 | yeah. so thats why I say, use minikube. | 03:21 |
| kfox1111 | anything more is way too much for what they need to kick the tires with. | 03:21 |
| portdirect | minikube is a bit iffy these days - they have removed ipset and a few other things from the kernel a few times :/ | 03:23 |
| srwilkers | a wild portdirect appears | 03:23 |
| portdirect | so anything that uses thes system calls gets fubared :( | 03:23 |
| kfox1111 | easiest way is to write/contribute regression tests and upstream them. if you care about something, make sure it gets tested. :) | 03:24 |
| kfox1111 | then its not accidental breakage at least. | 03:24 |
| *** eanylin has joined #openstack-kolla | 03:25 | |
| portdirect | yeah - i filed a few bugs, but it keeps on going :) | 03:25 |
| *** inc0 has quit IRC | 03:25 | |
| kfox1111 | yeah. if its a voting test, they have to try really hard to break it. | 03:25 |
| portdirect | this is what I'm using atm - which may be some help: https://github.com/portdirect/dockerfiles/tree/kubeadm/aio-deploy/kubeadm-aio | 03:26 |
| sdake | this is what kube-dns does to me kfox1111 kube-dns-2924299975-kbtz7 3/4 Running 15 27m | 03:27 |
| kfox1111 | portdirect: cool. thanks. I'll have a look. | 03:27 |
| srwilkers | 75% is still a passing grade most places | 03:27 |
| kfox1111 | sdake: that was what we were seeing in gate before the fix. | 03:27 |
| sdake | kfox1111 pretty sure i downloaded the fix | 03:28 |
| *** manheim has quit IRC | 03:28 | |
| sdake | kfox1111 however - i can give it another go | 03:28 |
| kfox1111 | sdake: can happen for other reasons too. | 03:28 |
| kfox1111 | issues with ip's, | 03:28 |
| kfox1111 | iptables in the way, etc. | 03:28 |
| srwilkers | portdirect, hows it going | 03:28 |
| sdake | tiller-deploy-3210613906-bksx2 0/1 CrashLoopBackOff 4 2m | 03:29 |
| sdake | however helm version returns the proper version | 03:29 |
| portdirect | srwilkers: better than that | 03:29 |
| kfox1111 | sdake: that seems like a sdn issue. | 03:29 |
| kfox1111 | tiller the way helm init deploys it uses sdn. | 03:30 |
| kfox1111 | kube-dns also does. | 03:30 |
| kfox1111 | so, symptoms of networkign not working. | 03:30 |
| *** dave-mccowan has quit IRC | 03:30 | |
| sdake | i love when people explode m80's right next door | 03:31 |
| sdake | makes me so happy | 03:31 |
| sdake | kfox1111 you got any bone to throw me beyond that | 03:32 |
| sdake | kfox1111 the sooner i get this documented the less often people ask questions :) | 03:33 |
| kfox1111 | what distro? | 03:33 |
| sdake | centos 7 | 03:33 |
| sdake | kfox1111 webex screen share? | 03:33 |
| kfox1111 | sdake: I so wish that were true.... there always are more questions. ;) | 03:33 |
| sdake | kfox1111 agreed- although people are struggling | 03:33 |
| kfox1111 | do you have firewalld disabled? | 03:33 |
| sdake | yes | 03:33 |
| sdake | and selinux as well | 03:33 |
| kfox1111 | iptables-service not installed? | 03:33 |
| sdake | which is my #1 and #2 questions to people | 03:34 |
| sdake | yes iptables is disables | 03:34 |
| sdake | disabled | 03:34 |
| kfox1111 | what version of kubeadm? | 03:34 |
| sdake | [sdake@kolla kolla-bringup]$ kubeadm version | 03:35 |
| sdake | kubeadm version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2074+a092d8e0f95f52", GitCommit:"a092d8e0f95f5200f7ae2cba45c75ab42da36537", GitTreeState:"clean", BuildDate:"2016-12-13T17:03:18Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} | 03:35 |
| kfox1111 | hmm... lets see. | 03:35 |
| sdake | 1366 sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/16 | 03:36 |
| sdake | 1377 kubeadm aply -f canal2.yaml | 03:36 |
| kfox1111 | anyhthing you missed from setup_kubernetes? | 03:36 |
| sdake | 1382 kubectl taint nodes --all dedicated- | 03:36 |
| sdake | 1383 ALLINONENODE=$(hostname) | 03:36 |
| sdake | 1384 kubectl label node $ALLINONENODE kolla_compute=true | 03:36 |
| sdake | 1385 kubectl label node $ALLINONENODE kolla_controller=true | 03:36 |
| sdake | 1386 sudo kolla-ansible genconfig | 03:36 |
| kfox1111 | sed -i 's/10.96.0.10/172.16.128.10/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | 03:36 |
| kfox1111 | perhaps? | 03:36 |
| sdake | there clearly has to be a better way to specify the dns server then hacking kubeadm.conf in systemd | 03:37 |
| portdirect | sdake: is that output copied from your history? looks like 1377 may be a typo? | 03:38 |
| kfox1111 | I put in a feature request for that. not really my problem to solve. | 03:38 |
| sdake | portdirect right - that is the wrong line - it should be apply (later in my history I use apply) | 03:38 |
| sdake | kfox1111 ok - so what should it be set to in kubeadm.conf? | 03:38 |
| kfox1111 | sdake: preferably, someone would take setup_kubernetes and make it generic enough for your document for those just wanting to crank out something quickly. | 03:39 |
| kfox1111 | the service ip of the dns server. | 03:39 |
| sdake | kfox1111 how is that known ahead of time? | 03:39 |
| portdirect | sdake: 'kubectl create -f canal2.yaml' ? | 03:39 |
| kfox1111 | take your service ip space and use the .10 address out of it. | 03:39 |
| kfox1111 | kubeadm defaults to usiong .10 for kube-dns. | 03:40 |
| sdake | kfox1111 thats helpful - thanks :) | 03:40 |
| sdake | portdirect i'll try that next go | 03:40 |
| kfox1111 | it basically uses the start of the subnet + 10. | 03:41 |
| kfox1111 | and kube-api gets .1 | 03:41 |
| sdake | kfox1111 intuitive ;) | 03:41 |
| kfox1111 | no idea why. | 03:41 |
| kfox1111 | probably leaving room for mutlple controllers or something. | 03:41 |
| spsurya | sup guys | 03:46 |
| sdake | kube-dns-2924299975-cxw89 3/4 Running 0 3m | 03:47 |
| sdake | still 3 | 03:47 |
| portdirect | are there logs from the container thats not running? | 03:47 |
| sdake | [sdake@kolla ~]$ kubectl run -i -t $(uuidgen) --image=busybox --restart=Never | 03:47 |
| sdake | Waiting for pod default/da72cb0f-0bac-4871-bb0f-0c268b171d53 to be running, status is Pending, pod ready: false | 03:47 |
| sdake | Waiting for pod default/da72cb0f-0bac-4871-bb0f-0c268b171d53 to be running, status is Pending, pod ready: false | 03:47 |
| *** dave-mccowan has joined #openstack-kolla | 03:49 | |
| portdirect | sdake: if you can 'decribe' the dns deployment, you should be able to see which container it is thats failing | 03:49 |
| portdirect | 'describe' whoops :) | 03:49 |
| portdirect | it would also be good to get the logs from that container using docker log on the host its trying to run on | 03:49 |
| portdirect | you will then have a better idea of whats up with your deployment | 03:50 |
| sdake | 4m4m3{kubelet kolla}WarningFailedSyncError syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-cxw89_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-cxw89_kube-system(34e95d2f-110d-11e7-865f-9890969934b8)\" using network plugins \"cni\": cni config unintialized; Skipping pod" | 03:50 |
| portdirect | it looks (from reading the scrollback) like to have an issue talking tot the api server from within a pod | 03:50 |
| portdirect | ah! right | 03:51 |
| sdake | works with weave ;) | 03:51 |
| sdake | not with calico | 03:51 |
| portdirect | can you dump the output of 'kubectl --all-namespaces get all' | 03:51 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix api_paste_config option https://review.openstack.org/449859 | 03:51 |
| *** inc0 has joined #openstack-kolla | 03:52 | |
| sdake | http://sprunge.us/BRIO | 03:52 |
| *** dave-mccowan has quit IRC | 03:53 | |
| sdake | betting etcd needs to be 10.3.0.100 | 03:53 |
| sdake | but not really sure | 03:53 |
| portdirect | sdake: ok - cool, what pod and service cidr did you specify when you launched kubeadm? | 03:54 |
| * portdirect is not that familiar with canal, just uses calico | 03:54 | |
| sdake | 1410 sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/16 | 03:54 |
| portdirect | ok - and where did you get the canal2.yaml? did you modify it? | 03:54 |
| sdake | yes it was modified | 03:55 |
| portdirect | cool - any chance you could dump it as well | 03:55 |
| kfox1111 | yeah. etcd probably shoudl be 10.3.0.100 and dns at 10.3.0.10 | 03:55 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix secure_proxy_ssl_header option https://review.openstack.org/449859 | 03:55 |
| sdake | 19:02:51] <sdake>1255 url="https://raw.githubusercontent.com/projectcalico/canal/7deb07cda04147ab49115f437151705c747d0374" | 03:55 |
| sdake | [19:02:51] <sdake> 1256 url="$url/k8s-install/kubeadm/canal.yaml" | 03:55 |
| sdake | [19:02:51] <sdake> 1257 wget $url | 03:55 |
| portdirect | sdake that uri doesnt work for me :( | 03:56 |
| sdake | portdirect its a nested url - have to place in your env | 03:56 |
| sdake | url = '$URL" | 03:56 |
| portdirect | whoops :) | 03:56 |
| sdake | 1256 puts together an url from 1255 | 03:57 |
| portdirect | how did you modify that file? | 03:58 |
| sdake | [sdake@kolla ~]$ diff canal.yaml canal2.yaml | curl -F 'sprunge=<-' http://sprunge.us | 03:58 |
| sdake | http://sprunge.us/jMSc | 03:58 |
| * sdake wtb a working pastebinit or fpaste | 03:58 | |
| sdake | pretty sure i dont have vxlan in my network | 03:59 |
| sdake | not sure if that matters | 03:59 |
| portdirect | so you understand canal arch? | 04:00 |
| sdake | not at all | 04:00 |
| portdirect | its flanneld providing networking, with calico's policy controller for security | 04:00 |
| sdake | just using "what works in the gate" to write some viable docs | 04:00 |
| portdirect | so you should have vxlan networks being created with that config | 04:00 |
| sdake | does underlying switch need to support vxlan? | 04:01 |
| portdirect | yes | 04:01 |
| sdake | (the hardware switch) | 04:01 |
| * sdake groans | 04:01 | |
| sdake | is there a way to avoid that | 04:01 |
| *** caoyuan has quit IRC | 04:01 | |
| portdirect | yup | 04:01 |
| portdirect | you can change it (2 sec) | 04:01 |
| sdake | thanks portdirect | 04:01 |
| *** fooliouno has joined #openstack-kolla | 04:02 | |
| portdirect | sdake: https://github.com/coreos/flannel#backends | 04:02 |
| sdake | so udp than with a dummy switch? | 04:02 |
| portdirect | sdake: I'd really recommend calico for dev/testing - its been working really well for me, and works pretty much anywhere | 04:03 |
| portdirect | sdake: that may work | 04:03 |
| portdirect | sdake: if you just want an aio node, this is by far the simplest way to go: https://raw.githubusercontent.com/portdirect/dockerfiles/kubeadm/aio-deploy/kubeadm-aio/assets/usr/bin/kubeadm-aio | 04:04 |
| sdake | portdirect i looked at that repo | 04:04 |
| portdirect | I'm running that on centos/fedora/ubuntu with the same version of kubeadm as you are using | 04:04 |
| sdake | portdirect i really do want to undertand aio -> multinode | 04:04 |
| eanylin | sdake: I just tested what kfox1111 talked about earlier, it works fine | 04:04 |
| portdirect | super easy | 04:04 |
| eanylin | https://raw.githubusercontent.com/projectcalico/canal/7deb07cda04147ab49115f437151705c747d0374/k8s-install/kubeadm/canal.yaml | 04:04 |
| eanylin | this config works | 04:04 |
| portdirect | sdake: i've scaled that out from single to 5 nodes and then back down again | 04:05 |
| sdake | eanylin that works - if you ahve vxlan switches | 04:05 |
| eanylin | ok | 04:05 |
| sdake | eanylin do you have vxlan switches? | 04:05 |
| eanylin | no, I am just using a normal CentOS VM | 04:05 |
| eanylin | it looks similar to the end results that we saw earlier with weave-net | 04:06 |
| sdake | eanylin that doesn't seem to work for me - dns doesn't come up | 04:06 |
| eanylin | the kubernetes dashboard came up fine too | 04:06 |
| portdirect | sdake, all you need to do is run line 77 on any node you want to join the cluster https://www.irccloud.com/pastebin/pwSByriB/ | 04:06 |
| eanylin | strange though, you want to try and drain your set up like what we did earlier? | 04:07 |
| eanylin | better to start clean? | 04:07 |
| sdake | eanylin i have done that | 04:07 |
| eanylin | ok | 04:07 |
| eanylin | I did the following | 04:08 |
| eanylin | kubeadm init --skip-preflight-checks | 04:09 |
| fooliouno | sdake: can you post the output of "ip a" | 04:09 |
| eanylin | kubectl apply -f canal.yaml | 04:09 |
| *** kolla-slack has quit IRC | 04:14 | |
| srwilkers | bye kolla-slack | 04:20 |
| *** kolla-slack has joined #openstack-kolla | 04:21 | |
| sdake | 4m1m179{kubelet kolla}WarningFailedSyncError syncing pod, skipping: failed to "SetupNetwork" for | 04:24 |
| sdake | "kube-dns-2924299975-6np0x_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-6np0x_kube-system(4fc4218b-1112-11e7-ab3b-989096993 | 04:24 |
| sdake | 4b8)\" using network plugins \"cni\": cni config unintialized; Skipping pod" | 04:24 |
| sdake | this reminds me of halycon errors :) | 04:24 |
| sdake | although now I understand it was kube-dns throwing the error | 04:25 |
| sdake | rather then "kubernetes" | 04:25 |
| *** cuongnv has joined #openstack-kolla | 04:26 | |
| portdirect | sdake: "cni config unintialized" should tell you a lot | 04:26 |
| portdirect | I'm not sure why you are having so much trouble with the network underlay :( | 04:27 |
| portdirect | it this a clean environement you are starting from? | 04:28 |
| portdirect | if you like i could have a look at the box? | 04:29 |
| portdirect | or give you access to one of mine - and we could go through the process together? | 04:30 |
| portdirect | it would be great to get the entire kolla-k8s core team able to deploy k8s confidently | 04:30 |
| sdake | kube-dns-2924299975-366sm 4/4 Running 0 1m | 04:31 |
| portdirect | w00t! | 04:31 |
| sdake | kubectl create -f http://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/calico.yaml | 04:31 |
| sdake | thats without flannel | 04:31 |
| portdirect | from the script i sent you? | 04:31 |
| portdirect | nice :) | 04:31 |
| sdake | yup | 04:31 |
| portdirect | three lines to joy :D | 04:31 |
| sdake | 1043 sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/16 | 04:32 |
| sdake | 1044 kubectl taint nodes --all dedicated- | 04:32 |
| sdake | 1045 kubectl create -f http://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/calico.yaml | 04:32 |
| sdake | 1046 kubectl get pods -n kube-system | 04:32 |
| sdake | 1047 watch kubectl get pods -n kube-system | 04:32 |
| sdake | is what I did | 04:32 |
| sdake | the defaults for kubeadm break my networking | 04:32 |
| portdirect | thats fantastic dude - really happy for you | 04:33 |
| srwilkers | :) | 04:33 |
| sdake | adly ping 8.8.8.8 from busybox doesn't work | 04:37 |
| sdake | sadly that is | 04:37 |
| sdake | and tiller doesn't do its thing | 04:37 |
| sdake | so - kubedns starts - but no networking | 04:37 |
| portdirect | sdake: sad panda | 04:38 |
| sdake | pretty much | 04:38 |
| portdirect | lets have a look at whats going on | 04:38 |
| portdirect | so - can you nslookup kubernetes from the busybox pod? | 04:39 |
| sdake | dns appears dead: / # nslookup kubernetes | 04:39 |
| sdake | Server: 10.3.0.10 | 04:39 |
| portdirect | ok - can we flush out that node? | 04:39 |
| sdake | server is what i set in systemd hacking | 04:39 |
| sdake | you mean completely reload it? | 04:40 |
| portdirect | well ideally yes - I should like that things been through the ringer | 04:40 |
| portdirect | but I can send you a semi safe set of commands | 04:40 |
| portdirect | https://github.com/portdirect/dockerfiles/tree/kubeadm/aio-deploy/kubeadm-aio#cleanup-host | 04:41 |
| portdirect | systemctl stop kubelet 1st | 04:41 |
| portdirect | then those should clear out your shiz, I expect that your cni dir is full of stuff from failed deployments | 04:41 |
| portdirect | (its only really the last 2 you need to run from that link) | 04:42 |
| *** kolla-slack has quit IRC | 04:42 | |
| sdake | portdirect here is my tidy script: | 04:46 |
| sdake | [sdake@kolla ~]$ more /home/sdake/tidy | 04:46 |
| sdake | docker stop $@ `docker ps -a -q` | 04:46 |
| sdake | docker rm $@ `docker ps -a -q` | 04:46 |
| sdake | docker rmi $@ `docker images -a -q` | 04:46 |
| eanylin | sbezverk_: How do we deploy nova cell0 and nova placement? You mentioned about some scripts in the ironic gate job? | 04:46 |
| portdirect | sdake: you need to clean your *host* not just the docker images | 04:47 |
| sdake | i did run the secon dcommand portdirect | 04:47 |
| portdirect | sweet | 04:47 |
| sdake | and also uninstalled the kube stuff | 04:47 |
| sdake | going to try out your aio container | 04:47 |
| sdake | see if that implodes my system or not :) | 04:47 |
| portdirect | lol - its worked on a lot of other ones - lets see if it survives the sdake acid test :) | 04:48 |
| portdirect | sdake: you need to run this: https://github.com/portdirect/dockerfiles/tree/kubeadm/aio-deploy/kubeadm-aio#common-host-setup as otherwise secrets wont work | 04:49 |
| sdake | nslookup: can't resolve 'kubernetes' | 04:56 |
| sdake | ok - enough fu nfor one week - hae a nice weekend folks :) | 04:57 |
| sdake | enjoy | 04:57 |
| portdirect | lets get this sorted next week dude - I really want all of us to be able to deploy k8s, as otherwise I'm a bit concerned about the ability of us to make and take technical direction and desisions | 04:58 |
| portdirect | catch you soon man - we'll get there :) | 04:58 |
| *** cuongnv has quit IRC | 04:59 | |
| *** srwilkers has quit IRC | 05:00 | |
| sdake | portdirect I can deploy kubernetes just fine on different hardware, just not my home lab | 05:00 |
| sdake | the differnet hardware = real switches | 05:01 |
| *** iceyao has joined #openstack-kolla | 05:01 | |
| sdake | portdirect agreed tho - its a problem -I've been hoping would have been solved.... 3 months ago | 05:01 |
| portdirect | I'm pretty snowed under (as always) but can make some time on tueday i think if you want to have a debugging session? | 05:02 |
| sdake | i have meetings all day tuesday from 7am until 6pm | 05:03 |
| sdake | tuesday funday for me | 05:03 |
| portdirect | same :( | 05:03 |
| portdirect | but free after then | 05:03 |
| portdirect | during office hours is pretty much a no-go for me, just too much on atm to play with a bro's home lab :( | 05:04 |
| portdirect | anyway better go too - catch ya later | 05:04 |
| *** jascott1 has quit IRC | 05:19 | |
| *** jascott1 has joined #openstack-kolla | 05:24 | |
| *** zhubingbing has joined #openstack-kolla | 05:32 | |
| *** hieulq_ has joined #openstack-kolla | 05:32 | |
| *** tovin07 has quit IRC | 05:33 | |
| *** hieulq_ has quit IRC | 05:34 | |
| *** hieulq_ has joined #openstack-kolla | 05:36 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 05:40 |
| zhubingbing | guys | 05:40 |
| zhubingbing | hi guys | 05:40 |
| *** eanylin has quit IRC | 06:00 | |
| *** jascott__ has joined #openstack-kolla | 06:03 | |
| *** sayantani01 has joined #openstack-kolla | 06:04 | |
| *** jascott1_ has quit IRC | 06:04 | |
| *** caoyuan has joined #openstack-kolla | 06:05 | |
| *** yangyapeng has quit IRC | 06:05 | |
| *** caoyuan has quit IRC | 06:06 | |
| *** lucasxu has joined #openstack-kolla | 06:14 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Set use kuerbernetes version in gate https://review.openstack.org/449873 | 06:27 |
| *** hieulq_ has quit IRC | 06:31 | |
| *** hieulq_ has joined #openstack-kolla | 06:43 | |
| *** lucasxu has quit IRC | 06:45 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Set use kuerbernetes version in gate https://review.openstack.org/449873 | 07:00 |
| *** hieulq_ has quit IRC | 07:00 | |
| *** shashank_t_ has joined #openstack-kolla | 07:02 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 07:02 |
| *** shashank_t_ has quit IRC | 07:06 | |
| *** yee37915 has joined #openstack-kolla | 07:18 | |
| *** yee379 has quit IRC | 07:19 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware vSphere support https://review.openstack.org/449529 | 07:37 |
| *** vhosakot has quit IRC | 08:01 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 08:09 |
| *** manheim has joined #openstack-kolla | 08:16 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 08:20 |
| *** manheim has quit IRC | 08:21 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware vSphere support https://review.openstack.org/449529 | 08:27 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 08:32 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 08:36 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 08:40 |
| *** shashank_t_ has joined #openstack-kolla | 08:50 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 08:54 |
| *** shashank_t_ has quit IRC | 08:55 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware vSphere support https://review.openstack.org/449529 | 08:59 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 09:03 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 09:09 |
| *** mgoddard has joined #openstack-kolla | 09:11 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 09:17 |
| openstackgerrit | Lingyong Xu proposed openstack/kolla-kubernetes master: Drop py34 target in tox.ini https://review.openstack.org/449898 | 09:20 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware vSphere support https://review.openstack.org/449529 | 09:22 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 09:30 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 09:40 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware vSphere support https://review.openstack.org/449529 | 09:45 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix gate Error helm plugin "template" exited with error https://review.openstack.org/449904 | 09:47 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 09:55 |
| zhubingbing | sup sdake | 09:56 |
| *** klindgren_ has joined #openstack-kolla | 09:59 | |
| *** bmace has quit IRC | 10:01 | |
| *** klindgren has quit IRC | 10:02 | |
| *** bmace has joined #openstack-kolla | 10:02 | |
| *** jascott__ has quit IRC | 10:03 | |
| *** jascott1_ has joined #openstack-kolla | 10:04 | |
| *** kristian__ has joined #openstack-kolla | 10:04 | |
| *** jascott1_ has quit IRC | 10:09 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware vSphere support https://review.openstack.org/449529 | 10:11 |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: test gate https://review.openstack.org/449906 | 10:12 |
| *** manheim has joined #openstack-kolla | 10:13 | |
| *** mgoddard has quit IRC | 10:16 | |
| *** pbourke has quit IRC | 10:19 | |
| *** pbourke has joined #openstack-kolla | 10:21 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 10:28 |
| *** manheim has quit IRC | 10:31 | |
| *** zhubingbing has quit IRC | 10:35 | |
| *** manheim has joined #openstack-kolla | 10:49 | |
| *** manheim has quit IRC | 10:58 | |
| *** kristian__ has quit IRC | 11:15 | |
| *** kristian__ has joined #openstack-kolla | 11:15 | |
| *** kristian__ has quit IRC | 11:20 | |
| *** kristian__ has joined #openstack-kolla | 11:21 | |
| *** yangyapeng has joined #openstack-kolla | 11:26 | |
| *** yangyapeng has quit IRC | 11:29 | |
| *** kristian__ has quit IRC | 11:42 | |
| *** caoyuan has joined #openstack-kolla | 11:55 | |
| *** caoyuan has quit IRC | 12:17 | |
| *** yangyapeng has joined #openstack-kolla | 12:32 | |
| *** Pavo has joined #openstack-kolla | 12:48 | |
| *** yangyapeng has quit IRC | 12:58 | |
| *** jtriley has quit IRC | 13:03 | |
| *** yangyapeng has joined #openstack-kolla | 13:04 | |
| *** caoyuan has joined #openstack-kolla | 13:04 | |
| *** zhurong has joined #openstack-kolla | 13:26 | |
| *** iceyao has quit IRC | 13:33 | |
| *** zhubingbing has joined #openstack-kolla | 13:41 | |
| mnaser | has anyone successfully ran kolla with gre tunnels instead of vxlan | 13:44 |
| mnaser | the br-tun interfaces are not going up | 13:45 |
| mnaser | i dont see any of them being established | 13:45 |
| *** zhubingbing_ has joined #openstack-kolla | 13:45 | |
| *** zhubingbing has quit IRC | 13:46 | |
| *** eanylin has joined #openstack-kolla | 13:58 | |
| *** zhurong has quit IRC | 14:19 | |
| *** eanylin has quit IRC | 14:25 | |
| *** eanylin has joined #openstack-kolla | 14:25 | |
| *** yangyapeng has quit IRC | 14:26 | |
| *** dave-mccowan has joined #openstack-kolla | 14:48 | |
| *** yangyapeng has joined #openstack-kolla | 14:53 | |
| *** dave-mccowan has quit IRC | 14:55 | |
| *** dave-mccowan has joined #openstack-kolla | 14:58 | |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Pin Back Canal https://review.openstack.org/449843 | 14:58 |
| *** dave-mcc_ has joined #openstack-kolla | 15:11 | |
| *** dave-mccowan has quit IRC | 15:14 | |
| *** iceyao has joined #openstack-kolla | 15:20 | |
| *** yangyapeng has quit IRC | 15:27 | |
| *** hrw has quit IRC | 15:34 | |
| *** hrw has joined #openstack-kolla | 15:36 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 15:41 |
| *** Pavo has quit IRC | 15:41 | |
| sdake | morning folks | 15:42 |
| sdake | sbezverk_ why no sbezverk? | 15:42 |
| *** caoyuan has quit IRC | 15:43 | |
| *** kolla-slack has joined #openstack-kolla | 15:46 | |
| *** Pavo has joined #openstack-kolla | 15:52 | |
| *** sbezverk_ is now known as sbezverk | 15:57 | |
| *** dave-mccowan has joined #openstack-kolla | 16:04 | |
| openstackgerrit | Mohammed Naser proposed openstack/kolla stable/newton: Fix neutron-vpnaas for redhat derived bases https://review.openstack.org/449952 | 16:05 |
| *** schwicht has joined #openstack-kolla | 16:06 | |
| *** dave-mcc_ has quit IRC | 16:06 | |
| *** lamt has joined #openstack-kolla | 16:10 | |
| *** zhubingbing__ has joined #openstack-kolla | 16:14 | |
| *** zhubingbing_ has quit IRC | 16:17 | |
| *** dave-mccowan has quit IRC | 16:19 | |
| spsurya | morning sdake | 16:20 |
| *** schwicht has quit IRC | 16:23 | |
| *** eaguilar has joined #openstack-kolla | 16:24 | |
| mnaser | is there any reason why a chown command would fail in kolla | 16:28 |
| mnaser | trying to debug vpnaas | 16:28 |
| *** schwicht has joined #openstack-kolla | 16:28 | |
| mnaser | ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: chown: changing ownership of '/var/lib/neutron/ipsec/94df0215-75d1-458b-a8d3-8c9ae689a520/etc/ipsec.secrets': Operation not permitted | 16:28 |
| *** iceyao has quit IRC | 16:30 | |
| mnaser | 2017-03-25 16:30:18.534 7 DEBUG neutron.agent.linux.utils [-] Running command: ['chown', '--from=998', 'root:root', '/var/lib/neutron/ipsec/94df0215-75d1-458b-a8d3-8c9ae689a520/etc/ipsec.secrets'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:89 | 16:31 |
| mnaser | this is the failing command | 16:31 |
| mnaser | interestingly enough if i run it manually, it works | 16:31 |
| *** manheim has joined #openstack-kolla | 16:40 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 16:47 |
| *** eaguilar has quit IRC | 16:48 | |
| *** jascott1_ has joined #openstack-kolla | 17:00 | |
| *** lamt has quit IRC | 17:03 | |
| *** jascott1_ has quit IRC | 17:05 | |
| *** haplo37 has quit IRC | 17:08 | |
| *** haplo37 has joined #openstack-kolla | 17:17 | |
| *** dave-mccowan has joined #openstack-kolla | 17:24 | |
| sdake | sup peeps | 17:26 |
| sdake | ok - the instructions are shaping up | 17:26 |
| sdake | although still incomplete | 17:26 |
| *** Pavo has quit IRC | 17:28 | |
| sdake | sbezverk any idea what this should be set to: dns_name: "192.168.1.101" | 17:28 |
| sdake | sbezverk and this: external_vip: "192.168.1.101" | 17:28 |
| *** manheim has quit IRC | 17:29 | |
| *** Pavo has joined #openstack-kolla | 17:52 | |
| *** lamt has joined #openstack-kolla | 18:00 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 18:00 |
| sbezverk | sdake: are you kidding me?? yesterday you asked the same question, I answered and now we are bakc to square 1?? Please read the scroll back from yesterday | 18:02 |
| sdake | i have a cloud yaml | 18:02 |
| sdake | you said to set them | 18:02 |
| sdake | you didn't say what they do | 18:02 |
| sdake | i need to know what they do to set them properlly | 18:02 |
| sdake | sbezverk i have set dns_name to my cluster ip and external_vip to my cluster ip (my "eth0" on my cluster is 192.168.1.101" | 18:04 |
| sdake | I am not convinced this is correct | 18:04 |
| sdake | sbezverk I have almost the entire workflow documented, atleast in my mind, minus these two values plus whatever charts are needed to get cells0 and apiplacement going | 18:04 |
| sdake | i cna sort out cells0 and api placement on my own | 18:04 |
| sdake | trust me when I say that nobody else in the community knows what to set these values to either - had 10-15 people on a call yesterday - none of them know what to set them too either | 18:05 |
| sdake | sbezverk in other words, it isn't documented anywhere - I am documenting it so this question isn't asked over and over and over | 18:05 |
| *** dave-mccowan has quit IRC | 18:08 | |
| *** Pavo has quit IRC | 18:08 | |
| *** lamt has quit IRC | 18:11 | |
| *** kolla-slack has quit IRC | 18:18 | |
| *** zhubingbing__ has quit IRC | 18:20 | |
| *** Pavo has joined #openstack-kolla | 18:26 | |
| *** Pavo has quit IRC | 18:30 | |
| *** lamt has joined #openstack-kolla | 18:45 | |
| *** schwicht has quit IRC | 19:06 | |
| thomas_oneill | hey how did some of you guys get your contributions to show up against your github accounts in the github commit log? | 19:13 |
| *** lamt has quit IRC | 19:21 | |
| *** jascott1_ has joined #openstack-kolla | 19:28 | |
| portdirect | Hey up kolla peeps o/ | 19:38 |
| eanylin | sbezverk_: I looked at the shell scripts in the gate_common.sh and updated the placement_api config, deployed the nova-placement-deployment helm chart and I can see that the nova compute stopped crashing after that | 19:47 |
| eanylin | I can see that the nova scheduler is not able to find a valid host to spin the cirros VM | 19:47 |
| eanylin | but I did deploy the microservice for cell0 | 19:47 |
| eanylin | the db for nova cell0 got created but its empty | 19:48 |
| eanylin | is this the reason why its failing? | 19:48 |
| portdirect | thomas_oneill: you just need to have the same details for not github and openstack's gerrit. No magic | 19:56 |
| portdirect | (Extra 'not' in that... Weird) | 19:56 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures https://review.openstack.org/434817 | 20:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu https://review.openstack.org/432787 | 20:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add support for non-x86 architectures (aarch64, ppc64le) https://review.openstack.org/430940 | 20:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base/kolla-toolbox: use mariadb-devel for CentOS and !x86-64 https://review.openstack.org/444832 | 20:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: handle ppc64le https://review.openstack.org/434810 | 20:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ceph: move ceph-fuse package to cephfs-fuse image https://review.openstack.org/440634 | 20:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: kubernetes: disable for architectures other than x86-64 https://review.openstack.org/445638 | 20:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-compute: handle rtslib(-fb) for debian-binary builds too https://review.openstack.org/435941 | 20:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: move to stretch https://review.openstack.org/434453 | 20:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base: Percona-Server is x86-64 only https://review.openstack.org/449965 | 20:01 |
| *** lamt has joined #openstack-kolla | 20:02 | |
| *** jtriley has joined #openstack-kolla | 20:06 | |
| *** manheim has joined #openstack-kolla | 20:21 | |
| *** lamt has quit IRC | 20:23 | |
| *** manheim has quit IRC | 21:02 | |
| *** jtriley has quit IRC | 21:09 | |
| *** eaguilar has joined #openstack-kolla | 21:21 | |
| *** eaguilar has quit IRC | 21:37 | |
| *** lamt has joined #openstack-kolla | 21:43 | |
| *** lamt has quit IRC | 22:17 | |
| sdake | eanylin would you mind adding your steps to the review here: https://review.openstack.org/#/c/447356/ (for cells0 and placement api) | 22:19 |
| sdake | eanylin line 155 | 22:20 |
| sdake | sup portdirect | 22:20 |
| sdake | sbezverk note i found the references to the data i was asking for in the scrollback - thanks | 22:22 |
| sdake | sbezverk they are in the review now | 22:23 |
| *** sean-k-mooney has quit IRC | 22:23 | |
| sdake | sbezverk if you have a chance you might have a look at it | 22:23 |
| *** sean-k-mooney has joined #openstack-kolla | 22:23 | |
| sdake | eanylin i think it is possible a configmap is missing for cell0 and placement api - although I am uncertain | 22:23 |
| sdake | eanylin i'd dig into that myself but going out | 22:23 |
| eanylin | sdake: Ok, I am trying same steps on another laptop, I am not sure if those are the correct steps though, since I wasnt able to get the Cirros up | 22:25 |
| sdake | eanylin if which are the correct steps | 22:25 |
| sdake | eanylin the ones you havve? | 22:26 |
| sdake | eanylin if you ahve steps to contribute, it may help others make forward progress - you can always add a preface that says "not sure if this is right, but this got me further" | 22:27 |
| eanylin | sure | 22:28 |
| eanylin | ok, I will do it after I get my dinner | 22:28 |
| sdake | eanylin thanks fine sir :) | 22:28 |
| eanylin | :) | 22:28 |
| sdake | i'll give em a runthrough with weae | 22:28 |
| sdake | since that works | 22:28 |
| sdake | (for me) | 22:28 |
| eanylin | yeah, so I tested with the master.sh script under kube-deploy | 22:29 |
| eanylin | seems like that one will work too, with flannel | 22:29 |
| sdake | which is master.sh? | 22:30 |
| eanylin | but that one doesnt use kubeadm though | 22:30 |
| eanylin | its inside the official document | 22:30 |
| eanylin | https://docs.openstack.org/developer/kolla-kubernetes/kubernetes-setup.html#single-node-kubernetes-setup | 22:30 |
| sdake | cool - i wasn't sure which script you were talking about | 22:32 |
| sdake | that script is from kube-deploy | 22:32 |
| eanylin | yeah | 22:32 |
| sdake | eanylin we have many scripts ;) | 22:32 |
| sdake | and many are third party | 22:32 |
| eanylin | I know, thats why its confusing after a while ;-) | 22:32 |
| sdake | ya I'd like to unify that into one doc and guide so people can understand how deployment works | 22:33 |
| eanylin | I know this script works pretty well, at least for AIO | 22:33 |
| sdake | cool | 22:33 |
| eanylin | agree | 22:33 |
| *** eaguilar has joined #openstack-kolla | 22:55 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 23:14 |
| *** eaguilar_ has joined #openstack-kolla | 23:17 | |
| *** eaguilar has quit IRC | 23:18 | |
| *** schwicht has joined #openstack-kolla | 23:20 | |
| *** eaguilar_ has quit IRC | 23:46 | |
| *** Pavo has joined #openstack-kolla | 23:46 | |
| *** lamt has joined #openstack-kolla | 23:49 | |
| *** eaguilar has joined #openstack-kolla | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!