| *** MarkBaker has joined #openstack-helm | 00:43 | |
| *** MarkBaker has quit IRC | 00:48 | |
| *** julim has joined #openstack-helm | 00:56 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Support services in different namespaces https://review.openstack.org/510810 | 01:22 |
|---|---|---|
| *** MarkBaker has joined #openstack-helm | 02:16 | |
| *** petevg is now known as petevg|afk | 02:17 | |
| openstackgerrit | Vlad Naboichenko proposed openstack/openstack-helm master: Upgrade kubernetes and helm versions https://review.openstack.org/517161 | 02:32 |
| *** MarkBaker has quit IRC | 02:33 | |
| openstackgerrit | Vlad Naboichenko proposed openstack/openstack-helm master: Upgrade kubernetes and helm versions https://review.openstack.org/517161 | 02:33 |
| *** larainema has joined #openstack-helm | 02:45 | |
| *** MarkBaker has joined #openstack-helm | 03:16 | |
| *** MarkBaker has quit IRC | 03:41 | |
| *** jistr has quit IRC | 03:42 | |
| *** jistr has joined #openstack-helm | 03:43 | |
| *** MarkBaker has joined #openstack-helm | 03:51 | |
| *** coolsvap has joined #openstack-helm | 04:06 | |
| *** MarkBaker has quit IRC | 04:24 | |
| *** yamamoto has joined #openstack-helm | 04:24 | |
| openstackgerrit | Hyunsun Moon proposed openstack/openstack-helm master: Add option to set external policy to local for openstack services https://review.openstack.org/517171 | 06:08 |
| *** MarkBaker has joined #openstack-helm | 06:28 | |
| *** julim has quit IRC | 06:55 | |
| *** julim has joined #openstack-helm | 06:57 | |
| *** hachi has joined #openstack-helm | 07:07 | |
| *** hachi__ has joined #openstack-helm | 07:11 | |
| *** hachi has quit IRC | 07:13 | |
| *** huxinhui__ has joined #openstack-helm | 07:32 | |
| *** huxinhui__ is now known as huxinhui | 07:34 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 08:14 |
| *** MarkBaker has quit IRC | 08:29 | |
| openstackgerrit | Dae Seong Kim proposed openstack/openstack-helm master: Enable cell service https://review.openstack.org/516810 | 08:36 |
| *** bjolo has joined #openstack-helm | 08:36 | |
| openstackgerrit | Mateusz Blaszkowski proposed openstack/openstack-helm master: Log based metrics using fluent-plugin-prometheus https://review.openstack.org/514938 | 08:40 |
| *** MarkBaker has joined #openstack-helm | 08:56 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: RBAC authorization support https://review.openstack.org/464630 | 09:05 |
| *** yamamoto has quit IRC | 09:18 | |
| *** yamamoto has joined #openstack-helm | 09:24 | |
| *** gouthamr has quit IRC | 09:26 | |
| *** yamamoto has quit IRC | 09:28 | |
| *** yamamoto has joined #openstack-helm | 09:35 | |
| *** yamamoto has quit IRC | 09:35 | |
| *** yamamoto has joined #openstack-helm | 09:57 | |
| *** yamamoto has quit IRC | 11:16 | |
| *** yamamoto has joined #openstack-helm | 11:19 | |
| *** yamamoto has quit IRC | 11:24 | |
| *** huxinhui has quit IRC | 11:31 | |
| *** yamamoto has joined #openstack-helm | 12:11 | |
| *** yamamoto has quit IRC | 12:15 | |
| *** yamamoto has joined #openstack-helm | 12:15 | |
| *** yamamoto has quit IRC | 12:20 | |
| *** krypto has joined #openstack-helm | 12:26 | |
| *** krypto has joined #openstack-helm | 12:26 | |
| *** jaypipes has joined #openstack-helm | 12:27 | |
| *** yamamoto has joined #openstack-helm | 12:47 | |
| *** yamamoto has quit IRC | 12:47 | |
| *** yamamoto has joined #openstack-helm | 12:50 | |
| *** yamamoto has quit IRC | 13:24 | |
| *** yamamoto has joined #openstack-helm | 13:28 | |
| openstackgerrit | Sungil Im proposed openstack/openstack-helm master: Fluent-logging helm chart https://review.openstack.org/514622 | 13:37 |
| *** alraddarla has joined #openstack-helm | 14:10 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 14:24 |
| *** coolsvap has quit IRC | 14:26 | |
| *** felipemonteiro_ has joined #openstack-helm | 14:48 | |
| *** felipemonteiro__ has joined #openstack-helm | 14:49 | |
| *** krypto has quit IRC | 14:51 | |
| *** krypto has joined #openstack-helm | 14:52 | |
| *** alraddarla has left #openstack-helm | 14:52 | |
| *** felipemonteiro_ has quit IRC | 14:53 | |
| *** lrensing has joined #openstack-helm | 14:56 | |
| *** krypto has quit IRC | 14:57 | |
| *** krypto has joined #openstack-helm | 14:57 | |
| *** krypto has quit IRC | 14:57 | |
| *** krypto has joined #openstack-helm | 14:57 | |
| *** lrensing has quit IRC | 15:00 | |
| *** krypto has quit IRC | 15:02 | |
| *** krypto has joined #openstack-helm | 15:03 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 15:04 |
| korzen | Hi portdirect | 15:22 |
| korzen | portdirect, I have seen that ceph-mon-check is not working properly: http://logs.openstack.org/28/506028/8/check/legacy-openstack-helm-aio-basic-ovs-rbd/8dde683/logs/k8s/pods/ceph-ceph-mon-check-1041216057-0p8vq-ceph-mon.txt | 15:24 |
| korzen | Traceback (most recent call last): | 15:25 |
| korzen | File "/check_zombie_mons.py", line 31, in <module> | 15:25 |
| korzen | expected_mons = extract_mons_from_kubeapi() | 15:25 |
| korzen | File "/check_zombie_mons.py", line 28, in extract_mons_from_kubeapi | 15:25 |
| korzen | return json.loads(kubemap) | 15:25 |
| korzen | File "/usr/lib/python2.7/json/__init__.py", line 339, in loads | 15:25 |
| portdirect | hmmm | 15:25 |
| korzen | return _default_decoder.decode(s) | 15:25 |
| korzen | File "/usr/lib/python2.7/json/decoder.py", line 367, in decode | 15:25 |
| korzen | raise ValueError(errmsg("Extra data", s, end, len(s))) | 15:25 |
| korzen | ValueError: Extra data: line 1 column 4 - line 1 column 110 (char 3 - 109) | 15:25 |
| portdirect | you have any idea when that came in? | 15:25 |
| portdirect | totally flew under my radar | 15:25 |
| korzen | no idea | 15:25 |
| *** lrensing has joined #openstack-helm | 15:26 | |
| korzen | your commit Change-Id: Ibc445e422c1ebad972de59b47955511d14dd6d3c | 15:26 |
| korzen | 5 september | 15:27 |
| korzen | or it is the line you have edited: https://github.com/openstack/openstack-helm/blob/master/ceph/templates/bin/_check_zombie_mons.py.tpl#L10 | 15:29 |
| portdirect | yeah i think thats it | 15:29 |
| portdirect | need to check and see if that var actually exists. | 15:29 |
| osh-chatbot1 | <lrensing> https://github.com/openstack/openstack-helm/blob/master/ceph/templates/bin/_check_zombie_mons.py.tpl#L40 | 15:29 |
| osh-chatbot1 | <lrensing> we changed ‘cluster’ to namespace, didn’t we? | 15:30 |
| korzen | the problem is that the template is not working | 15:30 |
| *** hachi__ has quit IRC | 15:30 | |
| portdirect | its somewhere between the two there :/ | 15:32 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: Add Elasticsearch and Kibana https://review.openstack.org/512881 | 15:32 |
| portdirect | I wont have time to look properly until this afternoon - but if you could have a look lrensing it would be awesome | 15:32 |
| osh-chatbot1 | <lrensing> not sure i will either today - i’ll put it on my laundry list | 15:33 |
| korzen | I do not know exactly wht this script is doing | 15:34 |
| korzen | but it does not seem to be doing the right thing ;) | 15:34 |
| korzen | I have fixed the template and cluter->namespace and this is the output: | 15:35 |
| korzen | root@ceph-mon-check-4262613270-m2pbx:/etc/ceph# /check_zombie_mons2.py | 15:35 |
| korzen | got monmap epoch 1 | 15:35 |
| korzen | kubectl get pods --namespace=${NAMESPACE} -l component=mon -l application=ceph -o template --template="{ {{range \$i, \$v := .items}} {{ if \$i}} , {{ end }} \"{{\$v.spec.nodeName}}\": \"{{\$v.status.podIP}}\" {{end}} }" | 15:35 |
| korzen | { "node1": "192.168.166.167" , "node1": "192.168.166.181" , "node1": "172.16.8.5" , "node1": "172.16.8.5" , "node1": "192.168.166.184" } | 15:35 |
| korzen | current mons: {'node1': '172.16.8.5'} | 15:35 |
| korzen | expected mons: {u'node1': u'192.168.166.184'} | 15:35 |
| korzen | ip change dedected for pod node1 | 15:35 |
| korzen | Error from server (NotFound): pods "node1" not found | 15:35 |
| korzen | deleted mon node1 via the kubernetes api | 15:35 |
| korzen | no idea why current mons are not equal expected mons, and why expected is 192.168.166.184 | 15:36 |
| korzen | also, the questino why this script is trying to delete the non existing pod ;) | 15:36 |
| portdirect | can you run `kubectl get nodes` | 15:37 |
| korzen | node1 is the node I have, yes | 15:37 |
| portdirect | do both those ips exist on the node | 15:38 |
| portdirect | if so i think I know whats going on | 15:38 |
| korzen | ubuntu@node1:~/helm/openstack-helm$ ip a | grep 192 | 15:38 |
| korzen | inet 192.168.166.128/32 scope global tunl0 | 15:38 |
| korzen | this is the calico CNI network | 15:38 |
| korzen | NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE | 15:39 |
| korzen | ceph ceph-mds-915759678-j2cwj 1/1 Running 0 3h 192.168.166.167 node1 | 15:39 |
| korzen | ceph ceph-mon-check-4262613270-m2pbx 1/1 Running 0 3h 192.168.166.181 node1 | 15:39 |
| korzen | ceph ceph-mon-jd5m2 1/1 Running 0 3h 172.16.8.5 node1 | 15:39 |
| korzen | ceph ceph-osd-gj5pr 1/1 Running 0 3h 172.16.8.5 node1 | 15:39 |
| korzen | ceph ceph-rbd-provisioner-2986409607-ph2dq 1/1 Running 0 3h 192.168.166.172 node1 | 15:39 |
| korzen | ceph ceph-rgw-3394275655-v25tk 1/1 Running 0 3h 192.168.166.184 node1 | 15:40 |
| portdirect | wait | 15:40 |
| *** yamamoto has quit IRC | 15:41 | |
| portdirect | should ` -l component=mon -l application=ceph` acutally be ` -l component=mon ,application=ceph` | 15:41 |
| portdirect | sorry ` -l component=mon,application=ceph` | 15:41 |
| portdirect | can you give the output of the kubectl command with those labels applied | 15:42 |
| *** yamamoto has joined #openstack-helm | 15:43 | |
| korzen | root@ceph-mon-check-4262613270-m2pbx:/# /check_zombie_mons2.py | 15:44 |
| korzen | got monmap epoch 1 | 15:44 |
| korzen | kubectl get pods --namespace=${NAMESPACE} -l component=mon,application=ceph -o template --template="{ {{range \$i, \$v := .items}} {{ if \$i}} , {{ end }} \"{{\$v.spec.nodeName}}\": \"{{\$v.status.podIP}}\" {{end}} }" | 15:44 |
| korzen | { "node1": "172.16.8.5" } | 15:44 |
| korzen | current mons: {'node1': '172.16.8.5'} | 15:44 |
| korzen | expected mons: {u'node1': u'172.16.8.5'} | 15:44 |
| korzen | no zombie mons found ... | 15:44 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: Add Elasticsearch and Kibana https://review.openstack.org/512881 | 15:46 |
| *** yamamoto has quit IRC | 15:48 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: Ceph: check zombie mons fix https://review.openstack.org/517360 | 15:58 |
| *** yamamoto has joined #openstack-helm | 16:04 | |
| *** julim has quit IRC | 16:08 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 16:18 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 16:22 |
| *** petevg|afk is now known as petevg | 16:25 | |
| *** julim has joined #openstack-helm | 16:52 | |
| *** julim_ has joined #openstack-helm | 17:02 | |
| *** julim has quit IRC | 17:02 | |
| *** yamamoto has joined #openstack-helm | 17:05 | |
| *** yamamoto has quit IRC | 17:12 | |
| openstackgerrit | Vlad Naboichenko proposed openstack/openstack-helm master: Upgrade kubernetes and helm versions https://review.openstack.org/517161 | 17:23 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 17:32 |
| *** gouthamr has joined #openstack-helm | 18:00 | |
| *** lukepatrick has joined #openstack-helm | 18:04 | |
| *** lrensing has quit IRC | 18:42 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: WIP: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 18:57 |
| *** hachi has joined #openstack-helm | 18:58 | |
| *** julim has joined #openstack-helm | 19:06 | |
| *** julim_ has quit IRC | 19:08 | |
| *** julim has quit IRC | 19:08 | |
| *** lrensing has joined #openstack-helm | 19:26 | |
| *** julim has joined #openstack-helm | 19:36 | |
| *** hachi has quit IRC | 20:13 | |
| *** julim has quit IRC | 20:17 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 20:20 |
| openstackgerrit | Vlad Naboichenko proposed openstack/openstack-helm master: Upgrade kubernetes version https://review.openstack.org/517161 | 20:20 |
| openstackgerrit | Vlad Naboichenko proposed openstack/openstack-helm master: Upgrade helm version to 2.7.0 https://review.openstack.org/517422 | 20:20 |
| *** felipemonteiro_ has joined #openstack-helm | 20:42 | |
| openstackgerrit | Andrey Pavlov proposed openstack/openstack-helm master: fix veriable definition https://review.openstack.org/517433 | 20:44 |
| openstackgerrit | Andrey Pavlov proposed openstack/openstack-helm master: extract checking VM to common functions https://review.openstack.org/517434 | 20:44 |
| openstackgerrit | Andrey Pavlov proposed openstack/openstack-helm master: add info what happens and timeouts to dump_logs.sh https://review.openstack.org/517435 | 20:44 |
| *** felipemonteiro__ has quit IRC | 20:44 | |
| openstackgerrit | Andrey Pavlov proposed openstack/openstack-helm master: fix variable definition in gating script https://review.openstack.org/517433 | 21:03 |
| openstackgerrit | Andrey Pavlov proposed openstack/openstack-helm master: extract checking VM to common functions https://review.openstack.org/517434 | 21:03 |
| openstackgerrit | Andrey Pavlov proposed openstack/openstack-helm master: add info what happens and timeouts to dump_logs.sh https://review.openstack.org/517435 | 21:03 |
| openstackgerrit | Andrey Pavlov proposed openstack/openstack-helm master: [WIP] add gating for nfs backend and try to fix it https://review.openstack.org/517438 | 21:04 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 21:17 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: Image local repo jobs and multiple namespace support https://review.openstack.org/517202 | 21:24 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: RBAC: enforce standard k8s RBAC policy https://review.openstack.org/517441 | 21:36 |
| *** julim has joined #openstack-helm | 21:48 | |
| *** yamamoto has joined #openstack-helm | 21:51 | |
| *** felipemonteiro_ has quit IRC | 21:56 | |
| *** jaypipes has quit IRC | 22:29 | |
| *** lukepatrick has quit IRC | 22:50 | |
| *** felipemonteiro_ has joined #openstack-helm | 23:08 | |
| *** lrensing has quit IRC | 23:18 | |
| *** felipemonteiro_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!