| *** kbaegis has joined #openstack-kolla | 00:00 | |
| *** mnasiadka has joined #openstack-kolla | 00:00 | |
| *** mnasiadka has quit IRC | 00:05 | |
| *** huzhengchuan_ has joined #openstack-kolla | 00:08 | |
| *** huzhengchuan has joined #openstack-kolla | 00:08 | |
| sdake | kfox1111 mariadb/rabbitmq are working with tip of ocata | 00:08 |
|---|---|---|
| sdake | kfox1111 i am struggling to override glance: | 00:08 |
| sdake | + glance-api: | 00:08 |
| sdake | + all: | 00:08 |
| sdake | + image_full: "docker.io/kolla/centos-source-glance-api:4.0.2" | 00:08 |
| sdake | + glance-registry: | 00:08 |
| sdake | + all: | 00:08 |
| sdake | + image_full: "docker.io/kolla/centos-source-glance-registry:4.0.2" | 00:08 |
| *** itlinux has joined #openstack-kolla | 00:09 | |
| kfox1111 | I don't have my laptop handy... | 00:09 |
| kfox1111 | check the search path on each of those. | 00:09 |
| *** eaguilar has quit IRC | 00:10 | |
| *** chas has joined #openstack-kolla | 00:18 | |
| *** chas has quit IRC | 00:23 | |
| *** manheim has joined #openstack-kolla | 00:28 | |
| *** manheim has quit IRC | 00:32 | |
| *** Pavo has joined #openstack-kolla | 00:35 | |
| *** eaguilar has joined #openstack-kolla | 00:35 | |
| *** itlinux has quit IRC | 00:42 | |
| kbaegis | So does kolla-build assume a python version? How about kolla-ansible? | 00:46 |
| kbaegis | Some of the stable/ocata scripting for k-a seems to assume py3.5, whereas kolla-build seems to prefer 2.7 | 00:47 |
| kbaegis | Is that inaccurate? | 00:47 |
| *** cuongnv has joined #openstack-kolla | 00:48 | |
| *** duonghq has joined #openstack-kolla | 00:49 | |
| duonghq | morning guys | 00:49 |
| kbaegis | duonghq: Morning | 00:49 |
| kbaegis | 🔮 | 00:50 |
| *** eaguilar has quit IRC | 00:51 | |
| *** eaguilar has joined #openstack-kolla | 00:54 | |
| *** chas has joined #openstack-kolla | 00:55 | |
| *** manheim has joined #openstack-kolla | 00:55 | |
| *** tovin07_ has joined #openstack-kolla | 00:56 | |
| duonghq | hi kbaegis | 00:56 |
| *** imcsk8 has quit IRC | 00:56 | |
| *** imcsk8_ has joined #openstack-kolla | 00:56 | |
| kbaegis | duonghq: What's up? | 00:56 |
| *** dixiaoli has joined #openstack-kolla | 00:56 | |
| duonghq | everything happens as usual :P | 00:57 |
| duonghq | how about you | 00:57 |
| kbaegis | duonghq: Trying to get kolla up on gentoo again. | 00:57 |
| kbaegis | requires venvs which I'm getting used to | 00:57 |
| duonghq | kbaegis, nice | 00:57 |
| kbaegis | kolla-build works perfectly, but kolla-ansible is giving me some headaches | 00:58 |
| kbaegis | I can pass prechecks just fine, but deploy is still breaking | 00:59 |
| duonghq | kbaegis, what kind of problem are you facing? | 00:59 |
| *** chas has quit IRC | 00:59 | |
| *** manheim has quit IRC | 01:00 | |
| kbaegis | https://hastebin.com/owomofepib.rb | 01:00 |
| *** yuanying_ has joined #openstack-kolla | 01:01 | |
| kbaegis | https://hastebin.com/fuyacolawo.vbs | 01:01 |
| *** mnasiadka has joined #openstack-kolla | 01:01 | |
| *** yuanying has quit IRC | 01:02 | |
| kbaegis | So mostly python issues it looks like | 01:02 |
| duonghq | kbaegis, can you try py34? | 01:06 |
| *** mnasiadka has quit IRC | 01:06 | |
| kbaegis | Sure. I can try rolling back | 01:08 |
| kbaegis | Give me 20 minutes | 01:08 |
| kbaegis | Just tried it as root instead of my user | 01:08 |
| *** stefan_bo has joined #openstack-kolla | 01:10 | |
| duonghq | ya I'm also working on https://review.openstack.org/#/c/398682/, a few steps before is merged | 01:11 |
| *** daidv has joined #openstack-kolla | 01:11 | |
| *** MarginHu has joined #openstack-kolla | 01:12 | |
| *** dixiaoli has quit IRC | 01:14 | |
| *** stefan_bo has quit IRC | 01:15 | |
| kbaegis | same failure under 3.4 | 01:16 |
| kbaegis | I actually disabled the haproxy service | 01:17 |
| kbaegis | So I don't know why it's creating a mysql user | 01:17 |
| *** dave-mccowan has joined #openstack-kolla | 01:21 | |
| kbaegis | Nice :D | 01:22 |
| kbaegis | So are there assumptions about the kernel config and sysfs? | 01:23 |
| *** ducttape_ has joined #openstack-kolla | 01:24 | |
| *** eanylin has quit IRC | 01:25 | |
| kbaegis | *sigh* | 01:26 |
| kbaegis | Have a good night | 01:27 |
| *** kbaegis has quit IRC | 01:27 | |
| *** eaguilar has quit IRC | 01:27 | |
| *** jascott1 has quit IRC | 01:28 | |
| MarginHu | hi, I have a question about openstack ha in kolla | 01:30 |
| *** yuanying_ has quit IRC | 01:30 | |
| *** chas has joined #openstack-kolla | 01:31 | |
| *** hrw has quit IRC | 01:32 | |
| *** yuanying has joined #openstack-kolla | 01:34 | |
| *** itlinux has joined #openstack-kolla | 01:34 | |
| *** hrw has joined #openstack-kolla | 01:34 | |
| *** chas has quit IRC | 01:35 | |
| *** dave-mccowan has quit IRC | 01:36 | |
| *** itlinux has quit IRC | 01:38 | |
| *** zhubingbing_ has joined #openstack-kolla | 01:39 | |
| *** ducttape_ has quit IRC | 01:41 | |
| *** m1dev has quit IRC | 01:49 | |
| *** ducttape_ has joined #openstack-kolla | 01:50 | |
| *** zhurong has joined #openstack-kolla | 01:51 | |
| *** chas has joined #openstack-kolla | 01:52 | |
| *** ducttape_ has quit IRC | 01:53 | |
| *** ducttape_ has joined #openstack-kolla | 01:56 | |
| *** chas has quit IRC | 01:56 | |
| *** ducttape_ has quit IRC | 02:01 | |
| *** ducttape_ has joined #openstack-kolla | 02:03 | |
| *** ducttape_ has quit IRC | 02:03 | |
| *** Pavo has quit IRC | 02:04 | |
| *** ducttape_ has joined #openstack-kolla | 02:06 | |
| *** stefan_bo has joined #openstack-kolla | 02:11 | |
| *** unicell1 has quit IRC | 02:15 | |
| *** ducttape_ has quit IRC | 02:17 | |
| *** ducttape_ has joined #openstack-kolla | 02:20 | |
| *** manheim has joined #openstack-kolla | 02:21 | |
| *** eanylin has joined #openstack-kolla | 02:24 | |
| *** eaguilar has joined #openstack-kolla | 02:26 | |
| *** manheim has quit IRC | 02:26 | |
| *** ducttape_ has quit IRC | 02:26 | |
| *** chas has joined #openstack-kolla | 02:27 | |
| jamesPR | MarginHu, ? | 02:28 |
| MarginHu | hi, jamesPR | 02:28 |
| jamesPR | hi | 02:29 |
| jamesPR | :) | 02:29 |
| MarginHu | I don't find any fence mechanism in haproxy+keepalived , how to handle split-brain ? | 02:29 |
| *** ducttape_ has joined #openstack-kolla | 02:32 | |
| *** chas has quit IRC | 02:32 | |
| jamesPR | what do you mean with split-brain? | 02:33 |
| jamesPR | network issues? | 02:34 |
| *** zhubingbing_ has quit IRC | 02:35 | |
| *** zhubingbing has joined #openstack-kolla | 02:35 | |
| SamYaple | MarginHu: there is no fencing involved, you are correct. youll need to ensure that yourself | 02:36 |
| MarginHu | stateful service like database or message queue , how to ensure data consistency in a cluster? | 02:38 |
| SamYaple | MarginHu: the database uses galera which ensures its consistency | 02:39 |
| SamYaple | message queue (rabbitmq) does not have any mechanisms for that. that is by design | 02:39 |
| SamYaple | well for syncing messages that is | 02:39 |
| SamYaple | each queue has a master, and all are copied around | 02:39 |
| *** ducttape_ has quit IRC | 02:40 | |
| SamYaple | if you reboot rabbitmq nodes too fast you can lose messages | 02:40 |
| MarginHu | jamesPR, now I can't enumerate, but split-brain or partition is a usual thing. | 02:40 |
| *** zhubingbing has quit IRC | 02:41 | |
| MarginHu | SamYaple, I see openstack ha in mainstream is using pacemaker,corosync | 02:42 |
| SamYaple | youre welcome to use and implement those | 02:42 |
| SamYaple | that is not the _common_ deployment though | 02:42 |
| SamYaple | personally, i do not use haproxy/keepalived. i use an external loadbalancer | 02:43 |
| *** manheim has joined #openstack-kolla | 02:43 | |
| MarginHu | how to ensure it by myself ? | 02:44 |
| *** ducttape_ has joined #openstack-kolla | 02:44 | |
| SamYaple | well you can setup pacemaker,corosync outside of kolla on each of hte loadbalancer nodes to do the fencing | 02:44 |
| SamYaple | just have it start/stop the containers itself when it detects split-brain | 02:44 |
| SamYaple | but ingeneral, you just want to ensure the vip lives on only a single node | 02:45 |
| *** xinliang has quit IRC | 02:46 | |
| jamesPR | pacemaker, corosync sucks :) | 02:47 |
| *** ducttape_ has quit IRC | 02:47 | |
| *** manheim has quit IRC | 02:47 | |
| SamYaple | jamesPR: i wont disagree there | 02:47 |
| MarginHu | ok, If I don't setup pacemaker ,corosync , which risks is there existed in production environment ? | 02:47 |
| *** ducttape_ has joined #openstack-kolla | 02:47 | |
| SamYaple | MarginHu: lots of people use keepalived in production without pacemaker/corosync | 02:47 |
| SamYaple | so the risk would be the vip ends up on two nodes | 02:48 |
| SamYaple | that will cause unpredicatable network issues | 02:48 |
| MarginHu | yes , that's why i have this question. | 02:48 |
| MarginHu | I feel strange about that. | 02:49 |
| MarginHu | ok, Is there a plan for adding pacemaker and corosync in kolla ? | 02:50 |
| SamYaple | no, but no one is blocking you from writing it, or doing it external to kolla | 02:51 |
| *** yuanying has quit IRC | 02:51 | |
| *** spsurya has joined #openstack-kolla | 02:52 | |
| MarginHu | yes, maybe I will do that for my production enrionment. | 02:52 |
| *** ducttape_ has quit IRC | 02:53 | |
| *** jamesPR has quit IRC | 02:54 | |
| *** yuanying_ has joined #openstack-kolla | 02:55 | |
| *** ducttape_ has joined #openstack-kolla | 02:55 | |
| *** mnasiadka has joined #openstack-kolla | 03:02 | |
| *** sayantan_ has quit IRC | 03:03 | |
| *** sayantan_ has joined #openstack-kolla | 03:03 | |
| *** chas has joined #openstack-kolla | 03:04 | |
| *** manheim has joined #openstack-kolla | 03:06 | |
| *** mnasiadka has quit IRC | 03:07 | |
| *** ducttape_ has quit IRC | 03:08 | |
| *** chas has quit IRC | 03:08 | |
| *** manheim has quit IRC | 03:10 | |
| inc0 | good evening | 03:17 |
| duonghq | evening inc0 | 03:17 |
| inc0 | can I have some reviews plz?:) https://review.openstack.org/#/c/466007/ | 03:18 |
| inc0 | Jeffrey4l: ^ | 03:18 |
| duonghq | inc0, if you have spare time, I hope that I can get some suggestion on https://review.openstack.org/#/c/398684/15 | 03:21 |
| inc0 | kk duonghq I'll check it out | 03:22 |
| *** sayantan_ has quit IRC | 03:27 | |
| *** sayantan_ has joined #openstack-kolla | 03:27 | |
| *** stefan_bo has quit IRC | 03:34 | |
| *** eaguilar has quit IRC | 03:37 | |
| *** manheim has joined #openstack-kolla | 03:38 | |
| *** chas has joined #openstack-kolla | 03:40 | |
| *** manheim has quit IRC | 03:43 | |
| *** chas has quit IRC | 03:44 | |
| duonghq | thank inc0 | 03:47 |
| duonghq | inc0, first 2 ps in 3 are intended to make default roles work (I mean the service which is enabled by default) | 03:48 |
| duonghq | I have plan to squash 1st and 2nd ps | 03:48 |
| duonghq | 3rd ps is for other roles :P | 03:48 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 03:52 |
| sdake | morning folks | 03:52 |
| sdake | evening | 03:52 |
| sdake | all the same :) | 03:52 |
| sdake | inc0 i had a look at your review - i think once Jeffrey4l 's comments are fixed, its GTG | 03:55 |
| *** sayantan_ has quit IRC | 03:56 | |
| *** skramaja has joined #openstack-kolla | 03:56 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 03:57 |
| *** sayantan_ has joined #openstack-kolla | 03:59 | |
| *** huzhengchuan_ has quit IRC | 04:03 | |
| *** huzhengchuan has quit IRC | 04:03 | |
| *** zhurong has quit IRC | 04:07 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Conditionalize pv creation https://review.openstack.org/468264 | 04:10 |
| *** huzhengchuan_ has joined #openstack-kolla | 04:10 | |
| *** huzhengchuan has joined #openstack-kolla | 04:10 | |
| *** manheim has joined #openstack-kolla | 04:14 | |
| *** chas has joined #openstack-kolla | 04:16 | |
| *** manheim has quit IRC | 04:18 | |
| *** huzhengchuan has quit IRC | 04:19 | |
| *** huzhengchuan_ has quit IRC | 04:20 | |
| *** chas has quit IRC | 04:20 | |
| *** unicell has joined #openstack-kolla | 04:22 | |
| *** mnasiadka has joined #openstack-kolla | 04:24 | |
| *** mnasiadka has quit IRC | 04:28 | |
| *** unicell has quit IRC | 04:29 | |
| *** ddurst has joined #openstack-kolla | 04:31 | |
| *** zhurong has joined #openstack-kolla | 04:45 | |
| *** unicell has joined #openstack-kolla | 04:47 | |
| *** manheim has joined #openstack-kolla | 04:47 | |
| *** manheim has quit IRC | 04:52 | |
| *** gfidente has quit IRC | 04:55 | |
| *** yuanying_ has quit IRC | 04:57 | |
| *** yuanying has joined #openstack-kolla | 04:57 | |
| *** janki has joined #openstack-kolla | 05:04 | |
| *** salv-orlando has joined #openstack-kolla | 05:06 | |
| *** chas has joined #openstack-kolla | 05:07 | |
| *** chas has quit IRC | 05:12 | |
| *** manheim has joined #openstack-kolla | 05:14 | |
| *** stefan_bo has joined #openstack-kolla | 05:16 | |
| *** manheim has quit IRC | 05:18 | |
| sdake | Jeffrey4l ping | 05:20 |
| *** yuanying has quit IRC | 05:23 | |
| *** chas has joined #openstack-kolla | 05:28 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 05:28 |
| *** yuanying has joined #openstack-kolla | 05:29 | |
| *** manheim has joined #openstack-kolla | 05:30 | |
| *** chas has quit IRC | 05:32 | |
| *** salv-orlando has quit IRC | 05:38 | |
| *** manheim has quit IRC | 05:39 | |
| *** manheim has joined #openstack-kolla | 05:40 | |
| *** mewald has joined #openstack-kolla | 05:42 | |
| *** pcaruana has joined #openstack-kolla | 05:43 | |
| *** manheim has quit IRC | 05:44 | |
| *** manheim has joined #openstack-kolla | 05:52 | |
| *** mnasiadka has joined #openstack-kolla | 05:58 | |
| *** mnasiadka has quit IRC | 05:59 | |
| *** yuanying has quit IRC | 06:02 | |
| *** chas has joined #openstack-kolla | 06:04 | |
| *** yuanying has joined #openstack-kolla | 06:05 | |
| Jeffrey4l | sdake, yep? | 06:08 |
| sdake | nm | 06:09 |
| *** chas has quit IRC | 06:09 | |
| *** manheim has quit IRC | 06:13 | |
| *** manheim has joined #openstack-kolla | 06:17 | |
| *** MarginHu has quit IRC | 06:18 | |
| *** manheim has quit IRC | 06:21 | |
| *** korzen has joined #openstack-kolla | 06:30 | |
| *** salv-orlando has joined #openstack-kolla | 06:32 | |
| *** chas has joined #openstack-kolla | 06:40 | |
| *** n0isyn0ise has quit IRC | 06:43 | |
| *** n0isyn0ise has joined #openstack-kolla | 06:44 | |
| *** chas has quit IRC | 06:45 | |
| *** manheim has joined #openstack-kolla | 06:53 | |
| *** unicell has quit IRC | 06:53 | |
| *** unicell has joined #openstack-kolla | 06:54 | |
| *** zhurong has quit IRC | 06:54 | |
| *** manheim has quit IRC | 06:57 | |
| *** zhurong has joined #openstack-kolla | 06:58 | |
| *** Serlex has joined #openstack-kolla | 07:00 | |
| openstackgerrit | jangseon ryu proposed openstack/kolla-ansible master: Support for increasing thread cache size in TCMalloc. https://review.openstack.org/468284 | 07:02 |
| *** salv-orl_ has joined #openstack-kolla | 07:04 | |
| *** jaosorior has joined #openstack-kolla | 07:06 | |
| *** salv-orlando has quit IRC | 07:06 | |
| *** shardy has joined #openstack-kolla | 07:07 | |
| *** sayantan_ has quit IRC | 07:08 | |
| *** manheim has joined #openstack-kolla | 07:14 | |
| *** chas has joined #openstack-kolla | 07:16 | |
| *** athomas has joined #openstack-kolla | 07:18 | |
| *** manheim has quit IRC | 07:18 | |
| *** chas has quit IRC | 07:20 | |
| *** sayantan_ has joined #openstack-kolla | 07:26 | |
| *** salv-orl_ has quit IRC | 07:27 | |
| *** manheim has joined #openstack-kolla | 07:27 | |
| *** egonzalez has joined #openstack-kolla | 07:27 | |
| *** manheim has quit IRC | 07:31 | |
| *** manheim has joined #openstack-kolla | 07:32 | |
| *** manheim has quit IRC | 07:32 | |
| *** manheim has joined #openstack-kolla | 07:37 | |
| *** manheim has quit IRC | 07:41 | |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** chas has joined #openstack-kolla | 07:52 | |
| *** openstackgerrit has joined #openstack-kolla | 07:55 | |
| openstackgerrit | Merged openstack/kolla-ansible master: gzip the ARA report after generating it https://review.openstack.org/468155 | 07:55 |
| *** chas has quit IRC | 07:56 | |
| *** jascott1 has joined #openstack-kolla | 07:59 | |
| *** tvignaud_pto has quit IRC | 08:02 | |
| *** gfidente has joined #openstack-kolla | 08:04 | |
| *** gfidente has quit IRC | 08:04 | |
| *** gfidente has joined #openstack-kolla | 08:04 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla master: Fix the read dnsmasq log permission issue for fluentd https://review.openstack.org/468307 | 08:05 |
| *** manheim has joined #openstack-kolla | 08:13 | |
| *** tvignaud_pto has joined #openstack-kolla | 08:15 | |
| *** manheim has quit IRC | 08:18 | |
| *** skramaja is now known as skramaja_afk | 08:19 | |
| *** mgoddard has joined #openstack-kolla | 08:23 | |
| *** manheim has joined #openstack-kolla | 08:27 | |
| *** chas has joined #openstack-kolla | 08:28 | |
| *** pbourke has quit IRC | 08:30 | |
| *** pbourke has joined #openstack-kolla | 08:30 | |
| *** salv-orlando has joined #openstack-kolla | 08:31 | |
| *** manheim has quit IRC | 08:31 | |
| *** chas has quit IRC | 08:32 | |
| *** manheim has joined #openstack-kolla | 08:35 | |
| *** skramaja_afk is now known as skramaja | 08:36 | |
| *** manheim has quit IRC | 08:39 | |
| *** manheim has joined #openstack-kolla | 08:39 | |
| *** manheim has quit IRC | 08:43 | |
| *** manheim has joined #openstack-kolla | 08:44 | |
| *** manheim has quit IRC | 08:45 | |
| *** manheim has joined #openstack-kolla | 08:49 | |
| *** chas has joined #openstack-kolla | 08:50 | |
| *** manheim has quit IRC | 08:54 | |
| *** imcsk8_ has quit IRC | 08:56 | |
| *** jascott1 has quit IRC | 08:56 | |
| *** jascott1 has joined #openstack-kolla | 08:57 | |
| *** chas has quit IRC | 08:59 | |
| *** jascott1 has quit IRC | 09:02 | |
| *** spsurya has quit IRC | 09:08 | |
| *** imcsk8 has joined #openstack-kolla | 09:13 | |
| *** zhurong has quit IRC | 09:19 | |
| *** sambetts|afk is now known as sambetts | 09:20 | |
| *** rmart04 has joined #openstack-kolla | 09:21 | |
| *** manheim has joined #openstack-kolla | 09:25 | |
| *** haplo37 has quit IRC | 09:29 | |
| hrw | morning | 09:29 |
| *** manheim has quit IRC | 09:30 | |
| *** yingjun has quit IRC | 09:30 | |
| *** lpetrut has joined #openstack-kolla | 09:34 | |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Add option to enable ironic inspector LLDP collection https://review.openstack.org/468335 | 09:37 |
| *** haplo37 has joined #openstack-kolla | 09:39 | |
| *** tovin07_ has quit IRC | 09:48 | |
| *** manheim has joined #openstack-kolla | 09:48 | |
| *** kbaegis has joined #openstack-kolla | 09:52 | |
| kbaegis | Morning all | 09:52 |
| *** manheim has quit IRC | 09:52 | |
| *** stefan_bo has quit IRC | 09:54 | |
| *** sayantan_ has quit IRC | 09:55 | |
| *** MarginHu has joined #openstack-kolla | 09:59 | |
| MarginHu | hi | 09:59 |
| MarginHu | why kolla ask that disable libvirtd on the target deploy node ? | 10:00 |
| egonzalez | MarginHu, to avoid conflict between libvirt process launched by kolla and libvirt in the host | 10:00 |
| kbaegis | Yeah, because kolla manages that for you | 10:01 |
| kbaegis | egonzalez: Morning | 10:01 |
| MarginHu | but I want to run kolla in a kvm vm | 10:02 |
| kbaegis | MarginHu: You can run qemu. I don't know if KVM nesting is a good idea yet | 10:02 |
| hrw | MarginHu: so that kvm vm will be your target deploy node not it's host, right? | 10:02 |
| MarginHu | no, I want kolla is runing a node which is my target deploy node also. | 10:03 |
| MarginHu | I want kolla is runing in a node which is my target deploy node also. | 10:04 |
| MarginHu | kolla is a kvm vm. | 10:04 |
| kbaegis | MarginHu: https://docs.openstack.org/developer/kolla/mitaka/quickstart.html | 10:06 |
| kbaegis | Ha. So HAProxy is dependent on the sysctl interface? Dang. | 10:08 |
| *** daidv has quit IRC | 10:08 | |
| *** lucasxu has joined #openstack-kolla | 10:09 | |
| kbaegis | That python module won't build for me. "ctl_size" undefined | 10:11 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: gnocchi: take care of centos/source on non-x86 https://review.openstack.org/468348 | 10:13 |
| hrw | egonzalez, sdake, duonghq: https://review.openstack.org/468348 is simple to review - follows debian/ubuntu with installing blas-devel/lapack-devel for !x86 | 10:14 |
| *** MarginHu has quit IRC | 10:15 | |
| kbaegis | If anyone is willing to help, I can't build sysctl which is a requirement for haproxy's container: https://hastebin.com/xusadalule.lua | 10:15 |
| *** caowei has joined #openstack-kolla | 10:17 | |
| hrw | another day looking at http://people.linaro.org/~marcin.juszkiewicz/kolla/ and filling empty spaces | 10:21 |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Add option to enable ironic inspector LLDP collection https://review.openstack.org/468335 | 10:22 |
| *** manheim has joined #openstack-kolla | 10:26 | |
| *** ddyer2 has joined #openstack-kolla | 10:29 | |
| *** ddyer has quit IRC | 10:29 | |
| *** salv-orlando has quit IRC | 10:30 | |
| *** manheim has quit IRC | 10:30 | |
| *** salv-orlando has joined #openstack-kolla | 10:30 | |
| *** duonghq has quit IRC | 10:38 | |
| *** lucasxu has quit IRC | 10:40 | |
| *** cuongnv has quit IRC | 10:41 | |
| *** manheim has joined #openstack-kolla | 10:45 | |
| *** manheim has quit IRC | 10:55 | |
| *** mewald has quit IRC | 10:56 | |
| *** mewald has joined #openstack-kolla | 10:57 | |
| *** mewald has quit IRC | 10:58 | |
| *** manheim has joined #openstack-kolla | 10:59 | |
| *** manheim has quit IRC | 11:01 | |
| *** shardy has quit IRC | 11:03 | |
| *** mewald has joined #openstack-kolla | 11:07 | |
| *** manheim has joined #openstack-kolla | 11:09 | |
| *** rwsu has quit IRC | 11:11 | |
| *** manheim has quit IRC | 11:13 | |
| *** rhallisey has joined #openstack-kolla | 11:13 | |
| *** caowei has quit IRC | 11:13 | |
| *** salv-orlando has quit IRC | 11:18 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Use internal interface to connect with keystone_ssh https://review.openstack.org/466674 | 11:20 |
| *** rwsu has joined #openstack-kolla | 11:27 | |
| sdake | Jeffrey4l sturggling with this ansible: http://logs.openstack.org/31/464031/40/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/55495a1/console.html#_2017-05-26_06_06_32_788086 | 11:29 |
| openstackgerrit | James McCarthy proposed openstack/kolla-ansible master: Change default permissions in jinja templates. https://review.openstack.org/445797 | 11:29 |
| *** rwellum has joined #openstack-kolla | 11:33 | |
| *** skramaja has quit IRC | 11:37 | |
| *** powerd has joined #openstack-kolla | 11:39 | |
| hrw | http://people.linaro.org/~marcin.juszkiewicz/kolla/ got row with status numbers. aarch64/debian == x86-64/debian ;d | 11:42 |
| *** manheim has joined #openstack-kolla | 11:45 | |
| *** manheim has quit IRC | 11:49 | |
| *** zhubingbing has joined #openstack-kolla | 11:55 | |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant and provider VLANs https://review.openstack.org/464675 | 11:56 |
| *** masber has quit IRC | 12:00 | |
| *** spsurya has joined #openstack-kolla | 12:04 | |
| *** bmaltais_ has joined #openstack-kolla | 12:06 | |
| bmaltais_ | Good morning. I have a question regarding the removal of nodes from the inventory. | 12:06 |
| bmaltais_ | Is the kolla-ansible script handling it gracefully? I tried removing a Ceph node yesterday and nothing changed in the ceph ods list | 12:07 |
| bmaltais_ | The removed node was still listed and used even after running a deploy without the node in the inventory file. | 12:07 |
| bmaltais_ | I had to manually remove the osd, wait for rebalance of storage, delete the containers and finally drop the osd from ceph | 12:08 |
| bmaltais_ | Is it supposed to be that way? So each node type, when removed, require some manual reconfig to complete? | 12:08 |
| *** Pavo has joined #openstack-kolla | 12:09 | |
| bmaltais_ | I documented the steps... I should probably contribute it to the kolla-ansible operations document. | 12:10 |
| *** manheim has joined #openstack-kolla | 12:10 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Updated style of proposed changes for better readability. https://review.openstack.org/466881 | 12:12 |
| egonzalez | bmaltais, if remove a node from inventory openstack or ceph remains configured unless you destroy that node | 12:14 |
| *** manheim has quit IRC | 12:15 | |
| *** manheim has joined #openstack-kolla | 12:15 | |
| *** manheim has quit IRC | 12:20 | |
| *** ducttape_ has joined #openstack-kolla | 12:20 | |
| bmaltais_ | egonzalez: Morning ;-) What do you mean by "destroy"? Do you mean if it is unreachable because it is down? | 12:22 |
| bmaltais_ | egonzalez: Or by running the kolla-ansible destroy command on that node alone? | 12:22 |
| egonzalez | bmaltais, no no! | 12:23 |
| egonzalez | kolla-ansible desstroy use nodes from inventory | 12:23 |
| egonzalez | if no limit to which nodes what to destroy will remove the whole cluster | 12:23 |
| bmaltais_ | egonzalez: That is what I tought. So kolla-ansible destroy is not the way to do it then | 12:24 |
| bmaltais_ | egonzalez: What is the --limit flag is use just for that node? | 12:24 |
| egonzalez | bmaltais, you can do it by: kolla-ansible destroy --limit <node> | 12:24 |
| bmaltais_ | egonzalez: OK. And that will take care of running the ceph commands to remove the osd from ceph as well? | 12:25 |
| egonzalez | or executing on each node the scripts at tools folder (cleanup-host, cleanup-containers, cleanup-images) | 12:25 |
| egonzalez | bmaltais, that no | 12:25 |
| *** ducttape_ has quit IRC | 12:25 | |
| egonzalez | you'll need to remove the osds from ceph | 12:25 |
| *** ducttape_ has joined #openstack-kolla | 12:26 | |
| bmaltais_ | egonzalez: OK. So manually running all the ceph osd commands is still required. I am writing an ops guide for a few students to learn and I am trying to get 80% of it right ;-) | 12:26 |
| bmaltais_ | egonzalez: Leaving them the other 20% to figure out | 12:26 |
| *** mewald has quit IRC | 12:27 | |
| egonzalez | hehe, yep, those scripts will only remove containers, fstab entries, etc. but from the cluster | 12:27 |
| *** dciabrin has quit IRC | 12:27 | |
| bmaltais_ | egonzalez: Then run the destroy --limit node once ceph osd are cleanly removed. | 12:27 |
| egonzalez | yep, by node mean <inventory_hostname> | 12:28 |
| bmaltais_ | egonzalez: Would you say kolla-ansible is ready from light production use? Do you find it stable enough for that? | 12:28 |
| *** chas has joined #openstack-kolla | 12:28 | |
| egonzalez | bmaltais, i have it ;) | 12:28 |
| bmaltais_ | egonzalez: I would like to get the students to build an openstack clound to offer lab services | 12:28 |
| bmaltais_ | egonzalez: Great. Sound like it is doable then | 12:29 |
| bmaltais_ | s/clound/cloud | 12:29 |
| *** chas has quit IRC | 12:33 | |
| *** zhubingbing has quit IRC | 12:35 | |
| *** mewald has joined #openstack-kolla | 12:35 | |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Remove inspector files and configs from ironic-conductor https://review.openstack.org/468403 | 12:37 |
| *** mewald1 has joined #openstack-kolla | 12:40 | |
| *** ducttape_ has quit IRC | 12:40 | |
| *** ducttape_ has joined #openstack-kolla | 12:40 | |
| *** mewald has quit IRC | 12:41 | |
| *** ducttape_ has quit IRC | 12:42 | |
| *** powerd has quit IRC | 12:42 | |
| *** manheim has joined #openstack-kolla | 12:44 | |
| bmaltais_ | egonzalez: So there is no point in actually re-running a kolla-ansible deploy after removing the node from the inventory file? | 12:46 |
| egonzalez | bmaltais, if re run deploy without the node in the inventory ansible dont know about that node | 12:46 |
| bmaltais_ | egonzalez: Just destroying the node (if you want to deploy it again as something else) and removing it fro the inventory is enough? | 12:47 |
| *** powerd has joined #openstack-kolla | 12:47 | |
| egonzalez | what is going to happen is, for list of hosts as rabbitmq connection, is that config files wont have that node in the list | 12:47 |
| bmaltais_ | egonzalez: But essentially it does not reconfigure the other nodes to let them know it was removed. That reconfiguration is left to the operator | 12:47 |
| egonzalez | bmaltais, ansible will only remove the node from lists where before appear, but some services need to remove internally as ceph | 12:48 |
| *** manheim has quit IRC | 12:49 | |
| bmaltais_ | egonzalez: So there is still some benefit of doing it then just to push back config on the other nodes that no longer include the removed node? | 12:49 |
| bmaltais_ | egonzalez: Or it is just a waste of time as it does not really impact the running environment? Just trying to establish best ops practices to remove nodes | 12:50 |
| egonzalez | in the case of a compute node, nova internally still discovering the node through rpc | 12:50 |
| bmaltais_ | egonzalez: And need some cleanup following something like: https://ask.openstack.org/en/question/97941/how-to-remove-compute-host-in-controller-unfortunately-i-removed-that-host-but-name-still-available/ | 12:52 |
| egonzalez | computes are the easiest example: after migrate all instances, destroy the node with kolla-ansible destroy --limit and then remove the service from nova internals (manually) | 12:52 |
| bmaltais_ | egonzalez: But no real need to do a "deploy" without the node in the inventory file as it does not really make a real difference. | 12:53 |
| *** manheim has joined #openstack-kolla | 12:53 | |
| *** fguillot has joined #openstack-kolla | 12:53 | |
| egonzalez | in the case of a compute node not(if is not shared by other service ofc) | 12:53 |
| egonzalez | but if remove a rabbit, all openstack services have a setting: transport_url: rabbitq, rabbit2, etc | 12:54 |
| bmaltais_ | egonzalez: What would you do in your prod environment if you had to remove a compute node? I think rabitmq is deploy when you tag a node as compute | 12:55 |
| egonzalez | bmaltais, migrate all instances, disable the node at nova's services, shuttoff the node(or destroy) and remove the compute entry from nova services | 12:58 |
| *** powerd has quit IRC | 12:59 | |
| *** manheim has quit IRC | 13:03 | |
| *** eanylin has quit IRC | 13:06 | |
| *** korzen is now known as rootkowski | 13:07 | |
| *** dims has quit IRC | 13:07 | |
| *** eaguilar has joined #openstack-kolla | 13:07 | |
| *** rootkowski is now known as korzen | 13:07 | |
| *** srwilkers has joined #openstack-kolla | 13:09 | |
| *** rhallisey has quit IRC | 13:13 | |
| *** lucasxu has joined #openstack-kolla | 13:14 | |
| *** shardy has joined #openstack-kolla | 13:18 | |
| *** Pavo has quit IRC | 13:19 | |
| *** Pavo has joined #openstack-kolla | 13:20 | |
| *** korzen has quit IRC | 13:20 | |
| *** srwilkers has quit IRC | 13:21 | |
| *** srwilkers has joined #openstack-kolla | 13:22 | |
| *** eanylin has joined #openstack-kolla | 13:25 | |
| *** Pavo has quit IRC | 13:25 | |
| *** srwilkers has quit IRC | 13:28 | |
| *** ducttape_ has joined #openstack-kolla | 13:28 | |
| openstackgerrit | Juan J. MartÃnez proposed openstack/kolla master: Fix sed replacement command in Ubuntu's install_docker https://review.openstack.org/468026 | 13:29 |
| *** ducttap__ has joined #openstack-kolla | 13:30 | |
| *** srwilkers has joined #openstack-kolla | 13:30 | |
| *** ducttape_ has quit IRC | 13:33 | |
| *** inc0 has quit IRC | 13:35 | |
| *** Pavo has joined #openstack-kolla | 13:37 | |
| *** cristicalin has joined #openstack-kolla | 13:37 | |
| *** ducttap__ has quit IRC | 13:38 | |
| *** jtriley has joined #openstack-kolla | 13:39 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix unrecognized --limit option https://review.openstack.org/468423 | 13:40 |
| *** ducttape_ has joined #openstack-kolla | 13:43 | |
| *** stefan_bo has joined #openstack-kolla | 13:51 | |
| *** emccormick has quit IRC | 13:53 | |
| *** stefan_bo has quit IRC | 13:55 | |
| kbaegis | Hey guys, so I can't get the python sysctl module built. I can disable haproxy. Then I run into this problem: https://hastebin.com/hibiconizi.vbs | 13:56 |
| kbaegis | Anyone know why kolla-toolbox would be handing me a Type error? | 13:56 |
| *** iniazi has quit IRC | 13:58 | |
| *** jascott1 has joined #openstack-kolla | 14:00 | |
| *** emccormick has joined #openstack-kolla | 14:01 | |
| *** manheim has joined #openstack-kolla | 14:03 | |
| *** dims has joined #openstack-kolla | 14:04 | |
| *** jascott1 has quit IRC | 14:04 | |
| openstackgerrit | Merged openstack/kolla master: use proper package name for python-oslo.vmware https://review.openstack.org/467521 | 14:04 |
| *** haplo37 has quit IRC | 14:05 | |
| *** jtriley has quit IRC | 14:05 | |
| *** powerd has joined #openstack-kolla | 14:06 | |
| kbaegis | Is there any documentation specifically on kolla-toolbox? | 14:12 |
| kbaegis | i.e. - how to run commands manually? | 14:12 |
| kbaegis | Or troubleshooting syntax in commands? | 14:12 |
| egonzalez | kbaegis, there is no document about that afik | 14:16 |
| egonzalez | kbaegis, some time ago opened a blueprint to document command line usage, maybe ansible modules can be there too https://blueprints.launchpad.net/kolla-ansible/+spec/document-comand-line-usage | 14:17 |
| kbaegis | Would be super helpful since kolla_toolbox is having a parsing issue or something: https://hastebin.com/hibiconizi.vbs | 14:17 |
| kbaegis | egonzalez: Great blueprint. Hope it gets accepted | 14:17 |
| manheim | yes, documentation on cmd usage would be nice | 14:18 |
| *** salv-orlando has joined #openstack-kolla | 14:19 | |
| *** kbyrne has joined #openstack-kolla | 14:19 | |
| manheim | egonzalez, which is the way to access the Hyperv IP on which I connect on? | 14:21 |
| *** jascott1 has joined #openstack-kolla | 14:21 | |
| manheim | is related bindaddr = {{ hostvars[inventory_hostname]['ansible_ip_addresses'][0] }} | 14:21 |
| manheim | i cannot user api_interface_address since this is on linux deployer side, or? | 14:21 |
| egonzalez | manheim, I mean, how to know that IP [0] on the Ip addresses list is the correct one | 14:22 |
| egonzalez | manheim, is there a way to retrieve the IP based on interface? | 14:23 |
| manheim | yes, you are true, i can have multiple nics, i understand the issue | 14:23 |
| kbaegis | It would be at least nice if ansible.log had some useful information. Kolla toolbox is failing the mariadb user create step and it doesn't give me any useful information to troubleshoot | 14:23 |
| manheim | well, i know the IP, i it in the inventori file | 14:23 |
| manheim | is there a way to retrieve it from there? | 14:24 |
| *** hongbin has joined #openstack-kolla | 14:24 | |
| egonzalez | if is a hostname or fqdn in inventory? | 14:24 |
| manheim | Ip | 14:24 |
| *** manheim has quit IRC | 14:24 | |
| *** iniazi has joined #openstack-kolla | 14:25 | |
| *** manheim has joined #openstack-kolla | 14:25 | |
| *** jascott1 has quit IRC | 14:25 | |
| *** srwilkers has quit IRC | 14:26 | |
| *** manheim_ has joined #openstack-kolla | 14:28 | |
| *** manheim_ has quit IRC | 14:29 | |
| *** manheim_ has joined #openstack-kolla | 14:29 | |
| *** manheim has quit IRC | 14:29 | |
| *** salv-orl_ has joined #openstack-kolla | 14:30 | |
| rwellum | ping sdake | 14:30 |
| *** salv-orlando has quit IRC | 14:32 | |
| rwellum | sdake sbezberk: I'm adding these micro-services as per the deployment guide but they now seem to be installed as part of a service chart. https://www.irccloud.com/pastebin/iMceqonS/ | 14:32 |
| *** manheim_ has quit IRC | 14:33 | |
| *** manheim has joined #openstack-kolla | 14:35 | |
| *** sbezverk has quit IRC | 14:37 | |
| *** jamesbenson has joined #openstack-kolla | 14:38 | |
| *** manheim has quit IRC | 14:40 | |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Remove inspector files and configs from ironic-conductor https://review.openstack.org/468403 | 14:40 |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Remove inspector files and configs from ironic-conductor https://review.openstack.org/468403 | 14:42 |
| *** manheim has joined #openstack-kolla | 14:50 | |
| *** jtriley has joined #openstack-kolla | 14:50 | |
| *** spsurya has quit IRC | 14:51 | |
| pbourke | egonzalez: can you please +a this https://review.openstack.org/#/c/463798/ | 14:53 |
| *** ducttape_ has quit IRC | 14:53 | |
| egonzalez | pbourke, sure, forgot to -a once gates passed | 14:54 |
| pbourke | thanks | 14:54 |
| *** glycerine has joined #openstack-kolla | 14:55 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Kibana post_config checks only need to run locally https://review.openstack.org/463798 | 14:57 |
| *** lpetrut has quit IRC | 14:57 | |
| *** dciabrin has joined #openstack-kolla | 14:57 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Implement blazar Docker images https://review.openstack.org/454745 | 14:58 |
| *** masuberu has joined #openstack-kolla | 14:58 | |
| *** manheim has quit IRC | 14:58 | |
| *** salv-orl_ has quit IRC | 14:59 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Implement blazar Docker images https://review.openstack.org/454745 | 15:00 |
| *** emccormick has quit IRC | 15:01 | |
| *** jamesbenson has quit IRC | 15:03 | |
| *** shaggydog97 has joined #openstack-kolla | 15:05 | |
| *** salv-orlando has joined #openstack-kolla | 15:07 | |
| sdake | morning pbourke | 15:07 |
| sdake | pbourke LTNS ! | 15:07 |
| sdake | sup rwellum | 15:07 |
| pbourke | hey sdake | 15:07 |
| sdake | rwellum those two microcharts wer eintegrated into the nova service charts | 15:07 |
| sdake | rwellum i'm not totally convinced it works integrated the way it is - as some ordering must be done wrt cell launching | 15:07 |
| sdake | rwellum and i'm not sure the gate is confirming the implementaiton | 15:08 |
| *** janki has quit IRC | 15:08 | |
| sdake | rwellum in the short term you can turn off the cells config and run asy ou nomrally do | 15:08 |
| *** rhallisey has joined #openstack-kolla | 15:09 | |
| *** sbezverk has joined #openstack-kolla | 15:10 | |
| sbezverk | rwellum: right we added recently to service charts | 15:10 |
| *** powerd has quit IRC | 15:13 | |
| kbaegis | Is there a way to get the ansible scripts that kolla-toolbox gets passed? | 15:16 |
| kbaegis | it's passing "EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1495811694.7199917-186648925687992/kolla_toolbox.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1495811694.7199917-186648925687992/" > /dev/null 2>&1 && sleep 0'" which is annoying because I can't read the file | 15:16 |
| *** srwilkers has joined #openstack-kolla | 15:17 | |
| kbaegis | to figure out why no output is being passed to it | 15:18 |
| *** mewald1 has quit IRC | 15:20 | |
| shaggydog97 | hello all, noob here having issues deploying 4.0.1 clean install when building the mariadb container… build fails on calling mysql_install_db https://github.com/openstack/kolla/blob/master/docker/mariadb/extend_start.sh#L55. | 15:22 |
| shaggydog97 | Seems it can’t find mysql_install_db | 15:22 |
| *** sayantan_ has joined #openstack-kolla | 15:23 | |
| *** powerd has joined #openstack-kolla | 15:24 | |
| *** cristicalin has quit IRC | 15:25 | |
| *** manheim has joined #openstack-kolla | 15:30 | |
| rwellum | Ah sbezverk sdake - how do I keep up with that kind of thing? Was it an obvious ps I missed? | 15:32 |
| kfox1111 | morning. | 15:33 |
| kbaegis | morning | 15:34 |
| *** manheim has quit IRC | 15:34 | |
| *** egonzalez has quit IRC | 15:35 | |
| sdake | rwellum it was int he review queue - i dont htink it was anythign obvious you missed | 15:35 |
| sdake | rwellum if you missed the review you didn't know about the change | 15:36 |
| rwellum | ok sdake | 15:36 |
| *** Serlex has quit IRC | 15:38 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Fix "key_repository is world readable: /etc/keystone/fernet-keys/" warning https://review.openstack.org/468452 | 15:42 |
| *** egonzalez has joined #openstack-kolla | 15:42 | |
| sean-k-mooney | sdake: on turning off cells that is not surpport by nova after ocata correct. there intentio was to makes cells v2 always required going forworad | 15:42 |
| egonzalez | sean-k-mooney yep | 15:42 |
| egonzalez | placement api and cells v2 are required | 15:43 |
| sean-k-mooney | yes in netwon placement was optional in ocata+ placement and cells v2 are requred as all deployment are now cells deployment of 1+ cells | 15:44 |
| sean-k-mooney | the intention being that if you want to 2+ cells and you have a cells deployment of 1 cell its a lot eaisre then if you dont have any cells | 15:45 |
| *** janki has joined #openstack-kolla | 15:46 | |
| sean-k-mooney | egonzalez: oh by the way the mariadb change i made can we backport that to ocata? | 15:46 |
| sean-k-mooney | stable ocata and 3.0.3 fail to build without that chage | 15:46 |
| egonzalez | yes | 15:47 |
| sean-k-mooney | should i cherrypick it for the ui or will i leave that to the core/stable team to do? | 15:47 |
| kbaegis | looks like ansible prechecks have undocumented dependencies on one of: weakrefset weakreflist weakref_property | 15:48 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla stable/ocata: pin mariadb version for rpm distros https://review.openstack.org/468454 | 15:48 |
| sean-k-mooney | egonzalez: thanks :) | 15:49 |
| kbaegis | was breaking prechecks; can someone please add the pertinent one to requirements? | 15:49 |
| *** powerd has quit IRC | 15:53 | |
| rwellum | sdake: my nova api cell restarted 5x before it was stable - with this changed service chart. | 16:04 |
| shaggydog97 | egonzalez: your proposed change just fixed my mariadb issue… thank you for this | 16:05 |
| *** inc0 has joined #openstack-kolla | 16:05 | |
| inc0 | hey | 16:05 |
| kfox1111 | hi | 16:05 |
| inc0 | comcast can't report my outage because...they have outage;) | 16:06 |
| inc0 | I found that amusing | 16:06 |
| sbezverk | kfox1111 inc0 good morning | 16:06 |
| kfox1111 | hehe | 16:06 |
| kfox1111 | sbezverk: hey. :) | 16:06 |
| kfox1111 | sbezverk: found out someting interetsting this morning. | 16:07 |
| kfox1111 | a svc's targetPort can be a string, that references a named port in a pod. | 16:07 |
| kfox1111 | so the port numbers don't always have to match up between all pods in an svc. | 16:07 |
| sbezverk | kfox1111 right I saw some yaml files where they use HTTP instead of 80 | 16:09 |
| kfox1111 | yeah. | 16:10 |
| *** manheim has joined #openstack-kolla | 16:12 | |
| *** japestinho has joined #openstack-kolla | 16:14 | |
| *** manheim has quit IRC | 16:17 | |
| *** mewald has joined #openstack-kolla | 16:19 | |
| *** jascott1 has joined #openstack-kolla | 16:22 | |
| *** jascott1 has quit IRC | 16:26 | |
| *** unicell has quit IRC | 16:28 | |
| *** chas has joined #openstack-kolla | 16:28 | |
| *** david-lyle has joined #openstack-kolla | 16:30 | |
| sdake | rwellum ya i think it is flakey tbh :) | 16:32 |
| *** chas has quit IRC | 16:33 | |
| sdake | kfox1111 re log forwarding - proceeding on that front | 16:34 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Fix "key_repository is world readable: /etc/keystone/fernet-keys/" warning https://review.openstack.org/468452 | 16:34 |
| sdake | kfox1111 I think getting fluentd enabled is fairly easy and think that owuld give us alot of value | 16:34 |
| sdake | kfox1111 any ojections? | 16:34 |
| kfox1111 | sdake: there are kind of two path forwards there... | 16:35 |
| *** mgoddard has quit IRC | 16:35 | |
| sdake | shoot me :) | 16:35 |
| *** eaguilar has quit IRC | 16:35 | |
| kfox1111 | one is to change all the services to just export to the docker logs. | 16:35 |
| kfox1111 | I have a patch set up for that. | 16:35 |
| sdake | i've seen it | 16:35 |
| kfox1111 | then a log shipper like: | 16:35 |
| kfox1111 | https://github.com/kubernetes/charts/pull/895 | 16:36 |
| kfox1111 | can pull them from docker/push them up wherever they need to go. | 16:36 |
| kfox1111 | the other route is kind of whats half implemented already. side car fluentd/fluent-bit and grab/ship the logs. | 16:36 |
| kfox1111 | it still needs a log roller/deleter though added to function properly. | 16:36 |
| kfox1111 | the formers very simple, which I like. today its less flexible though, but talking with the dev to make it just as flexible. | 16:37 |
| kbaegis | Hey guys, should I open up a bug for this? https://paste.pound-python.org/show/OTDBCL5Er6VGWdeUR6dQ/ | 16:37 |
| kfox1111 | the latter is much more complicated but more flexible today. | 16:37 |
| kfox1111 | I'm kind of on the fence. thats why I started implementing the former, to see if it would fly at all. | 16:38 |
| kfox1111 | it seems like it all will work in the ned. | 16:38 |
| kfox1111 | end. | 16:38 |
| sdake | fluentd-bit is fluentd? | 16:38 |
| kfox1111 | fluent-bit is to fluentd what filebeat is to logstash. | 16:39 |
| sdake | i thought fluentd was a log forwarder? (td-agent) | 16:39 |
| kfox1111 | understand, or need more explienation? | 16:39 |
| *** kbyrne has quit IRC | 16:39 | |
| sdake | need more minerals plz | 16:39 |
| kfox1111 | all 4 there are log forwarders. | 16:40 |
| kfox1111 | all 4 can foward logs. | 16:40 |
| kfox1111 | but the split is between memory usage and flexibility. | 16:40 |
| kfox1111 | the former two are extremely light weight, but have limited processing capabilities. they basically just ship. | 16:40 |
| sdake | the former two being :) | 16:41 |
| kfox1111 | the latter two can slice/dice the data. but they are heavy weight processes. | 16:41 |
| sdake | sorry having trouble tracking the relationships here ;) | 16:41 |
| kfox1111 | fluent-bit and filebeat are basically pure shippers. | 16:41 |
| sdake | the light weight components are which? | 16:41 |
| sdake | good | 16:41 |
| sdake | lets go with those imo | 16:41 |
| kfox1111 | fluentd and logstash are logshippers+log processors. | 16:41 |
| kbaegis | not sure who wrote kolla-toolbox but that loop is broken for me | 16:41 |
| kfox1111 | generally what you do is, | 16:41 |
| *** gfidente has quit IRC | 16:42 | |
| kfox1111 | use the light weight ones everywhere, then ship them to a central fluentd/logstash for processing. | 16:42 |
| *** kbyrne has joined #openstack-kolla | 16:42 | |
| kfox1111 | so you often will use both. | 16:42 |
| kfox1111 | tree style. | 16:42 |
| kfox1111 | fluent-bit on the edges, fluentd at the trunk. | 16:42 |
| *** jascott1 has joined #openstack-kolla | 16:44 | |
| *** inc01 has joined #openstack-kolla | 16:45 | |
| *** inc0 has quit IRC | 16:45 | |
| *** sambetts is now known as sambetts|afk | 16:46 | |
| bmaltais_ | Can someone help push this along: https://review.openstack.org/#/c/467695/ | 16:46 |
| *** inc01 has quit IRC | 16:47 | |
| bmaltais_ | I made a small change after the LGTM message and since then it is stuck at +1 | 16:47 |
| *** inc0 has joined #openstack-kolla | 16:47 | |
| sdake | kfox1111 what is needed to make the fluentd-bit stuff a reality? | 16:48 |
| sdake | kfox1111 need any hlep on the work? | 16:48 |
| bmaltais_ | Dang, sorry this is the change with the LGTM: https://review.openstack.org/#/c/466881/ | 16:48 |
| *** manheim has joined #openstack-kolla | 16:49 | |
| *** jaosorior has quit IRC | 16:49 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Fix "key_repository is world readable: /etc/keystone/fernet-keys/" warning https://review.openstack.org/468452 | 16:51 |
| kfox1111 | sdake: I think just that one ps needs completing. | 16:51 |
| kfox1111 | if you have time to take it over, go for it. :) | 16:51 |
| sdake | kfox1111 link plz | 16:51 |
| kfox1111 | we just need a way to make all lot files go out to stderr/out. | 16:51 |
| sdake | kfox1111 i may not take it over but find someone to do so :) | 16:52 |
| kfox1111 | https://review.openstack.org/#/c/458649/ | 16:52 |
| kfox1111 | that works too. | 16:52 |
| *** manheim has quit IRC | 16:53 | |
| sdake | kfox1111 in this fluentd-bit model, there is no sidecar? | 16:53 |
| sdake | only one copy of fluentd-bit? | 16:53 |
| kfox1111 | yeah. daemonset on the host. | 16:53 |
| *** ducttape_ has joined #openstack-kolla | 16:54 | |
| sdake | how doth it read /var/log/kolla - moutned to each pod? | 16:54 |
| sdake | oh i guess it reads stdout? | 16:54 |
| kfox1111 | it doth not. :) | 16:55 |
| kfox1111 | right. | 16:55 |
| sdake | does it read *all* stdout or just some of it? | 16:55 |
| kfox1111 | docker logs -f | 16:55 |
| sdake | right - so what does fluentd pick up? | 16:55 |
| kfox1111 | or kubectl logs | 16:55 |
| sdake | rather fluentd-bit? | 16:55 |
| *** unicell has joined #openstack-kolla | 16:55 | |
| kfox1111 | evertyhing spit out by the container on stderr/out. | 16:55 |
| sdake | how does it know which containers to select | 16:55 |
| kfox1111 | its k8s aware | 16:56 |
| sdake | select by namespace? | 16:56 |
| kfox1111 | http://fluentbit.io/kubernetes/ | 16:56 |
| sdake | reading brb | 16:56 |
| kfox1111 | I tested it with docker log rolling enabled, and it worked. so it should be a pretty easy to implement solution. | 16:57 |
| *** david-lyle has quit IRC | 16:58 | |
| sdake | i read the docs and clicke dthrough to the rest of the docs | 16:58 |
| sdake | assuming i have a million other htings runnign on my kubernetes cluster | 16:58 |
| sdake | it looks like this picks them all up? | 16:58 |
| *** itlinux has joined #openstack-kolla | 16:58 | |
| kfox1111 | yup. | 16:59 |
| kfox1111 | log ships all the things. | 16:59 |
| kfox1111 | the dev's pretty responsive. so if you wanted to have something like a pod attribute to tell it not to ship something, or other such features, it shouldn't be too hard I think? | 16:59 |
| sdake | select by namespace? | 17:00 |
| sdake | its good to hear the amin dev is respoive | 17:00 |
| kfox1111 | I asked for a feature, | 17:00 |
| kfox1111 | let each namespace have its own fluentd, and ship to that. | 17:00 |
| *** unicell has quit IRC | 17:00 | |
| kfox1111 | and a followon, leta pod specifiy a named fluentd svc. | 17:00 |
| openstackgerrit | Christian Berendt proposed openstack/kolla master: In set_configs.py support perm parameter in permissions section https://review.openstack.org/468472 | 17:00 |
| kfox1111 | so your namespace could have multipple fluentd's, and the pods could get their data routed to a particular set of them. | 17:01 |
| sdake | you asked for that feature where? | 17:02 |
| *** jascott1 has quit IRC | 17:02 | |
| kfox1111 | https://github.com/fluent/fluent-bit/issues/242 | 17:02 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 17:03 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Fix "key_repository is world readable: /etc/keystone/fernet-keys/" warning https://review.openstack.org/468452 | 17:04 |
| kbaegis | Nice. A hidden error I've discovered: https://paste.pound-python.org/show/j6sW4UhlSFbVj9Ti0Afz/ | 17:04 |
| kbaegis | re is failing | 17:04 |
| kbaegis | Anyone know if re is built in? | 17:06 |
| *** aagate has quit IRC | 17:08 | |
| sdake | kfox1111 your 3rd comment would allow tagging the raw logs per namespace? | 17:08 |
| sdake | (with fluentd magic) | 17:08 |
| kfox1111 | yes. | 17:09 |
| kfox1111 | more multitenant friendly. | 17:09 |
| sdake | i understand that part- i mean break apart the log item and generate es like entires that are generated today in kolla-ansible | 17:10 |
| kfox1111 | right now, in the docker shipped logs method, all the logs from all namespaces go to the same place. | 17:10 |
| kfox1111 | yeah. | 17:10 |
| sdake | and that component doing that parsing of the logs would be a fluentd container with approirate configuration | 17:10 |
| kfox1111 | yeah. | 17:10 |
| sdake | the fluentd bits are probably mostly reusable form kolla-ansible if 242 were to be implemented and merged? | 17:11 |
| kfox1111 | possibly, but no k8s there, so only would be able to get raw docker logs, not docker logs with additional attributes like with k8s. | 17:12 |
| *** athomas has quit IRC | 17:12 | |
| kfox1111 | not sure if those attributes wil be needed by fluentd to decide how to filter the logs best. | 17:13 |
| sdake | kolla-ansible breaks apart a log entry into process, error level, etc | 17:13 |
| sdake | it does htis with fluentd and then forwards the dictionary to ES | 17:14 |
| kfox1111 | that part would be no different I think. | 17:14 |
| sdake | sweet | 17:14 |
| sdake | so a path forward woudl be: | 17:15 |
| sdake | get 242 implemented | 17:15 |
| kfox1111 | well, except whats doing the breaking. | 17:15 |
| sdake | get above review implemented | 17:15 |
| kfox1111 | 242 doesn't have to be implmeneted to move forward. | 17:15 |
| *** ipsecguy has quit IRC | 17:15 | |
| kfox1111 | most clouds will be the only tenant on their cloud so user=admin. | 17:15 |
| *** ipsecguy has joined #openstack-kolla | 17:15 | |
| sdake | sorry - to have equiv fuciotnlaity to kolla-ansible | 17:15 |
| kfox1111 | still equiv without it. | 17:16 |
| kfox1111 | 242's really about multitenancy. | 17:16 |
| sdake | it implies adding fluentd | 17:16 |
| sdake | I guess we could add fluentd without it? | 17:16 |
| kfox1111 | it has fluentd support. | 17:16 |
| kfox1111 | 242 adds multiple fluentd support. | 17:16 |
| sdake | trying to picture a path where parsed logs get into ES | 17:16 |
| kfox1111 | keyed off of namespace. | 17:16 |
| kfox1111 | docker container spits crap out to stderr/out. | 17:17 |
| openstackgerrit | Merged openstack/kolla master: Fix sed replacement command in Ubuntu's install_docker https://review.openstack.org/468026 | 17:17 |
| kfox1111 | fluent-bit picks it up, adds k8s metadata, forwards it to fluentd. | 17:17 |
| kfox1111 | fluentd does whatever additional manipulations that are nessisary for ease of use, fowards it to es. | 17:17 |
| *** rmart04 has quit IRC | 17:17 | |
| kfox1111 | kibana displays stuff in es. | 17:17 |
| kfox1111 | proffit. :) | 17:18 |
| sdake | god i wish I could :) | 17:18 |
| *** unicell has joined #openstack-kolla | 17:18 | |
| bmaltais_ | In my kolla-ansible ceph deployment I get: HEALTH_WARN; too many PGs per OSD (616 > max 300) | 17:18 |
| kfox1111 | all the components exist today. | 17:18 |
| kfox1111 | just need to be glued together. | 17:18 |
| bmaltais_ | Is this something that should be fixed or is normal for the caontainer deployment? | 17:19 |
| *** srwilkers has quit IRC | 17:21 | |
| *** manheim has joined #openstack-kolla | 17:25 | |
| *** manheim has quit IRC | 17:29 | |
| *** unicell has quit IRC | 17:30 | |
| *** salv-orlando has quit IRC | 17:30 | |
| *** salv-orlando has joined #openstack-kolla | 17:31 | |
| *** shaggydog97 has quit IRC | 17:34 | |
| *** ducttape_ has quit IRC | 17:34 | |
| *** salv-orlando has quit IRC | 17:35 | |
| *** powerd has joined #openstack-kolla | 17:35 | |
| kbaegis | Speaking of kolla ansible | 17:37 |
| egonzalez | bmaltais_ probably need to decrease pgs per ceph pool | 17:37 |
| kbaegis | sdake, could I get some help understanding the issue? | 17:37 |
| egonzalez | there is a change under review to make this configurable | 17:37 |
| *** ducttape_ has joined #openstack-kolla | 17:38 | |
| *** itlinux_ has joined #openstack-kolla | 17:38 | |
| kbaegis | line 124 of kolla_toolbox.py doesn't like the info passed to it by roles/mariadb/tasks/register.yml | 17:39 |
| *** manheim has joined #openstack-kolla | 17:39 | |
| kbaegis | TypeError | 17:39 |
| egonzalez | kbaegis never seen that error, what python, OS, kolla , ansible versions are you using? | 17:40 |
| *** powerd has quit IRC | 17:40 | |
| kbaegis | py3.4.5, gentoo, stable/ocata, and 2.3.0.0 | 17:41 |
| *** itlinux has quit IRC | 17:42 | |
| *** manheim has quit IRC | 17:43 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 17:44 |
| *** salv-orlando has joined #openstack-kolla | 17:44 | |
| kbaegis | I've tried venvs for py2.7 and 3.5 | 17:45 |
| kbaegis | Same TypeError on the re module | 17:45 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 17:46 |
| openstackgerrit | Numan Siddique proposed openstack/kolla master: Support openvswitch-ovn docker images https://review.openstack.org/459179 | 17:48 |
| kbaegis | I need to downgrade ansible, don't I? | 17:48 |
| *** jascott1 has joined #openstack-kolla | 17:51 | |
| egonzalez | kbaegis i have 2.3 too and working fine | 17:52 |
| *** stefan_bo has joined #openstack-kolla | 17:52 | |
| rwellum | I had to use 2.2 fyi - can't recall why | 17:52 |
| kbaegis | egonzalez: Huh. All of my python versions have the TypeError for byte/string confusion on the kolla_toolbox import | 17:53 |
| kbaegis | prechecks pass fine | 17:53 |
| *** itlinux_ has quit IRC | 17:53 | |
| kbaegis | https://hastebin.com/urenemetir.vbs | 17:55 |
| *** stefan_bo has quit IRC | 17:57 | |
| *** manheim has joined #openstack-kolla | 17:57 | |
| *** ducttap__ has joined #openstack-kolla | 17:59 | |
| *** pcaruana has quit IRC | 18:00 | |
| *** ducttape_ has quit IRC | 18:00 | |
| portdirect | o/ | 18:00 |
| portdirect | hey think im being daft, but I'm trying to add apache2 to the base image | 18:01 |
| *** manheim has quit IRC | 18:01 | |
| portdirect | I would have thought that the following snippet would have been the correct way to do this: | 18:01 |
| portdirect | https://www.irccloud.com/pastebin/44a1K3D8/template-overrides.j2 | 18:02 |
| portdirect | but think i must be going crazy? :) | 18:02 |
| inc0 | portdirect: loogs good | 18:03 |
| inc0 | what's the issue? | 18:03 |
| *** ducttape_ has joined #openstack-kolla | 18:04 | |
| portdirect | inc0: https://review.openstack.org/#/c/468506/ | 18:05 |
| portdirect | https://www.irccloud.com/pastebin/Qbhf8b7r/ | 18:06 |
| portdirect | and of course it worked that time | 18:06 |
| portdirect | lol fml :D | 18:06 |
| *** ducttap__ has quit IRC | 18:07 | |
| inc0 | btw apache2 I assume is for APIs? | 18:08 |
| *** rmart04 has joined #openstack-kolla | 18:08 | |
| sdake | kfox1111 re https://review.openstack.org/#/c/468264/ | 18:10 |
| sdake | sbezverk ^^ | 18:10 |
| sdake | not really sure what is needed in the code | 18:10 |
| sdake | sup portdirect | 18:10 |
| portdirect | inc0: yeah | 18:11 |
| portdirect | hey sdake o/ | 18:11 |
| inc0 | also, if you guys want to do what I think you're doing | 18:11 |
| inc0 | and also want to join OpenStack at some point, you might want to check up ML | 18:11 |
| inc0 | there is long discussion about pushing images to dockerhub | 18:11 |
| portdirect | oh nice - I'll check it out for sure | 18:13 |
| sbezverk | sdake: I think you need to remove variable from values.yaml and keep only one all_values.yaml | 18:13 |
| sbezverk | in this case already existing logic will check all levels until it hits default, which is enable pv creation. | 18:13 |
| *** jtriley has quit IRC | 18:15 | |
| sdake | so basically just add the conditon variable to common: | 18:15 |
| sdake | global: | 18:15 |
| sdake | kolla: | 18:15 |
| sdake | all: | 18:15 |
| sdake | and then have a condtion on the variable? | 18:15 |
| sbezverk | yes | 18:15 |
| sdake | cool | 18:15 |
| sdake | incoming | 18:15 |
| *** manheim has joined #openstack-kolla | 18:15 | |
| sbezverk | you might want to add this variable to gate's common_workflows yamls | 18:16 |
| sbezverk | so gate jobs would still work correctly.. | 18:16 |
| sdake | i plan to default it to true | 18:16 |
| sdake | is that still needed inthat case? | 18:17 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Conditionalize pv creation https://review.openstack.org/468264 | 18:18 |
| sbezverk | sdake: here https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/common_workflow_config.sh#L55 | 18:19 |
| sdake | sbezverk right -that is for overrides | 18:19 |
| sbezverk | in this case for will be 100% sure | 18:19 |
| sdake | if it defaults to true, that wouldn't be necessary? | 18:19 |
| *** manheim has quit IRC | 18:20 | |
| sbezverk | sdake: ok let's see | 18:20 |
| sdake | was a noob mistake putting them i the service charts | 18:20 |
| *** japestinho has quit IRC | 18:20 | |
| sdake | I forgot about that all_yamls thign | 18:20 |
| sdake | all values that is | 18:20 |
| *** ducttape_ has quit IRC | 18:24 | |
| kbaegis | inc0: Any clue on this one? https://paste.pound-python.org/show/j6sW4UhlSFbVj9Ti0Afz/ | 18:28 |
| inc0 | kbaegis: check versions of python | 18:28 |
| inc0 | I'm not sure if ansible is happy with py3 | 18:28 |
| inc0 | or hold on | 18:29 |
| inc0 | that's inside container | 18:29 |
| inc0 | that's wierd | 18:29 |
| inc0 | can you exec to this container and check version of python? | 18:29 |
| kbaegis | 2.7.12 | 18:30 |
| kbaegis | I added the stdout, so ignore that. The TypeError is what is actually broken | 18:31 |
| kbaegis | sounds like re.compile is failing on the string it's passed | 18:31 |
| kbaegis | But it's failing on kolla-toolbox.py line 124 | 18:32 |
| kbaegis | stable/ocata on gentoo | 18:32 |
| *** manheim has joined #openstack-kolla | 18:33 | |
| *** ducttape_ has joined #openstack-kolla | 18:35 | |
| *** ducttape_ has quit IRC | 18:35 | |
| *** jtriley has joined #openstack-kolla | 18:36 | |
| *** ducttape_ has joined #openstack-kolla | 18:36 | |
| *** powerd has joined #openstack-kolla | 18:36 | |
| *** Pavo has quit IRC | 18:37 | |
| *** manheim has quit IRC | 18:38 | |
| *** powerd has quit IRC | 18:42 | |
| *** janki has quit IRC | 18:43 | |
| *** fooliouno has joined #openstack-kolla | 18:45 | |
| *** manheim has joined #openstack-kolla | 18:47 | |
| Reepicheep | hmm. when running bootstrap-server on some new nodes I get - 'dict object' has no attribute 'ipv4' | 18:48 |
| Reepicheep | it's when it it generating the /etc/hosts | 18:48 |
| *** ducttap__ has joined #openstack-kolla | 18:49 | |
| Reepicheep | how is it determining which interface to use and lookup the ip address? | 18:49 |
| kbaegis | Reepicheep: I believe that's either in your inventory or in globals.yml | 18:49 |
| *** ducttap__ has quit IRC | 18:50 | |
| *** ducttap__ has joined #openstack-kolla | 18:51 | |
| *** Pavo has joined #openstack-kolla | 18:52 | |
| *** ducttape_ has quit IRC | 18:52 | |
| Reepicheep | yeah.. so it uses the "network_interface" .. I think I see my issue | 18:53 |
| inc0 | guys, can I have eyes on this? https://review.openstack.org/#/c/466007/ | 18:53 |
| Reepicheep | one of the nodes dosn't have an ip on team0.811 after it rebooted | 18:54 |
| Reepicheep | who needs networking anyway :| | 18:54 |
| inc0 | kbaegis: gentoo? man you're brave;) | 18:55 |
| *** manheim has quit IRC | 18:56 | |
| kbaegis | inc0: :) Still confused about the issue | 18:56 |
| inc0 | kbaegis: right, it's kolla-toolbox module | 18:57 |
| inc0 | so it's gonna be python vertsion on the host | 18:57 |
| inc0 | not in container | 18:57 |
| inc0 | can you check it for me please? | 18:57 |
| kbaegis | inc0: I've tried venvs for 2.7, 3.4 (system default), and 3.5 | 18:57 |
| inc0 | it's gonna run 3.4 | 18:58 |
| inc0 | and that's the issue | 18:58 |
| inc0 | thing is | 18:58 |
| inc0 | when ansible ssh to node, it will run default python | 18:58 |
| *** lucasxu has quit IRC | 18:58 | |
| inc0 | but you can change that in inventory | 18:58 |
| kbaegis | inc0: I can't install ansible in a venv? | 18:58 |
| inc0 | so ansible you call | 18:59 |
| kbaegis | I wanted a post-merge hook like this https://paste.pound-python.org/show/R7IWq4kK7Zab8fSxR7v6/ | 18:59 |
| inc0 | will do ssh user@host -C python some_ansible_module.py | 18:59 |
| inc0 | in short | 18:59 |
| inc0 | and when you do ssh, it will use default python on host it ssh to | 18:59 |
| inc0 | you can specify ansible_python_interpreter=/usr/local/bin/python2.7 in inventory to change that | 19:00 |
| kbaegis | tyvm. I also just ran eselect python set 2.7 | 19:01 |
| inc0 | check if it helps | 19:01 |
| *** dciabrin has quit IRC | 19:01 | |
| inc0 | let me know if it won't we'll get back to drawing board;0 | 19:01 |
| *** eaguilar has joined #openstack-kolla | 19:03 | |
| *** salv-orl_ has joined #openstack-kolla | 19:04 | |
| *** salv-orlando has quit IRC | 19:06 | |
| kbaegis | inc0: :D Thanks | 19:07 |
| kbaegis | (worked) | 19:07 |
| inc0 | it's funny when you need to thing of 3 different pythons involved;) | 19:07 |
| inc0 | one in venv, one on host and one inside container | 19:07 |
| *** jamesbenson has joined #openstack-kolla | 19:08 | |
| *** rmart04 has quit IRC | 19:09 | |
| *** manheim has joined #openstack-kolla | 19:10 | |
| *** dciabrin has joined #openstack-kolla | 19:11 | |
| kbaegis | inc0: So fun fact about gentoo/portage: pip doesn't play nice and overwrites important stuff. Since kolla needs pip so heavily, yes you need a venv if you're going to set this up in a CI/CD model | 19:13 |
| *** manheim has quit IRC | 19:14 | |
| *** lucasxu has joined #openstack-kolla | 19:14 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multi-scenerio gates https://review.openstack.org/466815 | 19:15 |
| *** eaguilar_ has joined #openstack-kolla | 19:17 | |
| *** powerd has joined #openstack-kolla | 19:17 | |
| kbaegis | inc0: (I've also had to rebuild my kernel twice while troubleshooting ;)) | 19:18 |
| inc0 | haha | 19:18 |
| inc0 | hmm, since we're getting close to long weekend, maybe I'll spend it installing gentoo | 19:19 |
| inc0 | oorrrr....http://www.linuxfromscratch.org/ | 19:19 |
| *** eaguilar has quit IRC | 19:20 | |
| kbaegis | Hey, whichever distributions help me stay away from the scourge that is systemd | 19:21 |
| Reepicheep | I should be able to override the "network_interface" per node in the inventory file correct? | 19:21 |
| *** salv-orl_ has quit IRC | 19:21 | |
| kbaegis | Reepicheep: Yes | 19:21 |
| openstackgerrit | Sven Anderson proposed openstack/kolla master: Add ec2-api image https://review.openstack.org/468518 | 19:21 |
| *** salv-orlando has joined #openstack-kolla | 19:21 | |
| Reepicheep | "nodehostname network_interface=bond0.811" does not seem to have any effect it still is trying to use the value for that host from the globals.xml | 19:22 |
| *** powerd has quit IRC | 19:23 | |
| Reepicheep | kbaegis: btw.. I should be a bit ashamed because of the age of the vm but I have at least one gentoo box that has been running services for over 12 years in production | 19:23 |
| *** manheim has joined #openstack-kolla | 19:24 | |
| Reepicheep | it has never broken and I have never got around to replacing it | 19:24 |
| kbaegis | Reepicheep: lol I had a friend/mentor who used to have uptime competitions with gentoo | 19:25 |
| Reepicheep | I'm betting I may have some nodes in the running :) | 19:25 |
| *** salv-orlando has quit IRC | 19:25 | |
| kbaegis | Reepicheep: properly maintained, it's way more stable | 19:25 |
| kbaegis | Reepicheep: /usr/share/kolla-ansible/ansible/inventory/multinode has some comment style documentation you should use when writing your inventory | 19:26 |
| egonzalez | Reepicheep there is an issue with vars override, vars in globals are ansible extra-vars with higher priority, for make it work need to put all custom vars at the inventory or use common at group_vars | 19:26 |
| Reepicheep | stability was never an issue when I used to put production stuff on gentoo .. the issue became managing the number of nodes | 19:27 |
| kbaegis | inc0: I love it. Now everything except haproxy is working. Had some issues installing sysctl module | 19:27 |
| *** ducttap__ has quit IRC | 19:27 | |
| *** manheim has quit IRC | 19:28 | |
| Reepicheep | yeah I basicly followed that inline documentation but it keeps trying to use the value from globals.xml | 19:29 |
| sdake | what does anyone make of this error out of kubectl describe rabbitmq: | 19:29 |
| sdake | 2m 2m 4 {default-scheduler } Warning FailedScheduling [SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "rabbitmq", which is unexpected., SchedulerPredicates failed due to Pe | 19:29 |
| sdake | rsistentVolumeClaim is not bound: "rabbitmq", which is unexpected., SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "ra | 19:29 |
| sdake | bbitmq", which is unexpected.] | 19:29 |
| inc0 | Reepicheep: yml;) | 19:29 |
| Reepicheep | lol.. yeah .. since I was point that out to you a few weeks back | 19:29 |
| * Reepicheep hangs his head but laughs | 19:29 | |
| inc0 | karma | 19:30 |
| sdake | pv and pvc insepction looks good | 19:30 |
| Reepicheep | i'm sure I will do it again | 19:30 |
| sdake | kfox1111 ^ ? | 19:30 |
| kbaegis | If I write up a guide for setting kolla up on gentoo, who do I submit it to? | 19:31 |
| kbaegis | There are kernel config requirements, tricks, etc just like the other oses | 19:31 |
| *** jtriley_ has joined #openstack-kolla | 19:33 | |
| kbaegis | Anyone know of an alternative to py-sysctl? | 19:34 |
| *** jtriley has quit IRC | 19:35 | |
| kbaegis | This was the blocker breaking haproxy for me: https://paste.pound-python.org/show/wdmihGbbXgOtlcOK3VMH/ | 19:37 |
| *** manheim has joined #openstack-kolla | 19:37 | |
| Reepicheep | so it apears that if I override network_interfaces in the inventory file it does not apply to the other interfaces from globals.yml like api_interface: "{{ network_interface }}" | 19:38 |
| Reepicheep | is that to be expected? ^ | 19:39 |
| Reepicheep | It works when I set each of the interface options in the inventory file | 19:39 |
| *** cristicalin has joined #openstack-kolla | 19:40 | |
| kbaegis | Reepicheep: Separate per-host definitions for neutron_external_interface, api_interface, and storage_interface | 19:40 |
| kbaegis | Also tunnel_interface | 19:40 |
| Reepicheep | yeah.. that is what I ended up doing to make it work | 19:41 |
| *** manheim has quit IRC | 19:42 | |
| inc0 | if you specify this in globals, it will override inventory changes | 19:42 |
| *** sayantan_ has quit IRC | 19:43 | |
| *** ducttape_ has joined #openstack-kolla | 19:43 | |
| egonzalez | Reepicheep yep, or each in the inventory or the most common at group_vars, but not globals | 19:44 |
| *** ducttap__ has joined #openstack-kolla | 19:44 | |
| Reepicheep | ok.. thanks.. see I still have a lot to learn about ansible ;) | 19:45 |
| *** bmaltais_ has quit IRC | 19:45 | |
| *** manheim has joined #openstack-kolla | 19:47 | |
| *** Pavo has quit IRC | 19:47 | |
| kbaegis | sweet :D ok=601 changed=289 unreachable=0 failed=0 | 19:47 |
| inc0 | kolla on gentoo | 19:48 |
| *** Pavo has joined #openstack-kolla | 19:48 | |
| *** ducttape_ has quit IRC | 19:48 | |
| inc0 | that's golden | 19:48 |
| *** Pavo has quit IRC | 19:49 | |
| *** Pavo has joined #openstack-kolla | 19:50 | |
| *** salv-orlando has joined #openstack-kolla | 19:53 | |
| Reepicheep | kbaegis: so yeah had me curious .. here is one of my old gentoo box uptimes "14:51:51 up 1256 days, 23:39" | 19:55 |
| kbaegis | inc0: You actually mentioned that at the summit ;) | 19:56 |
| *** manheim has quit IRC | 19:56 | |
| kbaegis | I'll write a guide if it's helpful | 19:56 |
| *** powerd has joined #openstack-kolla | 19:56 | |
| kbaegis | Reepicheep: That's awesome | 19:56 |
| kbaegis | Yeah, on the span of 10 years power failures start to become the key factor | 19:57 |
| inc0 | kbaegis: that would make a great blogpost;) | 19:57 |
| kbaegis | Their competition ended when one of their basements flooded | 19:57 |
| Reepicheep | I think the only reason that it even rebooted then was because that is when we had a guy doing mainanace on the UPS and killed the entire server room :| | 19:57 |
| kbaegis | Reepicheep: Sounds like gentoo alright :) | 19:57 |
| Reepicheep | we didn't have anyone in there at the time I pretty sure pushed the lever the wrong way | 19:57 |
| Reepicheep | now we always have someone with them when they are doing the yearly maintenace | 19:58 |
| Reepicheep | lol the kernel is 2.6.23-hardened-r4 | 19:59 |
| kbaegis | Yeah, I never learned how to live patch or ksplice | 20:00 |
| kbaegis | kexec is awesome though | 20:00 |
| Reepicheep | since I'm looking at the interface definitions .. what exactly is the dns_interface used for? | 20:02 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multi-scenerio gates https://review.openstack.org/466815 | 20:04 |
| inc0 | Reepicheep: I believe it was sth to do with designate | 20:05 |
| *** manheim has joined #openstack-kolla | 20:05 | |
| Reepicheep | that makes sense | 20:05 |
| egonzalez | Reepicheep is the interface used by mDNS (designate) | 20:07 |
| inc0 | egonzalez: mind checking it up? https://review.openstack.org/#/c/466007/ | 20:08 |
| Reepicheep | egonzalez: I was looking at designate the other day.. I saw your blog. I remember something about the interface needs to be public is that because haproxy doesn't do udp? | 20:09 |
| *** manheim has quit IRC | 20:09 | |
| egonzalez | Reepicheep yes, haproxy only handle tcp load balancing | 20:09 |
| egonzalez | thats why a dns_interface is needed, to publish mDNS to instances or public network | 20:10 |
| Reepicheep | so since it by default is using the controller nodes they then would need to have a public IP like the network nodes if you need pubic access? | 20:11 |
| Reepicheep | all my testing the network nodes where the only hosts with access to the public networks | 20:11 |
| Reepicheep | but I never went beyond just reading about designate | 20:12 |
| egonzalez | hrm, maybe mdns containers need to go on network node... | 20:13 |
| egonzalez | never though about that | 20:13 |
| inc0 | egonzalez: that would be good | 20:13 |
| inc0 | in fact, network nodes can be only ones exposed to external network | 20:14 |
| egonzalez | i have a working change for designate, probably include the group change | 20:14 |
| Reepicheep | inc0: I hope that's the case because that is how I have been testing for the last month.. and I'm now working on a production deployment ;) | 20:15 |
| inc0 | Reepicheep: worst case scenerio that would be simple chagne in inventory | 20:16 |
| Reepicheep | true | 20:16 |
| inc0 | Reepicheep: https://github.com/openstack/kolla-ansible/blob/master/ansible/inventory/multinode#L542 just change this line to network and that's it | 20:16 |
| Reepicheep | thanks, I will give that a try.. I have been interested in the designate project but have yet to really try it because of the public IP stuff | 20:18 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Move mDNS to network nodes https://review.openstack.org/468534 | 20:20 |
| Reepicheep | that was quick ^ ^ thanks | 20:23 |
| egonzalez | Reepicheep ^^ the "fix', didn't tested yet, but by designate architecture should work just changing the group, will verify after the weekend | 20:23 |
| *** manheim has joined #openstack-kolla | 20:23 | |
| egonzalez | Reepicheep won't work :( , another change is needed, templates wont work unless is the same interface | 20:25 |
| *** manheim has quit IRC | 20:27 | |
| *** unicell has joined #openstack-kolla | 20:28 | |
| *** chas has joined #openstack-kolla | 20:29 | |
| inc0 | egonzalez: btw http://logs.openstack.org/07/466007/72/experimental/gate-kolla-ansible-dsvm-deploy-ceph-oraclelinux-binary-centos-7-2-node-nv/7ab7921/logs/kolla/nova/nova-compute.log this issue appears every now and then | 20:31 |
| inc0 | did you notice it? it's quite undeterministic | 20:32 |
| *** fooliouno has quit IRC | 20:32 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Move mDNS to network nodes https://review.openstack.org/468534 | 20:32 |
| *** chas has quit IRC | 20:34 | |
| *** manheim has joined #openstack-kolla | 20:37 | |
| inc0 | I'm thinking this might be race condition between scheduler starting up and deployment of nova-compute | 20:38 |
| inc0 | we deploy too fast;) | 20:38 |
| kfox1111 | heh. there are races all over the place. | 20:38 |
| inc0 | yeah | 20:38 |
| kfox1111 | 90% percent of the kolla-kubernetes gate work was all just figuring out how to get rid of them. :/ | 20:38 |
| inc0 | I've seen this one in particular several times during my gate work | 20:39 |
| inc0 | brb need to do some grocery shopping | 20:40 |
| inc0 | kfox1111: I'd appreciate your review too https://review.openstack.org/#/c/466007/ | 20:40 |
| *** egonzalez has quit IRC | 20:40 | |
| inc0 | I know it's heavy on ansible, but your gate expetise might help picking something I might've missed | 20:41 |
| kfox1111 | I'll have a quick look. | 20:41 |
| *** manheim has quit IRC | 20:41 | |
| inc0 | thanks | 20:41 |
| kfox1111 | can't say I will understand it all. ;) | 20:42 |
| inc0 | no worries, I don't | 20:42 |
| inc0 | ;) | 20:42 |
| kfox1111 | are you using ansible to run ansible? | 20:43 |
| inc0 | no | 20:43 |
| kfox1111 | self modifying run? | 20:43 |
| *** cristicalin has quit IRC | 20:43 | |
| kfox1111 | or one instantiation followed by another maybe... | 20:43 |
| inc0 | which lines are you referring to? | 20:43 |
| kbaegis | inc0 thanks for the help earlier! I owe you one | 20:44 |
| kfox1111 | just looking through. nothing in particular yet. | 20:44 |
| inc0 | I run localhost ansible to lay down configs that I'll use later with non-local ansible | 20:44 |
| inc0 | kbaegis: np, happy to help | 20:44 |
| kfox1111 | k | 20:44 |
| *** jascott1 has quit IRC | 20:45 | |
| inc0 | if you, or anyone here feeling that you owe me, feels like returning a favor, I'll accept payment in a writing something to docs;) | 20:45 |
| inc0 | cheat sheet entry, use case description, anything is appreciated | 20:46 |
| kbaegis | I'll write up a gentoo install guide | 20:46 |
| kfox1111 | inc0: any reason to support non systemd ubuntu anymore? | 20:46 |
| inc0 | that would be awesome:) thank you! | 20:46 |
| kbaegis | Including all the kernel parameters that are required for kolla | 20:46 |
| kbaegis | np ;) | 20:47 |
| *** kbaegis has quit IRC | 20:47 | |
| kfox1111 | not that its a large amount of code. | 20:47 |
| inc0 | kfox1111: hmm...not really, I see newton, so oldest non-eol is already 16.04 | 20:47 |
| inc0 | bbiaf, I really need to do groceries or face hungry wife after long day at work, it's a personal safety issue for me now;) | 20:49 |
| kfox1111 | inc0: ah. the real reason... your tyring to get rid of a big shell script. ;) | 20:49 |
| inc0 | kfox1111: that too | 20:49 |
| inc0 | multinode got me thinking | 20:49 |
| sdake | thats dangerous | 20:50 |
| inc0 | I mean ansible is glorified shell over ssh | 20:50 |
| inc0 | sdake: thinking? yes it is | 20:50 |
| kfox1111 | inc0: generally looks ok to me. | 20:50 |
| *** manheim has joined #openstack-kolla | 20:51 | |
| inc0 | after it merges I'll check kolla-k8s gates and see if there is any missing features in either gate jobs | 20:51 |
| inc0 | I'm quite happy with log downloading, maybe something other communities could use too | 20:51 |
| inc0 | anyway | 20:52 |
| * inc0 off | 20:52 | |
| *** jascott1 has joined #openstack-kolla | 20:53 | |
| *** emccormick has joined #openstack-kolla | 20:54 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla master: In set_configs.py support perm parameter in permissions section https://review.openstack.org/468472 | 20:55 |
| *** jtriley_ has quit IRC | 20:55 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla master: In set_configs.py support perm parameter in permissions section https://review.openstack.org/468472 | 20:56 |
| *** manheim has quit IRC | 20:59 | |
| *** glycerine has quit IRC | 21:02 | |
| *** manheim has joined #openstack-kolla | 21:04 | |
| *** masuberu has quit IRC | 21:07 | |
| *** manheim has quit IRC | 21:09 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla master: Add short README file for contrib/template-override directory https://review.openstack.org/422552 | 21:09 |
| *** emccormick has quit IRC | 21:10 | |
| *** Pavo has quit IRC | 21:11 | |
| berendt | Continuing to work on Kolla upstream :) | 21:11 |
| Reepicheep | lol.. I see why my interfaces are going down on one of the nodes which causes me other issues | 21:12 |
| Reepicheep | "ixgbe 0000:04:00.0 enp4s0f0: Network adapter has been stopped because it has over heated. Restart the computer. If the problem persists, power off the system and replace the adapte" | 21:13 |
| *** ducttap__ has quit IRC | 21:14 | |
| *** fguillot has quit IRC | 21:14 | |
| *** manheim has joined #openstack-kolla | 21:18 | |
| *** jamesbenson has quit IRC | 21:21 | |
| *** jamesbenson has joined #openstack-kolla | 21:22 | |
| *** eaguilar_ has quit IRC | 21:22 | |
| *** jamesbenson has quit IRC | 21:23 | |
| *** jamesbenson has joined #openstack-kolla | 21:23 | |
| *** jamesbenson has quit IRC | 21:24 | |
| *** lucasxu has quit IRC | 21:26 | |
| *** jamesben_ has joined #openstack-kolla | 21:26 | |
| *** manheim has quit IRC | 21:27 | |
| *** jamesben_ has quit IRC | 21:28 | |
| *** jamesbenson has joined #openstack-kolla | 21:28 | |
| sdake | i think mariadab is busted in stable/ocata | 21:30 |
| sdake | a package is named MariaDB-Galera-Server and antoehr is named mariadb-galera-server inthe repos | 21:31 |
| sdake | or some such nonsense | 21:31 |
| *** manheim has joined #openstack-kolla | 21:32 | |
| *** jamesbenson has quit IRC | 21:32 | |
| jascott1 | whatup sdake! | 21:34 |
| sdake | jascott1 living the dream | 21:34 |
| jascott1 | excellent | 21:35 |
| *** ducttape_ has joined #openstack-kolla | 21:40 | |
| *** manheim has quit IRC | 21:41 | |
| *** manheim has joined #openstack-kolla | 21:45 | |
| *** manheim has quit IRC | 21:50 | |
| *** salv-orlando has quit IRC | 21:51 | |
| *** salv-orlando has joined #openstack-kolla | 21:51 | |
| *** rwellum has quit IRC | 21:53 | |
| *** ducttape_ has quit IRC | 21:55 | |
| *** salv-orlando has quit IRC | 21:55 | |
| sdake | inc0 https://da.gd/PrVz -> https://paste.fedoraproject.org/paste/VopjD-TFK8zr-JNCpqs--l5M1UNdIGYhyRLivL9gydE=/ | 21:57 |
| sdake | INFO:kolla.image.build.mariadb:No package MariaDB-Galera-server available. | 21:57 |
| sdake | stable/ocata | 21:58 |
| *** manheim has joined #openstack-kolla | 21:59 | |
| *** ducttape_ has joined #openstack-kolla | 22:06 | |
| *** manheim has quit IRC | 22:08 | |
| *** manheim has joined #openstack-kolla | 22:18 | |
| *** manheim has quit IRC | 22:22 | |
| *** powerd has quit IRC | 22:33 | |
| *** manheim has joined #openstack-kolla | 22:35 | |
| *** powerd has joined #openstack-kolla | 22:37 | |
| *** jamesbenson has joined #openstack-kolla | 22:39 | |
| *** manheim has quit IRC | 22:40 | |
| *** ipsecguy has quit IRC | 22:41 | |
| *** ipsecguy has joined #openstack-kolla | 22:41 | |
| *** jamesbenson has quit IRC | 22:43 | |
| *** jrich523 has quit IRC | 22:43 | |
| *** jascott1 has quit IRC | 22:47 | |
| *** manheim has joined #openstack-kolla | 22:49 | |
| *** manheim has quit IRC | 22:53 | |
| *** manheim has joined #openstack-kolla | 23:03 | |
| *** ducttape_ has quit IRC | 23:05 | |
| openstackgerrit | Steven Dake proposed openstack/kolla master: Remove EPEL as a dependency https://review.openstack.org/468576 | 23:06 |
| *** manheim has quit IRC | 23:07 | |
| *** jascott1 has joined #openstack-kolla | 23:13 | |
| *** rhallisey has quit IRC | 23:19 | |
| *** hongbin has quit IRC | 23:21 | |
| *** manheim has joined #openstack-kolla | 23:21 | |
| *** eaguilar has joined #openstack-kolla | 23:22 | |
| *** salv-orlando has joined #openstack-kolla | 23:22 | |
| *** manheim has quit IRC | 23:25 | |
| *** powerd has quit IRC | 23:26 | |
| *** salv-orlando has quit IRC | 23:27 | |
| *** david-lyle has joined #openstack-kolla | 23:31 | |
| *** manheim has joined #openstack-kolla | 23:35 | |
| *** david-lyle has quit IRC | 23:36 | |
| *** manheim has quit IRC | 23:39 | |
| *** manheim has joined #openstack-kolla | 23:49 | |
| *** manheim has quit IRC | 23:53 | |
| *** stefan_bo has joined #openstack-kolla | 23:54 | |
| *** yingjun has joined #openstack-kolla | 23:58 | |
| *** stefan_bo has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!