| *** openstack has joined #openstack-kolla | 07:00 | |
| *** duonghq has joined #openstack-kolla | 07:03 | |
| *** stefan_bo has joined #openstack-kolla | 07:26 | |
| *** manheim has joined #openstack-kolla | 07:28 | |
| *** matrohon_ has joined #openstack-kolla | 07:28 | |
| *** duonghq has quit IRC | 07:29 | |
| *** manheim has quit IRC | 07:32 | |
| *** jascott1 has quit IRC | 07:33 | |
| *** duonghq has joined #openstack-kolla | 07:33 | |
| *** unicell has quit IRC | 07:37 | |
| openstackgerrit | Mathieu Rohon proposed openstack/kolla master: Add skydive service https://review.openstack.org/459995 | 07:39 |
|---|---|---|
| *** huzhengchuan has joined #openstack-kolla | 07:44 | |
| *** egonzalez has joined #openstack-kolla | 07:44 | |
| *** masber has quit IRC | 07:45 | |
| *** salv-orlando has joined #openstack-kolla | 07:46 | |
| *** pcaruana has joined #openstack-kolla | 07:47 | |
| *** matrohon_ has quit IRC | 07:49 | |
| *** matrohon has joined #openstack-kolla | 07:52 | |
| *** yuanying_ has quit IRC | 07:53 | |
| *** yuanying has joined #openstack-kolla | 07:53 | |
| *** mannamne has joined #openstack-kolla | 07:56 | |
| *** salv-orlando has quit IRC | 07:57 | |
| *** manheim has joined #openstack-kolla | 07:59 | |
| *** manheim has quit IRC | 07:59 | |
| *** manheim has joined #openstack-kolla | 07:59 | |
| *** mannamne has quit IRC | 08:00 | |
| *** cuongnv has joined #openstack-kolla | 08:03 | |
| *** stefan_bo has quit IRC | 08:07 | |
| *** jaosorior has joined #openstack-kolla | 08:09 | |
| manheim | hello | 08:19 |
| duonghq | hi manheim | 08:20 |
| *** jascott1 has joined #openstack-kolla | 08:33 | |
| *** jascott1 has quit IRC | 08:38 | |
| *** gomarivera has joined #openstack-kolla | 08:46 | |
| *** shardy has joined #openstack-kolla | 08:49 | |
| *** gomarivera has quit IRC | 08:50 | |
| *** xinliang has quit IRC | 08:52 | |
| *** xinliang has joined #openstack-kolla | 08:54 | |
| openstackgerrit | yunfeng zhou proposed openstack/kolla master: add libvirt-python for instance monitor. https://review.openstack.org/464542 | 08:58 |
| *** skramaja is now known as skramaja_afk | 09:02 | |
| openstackgerrit | yunfeng zhou proposed openstack/kolla master: add libvirt-python for instance monitor. https://review.openstack.org/464542 | 09:13 |
| openstackgerrit | jeck proposed openstack/kolla-ansible master: Modified based on not installed grafana environment upgrade grafana caused by mariadb database not initialized grafana problem When upgrading grafana we need to make sure grafana has been deployed https://review.openstack.org/464582 | 09:24 |
| *** Serlex has joined #openstack-kolla | 09:26 | |
| *** jistr_ is now known as jistr | 09:30 | |
| openstackgerrit | pangliye proposed openstack/kolla master: Install the plugin of head https://review.openstack.org/464592 | 09:36 |
| *** stefan_bo has joined #openstack-kolla | 09:46 | |
| *** tovin07_ has quit IRC | 09:48 | |
| openstackgerrit | pangliye proposed openstack/kolla master: Install the plugin of elasticsearch-head https://review.openstack.org/464592 | 09:49 |
| *** duonghq has quit IRC | 10:00 | |
| *** janki has quit IRC | 10:12 | |
| *** janki has joined #openstack-kolla | 10:13 | |
| *** caowei has quit IRC | 10:19 | |
| *** sambetts|afk is now known as sambetts | 10:24 | |
| *** shardy is now known as shardy_afk | 10:24 | |
| *** athomas has joined #openstack-kolla | 10:25 | |
| *** mannamne has joined #openstack-kolla | 10:25 | |
| *** jascott1 has joined #openstack-kolla | 10:35 | |
| *** jascott1 has quit IRC | 10:39 | |
| *** stefan_bo has quit IRC | 10:41 | |
| *** gomarivera has joined #openstack-kolla | 10:47 | |
| *** skramaja_afk is now known as skramaja | 10:49 | |
| *** gomarivera has quit IRC | 10:52 | |
| bmaltais | eanylin: I did configure the external network IP as required. The strange thing is that kolla-ansible work on my setup but I have never been able to get the kubernetes method working. | 10:53 |
| bmaltais | eanylin: The admin interface work, I can spawn VMs but the external network refuse to work. | 10:53 |
| bmaltais | eanylin: I will try to change the interface name from ensXXX to ethX and see if somehow there are no dependencies on using those two interface name in the setup. | 10:54 |
| sdake | morning | 11:20 |
| *** rwallner has joined #openstack-kolla | 11:21 | |
| *** gema_ is now known as gema | 11:36 | |
| *** gema has quit IRC | 11:37 | |
| *** gema has joined #openstack-kolla | 11:37 | |
| *** hieulq has quit IRC | 11:38 | |
| *** blallau has joined #openstack-kolla | 11:43 | |
| *** rwallner has quit IRC | 11:45 | |
| *** rwallner has joined #openstack-kolla | 11:47 | |
| *** shardy_afk is now known as shardy | 11:47 | |
| *** zhubingbing_ has joined #openstack-kolla | 11:47 | |
| *** schwicht_at_work has quit IRC | 11:51 | |
| *** bmaltais_ has joined #openstack-kolla | 11:54 | |
| *** jamesPR has joined #openstack-kolla | 11:56 | |
| openstackgerrit | Pavel Gluschak (scsnow) proposed openstack/kolla-ansible master: Make cinder-backup service optional https://review.openstack.org/464620 | 11:58 |
| *** shardy_ has joined #openstack-kolla | 11:59 | |
| *** cuongnv has quit IRC | 12:00 | |
| *** bmaltais_ has quit IRC | 12:03 | |
| *** rwellum has joined #openstack-kolla | 12:05 | |
| *** rhallisey has joined #openstack-kolla | 12:06 | |
| *** sbezverk has quit IRC | 12:07 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/462547 | 12:09 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/462548 | 12:09 |
| *** bmaltais_ has joined #openstack-kolla | 12:09 | |
| *** rwallner has quit IRC | 12:17 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix lbaasv2 in service_plugins https://review.openstack.org/464288 | 12:18 |
| eanylin_ | bmaltais: Ok. The same approach worked for me for both kolla ansible and kolla kubernetes on KVM host and VirtualBox. I dont really think the name of the interface makes a difference. It was ensXXX for me in VirtualBox when I set up kolla ansible etc, and it worked the same way... But I think you can try it nonetheless... | 12:22 |
| *** rwallner has joined #openstack-kolla | 12:24 | |
| bmaltais_ | eanylin_: That is what I am thinking also. I tried changing the interface name and I get the same result. Very strange ;-( I did not tought the virtualisation platform should have an impact on the end result but perhaps it does. | 12:27 |
| bmaltais_ | eanylin_: I wonder what specific config under ESXi might be causing this. | 12:28 |
| eanylin_ | bmaltais: Yeah, it would be really strange though, if ESXi was the causing it to behave differently | 12:28 |
| openstackgerrit | Martin Mágr proposed openstack/kolla master: Sensu images https://review.openstack.org/463593 | 12:29 |
| bmaltais_ | eanylin_: Especially when using a script to deploy that removes the fat finger issues. It should be highly repeatable but it is not ;-( | 12:29 |
| eanylin_ | bmaltais: I think most of the testing was done on KVM host and I tested on virtualbox on my own, which was working properly too | 12:29 |
| eanylin_ | bmaltais: Yeah, agree | 12:29 |
| bmaltais_ | eanylin_: When I deploy I use setup_kube_AIO_new.sh 10.10.10.219 ens160 ens192 | 12:30 |
| bmaltais_ | 10.10.10.219 is my mgmt IP | 12:31 |
| eanylin_ | bmaltais: Yeah, that should be sufficient | 12:31 |
| bmaltais_ | ens160 is my management interface that has the 10.10.10.219 ip | 12:31 |
| bmaltais_ | And ens192 is my external interface | 12:31 |
| eanylin_ | ens192 will be the one for connecting to ext net which we dont assign IP to | 12:31 |
| bmaltais_ | If I put an IP on the external interface matching the network subnet I can ping the gateway using it... | 12:31 |
| bmaltais_ | So the connectivity is fine | 12:32 |
| *** chas_ has quit IRC | 12:32 | |
| eanylin_ | ok, agree. I used that approach as basic sanity test before I start any deployment as well | 12:32 |
| *** chas has joined #openstack-kolla | 12:32 | |
| eanylin_ | you are coming into the network from outside of the VM right? | 12:33 |
| *** sbezverk has joined #openstack-kolla | 12:34 | |
| eanylin_ | for instance, in my case, if I spin up the CentOS VM with kolla kubernetes on my KVM host, I will need to access the Cirros VM etc from my KVM host | 12:34 |
| eanylin_ | and not from the CentOS VM itself | 12:34 |
| bmaltais_ | eanylin_: Correct. Through a vyos router at 10.10.10.1 from my lab network. This is how I ssh into the VM to run the script | 12:34 |
| eanylin_ | I mean the Cirros VM that is spun up in OpenStack should be accessed from the KVM host in my case and not directly from within the CentOS VM | 12:34 |
| eanylin_ | ok, you are using VyOS for routing | 12:35 |
| manheim | egonzalez you on? | 12:35 |
| egonzalez | manheim, yep | 12:35 |
| bmaltais_ | Yes. And the cirros VM can't ping the 10.10.11.1 (other network) ip when I assign a public IP from the pool | 12:35 |
| bmaltais_ | I can't ping either the router IP from openstack on that network not the cirros IP from vyos | 12:36 |
| manheim | https://review.openstack.org/#/c/455684/ | 12:36 |
| manheim | if you remember the last issue, it was avout overriding nova.conf in config.yml between kvm and hyperv | 12:36 |
| bmaltais_ | It appear the ens192 interface is not attaching to the br-ex interface\ | 12:36 |
| bmaltais_ | Somehow that step is not working properly | 12:37 |
| sbezverk | morning gents | 12:37 |
| manheim | hello | 12:37 |
| bmaltais_ | Morning | 12:37 |
| *** chas has quit IRC | 12:37 | |
| sbezverk | bmaltais_: try to add that interface manually | 12:37 |
| eanylin_ | bmaltais: Ok. I havent seen that behavior so far. Usually even if I cannot access the Cirros VM from my host, I can ping my public router from within the OpenStack overlay network. | 12:37 |
| bmaltais_ | eanylin_: I will rebuild the VM with the script so I can better troubleshoot and provide info if needed. | 12:38 |
| sbezverk | bmaltais_: kubectl exec {your vswitchd pod} -n kolla -- ovs-vsctl add-port br-ex ens192 | 12:38 |
| eanylin_ | bmaltais_: Ok | 12:38 |
| egonzalez | manheim, remember, you can use in hyperv role for copy nova.conf: {{ role_path }}../nova/templates/nova.conf.j2 | 12:39 |
| manheim | yes, but i have a different solution | 12:39 |
| eanylin_ | Morning sbezverk | 12:39 |
| egonzalez | manheim, exactly as made in glance/nova/cinder https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/glance/tasks/ceph.yml#L11 | 12:39 |
| manheim | what of i name my nova.conf nova-hyperv.conf.j2 | 12:39 |
| manheim | and in the copy over, i put dst nova.conf | 12:39 |
| manheim | i works, o tried | 12:39 |
| bmaltais_ | eanylin_: OK, I will try that command if the resulting environment still does not have ens192 attached to br-ex | 12:39 |
| manheim | so in templates/nova-hyperv.conf.j2 | 12:40 |
| manheim | and when o copy the template over, I put the dst nova.conf | 12:40 |
| egonzalez | manheim, the issue here is that we are going to duplicate 60% of the file with main nova.conf | 12:40 |
| eanylin_ | bmaltais: Yeah, I think we can try sbezverk's suggestion. | 12:41 |
| manheim | hmm, this is true | 12:41 |
| manheim | that big of an issue? | 12:41 |
| egonzalez | and will be harder to maintain, each time a change into nova.conf is made will need to be also done at hyper nova conf | 12:41 |
| egonzalez | which mostly for sure will cause a not up to date hyperv nova.conf | 12:42 |
| manheim | i see | 12:43 |
| manheim | ok, one questions | 12:43 |
| *** schwicht has joined #openstack-kolla | 12:43 | |
| manheim | so i have vars specific to nova-hyperv role, do i have to put them in nova role as well? | 12:43 |
| *** flaper87 has quit IRC | 12:43 | |
| manheim | if i use nova/nova.conf.js | 12:43 |
| manheim | vars in defaults/main.yml | 12:44 |
| *** flaper87 has joined #openstack-kolla | 12:44 | |
| *** schwicht has quit IRC | 12:45 | |
| *** flaper87 has quit IRC | 12:45 | |
| *** schwicht has joined #openstack-kolla | 12:45 | |
| *** flaper87 has joined #openstack-kolla | 12:47 | |
| *** flaper87 has quit IRC | 12:47 | |
| *** flaper87 has joined #openstack-kolla | 12:48 | |
| egonzalez | i see.. that can be resolved with include_vars from nova role into hyperv role, at some point in time (when hyperv gets containerized) hyperv role will disappear and be merged into global nova role | 12:49 |
| *** rwallner has quit IRC | 12:50 | |
| *** jascott1 has joined #openstack-kolla | 12:50 | |
| *** dave-mccowan has joined #openstack-kolla | 12:51 | |
| egonzalez | I think we should try to reuse as much as we can from global nova role to avoid duplicate work imo | 12:51 |
| manheim | i see | 12:53 |
| *** zhubingbing_ has quit IRC | 12:54 | |
| *** goldyfruit has joined #openstack-kolla | 12:54 | |
| *** blallau has quit IRC | 12:55 | |
| *** jascott1 has quit IRC | 12:56 | |
| *** rwallner has joined #openstack-kolla | 12:57 | |
| *** schwicht has quit IRC | 12:59 | |
| manheim | won't i need to import from nova-hyperv? i mean nova.conf.j2 from nova/ is intepreted | 13:00 |
| manheim | and in nova/nova.conf.j2 i'll need stuff like rdp_port from nova-hyper | 13:01 |
| bmaltais_ | eanylin_: OK. Done with the setup. This is the state of br-ex when running kubectl exec openvswitch-vswitchd-network-3t37d -n kolla -- ovs-vsctl show http://paste.openstack.org/show/609560/ | 13:02 |
| egonzalez | manheim, maybe not, as all hyperv variables will be under `if enable_hyperv, not evaluated, endif` | 13:03 |
| manheim | i get this, but if enable_hyperv is evaluated | 13:03 |
| manheim | do I need to impot nova-hyperv vars in nova or not? | 13:04 |
| *** jtriley has quit IRC | 13:04 | |
| egonzalez | manheim, vars will be evaluated from hyperv defaults, and from included nova defaults | 13:05 |
| sbezverk | bmaltais_: kubectl exec openvswitch-vswitchd-network-3t37d -n kolla -- ovs-vsctl add-port br-ex ens192 | 13:05 |
| egonzalez | manheim, guess not | 13:05 |
| manheim | ok | 13:05 |
| sbezverk | bmaltais_: it is still missing interface | 13:05 |
| egonzalez | manheim, wait, yes because nova-api need those vars | 13:05 |
| *** rwallner has quit IRC | 13:05 | |
| manheim | i'll write the code and i'll test, thanks for the help | 13:06 |
| manheim | ok | 13:06 |
| bmaltais_ | sbezverk: After adding the interface to the pod: http://paste.openstack.org/show/609564/ | 13:08 |
| *** rwallner_ has joined #openstack-kolla | 13:08 | |
| bmaltais_ | sbezverk: I will setup a basic VM to test if all is fine now | 13:08 |
| sbezverk | bmaltais_: now it looks right imho :) | 13:08 |
| *** rwallner_ has quit IRC | 13:09 | |
| *** rwallner_ has joined #openstack-kolla | 13:09 | |
| *** mattmceuen_ has quit IRC | 13:09 | |
| *** rwallner_ has quit IRC | 13:10 | |
| *** rwallner has joined #openstack-kolla | 13:10 | |
| bmaltais_ | sbezverk: If this fixes it then I wonder why it does not get properly built. Might be good to figure it out as it might help others in the future | 13:10 |
| *** mgoddard has joined #openstack-kolla | 13:13 | |
| sbezverk | bmaltais_: if it fixes, we will have a debug session with you to see what is going on. | 13:13 |
| bmaltais_ | sbezverk: Let me know if there is particular log file I could look into to see why the interface does not get added properly to br-ex during the deployment | 13:13 |
| bmaltais_ | OK | 13:13 |
| bmaltais_ | No problem | 13:13 |
| sbezverk | bmaltais_: one thing would be helpfull is adding --debug keyword to all helm install commands in the script you are using | 13:13 |
| mgoddard | kfox1111: hi. seen this: https://thenewstack.io/openstack-boston-2017-finding-new-place-among-stacks/? | 13:13 |
| eanylin_ | bmaltais_:I think mine looks similar as yours as well, http://paste.openstack.org/show/609566/ | 13:13 |
| sbezverk | then we can see if command got generated to add interface | 13:13 |
| *** rwallner_ has joined #openstack-kolla | 13:13 | |
| *** rwallner has quit IRC | 13:14 | |
| *** bjolo has joined #openstack-kolla | 13:14 | |
| sbezverk | mgoddard: thanks for the link, out kfox1111 is a celebrity now ;) | 13:15 |
| *** jaosorior has quit IRC | 13:15 | |
| bmaltais_ | sbezverk: It works now | 13:16 |
| mgoddard | sbezverk: it appears so! | 13:16 |
| bmaltais_ | sbezverk: So somehow the interface fail to be added during the deployment | 13:16 |
| sbezverk | bmaltais_: ok in this case I am almost positive it has something to do with identation in cloud.yaml | 13:16 |
| bmaltais_ | sbezverk: Let me know how I can help to fix this | 13:17 |
| sbezverk | when helm looks for the variable add_port it does not find it and hence the interface does not get plugged. | 13:17 |
| bmaltais_ | sbezverk: The strange thing is that I use a script to deploy and the indent is working for others | 13:17 |
| eanylin_ | bmaltais_: How about pasting the cloud.yaml and see where it got off-indented? | 13:18 |
| pomac | docker restart nova_libvirt is not a good idea - it blocks new machines from being started. --- me and bjolo etc have been... debugging that for a ... while... | 13:18 |
| *** ducttape_ has joined #openstack-kolla | 13:18 | |
| sbezverk | bmaltais_: we need to confirm it by re-deploying openswitch with --debug keyword | 13:18 |
| pomac | any clues on things that could be affecting this? | 13:18 |
| bmaltais_ | sbezverk: Here is a copy of cloud.yaml: http://paste.openstack.org/show/609567/ | 13:19 |
| bmaltais_ | sbezverk: I will do a new deploy with --debug added to all helm commands | 13:20 |
| bmaltais_ | sbezverk: Where will the log files be stored so I can provide a copy of them? | 13:21 |
| sbezverk | bmaltais_: it should be spit to stdout | 13:22 |
| sbezverk | just make sure you capture it | 13:22 |
| bmaltais_ | sbezverk: OK, will do that and report the result | 13:24 |
| manheim | kolla-ansible reconfigure -t nova should reconfigure only nova, right? | 13:25 |
| manheim | coz it does for all roles | 13:25 |
| *** chas has joined #openstack-kolla | 13:27 | |
| *** bjolo_ has joined #openstack-kolla | 13:27 | |
| *** Serlex has quit IRC | 13:27 | |
| *** bjolo_ has left #openstack-kolla | 13:28 | |
| bmaltais_ | sbezverk: While the deployment run, I also found another issue with console access | 13:29 |
| *** chas has quit IRC | 13:29 | |
| *** chas has joined #openstack-kolla | 13:29 | |
| *** eanylin_ has quit IRC | 13:29 | |
| bmaltais_ | The url returned is not reachable as it does not point to the public ip | 13:30 |
| bmaltais_ | It return: http://nova-novncproxy:6080/ | 13:31 |
| sbezverk | bmaltais_: in cloud yaml you need port_external: true | 13:31 |
| bmaltais_ | Instead of http://ip:6080/ | 13:31 |
| bmaltais_ | sbezverk: according to the cloud.yml it should already be set: http://paste.openstack.org/show/609567/ | 13:32 |
| bmaltais_ | So maybe another issue with the parsing of my cloud.yaml during deployment | 13:33 |
| manheim | if i use this way, nova-hyperv stuff will be evaluated in nova role | 13:33 |
| manheim | hmmm | 13:33 |
| manheim | coz enable_hypev is enabled | 13:34 |
| manheim | but stuff is done during nova role | 13:34 |
| *** zhubingbing_ has joined #openstack-kolla | 13:36 | |
| zhubingbing_ | hi guys | 13:36 |
| manheim | hi | 13:36 |
| bmaltais_ | Hi | 13:36 |
| *** jtriley has joined #openstack-kolla | 13:38 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/462548 | 13:39 |
| sbezverk | bmaltais_: man there must be something with cloud.yaml then for sure | 13:40 |
| sbezverk | bmaltais_: maybe some characters got into it, like tabs or something else?! | 13:40 |
| bmaltais_ | Perhaps. I wonder if the script has a missing/too many spaces causing the issue | 13:41 |
| bmaltais_ | Is there a yml validator I vould use on it to check? | 13:41 |
| sbezverk | bmaltais_: you should be able to find one on the internet, do not have link handy | 13:42 |
| *** skramaja has quit IRC | 13:42 | |
| rwellum | bmaltais_: that's native functionality in emacs | 13:44 |
| bmaltais_ | sbezverk: OK. Here is the debug output as requested: http://paste.openstack.org/show/609570/ | 13:44 |
| bmaltais_ | sbezverk: Oups... look like paste.openstack.org can't handle the full length | 13:45 |
| bmaltais_ | Is there a specific portion you would like to see? | 13:46 |
| sbezverk | bmaltais_: yes, where openvswitch gets installed | 13:46 |
| bmaltais_ | sbezverk: Here is a link to the log: https://drive.google.com/file/d/0B5odRXvIkNfJdEZmdnlfVmdLdUE/view?usp=sharing | 13:49 |
| *** zhubingbing_ has quit IRC | 13:49 | |
| bmaltais_ | sbezverk: Still too much output to paste so a direct file access is better | 13:50 |
| bmaltais_ | sbezverk: Here is an interesting section: http://paste.openstack.org/show/609573/ | 13:53 |
| bmaltais_ | Look at how ens224 is interpreted as eth1... | 13:54 |
| bmaltais_ | sbezverk: Forget it... look like the section is redefined after with the right values | 13:55 |
| sbezverk | bmaltais_: http://paste.openstack.org/show/609574/ it is not picking up your values | 13:55 |
| sbezverk | bmaltais_: as a result it does not generate add interface to bridge command | 13:56 |
| bmaltais_ | sbezverk: So most likelly an issue with the cloud.yml file then? | 13:57 |
| *** tomatopeel has joined #openstack-kolla | 13:57 | |
| sbezverk | bmaltais_: or the way it got fed into compute kit | 13:57 |
| *** eanylin has joined #openstack-kolla | 13:58 | |
| bmaltais_ | sbezverk: This is the command to deploy openvswitch: helm install --debug /opt/kolla-kubernetes/helm/service/openvswitch --namespace kolla --name openvswitch --values /opt/c loud.yaml | 13:58 |
| sbezverk | but since partially it works, then something fishy is going on with it | 13:58 |
| bmaltais_ | I see the issue | 13:58 |
| bmaltais_ | Darn! | 13:58 |
| hrw | hi | 13:59 |
| bmaltais_ | A script error. Sorry folks. | 13:59 |
| bmaltais_ | sbezverk: I need to find a way to contribute to the source code to raise the issue with the proposed script | 14:01 |
| bmaltais_ | https://review.openstack.org/#/c/464312/ | 14:02 |
| eanylin | bmaltais_: Which line actually? | 14:03 |
| bmaltais_ | sbezverk: Sorry again... the "space" was just a copy/paste from the terminal | 14:03 |
| bmaltais_ | The script is actually fine | 14:03 |
| bmaltais_ | So this is not it | 14:03 |
| * hrw back to code review | 14:04 | |
| bmaltais_ | But I see a strange "CR/LF" for the helm install --debug /opt/kolla-kubernetes/helm/service/cinder-control line | 14:04 |
| eanylin | bmaltais: Ok, was checking actually, didnt see space | 14:05 |
| eanylin | bmaltais_: CR/LF means? | 14:05 |
| sbezverk | bmaltais_: play with cloud.yaml, add some changes | 14:05 |
| sbezverk | bmaltais_: I am positive the issue is there | 14:05 |
| bmaltais_ | eanylin: Carriage Return / Line Feed | 14:07 |
| bmaltais_ | It show when I do more <script> in the terminal window | 14:07 |
| bmaltais_ | Not sure if this can cantribute to the isse... probably not | 14:08 |
| *** schwicht has joined #openstack-kolla | 14:08 | |
| eanylin | bmaltais_: Ok | 14:09 |
| bmaltais_ | eanylin: Again, not showing the secong time I tried | 14:09 |
| bmaltais_ | Must be some terminal roll over oddities | 14:09 |
| bmaltais_ | I did stretch the output window so it might have caused the perceived CR/LF but it is not there. | 14:10 |
| bmaltais_ | OK. I will have a look at the cloud.yml file. I can share the actual file as it stand if you think it is worth looking at it. | 14:11 |
| eanylin | bmaltais_: Ok, cause when I read it in gerrit, it seems ok... | 14:15 |
| bmaltais_ | sbezverk: Using http://yaml-online-parser.appspot.com/ to parse the yml file content look good | 14:15 |
| eanylin | bmaltais_: Please let me know then, if you see problems with the cloud.yaml that got auto-generated | 14:15 |
| eanylin | the alignment seems ok to me... | 14:15 |
| bmaltais_ | eanylin: Same using http://yaml-online-parser.appspot.com/ | 14:16 |
| eanylin | ok | 14:16 |
| bmaltais_ | So the issue must be somewhere else | 14:16 |
| eanylin | bmaltais_: Ok | 14:17 |
| *** Serlex has joined #openstack-kolla | 14:17 | |
| bmaltais_ | sbezverk: http://paste.openstack.org/show/609574/ looking at line 65-82 | 14:17 |
| bmaltais_ | Is this where you see the issue? Could line 69 be causing the issue because it does not get defined as UP in section 79-82? | 14:18 |
| sbezverk | bmaltais_: F........ there is a type on openvswitch | 14:20 |
| sbezverk | check line 150 | 14:20 |
| sbezverk | I meant typo | 14:21 |
| sbezverk | line 162 openvwswitch: | 14:21 |
| bmaltais_ | 162 you mean? | 14:21 |
| hrw | egonzalez: if image is built only from binaries then should it use 'if type == binary' for installing or not? | 14:21 |
| sbezverk | there is extra 'w' | 14:21 |
| eanylin | line 162 | 14:22 |
| hrw | openwswitch | 14:22 |
| eanylin | agree | 14:22 |
| eanylin | I will update | 14:22 |
| bmaltais_ | Yep... | 14:22 |
| egonzalez | hrw, openvswitch is built binary for source and binary installs | 14:22 |
| bmaltais_ | So an issue in the script creating the cloud.yml | 14:22 |
| egonzalez | not need if type binary or will not install for source | 14:22 |
| bmaltais_ | eanylin: i guess ;-) | 14:23 |
| jamesPR | sup egonzalez, remember my issue with SSL I fixed with couple of lines in haproxy.cfg | 14:23 |
| jamesPR | for horizon | 14:23 |
| sdake | jamesPR which lines were needed | 14:23 |
| hrw | egonzalez: looking at ceilometer-ipmi code review. for me if it uses binary only then it should not have 'if type == binary' check. otherwise rsulting image will be empty for type=source builds. | 14:24 |
| egonzalez | jamesPR, something so specific for your env or a global kolla issue? | 14:24 |
| eanylin | I copied from Deployment Guide, seems like its wrong there too =X | 14:24 |
| bmaltais_ | Ha ha! | 14:24 |
| sbezverk | eanylin: :) it happens | 14:24 |
| egonzalez | hrw, in that case, ceilometer is build for source and binary, while building images (source is always build at ceilometer-base) | 14:24 |
| jamesPR | listen horizon | 14:24 |
| jamesPR | bind 10.50.85.254:80 | 14:24 |
| jamesPR | bind 10.50.85.254:443 ssl crt /etc/haproxy/haproxy.pem | 14:24 |
| jamesPR | mode http | 14:24 |
| jamesPR | option http-server-close | 14:24 |
| jamesPR | option forwardfor | 14:24 |
| jamesPR | balance source | 14:24 |
| jamesPR | http-request del-header X-Forwarded-Proto if { ssl_fc } | 14:24 |
| jamesPR | server node1 10.50.85.70:80 check inter 2000 rise 2 fall 5 | 14:24 |
| jamesPR | server node2 10.50.85.71:80 check inter 2000 rise 2 fall 5 | 14:25 |
| jamesPR | server node3 10.50.85.72:80 check inter 2000 rise 2 fall 5 | 14:25 |
| eanylin | sbezverk: Good catch :) | 14:25 |
| bmaltais_ | eanylin: Correct, the typo is indeed in the deployment guide | 14:25 |
| hrw | egonzalez: ceilometer-base is built properly. ceilometer-ipmi is empty. gave -1 with info why | 14:25 |
| egonzalez | if install_type is not in ipmi, then will install binary packages into source(not expected in this case) | 14:25 |
| eanylin | bmaltais_: Yeah, I think we need to correct it there too | 14:25 |
| sdake | curious what purpose of script is | 14:25 |
| openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: Script to Deploy Kolla Kubernetes AIO https://review.openstack.org/464312 | 14:26 |
| bmaltais_ | eanylin: I also have an issue with novncproxy: | 14:26 |
| *** mgoddard_ has joined #openstack-kolla | 14:26 | |
| jamesPR | nothing specific on my globals.yml | 14:26 |
| goldyfruit | Guys, any idea about this: https://bugs.launchpad.net/kolla/+bug/1690252 | 14:27 |
| openstack | Launchpad bug 1690252 in kolla-ansible "Docker L3 and VPN agent containers don't kill keealived with a SIGTERM" [Undecided,New] | 14:27 |
| egonzalez | jamesPR, so only added mode http | 14:27 |
| egonzalez | option http-server-close | 14:27 |
| egonzalez | option forwardfor | 14:27 |
| bmaltais_ | eanylin: It does not return the IP but rather http://nova-novncproxy:6080/ | 14:27 |
| bmaltais_ | Wonder if it is another typo | 14:27 |
| *** mgoddard has quit IRC | 14:27 | |
| eanylin | bmaltais_: Oh, I usually set this to the IP in my /etc/hosts | 14:27 |
| eanylin | so that it gets redirected properly | 14:27 |
| jamesPR | egonzalez, bind 10.50.85.254:443 ssl crt /etc/haproxy/haproxy.pem < yeah and this one too | 14:27 |
| *** mgoddard__ has joined #openstack-kolla | 14:27 | |
| egonzalez | jamesPR, but that config is in internal haproxy backend | 14:27 |
| openstackgerrit | Merged openstack/kolla master: Updated from global requirements https://review.openstack.org/462547 | 14:28 |
| egonzalez | ssl config is only on external | 14:28 |
| bmaltais_ | eanylin: Apparently it should put the IP in there based on the config in cloud.yml | 14:28 |
| eanylin | bmaltais_: Ok | 14:28 |
| jamesPR | ohh ok | 14:28 |
| manheim | https://bugs.launchpad.net/kolla/+bug/1537815 | 14:29 |
| openstack | Launchpad bug 1537815 in kolla "horizon does not load the css properly" [Undecided,Fix released] - Assigned to Jeffrey Zhang (jeffrey4l) | 14:29 |
| manheim | this was not backported to stable/ocata? | 14:29 |
| jamesPR | egonzalez, I cannot access horizon from external IP | 14:29 |
| manheim | i keep having this issue if I have a custom theme on first deploy | 14:29 |
| jamesPR | only from internal | 14:30 |
| manheim | even tho I set debug=true | 14:30 |
| egonzalez | jamesPR, horizon really does not use SSL, is HAproxy removes SSL headers before sending to haproxy internal iirc | 14:30 |
| bmaltais_ | eanylin: Saw you already pushed the update to the script. I will grab a copy and validate the deployment on my ESXi VM. | 14:30 |
| eanylin | bmaltais_: Thanks! | 14:31 |
| egonzalez | jamesPR, do you have listen horizon_external backend in your haproxy.conf? | 14:31 |
| *** mgoddard_ has quit IRC | 14:31 | |
| jamesPR | jamesPR, no because I try this but doesn't work cannot ping external IP from my network | 14:32 |
| bmaltais_ | eanylin: Nice work on the script by the way | 14:32 |
| egonzalez | jamesPR, need to add kolla_enable_tls_external: yes and haproxy_enable_external_vip: yes into globals.yml | 14:32 |
| bmaltais_ | eanylin: I wonder what would be needed to modify to make the deployment work on multiple kubernetes nodes | 14:33 |
| bmaltais_ | It is easy to add new nodes but right now the IP is attached to the interface of the master node | 14:33 |
| egonzalez | jamesPR, also kolla_external_vip_address: SOME_IP_ADDRESS | 14:33 |
| bmaltais_ | Would need an HAProxy in front | 14:33 |
| jamesPR | cat /etc/kolla/globals.yml |grep -v ^# | grep -v ^$ | grep -e "vip_address" -e "interface" | 14:34 |
| jamesPR | kolla_internal_vip_address: "10.50.85.254" | 14:34 |
| jamesPR | kolla_external_vip_address: "10.50.85.254" | 14:34 |
| jamesPR | network_interface: "eth1" | 14:34 |
| jamesPR | neutron_external_interface: "eth0" | 14:34 |
| jamesPR | kolla_external_vip_interface: "{{ network_interface }}" | 14:34 |
| jamesPR | api_interface: "{{ network_interface }}" | 14:34 |
| eanylin | bmaltais_: Thanks. I am thinking of creating multiple nodes, after getting single node to work | 14:34 |
| egonzalez | jamesPR, cannot be the same IP for internal and external | 14:34 |
| bmaltais_ | eanylin: I think putting a working haproxy in front of horizon even in single node mode would be a 1st step | 14:34 |
| eanylin | bmaltais_: Ok | 14:35 |
| jamesPR | egonzalez, I try with the external IP but doesn't work | 14:35 |
| egonzalez | jamesPR, haproxy will try to bind <IP><PORT>, a port cannot be bind twice in the same IP | 14:35 |
| bmaltais_ | eanylin: Then adding other minions should be easy | 14:35 |
| jamesPR | egonzalez, right now I cannot ping the external IP servers have | 14:36 |
| bmaltais_ | egonzalez: So is there an easy way to set it up like the ansible solution with a virtual IP? | 14:36 |
| eanylin | bmaltais_: Yeah, I did it with Jenkins myself but I think sbezverk feels that its better to use bash to reduce dependencies, which I agree | 14:36 |
| egonzalez | jamesPR, this is a working ssl working on ocata http://paste.openstack.org/show/609578/ | 14:37 |
| bmaltais_ | egonzalez: Sorry, the comment was not for you. | 14:38 |
| jamesPR | egonzalez, ok I will try | 14:38 |
| bmaltais_ | eanylin: I am not sure i understand the comment about bash | 14:38 |
| bmaltais_ | eanylin: Do youmean to deploy new minions using manual bash commands? | 14:39 |
| egonzalez | jamesPR, you should see the external VIP at keepalived master node | 14:39 |
| eanylin | as in use bash scripts to set up the whole thing instead of tools | 14:39 |
| eanylin | like Jenkins | 14:39 |
| eanylin | something like that | 14:39 |
| jamesPR | egonzalez, yes | 14:39 |
| jamesPR | but I cannot ping that IP from my network | 14:39 |
| bmaltais_ | eanylin: OK. What about making sure haproxy work for horizon access using a VIP? | 14:39 |
| *** gomarivera has joined #openstack-kolla | 14:40 | |
| eanylin | bmaltais_: Actually I think the same VIP will work (am still learning). Based on what I saw in my own setup, I can get 3 horizon across 3 nodes with replica set to 3 | 14:41 |
| bmaltais_ | eanylin: This way horizon is not bound to the master AIO IP but rather a floating one across multiple kubernetes nodes | 14:41 |
| eanylin | and it still works | 14:41 |
| eanylin | with kube proxy | 14:41 |
| bmaltais_ | eanylin: I am learning too ;-) So take what I say with a grain of salt | 14:41 |
| egonzalez | hrw, re ceilometer-ipmi, also is needed because deployment methods consume the image by name, also can be customized some images instead of ceilometer-base and add correct USER(base set as root) | 14:41 |
| eanylin | bmaltais_: :) | 14:42 |
| bmaltais_ | eanylin: But from what I see the IP used is the one from the master k8 node and I won't work to access horizon on other nodes if the master is down | 14:42 |
| eanylin | bmaltais_: Yeah, if master is down, its going to be tough | 14:42 |
| bmaltais_ | eanylin: This is why the kolla-ansible is using a VIP (like subnet.254) as the floating IP to access horizon | 14:43 |
| bmaltais_ | eanylin: The same need to be done for the AIO to make it multinode compliant I think | 14:44 |
| bmaltais_ | eanylin: This way the VIP attached to haproxy will balance access to horizon across all POD even if on different nodes | 14:44 |
| bmaltais_ | eanylin: Again, in theory given my understanding of haproxy at the moment | 14:45 |
| inc0 | good morning | 14:46 |
| eanylin | bmaltais_: Hmm, ok. We can discuss with sbezverk maybe | 14:46 |
| jamesPR | egonzalez, I can access floating ips without issues from external network | 14:47 |
| jamesPR | but cannot ping external VIP | 14:47 |
| *** schwicht has quit IRC | 14:47 | |
| sbezverk | eanylin: port_external: true | 14:48 |
| *** gomarivera has quit IRC | 14:48 | |
| sbezverk | then it should use external vip | 14:48 |
| *** schwicht has joined #openstack-kolla | 14:48 | |
| sbezverk | for vncproxy | 14:48 |
| egonzalez | jamesPR, can you share `ip a` from keepalived master node? | 14:49 |
| hrw | inc0: hi | 14:50 |
| inc0 | hey hrw :) | 14:51 |
| eanylin | sbezverk: Yeah, because we set it to external true, it uses external vip | 14:51 |
| jamesPR | egonzalez, https://pastebin.com/raw/G3evdGkz | 14:52 |
| sbezverk | inc0: could you check this ps, we need to restore the gate: https://review.openstack.org/464347 | 14:52 |
| egonzalez | jamesPR, change kolla_external_vip_address to a different IP address of kolla_internal_vip_address | 14:53 |
| *** gomarivera has joined #openstack-kolla | 14:53 | |
| *** itlinux has joined #openstack-kolla | 14:53 | |
| sbezverk | eanylin: but in case bmaltais_ he has that value set, but it is not get picked up | 14:53 |
| itlinux | does the kolla-ansible work on virtualbox to do the testing? Thanks | 14:53 |
| inc0 | oh joy, k8s is broken again:( | 14:53 |
| egonzalez | jamesPR, and do kolla-ansible deploy --tags haproxy | 14:54 |
| itlinux | does anyone have the guidance in case.. | 14:54 |
| eanylin | sbezverk: It will get auto directed to nova-novncproxy | 14:54 |
| sbezverk | it looks another issue of cloud.yaml ;) | 14:54 |
| inc0 | itlinux: it should work just fine | 14:54 |
| egonzalez | jamesPR, well, in ocata better use kolla-ansible reconfigure | 14:54 |
| bmaltais_ | sbezverk: Found the issue in the cloud.yml file | 14:54 |
| bmaltais_ | sbezverk: A typo is causing the issue | 14:54 |
| inc0 | I mean it doesn't really care if it's vbox, kvm or bare metal | 14:54 |
| jamesPR | egonzalez, ok trying | 14:54 |
| sbezverk | eanylin: hmm, I do not think so | 14:54 |
| itlinux | hi inc0. I get a strange virt error.. | 14:54 |
| sbezverk | outside that dns name is not known | 14:55 |
| inc0 | hmm, so by default I believe we use kvm | 14:55 |
| sbezverk | so it will break | 14:55 |
| inc0 | and nested virt is fun | 14:55 |
| eanylin | sbezverk: Hmm, ok. I seem to see that behavior with current set up. Maybe missing something? | 14:55 |
| itlinux | ahh I get the precheck ok but the deploy fails.. | 14:55 |
| eanylin | sbezverk: Agree. Which is why I usually get around it with updates to /etc/hosts | 14:56 |
| inc0 | itlinux: what's the error? | 14:56 |
| itlinux | it failed on fluentd so I did a manual install so it passed that part but the deploy fails.. | 14:56 |
| jamesPR | egonzalez, ok | 14:56 |
| sbezverk | eanylin: ahhh, it is because AIO | 14:56 |
| itlinux | check it now | 14:56 |
| sbezverk | but access it from outside will not work | 14:56 |
| eanylin | sbezverk: Ah, ok | 14:56 |
| sbezverk | eanylin: so it is safer to use VIP | 14:56 |
| egonzalez | jamesPR, worked? | 14:56 |
| *** bjolo has quit IRC | 14:56 | |
| jamesPR | egonzalez, running right now | 14:57 |
| eanylin | sbezverk: Ok. So where should I change in cloud.yaml to make it use VIP? | 14:58 |
| itlinux | inc0: now I get this fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Unknown error message: Tag 4.0.1 not found in repository docker.io/kolla/centos-binary-fluentd"} | 14:58 |
| inc0 | ahh I suspected as much, you didn't build images right? are you planning to deploy ocata? | 14:59 |
| inc0 | or master? | 14:59 |
| itlinux | it's pulling down directly, no image build yet | 14:59 |
| itlinux | where do I check if it's master or ocata... in the /etc/kolla ? | 15:00 |
| inc0 | how did you install kolla-ansible? | 15:00 |
| itlinux | I have not done much on kolla inc0: we talked last week at the summit.. | 15:00 |
| itlinux | inc0: git clone | 15:00 |
| sbezverk | eanylin: http://paste.openstack.org/show/609584/ line 7 | 15:00 |
| inc0 | so it's master:) | 15:00 |
| *** janki has quit IRC | 15:00 | |
| sbezverk | is needed | 15:00 |
| inc0 | way it works today is that it will assume that you install kolla and kolla-ansible in same way | 15:00 |
| inc0 | and we don't have dockerhub images for master yet (working on it) | 15:01 |
| *** gomarivera has quit IRC | 15:01 | |
| inc0 | I'd suggest to deploy ocata | 15:01 |
| itlinux | ok.. | 15:01 |
| inc0 | or build images | 15:01 |
| itlinux | ocata is fine.. what steps should I take? | 15:01 |
| inc0 | to deploy ocata: git checkout stable/ocata | 15:01 |
| inc0 | in kolla-ansible | 15:01 |
| inc0 | then pip install -U . | 15:01 |
| inc0 | and after that change openstack_release: "4.0.0" in globals.yml | 15:02 |
| inc0 | that should help | 15:02 |
| eanylin | sbezverk: Ok, it looks similar to what we pass in too actually, http://paste.openstack.org/show/609586/, line 49 | 15:02 |
| itlinux | do you have ok let me check | 15:02 |
| sbezverk | eanylin: in this case this value again does not get picked up from cloud.yaml | 15:03 |
| inc0 | I have plenty of "let me check"'s :) here, grab one! | 15:03 |
| eanylin | sbezverk: Meaning the format of the yaml has a problem or? | 15:03 |
| sbezverk | eanylin: we need to add --debug to helm chart deploying nova components | 15:03 |
| itlinux | shall see if that goes. inc0 | 15:03 |
| sbezverk | eanylin: right, something similar to we saw for openvswitch | 15:03 |
| eanylin | sbezverk: Debug is turned on by default as well | 15:04 |
| eanylin | sbezverk: Ok, I will re-check again | 15:04 |
| inc0 | brb, gonna grab some breakfast | 15:04 |
| sbezverk | eanylin: ok then if you can get logs generated for novncproxy | 15:04 |
| sbezverk | we can figure out what is going on | 15:04 |
| itlinux | inc0: super.. what are the steps or doc to follow to build a new image.. in case I want to have specs that are not there. TY | 15:05 |
| hrw | portdirect: https://review.openstack.org/#/c/464045/ - please take a look | 15:06 |
| eanylin | sbezverk: Ok, I might need to do it later, too many proxies in my office, things will not work properly... | 15:06 |
| sbezverk | eanylin: sounds good | 15:11 |
| openstackgerrit | pangliye proposed openstack/kolla master: Install the plugin of elasticsearch-head https://review.openstack.org/464592 | 15:12 |
| itlinux | inc0: get stopped at atal: [localhost]: FAILED! => {"attempts": 30, "changed": false, "cmd": ["docker", "exec", "openvswitch_db" | 15:17 |
| *** lpetrut has quit IRC | 15:17 | |
| jamesPR | egonzalez, doesn't work | 15:19 |
| jamesPR | cannot access external IP | 15:19 |
| jamesPR | lol | 15:19 |
| egonzalez | jamesPR, can you check if the ip is bind, share ip a on master node and content of /etc/kolla/keepalived/keepalived.conf | 15:20 |
| jamesPR | egonzalez, https://pastebin.com/raw/WCAj0KLR | 15:22 |
| egonzalez | is 10.50.85.252 bind? check docker logs keepalived | 15:23 |
| *** pcaruana has quit IRC | 15:25 | |
| jamesPR | egonzalez, https://pastebin.com/raw/yAEAysqY | 15:25 |
| itlinux | the openvswitch does not boot up looks like "kolla/centos-binary-openvswitch-db-server:4.0.0 "kolla_start" 11 minutes ago Restarting (1) 4 minutes ago openvswitch_db" inc0: ty | 15:28 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Temp workaround for kubelet 45613 bug https://review.openstack.org/464347 | 15:29 |
| *** jtriley has quit IRC | 15:30 | |
| *** StephenWang1991 has quit IRC | 15:32 | |
| Reepicheep | inc0: I was thinking it was .yml instead of .yaml, ;) but I'm to new to kolla to be sure | 15:34 |
| *** MarginHu has joined #openstack-kolla | 15:35 | |
| *** jascott1 has joined #openstack-kolla | 15:35 | |
| MarginHu | hi, I open a bug https://bugs.launchpad.net/kolla/+bug/1690842 | 15:35 |
| openstack | Launchpad bug 1690842 in kolla "need to add ulimit arg for kolla docker run when use elasticsearch 5.4 " [Undecided,New] | 15:35 |
| jascott1 | o/ | 15:35 |
| *** jamesPR has quit IRC | 15:36 | |
| *** m1dev has joined #openstack-kolla | 15:40 | |
| *** MarginHu has quit IRC | 15:43 | |
| itlinux | anyhow, I will try to rebuild the image today and see what and how it goes inc0: thank you for now.. | 15:44 |
| *** itlinux has quit IRC | 15:45 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove the deprecated option "event_connection" https://review.openstack.org/464673 | 15:47 |
| *** itlinux has joined #openstack-kolla | 15:47 | |
| *** jamesPR has joined #openstack-kolla | 15:48 | |
| *** jrich523 has joined #openstack-kolla | 15:50 | |
| *** jaosorior has joined #openstack-kolla | 15:51 | |
| *** itlinux has quit IRC | 15:52 | |
| *** jtriley has joined #openstack-kolla | 15:53 | |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant and provider VLANs https://review.openstack.org/464675 | 15:56 |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant and provider VLANs https://review.openstack.org/464675 | 16:01 |
| *** egonzalez has quit IRC | 16:04 | |
| *** manheim has quit IRC | 16:11 | |
| *** manheim has joined #openstack-kolla | 16:12 | |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant and provider VLANs https://review.openstack.org/464675 | 16:15 |
| *** erobell has joined #openstack-kolla | 16:15 | |
| *** matrohon has quit IRC | 16:15 | |
| *** manheim has quit IRC | 16:16 | |
| erobell | sdake Fyi, using 1.6.2 and the bare metal instructions still didn't work for me. Anthony Lin however, has posted a review for an AIO script, and that does seem to work. https://review.openstack.org/gitweb?p=openstack/kolla-kubernetes.git;a=commit;h=746180bd03d7196d3ba4696f54d5b7fdbdc1a0f4 | 16:17 |
| inc0 | kfox1111: back in action? | 16:18 |
| kfox1111 | inc0: hey. yeah, 90% back. | 16:20 |
| kfox1111 | able to walk again, and got into work ok. | 16:20 |
| kfox1111 | off the meds too, so my heads not so fuzzy. | 16:20 |
| inc0 | what happend? | 16:20 |
| inc0 | I haven't heard anything! | 16:20 |
| kfox1111 | I dared to cross the street. :) | 16:20 |
| inc0 | omg | 16:20 |
| inc0 | when? | 16:20 |
| kfox1111 | I stepeed off the curb and sproing, something let loose in my back. | 16:21 |
| inc0 | damn | 16:21 |
| kfox1111 | was with steve and ryan. | 16:21 |
| kfox1111 | wasn't too bad. sore, but was able to walk back to the hotel. | 16:21 |
| inc0 | that sucks man, I'm glad you're getting better | 16:21 |
| kfox1111 | but the next morning, I couldn't walk. | 16:21 |
| inc0 | well, I wanted to say that next item on my agenda is dockerhub publisher | 16:22 |
| kfox1111 | talked to my travel support. they wanted me to go to the hospital by ambulence. | 16:23 |
| inc0 | since pre-summit heat subsided, I want to get full speed ahead into this one | 16:23 |
| kfox1111 | inc0: yes please. :) | 16:23 |
| kfox1111 | I'll help with that however possible. | 16:23 |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant and provider VLANs https://review.openstack.org/464675 | 16:23 |
| kfox1111 | it sounded like it was very close already. | 16:23 |
| inc0 | cool, well, I've heard there are some issues we need to deal with, but fungi said that secrets aren't that problematic in fact | 16:24 |
| inc0 | I want to make little more comprehensive toolset for registry mgmt frankly | 16:24 |
| inc0 | like "pull images from dockerhub and load them to my local registry" | 16:25 |
| *** gfidente is now known as gfidente|afk | 16:26 | |
| kfox1111 | yeah, that would be cool. | 16:28 |
| kfox1111 | though, baby steps. | 16:28 |
| inc0 | yeah | 16:28 |
| inc0 | also we need extreme vetting of images | 16:29 |
| inc0 | so we won't push any bad hombres up | 16:29 |
| kfox1111 | hehe | 16:30 |
| bmaltais_ | sbezverk: A capture of the vnc is in there too: https://drive.google.com/file/d/0B5odRXvIkNfJdEZmdnlfVmdLdUE/view?usp=sharing | 16:30 |
| kfox1111 | that will be much easier with zuulv3 where we can do pipelines. but | 16:31 |
| inc0 | I'm a little on the fence with having voting deploy gates | 16:31 |
| kfox1111 | just running them through the koklla-kubernetes gate tests a whole lot of functionality. | 16:31 |
| inc0 | like, k8s change could block merging sth to kolla | 16:31 |
| bmaltais_ | eanylin: sbezverk: This is the same capture we used to trace the openvswitch typo | 16:31 |
| kfox1111 | inc0: yeah. I'm working on a fix for that. | 16:31 |
| kfox1111 | mirroring a known working set of k8s | 16:31 |
| kfox1111 | so if the repo's break, the gate doesn't. | 16:32 |
| bmaltais_ | eanylin: sbezverk: It also include the vnc config so any error shoud show up in there too | 16:32 |
| inc0 | we can have non-voting and expect cores to be careful with deployment gates red, we are today but reemphasize | 16:32 |
| inc0 | that'd work to | 16:32 |
| inc0 | too | 16:32 |
| kfox1111 | yeah. | 16:36 |
| *** gomarivera has joined #openstack-kolla | 16:37 | |
| kfox1111 | I'm working on the k8s cache anyway, just so they will stop breaking us all the time. | 16:37 |
| kfox1111 | it pushes the breakage to a time we can deal with it on our own time. | 16:37 |
| sbezverk | kfox1111: hey man, I compiled kubelet 1.6.3 with that fix to restore the gate | 16:37 |
| *** stefan_bo has joined #openstack-kolla | 16:39 | |
| kfox1111 | sbezverk: oh, cool. thanks. :) | 16:39 |
| sbezverk | kfox1111: I am get allergy on red ;) at the gate | 16:40 |
| kfox1111 | hehe. | 16:40 |
| kfox1111 | +1 | 16:40 |
| kfox1111 | its like walking into a datacenter and its quient. | 16:40 |
| kfox1111 | quiet. | 16:40 |
| kfox1111 | too quiet. | 16:41 |
| kfox1111 | very unnerving. | 16:41 |
| *** vhosakot has joined #openstack-kolla | 16:41 | |
| hrw | inc0, sdake, kfox1111: is there a way to rename image from inside of Dockerfile.j2? | 16:41 |
| inc0 | hrw: no, because images aren't named per se | 16:41 |
| kfox1111 | the image is named via the commandliine. I don't think it can be named at all in the dockerfile. | 16:42 |
| hrw | kfox1111: by commandline I tell which image(s) I want to build | 16:42 |
| inc0 | "name" of image is just random id, names are called tags | 16:42 |
| inc0 | and one image can have multiple | 16:42 |
| kfox1111 | hrw: sorry, was unclear. docker command line. the kolla commandline's different. | 16:42 |
| hrw | inc0: ok, tags | 16:42 |
| inc0 | docker tag << image_id>> << tag you want >> | 16:43 |
| inc0 | if I remember correctly | 16:43 |
| inc0 | that will add new name to existing image | 16:43 |
| *** stefan_bo has quit IRC | 16:43 | |
| hrw | the problem is with kube images. now we have kube-apiserver-amd64 which I renamed to kube-apiserver and would like to be able to build kube-apiserver-SOMEVARWITHARCH | 16:43 |
| *** harlowja has joined #openstack-kolla | 16:44 | |
| inc0 | potentially we could just add arch suffix to all images if arch != amd64 | 16:44 |
| inc0 | but we don't build kube-apiserver images right?;) | 16:44 |
| hrw | inc0: but we need arch mapping too | 16:45 |
| hrw | I plan to make a patch which will add such. so we have base_arch and kernel_arch or debian_arch | 16:45 |
| kfox1111 | I'm kind of hoping we can get rid of kube-apiserver asap. | 16:46 |
| hrw | so aarch64/arm64, x86_64/amd64 (ppc64le/ppc64le anyway) | 16:46 |
| *** david-lyle has quit IRC | 16:46 | |
| *** dklyle has joined #openstack-kolla | 16:46 | |
| kfox1111 | there is an example external cephfs provisioner. | 16:46 |
| kfox1111 | I'm wondering if we could adapt it to do rbd as well. | 16:46 |
| vhosakot | inc0: hope you and Tim had a good time in Boston on Friday. Let me know if you have time later today to discuss how I add contribute to the automated build of master once in 24 hours. | 16:46 |
| kfox1111 | then a stock apiserver would work. | 16:46 |
| inc0 | vhosakot: that's top of my personal list, also mnaser and kfox1111 want to be involved | 16:47 |
| hrw | kfox1111: if someone send patch to kill kube/ directory ;D | 16:47 |
| inc0 | so I think we have pretty strong team | 16:47 |
| inc0 | lots of poeple on summit were asking for it | 16:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/464393 | 16:47 |
| vhosakot | inc0: cool, is there a review/bp I can start looking? | 16:47 |
| inc0 | and I kinda promised that on project update;) | 16:47 |
| *** chas has quit IRC | 16:47 | |
| inc0 | there are specs, give me a second | 16:47 |
| *** chas has joined #openstack-kolla | 16:48 | |
| kfox1111 | vhosakot: the last roadblock is having a vm with a dockerhub credential. | 16:48 |
| kfox1111 | it sounds like infra is working on it and close. | 16:48 |
| kfox1111 | once thats in, most of the rest is relatively easy. | 16:48 |
| *** itlinux has joined #openstack-kolla | 16:48 | |
| vhosakot | kfox1111: ah ok, is the plan to bake the dockerhub creds into the VM, or send/inject the creds encrypted into the VM using Ansible? | 16:49 |
| kfox1111 | vhosakot: not sure the implementation details. all I know is we need them put there somehow securely. however's best for infra. | 16:49 |
| vhosakot | kfox1111: cool cool | 16:50 |
| kfox1111 | we just need to be able to write a cron job that runs there with the creds already there. | 16:50 |
| vhosakot | right | 16:50 |
| vhosakot | kfox1111: don't need to version the mater images right? just overwrite yesterday's master with today's mater? | 16:51 |
| *** StephenWang1991 has joined #openstack-kolla | 16:51 | |
| kfox1111 | yeah, I think latest there is probably fine. | 16:51 |
| vhosakot | cool | 16:51 |
| kfox1111 | well... | 16:51 |
| kfox1111 | I guess it depends... | 16:51 |
| kfox1111 | we have to answer the question, | 16:51 |
| kfox1111 | "is anyone crazy enough to run from trunk for production" ? :) | 16:52 |
| vhosakot | lol right | 16:52 |
| *** chas has quit IRC | 16:52 | |
| kfox1111 | if the answer is yes, we should at least date stamp the images. | 16:52 |
| *** gomarivera has quit IRC | 16:53 | |
| hrw | hm. looks like image_name can not be changed as is is just os.path.basename() | 16:54 |
| inc0 | so I'd suggest starting with pulling http://tarballs.openstack.org/kolla/images/ and pushing them up | 16:54 |
| vhosakot | it is for dev mainly I think and speedens build time at dev time as we can pull atmost 24 hrs old images from master | 16:54 |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant and provider VLANs https://review.openstack.org/464675 | 16:54 |
| inc0 | hrw: well we can modify that | 16:54 |
| inc0 | in python script | 16:54 |
| hrw | inc0: I know | 16:54 |
| inc0 | today we don't have this | 16:54 |
| *** StephenWang1991 has quit IRC | 16:55 | |
| hrw | inc0: kind of workaround would be creation of kube/kube-xyz(-{amd64,arm64}/ dirs which will include generic j2 | 16:55 |
| sbezverk | kfox1111: qq in configmap PS you removed in code configmap creation but I did not find corresponding config template | 16:55 |
| sbezverk | example keystone's fernet | 16:56 |
| *** unicell has joined #openstack-kolla | 16:56 | |
| kfox1111 | sbezverk: the ones not generated by kubectl? | 16:56 |
| sbezverk | kfox1111: right, before configmap for fernet was right in the template, now it is removed, but it should still exist somewhere, I am failing to find it | 16:57 |
| kfox1111 | sbezverk: oh. I think sdake did a rebase and it got lost along thee way? | 16:58 |
| kfox1111 | gotta meeting. bbiab. | 16:58 |
| inc0 | hrw: hmm...I have no good answers for you right now | 16:59 |
| sbezverk | kfox1111: cool, I wanted to confirm.. | 16:59 |
| vhosakot | inc0: should we keep http://tarballs.openstack.org/kolla/images/ update every 24 hrs as well? | 16:59 |
| vhosakot | updated* | 16:59 |
| inc0 | right now they are updated every time something merges | 17:00 |
| inc0 | which happens quite often | 17:00 |
| *** ddyer has quit IRC | 17:01 | |
| inc0 | but yeah at some point we'd need "cron" to kick off gates too | 17:01 |
| inc0 | I think that's possible with infra | 17:01 |
| vhosakot | inc0: cool | 17:01 |
| vhosakot | inc0: are you saying, pull from http://tarballs.openstack.org/kolla/images/ every 24 hrs, untar it, and push images in the untarred file into dockerhub? | 17:02 |
| inc0 | well, tar is tared registry | 17:02 |
| openstackgerrit | Merged openstack/kolla stable/ocata: Fix Incorrect timezone in ubuntu horizon images https://review.openstack.org/464214 | 17:02 |
| vhosakot | ah ok | 17:03 |
| inc0 | so pull tar.gz, untar to /tmp/untarred, docker run -v /tmp/untarred:/var/lib/registry -d registry:2 | 17:03 |
| inc0 | pull all images from this registry, as in docker pull 127.0.0.1:5000/lokolla/ubuntu-source-base:4.0.0 | 17:04 |
| vhosakot | got it | 17:04 |
| inc0 | notice change of namespace and tag | 17:04 |
| inc0 | that should be fixed at some point too | 17:04 |
| vhosakot | right | 17:04 |
| inc0 | then docker tag 127.0.0.1:5000/lokolla/ubuntu-source-base:4.0.0 docker.io/kolla/ubuntu-source-base:4.0.0 | 17:04 |
| vhosakot | inc0: can you send the pointer to the spec so i can read up | 17:04 |
| *** mgoddard__ has quit IRC | 17:05 | |
| inc0 | docker push docker.io/kolla/ubuntu-source-base:4.0.0 | 17:05 |
| *** ddyer has joined #openstack-kolla | 17:06 | |
| *** Serlex has quit IRC | 17:08 | |
| *** chas has joined #openstack-kolla | 17:08 | |
| openstackgerrit | Merged openstack/kolla master: introduce 'install_kubectl' macro https://review.openstack.org/464184 | 17:09 |
| hrw | o! | 17:10 |
| hrw | I have to admit that it this patch was more like "let's see does it make sense" | 17:11 |
| hrw | inc0: how hub.docker is updated with u/kolla/ images? CI job? | 17:12 |
| bmaltais_ | eanylin: The new script work fine now for external access. Looking into the VNC issue now. | 17:12 |
| inc0 | hrw: no, my laptop when I feel like it;)_ | 17:13 |
| inc0 | or rather, after releases | 17:13 |
| hrw | inc0: ok | 17:13 |
| inc0 | now we're talking about making it proper | 17:13 |
| hrw | inc0: asking bcause I am thinking of u/kolla-aarch64 u/kolla-ppc64le stuff | 17:13 |
| *** chas has quit IRC | 17:13 | |
| hrw | inc0: with this amount of images I think that it would be easiest way. no other namespaces needed | 17:14 |
| inc0 | vhosakot: https://review.openstack.org/#/c/447524/ | 17:14 |
| hrw | inc0: and I do not want to use u/linaro for that as other linaro images would drown in kolla flood | 17:14 |
| inc0 | yeah, with that we'll run into problems of not having gates | 17:14 |
| vhosakot | inc0: I have used "docker save" to convert an image into a tarfile and then "docker load" or "docker import" to convert this tar file back into an image. how are the tar files of the _entire_ registry created in http://tarballs.openstack.org/kolla/images/ in the first place ? using "docker save"? | 17:14 |
| hrw | inc0: xinliang is working on 3rdparty CI for openstack | 17:15 |
| vhosakot | inc0: cool, thanks for the link, I'll read it! | 17:15 |
| inc0 | vhosakot: no...using tar;) | 17:15 |
| inc0 | we really tar.gz volume with registry;0 | 17:15 |
| vhosakot | ah I see, cool | 17:15 |
| inc0 | taring image by image will cost you *a lot* of space | 17:15 |
| inc0 | and by a lot I mean 100gig+ | 17:15 |
| vhosakot | inc0: yes, so the entire directory /var/lib/registry in the registry containers is tarred? | 17:16 |
| inc0 | yeah | 17:16 |
| vhosakot | inc0: ah got it, cool. | 17:17 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: kubernetes: get it working on AArch64 as well https://review.openstack.org/464045 | 17:17 |
| hrw | portdirect: not needed line removed ;D | 17:17 |
| hrw | sdake, pbourke: https://review.openstack.org/#/c/464300/ is small cleanup of base/Dockerfile.j2 to make it a bit more readable. Can you take a look? | 17:19 |
| *** pbourke has quit IRC | 17:20 | |
| sdake | hrw enjoy | 17:20 |
| sdake | inc0 nice work :) | 17:21 |
| hrw | sdake: thanks | 17:21 |
| inc0 | which one sdake ? | 17:21 |
| hrw | inc0: split base into rpm_ apt_ | 17:21 |
| inc0 | yeah one thing we definetly need to work on is readibility | 17:22 |
| *** pbourke has joined #openstack-kolla | 17:22 | |
| *** egonzalez has joined #openstack-kolla | 17:22 | |
| *** mgoddard__ has joined #openstack-kolla | 17:24 | |
| kfox1111 | back. | 17:26 |
| kfox1111 | vhosakot: the kolla-kubernetes gate is building them with docker save. | 17:26 |
| kfox1111 | the kolla version is pushing them to a docker repo then saving the repo's backing store. | 17:26 |
| vhosakot | kfox1111: ah I see, you mean http://tarballs.openstack.org/kolla-kubernetes/gate/containers/ ? | 17:28 |
| kfox1111 | yeah. | 17:28 |
| vhosakot | cool, good to push tested containers in gate. | 17:28 |
| kfox1111 | yeah. its constantly retesting. | 17:29 |
| kfox1111 | though I don't have trun testing merged quite yet. once thats done, it should push the tested ones from kolla. | 17:30 |
| kfox1111 | then we will know for sure that they work. | 17:30 |
| kfox1111 | that tied in with docker hub pushing should close the loop and we'd have fully tested stuff released nightly. | 17:31 |
| *** jamesPR has quit IRC | 17:36 | |
| vhosakot | kfox1111: when are they created and re-tested? crated the end of a passed gate using "docker save"? where are they re-tested? at gate? | 17:37 |
| kfox1111 | there is a set of periodic gates taht run nightly. | 17:37 |
| *** dklyle is now known as david-lyle | 17:37 | |
| kfox1111 | they do a full kolla-kubernetes gate test with the pulled images, | 17:38 |
| kfox1111 | then if they are good, docker save everything and push it back up. | 17:38 |
| vhosakot | kfox1111: ah cool, yeah, I have heard about it.. cool | 17:38 |
| kfox1111 | I've still got a little logic to put into them to only push on changes. | 17:38 |
| vhosakot | cool | 17:40 |
| bmaltais_ | sbezverk: eanylin: I see a lot of this in the debug output... host: nova-novncproxy | 17:41 |
| bmaltais_ | sbezverk: eanylin: I think this is where the issue is coming from... | 17:41 |
| sbezverk | bmaltais_: you can email me the whole log sbezverk@cisco.com | 17:41 |
| bmaltais_ | sbezverk: eanylin: Is there a way to manually re-apply a halm chart so I would push an update to cloud.yml? | 17:42 |
| bmaltais_ | sbezverk: Sure. I am sending it. | 17:42 |
| sbezverk | bmaltais_: depending how it was deployed | 17:43 |
| sbezverk | remember helm works with release | 17:43 |
| kfox1111 | bmaltais_: helm upgrade <releasename> <chartname> --values cloud.yaml | 17:43 |
| sbezverk | release could be microchart/service chart or compute chart | 17:43 |
| kfox1111 | "upgrade" is a bit of a misnomer. | 17:43 |
| kfox1111 | "update" might have been a better word choice. | 17:43 |
| bmaltais_ | kfox1111: I was deployed using: helm install --debug /opt/kolla-kubernetes/helm/service/nova-compute --namespace kolla --name nova-compute --values /opt/cloud.yaml | 17:45 |
| bmaltais_ | kfox1111: So to update it would be: helm upgrade /opt/kolla-kubernetes/helm/service/nova-compute nova-compute --values /opt/cloud.yaml ? | 17:46 |
| sbezverk | bmaltais_: in this case you can potentially delete nova-control release | 17:46 |
| sbezverk | and then re-deploy it with new cloud.yaml | 17:46 |
| kfox1111 | something like: helm upgrade --namespace kolla nova-compute kolla/nova-compute --values /opt/cloud.yaml | 17:47 |
| sbezverk | novncproxy is part of nova-control | 17:47 |
| bmaltais_ | sbezverk: I guess it was part of compute... but I guess wrong. Which helm chart is it part of? | 17:48 |
| sbezverk | nova-control | 17:48 |
| *** itlinux has quit IRC | 17:50 | |
| sbezverk | bmaltais_: can you dump configmap for nova? | 17:53 |
| sbezverk | I think there might be a wrong url is coming from there | 17:53 |
| bmaltais_ | sbezverk: I am trying to push an update of cloud.yml | 17:55 |
| bmaltais_ | I think the issue is with novncproxy: | 17:55 |
| bmaltais_ | all: | 17:55 |
| bmaltais_ | host: nova-novncproxy | 17:55 |
| bmaltais_ | The host: value should be the public IP instead of nova-novncproxy | 17:56 |
| *** egonzalez has quit IRC | 17:57 | |
| bmaltais_ | sbezverk: But I might be totally wrong... Waiting for POD to come back up | 17:58 |
| *** shardy is now known as shardy_afk | 17:59 | |
| sbezverk | bmaltais_: cool, let me know the result | 17:59 |
| *** gfidente|afk is now known as gfidente | 18:00 | |
| *** shardy_afk is now known as shardy | 18:00 | |
| bmaltais_ | sbezverk: Well... I totally hosed my k8 POD trying to push the update | 18:00 |
| bmaltais_ | sbezverk: There are a bunch of POD in error and they won't come back | 18:01 |
| bmaltais_ | sbezverk: Will go back to a working snapshot | 18:01 |
| kfox1111 | sbezverk: sounds like 1.6.4 is imminant to fix our regression. | 18:02 |
| bmaltais_ | sbezverk: Perhaps I should just start fresh and do a deploy where the host: is specified as the IP instead | 18:02 |
| sbezverk | kfox1111: right, as soon as it is out I will remove the workaround | 18:02 |
| *** shardy is now known as shardy_afk | 18:02 | |
| kfox1111 | cool. | 18:03 |
| *** gomarivera has joined #openstack-kolla | 18:03 | |
| *** mannamne has quit IRC | 18:03 | |
| *** schwicht has quit IRC | 18:03 | |
| *** sambetts is now known as sambetts|afk | 18:04 | |
| *** hongbin has joined #openstack-kolla | 18:07 | |
| openstackgerrit | Merged openstack/kolla master: Clean up base Dockerfile https://review.openstack.org/464300 | 18:08 |
| hrw | yay! | 18:08 |
| *** schwicht has joined #openstack-kolla | 18:10 | |
| *** egonzalez has joined #openstack-kolla | 18:12 | |
| *** erobell_ has joined #openstack-kolla | 18:13 | |
| *** erobell has quit IRC | 18:16 | |
| *** bmace__ has quit IRC | 18:16 | |
| openstackgerrit | Pete Birley proposed openstack/kolla master: Base image: fix formatting of rpm_distros template https://review.openstack.org/464714 | 18:17 |
| *** bmace__ has joined #openstack-kolla | 18:17 | |
| *** krtaylor has quit IRC | 18:19 | |
| bmaltais_ | sbezverk: That was the issue. Working peachy now | 18:26 |
| sbezverk | bmaltais_: nice | 18:26 |
| sbezverk | bmaltais_: what was changed exactly with regards to vnc? too many things were discussed.. | 18:27 |
| bmaltais_ | eanylin: I have a proposed change for the script to fix the novnc issue | 18:27 |
| bmaltais_ | sbezverk: Here is the updated section for eanylin: script that take care of it: http://paste.openstack.org/show/609599/ | 18:28 |
| sbezverk | bmaltais_: cool | 18:29 |
| bmaltais_ | sbezverk: Now, if we could front the current IP with a VIP it would be really nice | 18:29 |
| bmaltais_ | sbezverk: That way horizon could run on any node in k8 and be reachable using the haproxy VIP | 18:30 |
| bmaltais_ | Similar to kolla-ansible | 18:30 |
| *** gomarivera has quit IRC | 18:31 | |
| bmaltais_ | sbezverk: I notice that haproxy is not enabled in the global file. Is there a reason why? Perhaps no helm chart ready for it yet? | 18:31 |
| *** gomarivera has joined #openstack-kolla | 18:32 | |
| bmaltais_ | eanylin: You can find the proposed change for VNC in http://paste.openstack.org/show/609599/ | 18:34 |
| *** mgoddard__ has quit IRC | 18:39 | |
| *** gomarivera has quit IRC | 18:41 | |
| *** gomarivera has joined #openstack-kolla | 18:43 | |
| jascott1 | haha portdirect said 'cloud lego' in the kk8s day2 talk | 18:49 |
| portdirect | great artists steal jascott1 :P | 18:50 |
| jascott1 | :D | 18:50 |
| jascott1 | portdirect good job btw | 18:50 |
| jascott1 | all you guys | 18:50 |
| portdirect | yeah sbezverk's demo was awesome | 18:51 |
| portdirect | and that guy srwilkers sounds like he knows his stuff too | 18:51 |
| portdirect | cheers dude | 18:51 |
| kfox1111 | bmaltais_: keepalived can run a vip on whichever hosts you want and endpoint svc's will get the traffic where it needs to go. | 18:57 |
| kfox1111 | you really should have haproxy enabled unless you are very tight on memory. | 18:58 |
| kfox1111 | its very useful. | 18:58 |
| bmaltais_ | kfox1111: Good to know. The AIO guide does not enable it and the VIP is pointing to the IP of the 1st NIC on the server... not very scalable right now | 18:59 |
| kfox1111 | bmaltais_: yeah, we should probably update the guide. | 19:00 |
| bmaltais_ | kfox1111: Really it should use a VIP that is not assigned to any of the k8 nodes and that one point to the horizon DB | 19:00 |
| kfox1111 | bmaltais_: exactly. | 19:00 |
| bmaltais_ | Once that is changes it should be easy to add node to k8 and make sure horizon remain available. ATM if the master node goes down then kaboum. | 19:01 |
| bmaltais_ | kfox1111: I tried setting the VIP to something else than the physical NIC but that failed. Is it because haproxy was not enabled? Is it that easy to make it work? | 19:03 |
| bmaltais_ | kfox1111: Obviously it was an IP part of the subnet... So instead of 10.10.10.219 I tried 10.10.10.254 | 19:03 |
| bmaltais_ | kfox1111: But no joy | 19:03 |
| *** manheim has joined #openstack-kolla | 19:06 | |
| *** manheim has quit IRC | 19:10 | |
| *** jamesPR has joined #openstack-kolla | 19:10 | |
| *** mgoddard__ has joined #openstack-kolla | 19:12 | |
| *** chas has joined #openstack-kolla | 19:14 | |
| *** devananda has quit IRC | 19:16 | |
| kfox1111 | yeah, something like that should work. | 19:16 |
| kfox1111 | gotta make sure the external ip's for all the services are pointing to it too. | 19:17 |
| kfox1111 | not sure if thats easily reconfigurable after its deployed. | 19:17 |
| bmaltais_ | kfox1111: Trying it right now. Enabled haproxy and configured vip to 10.10.10.254 | 19:17 |
| kfox1111 | man... you go away for a week and everything falls apart. :/ | 19:17 |
| bmaltais_ | kfox1111: The deployment script from eanylin: takes care of all those changes | 19:18 |
| kfox1111 | bmaltais_: cool. | 19:18 |
| *** chas has quit IRC | 19:18 | |
| kfox1111 | wouldn't totally bet on it though. the ip gets woven into config files and mariadb in places I think. | 19:18 |
| bmaltais_ | kfox1111: BEst would be to use a fqdn instead of IP | 19:19 |
| bmaltais_ | kfox1111: But that could also be problematic if DNS change | 19:19 |
| kfox1111 | bmaltais_: yeah. usually if AIO, you don't have too much dns access. | 19:20 |
| bmaltais_ | kfox1111: What I would like is to start with an AIO and then have the possibility to add more minions | 19:20 |
| kfox1111 | yeah. that would be good. not sure anyone's flushed out the docs well for that use case yet. but if you figure it out, lets get it doc'ed. :) | 19:21 |
| bmaltais_ | kfox1111: So no ties to any IP of the master | 19:21 |
| bmaltais_ | kfox1111: Sure | 19:21 |
| bmaltais_ | kfox1111: I can already see some issues if other nodes don't have matching internace numbering with master | 19:22 |
| bmaltais_ | kfox1111: This would probably break the openvswitch networking | 19:22 |
| bmaltais_ | kfox1111: Would be nice if this mapping was not so tied to actual interface name | 19:23 |
| bmaltais_ | kfox1111: But I might be wrong. I am fairly new to k8s. | 19:24 |
| bmaltais_ | kfox1111: Look like enabling haproxy and setting VIP to 10.10.10.254 does not work. I might be missing some specific config to enable it? | 19:25 |
| bmaltais_ | kfox1111: Nothing is actually listening on 10.10.10.254... look like haproxy does not know it need to listen on that IP | 19:26 |
| bmaltais_ | kfox1111: And what to front with it | 19:26 |
| kfox1111 | bmaltais_: no way to not tie it to an interface. but there is an override per host in the pod. | 19:27 |
| bmaltais_ | kfox1111: I see | 19:27 |
| kfox1111 | keepalived needs to put a vip up. | 19:27 |
| *** gomarivera has quit IRC | 19:28 | |
| bmaltais_ | kfox1111: How would I do this via cloud.yml? Is it where it would get set? | 19:28 |
| *** gomarivera has joined #openstack-kolla | 19:28 | |
| kfox1111 | which part are you asking about? | 19:28 |
| bmaltais_ | The keepalived for VIP part | 19:29 |
| kfox1111 | bmaltais_: https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/ceph_workflow.sh#L526 | 19:30 |
| bmaltais_ | kfox1111: So that 10.10.10.254 would be up and able to act as the VIP | 19:30 |
| *** krtaylor has joined #openstack-kolla | 19:30 | |
| *** mannamne has joined #openstack-kolla | 19:31 | |
| kfox1111 | I think you need a keepalived configmap too. | 19:31 |
| kfox1111 | probably from kollakube | 19:31 |
| pomac | Anyone with a clue of how qemu://system works? Is it shm based? | 19:31 |
| kfox1111 | pomac: what do you mean? | 19:31 |
| bmaltais_ | kfox1111: Thank's. Will do some reading | 19:31 |
| kfox1111 | pomac: like, how authentication works? | 19:32 |
| kfox1111 | bmaltais_: cool. ping me if you get stuck. | 19:32 |
| *** gomarivera has quit IRC | 19:33 | |
| pomac | kfox1111: we have a issue that is isolated to old qemu instances around (since they are spawned with systemd-nspawn) and a restarted nova_libvirt - basically the new instance of nova_libvirt will not work until all qemu instances are shutdown - docker says something about <some container fs>/shm on startup | 19:33 |
| pomac | kfox1111: basically, start instances, restart nova_libvirt - watch all new machines fail and all volume attachments fail - seems to happen with newer versions of docker | 19:34 |
| kfox1111 | pomac: oh, weird. | 19:34 |
| kfox1111 | never played with that particular set of interfaces. | 19:34 |
| pomac | kfox1111: exactly, I spent most of the day working that one out... My friend found it | 19:34 |
| *** chas has joined #openstack-kolla | 19:34 | |
| kfox1111 | inc0: have you seen that? ---^ | 19:34 |
| pomac | docker 17.5 (and all the way down to 17.3) seems to suffer from this issue | 19:36 |
| *** athomas has quit IRC | 19:39 | |
| *** chas has quit IRC | 19:39 | |
| *** mgoddard__ has quit IRC | 19:40 | |
| *** athomas has joined #openstack-kolla | 19:41 | |
| pomac | kfox1111, inc0: if you know anything that we should try to look at to get more details, please let me know | 19:43 |
| kfox1111 | docker 17.5? thats really new... | 19:43 |
| kfox1111 | I usually stick to the distro versions. 1.12 or so. | 19:44 |
| kfox1111 | docker's been going and doing some weird stuff recently. | 19:44 |
| vhosakot | kfox1111: docker changed versioning format... it is not year.month | 19:44 |
| vhosakot | not==mow | 19:44 |
| kfox1111 | most of the community is abandoning docker for containerd. (which is essentially docker 1.12) | 19:44 |
| vhosakot | now* | 19:44 |
| kfox1111 | docker got renamed to containerd, and docker swarm got renamed to docker. :/ | 19:45 |
| *** scheuk has joined #openstack-kolla | 19:45 | |
| kfox1111 | some details here: http://containerd.io/ if your interested. | 19:46 |
| *** bmaltais_ has quit IRC | 19:46 | |
| pomac | kfox1111: 1.12.6 i hope, it fixes a cve ;) | 19:46 |
| kfox1111 | yeah. redhat stays on top of things pretty well. | 19:47 |
| kfox1111 | I'm sure ubuntu does too. | 19:47 |
| kfox1111 | they just don't want to push forward with swarm integration. | 19:47 |
| kfox1111 | since it forces a competitors product into their own. | 19:47 |
| pomac | kfox1111: well, all i know is that it's called docker-engine now =) | 19:48 |
| pomac | kfox1111: but container.io is not a dropin replacement ala mariadb i assume? | 19:48 |
| kfox1111 | kind of. its always kind of had that name. | 19:48 |
| kfox1111 | 1.11 and 1.12 use it internally. | 19:48 |
| kfox1111 | then they are moving more and more functionality into it and in its place putting swarm stuff. so if you look at: http://containerd.io/ | 19:49 |
| kfox1111 | at the very bottom image, you can see where they want to go with it. | 19:49 |
| kfox1111 | docker sits on top of swarm, on top of containerd, | 19:49 |
| *** gomarivera has joined #openstack-kolla | 19:49 | |
| kfox1111 | and everyone else does too. but hardly anything thats using docker today is using docker anymore in that picture. | 19:49 |
| kfox1111 | k8s, mesos, cf and aws all are moving away from docker. | 19:50 |
| pomac | kfox1111: I'm fine with that, as long as it doesn't cause problems with kolla - atleast for now =) | 19:51 |
| dave-mccowan | i'm following the kolla-k8s deployement guide for my maiden voyage. | 19:51 |
| dave-mccowan | i'm getting the following error: | 19:51 |
| kfox1111 | yeah. not sure what ansible's going to do with respect with docker, but since redhat owns ansible, I"m pretty sure it will retarget containderd too. | 19:51 |
| dave-mccowan | error: unable to decode "canal.yaml": Object 'Kind' is missing in '{"404":"Not Found"}' | 19:51 |
| pomac | kfox1111: right now, the target is just getting thinsg working stable enough to set up a lab for further development but when users can't use it... | 19:51 |
| vhosakot | docker does not exist anymore :) https://github.com/docker/docker redirects to https://github.com/moby/moby | 19:51 |
| kfox1111 | dave-mccowan: have a look at canal.yaml in a text editor. did it fail to download ok? | 19:52 |
| pomac | vhosakot: yeah, noticed that earlier and there is nom 17.5 changelog | 19:52 |
| kfox1111 | vhosakot: moby's an even different animal. | 19:52 |
| kfox1111 | due to containerd, | 19:52 |
| kfox1111 | docker really is docker-swarm. | 19:52 |
| kfox1111 | so moby's the open source version fo docker-swarm. | 19:52 |
| vhosakot | ah cool | 19:52 |
| kfox1111 | for all those that want an open source version of docker swarm instead of just using k8s... | 19:53 |
| kfox1111 | not that I'm biased or anything. :) | 19:53 |
| kfox1111 | (*cough* k8s's better *cough* ;) | 19:53 |
| pomac | kfox1111: sounds like that should be </shameless-plug> | 19:53 |
| pomac | =) | 19:53 |
| dave-mccowan | kfox1111 yea, that's it. couldn't download canal.yaml. | 19:54 |
| kfox1111 | pomac: they really anoyed so many poeople when they renamed docker swarm to docker. most of their contributors left. | 19:54 |
| kfox1111 | so thats why they had to do a big anouncement around moby. to try and get non docker inc folks to try and contribute again. | 19:55 |
| pomac | kfox1111: ah | 19:55 |
| *** shardy_afk has quit IRC | 19:55 | |
| pomac | kfox1111: never really paid attention to docker stuff.... so ... =) | 19:56 |
| *** chas has joined #openstack-kolla | 19:56 | |
| kfox1111 | yeah. its rpetty recent too. | 19:57 |
| kfox1111 | only happened in the last few months. | 19:57 |
| kfox1111 | so all the companies are scrambling to get docker out. | 19:57 |
| kfox1111 | k8s 1.6 included a beta plugin by default that gets rid of direct docker access. | 19:57 |
| *** scheuk has quit IRC | 19:57 | |
| kfox1111 | they really want it gone asap. | 19:58 |
| kfox1111 | I don't really blame them. | 19:58 |
| *** scheuk has joined #openstack-kolla | 19:58 | |
| pomac | kfox1111: outch, sounds like a company-killing-move^tm | 19:58 |
| kfox1111 | it may end up really killing docker inc, but they have one thing going for them. | 19:59 |
| kfox1111 | people are still requesting "docker" training. | 19:59 |
| kfox1111 | which means you ask for it today, and they give you "swarm training" in descise. | 19:59 |
| kfox1111 | if you don't pay close attention, this locks you into their own product. | 19:59 |
| *** jamesPR has quit IRC | 19:59 | |
| *** scheuk has quit IRC | 19:59 | |
| kfox1111 | that may pay off for them if enough people aren't paying attention. :/ | 19:59 |
| pomac | kfox1111: i just remember seeing that curl somesite/file.sh |bash - which caused me to go... Naaah | 20:00 |
| *** chas has quit IRC | 20:00 | |
| rwellum | kfox1111: you make it sound like kolla-kubernetes is not using docker? | 20:00 |
| kfox1111 | rwellum: kolla-kubernetes is agnostic to runtime. should work with whatever k8s is setup to use. so containerd once ready. hyper shoudl in theory work now, though never tested. | 20:01 |
| rwellum | Right - but we haven't cut the cord yet | 20:01 |
| kfox1111 | thats why docker inc pushed so hard swarm. they realized k8s didn't really need docker. | 20:01 |
| kfox1111 | and if people adopted k8s then they didnt really need docker inc. | 20:01 |
| kfox1111 | so they are tryign to capitalize on the name very quickly before people realize whats happening. | 20:02 |
| kfox1111 | I think k8s 1.6 may work with containerd without docker. but havent tried it. | 20:02 |
| kfox1111 | k8s 1.6's default backend is cri based which I think talks directly to containerd. | 20:03 |
| kfox1111 | haven't verified though. | 20:03 |
| kfox1111 | but I know k8s is pushing as hard as they can to ditch it. I would be surprised if k8s 1.7 wasnt fully docker less out of the box. | 20:04 |
| rwellum | Very interesting | 20:04 |
| pomac | but containerd is/was docker? | 20:04 |
| kfox1111 | especially since containerd and k8s are both under the cncf now. | 20:04 |
| kfox1111 | yeah. containerd is basically docker from around 1.11. | 20:04 |
| kfox1111 | the stuff you normally think of as docker. | 20:04 |
| pomac | well, the link is github.com/docker/containerd | 20:05 |
| kfox1111 | docker run xxxx... docker stop xxxx. docker rm xxxx. etc. | 20:05 |
| pomac | that then redirects to containerd/containerd | 20:05 |
| kfox1111 | pomac: yeah. it was donated by docker to the cncf. see: https://www.cncf.io/ | 20:05 |
| pomac | AH | 20:05 |
| *** StephenWang1991 has joined #openstack-kolla | 20:06 | |
| kfox1111 | the new docker is docker-compose+docker-swarm together. a COE that competes with kubernetes. | 20:06 |
| kfox1111 | so, basically it will hurt less to just stick to the distro's docker version until containerd fully replaces it. | 20:07 |
| kfox1111 | unless you really want docker-swarm. | 20:07 |
| rwellum | Does that mean containerd is missing a lot of development / fixes? | 20:08 |
| pomac | kfox1111: i think we did the switch because of some snapshoting issue with btrfs | 20:08 |
| kfox1111 | no, its being actively worked on by all the companies that use to work on docker. | 20:08 |
| kfox1111 | so google, redhat, ubuntu, coreos, etc. | 20:08 |
| sdake | when you run docker, it runs containerd in the background | 20:08 |
| sdake | the docker daemon is being removed from the equation | 20:08 |
| *** athomas has quit IRC | 20:09 | |
| kfox1111 | the github site shows a commit 3 days ago. | 20:09 |
| pomac | kfox1111: i mean, thats why we upgraded, will reevaluate that decision tomorrow | 20:09 |
| sdake | i wouldn't worry about the docker daemon going anywhere soon tho :0 | 20:09 |
| pomac | sdake: i just want things to work - even with newer kernels, =) | 20:10 |
| kfox1111 | yeah. if you stick to the distro version of docker, it will be stable, and the distros will provide a smooth upgrade path to containerd once its fully standalone. | 20:10 |
| *** StephenWang1991 has quit IRC | 20:10 | |
| kfox1111 | if you go with docker inc's version, I'm sure they will try and keep you in swarm. | 20:10 |
| pomac | kfox1111: if i can avoid the errors with btrfs - then sure, the distro/epel version will be fine | 20:12 |
| pomac | kfox1111: i'd actually prefer it - just need something that works ;) | 20:12 |
| *** ducttap__ has joined #openstack-kolla | 20:13 | |
| *** ducttape_ has quit IRC | 20:14 | |
| kfox1111 | been running the distro ver with btrfs so far without incident. | 20:14 |
| kfox1111 | unlike the dm-thin provisioned thing. that ate my containers several times. | 20:15 |
| *** rwsu has quit IRC | 20:15 | |
| pomac | kfox1111: we had issues with cgroups and lockups - which has been fixed with more recent kernels, so we try to stick close to mainline | 20:15 |
| pomac | kfox1111: btrfs has been working nicely for me for maaaaaaaaany years | 20:15 |
| kfox1111 | ah. | 20:15 |
| kfox1111 | my brother had a btrfs go south on him about 5 years back. | 20:16 |
| kfox1111 | much has changed since then though. | 20:16 |
| pomac | kfox1111: (last time i had a incident, it caused chris to push a patch to stable ASAP) | 20:16 |
| *** jascott1 has quit IRC | 20:16 | |
| pomac | kfox1111: there was some issue a long way back, but been running it since it was merged almost - =) | 20:16 |
| pomac | kfox1111: more reliable than ext4 if you have poweroutages etc | 20:16 |
| kfox1111 | I run all my containers ephemeral anyway, so I figure if I loose one, its not so bad. | 20:17 |
| kfox1111 | I wouldn't use it for perminant storage yet though. | 20:17 |
| kfox1111 | ceph works really well for that though. | 20:17 |
| pomac | yeah, ceph tends to use xfs which is nice and performant but really annoying to have as root fs | 20:18 |
| kfox1111 | true. | 20:18 |
| pomac | talked to Cristoph about the differens back in... 2011? heck has it been that long... | 20:19 |
| *** manheim has joined #openstack-kolla | 20:19 | |
| kfox1111 | cockroachdb just hit 1.0... interesting... :) | 20:19 |
| rwellum | kfox1111: when generating the kolla config maps I am seeing: "Unable to locate nova-compute-ironic directory in search_paths=[/etc/kolla, /etc/kolla-kubernetes, /usr/lib/python2.7/kolla/etc/kolla, /usr/lib/python2.7/site-packages/etc/kolla-kubernetes]" - has something moved? | 20:21 |
| inc0 | yeah | 20:21 |
| inc0 | since cocroach is sql | 20:21 |
| inc0 | I wonder how hard would it be to use it instead of maria | 20:22 |
| *** ducttap__ has quit IRC | 20:23 | |
| *** ducttape_ has joined #openstack-kolla | 20:23 | |
| *** schwicht has quit IRC | 20:24 | |
| *** ducttap__ has joined #openstack-kolla | 20:24 | |
| *** ducttape_ has quit IRC | 20:25 | |
| *** jascott1 has joined #openstack-kolla | 20:28 | |
| sdake | got my rack put together - took 3-4 hours | 20:29 |
| sdake | and got one server put in it | 20:29 |
| sdake | ordered 2 more and they are in the mail :) | 20:29 |
| * sdake so excited | 20:29 | |
| jascott1 | woot | 20:29 |
| *** manheim has quit IRC | 20:29 | |
| *** manheim has joined #openstack-kolla | 20:30 | |
| sbezverk | kfox1111: I guess demo at summit was reason enough to go 1.0 ;) | 20:30 |
| *** gomarivera has quit IRC | 20:31 | |
| *** rhallisey has quit IRC | 20:33 | |
| rwellum | sbezverk: :) | 20:35 |
| *** manheim has quit IRC | 20:35 | |
| *** chas has joined #openstack-kolla | 20:37 | |
| *** stefan_bo has joined #openstack-kolla | 20:40 | |
| *** chas has quit IRC | 20:42 | |
| *** gomarivera has joined #openstack-kolla | 20:42 | |
| *** stefan_bo has quit IRC | 20:44 | |
| *** schwicht has joined #openstack-kolla | 20:50 | |
| *** dave-mccowan has quit IRC | 20:52 | |
| *** devananda has joined #openstack-kolla | 20:52 | |
| *** rwallner_ has quit IRC | 20:53 | |
| *** schwicht has quit IRC | 20:55 | |
| *** rwallner_ has joined #openstack-kolla | 20:56 | |
| *** Manheim has joined #openstack-kolla | 20:58 | |
| *** jtriley has quit IRC | 20:58 | |
| *** gomarivera has quit IRC | 20:59 | |
| *** srwilkers has joined #openstack-kolla | 21:00 | |
| *** manheim_ has joined #openstack-kolla | 21:02 | |
| *** gomarivera has joined #openstack-kolla | 21:04 | |
| *** nanliu has joined #openstack-kolla | 21:06 | |
| *** schwicht has joined #openstack-kolla | 21:06 | |
| *** scheuk has joined #openstack-kolla | 21:08 | |
| *** scheuk has quit IRC | 21:09 | |
| *** scheuk has joined #openstack-kolla | 21:10 | |
| *** jascott1 has quit IRC | 21:11 | |
| *** jascott1 has joined #openstack-kolla | 21:12 | |
| *** jascott1_ has joined #openstack-kolla | 21:15 | |
| *** jascott1 has quit IRC | 21:15 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Do Not Merge, kolla-kubernetes gate tests https://review.openstack.org/461063 | 21:15 |
| *** hrw has quit IRC | 21:17 | |
| vhosakot | sdake: cool, which servers? UCS C-240s? | 21:18 |
| *** chas has joined #openstack-kolla | 21:19 | |
| *** jascott1_ has quit IRC | 21:19 | |
| Reepicheep | inc0: just watched the Kolla Project Update, You mentioned for documentation that use cases would be helpfull. Where are those being collected? | 21:20 |
| inc0 | hey | 21:23 |
| inc0 | so | 21:23 |
| inc0 | https://github.com/openstack/kolla-ansible/tree/master/doc | 21:23 |
| inc0 | all these files corresponds with docs | 21:23 |
| inc0 | doc pages | 21:23 |
| *** chas has quit IRC | 21:24 | |
| inc0 | so you just edit these | 21:24 |
| vhosakot | inc0: I think Ryan said we need a new directory "Ops_Best_Practices" or something like that. Don't think we need to mix them with https://github.com/openstack/kolla-ansible/tree/master/doc which are our docs to standup dev env. | 21:26 |
| *** rwallner_ has quit IRC | 21:27 | |
| inc0 | well, yeah I guess you're right | 21:27 |
| *** rwallner has joined #openstack-kolla | 21:30 | |
| hawi | hi there. i do have question - why horizon dashboard keeps throwing me out, when i access it - it is some sort persistence problem for loadbalancer in front of horizon? | 21:37 |
| *** schwicht has quit IRC | 21:38 | |
| *** chas has joined #openstack-kolla | 21:40 | |
| *** mannamne has quit IRC | 21:42 | |
| *** chas has quit IRC | 21:44 | |
| *** krtaylor has quit IRC | 21:48 | |
| vhosakot | Reepicheep: are you planning to add a ops best practices doc? if so, cheers! :) | 21:50 |
| vhosakot | I'm planning to add two (one for blade servers, one for rack-mount servers). | 21:50 |
| *** jamesPR has joined #openstack-kolla | 21:51 | |
| jamesPR | hi all ;) | 21:51 |
| *** rhallisey has joined #openstack-kolla | 21:51 | |
| kfox1111 | back. | 21:53 |
| vhosakot | jamesPR: were you able to get SSL in horizon? :) | 21:54 |
| *** scheuk_ has joined #openstack-kolla | 21:55 | |
| *** scheuk has quit IRC | 21:55 | |
| *** srwilkers has quit IRC | 21:55 | |
| *** srwilkers has joined #openstack-kolla | 21:56 | |
| *** schwicht has joined #openstack-kolla | 21:59 | |
| *** scheuk_ has quit IRC | 21:59 | |
| *** rwallner has quit IRC | 21:59 | |
| jamesPR | vhosakot, no but I think I found the issue right now :D | 22:01 |
| vhosakot | cool | 22:01 |
| *** chas has joined #openstack-kolla | 22:01 | |
| *** rwallner_ has joined #openstack-kolla | 22:02 | |
| *** oanson has quit IRC | 22:03 | |
| *** oanson has joined #openstack-kolla | 22:03 | |
| *** ducttap__ has quit IRC | 22:04 | |
| *** rwallner_ has quit IRC | 22:04 | |
| *** rwallner has joined #openstack-kolla | 22:05 | |
| *** chas has quit IRC | 22:06 | |
| *** ducttape_ has joined #openstack-kolla | 22:06 | |
| jamesPR | cannot ping the external kolla VIP :-@ | 22:08 |
| *** gomarivera has quit IRC | 22:08 | |
| *** erobell_ has quit IRC | 22:09 | |
| *** rwallner has quit IRC | 22:10 | |
| *** ducttape_ has quit IRC | 22:10 | |
| inc0 | jamesPR: being able to ping address help tremendously with accessing it | 22:11 |
| Reepicheep | vhosakot: I would be happy to help where I can.. ops best practices would be a good start | 22:11 |
| vhosakot | Reepicheep: cool, I'll let you know :) | 22:12 |
| Reepicheep | sweet, thanks | 22:12 |
| vhosakot | jamesPR: did you run "kolla-ansible recongiure" with TLS enabled? | 22:12 |
| *** schwicht has quit IRC | 22:13 | |
| jamesPR | vhosakot, yes | 22:15 |
| vhosakot | jamesPR: cool | 22:16 |
| *** ducttape_ has joined #openstack-kolla | 22:16 | |
| vhosakot | jamesPR: can you ping the external kolla VIP wihtout TLS? ;) | 22:17 |
| *** dixiaoli has joined #openstack-kolla | 22:17 | |
| jamesPR | vhosakot, no | 22:17 |
| jamesPR | This site can’t be reached | 22:17 |
| jamesPR | 192.168.25.254 took too long to respond. | 22:17 |
| jamesPR | :S | 22:17 |
| jamesPR | lol | 22:17 |
| *** scheuk has joined #openstack-kolla | 22:17 | |
| jamesPR | I can see the ip assigned to the interface correctly | 22:18 |
| jamesPR | but cannot ping from outside | 22:18 |
| vhosakot | ah i see | 22:19 |
| sdake | vhosakot ucs c220m4 | 22:22 |
| vhosakot | sdake: cool, did you order a new TOR too? | 22:23 |
| sdake | vhosakot no | 22:23 |
| vhosakot | cool cool | 22:23 |
| *** goldyfruit has quit IRC | 22:23 | |
| *** myoung is now known as myoung|bbl | 22:24 | |
| Seyong | goor morning guys :) | 22:30 |
| Seyong | *good | 22:30 |
| *** scheuk has quit IRC | 22:32 | |
| jamesPR | morning! | 22:32 |
| jamesPR | :D | 22:32 |
| *** scheuk has joined #openstack-kolla | 22:33 | |
| jamesPR | here is 6:33PM | 22:33 |
| jamesPR | :D | 22:33 |
| bmaltais | kfox1111: I have been reading on keepalived and I don't think it is the right choice for a VIP for the horizon dashboard | 22:34 |
| bmaltais | kfox1111: The reason being that each instance of horizon can take user request... so it is a fit for loadbalancing | 22:34 |
| Seyong | Here is 7 in the morning ^^ | 22:35 |
| Seyong | 7:35 | 22:35 |
| bmaltais | kfox1111: keepalived is a fit where you only want one of the destination to be active while the other remain passive | 22:35 |
| jamesPR | Seyong, where? lol | 22:35 |
| Seyong | South Korea I am and you? | 22:36 |
| bmaltais | kfox1111: like it is described here: https://github.com/kubernetes/contrib/tree/master/keepalived-vip | 22:36 |
| jamesPR | Seyong, caribbean paradise Puerto Rico :D | 22:37 |
| *** scheuk has quit IRC | 22:37 | |
| Seyong | jamesPR: Good man | 22:38 |
| bmaltais | kfox1111: Darn, the more I read it the more I get confused... keepalived is a complement to haproxy. Not sure I totally get the relationship of both then. | 22:41 |
| *** chas has joined #openstack-kolla | 22:42 | |
| *** egonzalez has quit IRC | 22:44 | |
| *** Manheim has quit IRC | 22:47 | |
| *** manheim_ has quit IRC | 22:47 | |
| *** chas has quit IRC | 22:48 | |
| kfox1111 | bmaltais: keepalived makes an ip float between hosts. | 22:49 |
| kfox1111 | an ip can only ever be on one host at a time. | 22:49 |
| kfox1111 | haproxy is a l7 load balancer. it takes http requests in and balances them across web servers. | 22:50 |
| kfox1111 | k8s has an 'external service' which is a l3 load balancer. | 22:50 |
| kfox1111 | which takes in a tcp stream and load balances them across pods. | 22:50 |
| bmaltais | kfox1111: Right. So keepalived should point at haproxy that would then point to all the horizon dashboard container ip/ports | 22:50 |
| kfox1111 | so, in kolla-kubernetes, the config I recommend is, | 22:50 |
| kfox1111 | keepalived to allow a vip to float. | 22:50 |
| kfox1111 | the vip is backed by an external service allowing that vip to share traffic to all the pods. | 22:51 |
| bmaltais | kfox1111: Running many haproxy all pointing to those horizon dashboard and being fronted by keepalived would make it really ha | 22:51 |
| kfox1111 | haproxy is in each pod doing connection tracking so each service can seamlessly be rolling upgraded. | 22:51 |
| *** srwilkers has quit IRC | 22:51 | |
| *** manheim has joined #openstack-kolla | 22:52 | |
| kfox1111 | this setup is relatively easy to maintain, but bottlenecks at the maximum bandwidth of the node with the vip. | 22:52 |
| kfox1111 | but is ha all the way through. | 22:52 |
| *** srwilkers has joined #openstack-kolla | 22:54 | |
| bmaltais | kfox1111: OK. So https://github.com/kubernetes/contrib/tree/master/keepalived-vip look like a good place to start playing with this | 22:54 |
| kfox1111 | thats cool looking, but probably overkill. | 22:55 |
| kfox1111 | kolla-kubernetes has a keepalived implemented/integrated. | 22:55 |
| bmaltais | kfox1111: Ouch, I am really coming in this cold ;-) I did not realise. I am trying to put the peices of the puzzle as I start to learn about k8t. Maybe I am chewing more than I should at this time | 22:57 |
| bmaltais | kfox1111: I just find it sad that the guide is relying on a physical IP for the VIP instead of a true floating one. The current design prevent scaling to many minions. | 22:58 |
| *** jascott1 has joined #openstack-kolla | 22:58 | |
| *** jamesPR has quit IRC | 22:59 | |
| bmaltais | kfox1111: Sacling would work but losing the node with the IP will kill all access... so it need to be moved to a VIP to scale and be resilient. I tought I could make this change easilly but I am to cold with k8t to really understand all the moving parts | 22:59 |
| kfox1111 | bmaltais: the guide there now was written very recently and was not designed with scaling in mind. it doesn't take too many tweaks I think to get it to scale better/be more ha. just no one has tried it yet. | 23:00 |
| bmaltais | kfox1111: Even the kolla-ansible all-in-one make use of a VIP instead of the interface IP. | 23:00 |
| kfox1111 | yeah. cause it was written over a year ago, and they had time to tweak it to do vips. | 23:00 |
| *** kbyrne has quit IRC | 23:02 | |
| kfox1111 | the aio guide really should have keepalived in it I think. | 23:02 |
| kfox1111 | just no one has had time yet to put the instructions in. | 23:02 |
| bmaltais | kfox1111: I see. I guess I am too exited about this new openstack on kubernetes. I really like what it propose over the old methods. I see what Mirantis is doing but this is going to be a $ solution | 23:02 |
| kfox1111 | yeah. | 23:02 |
| bmaltais | kfox1111: Was looking to build something a little bit similar using pure opensource while learning about k8t in the process | 23:03 |
| kfox1111 | kolla-kubernetes is stil pretty green. the building blocks for a solid cloud are all there now. its mostly adding some polish and writing some exhaustive docs. | 23:03 |
| kfox1111 | yeah. openstack's been a good k8s learning experience. if you can make openstack work on k8s, should be able to make anything work. :) | 23:03 |
| bmaltais | kfox1111: I can certainly help with that and validating process as well. Probably not contribute to the actual config files just yet ;-) | 23:04 |
| bmaltais | kfox1111: I can certainly see kolla-kubernetes being almost there now. Just need some more things and it could almost be used as a prototype in a lab | 23:05 |
| kfox1111 | yeah. | 23:05 |
| bmaltais | kfox1111: This is my goal at this point, to install a lab based openstack service based on kolla-kubernetes and see how it resist to some light usage | 23:05 |
| *** rwallner has joined #openstack-kolla | 23:06 | |
| kfox1111 | I think if the deployment guide included using a vip from the begining, and deployed configmap keepalivd and the keepalived package, it might just work. | 23:06 |
| kfox1111 | thats basically what the gate does, and it works. | 23:06 |
| bmaltais | kfox1111: I think you are right. I already tried adding minions to k8t and services just got deployed to the new nodes based on them being controller, compute or both | 23:06 |
| bmaltais | So the missing piece would be a VIP for the api/dashboard side | 23:07 |
| kfox1111 | yeah. k8s makes that part really easy. :) | 23:07 |
| kfox1111 | yeah. | 23:07 |
| *** gomarivera has joined #openstack-kolla | 23:08 | |
| bmaltais | kfox1111: I will keep an eye to see how I can contribute. I could start by documenting how to scale the AIO to multinode... knowing the VIP will fail if the master node dies but it should be a beginning. | 23:09 |
| *** vhosakot has quit IRC | 23:09 | |
| * inc0 ragecoding | 23:09 | |
| inc0 | https://github.com/inc0/kolla_dockerhub_pusher | 23:09 |
| inc0 | so I just created tool to pull kolla registry from tarballs.o.o and run it locally | 23:09 |
| inc0 | now you can have full set of kolla images in less than 5min...depends on internet | 23:10 |
| kfox1111 | bmaltais: that would be great. :) | 23:10 |
| *** rwallner has quit IRC | 23:10 | |
| inc0 | aio to multinode might cause some interesting mariadb issues | 23:11 |
| *** kbyrne has joined #openstack-kolla | 23:11 | |
| kfox1111 | inc0: depends on if ceph backed or not. | 23:11 |
| kfox1111 | also, depends on what multinode means. | 23:11 |
| inc0 | hehe I guess | 23:11 |
| kfox1111 | one controller + multiple computes should "just work (TM)" | 23:11 |
| kfox1111 | multiple controllers, may need some more work. | 23:12 |
| *** jascott1 has quit IRC | 23:12 | |
| inc0 | Sir, I refuse to call single controller a multinode deployment. That will not stand sir! | 23:12 |
| *** denaitre has quit IRC | 23:12 | |
| *** jascott1 has joined #openstack-kolla | 23:13 | |
| kfox1111 | I won't tell you how many production multinode configs Ihave in that arangement. and I'm sure I'm not the only one. ;) | 23:13 |
| inc0 | please don't tell me | 23:13 |
| inc0 | are they running on devstack | 23:13 |
| inc0 | ? | 23:13 |
| inc0 | just to make things a little more spicy | 23:13 |
| *** denaitre has joined #openstack-kolla | 23:14 | |
| portdirect | inc0: I'm kinda with you on that - sorry kfox1111 :P | 23:14 |
| inc0 | lol | 23:14 |
| portdirect | I think its perfectly fine (if a little exciting) to have a single controller, multiple compute node deployment | 23:15 |
| portdirect | but to call it multitude is a bit of a misnomer | 23:15 |
| inc0 | hey, let's run it on k8s, then everything will be fine forever right? | 23:15 |
| kfox1111 | portdirect: I didn't say it was smart. just said it is very common. ;) | 23:15 |
| portdirect | lol - thats what my old deployment was untill i went HYPERCONVERGED(tm) | 23:16 |
| kfox1111 | heh. yeah, we had a hyperconverged cloud too... | 23:16 |
| inc0 | haha so not just control but also compute and stor on same node? | 23:16 |
| kfox1111 | that one did not end well. ;) | 23:16 |
| portdirect | it never does... | 23:16 |
| *** jascott1 has quit IRC | 23:17 | |
| inc0 | hyperconverge my cloud | 23:17 |
| kfox1111 | hyperconverge works well on a cloud of one node. ;) | 23:17 |
| bmaltais | inc0: Mariadb issue... oh oh... so it does net sync at the back between nodes then... | 23:17 |
| inc0 | cloud of one node | 23:17 |
| inc0 | I'll just let it sink in for a second | 23:17 |
| kfox1111 | bmaltais: not yet. we don't have galera support. | 23:18 |
| inc0 | speaking of which, portdirect do we want to kick the bikeshed? | 23:18 |
| inc0 | off | 23:18 |
| kfox1111 | bmaltais: but if backed by a pv that can float between nodes like ceph rbd's, then the db can failover to a different node. | 23:18 |
| inc0 | btw project can be called shed ... as it holds tools right? | 23:18 |
| portdirect | lol - my favorite new overused pharased is the infrastructure sandwich. 99% of the time the fully converged stuff is that sandwich so full of fillings theres no way to eat it without half of it ending up on the floor :) | 23:18 |
| *** dixiaoli has quit IRC | 23:22 | |
| *** hongbin has quit IRC | 23:23 | |
| kfox1111 | hehe. lets call it the "conspiracy" project. ;) | 23:24 |
| inc0 | ahh so you kfox1111 are already included in our very secretive plan that I just called out on public channel | 23:25 |
| kfox1111 | nope. just figure people will bike shed or read into it what they want. | 23:26 |
| kfox1111 | it just will cause those folks to spin their wheels even harder. :) | 23:26 |
| kfox1111 | which would be funny. :) | 23:26 |
| inc0 | don't worry, all shall be revealed soon;) | 23:27 |
| *** schwicht has joined #openstack-kolla | 23:28 | |
| kfox1111 | oh... I got it. | 23:29 |
| kfox1111 | other then breaking my rule of hating to pick words that I have a hard time spellilng.... | 23:29 |
| kfox1111 | 'symbiosis' :) | 23:29 |
| inc0 | dicker ? | 23:30 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla stable/ocata: Updated from global requirements https://review.openstack.org/464807 | 23:31 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible stable/ocata: Updated from global requirements https://review.openstack.org/452870 | 23:31 |
| kfox1111 | heh. I think that may have connotations to those not fully understanding english. | 23:32 |
| kfox1111 | I'm partial to the symbiosis project. | 23:32 |
| inc0 | well, it also means that o and i are near each other on keyboard | 23:32 |
| inc0 | symbiosis is good name | 23:33 |
| *** rwellum has quit IRC | 23:33 | |
| bmaltais | kfox1111: Depending on the objective of the project... Commensalism: One organism benefits and the other is neither harmed nor helped. | 23:34 |
| inc0 | I know... let's call it communism | 23:34 |
| *** ducttape_ has quit IRC | 23:34 | |
| inc0 | everything is shared | 23:34 |
| portdirect | oh gawd | 23:35 |
| inc0 | whether you like it or not | 23:35 |
| srwilkers | oh boy | 23:35 |
| kfox1111 | bmaltais: the idea is folks working together rather then apart when there are overlapping things we can share. | 23:35 |
| inc0 | let's paint the bikeshed red | 23:35 |
| kfox1111 | hehe | 23:35 |
| kfox1111 | inc0: wow.. just wow... ;) | 23:35 |
| inc0 | :D | 23:35 |
| srwilkers | im partial to blue | 23:35 |
| * portdirect is taking the chisels to his hard drives | 23:36 | |
| inc0 | how about conflux? | 23:36 |
| bmaltais | kfox1111: OK... so a form of Mutualism Symbiosis where Both organisms benefit. | 23:36 |
| kfox1111 | bmaltais: exactly. | 23:36 |
| srwilkers | bikeshedding over a name for bikeshed | 23:37 |
| srwilkers | fantastic | 23:37 |
| kfox1111 | srwilkers: hehe. exactly. :) | 23:37 |
| inc0 | wouldn't have it any other way | 23:37 |
| kfox1111 | there are only 2 hard problems in computer science. | 23:37 |
| inc0 | cache and naming things | 23:38 |
| kfox1111 | naming things, cache invalidation, and off by one errors. ;) | 23:38 |
| srwilkers | ;) | 23:38 |
| inc0 | haha | 23:38 |
| *** srwilkers has quit IRC | 23:38 | |
| *** ducttape_ has joined #openstack-kolla | 23:38 | |
| *** schwicht has quit IRC | 23:40 | |
| bmaltais | kfox1111: Osmosis - process by which molecules of a solvent tend to pass through a semipermeable membrane from a less concentrated solution into a more concentrated one, thus equalizing the concentrations on each side of the membrane. | 23:41 |
| kfox1111 | thats a good candidate too I think. | 23:42 |
| kfox1111 | inc0: so.... | 23:43 |
| *** ducttape_ has quit IRC | 23:43 | |
| kfox1111 | the pusher you wrote seems to assume a lot about how a docker repo is formatted. | 23:43 |
| kfox1111 | how fragile do you think that will be? it makes me kind of nervious. | 23:43 |
| inc0 | kfox1111: well, it's targetted for our tarballs today | 23:43 |
| kfox1111 | inc0: we have 2 types today. | 23:44 |
| kfox1111 | the ones that are registry backed, | 23:44 |
| kfox1111 | and those that are docker saved. | 23:44 |
| inc0 | yeah, this is registry backed one | 23:44 |
| inc0 | also, still inprog | 23:44 |
| inc0 | I barely started it this morning | 23:44 |
| kfox1111 | I'm kind of partial to the docker saved ones as they are much simpler / don't assume anything about the formatting. | 23:44 |
| kfox1111 | yeah. your code looks good. I'm just wondering about the base assumptions. | 23:44 |
| inc0 | but we have 200+ images | 23:45 |
| inc0 | and if you do docker save, it won't share stuff | 23:45 |
| *** jascott1 has joined #openstack-kolla | 23:45 | |
| kfox1111 | it does share stuff. | 23:45 |
| kfox1111 | you just make sure they are in the same tarball. | 23:45 |
| inc0 | if you do docker save? | 23:45 |
| kfox1111 | it dedups. | 23:45 |
| jascott1 | wow getting your pupils dilated is not for the faint of heart | 23:45 |
| kfox1111 | yeah. | 23:45 |
| kfox1111 | jascott1: hehe. yeah. | 23:45 |
| jascott1 | Me: "I have to drive home" Dr: "Oh you will be fine" | 23:45 |
| *** harlowja has quit IRC | 23:46 | |
| kfox1111 | inc0: I had the same question about them. someone (portdirect?) I think validiated that it collapses them. | 23:46 |
| inc0 | I thought docker save just tars dir of particular image | 23:46 |
| * portdirect cant rembemer - though knows it did *things* | 23:46 | |
| kfox1111 | inc0: see: http://tarballs.openstack.org/kolla-kubernetes/gate/containers/ | 23:47 |
| kfox1111 | the tarballs aren't huge. | 23:47 |
| kfox1111 | inc0: it does, but all deps too. | 23:47 |
| kfox1111 | so if they share deps, it only has the one copy. | 23:47 |
| kfox1111 | works the same way as the registry dump. | 23:47 |
| inc0 | I just ran docker save <<one of images>> | 23:48 |
| inc0 | and it's 800+ megs | 23:48 |
| kfox1111 | inc0: yeah. but do docker save image1 image2 | 23:48 |
| inc0 | ahh | 23:48 |
| kfox1111 | it won't be sizeof(image1)+sizeof(image2) | 23:48 |
| *** manheim has quit IRC | 23:49 | |
| inc0 | trying | 23:49 |
| kfox1111 | so if we use docker save, we can do the same as the registry size wise, but without needing to ever know the interneals/spawn a repo server to extract stuff. | 23:49 |
| inc0 | yeah that would be much better | 23:49 |
| kfox1111 | the cron job then is. "docker load; for each image in docker images; docker push image; done; delete images. | 23:50 |
| inc0 | trying stack of kolla images | 23:50 |
| kfox1111 | inc0: should look a lot like: http://tarballs.openstack.org/kolla-kubernetes/gate/containers/centos-binary-2-ceph.tar.bz2 :) | 23:51 |
| kfox1111 | containing something like: http://tarballs.openstack.org/kolla-kubernetes/gate/containers/centos-binary-2-ceph-containers.txt :) | 23:51 |
| inc0 | well tared registry of all the images is 2.4gig | 23:51 |
| inc0 | for ubuntu source | 23:51 |
| inc0 | let see if docker will beat that | 23:51 |
| kfox1111 | yeah. that has more containers then what I've been building. | 23:51 |
| kfox1111 | I've only been doing the ones kolla-kubernetes tests in the gate. | 23:52 |
| inc0 | well having them tared would indeed be much less complex to manage | 23:53 |
| inc0 | but size grows significantly | 23:54 |
| kfox1111 | you sure? | 23:54 |
| inc0 | just tried, subset of images (profile default) is almost 4 gigs | 23:54 |
| kfox1111 | I'm not seeing that in the kolla-kubernetes ones. | 23:54 |
| inc0 | as opposed to full set being less than 3 | 23:54 |
| kfox1111 | were they all built at exactly the same time? | 23:54 |
| kfox1111 | (different parents?) | 23:54 |
| *** rhallisey has quit IRC | 23:55 | |
| inc0 | well, no but that was my own build locally | 23:55 |
| inc0 | vs what I downloaded from tarballs.o.o | 23:55 |
| kfox1111 | you might clear your imags out then build fresh and then try the tar again. | 23:55 |
| kfox1111 | it may be you have some uncommon parents making the tar bigger. | 23:55 |
| inc0 | well I only tared images from grep | 23:56 |
| inc0 | ahh | 23:56 |
| *** goldyfruit has joined #openstack-kolla | 23:56 | |
| inc0 | I see whay you're saying | 23:56 |
| inc0 | yes, all of images tared were build in same run | 23:56 |
| kfox1111 | hmm.... I wonder whats the difference then. | 23:58 |
| inc0 | I'll try again later | 23:58 |
| inc0 | clear everything and do proper test | 23:58 |
| kfox1111 | k. | 23:58 |
| kfox1111 | would be curious the results. | 23:59 |
| kfox1111 | if it is equal, or close, the docker save would be sooo much easier to deal with. | 23:59 |
| inc0 | I agree | 23:59 |
| kfox1111 | but if its significantly bigger, then the registry stuff would probably be worth it. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!