| *** zhurong has quit IRC | 00:07 | |
| *** fguillot_ has joined #openstack-kolla | 00:12 | |
| *** sdake has joined #openstack-kolla | 00:14 | |
| sdake | yo | 00:15 |
|---|---|---|
| *** gfhellma has quit IRC | 00:18 | |
| sbezverk_ | sdake: need help from vxlan expert, do you know any? | 00:20 |
| sdake | god i wish | 00:20 |
| sdake | daneyon is your best shot there | 00:20 |
| sdake | but i you dont know it, then i ont think daneyon would either | 00:20 |
| sbezverk_ | :-( | 00:21 |
| sdake | he is a ccie and pretty strong network guy - worth asking | 00:21 |
| sbezverk_ | sdake: this one of those x-files type of issue | 00:21 |
| sbezverk_ | when all config looks ok, no any errors in the log but it does not work.. | 00:21 |
| sdake | where is the vikram | 00:22 |
| sbezverk_ | no idea, tried to ping him earlier | 00:22 |
| sdake | he sent me mail 30 minutes ago | 00:23 |
| *** frzn_ has quit IRC | 00:24 | |
| aNupoisc | Hi Folks, When i am trying to deploy ceph with kolla it is forever staying in "Looking up disks to bootstrap for Ceph OSDs" task. I have one partition named as "KOLLA_CEPH_OSD_BOOTSTRAP". | 00:26 |
| aNupoisc | Any idea whether it is looking for something | 00:26 |
| sdake | it should be fast at that step | 00:26 |
| sdake | are you using mitaka branch? | 00:26 |
| aNupoisc | sdake: yes | 00:27 |
| sdake | how many nodes? | 00:27 |
| aNupoisc | this is the paste where it is sitting forever http://pastebin.com/rYqvXjgA | 00:27 |
| aNupoisc | single node | 00:27 |
| *** fragatin_ has quit IRC | 00:27 | |
| sdake | single node requires special ceph config | 00:27 |
| aNupoisc | i am using all-in-one inventory file sdake | 00:27 |
| aNupoisc | sdake: I am following https://github.com/openstack/kolla/blob/master/doc/ceph-guide.rst#managing-ceph | 00:27 |
| *** fragatina has joined #openstack-kolla | 00:28 | |
| aNupoisc | sdake: what is that can you tell? I have changed globals.yml and ceph.conf to tell one pool | 00:28 |
| aNupoisc | for osd | 00:28 |
| sdake | [global] | 00:28 |
| sdake | osd pool default size = 1 | 00:28 |
| sdake | osd pool default min size = 1 | 00:28 |
| aNupoisc | sdake: yes doing that | 00:29 |
| sdake | ok | 00:29 |
| aNupoisc | also running this "docker exec ceph_mon ceph osd pool set rbd size 1" | 00:29 |
| aNupoisc | to tell one copy | 00:29 |
| sdake | you should not need to do that if you set the globals.yml | 00:29 |
| aNupoisc | okay let me paste my globals.yml | 00:30 |
| *** duonghq has joined #openstack-kolla | 00:30 | |
| aNupoisc | sdake: http://pastebin.com/gbPE33hC | 00:31 |
| aNupoisc | that is my globals | 00:31 |
| aNupoisc | sdake: i ran docker exec ceph_mon ceph osd pool set rbd size 1 too. | 00:32 |
| aNupoisc | sdake: but that won't affect right as it will read globals.yml | 00:32 |
| sdake | what is that spposed to do | 00:32 |
| duonghq | sdake: centos is ok? | 00:32 |
| sdake | duonghq no iea just woke up | 00:33 |
| duonghq | oops | 00:33 |
| sdake | duonghq i was hoping Daviey or vhosakot would e up still | 00:33 |
| sdake | i got a email indicating the nodes are deplouyed | 00:33 |
| aNupoisc | sdake: it is just tells the number of copies should be one inside pool | 00:33 |
| duonghq | sdake: you just took a quick nap or full sleep? | 00:33 |
| sdake | 2-3 hrs | 00:33 |
| sdake | 2 hrs sleep 12 hrs work in cycle since wednesday | 00:34 |
| aNupoisc | sdake: you are what timezone? | 00:34 |
| sdake | arizona gmt-7 pst | 00:34 |
| duonghq | hope that next time I can join OSIC driver team | 00:34 |
| aNupoisc | sdake: ahh | 00:34 |
| sdake | aNupoisc i need to get back to you - today is our lat day on the osic gear | 00:35 |
| sdake | aNupoisc and i guess it has beeen deplooyed on centos | 00:35 |
| sdake | so I need to measure some baselines | 00:35 |
| duonghq | sdake: how many hours do we left? | 00:35 |
| sdake | about 6-12 | 00:35 |
| sdake | enough to run 1 baseline | 00:35 |
| duonghq | tight schedule | 00:35 |
| *** frzn_ has joined #openstack-kolla | 00:36 | |
| aNupoisc | sdake: i am running on ubuntu | 00:36 |
| sdake | aNupoisc its not that i dont want to help yu - osic gear disappaers tomorrow | 00:36 |
| duonghq | but, what wrong with you aNupoisc? | 00:36 |
| sdake | and i need to make use of it while we have it | 00:36 |
| duonghq | see if I can help | 00:36 |
| aNupoisc | sdake: Oh okay i don't have an idea what is osic gear? | 00:37 |
| aNupoisc | sdake: okay np | 00:37 |
| aNupoisc | duonghq: Hi yeah so | 00:37 |
| aNupoisc | duonghq: I am running kolla with ceph and my deploy is sitting at " Looking up disks to bootstrap for Ceph OSDs" forever | 00:38 |
| aNupoisc | duonghq: I am running all-in-one node setup | 00:38 |
| duonghq | which ceph backend do you use? | 00:38 |
| aNupoisc | radosgw | 00:38 |
| sdake | aNupoisc osic gear is 131 node cluster we have to scale tet kolla | 00:39 |
| aNupoisc | sdake: Ahh awesome good luck :) | 00:39 |
| sdake | we have done alot of measurement already | 00:39 |
| sdake | 20 minutes to deploy 121 nodes | 00:39 |
| sdake | 123 nodes that is | 00:39 |
| *** jtriley has joined #openstack-kolla | 00:39 | |
| aNupoisc | duonghq: here is my globals.yml http://pastebin.com/gbPE33hC | 00:39 |
| aNupoisc | sdake: I am trying to bring up ceph for 2 days and some thing is failing. We can talk on it tomorrow if not resolved by today | 00:40 |
| aNupoisc | sdake: good luck | 00:40 |
| duonghq | I do not ceph yet, sorry | 00:41 |
| duonghq | but did you check the container log? | 00:41 |
| aNupoisc | duonghq: not sure where they go | 00:41 |
| aNupoisc | let me do that | 00:41 |
| duonghq | docker exec -it heka /bin/bash | 00:42 |
| duonghq | and look in /var/log/kolla/ceph | 00:42 |
| aNupoisc | duonghq: not returning anything | 00:44 |
| aNupoisc | i have two files ceph-client.admin.log and ceph-mon.log | 00:45 |
| aNupoisc | both are empty | 00:45 |
| duonghq | hmm, did you try: docker logs ceph-osd? | 00:45 |
| duonghq | or bootstrap ceph | 00:45 |
| duonghq | do not remember exact name | 00:45 |
| aNupoisc | should those be inside docker | 00:46 |
| aNupoisc | ? | 00:46 |
| duonghq | no, the outer shell | 00:46 |
| aNupoisc | saying no such container | 00:46 |
| duonghq | docker ps -a | grep cehph | 00:47 |
| duonghq | *ceph | 00:47 |
| duonghq | what does it say? | 00:47 |
| *** kbaegis has joined #openstack-kolla | 00:50 | |
| kbaegis | Hey guys. From current master, horizon | 00:50 |
| duonghq | kbaegis? what's wrong? | 00:52 |
| *** zhubingbing has joined #openstack-kolla | 00:53 | |
| kbaegis | 500's. Traceback: http://hastebin.com/dafahevori.rb | 00:53 |
| aNupoisc | duonghq: It says ceph is running 8d4f59263708 kolla/ubuntu-binary-ceph-mon:2.0.2 "kolla_start" | 00:55 |
| aNupoisc | from 3 days | 00:55 |
| aNupoisc | duonghq: it is somehow unable to find the partition | 00:56 |
| aNupoisc | for osd | 00:56 |
| aNupoisc | thats the problem even though i have the partition named as it is looking | 00:56 |
| duonghq | kbaegis?latest master? did you try clean and rebuild image | 00:57 |
| duonghq | aNupoisc: can you try docker logs <container-name> | 00:58 |
| duonghq | ceph-mon, I guess | 00:58 |
| *** v1k0d3n has quit IRC | 00:58 | |
| kbaegis | duonghq: Did rebuild, didn't clean | 00:59 |
| aNupoisc | it is unable to find ceph-mon container | 00:59 |
| aNupoisc | duonghq | 00:59 |
| aNupoisc | duonghq: i forget to mention the name at start. lol | 01:00 |
| *** vhosakot has joined #openstack-kolla | 01:00 | |
| aNupoisc | duonghq: do you know what extra things are required when doing ceph on single node | 01:00 |
| duonghq | kbaegis: sometime docker image and ansible need to in sync :) | 01:01 |
| duonghq | aNupoisc: I don't know, sorry | 01:02 |
| duonghq | but ceph-mon is running but cannot docker logs is wired | 01:02 |
| aNupoisc | duonghq: I am not sure. Do we need to mention seperately in any conf file or in globals for that? | 01:03 |
| aNupoisc | duonghq: yeah seems ceph-mon is started but logs are not available | 01:03 |
| duonghq | afaik it's quite unussual | 01:04 |
| duonghq | not sure, sorry | 01:04 |
| aNupoisc | duonghq: hmm. | 01:04 |
| aNupoisc | no problem | 01:04 |
| aNupoisc | thanks | 01:04 |
| vhosakot | sdake: ping.. should we run scenario # 2 on centos ? | 01:05 |
| sdake | vhosakot need to stup network first | 01:05 |
| *** jtriley has quit IRC | 01:06 | |
| vhosakot | sdake: yep, I'm searching that playbook.. do you know its name? | 01:06 |
| sdake | i dont know i there is one | 01:06 |
| vhosakot | sdake: I'll check.. | 01:06 |
| *** jtriley has joined #openstack-kolla | 01:06 | |
| sdake | the interfaces are named differenelty as well | 01:06 |
| vhosakot | 1 min... switching VPNs | 01:06 |
| *** xwang has quit IRC | 01:07 | |
| *** Pavo has quit IRC | 01:09 | |
| *** zhurong has joined #openstack-kolla | 01:13 | |
| *** Pavo has joined #openstack-kolla | 01:13 | |
| *** sdake has quit IRC | 01:14 | |
| *** frzn_ has left #openstack-kolla | 01:14 | |
| rkrum | Has anyone seen neutron agent errors like RTNETLINK answers: Invalid argument with Kolla and Centos 7? | 01:16 |
| rkrum | It seems specific to running neutron in docker atm. | 01:16 |
| *** mtaylor22 has joined #openstack-kolla | 01:18 | |
| *** mtaylor22 has quit IRC | 01:18 | |
| *** mtaylor22 has joined #openstack-kolla | 01:19 | |
| *** sdake has joined #openstack-kolla | 01:21 | |
| sdake | vhosakot - ok was delaing with yard dude | 01:22 |
| sdake | i am back | 01:22 |
| vhosakot | sdake: cool, I'm still searching the playbook... | 01:22 |
| vhosakot | do we have the globals.yml ready for scenario 2 ? | 01:22 |
| sdake | not really ready | 01:22 |
| sdake | but its prtty rclose | 01:23 |
| vhosakot | sdake: ok... | 01:23 |
| sdake | the interface names are diferent on that machine | 01:23 |
| sdake | on centos the interface names ae different | 01:23 |
| vhosakot | that machine = ? deploy node? | 01:23 |
| vhosakot | 253? | 01:23 |
| sdake | all machines deployed with centos have a differnet interface name | 01:24 |
| sdake | enp1s0 | 01:24 |
| sdake | vs eth0 | 01:24 |
| sdake | etc | 01:24 |
| vhosakot | ok.. | 01:24 |
| vhosakot | sdake: 1 sec... i'll be back... | 01:24 |
| *** vhosakot has quit IRC | 01:25 | |
| *** vhosakot_ has joined #openstack-kolla | 01:25 | |
| *** zhiwei has joined #openstack-kolla | 01:27 | |
| vhosakot_ | sdake: back | 01:27 |
| sdake | vhosakot_ did you capture ipconfig on one of the nodes before resetting them? | 01:27 |
| sdake | ifconfig thatis | 01:27 |
| vhosakot_ | sdake: no | 01:27 |
| sdake | britthouser would be helpful here | 01:28 |
| sdake | ca nyou call him | 01:28 |
| sdake | he knows how the networks are setup i htink | 01:28 |
| vhosakot_ | yes, britthouser said he can help after 9 pm EST today and it is 9:28 pm EST here now | 01:28 |
| sdake | p4p1.702 Link encap:Ethernet HWaddr 68:05:ca:38:65:64 | 01:29 |
| sdake | inet addr:172.21.8.253 Bcast:172.21.11.255 Mask:255.255.252.0 | 01:29 |
| sdake | p4p1.733 Link encap:Ethernet HWaddr 68:05:ca:38:65:64 | 01:29 |
| sdake | inet addr:172.21.112.250 Bcast:0.0.0.0 Mask:255.255.252.0 | 01:29 |
| sdake | no idea what network topology of this gear i | 01:30 |
| vhosakot_ | wow, that is the sub-interface for VLAN 702 I think... I doubt the cobbler ks has VLAN info... will the playbook set this up ? | 01:30 |
| sdake | is | 01:30 |
| *** fragatina has quit IRC | 01:30 | |
| sdake | there is no playbook that i can see | 01:30 |
| sdake | it was deployed by ansible i'm certain of it | 01:30 |
| sdake | becuse the interface file aid at the top "Deployed by ansible" | 01:30 |
| sdake | there are other vlans we have as ewll | 01:31 |
| *** jtriley has quit IRC | 01:31 | |
| sdake | setup-insecure-inv.yml | 01:32 |
| sdake | that is one file | 01:32 |
| sdake | I think it needs modification | 01:32 |
| *** salv-orlando has joined #openstack-kolla | 01:32 | |
| vhosakot_ | ok | 01:32 |
| vhosakot_ | let me check | 01:32 |
| *** huikang has joined #openstack-kolla | 01:33 | |
| *** frzn_ has joined #openstack-kolla | 01:33 | |
| sdake | how aobut we run the host setup playbook | 01:33 |
| sdake | and see what happens | 01:33 |
| sdake | moment | 01:33 |
| sdake | 2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 | 01:34 |
| sdake | link/ether 3c:fd:fe:9c:6f:1c brd ff:ff:ff:ff:ff:ff | 01:34 |
| sdake | inet 172.21.0.24/22 brd 172.21.3.255 scope global dynamic ens1f0 | 01:34 |
| sdake | valid_lft 18192sec preferred_lft 18192sec | 01:34 |
| sdake | inet6 fe80::3efd:feff:fe9c:6f1c/64 scope link | 01:34 |
| sdake | valid_lft forever preferred_lft forever | 01:34 |
| sdake | 3: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 | 01:34 |
| sdake | link/ether 3c:fd:fe:9c:6f:1d brd ff:ff:ff:ff:ff:ff | 01:34 |
| sdake | 4: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 | 01:34 |
| sdake | link/ether 3c:fd:fe:9c:73:14 brd ff:ff:ff:ff:ff:ff | 01:34 |
| sdake | 5: ens4f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 | 01:34 |
| sdake | link/ether 3c:fd:fe:9c:73:15 brd ff:ff:ff:ff:ff:ff | 01:34 |
| sdake | those are all the interfaces on the machine | 01:34 |
| vhosakot_ | sdake: setup-insecure-inv.yml starts the docker insecure registry | 01:34 |
| sdake | i know - i looked at it | 01:34 |
| *** banix has joined #openstack-kolla | 01:34 | |
| vhosakot_ | sdake: who setup the netwporking in ubuntu | 01:34 |
| sdake | it doesn't start the registry | 01:35 |
| sdake | it sets up insecure registry on all the nodes | 01:35 |
| sdake | no idea | 01:35 |
| sdake | i think inc0 or pburke | 01:35 |
| sdake | pbourke | 01:35 |
| sdake | but britthouser was involved | 01:35 |
| vhosakot_ | right, setup-insecure-inv.yml updates DOCKEROPTS /etc/default/docker... | 01:35 |
| vhosakot_ | sdake: in centos, the config file is not /etc/default/docker... I think it is different.. let me check | 01:36 |
| sdake | would have been handy to keep an interface file on the ubuntu machines | 01:36 |
| *** mtaylor22 has quit IRC | 01:36 | |
| sdake | are there any machines you didn't reload? | 01:36 |
| *** caowei has joined #openstack-kolla | 01:37 | |
| *** dwalsh has joined #openstack-kolla | 01:37 | |
| vhosakot_ | sdake: only deploy node (253) was not touched | 01:37 |
| sdake | i dunno - dead in the water | 01:38 |
| sdake | need britthouser | 01:38 |
| vhosakot_ | shall I setup a webex with you and britthouser ? | 01:39 |
| sdake | i think irc should be good enough | 01:39 |
| vhosakot_ | cool | 01:39 |
| *** signed8b_ has quit IRC | 01:39 | |
| sdake | is britthouser here? | 01:39 |
| sdake | my cll is out of juice - so i can't call him | 01:39 |
| sdake | cell | 01:40 |
| vhosakot_ | 1 sec... need to switch VPN again to find Britt;s cell number | 01:41 |
| *** vhosakot_ has quit IRC | 01:43 | |
| *** salv-orlando has quit IRC | 01:43 | |
| *** vhosakot has joined #openstack-kolla | 01:43 | |
| vhosakot | sdake: ok back | 01:44 |
| vhosakot | sdake: wow, britthouser does not have his cell number in the compant dir | 01:45 |
| vhosakot | company* | 01:45 |
| vhosakot | sdake: I emailed britthouser | 01:47 |
| britthouser | I am here now sdake and vhosakot | 01:47 |
| vhosakot | :) | 01:48 |
| sdake | cool - wife just brought home dinner | 01:48 |
| sdake | britthouser do you recall how the networks were configured for kolla | 01:48 |
| vhosakot | britthouser: so, all the 130 nodes have CentOS now and need networking/VLAN settings | 01:48 |
| sdake | there were some vlans and stuff | 01:48 |
| britthouser | I took some notes, lemme look | 01:49 |
| vhosakot | 1 sec.. switching VPNs... I'll be back | 01:49 |
| vhosakot | Cisco<-->F5 VPN switching is a game! | 01:49 |
| britthouser | So have a look at the etherpoad | 01:49 |
| britthouser | there is a section called "Network Layout" | 01:49 |
| britthouser | p4p2 and p1p2 should be bonded into bond0 | 01:50 |
| britthouser | p1p1 is our external | 01:50 |
| britthouser | p4p1 is the ceph cluster (on vlan 733) | 01:50 |
| sdake | line 128 | 01:52 |
| *** vhosakot_ has joined #openstack-kolla | 01:52 | |
| *** vhosakot has quit IRC | 01:52 | |
| sdake | britthouser do you know how to do that on centos on 123 nodes via ansible | 01:52 |
| sdake | note centos has different interface names | 01:53 |
| sdake | britthouser so weprobaby need a mapping of some type | 01:53 |
| vhosakot_ | back.. | 01:53 |
| vhosakot_ | britthouser: which tmux session are you in? 0 or 9? | 01:53 |
| *** unicell has quit IRC | 01:55 | |
| vhosakot_ | the centos source images built can be seen in tmux session 0 window 3 | 01:55 |
| britthouser | I'll login to zero | 01:55 |
| vhosakot_ | cool | 01:56 |
| sdake | cn you cats make yur screens bigger | 01:58 |
| *** aNupoisc has quit IRC | 02:01 | |
| britthouser | Agreed sdake and vhosakot - lets stick with the interface names we have | 02:02 |
| sdake | too complicted to rename | 02:02 |
| sdake | and doens't work particualrly well | 02:02 |
| vhosakot_ | cool... | 02:04 |
| vhosakot_ | britthouser: so, what "extra" networking is needed than how it is now ? | 02:04 |
| britthouser | sdake - my proposal is that we work out the bondign config on one host, and then write the ansible todo the rest | 02:04 |
| britthouser | lets login to one of the nodes and see | 02:04 |
| vhosakot_ | britthouser: should we bond and sub-inerface for vlan ? | 02:04 |
| sdake | wfm | 02:04 |
| *** daneyon has joined #openstack-kolla | 02:08 | |
| vhosakot_ | britthouser: window 3 was deploy node.. now, it is not.. what happened ? | 02:08 |
| * britthouser shrug | 02:09 | |
| duonghq | anybody know where the baremetal roles is for ubuntu 14.04 or 16.04 one? | 02:09 |
| vhosakot_ | britthouser: :) cool... | 02:09 |
| duonghq | *whether | 02:09 |
| sdake | duonghq its for 14.04 | 02:09 |
| duonghq | thank sdake | 02:10 |
| sdake | which interfaces aer e bonding | 02:11 |
| britthouser | ens1f1 and ens4f1 | 02:11 |
| britthouser | old p1p2 and p4p2 | 02:11 |
| vhosakot_ | britthouser: ah, did you SSH into one of the target nodes (.45) in window 3 ? | 02:11 |
| britthouser | someone must have vhosakot_ | 02:12 |
| vhosakot_ | cool.. | 02:12 |
| sdake | may have been me looking for interface files for ubuntu | 02:12 |
| *** duonghq has left #openstack-kolla | 02:12 | |
| vhosakot_ | cool.. | 02:13 |
| *** daneyon has quit IRC | 02:13 | |
| openstackgerrit | Lu lei proposed openstack/kolla: Fix oslo.i18n in kolla project https://review.openstack.org/355342 | 02:13 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 | 02:15 |
| vhosakot_ | britthouser: of the 130 nodes, on which nodes are the bonds configured? | 02:16 |
| sdake | britthouser pretty sur eyu can get rid of uuid as well as long as hou ahve the name | 02:16 |
| britthouser | all | 02:16 |
| vhosakot_ | wow, all 130 needs bond.. ok | 02:16 |
| britthouser | I think you need either HWADDRES or UUID | 02:16 |
| openstackgerrit | dupengfei proposed openstack/kolla: * Fix lost parameter --remote in generated start_ovsdb_server.sh https://review.openstack.org/362027 | 02:17 |
| * britthouser crosses fingers | 02:17 | |
| *** zhubingbing has quit IRC | 02:17 | |
| vhosakot_ | SSH did not drop. cool! | 02:17 |
| *** duonghq has joined #openstack-kolla | 02:18 | |
| vhosakot_ | nice! | 02:18 |
| britthouser | oh duh | 02:18 |
| vhosakot_ | britthouser: we can refer VIM's bond0 as sample? | 02:19 |
| vhosakot_ | cool | 02:19 |
| britthouser | that's where I'm cutting pasteing | 02:19 |
| vhosakot_ | :) | 02:19 |
| sdake | nice mii statu ftw | 02:20 |
| britthouser | Partner Mac Address: 00:23:04:ee:be:b9 | 02:20 |
| britthouser | we are bnoded with teh switch | 02:20 |
| vhosakot_ | britthouser: is this mac the mac of the remote end ? | 02:20 |
| britthouser | right...that is the switch_id sent by lacp process on the ToR | 02:21 |
| vhosakot_ | cool | 02:21 |
| britthouser | if its all 00:00:00:00 | 02:21 |
| vhosakot_ | shouldn;t bond0 have an IP ? | 02:21 |
| britthouser | no | 02:21 |
| britthouser | we now need to create bond0.702 | 02:21 |
| sdake | the power of 100 lemons!! | 02:21 |
| britthouser | bond0.730 | 02:21 |
| britthouser | bond0.731 | 02:21 |
| britthouser | ROFL | 02:21 |
| britthouser | bond0.732 | 02:21 |
| vhosakot_ | yes, I saw that issue the other day.. partner MAC was 00:00:00:00 and port-channel on the other end was down due to bad networking | 02:21 |
| britthouser | and ens4f1.733 | 02:22 |
| *** zhubingbing has joined #openstack-kolla | 02:22 | |
| vhosakot_ | britthouser: how do we redo these steps on the other 129 nodes... should we write a bash script and use sed to replace lines in files ? | 02:22 |
| *** kbyrne has quit IRC | 02:22 | |
| britthouser | probably ansible lineinfile? | 02:22 |
| vhosakot_ | cool | 02:22 |
| britthouser | we should look back how it was done in ubuntu | 02:23 |
| sdake | lineinfile | 02:23 |
| britthouser | it will be similar, just different files instead of all in one file | 02:23 |
| sdake | britthouser i don't see any networ playbooks on 253 | 02:23 |
| vhosakot_ | are 730, 731 and 732 the VLAN Ids trunked on bond0? | 02:23 |
| *** v1k0d3n has joined #openstack-kolla | 02:24 | |
| *** kbyrne has joined #openstack-kolla | 02:24 | |
| britthouser | hmm....not sure how inc0 did that then. | 02:24 |
| britthouser | oh...lets check the lxc-container | 02:25 |
| britthouser | he might have done it in there | 02:25 |
| *** zhubingbing has quit IRC | 02:26 | |
| *** duonghq_ has joined #openstack-kolla | 02:26 | |
| *** Dupengfei has joined #openstack-kolla | 02:26 | |
| Dupengfei | Hello, everybody, I am a newer in openstack contributor. | 02:28 |
| sdake | bingo | 02:28 |
| *** pbourke has quit IRC | 02:29 | |
| *** duonghq has quit IRC | 02:29 | |
| sdake | hey guys need to run to store for 10 mins bbiaf ok? | 02:30 |
| britthouser | sure thing! | 02:30 |
| *** duonghq_ is now known as duonghq | 02:32 | |
| *** zhubingbing_ has joined #openstack-kolla | 02:33 | |
| *** frzn_ has quit IRC | 02:33 | |
| britthouser | So in ubuntu, there is just a single file which has all the interfaces | 02:33 |
| britthouser | In Centos, each interface has its own file. | 02:34 |
| *** dwalsh has quit IRC | 02:34 | |
| duonghq | britthouser, /etc/network/interfaces | 02:34 |
| *** fguillot_ has quit IRC | 02:34 | |
| vhosakot_ | britthouser: did the steps you did on the targetnnode .45 work ? If yes, can we try to add those steps in lineinfile ? | 02:34 |
| britthouser | Right duonghq, taht the ubuntu file | 02:34 |
| britthouser | vhosakot_ - they worked, but they aren't everything | 02:34 |
| vhosakot_ | britthouser: ahok | 02:34 |
| britthouser | we need to divide the playbooks into two categories: 1) interfaces files that already exist and just need some lineinefile plays | 02:35 |
| duonghq | ah, I just blackout for awhile, so I lost the context | 02:35 |
| britthouser | 2) interfaces that don't yet exist and we can use templates to create | 02:35 |
| britthouser | make sense? | 02:35 |
| britthouser | So we can ansiblize the ens1f1 and ens4f1 config changes using lineinfile | 02:35 |
| britthouser | and the templatie the bond0, bond0.whatever and ens4f1.whatever files. | 02:36 |
| vhosakot_ | britthouser: yes, for 2), do we write a shell script and call ansible's "shell" module to run the shell script on all the nodes ? | 02:36 |
| britthouser | no, lets use template simliar to what we have with interfaces.j2 | 02:36 |
| britthouser | b/c the Ip address logic will be the same | 02:36 |
| britthouser | we can reuse | 02:36 |
| britthouser | we'll just have interface.bond0.j2 and interface.bond0.733.j2 etc | 02:37 |
| vhosakot_ | right right.. cool | 02:37 |
| britthouser | make sense? | 02:37 |
| vhosakot_ | yes... | 02:37 |
| duonghq | which host distro do we support? Ubuntu 14.04/16.04 and??? | 02:37 |
| vhosakot_ | so, three templates to create 3 new files on each node | 02:37 |
| vhosakot_ | right ? | 02:37 |
| *** srwilkers_ has joined #openstack-kolla | 02:37 | |
| britthouser | I think 6? | 02:37 |
| *** srwilkers_ has left #openstack-kolla | 02:37 | |
| britthouser | bond0, bond0.702, bond0.730, bond0.731, bond0.7732, ens4f1.733 | 02:38 |
| britthouser | 7732=732 | 02:38 |
| vhosakot_ | cool, and edit for the ens1f1 and ens4f1 files are done using lineinfile ? | 02:38 |
| britthouser | that's my thoughts. | 02:38 |
| vhosakot_ | cool, yes.. thats a great idea... | 02:39 |
| britthouser | You wanna start with the lineinfile part, and I'll watch? | 02:39 |
| *** huikang has quit IRC | 02:39 | |
| britthouser | or vice versa? what do you think? | 02:39 |
| vhosakot_ | yes... I need to know the lines to change... I can do the lineinfile part.. cool | 02:39 |
| britthouser | just diff the file and file.orig | 02:39 |
| britthouser | I think you can pick out the differences? | 02:40 |
| *** esmiurium has quit IRC | 02:41 | |
| sdake | back | 02:42 |
| britthouser | vhosakot_ I think instead of commenting out, we can just remove | 02:42 |
| britthouser | Hey sdake | 02:42 |
| britthouser | I think we have a plan | 02:43 |
| britthouser | need to me re-iterate? or do yo have scrollback? | 02:43 |
| sdake | let me read | 02:43 |
| vhosakot_ | sdake: hey.. still playing with interface config files | 02:43 |
| vhosakot_ | britthouser: cool | 02:43 |
| sdake | ya keep at it - if we finish by midnight pst i should be able to run a baseline :) | 02:43 |
| britthouser | vhosakot_ lets copy the current files to .bond | 02:43 |
| vhosakot_ | britthouser: can I delete the lines commented out (with #) | 02:43 |
| britthouser | Yeah vhosakot_ lets delete the files with # | 02:43 |
| britthouser | and then copy to *.bond | 02:44 |
| vhosakot_ | coolsvap: | 02:44 |
| vhosakot_ | cool | 02:44 |
| britthouser | then copy the orig back | 02:44 |
| britthouser | and we can run our playbook and diff against the .bond file to make sure it turned out right. | 02:44 |
| *** dave-mccowan has quit IRC | 02:44 | |
| sdake | vhosakot_ if your using a tempalte you just use template and a globals..yml to go with it | 02:45 |
| *** esmiurium has joined #openstack-kolla | 02:45 | |
| vhosakot_ | cool | 02:45 |
| britthouser | lemme change one more thing vhosakot_ | 02:46 |
| vhosakot_ | britthouser: sure, go ahead | 02:46 |
| britthouser | ok that's cleaner | 02:46 |
| *** banix has quit IRC | 02:46 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add OSIC Scale Testing Documentation https://review.openstack.org/352101 | 02:46 |
| britthouser | Ok so it looks like we need to remove 10 file, change 2 lines, and add 3 lines. yes? | 02:47 |
| britthouser | remove 10 lines* | 02:48 |
| sdake | britthouser Jeffrey4l_ is here | 02:48 |
| sdake | Jeffrey4l_ can probbly helpl very fast get tihs problem sorted | 02:48 |
| britthouser | Hey Jeffrey4l_ | 02:48 |
| Jeffrey4l_ | sdake, britthouser hey | 02:48 |
| Jeffrey4l_ | what's wrong? | 02:48 |
| sdake | Jeffrey4l_ centos is deployed | 02:48 |
| *** banix has joined #openstack-kolla | 02:48 | |
| sdake | now need to configure netowrkign on 130 nodes | 02:49 |
| sdake | we have a reference impementation on one of the nodes of the cnfig files | 02:49 |
| sdake | need that rolled across 130 nodes | 02:49 |
| Jeffrey4l_ | write a ansible templates? | 02:49 |
| openstackgerrit | dupengfei proposed openstack/kolla: Fix Bug #1618294, Live migration failure bug. Modify vncserver_listen and server_listen to 0.0.0.0. Closes-Bug: #1618294 https://review.openstack.org/362577 | 02:49 |
| openstack | bug 1618294 in kolla "Live migration failure bug." [Undecided,New] https://launchpad.net/bugs/1618294 - Assigned to Pengfei Du (58750307-3) | 02:49 |
| sdake | that is what vhosakot_ is doing atm | 02:49 |
| britthouser | If you're in tmux 0, then on the screen you see the diff of the manual changes we made that need ansiblizing | 02:49 |
| Jeffrey4l_ | use ansible tempalte? | 02:49 |
| britthouser | We discussed, and I think there are two sets of changes we need. | 02:50 |
| sdake | Jeffrey4l_ yup - any chance yuo can jump in and write it quckly | 02:50 |
| Jeffrey4l_ | i am in the tmux 0 now. | 02:50 |
| britthouser | 1) changes to existing files (what you see now) | 02:50 |
| britthouser | 2) additional files that don't exist yet | 02:50 |
| britthouser | so I think templates for #2 | 02:50 |
| vhosakot_ | britthouser: I'm writing lineinfile to edit lines | 02:50 |
| sdake | you cn doo all of those with tmeplates | 02:50 |
| britthouser | I think we need UUID or HW_ADDRESS | 02:50 |
| britthouser | ok it sounds like vhosakot_ is writing some ansible not in the tmux. =) | 02:51 |
| sdake | dont need uuid but may need mac address | 02:51 |
| vhosakot_ | britthouser: I'm in tmux session 0 window 3 :) | 02:51 |
| britthouser | it works with UUID, so rather not rock the boat | 02:51 |
| vhosakot_ | wow, mac address need to be read from cobbler config for each node then | 02:51 |
| Jeffrey4l_ | uuid and mam doesn't matter. i think | 02:51 |
| Jeffrey4l_ | mam/mac | 02:52 |
| Jeffrey4l_ | if you do not point the mac, it use the physical mac address. | 02:52 |
| britthouser | well, what we narrowed down for these files Jeffrey4l_ was we need to remove 10 lines, change 2 lines and add 3 lines | 02:52 |
| vhosakot_ | britthouser: shall I write ansible lineinfile code to remove lines first? | 02:52 |
| britthouser | so it should be pretty easy playbook, and I think vhosakot_ is already writing it. | 02:52 |
| britthouser | go for it vhosakot_ | 02:52 |
| vhosakot_ | britthouser: cool | 02:52 |
| Jeffrey4l_ | ok | 02:52 |
| britthouser | Are you writing it not in the tmux? | 02:52 |
| britthouser | vhosakot_ ^^ | 02:53 |
| britthouser | if so, then Jeffrey4l_ and I can look at the 6 files we do need to create from template? | 02:53 |
| vhosakot_ | I am.. session 0 window 2 | 02:53 |
| vhosakot_ | britthouser: ^^ | 02:53 |
| sdake | btw Jeffrey4l_ i captured results for the fio test | 02:53 |
| Jeffrey4l_ | sdake, cool. and i captured too. and push it into the PS just now | 02:53 |
| vhosakot_ | britthouser: aren't you in session 0 window 3? | 02:53 |
| sdake | ok | 02:53 |
| britthouser | Ok, vhosakot_ I'll let you work on that uninterrupted. Ping us when you get it written and we can test | 02:54 |
| Jeffrey4l_ | sdake, i added scenario-11 ( fio test) and scenario-12( nova boot test) | 02:54 |
| vhosakot_ | britthouser: cool... I'll let you guys know.. | 02:54 |
| *** Pavo has quit IRC | 02:55 | |
| *** banix has quit IRC | 02:55 | |
| sbezverk_ | Jeffrey4l_: do you know ryu? | 02:56 |
| *** Pavo has joined #openstack-kolla | 02:56 | |
| Jeffrey4l_ | sbezverk_, no :( what happen? | 02:56 |
| sbezverk_ | Jeffrey4l_: I am struggling with vxlan tunnel between two compute nodes in kube, suspecting ryu which is openflow controller.. | 02:57 |
| sbezverk_ | Jeffrey4l_: but I have never dealt with it before, so I was hoping you might know.. | 02:58 |
| Jeffrey4l_ | sbezverk_, why introduce it? is it necessary? | 02:59 |
| sbezverk_ | Jeffrey4l_: I did not introduce it, it is in neutron | 03:03 |
| sbezverk_ | it is in kolla's build image | 03:03 |
| Jeffrey4l_ | sbezverk_, but i find nothing in the kolla' code about ryu? which line install that? | 03:05 |
| britthouser | @vhosakot_ feel free to overwrite the ifcfg-ens1f1 adn ifcfg-ens4f1 files in your play. I have made backup copy. | 03:06 |
| sbezverk_ | Jeffrey4l_: do you have by any chance ml2_conf.ini from multinode with vlan used for tenants and vxlan for tunnel? | 03:06 |
| vhosakot_ | britthouser: yep... cool | 03:06 |
| sbezverk_ | here you go: INFO:kolla.image.build.neutron-base:Installing collected packages: waitress, beautifulsoup4, WebTest, httplib2, logutils, neutron-lib, oslo.cache, psutil, oslo.reports, oslo.rootwrap, oslo.versionedobjects, ovs, singledispatch, pecan, ryu, neutron | 03:07 |
| sdake | Dupengfei hey there, looking to contribute to kolla or looking to find where you would like to contribute? | 03:07 |
| Jeffrey4l_ | sbezverk_, yep. I have a env use that env | 03:07 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 | 03:08 |
| sbezverk_ | Jeffrey4l_: as you can see it gets installed as part of neutron.. if you can share that ml2 conf, that would be awesome.. | 03:08 |
| Jeffrey4l_ | sbezverk_, http://paste.openstack.org/show/564724/ | 03:09 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Change package installation to use ansible package task https://review.openstack.org/362579 | 03:10 |
| sbezverk_ | Jeffrey4l_: thank you very much | 03:10 |
| Jeffrey4l_ | sbezverk_, neutron depends ryu. the requirements.txt has: ryu!=4.1,!=4.2,!=4.2.1,!=4.4,>=3.30 # Apache-2.0 | 03:11 |
| Jeffrey4l_ | OK. I got the reason. the openvswitch use the native openflow driver, which depend on ryu packages. | 03:13 |
| sbezverk_ | Jeffrey4l_: confused by your config. physnet1 is flat and vlan, never seen it before | 03:13 |
| sbezverk_ | this one from compute node right? | 03:14 |
| Jeffrey4l_ | sbezverk_, compute node and control node are the same | 03:14 |
| Jeffrey4l_ | I use physnet1 as the flat interface and value interface, that's OK | 03:14 |
| sbezverk_ | 172.18.2.101 this ip address is assigned to which interface? | 03:14 |
| Jeffrey4l_ | sbezverk_, customize the config by /etc/kolla/config/neutron/ml2_conf.ini check here http://paste.openstack.org/show/564725/ | 03:15 |
| Jeffrey4l_ | sbezverk_, eth5, another interface. | 03:15 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Change package installation to use ansible package task https://review.openstack.org/362579 | 03:16 |
| Jeffrey4l_ | sbezverk_, br-ex bridged to eth1 | 03:16 |
| *** mtaylor22 has joined #openstack-kolla | 03:16 | |
| sdake | Jeffrey4l_ you up for debugging the host install playbook for centos | 03:16 |
| Jeffrey4l_ | sdake, vhosakot_ is working on it. | 03:17 |
| sdake | Jeffrey4l_ or seeing it in action | 03:17 |
| sdake | no he is working on the network stup | 03:17 |
| sdake | our host playbook doesn't do host network sstup | 03:17 |
| Jeffrey4l_ | OK> install playbook. | 03:17 |
| vhosakot_ | sdake: I'm working on it.. almost done | 03:17 |
| Jeffrey4l_ | hmm. my osic_playbooks should work with centos. | 03:18 |
| Jeffrey4l_ | let me check | 03:18 |
| openstackgerrit | Merged openstack/kolla: Make the kolla_keystone_service can update fields https://review.openstack.org/348382 | 03:18 |
| *** zhubingbing_ has quit IRC | 03:18 | |
| sdake | Jeffrey4l_ - we have full hot playbooks i'd like to tyr out | 03:18 |
| sdake | Jeffrey4l_ installs docker, configures docker, intalls /etc/hostss etc | 03:18 |
| sdake | Jeffrey4l_ how do you creat ea new tmux? | 03:19 |
| openstackgerrit | Merged openstack/kolla: Enable the nova microversion api https://review.openstack.org/348432 | 03:19 |
| sdake | nm i got new tmux open | 03:19 |
| sdake | feel free to join :) | 03:19 |
| *** zhubingbing has joined #openstack-kolla | 03:22 | |
| Dupengfei | @sdake, hello , I am in | 03:23 |
| *** zhubingbing has quit IRC | 03:24 | |
| *** zhubingbing has joined #openstack-kolla | 03:24 | |
| britthouser | vhosakot_ - It hink I forget we need to also edit the ifcfg-enp4f0 file | 03:26 |
| Dupengfei | I 'd like to contribute to kolla and kolla-k8s preffered. | 03:26 |
| vhosakot_ | britthouser: cool..once the playbook is done.. we can run add more interfaces,... cool | 03:27 |
| britthouser | sounds good vhosakot_ | 03:28 |
| britthouser | I think I have the templates created for the other interfaces | 03:28 |
| britthouser | so should be pretty easy to add plays to render those when you're ready | 03:29 |
| sdake | Dupengfei cool - so best thing to do is ramp up | 03:29 |
| vhosakot_ | cool.. I'm almost done with lineinfile to remove, edit and add lines in ifcfg-* | 03:29 |
| sdake | Dupengfei have you got an aio deployment under yourbelet? | 03:29 |
| sdake | that is step 1 ;) | 03:29 |
| Dupengfei | sdake: I have a multinode deployment | 03:30 |
| Dupengfei | sdake: and work well. | 03:31 |
| sdake | Dupengfei nice! | 03:31 |
| Dupengfei | sdake: I tested most features on openstack. | 03:31 |
| Dupengfei | sdake: Sorry, I say I had a multinode deployment with kolla-ansible | 03:32 |
| sdake | Dupengfei we are rihgt at the end of our cycle | 03:32 |
| sdake | milestone 3 is the 31st | 03:32 |
| sdake | Dupengfei so as far as feature dev goes we dont need alot at this point - mostly testint bug fixes for kolla-ansible for newton | 03:32 |
| Dupengfei | sdake: kolla-k8s is not ready. | 03:32 |
| sdake | ya kolla-kubernetes wont be ready for some time | 03:33 |
| sdake | that is why its not 1.0.0 :) | 03:33 |
| sdake | its at pre-release versioining | 03:33 |
| Dupengfei | sdake: I see, my team is willing to use and update kolla-k8s. | 03:33 |
| sdake | nice | 03:34 |
| sdake | we need devs on that deliverable | 03:34 |
| britthouser | looks right vhosakot_ ? | 03:34 |
| sdake | rhallisey or sbezverk_ are the guys to tlak to about what work needs to be done | 03:34 |
| sdake | sbezverk_ is here right now | 03:34 |
| Dupengfei | sdake: so as far as feature dev goes we dont need alot at this point - mostly testint bug fixes for kolla-ansible for newton | 03:35 |
| Dupengfei | sdake: What's meaning, I can't understand. | 03:35 |
| sdake | we are not doing feature dev on the ansible code base for about another 6 weeks until we releases | 03:35 |
| sdake | or whenever it is | 03:35 |
| sdake | just bug fixes | 03:35 |
| Dupengfei | sdake: OH, I see. Forgive my poor english | 03:36 |
| sdake | britthouser in the other tmux - you can see the bootstrap playbook we hae in kolla proper | 03:38 |
| sdake | which installs the deps and whatnot | 03:38 |
| sdake | anddoes baic configuration | 03:38 |
| sdake | took about 7 minutes to run | 03:41 |
| britthouser | "in the other tmux" ? sdake? | 03:41 |
| vhosakot_ | ok britthouser, the lineinfile now works anf edits config files for ens1f1 and ens4f1.. should we edit files of ens1f0 and ens4f0 as well ? | 03:41 |
| vhosakot_ | anf=and* | 03:42 |
| britthouser | just ens4f0 | 03:42 |
| britthouser | and it will be similar but subset of what you did vhosakot_ | 03:42 |
| britthouser | check out the ifcfg-ens4f0 and ifcfg-ens4f0.orig | 03:42 |
| britthouser | it will be the same line changes, lien remove, but NOT the line adds | 03:42 |
| vhosakot_ | ah ok.. | 03:42 |
| britthouser | ok I'm in the other tmux sdake | 03:42 |
| vhosakot_ | britthouser: ens1f0 has the IP 172.21.0.45 from cobbler.. no need to edit its file ? | 03:43 |
| sdake | it already finished | 03:43 |
| britthouser | correct no need to edit that file | 03:43 |
| *** Dupengfei has left #openstack-kolla | 03:43 | |
| vhosakot_ | yes britthouser, I see the different edits needed in ifcfg-ens4f0.orig... just edits and removals, no additions.. | 03:43 |
| vhosakot_ | I'll add them to the inlineinfile playbook | 03:44 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Change package installation to use ansible package task https://review.openstack.org/362579 | 03:44 |
| britthouser | yeah and should be same edits and removals as the other files | 03:44 |
| britthouser | easy cust/paste It hnk | 03:44 |
| vhosakot_ | britthouser: cool | 03:44 |
| britthouser | or start using loops | 03:44 |
| britthouser | i.e. the ansible with: | 03:44 |
| britthouser | What are you the two issues you're seeing sdake? | 03:44 |
| sdake | docker-py is not installed thus far | 03:45 |
| sdake | openssl-devel and libffi-devel are installed as well which is wrong | 03:45 |
| *** salv-orlando has joined #openstack-kolla | 03:45 | |
| britthouser | the playbook looks like it installs docker[-py, but it actually doesn't? | 03:45 |
| sdake | its called python-docker-py | 03:46 |
| britthouser | yes...RH renamed that | 03:46 |
| britthouser | were openssl-devel and libffi-devel installed as dependancies? | 03:47 |
| *** huikang has joined #openstack-kolla | 03:47 | |
| britthouser | sdake: I'm headed back to tmux0 to see how vhosakot_ is getting along | 03:50 |
| sdake | sounds good | 03:50 |
| sdake | i think we are done in this one | 03:50 |
| vhosakot_ | britthouser: Im done.. testing on 45 | 03:50 |
| *** Dupengfei has joined #openstack-kolla | 03:51 | |
| britthouser | looks good vhosakot_ ! | 03:52 |
| britthouser | wanna do a live run? | 03:52 |
| vhosakot_ | britthouser: two things.. need to point to real files instead of *.orig.test files.. and need to point to entier inventory with 130 nodes, not to just 45 (test target node) | 03:53 |
| britthouser | well, lets get the other tempaltes in there too before the 130 nodes | 03:53 |
| *** salv-orlando has quit IRC | 03:53 | |
| britthouser | also, wanna clean it up a bit and make a loop? | 03:54 |
| vhosakot_ | britthouser: yep.. | 03:54 |
| britthouser | that's really up2 you. it works, don't break it. =) | 03:54 |
| vhosakot_ | yeah.. it works as is.. :) we can add loop and variable if you'd like | 03:55 |
| britthouser | meh...I'm on the fence vhosakot_ | 03:55 |
| britthouser | lets clean up later | 03:55 |
| sdake | the famous last words of every great inventor ;) | 03:56 |
| britthouser | ROFL | 03:56 |
| sdake | 50 years later people still cleaning up their messes :) | 03:56 |
| sdake | but in this case - no need to be perfect | 03:56 |
| sdake | whatever works ;) | 03:56 |
| britthouser | True. =) Fortunately (unfortunately?) we have a short deadline. | 03:56 |
| britthouser | yeah | 03:56 |
| vhosakot_ | britthouser: cool, functionality first, looks later :) | 03:56 |
| vhosakot_ | ok, setup_centos_networking.yml is ready and points to real files | 03:56 |
| vhosakot_ | shall we run on all 130 nodes | 03:56 |
| britthouser | not yet. | 03:56 |
| britthouser | Lets add 6 more plays to do the templates | 03:57 |
| vhosakot_ | sdake: you can see in tmux session 0 window 3 if you'd like | 03:57 |
| vhosakot_ | britthouser: cool | 03:57 |
| sdake | get the anestaphine from the medlock | 03:57 |
| vhosakot_ | lol | 03:57 |
| vhosakot_ | britthouser: can't we run setup_centos_networking.yml first and make the edit/replces/additions ? | 03:57 |
| britthouser | yes lets test on real files on one host | 03:58 |
| britthouser | then test adding templates on one host. | 03:58 |
| sdake | ya there has to be a better way long term | 03:58 |
| sdake | i think we will need network setup in our playbooks at some point | 03:58 |
| britthouser | then once we're happy, fire off the 130 | 03:58 |
| sdake | so keep a copy of that | 03:58 |
| britthouser | lemme revert the ifcfg-interfaces files | 03:58 |
| britthouser | and then you can try your places on real files | 03:58 |
| britthouser | ok go ahead with your live test vhosakot_ | 03:59 |
| britthouser | on just that one host | 03:59 |
| vhosakot_ | britthouser: sdake: inventory-ips-only is the right inventory file ? | 04:00 |
| sdake | roger vhosakot_ | 04:00 |
| vhosakot_ | britthouser: why do you want to revert your changes on the test target node (45)... if no matches found, lineinfile will not fail and continue to next line/file | 04:01 |
| *** zhubingbing has quit IRC | 04:01 | |
| britthouser | Call me extra careful | 04:01 |
| vhosakot_ | britthouser: cool :) yeah, actually back them up.. they are working configs | 04:02 |
| britthouser | They are backedup as .bond | 04:02 |
| vhosakot_ | cool.. shall i run for all 130 nodes then ? | 04:02 |
| britthouser | no | 04:02 |
| vhosakot_ | britthouser: ok | 04:02 |
| *** fragatina has joined #openstack-kolla | 04:02 | |
| britthouser | run just that one node again | 04:02 |
| vhosakot_ | britthouser: one node.. ok | 04:02 |
| britthouser | and then lets add all the vlan interfaces | 04:02 |
| vhosakot_ | cool | 04:02 |
| britthouser | on just that one node too | 04:03 |
| britthouser | and THEN go 130 nodes | 04:03 |
| vhosakot_ | britthouser: ran on one node | 04:03 |
| britthouser | +1 | 04:03 |
| britthouser | ok lets add the playbooks to render those tempaltes | 04:04 |
| britthouser | Since the interfaces.j2 tempalte was in teh container | 04:04 |
| vhosakot_ | britthouser: should I add the command to restart network service in the playbook ? | 04:04 |
| Dupengfei | byby, every one | 04:04 |
| *** Dupengfei has left #openstack-kolla | 04:04 | |
| britthouser | Yeah vhosakot_ | 04:04 |
| vhosakot_ | britthouser: cool | 04:04 |
| britthouser | also to modeprobe bonding | 04:04 |
| britthouser | modprobe 802.1x | 04:04 |
| britthouser | errr | 04:05 |
| britthouser | modprobe 8021q | 04:05 |
| *** fragatina has quit IRC | 04:07 | |
| britthouser | vhosakot_ I copied all the templates into /root on your deploy node | 04:07 |
| britthouser | have a look at one of them and tell me what you think | 04:08 |
| vhosakot_ | britthouser: cool.. I see ens4f0 is not part of bond0.. is this fine ? | 04:08 |
| britthouser | correct | 04:08 |
| vhosakot_ | cool | 04:08 |
| britthouser | only the *f1 interfaces are bonded | 04:08 |
| vhosakot_ | britthouser: did you delete the setup_centos_networking.loop.yml.. I dont see it | 04:09 |
| britthouser | yeah | 04:09 |
| vhosakot_ | cool | 04:09 |
| britthouser | since we weren't gonna do that | 04:09 |
| vhosakot_ | cool.. we can do later.. sure... | 04:09 |
| britthouser | you caught me. =P | 04:09 |
| vhosakot_ | ok, I now see you *.j2 files britthouser on the deploy node.. cool | 04:09 |
| britthouser | so we need to have plays to render those tempaltes on each node | 04:09 |
| vhosakot_ | yes.. there is a playbook.. 1 sec.. let me check | 04:10 |
| britthouser | in the osic-prep | 04:10 |
| britthouser | there is configure-bonds.yml | 04:11 |
| britthouser | which we can cut/paste the play | 04:11 |
| britthouser | it looks like this vhosakot_ | 04:12 |
| britthouser | https://paste.fedoraproject.org/417104/47253031/ | 04:12 |
| vhosakot_ | britthouser: cool.. thanks! | 04:12 |
| britthouser | so we just add 6 templates to your existing playbook | 04:12 |
| britthouser | we can ignore the ifdown/ifup since we restart networking | 04:13 |
| vhosakot_ | britthouser: should we run your templates on just one node first to test on the test targte node (.45) ? | 04:13 |
| britthouser | yes please. =) | 04:13 |
| vhosakot_ | britthouser: cool.. I'll do it now | 04:13 |
| britthouser | If you haven't caught on yet...I'm risk averse. =P | 04:13 |
| openstackgerrit | dupengfei proposed openstack/kolla: Modify Live migration failure bug. https://review.openstack.org/362577 | 04:14 |
| britthouser | dest is wrong @vhosakot_ | 04:16 |
| vhosakot_ | britthouser: what should it be ? | 04:16 |
| vhosakot_ | the older path was ubuntu's | 04:16 |
| britthouser | the first one should be /etc/sysconfig/network-scripts/ifcfg-bond0 | 04:16 |
| *** pbourke has joined #openstack-kolla | 04:16 | |
| britthouser | morning pbourke ! | 04:16 |
| sdake | britthouser my first director at intel called me a risk taker and green | 04:17 |
| britthouser | second one same but bond0.732 | 04:17 |
| vhosakot_ | britthouser: ah ok .. filename | 04:17 |
| britthouser | I've never been called a risk taker sdake. =P Probably why I haven't advanced as fast career wise. But for me, slow and steady wins the race. =) | 04:17 |
| sdake | britthouser the key with risk taking is its painful when your green | 04:18 |
| sdake | but when your not green and you take risks all your life - itleads to a 80-90% hit rate | 04:18 |
| sdake | all those risks taken = serious punishment | 04:18 |
| sdake | you know when you can win | 04:18 |
| sdake | and when to play no foldem holdem ;) | 04:18 |
| britthouser | With house paid for, I could probably stand to take more risk, but unfortunately I'm also change-averse. | 04:18 |
| britthouser | I really like my ruts | 04:19 |
| sdake | i dont mind change | 04:19 |
| vhosakot_ | britthouser is a rockstar :) | 04:19 |
| britthouser | ROFL | 04:19 |
| britthouser | I could eat the same cereal for breakfast my entire life and be happy. Drives my wife cray | 04:20 |
| britthouser | crazy | 04:20 |
| britthouser | one more vhosakot_ | 04:20 |
| britthouser | ifcfg-ens4f0.733.j2 | 04:20 |
| vhosakot_ | britthouser: ah ok | 04:21 |
| vhosakot_ | let me add it | 04:21 |
| britthouser | You like my back seat driving? I feel kinda like hyacinth bucket | 04:21 |
| vhosakot_ | lolol | 04:21 |
| vhosakot_ | ok britthouser... does it look good now ? | 04:22 |
| * britthouser crosses fingers | 04:22 | |
| vhosakot_ | britthouser: let me add the task to restart network service | 04:22 |
| britthouser | and modprobe? | 04:22 |
| britthouser | maybe you added that and I didn't see it. | 04:22 |
| britthouser | I'm 90% sure CentOS will do that automagically, but better safe than sorry | 04:23 |
| britthouser | you want bonding and 8021q | 04:23 |
| vhosakot_ | britthouser: just "modprobe bonding" is good | 04:23 |
| vhosakot_ | ? | 04:23 |
| sdake | vhosakot_ he menas in the plaboook | 04:23 |
| britthouser | yeah ^^ | 04:24 |
| sdake | i already insserted that module on45 | 04:24 |
| vhosakot_ | sdake: yes.. I asked it for the playbook :) | 04:24 |
| sdake | centos does not do it automatically - atleast not on systemctl restart network | 04:24 |
| vhosakot_ | cool.. let me add it | 04:24 |
| *** berendt has joined #openstack-kolla | 04:25 | |
| britthouser | vhosakot_ add 8021q after bonding | 04:25 |
| vhosakot_ | britthouser: like "modprobe bonding 8021q" | 04:25 |
| vhosakot_ | ? | 04:25 |
| britthouser | yup vhosakot_ | 04:25 |
| vhosakot_ | britthouser: cool looks good? | 04:26 |
| * britthouser crosses fingers again | 04:26 | |
| britthouser | yup | 04:26 |
| vhosakot_ | ok, shall we run the play book on 45 ? | 04:26 |
| britthouser | yup | 04:26 |
| vhosakot_ | and may be on 46 as well (that has all the origin configs from cobbler).. | 04:26 |
| britthouser | sounds good | 04:26 |
| vhosakot_ | britthouser: cool.. here I go! | 04:27 |
| britthouser | I htink we just ignore errors there | 04:28 |
| britthouser | b/c they all got added | 04:28 |
| britthouser | hmm...the bond0 didn't get their IPs | 04:28 |
| sdake | double check their content | 04:29 |
| sdake | ya yu cn't just add bonds and systemctl restart network | 04:29 |
| sdake | hve to reboot | 04:30 |
| vhosakot_ | there is no IP inifcfg-bond0 britthouser | 04:30 |
| sdake | because network tries to down the interface | 04:30 |
| vhosakot_ | ah ok | 04:30 |
| sdake | and the interface is new | 04:30 |
| sdake | but make sure the contents o the file are correct | 04:30 |
| sdake | or the reboot will crater | 04:30 |
| sdake | i mean on the target.. | 04:30 |
| britthouser | yeah the files look good | 04:31 |
| britthouser | we're missing something though.... | 04:31 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 | 04:31 |
| vhosakot_ | britthouser: yeah.. none of the sub-interafces for IPs | 04:31 |
| vhosakot_ | hmmmm | 04:31 |
| vhosakot_ | why? | 04:31 |
| britthouser | ens4f0.733 does | 04:32 |
| britthouser | just none of teh bonds | 04:32 |
| berendt | good morning | 04:32 |
| sdake | hey berendt | 04:32 |
| britthouser | good morning berendt | 04:32 |
| sdake | if files look good reboot | 04:33 |
| sdake | ON THE TARGET | 04:33 |
| sdake | if they dont look good on the target | 04:33 |
| sdake | dont reboot | 04:33 |
| *** jrist has quit IRC | 04:33 | |
| vhosakot_ | sdake: britthouser: shall I delete all the interfaces and rerrun | 04:33 |
| britthouser | no | 04:33 |
| vhosakot_ | ok | 04:33 |
| britthouser | # ifup bond0.731 | 04:33 |
| britthouser | Error: Connection activation failed: Connection 'Vlan bond0.731' is not available on the device bond0.731 at this time. | 04:33 |
| sdake | you cannot down a network with a bond after adding a bond later to the system | 04:33 |
| vhosakot_ | see I cannot retsrat network.serice manually as well | 04:34 |
| britthouser | I wonder if this is network manager again. | 04:34 |
| *** jrist has joined #openstack-kolla | 04:34 | |
| *** senk has joined #openstack-kolla | 04:34 | |
| britthouser | lets try adding NM_CONTROLLED=no | 04:34 |
| britthouser | and restart networking | 04:34 |
| britthouser | I think netowrking manager is trying to interfere | 04:34 |
| vhosakot_ | hmmm. I have seen it in past | 04:34 |
| britthouser | yeah taht is it. | 04:35 |
| britthouser | check out bond0.731 | 04:35 |
| britthouser | it had ip address now | 04:35 |
| britthouser | so lets edit our tempaltes | 04:35 |
| britthouser | add NM_CONTROLLED=no | 04:35 |
| britthouser | and rerun | 04:35 |
| britthouser | you are one step ahead of me vhosakot_ | 04:35 |
| vhosakot_ | britthouser: looks good? | 04:35 |
| vhosakot_ | haha | 04:36 |
| britthouser | yes | 04:36 |
| vhosakot_ | cool, let me save and rerun | 04:36 |
| *** ntpttr- has quit IRC | 04:36 | |
| vhosakot_ | should we retsart NetworkManager as well ? | 04:37 |
| britthouser | all of the tempaltes need it | 04:37 |
| britthouser | one sec I'll add | 04:37 |
| vhosakot_ | cool, please | 04:37 |
| sdake | uninstall network manager more like it :) | 04:37 |
| vhosakot_ | lol... | 04:37 |
| sdake | NetworkManager is really a misnomer | 04:37 |
| sdake | it should be called: | 04:37 |
| sdake | NetworkMismanager | 04:37 |
| berendt | don't write it | 04:37 |
| berendt | :) | 04:37 |
| vhosakot_ | yeah, I have seen two services trouble netowking in Centos.. why does centos have two..network.service and NetworkManager.service | 04:38 |
| vhosakot_ | hahaha | 04:38 |
| vhosakot_ | NetworkMismanager.. lol | 04:38 |
| britthouser | ok try again vhosakot_ | 04:38 |
| vhosakot_ | britthouser: cool.. what did you do ? | 04:38 |
| britthouser | if we uninstall network manager, we'll probalby lose the DHCP address on ens1f0 | 04:38 |
| britthouser | I added NM_CONTROLLED=no to all the tempaltes | 04:38 |
| vhosakot_ | ah right | 04:38 |
| vhosakot_ | some already had it right ? | 04:38 |
| britthouser | yeah | 04:39 |
| vhosakot_ | cool.. re-running | 04:39 |
| vhosakot_ | worked! | 04:39 |
| vhosakot_ | yes!! | 04:39 |
| britthouser | hmm...730 has two addresses | 04:39 |
| vhosakot_ | hmmm | 04:39 |
| britthouser | I think I might have a typo in the template | 04:39 |
| britthouser | lemme check | 04:39 |
| vhosakot_ | cool | 04:39 |
| britthouser | # grep 730 ifcfg-* | 04:40 |
| britthouser | ifcfg-bond0.730.j2:DEVICE=bond0.730 | 04:40 |
| sdake | i know whats coming, cultavanue | 04:40 |
| britthouser | ifcfg-bond0.732.j2:DEVICE=bond0.730 | 04:40 |
| britthouser | yup | 04:40 |
| britthouser | woops | 04:40 |
| sdake | the serpents | 04:40 |
| britthouser | ok one more time | 04:40 |
| *** ntpttr- has joined #openstack-kolla | 04:40 | |
| vhosakot_ | cool.. | 04:40 |
| vhosakot_ | shall I run britthouser? | 04:40 |
| britthouser | yes | 04:40 |
| vhosakot_ | cool | 04:40 |
| britthouser | w00t | 04:41 |
| *** srwilkers_ has joined #openstack-kolla | 04:41 | |
| britthouser | Ok just for good measure, lets reboot .45 | 04:41 |
| vhosakot_ | !!! | 04:41 |
| *** ntpttr has quit IRC | 04:41 | |
| openstack | vhosakot_: Error: "!!" is not a valid command. | 04:41 |
| britthouser | I can do that while you run on .46 | 04:41 |
| britthouser | sound good? | 04:41 |
| *** ntpttr- is now known as ntpttr | 04:41 | |
| *** salv-orlando has joined #openstack-kolla | 04:41 | |
| sdake | !/bin/bash | 04:41 |
| openstack | sdake: Error: "/bin/bash" is not a valid command. | 04:41 |
| *** ntpttr__ has joined #openstack-kolla | 04:41 | |
| britthouser | .45 going down..... | 04:41 |
| vhosakot_ | britthouser: yes.. I'll run on 46 and may be 47 too.. can you check the bonding in 46 and 47 after I run.. | 04:41 |
| vhosakot_ | britthouser: r u rebooting 45 ? | 04:42 |
| britthouser | yes 45 is rebooting | 04:42 |
| britthouser | go ahead iwth 46 | 04:42 |
| vhosakot_ | cool.. | 04:42 |
| vhosakot_ | ok.. | 04:42 |
| vhosakot_ | once 45, 46, 47 are good... we'll blast it for all 130 modes | 04:42 |
| vhosakot_ | cool | 04:42 |
| britthouser | you config file is hardcoded vhosakot_ | 04:43 |
| britthouser | there you go. = | 04:43 |
| vhosakot_ | britthouser: .. we need to chanage it to "all" | 04:43 |
| vhosakot_ | running on 46 | 04:44 |
| vhosakot_ | ah, modprobe bombed | 04:44 |
| *** ntpttr has quit IRC | 04:44 | |
| * britthouser scratches my head | 04:44 | |
| britthouser | it worked fine manually | 04:45 |
| vhosakot_ | yeah | 04:45 |
| vhosakot_ | hmmm | 04:45 |
| britthouser | lemme remove and try it again? | 04:45 |
| vhosakot_ | britthouser: yes | 04:45 |
| *** srwilkers_ has quit IRC | 04:46 | |
| *** ntpttr has joined #openstack-kolla | 04:46 | |
| britthouser | add them one at a time | 04:46 |
| vhosakot_ | all 130 ? | 04:46 |
| britthouser | its putting them in the same command that is messing it up | 04:47 |
| britthouser | no the modprobe | 04:47 |
| britthouser | split it into two commands | 04:47 |
| britthouser | reboot on .45 looks good | 04:47 |
| vhosakot_ | britthouser: cool.. all the IP and bonds r good on 45 after reboot ? | 04:47 |
| britthouser | correct vhosakot_ | 04:47 |
| vhosakot_ | britthouser: cool.. do the bonds have partner MAC ? | 04:47 |
| britthouser | yes | 04:48 |
| vhosakot_ | britthouser: cool! what do you mean by add two modprobe commands | 04:48 |
| sdake | vhosakot_ ope nplaybook | 04:48 |
| britthouser | so just split the modprobe command into two | 04:48 |
| britthouser | run on 46 | 04:48 |
| britthouser | and I thnk we'll be ready to shoot all 130 of our toes off. =P | 04:48 |
| vhosakot_ | britthouser: ah, add just the modprobe command in a separate playbook? got it.. cool | 04:49 |
| britthouser | Your play where you do the modprobe vhosakot_ | 04:49 |
| britthouser | need to split it into two | 04:49 |
| britthouser | yes | 04:49 |
| *** daneyon has joined #openstack-kolla | 04:50 | |
| vhosakot_ | done.. shall I run on 46 again britthouser? | 04:51 |
| britthouser | you didn't make two modprobe command yet | 04:51 |
| sdake | if this didn't work try using shelll | 04:51 |
| sdake | shell module | 04:51 |
| britthouser | I didn't mean put it in a different playbook | 04:51 |
| vhosakot_ | I moved modprobe and networkstart to separate playbook.. | 04:51 |
| vhosakot_ | haha.. then what? :) | 04:52 |
| britthouser | I mean run two commands in same playbook | 04:52 |
| britthouser | modprobe bonding | 04:52 |
| britthouser | modprobe 8021q | 04:52 |
| vhosakot_ | ah ok ok | 04:52 |
| britthouser | insert each kernel module as its own command | 04:52 |
| sdake | or use shell module ;) | 04:52 |
| vhosakot_ | sdake: yes.. we can shell module as well to run them on all nodes.. :) | 04:53 |
| britthouser | even when I try to load both manually from shell I get error sdake | 04:53 |
| vhosakot_ | just adding to playbook | 04:53 |
| sdake | britthouser roger | 04:53 |
| * britthouser crosses fingers | 04:53 | |
| britthouser | clear to luanch on 46 | 04:54 |
| vhosakot_ | cool.. yep.. lauching now on 46! | 04:54 |
| sdake | vhosakot_ a bit cavalier with that -rf operation ;) | 04:54 |
| *** zhurong has quit IRC | 04:54 | |
| vhosakot_ | haha.. yeah... :) | 04:54 |
| britthouser | lol | 04:54 |
| britthouser | +2 | 04:55 |
| britthouser | clear to launch on all 130 | 04:55 |
| sdake | i'e learned not to use -rf because it results in thigns like rm -rf /var/lib/docker * | 04:55 |
| vhosakot_ | britthouser: does it all look good? | 04:55 |
| britthouser | yes vhosakot_ looks good | 04:55 |
| sdake | just becuae i like typing extra spaces! | 04:55 |
| britthouser | extra space there sdake ? | 04:55 |
| britthouser | lol | 04:55 |
| *** daneyon has quit IRC | 04:55 | |
| vhosakot_ | sdake: yes.. yeah, I deleted evetying once using rm -rf.. | 04:55 |
| sdake | happend more then once | 04:56 |
| vhosakot_ | britthouser: should we ping the switch (the other end of bond0) ? | 04:56 |
| sdake | i use -r, but not -f | 04:56 |
| britthouser | no vhosakot_ | 04:56 |
| britthouser | I'm satisifed we'll have a clean run on the rest | 04:56 |
| vhosakot_ | britthouser: sdake ok! should I run on 47 as well.. or, shall we fire on 130 ? | 04:57 |
| sdake | do all 130 | 04:57 |
| britthouser | I feel good about doing the 130 | 04:57 |
| sdake | time it plz | 04:57 |
| vhosakot_ | yep | 04:57 |
| vhosakot_ | should we reboot 46 and test britthouser sdake? | 04:57 |
| britthouser | no...the 45 reboot was sufficient | 04:57 |
| britthouser | dno't forget to remove your hardcode | 04:57 |
| britthouser | host | 04:58 |
| vhosakot_ | britthouser: yes | 04:58 |
| vhosakot_ | sdake: inventory-ips-only is the right inventory pointing to 130 nodes woth 130 IPs, right ? | 04:58 |
| vhosakot_ | with* | 04:58 |
| sdake | vhosakot_ more or less | 04:58 |
| vhosakot_ | coo | 04:58 |
| vhosakot_ | cool | 04:58 |
| sdake | some nodes are ubuntu - we have those commented out | 04:58 |
| *** huikang has quit IRC | 04:58 | |
| *** ayoung has quit IRC | 04:58 | |
| vhosakot_ | sdake: cool.. I do see 170 IPs in inventory-ips-only.. cool | 04:59 |
| vhosakot_ | britthouser: yeah.. I'll remove the hardcoded IP and change it to "all" | 04:59 |
| sdake | 170 ips? | 04:59 |
| vhosakot_ | sorry 130 | 05:00 |
| vhosakot_ | "cat inventory-ips-only | more | grep 172 | wc -l" returns 130 sdake | 05:00 |
| vhosakot_ | cool | 05:00 |
| vhosakot_ | britthouser: sdake: shall I fire for the entire cluster? | 05:00 |
| britthouser | yes | 05:00 |
| britthouser | You are clear for launch vhosakot_ | 05:01 |
| vhosakot_ | cool and thanks for adding the "time" sdake! :) | 05:01 |
| vhosakot_ | started!! | 05:01 |
| * britthouser britthouser is disconnected | 05:01 | |
| britthouser | ....j/k | 05:02 |
| vhosakot_ | noooo.. | 05:02 |
| vhosakot_ | :)_ | 05:02 |
| vhosakot_ | this is the power of a config mgmt tool like ansible.. :) | 05:02 |
| vhosakot_ | althought this would have been totally posisble using python, pexpect, TCL, bash... Ansible is very easy | 05:02 |
| britthouser | I sorta wish you could make libraries out of plays. But the plays are so simple to write, its not that much effort to re-invent the wheel | 05:03 |
| vhosakot_ | right | 05:03 |
| vhosakot_ | done! | 05:03 |
| sdake | cool 2 min | 05:03 |
| britthouser | dang....my screen is still scrolling | 05:03 |
| sdake | an d that oculd be highly optimized | 05:03 |
| * britthouser shakes fist at DSL | 05:04 | |
| sdake | britthouser are yo urnning through the docker proxy? | 05:04 |
| vhosakot_ | true | 05:04 |
| sdake | i.e. 28.8k modem | 05:04 |
| britthouser | LOL | 05:04 |
| vhosakot_ | did 130 nodes run in parallel? | 05:04 |
| britthouser | its about that sdake. | 05:04 |
| sdake | vhosakot_ default is 8 threads i think | 05:04 |
| vhosakot_ | ah ok sdake | 05:04 |
| britthouser | now I see its done | 05:04 |
| sdake | ok now i guess its tie to deploy | 05:04 |
| britthouser | sheesh | 05:04 |
| vhosakot_ | sdake: should we add the OSIC inventory file to the kolla repo incase someone wants to deploy multinode with 100+ noes to scale | 05:05 |
| sdake | after we conigure globals.yml | 05:05 |
| vhosakot_ | sdake: let us deploy... | 05:05 |
| vhosakot_ | scenario 2 ? is the globals.yml ready ? | 05:05 |
| sdake | need to configure gloals.yml | 05:05 |
| britthouser | So this is where my expertise drops off, but I'd like to watch | 05:05 |
| sdake | yup | 05:05 |
| sdake | let me drive | 05:05 |
| vhosakot_ | sure, please sdake | 05:05 |
| vhosakot_ | britthouser: sdake: are you sure we dont need to check networking on the nodes after the playbook ended? | 05:06 |
| britthouser | I think we can do that in background while sdake forges ahead | 05:06 |
| sdake | nah i need help on globals.yml | 05:07 |
| vhosakot_ | sdake: I think we need to add this globals.yml and inventory we used in OSIC | 05:07 |
| sdake | vhosakot_ i am making a tarball of /root when we wrap up | 05:07 |
| vhosakot_ | sdake: neutron_external_interface shold be the interface without any IP, right ? | 05:07 |
| britthouser | p4p1 == ens4f1 | 05:07 |
| sdake | so we will have a full archive | 05:07 |
| vhosakot_ | sdake: cool | 05:07 |
| britthouser | 4 before the f sdake | 05:07 |
| vhosakot_ | sdake: copy from ip a | grep ens | 05:08 |
| britthouser | doh | 05:10 |
| vhosakot_ | no apt-get? | 05:10 |
| britthouser | yum | 05:10 |
| vhosakot_ | ah | 05:10 |
| vhosakot_ | cool :) | 05:10 |
| vhosakot_ | sdake: on which OS do you expect better kolla performance ? centOS or ubuntu ? | 05:11 |
| sdake | centos vs 14.04? | 05:11 |
| sdake | centos | 05:11 |
| sdake | more modern compiler | 05:11 |
| vhosakot_ | cool | 05:11 |
| vhosakot_ | was the ubuntu on the target nodes in OSIC 14.04 or xenial (16) ? | 05:12 |
| vhosakot_ | sdake: ^^ | 05:12 |
| vhosakot_ | sdake: password? | 05:13 |
| sdake | 14.04 | 05:13 |
| sdake | xenail we can't get to work | 05:13 |
| vhosakot_ | sdake: cool.. what about kolla-genpwd? | 05:13 |
| sdake | its already configured | 05:13 |
| vhosakot_ | cool | 05:13 |
| vhosakot_ | nice!! | 05:13 |
| vhosakot_ | sdake: how long will with deploy take ? | 05:13 |
| sdake | 20 minutes typically | 05:14 |
| vhosakot_ | cool.. i'll be there.. after that, will you run your script that calculates time taken by each kolla-ansible operation ? | 05:14 |
| sdake | nah - i'll run a rally report | 05:14 |
| vhosakot_ | cool | 05:14 |
| sdake | that takes 6 hours | 05:15 |
| sdake | hopefully we will still hvae the cluster by then | 05:15 |
| sdake | Jeffrey4l_ need help | 05:15 |
| sdake | Jeffrey4l_ something failed in deploy | 05:15 |
| sdake | i've seen this before on the bootstrap playbook | 05:16 |
| britthouser | when I add 'ip addr' | 05:16 |
| britthouser | kernel is calling that interface bond0.731@bond0 | 05:16 |
| britthouser | maybe that what should be in the play? | 05:16 |
| britthouser | sdake ^^ | 05:17 |
| vhosakot_ | sdake: are the ceph storage nodes setup for ceph ? ceph-osd failed on storage nodes | 05:17 |
| *** unicell has joined #openstack-kolla | 05:18 | |
| britthouser | probably the rest there too | 05:18 |
| vhosakot_ | britthouser: cool catch on @bond0 | 05:19 |
| vhosakot_ | britthouser is james bond :) | 05:19 |
| britthouser | well don't thank me yet | 05:19 |
| britthouser | could be red herring | 05:19 |
| *** unicell has quit IRC | 05:19 | |
| vhosakot_ | sdake: what is wrong? | 05:22 |
| britthouser | could the @ be throwing it off sdake? | 05:22 |
| sdake | who knows | 05:23 |
| britthouser | maybe \@ | 05:23 |
| sdake | unlikely | 05:23 |
| * britthouser throws some darts at well | 05:23 | |
| sdake | i saw the interface in a lower-up state | 05:23 |
| sdake | then it went to up | 05:23 |
| sdake | so i dont get how that happened | 05:23 |
| vhosakot_ | sdake: do you haven working gloabsl.yml with bond from ubuntu testing? | 05:23 |
| sdake | no | 05:23 |
| vhosakot_ | sdake: did you @ bond0in ubuntu | 05:24 |
| britthouser | no ubuntu didn't have @bond0@ in the interface name | 05:24 |
| britthouser | that is something rhel kernel is adding | 05:24 |
| vhosakot_ | britthouser: ah ok | 05:24 |
| *** unicell has joined #openstack-kolla | 05:24 | |
| britthouser | can we see the error again sdake? | 05:25 |
| britthouser | can we get the inventory of one of those nodes | 05:26 |
| sdake | moment thinking | 05:27 |
| *** zhubingbing has joined #openstack-kolla | 05:28 | |
| sdake | no idea why its running that on .23 | 05:28 |
| sdake | Jeffrey4l_ wtb some assisstance | 05:28 |
| sdake | Jeffrey4l_ i knwo your beat - i am too, the pain is almost over | 05:28 |
| britthouser | whenever I've seen that error, its b/c the var I'm referencing isn't in teh inventory | 05:29 |
| britthouser | maybe we can browse the inventory and see what it thinks that interface name is | 05:29 |
| sdake | how would bond0 be in the invetory? | 05:29 |
| sdake | intreface names are on globals.yml | 05:29 |
| britthouser | no i mean where you say, "ansible tell me everything you know about this host" | 05:29 |
| britthouser | maybe inventory isn't the right term | 05:29 |
| vhosakot_ | all interfaces in LOWER_UP.. is this good? | 05:30 |
| *** zhurong has joined #openstack-kolla | 05:30 | |
| britthouser | ansible -m <host> setup | 05:30 |
| britthouser | will dump the vars | 05:31 |
| vhosakot_ | sdake: can I rerun with verbose? -vvvv ? | 05:31 |
| sdake | you can - but you need to cleanup and relabel | 05:31 |
| sdake | who is driving atm? | 05:31 |
| britthouser | not me | 05:31 |
| vhosakot_ | sdake: yes, doing it | 05:31 |
| vhosakot_ | runinng cleanuo | 05:31 |
| britthouser | try ansible 172.21.0.27 -m setup -i inventory-ips-only | 05:32 |
| britthouser | it shoudl be ansible_bond0.731 | 05:32 |
| vhosakot_ | britthouser: ah ok.. I'll run it once the playbook ends | 05:33 |
| britthouser | I sent us on wild goose chase with teh @bond0@ nonsense | 05:33 |
| sdake | it definately shouldn't e ansible_bond0 :) | 05:33 |
| britthouser | right sdake | 05:33 |
| britthouser | that is the variable name | 05:33 |
| sdake | what I mean is it shuldn't be prefixed with ansible_ | 05:33 |
| britthouser | right | 05:33 |
| britthouser | it should just be bond0.731 | 05:33 |
| britthouser | and then playbook should be referencing ansible_bond0.731 | 05:33 |
| sdake | that is what we had originally | 05:33 |
| britthouser | yeah. =/ | 05:34 |
| sdake | the playbook automatically references ansible_ | 05:34 |
| britthouser | If you look at ansible 172.21.0.27 -m setup -i inventory-ips-only | 05:34 |
| zhubingbing | hi | 05:34 |
| britthouser | you can see all variables | 05:34 |
| sdake | i have sseen this problem and inc0 fixed it | 05:34 |
| sdake | but not sure what he did | 05:35 |
| vhosakot_ | britthouser: I ran it.. | 05:35 |
| vhosakot_ | I see ansible_bond0.731 | 05:35 |
| britthouser | so in etherpad | 05:35 |
| britthouser | we have p4p1.733 being the cluster network | 05:35 |
| britthouser | is that what storage_network is? the ceph cluster network? | 05:35 |
| sdake | where is the depploy -vvv output | 05:35 |
| vhosakot_ | britthouser: but, the storage node does not have any itnerface ncamed p4p1.733 | 05:36 |
| vhosakot_ | sdake: further up | 05:36 |
| sdake | ok taking a break - keep a tit guys :) | 05:36 |
| vhosakot_ | sdake: see now | 05:36 |
| sdake | get Jeffrey4l_ involved - he can debug easily | 05:36 |
| vhosakot_ | sdake: this is the -vvv output | 05:36 |
| britthouser | ok thx sake | 05:36 |
| britthouser | sdake | 05:36 |
| sdake | i'll be back in 15 mins or so | 05:36 |
| vhosakot_ | sdake: cool | 05:36 |
| britthouser | can we look at the error again vhosakot_ ? | 05:37 |
| vhosakot_ | britthouser: sure... | 05:37 |
| vhosakot_ | the error and debug (-vvvv) is in tmux session 0 window 2 | 05:37 |
| vhosakot_ | window 3, not 2 | 05:37 |
| britthouser | ok taht's stil got teh @bond0@ in the name | 05:38 |
| britthouser | lets revert that | 05:38 |
| britthouser | and rerun | 05:38 |
| vhosakot_ | britthouser: cool.. let me rerun | 05:38 |
| *** DuPengfei has joined #openstack-kolla | 05:39 | |
| vhosakot_ | britthouser: should I remove all @bond0, or just the storage network? | 05:39 |
| britthouser | all the bond0 | 05:39 |
| vhosakot_ | britthouser: cool | 05:39 |
| britthouser | and make cluster_interface ens4f0.733 | 05:39 |
| britthouser | no wait | 05:40 |
| britthouser | vhosakot_ | 05:40 |
| vhosakot_ | britthouser: ah ok | 05:40 |
| britthouser | lemme just dobule check all these one sec | 05:40 |
| vhosakot_ | britthouser: sure, please | 05:40 |
| vhosakot_ | britthouser: is it good? | 05:41 |
| britthouser | neutron external should be bond0.732 I think | 05:41 |
| britthouser | unless the etherpad is out of date.... | 05:42 |
| vhosakot_ | let me change it to bond0.732 britthouser | 05:42 |
| britthouser | ok give it a shot | 05:43 |
| vhosakot_ | cool | 05:43 |
| vhosakot_ | britthouser: now it complains that ansible_bond0.731 is not found | 05:45 |
| britthouser | my screen still scrolls..... | 05:46 |
| britthouser | which host/ | 05:46 |
| *** sdake has quit IRC | 05:47 | |
| vhosakot_ | britthouser: all storage hosts complain the same.. same error | 05:47 |
| britthouser | but taht is cleary in the variables for that host | 05:48 |
| britthouser | Jeffrey4l_ got a minute? | 05:48 |
| *** sdake has joined #openstack-kolla | 05:50 | |
| vhosakot_ | britthouser: I'll add a debug in the ceph playbook | 05:50 |
| sdake | do wehave the correct vereson of anible | 05:50 |
| vhosakot_ | I'm printing all the variables in the playbook itself.. | 05:51 |
| britthouser | yeah its sthe same deploy ndoe, right? | 05:51 |
| britthouser | # pip list | grep ansible | 05:51 |
| britthouser | ansible (2.1.1.0) | 05:51 |
| sdake | lets run yum upgrade on all the nodes | 05:53 |
| sdake | and reboot them | 05:53 |
| britthouser | I'll run manually on .45 first | 05:53 |
| sdake | ok | 05:53 |
| sdake | yum -y upgrade should do it | 05:54 |
| britthouser | yum update && yum -y upgrade | 05:55 |
| britthouser | is what I did | 05:55 |
| vhosakot_ | yeah that should do | 05:55 |
| britthouser | and then probably need touch /.autorelabel and reboot | 05:55 |
| vhosakot_ | I'm runnign with debugs in playbook | 05:55 |
| britthouser | but I'll wait till yoru plays are done vhosakot_ | 05:55 |
| vhosakot_ | britthouser: cool | 05:55 |
| *** mewald has joined #openstack-kolla | 05:56 | |
| britthouser | can i reboot vhosakot_ ? | 05:56 |
| britthouser | just .45 | 05:57 |
| sdake | might as well wait until its up | 05:57 |
| britthouser | nvm on the autorelabel | 05:57 |
| sdake | and run playbook | 05:57 |
| britthouser | selinux is disabled | 05:57 |
| sdake | ya we do that atm | 05:57 |
| sdake | although msater can run with xelinux enabled | 05:57 |
| sdake | if the disk is labeled properly ;-) | 05:58 |
| britthouser | taht's what /.autorelabel is for. =) | 05:58 |
| sdake | i'd lilke something more surgical ;-)o | 05:58 |
| britthouser | i.e. if you're been running with it off for awhile, or sometimes when your docker selinux policy changes drastically | 05:58 |
| britthouser | we've hit both cases | 05:58 |
| britthouser | ok, rebooting vhosakot_ | 05:59 |
| vhosakot_ | britthouser: wait | 05:59 |
| * britthouser takes finger off return key | 05:59 | |
| vhosakot_ | britthouser: ok reboot... got for it! | 05:59 |
| *** jrich523 has quit IRC | 06:00 | |
| vhosakot_ | I do see the variabel ""storage_interface": "bond0.731", " at runtime in the playbbok | 06:00 |
| vhosakot_ | so not like the variable is missing at runtime | 06:00 |
| sdake | could be nasible bug | 06:00 |
| sdake | i'd like Jeffrey4l_ to take a look | 06:00 |
| sdake | he would have a good handle on thedetila of this one | 06:00 |
| sdake | he is typically around this time | 06:01 |
| sdake | maybe at lunch | 06:01 |
| Jeffrey4l_ | sdake, what's the bug? | 06:02 |
| britthouser | Jeffrey4l_ ansible is spitting back it can't find a dict value | 06:02 |
| sdake | Jeffrey4l_ we are doing our laswt centos test for the osic cluster | 06:02 |
| britthouser | but we easily find it in the host invenetory | 06:02 |
| sdake | Jeffrey4l_ and ansible is complaining about bnod0-731 | 06:02 |
| Jeffrey4l_ | britthouser, could u show me? | 06:03 |
| sdake | bound0.731 i mean | 06:03 |
| vhosakot_ | Jeffrey4l_: this is error | 06:03 |
| vhosakot_ | failed: [172.21.0.143] (item=ceph-osd) => {"failed": true, "invocation": {"module_args": {"dest": "/etc/kolla//ceph-osd/config.json", "src": "ceph-osd.json.j2"}, "module_name": "template"}, "item": "ceph-osd", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute u'ansible_bond0.731'"} | 06:03 |
| britthouser | but then in the variables: | 06:03 |
| britthouser | # ansible 172.21.0.143 -m setup -i inventory-ips-only | grep bond0.731 | 06:03 |
| britthouser | "ansible_bond0.731": { | 06:03 |
| britthouser | "device": "bond0.731", | 06:03 |
| britthouser | "bond0.731", | 06:03 |
| vhosakot_ | sdake: did you use bond0.731 as storage_interafec in globals.yml on ubuntu ? | 06:05 |
| sdake | vhosakot_ no idea | 06:05 |
| vhosakot_ | cool | 06:05 |
| sdake | maybe version mismatch in the fact caching | 06:05 |
| sdake | which sucks | 06:06 |
| vhosakot_ | hmm | 06:06 |
| vhosakot_ | ansible fact caching? | 06:06 |
| britthouser | python 2.7.6 on the deploy node, and 2.7.5 on centos nodes | 06:06 |
| Jeffrey4l_ | that's should be OK | 06:06 |
| britthouser | Jeffrey4l_ ifyou log into the tmux0 | 06:06 |
| Jeffrey4l_ | i am in britthouser | 06:06 |
| sdake | sorry guys i can barely see straight | 06:07 |
| britthouser | Ok.... do you want to drive for a bit Jeffrey4l_ | 06:07 |
| Jeffrey4l_ | i removed the /tmp/ansible_fact_cache folder. | 06:07 |
| vhosakot_ | cool | 06:07 |
| vhosakot_ | I re-ran | 06:07 |
| * britthouser crosses fingers | 06:07 | |
| Jeffrey4l_ | mtaylor22, ping still around? | 06:07 |
| * vhosakot_ vhosakot has bitten his nails and fingers | 06:07 | |
| britthouser | one odd things that we think is just a red herring, the otuput of 'ip addr' shows 'bond0.731@bond0@' | 06:08 |
| *** salv-orl_ has joined #openstack-kolla | 06:08 | |
| britthouser | somthing different the rhel kernel is doing | 06:08 |
| vhosakot_ | looks like is it working | 06:08 |
| vhosakot_ | it never went this far, right? | 06:08 |
| sdake | annoying | 06:08 |
| vhosakot_ | Jeffrey4l_: what is this /tmp/ansible_fact_cache folder ? | 06:08 |
| britthouser | ahh...so since we ran ansible to create those interfaces | 06:08 |
| sdake | well let see if it runs rally now ;) | 06:09 |
| britthouser | the cached inventory didn't have them | 06:09 |
| mewald | Please review https://review.openstack.org/#/c/346449 | 06:09 |
| sdake | mewald i know you need reviews | 06:09 |
| britthouser | we should add removing that to the cleanup. | 06:09 |
| sdake | mewald core team supe rbusy with osic - we have 6-12 hours left of time | 06:09 |
| Jeffrey4l_ | there must some thing in ansible. | 06:09 |
| sdake | mewald then we are burning through the review queue - ok? | 06:09 |
| Jeffrey4l_ | the fact cache should only keep 3600s | 06:09 |
| vhosakot_ | yes, britthouser is right.. I think we must delete the /tmp/ansible_fact_cache folder in the cleanup script | 06:10 |
| Jeffrey4l_ | this is configuration in /root/.ansible.cfg, to speed up the deployment | 06:10 |
| sdake | is it still deploying? | 06:10 |
| mewald | sdake: yeah I am just pinging once in a while to remind everybody :) sure take your time! The chance to have OSIC must be used :) | 06:10 |
| vhosakot_ | sdake: yes.. | 06:10 |
| britthouser | Thanks Jeffrey4l_ you da man! | 06:10 |
| vhosakot_ | it did not come this far at all.. it dies within a min | 06:10 |
| *** salv-orlando has quit IRC | 06:11 | |
| sdake | ya we need to ctrol -c this | 06:11 |
| sdake | remove the debug junk | 06:11 |
| vhosakot_ | sdake: why? | 06:11 |
| vhosakot_ | sdake: ok | 06:11 |
| sdake | so i can get a deploy time | 06:11 |
| vhosakot_ | shall I remove my debug as well and re-run? | 06:11 |
| sdake | yes pleae | 06:11 |
| vhosakot_ | my debug will add time | 06:11 |
| sdake | but don't git reeset | 06:11 |
| vhosakot_ | sdake: ok, will do it | 06:11 |
| vhosakot_ | sdake: cool | 06:11 |
| *** salv-orlando has joined #openstack-kolla | 06:12 | |
| sdake | run deploy in script please | 06:13 |
| sdake | and save output | 06:13 |
| sdake | i nee to grab a 30 min nap - and then i'll caputre the data | 06:13 |
| vhosakot_ | sdake: which script? | 06:14 |
| mtaylor22 | Jeffrey4l_: yep sorry, I didn't have notifications enabled for irc | 06:15 |
| britthouser | vhosakot_ and sdake | 06:16 |
| sdake | Jeffrey4l_ - any chance you can finishthe rally tests | 06:16 |
| britthouser | should we yum upgrade and reboot all hosts | 06:16 |
| britthouser | or just leave .45 as the sole one upgraded? | 06:16 |
| sdake | upgrade all please | 06:17 |
| *** salv-orl_ has quit IRC | 06:17 | |
| sdake | here is what I'd liek | 06:17 |
| mtaylor22 | just out of curiosity, anyone had dramas with nova resize + ceph (volumes) not actually resizing the root fs? | 06:17 |
| sdake | upgrade lall nodes, reboot, run script (the command) to capture he deploy output (type exit to exit wscript) | 06:17 |
| *** salv-orlando has quit IRC | 06:17 | |
| vhosakot_ | sdake: the ceph cleanup complained that partion cannot be cleaned up and a reboot is needed.. hence, I'm rebooting the deploy node.. all tmux sessions will die | 06:17 |
| sdake | run it timed | 06:18 |
| sdake | no | 06:18 |
| sdake | that isn't the deploy node | 06:18 |
| sdake | that is the ceph nodde | 06:18 |
| sdake | all of the cluster nodes need to be rebooted | 06:19 |
| sdake | and deploy time caputred | 06:19 |
| sdake | jeffrey we are all supe tired here | 06:19 |
| sdake | Jeffrey4l_ any chance you could do what needs to be done her ewith the benchmarking? | 06:19 |
| vhosakot_ | sdake: I rebooted the deploy node.. the error showed the IP of the deploy node, not the ceph node | 06:20 |
| vhosakot_ | sdake: the deploy node is rebooted and back up fine | 06:20 |
| vhosakot_ | britthouser: I think all nodes must be upgraded | 06:21 |
| britthouser | yes | 06:21 |
| britthouser | yum update && yum -y upgrade | 06:21 |
| britthouser | reboot | 06:21 |
| britthouser | that's what I did on .45, and it went fine | 06:21 |
| vhosakot_ | sdake: which script must be used to deploy? | 06:21 |
| Jeffrey4l_ | sorry need be out for about 2 hours. | 06:27 |
| Jeffrey4l_ | i can take over this after back. | 06:27 |
| sdake | vhosakot_ cool thanks - i think Jeffrey4l_ can take it over nad i can get some real rest and hou gusy can go to bed :) | 06:28 |
| sdake | its like 3am there | 06:28 |
| vhosakot_ | sdake: yep.. going to bed.. good night.. thanks for all the help! | 06:28 |
| britthouser | good night vhosakot_ | 06:28 |
| britthouser | good night sdake | 06:29 |
| Jeffrey4l_ | good night | 06:29 |
| britthouser | Jeffrey4l_ - just to reiterate: all ndoes need to be yum upgraded, and then rebooted before deploy | 06:29 |
| vhosakot_ | Jeffrey4l_: you can add the commands to setup_centos_networking.yml in /root | 06:31 |
| *** vhosakot_ has quit IRC | 06:31 | |
| *** sdake has quit IRC | 06:44 | |
| britthouser | Ok I'm off to bed everone | 06:59 |
| *** apuimedo|away is now known as apuimedo | 07:00 | |
| duonghq | anyboday try sean k moony's boostrap-servers | 07:00 |
| zhubingbing | ping Eduardo Gonzalez | 07:01 |
| duonghq | he is not online zhubingbing | 07:03 |
| zhubingbing | thanks | 07:03 |
| zhubingbing | duonghq I found you always on line. | 07:04 |
| *** mtaylor22 has quit IRC | 07:05 | |
| duonghq | zhubingbing: our timezone is closer than American guys | 07:05 |
| *** egonzalez90 has joined #openstack-kolla | 07:12 | |
| *** mnasiadka has joined #openstack-kolla | 07:15 | |
| openstackgerrit | Merged openstack/kolla: Dockerfiles for collectd Container https://review.openstack.org/358632 | 07:15 |
| zhubingbing | ping egonzalez90 | 07:17 |
| *** rmart04 has joined #openstack-kolla | 07:20 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 07:21 |
| *** rkrum has quit IRC | 07:21 | |
| *** Mr_Broken has joined #openstack-kolla | 07:22 | |
| egonzalez90 | morning zhubingbing | 07:23 |
| zhubingbing | morning | 07:23 |
| zhubingbing | https://review.openstack.org/349351 | 07:24 |
| zhubingbing | can you review it | 07:24 |
| Mr_Broken | hey guys | 07:24 |
| Mr_Broken | anyone have time to look into this ? https://bugs.launchpad.net/kolla/+bug/1617188 | 07:25 |
| openstack | Launchpad bug 1617188 in kolla "neutron-ovs-cleanup missing" [Undecided,New] | 07:25 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 | 07:30 |
| egonzalez90 | zhubingbing, gnocchi_statsd_resource_id are random uuids or something like domain_id=default? | 07:32 |
| *** b_bezak has joined #openstack-kolla | 07:33 | |
| zhubingbing | random uuids | 07:33 |
| *** matrohon has joined #openstack-kolla | 07:33 | |
| *** mtaylor22 has joined #openstack-kolla | 07:33 | |
| *** neilus1 has joined #openstack-kolla | 07:34 | |
| *** neilus has quit IRC | 07:35 | |
| zhubingbing | https://github.com/openstack/gnocchi/blob/master/gnocchi/statsd.py | 07:35 |
| zhubingbing | egonzalez90 you can read gnocchi code | 07:36 |
| zhubingbing | You can look at this section of the code on the resouce_id ， link https://github.com/openstack/gnocchi/blob/master/gnocchi/statsd.py | 07:36 |
| * Daviey checks in | 07:43 | |
| zhubingbing | ok | 07:44 |
| *** senk has quit IRC | 07:52 | |
| *** mtaylor22 has quit IRC | 07:54 | |
| *** Serlex has joined #openstack-kolla | 07:56 | |
| *** shardy has joined #openstack-kolla | 08:03 | |
| *** Satya has joined #openstack-kolla | 08:03 | |
| openstackgerrit | Yao Lu proposed openstack/kolla: delete virtual bridge artifacts when cleaning hosts https://review.openstack.org/358924 | 08:09 |
| *** zhubingbing has quit IRC | 08:10 | |
| *** zhubingbing has joined #openstack-kolla | 08:17 | |
| *** phuongnh has joined #openstack-kolla | 08:19 | |
| *** gfidente has joined #openstack-kolla | 08:25 | |
| *** daneyon has joined #openstack-kolla | 08:27 | |
| zhubingbing | ping egonzalez90 | 08:30 |
| zhubingbing | https://review.openstack.org/#/c/349351/ | 08:30 |
| zhubingbing | can you review it ? | 08:30 |
| zhubingbing | https://review.openstack.org/#/c/348388/ | 08:31 |
| *** daneyon has quit IRC | 08:32 | |
| *** DuPengfei has quit IRC | 08:38 | |
| *** egonzalez90 has quit IRC | 08:59 | |
| *** dwalsh has joined #openstack-kolla | 09:08 | |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Allow use of database for Horizon sessions https://review.openstack.org/359794 | 09:23 |
| zhubingbing | ping sdake | 09:33 |
| zhubingbing | ping Daviey | 09:35 |
| duonghq | zhubingbing: OSIC made them very tired now | 09:41 |
| zhubingbing | － － | 09:41 |
| Daviey | zhubingbing: hey | 09:45 |
| zhubingbing | I have a question for rally that needs your help. | 09:46 |
| *** berendt has quit IRC | 09:47 | |
| zhubingbing | Rally install one step to initialize the database operation (rally-manage db recreate), this step operation, how should I add? | 09:47 |
| Daviey | umm | 09:48 |
| zhubingbing | But he's not as complicated as the other components, and he's just executing a command. | 09:49 |
| Daviey | zhubingbing: sorry, can you phrase the question again? | 09:49 |
| zhubingbing | rally-manage db recreate you think we should how to add for kolla | 09:50 |
| Daviey | zhubingbing: I'm sorry, but i'm not understanding the question | 09:50 |
| zhubingbing | Add rally role | 09:50 |
| zhubingbing | - - | 09:50 |
| Daviey | Oh! | 09:50 |
| Daviey | You want to add a permanent rally role to kolla | 09:50 |
| zhubingbing | yeah | 09:50 |
| Daviey | i see | 09:51 |
| Daviey | sorry, i thought you were trying to use rally | 09:51 |
| Daviey | zhubingbing: so, why does this differ from other services? | 09:51 |
| Daviey | zhubingbing: rally supports mysql, right? | 09:51 |
| zhubingbing | yes | 09:51 |
| Daviey | zhubingbing: so create the database, and then bootstrap rally-manage db recreate ? | 09:52 |
| Daviey | zhubingbing: I mean, you just need to mimic this to create the database: https://github.com/openstack/kolla/blob/0ff90a6f10494f9f33d665a3367dfaf0486b94df/ansible/roles/watcher/tasks/bootstrap.yml | 09:53 |
| zhubingbing | It seems as long as the implementation of this command rally-manage db recreate, do not need to create a database | 09:54 |
| zhubingbing | If this is so, I have no doubt, but rally initialization database on a command, do not create their own database | 09:55 |
| Daviey | zhubingbing: rally can use sqlite or mysql right? | 09:59 |
| zhubingbing | use qlite | 09:59 |
| Daviey | zhubingbing: I think it is reasonable for kolla rally role to use mysql | 09:59 |
| *** egonzalez90 has joined #openstack-kolla | 10:00 | |
| zhubingbing | i see， you advice to use mysql | 10:00 |
| zhubingbing | ok | 10:00 |
| *** Satya has quit IRC | 10:01 | |
| *** duonghq has left #openstack-kolla | 10:01 | |
| *** zhurong has quit IRC | 10:03 | |
| *** zhiwei has quit IRC | 10:06 | |
| Daviey | zhubingbing: i think so... it makes sense to use the databases we already use | 10:15 |
| Daviey | zhubingbing: and sqlite is going to be a pain | 10:15 |
| zhubingbing | ok | 10:15 |
| zhubingbing | good | 10:15 |
| *** daneyon has joined #openstack-kolla | 10:16 | |
| *** daneyon has quit IRC | 10:20 | |
| *** salv-orlando has joined #openstack-kolla | 10:27 | |
| openstackgerrit | Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 | 10:29 |
| *** apuimedo is now known as apuimedo|lunch | 10:29 | |
| *** ntpttr has quit IRC | 10:30 | |
| *** zhubingbing has quit IRC | 10:33 | |
| *** ntpttr has joined #openstack-kolla | 10:35 | |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 10:35 |
| *** phuongnh has quit IRC | 10:39 | |
| britthouser | Hey Jeffrey4l_ - how did OSIC go overnight | 10:51 |
| *** salv-orl_ has joined #openstack-kolla | 10:52 | |
| *** salv-orl_ has quit IRC | 10:54 | |
| *** apuimedo|lunch is now known as apuimedo | 10:54 | |
| *** salv-orlando has quit IRC | 10:56 | |
| Jeffrey4l_ | britthouser, i got a success deploy | 10:56 |
| britthouser | Hurray! | 10:57 |
| britthouser | Thanks for taking over last night | 10:57 |
| *** caowei has quit IRC | 10:57 | |
| *** rhallisey has joined #openstack-kolla | 10:59 | |
| *** berendt has joined #openstack-kolla | 11:00 | |
| *** dwalsh has quit IRC | 11:05 | |
| Jeffrey4l_ | britthouser, do u change anything in osic? | 11:09 |
| *** wznoinsk has quit IRC | 11:10 | |
| *** banix has joined #openstack-kolla | 11:26 | |
| *** wznoinsk has joined #openstack-kolla | 11:28 | |
| *** banix has quit IRC | 11:29 | |
| *** banix has joined #openstack-kolla | 11:30 | |
| britthouser | I have not Jeffrey4l_ | 11:38 |
| *** dwalsh has joined #openstack-kolla | 11:38 | |
| *** banix has quit IRC | 11:39 | |
| Jeffrey4l_ | ok. thanks. | 11:39 |
| *** dwalsh has quit IRC | 12:12 | |
| *** zhurong has joined #openstack-kolla | 12:15 | |
| *** banix has joined #openstack-kolla | 12:17 | |
| *** banix has quit IRC | 12:27 | |
| Daviey | Jeffrey4l_: hey, did you see that nova change for vnc ip? | 12:43 |
| Jeffrey4l_ | Daviey, yep. I have -2 for my PS. let's wait the nova bug is fixed. | 12:45 |
| *** egonzalez90 has quit IRC | 12:57 | |
| *** daneyon has joined #openstack-kolla | 12:58 | |
| *** sbezverk_ has quit IRC | 13:02 | |
| *** daneyon has quit IRC | 13:02 | |
| Daviey | Jeffrey4l_: Are you in a position to be able to validate the nova fix before it lands? | 13:03 |
| Daviey | Jeffrey4l_: i sensed they wanted us to | 13:03 |
| *** zhiwei has joined #openstack-kolla | 13:04 | |
| Jeffrey4l_ | Daviey, maybe when i am free :) | 13:04 |
| Daviey | Jeffrey4l_: ok, let me know.. as i will try and do it later in the week if not | 13:07 |
| *** msimonin has joined #openstack-kolla | 13:08 | |
| *** zhiwei has quit IRC | 13:09 | |
| *** david-lyle has quit IRC | 13:10 | |
| Jeffrey4l_ | OK | 13:10 |
| *** celo_r has joined #openstack-kolla | 13:17 | |
| *** david-lyle has joined #openstack-kolla | 13:18 | |
| *** dave-mccowan has joined #openstack-kolla | 13:24 | |
| *** jtriley has joined #openstack-kolla | 13:30 | |
| kbaegis | Did a full kolla rebuild. Horizon master is broken. | 13:32 |
| *** sbezverk has joined #openstack-kolla | 13:33 | |
| kbaegis | Returning 500 for any query, even curl from localhost | 13:33 |
| kbaegis | traceback: http://hastebin.com/ezixuzeqil.vbs | 13:34 |
| *** huikang has joined #openstack-kolla | 13:38 | |
| *** mgoddard_ has joined #openstack-kolla | 13:45 | |
| *** mgoddard has quit IRC | 13:48 | |
| *** celo_r has quit IRC | 13:52 | |
| *** ayoung has joined #openstack-kolla | 13:52 | |
| irtermite | mornin all... how goes? | 13:52 |
| kbaegis | irtermite: Morning | 13:53 |
| *** Mr_Broken has quit IRC | 14:01 | |
| *** sbezverk has quit IRC | 14:01 | |
| *** Mr_Broken has joined #openstack-kolla | 14:01 | |
| *** neilus1 has quit IRC | 14:01 | |
| *** celo_r has joined #openstack-kolla | 14:02 | |
| *** inc0 has joined #openstack-kolla | 14:02 | |
| *** neilus has joined #openstack-kolla | 14:02 | |
| *** inc0_ has joined #openstack-kolla | 14:05 | |
| inc0_ | good morning | 14:05 |
| kbaegis | inc0_: Good morning_ | 14:05 |
| *** Mr_Broken has quit IRC | 14:06 | |
| *** inc0 has quit IRC | 14:07 | |
| *** sdake has joined #openstack-kolla | 14:08 | |
| *** sbezverk has joined #openstack-kolla | 14:10 | |
| *** sdake_ has joined #openstack-kolla | 14:12 | |
| *** sdake has quit IRC | 14:13 | |
| *** Liuqing has joined #openstack-kolla | 14:18 | |
| britthouser | good morning inc0 | 14:20 |
| *** banix has joined #openstack-kolla | 14:23 | |
| inc0_ | sdake_, did you secure all the data we have on OSIC? | 14:23 |
| sdake_ | inc0_ yes just now | 14:23 |
| sdake_ | the /root directorry only | 14:24 |
| inc0_ | yeah I think that's enough | 14:24 |
| inc0_ | we need ton of time to analyze our results too | 14:24 |
| *** huikang_ has joined #openstack-kolla | 14:25 | |
| sdake_ | the sad part is centos was deployed lasst night | 14:25 |
| inc0_ | yeah, that was also bad organization from out side | 14:26 |
| inc0_ | kinda sucks that we got this at the end of cycle, but well, that's on me | 14:26 |
| inc0_ | sdake_, we need to tag this week right? | 14:27 |
| sdake_ | yup | 14:27 |
| inc0_ | https://review.openstack.org/#/c/358042/ then this is important | 14:28 |
| inc0_ | unless you want your repo customization | 14:28 |
| *** michauds has joined #openstack-kolla | 14:32 | |
| *** mgoddard_ has quit IRC | 14:32 | |
| *** huikang_ has quit IRC | 14:32 | |
| *** mgoddard has joined #openstack-kolla | 14:32 | |
| *** huikang_ has joined #openstack-kolla | 14:33 | |
| inc0_ | sdake_, I hope that we did copy globals.yml along with the tests? | 14:36 |
| inc0_ | in any case, please copy /etc/kolla too | 14:36 |
| *** huikang_ has quit IRC | 14:37 | |
| *** mnasiadka has quit IRC | 14:40 | |
| *** neilus has quit IRC | 14:40 | |
| *** neilus has joined #openstack-kolla | 14:40 | |
| *** david-lyle has quit IRC | 14:43 | |
| inc0_ | mewald, ping | 14:45 |
| *** diogogmt has joined #openstack-kolla | 14:45 | |
| mewald | inc0_: ping | 14:46 |
| *** sdake_ has quit IRC | 14:47 | |
| inc0_ | https://review.openstack.org/#/c/346449/23 I wanted to ask you about inventory | 14:47 |
| mewald | sure | 14:47 |
| inc0_ | so you created brand new group for monitoring, why? | 14:47 |
| inc0_ | why not use control:children? | 14:47 |
| inc0_ | and if it actually worth to create new group for it, why not add monitoring01 as example hostname? | 14:48 |
| *** david-lyle has joined #openstack-kolla | 14:48 | |
| openstackgerrit | Merged openstack/kolla: Customization for base https://review.openstack.org/358042 | 14:49 |
| mewald | I don't really love the idea to running ElasticSearch and Kibana on the controller nodes. Imho they should only have to deal with OpenStack controller stuff. In my first patch I added a monitoring01 host to deploy to, but then someone gave -1 for adding new server types. So I ended up reusing the control group and only deploying to the first node, as I am currently not clustering influxdb and Grafana. | 14:50 |
| *** Liuqing has quit IRC | 14:51 | |
| *** neilus has quit IRC | 14:51 | |
| inc0_ | so this someone made a wrong call in my opinion;) I agree that monitoring can produce enough data to be on a dedicated node, however if we want to encourage that, we should | 14:51 |
| mewald | good :D I will go that way then. I like it better anyway :) | 14:52 |
| inc0_ | can we cluster influx tho? isn't this paid license? | 14:52 |
| mewald | what do you think about enable_performance_monitoring? | 14:52 |
| inc0_ | I'll make comment in review so you can blame it on me | 14:52 |
| mewald | ok ^^ | 14:52 |
| inc0_ | as single command to rule them all? that's ok, we did something like that for central logging | 14:52 |
| mewald | I think you can cluster influx using that relay service but it's very complex and I personally wouldnt need that kind of availability for monitoring | 14:53 |
| mewald | about enable_performance_monitoring: full ack, I have seen that for logging and kept the pattern | 14:53 |
| inc0_ | well, at least some sort of data replication across servers would be nice | 14:53 |
| *** jtriley has quit IRC | 14:55 | |
| inc0_ | berendt, ping;) | 14:56 |
| inc0_ | (I'm doing house cleaning, sorry guys) | 14:56 |
| inc0_ | berendt, https://review.openstack.org/#/c/350777/ will you continue? If not that's ok, but let me know, this one is extremely important for me. I can finish it off if you don't have time for it | 14:58 |
| *** huikang_ has joined #openstack-kolla | 15:00 | |
| *** mnasiadka has joined #openstack-kolla | 15:02 | |
| *** david-lyle has quit IRC | 15:02 | |
| *** huikang_ has quit IRC | 15:04 | |
| *** rhallisey has quit IRC | 15:05 | |
| *** huikang_ has joined #openstack-kolla | 15:05 | |
| *** jtriley has joined #openstack-kolla | 15:05 | |
| *** diogogmt has quit IRC | 15:06 | |
| *** coolsvap is now known as coolsvap_ | 15:07 | |
| *** neilus has joined #openstack-kolla | 15:08 | |
| *** diogogmt has joined #openstack-kolla | 15:08 | |
| *** sdake has joined #openstack-kolla | 15:09 | |
| *** huikang_ has quit IRC | 15:10 | |
| *** rmart04 has quit IRC | 15:10 | |
| *** huikang_ has joined #openstack-kolla | 15:11 | |
| Jeffrey4l_ | hi sdake , i have deployed a openstack env in osic now. | 15:12 |
| Jeffrey4l_ | boot a instance and pint that vms successfully. | 15:12 |
| Jeffrey4l_ | *ping | 15:12 |
| sdake | Jeffrey4l_ tests take 6 hours to run | 15:13 |
| sdake | i think we have to give backgear now | 15:13 |
| Jeffrey4l_ | But rally seem has some issue. I can not run rally test now. no idea the root cuase. | 15:13 |
| Jeffrey4l_ | OK.. | 15:13 |
| sdake | can you copy /etc/kolla? | 15:13 |
| Jeffrey4l_ | sure. copy to where? | 15:14 |
| sdake | anywhere - my computer is fine | 15:14 |
| Jeffrey4l_ | ok. i will keep one copy in my local. and make a copy to you node.. | 15:14 |
| *** celo_r has quit IRC | 15:16 | |
| *** b_bezak has quit IRC | 15:17 | |
| Jeffrey4l_ | copied to /home/jeffrey/testresult/kolla | 15:18 |
| Jeffrey4l_ | sdake, ^ | 15:18 |
| berendt | inc0_ take over if you need to finish it today, i'm end of business now | 15:22 |
| berendt | inc0_ i can take care of it tomorrow morning | 15:22 |
| inc0_ | berendt, more like end of week | 15:22 |
| *** david-lyle has joined #openstack-kolla | 15:22 | |
| berendt | inc0_ ok, than have a look on it tomorrow | 15:22 |
| inc0_ | but yeah, I'll fix it up and you can look tomorrow morning for any incomming reviews | 15:22 |
| inc0_ | ok? | 15:22 |
| *** celo_r has joined #openstack-kolla | 15:22 | |
| inc0_ | we I'll co-author it if that's ok with you | 15:22 |
| berendt | inc0_ of course this if fine for me | 15:23 |
| inc0_ | ok, so let's do that, I think this is only customization patch left from core services | 15:23 |
| *** celo_r has quit IRC | 15:23 | |
| berendt | great | 15:23 |
| *** celo_r has joined #openstack-kolla | 15:24 | |
| *** vbel has quit IRC | 15:24 | |
| *** vbel has joined #openstack-kolla | 15:24 | |
| kfox1111 | sbezverk: alive? | 15:24 |
| sbezverk | kfox1111: yes | 15:27 |
| kfox1111 | I can take a quick stab at doing a deployment for novncproxy if you haven't started one. | 15:27 |
| sbezverk | kfox1111: when I switched to vxlan for tenants as well, now instances from both compute nodes have connectivity.. | 15:28 |
| kfox1111 | very cool. | 15:28 |
| sbezverk | kfox1111: I have not started, I was busy with vxlan | 15:28 |
| kfox1111 | so multinode breaths life. :) | 15:29 |
| *** vhosakot has joined #openstack-kolla | 15:29 | |
| sbezverk | kfox1111: oh yes.. the pulse is normal ;-) | 15:29 |
| kfox1111 | hehe. but its a start. :) | 15:30 |
| *** senk has joined #openstack-kolla | 15:30 | |
| *** kbaegis has quit IRC | 15:30 | |
| *** sdake has quit IRC | 15:31 | |
| *** banix has quit IRC | 15:31 | |
| *** banix has joined #openstack-kolla | 15:31 | |
| sbezverk | kfox1111: about your ps for changing vncproxy.. I still think we need to use compute node's ip address.. the reason being, it is used in openstack documentation.. | 15:31 |
| openstackgerrit | Merged openstack/kolla: Vagrant doc updates https://review.openstack.org/353240 | 15:31 |
| *** Jeffrey4l_ has quit IRC | 15:32 | |
| vhosakot | Jeffrey4l_: did you get a chance to deploy kolla on centos? | 15:32 |
| kfox1111 | for which field? I'm absolutely positive on base_url's. | 15:32 |
| kfox1111 | I've manually had to fix every novnc setup I've deployed with packstack. | 15:32 |
| vhosakot | I meant in the OSIC cluster | 15:32 |
| sbezverk | kfox1111: do you have your PS link handy? | 15:33 |
| kfox1111 | yeah. sec | 15:33 |
| kfox1111 | https://review.openstack.org/#/c/362449/ | 15:34 |
| sbezverk | kfox1111: here is what openstack doc says http://paste.openstack.org/show/564848/, we should probably follow it.. | 15:36 |
| kfox1111 | looking. | 15:36 |
| kfox1111 | yeah, I have vncserver_proxyclient_address overriden still in the PS. | 15:38 |
| kfox1111 | and novncproxy_base_url is set to the controller in that example. which is what kolla should be generating. | 15:38 |
| kfox1111 | same with vncserver_listen. | 15:38 |
| kfox1111 | so I think that snippet is how we are configuring it. | 15:39 |
| sbezverk | vnc_listen is bound to ip address, we should change it to 0.0.0.0 as per doc | 15:39 |
| sbezverk | here is from your ps | 15:39 |
| sbezverk | crudini --set /nova/nova.conf vnc vncserver_listen $IP; | 15:39 |
| kfox1111 | well, it shouldn't matter. for the container there are only 2 addresses. 127.0.0.1 and the eth0 one. | 15:40 |
| kfox1111 | nothings going to touch it on 127.0.0.1 so they are basically equiv. | 15:40 |
| *** salv-orlando has joined #openstack-kolla | 15:40 | |
| *** david-lyle has quit IRC | 15:40 | |
| kfox1111 | is that correct? | 15:41 |
| *** akscram has quit IRC | 15:42 | |
| sbezverk | kfox1111: in this specific case you are right, but I think with 0.0.0.0 it will be more flexible for other possible confiugration | 15:42 |
| sbezverk | which we do not consider | 15:42 |
| kfox1111 | true. | 15:42 |
| *** unicell has quit IRC | 15:42 | |
| sbezverk | I mean it is more to be inline with opensatckj doc that is all | 15:42 |
| kfox1111 | If its not specific to kolla-kubernetes, then I think it should be set in kolla's genconfig so we should just remove the line entirely. | 15:42 |
| kfox1111 | fyi, the docs arn't aways the most security hardened of things | 15:43 |
| kfox1111 | they focus on getting somethign working. | 15:43 |
| kfox1111 | not nessisarily how you'd want to run it in production. | 15:43 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Customizations for Neutron https://review.openstack.org/350777 | 15:43 |
| kfox1111 | they have gotten better over the years though. | 15:43 |
| sbezverk | kfox1111: agree, but config should not force people, there should be a choice if they want to make it more secure.. | 15:44 |
| *** b_bezak has joined #openstack-kolla | 15:44 | |
| sbezverk | kfox1111: agree that it make more sense to address from genconfig | 15:45 |
| kfox1111 | agreed. | 15:45 |
| Daviey | sbezverk: uhm, what is this about? | 15:45 |
| sbezverk | Daviey: kubernetes things :-) | 15:46 |
| kfox1111 | Daviey: we've been having to do some config overriding in kolla-kubernetes init as only kubernetes knows a few things like what ip the container has at runtime. | 15:46 |
| kfox1111 | we're still coming up with the right balance of having kolla do almost all of the config generation and having kolla-kubernetes+k8s do a few overrides at runtime to get stuff working. | 15:46 |
| *** b_bezak has quit IRC | 15:46 | |
| Daviey | kfox1111: ah, i see | 15:47 |
| kfox1111 | sbezverk: ok. so I'll remove vncserver_listen from my ps? | 15:47 |
| Daviey | I thought it might have been in reference to vnc listen address being 0.0.0.0 | 15:47 |
| kfox1111 | Daviey: yeah. it is in this particular case. we're deciding whether to set it to 0.0.0.0 in kolla and not override it it k8s, | 15:48 |
| kfox1111 | or set it to the specific ip of the container in k8s. | 15:48 |
| sbezverk | Daviey: I am for following openstack docs on this regard | 15:48 |
| sbezverk | and give people a choice if they want to change it | 15:49 |
| kfox1111 | for this case, I don't think it matters much. | 15:49 |
| kfox1111 | the only extra interface 0.0.0.0 gives is 127.0.0.1. | 15:49 |
| *** akscram has joined #openstack-kolla | 15:49 | |
| kfox1111 | if its less code to maintain in kolla-kubernetes though, I'm for it. :) | 15:49 |
| *** zhurong has quit IRC | 15:49 | |
| *** unicell has joined #openstack-kolla | 15:50 | |
| kfox1111 | sbezverk: oh, one thing | 15:50 |
| kfox1111 | novncproxy_base_url = http://controller:6080/vnc_auto.html is wrong. | 15:50 |
| *** david-lyle has joined #openstack-kolla | 15:50 | |
| kfox1111 | in two ways... | 15:50 |
| *** banix has quit IRC | 15:50 | |
| Daviey | sbezverk: Hardened by default if possible is never a bad idea :) | 15:50 |
| kfox1111 | 1. the address is handed directly to a browser via horizon. | 15:50 |
| kfox1111 | so it at minimum has to be http://<external_vip>:6080/vnc_auto.html | 15:50 |
| kfox1111 | but probably should be a dns entry. | 15:51 |
| kfox1111 | 2. its a console where passwords tend to go across. it is really really important to use https for it. | 15:51 |
| kfox1111 | either way, I think kolla genconfig shoudl create the value. | 15:51 |
| *** mewald has left #openstack-kolla | 15:54 | |
| *** ntpttr__ has quit IRC | 15:54 | |
| sbezverk | kfox1111: since you will be taken care of vnc, I will start working on cinder | 15:55 |
| kfox1111 | k. | 15:55 |
| kfox1111 | when you do cinder-api, can you use glance-registry as a starting point? | 15:56 |
| kfox1111 | that will be one less thing I'd have to convert later then. | 15:56 |
| *** sean-k-mooney has joined #openstack-kolla | 15:56 | |
| kfox1111 | it should mostly be a copy and a sed 's/glance-registry/cinder-api/' | 15:56 |
| *** jrich523 has joined #openstack-kolla | 15:57 | |
| kfox1111 | we really need to get conventions agreed to. its really hard to debug all this stuff when the names are so inconsistant all over the place. :/ | 15:58 |
| inc0_ | kfox1111, quick question...when you would want to expose libvirt over tcp? | 15:58 |
| inc0_ | live migration? | 15:58 |
| kfox1111 | inc0_: personally? never. | 15:59 |
| kfox1111 | I've only seen it configured to ssh tunnel the traffic from local to local. | 15:59 |
| kfox1111 | thats why I added the switch to disable libvirt tcp. I don't need/want it in my env. | 16:00 |
| inc0_ | well, do you know if tunneled LM requires that? | 16:00 |
| kfox1111 | I don't believe it does. | 16:00 |
| kfox1111 | exposesd libvirt tcp is a security issue imho. | 16:00 |
| *** celo_r has quit IRC | 16:00 | |
| inc0_ | yeah I don't think there is any auth out there right | 16:01 |
| inc0_ | ? | 16:01 |
| kfox1111 | as far as I know, no auth. | 16:01 |
| inc0_ | yay | 16:01 |
| inc0_ | then I'd go for "don't expose it at all" | 16:01 |
| *** Serlex has quit IRC | 16:01 | |
| inc0_ | on that note...let's see if Kolla exposes it over tcp | 16:02 |
| kfox1111 | thats where I was at, but I gave up fighting it and opted for a config option to at least disable it. | 16:02 |
| *** huikang_ has quit IRC | 16:03 | |
| *** huikang_ has joined #openstack-kolla | 16:03 | |
| inc0_ | well there is some auth | 16:04 |
| inc0_ | configurable | 16:04 |
| kfox1111 | is it more then just plain text over the wire? | 16:04 |
| inc0_ | https://libvirt.org/auth.html hard to tell | 16:05 |
| sbezverk | kfox1111: ok I will use glance-apoi as base | 16:06 |
| inc0_ | I doubt it uses any encryption over network | 16:06 |
| inc0_ | uhh kolla suffers from it too | 16:06 |
| kfox1111 | hmm.... its using sasl for basic username/password | 16:06 |
| sbezverk | inc0_ Jeffrey4l was advocating tcp socket as more efficient way | 16:06 |
| inc0_ | sbezverk, but much less secure | 16:07 |
| sean-k-mooney | inc0_: sdake: i figured out why the bifrost change is broken. currently you cannt install mysql-server on any image built off of kollas base image on ubuntu 16.04 | 16:07 |
| kfox1111 | libvirt communication doesn't have to be very performant I think. | 16:07 |
| inc0_ | sean-k-mooney, duh...why is that? | 16:07 |
| kfox1111 | secure on the other hand. | 16:07 |
| sbezverk | inc0_: hence config option, people after secuirty, can use it | 16:07 |
| sbezverk | to disable it, I do not care about secuirty, so I am after efficiency | 16:07 |
| kfox1111 | sbezverk: -1. ok with a config option, but security shoudl be the default. performance for those at extreme scale that may need it. | 16:08 |
| *** huikang_ has quit IRC | 16:08 | |
| inc0_ | btw sean-k-mooney https://review.openstack.org/#/c/359364/ mind taking a look? I answered your comment | 16:08 |
| kfox1111 | "premature optomization is the root of all evil". | 16:08 |
| sean-k-mooney | inc0_: http://paste.openstack.org/show/564851/ | 16:08 |
| sean-k-mooney | and sure ill take a look now | 16:08 |
| sbezverk | kfox1111: I am fine as long as config option exists | 16:08 |
| kfox1111 | on a small cloud, the performance overhad of it is unnoticable. the extra security is noticable. | 16:08 |
| kfox1111 | k. | 16:08 |
| inc0_ | agree with Kevin here, everybody should be after security | 16:09 |
| inc0_ | and we don't require our deployers to know every config option to increase it | 16:09 |
| inc0_ | we should default to as secure as we can (within reasonable limits) | 16:09 |
| kfox1111 | I'm ok with an option so long as its defaulted to security. if you want to turn it off, thats ok with me. | 16:09 |
| sbezverk | kfox1111: agreed | 16:09 |
| inc0_ | sean-k-mooney, can we use mariadb btw? I know you had issues with it, mind reminding me which were these? | 16:10 |
| inc0_ | btw I see issue with your configs guys:) | 16:11 |
| sean-k-mooney | inc0_: i replace mysql with mariadb later but mysql-server is installed by bifrost and i dont think i can change it | 16:11 |
| kfox1111 | inc0_: which issues? | 16:11 |
| inc0_ | you use "yes" "no" config type without ansible's | bool filter | 16:11 |
| kfox1111 | inc0_: its j2 not ansible I think. | 16:11 |
| inc0_ | people might write true there and that would work with ansible | 16:11 |
| inc0_ | bool filter is in ansible unfortunately | 16:12 |
| kfox1111 | yeah, we probably shoudl fix that. | 16:12 |
| kfox1111 | I added another ansible like filter to the system already. | 16:12 |
| kfox1111 | it woudl be easy to add a | bool | 16:12 |
| kfox1111 | maybe we make a blueprint for it? | 16:12 |
| inc0_ | yeah, that's super simple one | 16:12 |
| sean-k-mooney | inc0_: ill apply https://review.openstack.org/#/c/359364/ locally and test it with my 3 vm setup and give you my +1 if it works. i just want to be sure that it work when you have different barmetal nics on each host | 16:12 |
| inc0_ | bp will take longer than this filter;) | 16:12 |
| kfox1111 | inc0_: filter yes, rewriting everything to use it, no. | 16:13 |
| inc0_ | sean-k-mooney, sure thing, just make sure to remove it from globals | 16:13 |
| inc0_ | and add it in inventoryu | 16:13 |
| inc0_ | kfox1111, fair enough | 16:13 |
| kfox1111 | https://blueprints.launchpad.net/kolla-kubernetes/+spec/bool-filter | 16:14 |
| sean-k-mooney | inc0_: yep that what i have currently as i have a ubuntu 14.04 vm, 16.04 vm and centos7 so the values are different and set in my inventory. | 16:14 |
| inc0_ | ok | 16:15 |
| inc0_ | btw sean-k-mooney this play took little over 5min on 130node osic:) | 16:15 |
| *** huikang_ has joined #openstack-kolla | 16:15 | |
| inc0_ | ok, I take that back, I need to optimize one thing that was horribly long:P | 16:16 |
| inc0_ | after optimization it was 5min | 16:16 |
| sean-k-mooney | oh cool. and after it was done was it fuctional? | 16:16 |
| *** huikang_ has quit IRC | 16:16 | |
| inc0_ | yeah | 16:16 |
| sean-k-mooney | :) oh what was the slow part out of interest | 16:16 |
| inc0_ | issue is hostfile | 16:16 |
| sean-k-mooney | ah | 16:16 |
| inc0_ | only controllers needs to be talking to each other | 16:17 |
| inc0_ | you did that for everything and it was N^2 connections | 16:17 |
| inc0_ | with 130 nodes it became...significant;) | 16:17 |
| sean-k-mooney | yes that n*n in current implemntation. i guess its simple to restict that to nodes in the controler group only | 16:17 |
| inc0_ | yeah | 16:18 |
| inc0_ | that's what I'm going to do | 16:18 |
| sean-k-mooney | inc0_: does it need to be done for any of the other services. its mainly for rabbit but does ceph or any of the storage backend require hostnames to be resolveable? | 16:19 |
| *** huikang_ has joined #openstack-kolla | 16:19 | |
| *** banix has joined #openstack-kolla | 16:19 | |
| *** matrohon has quit IRC | 16:19 | |
| inc0_ | I don't think ceph even needs that now | 16:20 |
| inc0_ | afair it's only rabbit | 16:20 |
| sean-k-mooney | cool well its easy to rexpand if needed | 16:20 |
| inc0_ | also I think we can optimize it yet again | 16:21 |
| inc0_ | create file once and copy over | 16:21 |
| *** david-lyle has quit IRC | 16:21 | |
| inc0_ | that will lower the complexity to N | 16:21 |
| kfox1111 | darn... gota rebase this... | 16:21 |
| sean-k-mooney | yes though instead of a copy it would be better to append to the end of the existing file. | 16:22 |
| inc0_ | yeah, that's true | 16:22 |
| kfox1111 | inc0_: what do you think of https://review.openstack.org/#/c/362443 | 16:23 |
| *** david-lyle has joined #openstack-kolla | 16:23 | |
| *** huikang_ has quit IRC | 16:23 | |
| sean-k-mooney | inc0_: do you want me to do that? i can open a tech debt bug and submit a patch. | 16:23 |
| inc0_ | kfox1111, let me put it this way - make configs right >> crudini >> sed | 16:23 |
| kfox1111 | I need to rebase the libvirt tcp one, and it would be much easier if review 362443 was in. | 16:23 |
| kfox1111 | inc0_: agreed. | 16:23 |
| inc0_ | sean-k-mooney, I'll write it down later today | 16:24 |
| inc0_ | at least restrict to control one | 16:24 |
| inc0_ | this will make it managable | 16:24 |
| kfox1111 | inc0_: if it looks good enough to you, can you +2/workflow it? then I can rebase the libvirt tcp one easier. | 16:24 |
| inc0_ | there ya go | 16:24 |
| sean-k-mooney | inc0_: ok cool, for bifrost i need to overide https://github.com/openstack/bifrost/blob/1ddc801c65c8c8c8ba28caf9cc7f1e8ec3a6ad71/playbooks/roles/bifrost-ironic-install/defaults/required_defaults_Debian_family.yml#L13-L44 | 16:24 |
| kfox1111 | thx. | 16:25 |
| inc0_ | sean-k-mooney, btw...can bifrost work on sqlite? | 16:25 |
| inc0_ | I see little value in having mysql there tbh | 16:25 |
| *** msimonin has quit IRC | 16:25 | |
| sean-k-mooney | inc0_: if ironic can work with sqllite the bifrost should be able to | 16:26 |
| sean-k-mooney | bifrost jsut installs ironic in standalone mode. | 16:26 |
| inc0_ | worth to try later | 16:26 |
| inc0_ | seems like it can | 16:26 |
| inc0_ | would be much easier to deploy tbh | 16:26 |
| inc0_ | btw I'd be ok with seding this mysql installation off. I think requiring mysql is bad in the first place | 16:27 |
| sean-k-mooney | inc0_: ya it would i just implmented as vanila a bifrost install as i code but im sure there is lots we can do to optimise | 16:27 |
| inc0_ | ofc, let's get it working, we still need our jobs following cycle;) | 16:27 |
| inc0_ | so let's not do everything today;) | 16:28 |
| sean-k-mooney | inc0_: so for the mysql issue what do you think is the best approch. fix kolla base so installing works, overdied the package list in kolla so it uses mariabdb or submit a change to bifrost to change to mariadb? | 16:29 |
| inc0_ | in fact, best solution would be to submit change to bifrost to make this configurable, and for kolla use sqlite | 16:30 |
| inc0_ | setting up mariadb is complicated compared to just use local sqlite | 16:30 |
| *** unicell has quit IRC | 16:31 | |
| sean-k-mooney | yep thats an option too though i think we are pasted the ironic code freeze so not sure it can be done this cycle but i can ask in #openstack-ironic | 16:31 |
| *** huikang_ has joined #openstack-kolla | 16:31 | |
| inc0_ | we can do this next release imho | 16:32 |
| *** daneyon has joined #openstack-kolla | 16:34 | |
| *** cloudnull is now known as wq | 16:35 | |
| *** david-lyle_ has joined #openstack-kolla | 16:36 | |
| *** daneyon has quit IRC | 16:39 | |
| *** david-lyle has quit IRC | 16:39 | |
| *** wq is now known as cloudnull | 16:39 | |
| *** cloudnull is now known as wq | 16:39 | |
| *** wq is now known as cloudnull | 16:39 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Switch to crudini for easier readability https://review.openstack.org/362443 | 16:43 |
| *** huikang_ has quit IRC | 16:46 | |
| *** huikang_ has joined #openstack-kolla | 16:47 | |
| *** aNupoisc has joined #openstack-kolla | 16:47 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Limit hostfile generation to control nodes https://review.openstack.org/363028 | 16:47 |
| *** fragatina has joined #openstack-kolla | 16:50 | |
| *** fragatin_ has joined #openstack-kolla | 16:51 | |
| *** huikang_ has quit IRC | 16:51 | |
| *** diogogmt has quit IRC | 16:52 | |
| *** gfhellma has joined #openstack-kolla | 16:53 | |
| *** kbaegis has joined #openstack-kolla | 16:53 | |
| *** gfidente has quit IRC | 16:54 | |
| *** fragatina has quit IRC | 16:55 | |
| *** celo_r has joined #openstack-kolla | 16:55 | |
| *** aNupoisc has quit IRC | 16:55 | |
| *** rhallisey has joined #openstack-kolla | 16:57 | |
| *** unicell has joined #openstack-kolla | 16:57 | |
| *** gfhellma1 has joined #openstack-kolla | 17:01 | |
| *** celo_r has quit IRC | 17:01 | |
| *** hieulq_ has joined #openstack-kolla | 17:02 | |
| *** gfhellma has quit IRC | 17:03 | |
| *** celo_r has joined #openstack-kolla | 17:04 | |
| *** unicell has quit IRC | 17:06 | |
| *** mgoddard_ has joined #openstack-kolla | 17:06 | |
| *** aNupoisc has joined #openstack-kolla | 17:06 | |
| *** wally__ has joined #openstack-kolla | 17:07 | |
| *** diogogmt has joined #openstack-kolla | 17:07 | |
| *** wally__ is now known as Guest32748 | 17:07 | |
| *** Guest32748 has quit IRC | 17:07 | |
| *** wally2 has joined #openstack-kolla | 17:08 | |
| *** wally2 has quit IRC | 17:08 | |
| *** wally2 has joined #openstack-kolla | 17:08 | |
| *** mgoddard has quit IRC | 17:09 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 | 17:10 |
| *** huikang_ has joined #openstack-kolla | 17:14 | |
| *** hieulq_ has quit IRC | 17:19 | |
| *** huikang_ has quit IRC | 17:23 | |
| *** huikang_ has joined #openstack-kolla | 17:24 | |
| *** mnasiadka has quit IRC | 17:24 | |
| *** sdake has joined #openstack-kolla | 17:27 | |
| *** huikang_ has quit IRC | 17:28 | |
| *** celo_r has quit IRC | 17:30 | |
| inc0_ | sdake, can you plz take a look? https://review.openstack.org/#/c/360796/ | 17:31 |
| inc0_ | people keep asking same questions;) | 17:31 |
| *** huikang_ has joined #openstack-kolla | 17:31 | |
| wally2 | hi sdake, I re-config my network, and re-do again now | 17:33 |
| *** celo_r has joined #openstack-kolla | 17:33 | |
| wally2 | just finish, going to run the init-runonce, do u have a copy that I can have a look? | 17:34 |
| sdake | hey cats- taking today off mostly - been grinding away on osic for 6 days straight- palan on sleepingtoday | 17:34 |
| inc0_ | sdake, https://www.youtube.com/watch?v=HKtsdZs9LJo | 17:35 |
| vhosakot | take rest sdake | 17:39 |
| *** unicell has joined #openstack-kolla | 17:42 | |
| wally2 | hi sdaked, i got my vm workking, it can go out now,:) | 17:48 |
| *** michauds has quit IRC | 17:52 | |
| *** b_bezak has joined #openstack-kolla | 18:00 | |
| *** huikang_ has quit IRC | 18:01 | |
| *** senk has quit IRC | 18:02 | |
| *** gfhellma1 has quit IRC | 18:05 | |
| *** banix has quit IRC | 18:05 | |
| *** huikang_ has joined #openstack-kolla | 18:05 | |
| *** gfhellma has joined #openstack-kolla | 18:06 | |
| *** banix has joined #openstack-kolla | 18:06 | |
| *** celo_r has quit IRC | 18:07 | |
| *** salv-orl_ has joined #openstack-kolla | 18:08 | |
| *** michauds has joined #openstack-kolla | 18:09 | |
| *** salv-orlando has quit IRC | 18:11 | |
| inc0_ | wally2, what was the issue? | 18:16 |
| *** celo_r has joined #openstack-kolla | 18:18 | |
| wally2 | just network configure problem | 18:21 |
| inc0_ | ok I'm glad you got this fixed | 18:21 |
| kfox1111 | sbezverk: I just tried firing up the nova-scheduler. | 18:22 |
| kfox1111 | it fails because command: ["sh", "-c"] isn't a valid shell command. | 18:22 |
| kfox1111 | it looks like this was debug code left in by mistake? | 18:22 |
| kfox1111 | should I just pull it? | 18:22 |
| *** aNupoisc has quit IRC | 18:27 | |
| *** b_bezak has quit IRC | 18:29 | |
| *** b_bezak has joined #openstack-kolla | 18:30 | |
| sbezverk | kfox1111: there should not be any sh other than in init container | 18:31 |
| *** mgoddard_ has quit IRC | 18:31 | |
| sbezverk | kfox1111: yep, damn left overs :-( | 18:32 |
| sbezverk | will fix it | 18:32 |
| *** wally2 has quit IRC | 18:33 | |
| *** b_bezak has quit IRC | 18:35 | |
| kfox1111 | k. thx. | 18:36 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: PS cleans up nova scheduler container from left over command https://review.openstack.org/363113 | 18:36 |
| kfox1111 | do we need 2 +2's in this case? or is it trivial enough/critical enough to just workflow now? | 18:37 |
| inc0_ | always | 18:37 |
| inc0_ | but just acked it | 18:37 |
| kfox1111 | k. | 18:37 |
| *** fragatin_ has quit IRC | 18:38 | |
| inc0_ | ok, I need to make at least one commit in kolla-k8s, I have +2 powers - it is my moral obligation;) | 18:38 |
| kfox1111 | :) | 18:39 |
| kfox1111 | bool? :) | 18:39 |
| inc0_ | duh | 18:39 |
| inc0_ | or fencing script for ceph | 18:40 |
| kfox1111 | ah. ceph support for k8s would be sweet. :) | 18:40 |
| inc0_ | well, it is our requirement right? | 18:40 |
| inc0_ | to have ceph pre-installed? | 18:41 |
| kfox1111 | for now. we'd like not to have to require that long term. | 18:41 |
| kfox1111 | have a way to use kolla-kubernetes to deploy ceph if you don't already have one. | 18:41 |
| inc0_ | but we want to keep ceph as requirement right? for HA | 18:41 |
| kfox1111 | yeah. | 18:42 |
| inc0_ | btw I'm not super convinced if ceph should be deployed by k8s | 18:42 |
| inc0_ | not much value | 18:42 |
| kfox1111 | I've got it running in k8s on my laptop. :) | 18:42 |
| kfox1111 | using the kolla containers. :) | 18:42 |
| inc0_ | ok, but why, that's my question:P | 18:42 |
| inc0_ | it's not like osds will float around | 18:43 |
| kfox1111 | why containers, or why k8s? | 18:43 |
| inc0_ | why k8s | 18:43 |
| kfox1111 | because its one tool for an op to learn. | 18:43 |
| kfox1111 | its easier to debug if you already have to learn how to debug things in k8s. | 18:43 |
| inc0_ | fair enough | 18:44 |
| inc0_ | but I don't think we' | 18:44 |
| inc0_ | we coudl do this without petsets | 18:44 |
| inc0_ | ceph.conf needs list of running mons | 18:45 |
| kfox1111 | petsets or net=host. | 18:45 |
| kfox1111 | well... daemonsets with net=host? | 18:45 |
| inc0_ | however, how will you know the IPs prior to scheduling? | 18:45 |
| kfox1111 | I haven't tried much, but can you use dns? | 18:46 |
| kfox1111 | a petset would work then. | 18:46 |
| inc0_ | I think ceph likes dns | 18:46 |
| inc0_ | yeah that's what I mean | 18:46 |
| kfox1111 | if not, a daemonset + net=host would work as its the host ip which is static. | 18:46 |
| kfox1111 | well, a petset won't work as it will float around though. | 18:47 |
| inc0_ | yeah, but not really k8s'y | 18:47 |
| inc0_ | with petsets at least mons can float around | 18:47 |
| kfox1111 | it really really needs to be pinned to the particular host. | 18:47 |
| kfox1111 | ah. true. | 18:47 |
| sbezverk | kfox1111: for sure | 18:47 |
| kfox1111 | well... | 18:47 |
| inc0_ | although...not sure, it does have persistent data (monmap) | 18:47 |
| kfox1111 | a petset mon would work only if you could persist the data | 18:47 |
| kfox1111 | (hehe) | 18:47 |
| inc0_ | soooo...we can't put it on ceph;) | 18:47 |
| sbezverk | unless you have shared storage and it can be access from multiple compute nodes | 18:47 |
| inc0_ | so we need ceph to deploy ceph;) | 18:48 |
| kfox1111 | deamonset + net=host then. | 18:48 |
| kfox1111 | it should work in all cases. | 18:48 |
| inc0_ | tbh kfox1111 I'm not big fan of using k8s just to use k8s | 18:48 |
| sbezverk | kfox1111: why do you need daemonset? | 18:48 |
| inc0_ | if something doesn't make sense to be ran on k8s, I'd not run it on k8s | 18:48 |
| kfox1111 | sbezverk: so you can alwys land a container on a host where the data resides. | 18:49 |
| sbezverk | what about using labels and nodeselect? | 18:49 |
| *** salv-orl_ has quit IRC | 18:49 | |
| kfox1111 | I guess you could do it with a host selector label on a deployment too and just do one deployment per mon/osd | 18:49 |
| sbezverk | you mark compute with label, say compute | 18:49 |
| kfox1111 | :) | 18:49 |
| sbezverk | sorry I meant storage | 18:49 |
| sbezverk | and that is it | 18:49 |
| kfox1111 | daemonset + hostlabel + net=host, yeah. | 18:50 |
| sbezverk | not that I against daemonset, I love them :-) | 18:50 |
| inc0_ | sooo....you'll reimplement ansible in k8s | 18:50 |
| inc0_ | not a fan. | 18:50 |
| sbezverk | inc0_ do not you think that ansible actually is kind of kube killer? | 18:51 |
| *** huikang_ has quit IRC | 18:51 | |
| sbezverk | because ansible cannot keep up with dynamic nature of kube | 18:51 |
| *** PyroMani has quit IRC | 18:51 | |
| kfox1111 | the issue with it is, requiring ansible when you only need it for the last 2%. maybe that's ok, but it is a potentially largish cost for an op to learn. | 18:51 |
| inc0_ | sbezverk, there are services that can't keep up with dynamic nature of kube:) | 18:51 |
| *** PyroMani has joined #openstack-kolla | 18:51 | |
| sbezverk | example you want to scale up, do you need ansible for this in kube? absolutely not | 18:51 |
| inc0_ | ceph being an example | 18:51 |
| *** vbel has quit IRC | 18:52 | |
| *** jrist has quit IRC | 18:52 | |
| *** NobodyCam has quit IRC | 18:52 | |
| inc0_ | kfox1111, I was rather thinking...you also need to deploy k8s | 18:52 |
| inc0_ | so if you use ansible to deploy k8s itself + ceph | 18:52 |
| inc0_ | that makes more sense to me | 18:52 |
| kfox1111 | inc0_: other way around I think. kube 1.0 wasn't designed to handle it. they are working on growing backwards to support more pet kind of things. hence daemonsets/petsets. | 18:52 |
| kfox1111 | inc0_: ansible is overkill I think to deploy kubernetes. its really easy to do. | 18:52 |
| kfox1111 | heat template or kickstart file and good to go. | 18:53 |
| inc0_ | rhallisey, ^ ;) | 18:53 |
| kfox1111 | kolla-kubernetes can deploy the rest on top. | 18:53 |
| *** aNupoisc has joined #openstack-kolla | 18:53 | |
| *** NobodyCam has joined #openstack-kolla | 18:54 | |
| inc0_ | sooo...I think you need to tell kubelets where kube mons (or whatever it's called) are | 18:54 |
| inc0_ | which means you need orchiestration | 18:54 |
| *** huikang_ has joined #openstack-kolla | 18:54 | |
| kfox1111 | no. | 18:54 |
| inc0_ | it detects it? | 18:55 |
| *** banix has quit IRC | 18:55 | |
| kfox1111 | ceph support in k8s is done at the pod or pvc level. | 18:55 |
| openstackgerrit | Merged openstack/kolla-kubernetes: PS cleans up nova scheduler container from left over command https://review.openstack.org/363113 | 18:55 |
| kfox1111 | you don't have to preconfig the kubelets. | 18:55 |
| kfox1111 | in fact, because of that, you can have one kube system target multiple cephs, potentially one per tenant. :) | 18:55 |
| rhallisey | I think ceph externally is perfect. If you want to deploy with ceph and have it manged by kube go ahead | 18:56 |
| rhallisey | I don't see a big gain in having k8s manage ceph | 18:57 |
| rhallisey | other than convince | 18:57 |
| kfox1111 | thats ok. :) | 18:57 |
| rhallisey | but that's not a reason not to have it | 18:57 |
| kfox1111 | I may take a stab at it after a bunch of other stuff is working. :) | 18:57 |
| kfox1111 | but my clouds all have ceph external, so I can wait. | 18:57 |
| *** intr1nsic has quit IRC | 18:58 | |
| sbezverk | rhallisey: what about secret generator, inc0 is pushing for integrating it with password gen tool of kolla. Does it make sense to merge these two? I think they should be separte. | 18:59 |
| *** salv-orlando has joined #openstack-kolla | 18:59 | |
| rhallisey | depends, is kolla moving that out of kolla-ansible? | 18:59 |
| sbezverk | cuase the same password gen will be used for ansible as well | 18:59 |
| *** banix has joined #openstack-kolla | 18:59 | |
| kfox1111 | I could go either way. but right now, it kind of feels like a premature optomization? | 19:00 |
| inc0_ | rhallisey, I don't believe it is | 19:00 |
| rhallisey | what there is now is fine | 19:00 |
| sbezverk | especially I do not see anybody taking to actually do it ;-) | 19:00 |
| rhallisey | inc0_, Ideally kolla-kube would consume kolla-host, kolla-config, and kolla | 19:00 |
| rhallisey | and kolla-ansible would do the same | 19:00 |
| rhallisey | unless kolla-kube and kolla-ansible are intertwined | 19:01 |
| *** intr1nsic has joined #openstack-kolla | 19:01 | |
| *** huikang has quit IRC | 19:01 | |
| rhallisey | but kolla-ansible doesn't need anything from kolla-kube | 19:01 |
| kfox1111 | I think kolla-kube will have issues gaining traction if it depends on kolla-ansible. | 19:01 |
| rhallisey | ya I agree | 19:01 |
| sbezverk | rhallisey: exactly my point too | 19:01 |
| inc0_ | well genconfig is a scrtipt that sits in kolla | 19:03 |
| inc0_ | not ansible | 19:03 |
| inc0_ | so I see no reason to not consume it | 19:03 |
| inc0_ | we already established that passwords.yml and globals.yml will look more or less similar | 19:03 |
| kfox1111 | to me, there are 4 parts to the equation. | 19:03 |
| rhallisey | ya but we talked about splitting out config | 19:03 |
| kfox1111 | workflow, config, deployment tool, containers. | 19:03 |
| inc0_ | yeah I hope config split will happen too | 19:04 |
| inc0_ | probably next release | 19:04 |
| openstackgerrit | Sandhya Dasu proposed openstack/kolla: Removing development packages from Baremetal install defaults https://review.openstack.org/363138 | 19:04 |
| rhallisey | kolla, kolla-config, kolla-host, kolla-ansible, kolla-kube | 19:04 |
| kfox1111 | kolla-ansible tends to do three things, workflow engine, config and deployment. | 19:04 |
| inc0_ | well I wouldn't make 3 projects | 19:04 |
| kfox1111 | kolla-kubernetes should minimially do just deployment and containers. | 19:04 |
| inc0_ | rather kolla-config and kolla-host will be simply kolla | 19:04 |
| kfox1111 | I plan on only using it in that mode for a while. | 19:04 |
| *** vbel has joined #openstack-kolla | 19:05 | |
| kfox1111 | I've got system configs already and will use genconfig once and overwrite the configs with my own for now. and do manuall workflow. | 19:05 |
| rhallisey | inc0_, kolla-host would have docs/scripts to setup kolla-kube and kolla-kube would reuse the bm | 19:05 |
| rhallisey | kolla-host could be merged with kolla-config | 19:05 |
| *** david-lyle_ is now known as david-lyle | 19:05 | |
| kfox1111 | next step will be getting configgen to be able to produce configs like mine. then I might replace my hand crafted ones with kolla configgen ones. | 19:05 |
| kfox1111 | and last would be workflow, once that matures enough I'm confortable with. | 19:05 |
| kfox1111 | but just like the containers are seperate from ansible, I kind of think kolla-kubernetes should too. | 19:06 |
| kfox1111 | workflow is a different step that can be done by many differnet tools. | 19:06 |
| inc0_ | btw since kolla-host is well...ansible;) | 19:06 |
| kfox1111 | my preference I think would be ansible too. but, | 19:06 |
| inc0_ | we can keep it there | 19:06 |
| kfox1111 | it could be mistral, or any number of other things. | 19:06 |
| sbezverk | kfox1111: generating good config for kube would be huge work, right now there is no a signle service which would have its configs consumable by kube without manual editing | 19:06 |
| rhallisey | inc0_, but it's a different step in the deploymen tprocess | 19:06 |
| inc0_ | different playbook | 19:06 |
| inc0_ | like it is today | 19:07 |
| rhallisey | 1) vm deploy 2) configure 3) deploy | 19:07 |
| kfox1111 | sbezverk: yup. I've got it mostly working though. :) | 19:07 |
| rhallisey | s/vm/bm/ | 19:07 |
| *** jrist has joined #openstack-kolla | 19:07 | |
| *** huikang_ has quit IRC | 19:08 | |
| *** huikang has joined #openstack-kolla | 19:08 | |
| sbezverk | inc0_ are you ok now for secret generator? | 19:08 |
| inc0_ | sbezverk, I just don't want to have x scripts doing the same | 19:09 |
| inc0_ | thing | 19:09 |
| inc0_ | it's not scallable | 19:09 |
| kfox1111 | inc0_: agreed. its not actually generating secrets though, or shouldnt. | 19:09 |
| kfox1111 | it should just take the ones created by kolla and upload them. | 19:09 |
| kfox1111 | its just a re-formatter. | 19:09 |
| inc0_ | from yaml to whatever-k8s-likes? | 19:10 |
| kfox1111 | as the secrets it uploads into k8s should match the ones in the configs kolla generates. | 19:10 |
| inc0_ | that's ok | 19:10 |
| inc0_ | I just dont want x generators and possibly keep same config files | 19:10 |
| kfox1111 | +1 | 19:10 |
| *** coolsvap_ has quit IRC | 19:12 | |
| *** huikang has quit IRC | 19:15 | |
| *** sbezverk has quit IRC | 19:16 | |
| kfox1111 | ok... got nova api services working on my test box. :) | 19:16 |
| *** daneyon has joined #openstack-kolla | 19:17 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Make libvirt over tcp an option https://review.openstack.org/362457 | 19:18 |
| sdake | kfox1111 - nova + ovs is a huge accomplishment | 19:19 |
| sdake | much further then we got in 9 months of doing kolla work initially | 19:20 |
| kfox1111 | yes, but getting the containers there was a huge amount of work too. don't sell that short. :) | 19:20 |
| sdake | ya or the structure of the containers so they are esaily consumable | 19:20 |
| kfox1111 | yeah. | 19:20 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Nova Services https://review.openstack.org/363152 | 19:21 |
| *** daneyon has quit IRC | 19:22 | |
| kfox1111 | ok. with that one, I can replace my on host nova-api with kube-k8s container. | 19:22 |
| inc0_ | kfox1111, fix your git;) | 19:23 |
| kfox1111 | inc0_: oops | 19:23 |
| kfox1111 | yeah. :) | 19:24 |
| kfox1111 | I'm usually so careful of that too. :/ | 19:24 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Make libvirt over tcp an option https://review.openstack.org/362457 | 19:25 |
| kfox1111 | ok. take 2. | 19:25 |
| *** dwalsh has joined #openstack-kolla | 19:27 | |
| sdake | there is 1 project - deliverables inside a project | 19:30 |
| sean-k-mooney | sdake: o/ this is whats breaking the bifrost container build https://github.com/openstack/kolla/blob/master/docker/base/sources.list.ubuntu#L14 | 19:36 |
| sdake | sean-k-mooney hrm | 19:36 |
| sdake | sean-k-mooney there is an internal mirror f or mariadb now | 19:36 |
| kfox1111 | can I get some more eyes on this: https://review.openstack.org/#/c/361186/ | 19:37 |
| sean-k-mooney | sdake: bifrost is trying to install mysql which i replace with mariadb at a later date but because we enable that repo it case the install of mysql-server to fail in bifrosts install playbook | 19:38 |
| sean-k-mooney | sdake: ubuntu provides mysql-server-5.7 this is enableing mariadb version 10 | 19:39 |
| *** vhosakot has quit IRC | 19:40 | |
| sdake | can bifrost run without mariadb 10? | 19:40 |
| sean-k-mooney | biforst is not enableing mariadb 10 kolla is | 19:41 |
| sdake | if so - ust remoe the mariadb repo file at the stat of the container | 19:41 |
| sdake | start of container build i mean | 19:41 |
| sean-k-mooney | oh am well i had issue using mysql instad of kolla in the past but i can try that | 19:41 |
| sean-k-mooney | /kolla/mariadb/ | 19:42 |
| sdake | maybe removal needs to be conditional | 19:42 |
| sdake | or maybe there is a way to versoinpin on ubuntu | 19:42 |
| sdake | i am not an ubuntu expert unfortunately | 19:42 |
| *** huikang has joined #openstack-kolla | 19:42 | |
| *** fragatina has joined #openstack-kolla | 19:43 | |
| sdake | sean-k-mooney in the gate log where is the output which makess you think that is the problem? | 19:43 |
| sean-k-mooney | http://paste.openstack.org/show/564851/ | 19:44 |
| sdake | sean-k-mooney - btw tested bootsstrap - works weell | 19:44 |
| sean-k-mooney | i ran it locally and found the real error. | 19:44 |
| sdake | sean-k-mooney we got a few patche up for bootstrap | 19:44 |
| sean-k-mooney | sdake: inc0_: has a few tweaks to it to make it faster | 19:44 |
| sdake | sean-k-mooney its already fast - 5-7 minutes | 19:45 |
| sean-k-mooney | sdake: mainly around how the hosts file is generated. currently its O(n*n) but it can be made O(1) | 19:46 |
| sdake | yes i saw his change | 19:46 |
| sdake | we were in tmuxtogther when it was discussed | 19:47 |
| sdake | whatever numbers we produce we have to multiply by 9.2!! | 19:47 |
| sdake | dave-mccowan interesting result with tls no? | 19:48 |
| inc0_ | sdake, it wasn't 5min | 19:49 |
| inc0_ | I hacked it to be 5min | 19:49 |
| sean-k-mooney | why 9.2? | 19:49 |
| inc0_ | now I proposed my former hack to upstream | 19:49 |
| sdake | ran again o centos | 19:49 |
| inc0_ | sean-k-mooney, inside joke, I'll tell you over beer | 19:49 |
| sean-k-mooney | ok | 19:49 |
| sean-k-mooney | beer good :) | 19:50 |
| inc0_ | cervesa con limon - I'm drinking that in Barcelona | 19:50 |
| inc0_ | and ciara | 19:50 |
| inc0_ | man I love this city | 19:50 |
| inc0_ | sean-k-mooney, btw I just limited hostname generation to controllers for now | 19:51 |
| inc0_ | quick fix and get's us there | 19:51 |
| inc0_ | we'll improve later when we need | 19:51 |
| inc0_ | sean-k-mooney, https://review.openstack.org/#/c/363028/ | 19:51 |
| *** celo_r has quit IRC | 19:51 | |
| sean-k-mooney | cool ya the change makes sense to me | 19:53 |
| sean-k-mooney | sdake: ill try to revert to mysql instead of mariadb and disable the mariadb repo in the bifrost containter for now and see if that works. | 19:54 |
| sean-k-mooney | sdake: diableing the repo allows mysql-server to be installed but im not sure if i will hit the service file issue i had before. | 19:55 |
| sean-k-mooney | sdake: im heading home to grab something to eat but ill try and test this soon and push a patch tonight with a fix | 19:56 |
| *** aNupoisc has quit IRC | 19:57 | |
| sean-k-mooney | before i run did support for using msdos partitions for ceph ever land? | 19:57 |
| *** harlowja has quit IRC | 19:58 | |
| *** vhosakot has joined #openstack-kolla | 19:58 | |
| *** msimonin has joined #openstack-kolla | 19:58 | |
| *** inc0_ has quit IRC | 19:59 | |
| *** sean-k-mooney is now known as sean-k-mooneyAFK | 19:59 | |
| *** berendt has quit IRC | 20:02 | |
| *** shardy has quit IRC | 20:02 | |
| *** shardy has joined #openstack-kolla | 20:05 | |
| *** gfhellma has quit IRC | 20:05 | |
| *** fguillot has quit IRC | 20:06 | |
| *** gfhellma has joined #openstack-kolla | 20:06 | |
| *** sdake has quit IRC | 20:11 | |
| *** sdake has joined #openstack-kolla | 20:12 | |
| sdake | sean-k-mooneyAFK souns good | 20:12 |
| *** sbezverk has joined #openstack-kolla | 20:16 | |
| sbezverk | damn internet, dropped right in the middle of message I typed, appologies.. | 20:17 |
| sdake | irtermite thanks for access to the osic gear - I am pretty sur we had everything we neededd around 8am | 20:21 |
| sdake | irtermite so feel free (if you haavent already) to take back control :) | 20:22 |
| *** dwalsh has quit IRC | 20:27 | |
| *** sbezverk_ has joined #openstack-kolla | 20:29 | |
| *** sbezverk has quit IRC | 20:29 | |
| *** celo_r has joined #openstack-kolla | 20:30 | |
| *** aNupoisc has joined #openstack-kolla | 20:32 | |
| *** banix has quit IRC | 20:33 | |
| *** rstarmer has joined #openstack-kolla | 20:34 | |
| openstackgerrit | Merged openstack/kolla: Removing development packages from Baremetal install defaults https://review.openstack.org/363138 | 20:35 |
| *** jtriley has quit IRC | 20:39 | |
| irtermite | sdake: much thanks! it's still yours for the next 2-3 hours or so. All good! | 20:40 |
| irtermite | glad it worked out for you all | 20:40 |
| Daviey | irtermite: Great. deploying bitcoin mining now | 20:42 |
| irtermite | hah | 20:43 |
| * irtermite blocks port 8333 | 20:44 | |
| openstackgerrit | Merged openstack/kolla: Customizations for dind https://review.openstack.org/353203 | 20:45 |
| irtermite | Daviey: ^^ | 20:45 |
| Daviey | :) | 20:46 |
| sdake | Daviey 130 nodes of that gear of bitcoin softare would make yo uabout 2 cents a day | 20:52 |
| sdake | Daviey i had a knc 600mhz miner - was makingabout 400bucks a day for the first month | 20:52 |
| sdake | asic based | 20:53 |
| Daviey | sdake: My mum always told me.. look after the pennies and the pounds will look after themselves | 20:53 |
| sdake | Daviey gret idiom ;) | 20:53 |
| sdake | Daviey i wish i coudl get my wife to stop spendin money 300 bucks a crack | 20:53 |
| sdake | 300 bucks 300 bucks adds up over a month | 20:53 |
| *** shardy has quit IRC | 20:53 | |
| Daviey | sdake: I get my crack much cheaper | 20:54 |
| *** celo_r has quit IRC | 20:55 | |
| *** v1k0d3n has quit IRC | 21:00 | |
| *** v1k0d3n has joined #openstack-kolla | 21:01 | |
| aNupoisc | Hi All, my bootstrap_osd.yml seems to run a command "docker exec -t kolla_toolbox sudo -E /usr/bin/ansible localhost | 21:04 |
| aNupoisc | -m find_disk" which is requesting password for ansible, do we need to create seperate user as ansible? I am trying to deploy ceph with kolla on a single node | 21:04 |
| *** huikang has quit IRC | 21:08 | |
| *** huikang has joined #openstack-kolla | 21:09 | |
| *** huikang has quit IRC | 21:13 | |
| *** rhallisey has quit IRC | 21:16 | |
| *** harlowja has joined #openstack-kolla | 21:17 | |
| *** rstarmer has quit IRC | 21:18 | |
| *** ayoung has quit IRC | 21:19 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Script generating kubernetes secrets from passwords.yml https://review.openstack.org/354199 | 21:19 |
| *** huikang has joined #openstack-kolla | 21:24 | |
| *** sdake has quit IRC | 21:26 | |
| *** salv-orlando has quit IRC | 21:26 | |
| *** fguillot has joined #openstack-kolla | 21:30 | |
| *** sdake has joined #openstack-kolla | 21:32 | |
| openstackgerrit | Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 | 21:35 |
| *** vhosakot has quit IRC | 21:37 | |
| *** fguillot has quit IRC | 21:40 | |
| *** vhosakot has joined #openstack-kolla | 21:40 | |
| *** aNupoisc has quit IRC | 21:43 | |
| *** gfhellma1 has joined #openstack-kolla | 21:44 | |
| *** lyncos has quit IRC | 21:44 | |
| *** gfhellma has quit IRC | 21:44 | |
| *** gfhellma has joined #openstack-kolla | 21:48 | |
| *** chrichip has joined #openstack-kolla | 21:49 | |
| *** salv-orlando has joined #openstack-kolla | 21:49 | |
| *** gfhellma1 has quit IRC | 21:50 | |
| *** banix has joined #openstack-kolla | 21:53 | |
| *** aNupoisc has joined #openstack-kolla | 21:54 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Make libvirt over tcp an option https://review.openstack.org/362457 | 21:57 |
| *** michauds has quit IRC | 22:03 | |
| *** zhiwei has joined #openstack-kolla | 22:04 | |
| *** fguillot has joined #openstack-kolla | 22:05 | |
| *** chrichip has quit IRC | 22:06 | |
| *** huikang has quit IRC | 22:06 | |
| *** huikang has joined #openstack-kolla | 22:07 | |
| *** rstarmer has joined #openstack-kolla | 22:07 | |
| *** zhiwei has quit IRC | 22:09 | |
| sbezverk_ | kfox1111: ping | 22:10 |
| *** huikang has quit IRC | 22:12 | |
| openstackgerrit | Merged openstack/kolla: Architecture guide doc https://review.openstack.org/360796 | 22:17 |
| *** diogogmt has quit IRC | 22:20 | |
| *** msimonin has quit IRC | 22:24 | |
| openstackgerrit | Merged openstack/kolla: Add defaults for interfaces to all.yml https://review.openstack.org/354173 | 22:26 |
| *** vhosakot has quit IRC | 22:31 | |
| *** jrist has quit IRC | 22:32 | |
| *** diogogmt has joined #openstack-kolla | 22:32 | |
| *** sdake has quit IRC | 22:36 | |
| *** gfhellma has quit IRC | 22:38 | |
| *** gfhellma has joined #openstack-kolla | 22:39 | |
| *** jrist has joined #openstack-kolla | 22:39 | |
| *** xwang has joined #openstack-kolla | 22:59 | |
| *** ayoung has joined #openstack-kolla | 23:00 | |
| *** rkrum has joined #openstack-kolla | 23:12 | |
| *** huikang has joined #openstack-kolla | 23:13 | |
| openstackgerrit | sean mooney proposed openstack/kolla: generate bifrost yaml configs https://review.openstack.org/349684 | 23:20 |
| openstackgerrit | sean mooney proposed openstack/kolla: add bifrost deploy ansible support https://review.openstack.org/349685 | 23:20 |
| openstackgerrit | sean mooney proposed openstack/kolla: introduce bifrost-deploy container https://review.openstack.org/319703 | 23:20 |
| openstackgerrit | sean mooney proposed openstack/kolla: adds bifrost ssh key generation https://review.openstack.org/350692 | 23:20 |
| openstackgerrit | sean mooney proposed openstack/kolla: add deploy-servers command https://review.openstack.org/349729 | 23:20 |
| *** salv-orlando has quit IRC | 23:21 | |
| *** huikang has quit IRC | 23:24 | |
| *** huikang has joined #openstack-kolla | 23:25 | |
| *** huikang has quit IRC | 23:29 | |
| *** v1k0d3n has quit IRC | 23:35 | |
| *** v1k0d3n has joined #openstack-kolla | 23:36 | |
| sbezverk_ | kfox1111: I see issue with haproxy http://paste.openstack.org/show/564919/ | 23:40 |
| *** zhurong has joined #openstack-kolla | 23:44 | |
| *** v1k0d3n has quit IRC | 23:47 | |
| *** banix has quit IRC | 23:47 | |
| *** v1k0d3n has joined #openstack-kolla | 23:49 | |
| *** v1k0d3n has quit IRC | 23:59 | |
| *** aNupoisc has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!