Monday, 2017-11-20

*** jascott1 has quit IRC00:00
*** yingjun has quit IRC00:05
*** salv-orlando has quit IRC00:14
*** yingjun has joined #openstack-kolla00:21
*** Pavo has quit IRC00:35
*** itlinux has quit IRC00:37
*** Pavo has joined #openstack-kolla00:43
*** zshi has joined #openstack-kolla00:50
*** hieulq has joined #openstack-kolla00:54
*** itlinux has joined #openstack-kolla00:56
*** tovin07_ has joined #openstack-kolla00:58
*** hieulq has quit IRC01:02
*** hieulq has joined #openstack-kolla01:04
*** itlinux has quit IRC01:06
masberI tried deploy openstack with external ceph using kolla-ansible. Deployment is successful but I can't upload/create an image https://bpaste.net/show/77d2ab681ebc any thoughts?01:11
*** salv-orlando has joined #openstack-kolla01:15
masberok, I found the error https://bpaste.net/show/65419b66706d01:28
masberso no permission to connect to ceph cluster, my question is which container is supposed to have the ceoh.client.glance.keyring and ceph.conf files?01:28
*** itlinux has joined #openstack-kolla01:30
*** kiennt26 has joined #openstack-kolla01:33
*** caowei has joined #openstack-kolla01:41
*** salv-orlando has quit IRC01:49
*** zhangfei has joined #openstack-kolla01:56
masberhi, it is me again with another question. Do I need to create 2 users cinder and cinder-backup or just 1 user called cinder and give him access to both volumes and backups volumes?02:17
masberthis is for kolla-ansible ceph external02:17
*** daidv has joined #openstack-kolla02:21
*** duonghq has joined #openstack-kolla02:22
*** Pavo has quit IRC02:25
Jeffrey4lduonghq, please review https://review.openstack.org/521372 which blocks the gate.02:31
lxdongfailed: [compute01] (item=/usr/share/kolla-ansible/ansible/roles/haproxy/templates/haproxy.cfg.j2) => {"changed": false, "failed": true, "item": "/usr/share/kolla-ansible/ansible/roles/haproxy/templates/haproxy.cfg.j2", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'api_interface'"}02:34
duonghqJeffrey4l, done02:34
Jeffrey4lthx02:34
duonghqnp02:34
Jeffrey4llxdong, guess wrong configurations. no ip address on the api_interface on the copute01 node?02:34
lxdongconfiguration didn't change.02:36
lxdongboth compute01 and compute02 came across this failure.02:38
Jeffrey4llxdong, are you using a fact cache? or run some cli to check the whether there is a address on api_interface on these two nodes02:40
*** salv-orlando has joined #openstack-kolla02:46
lxdongWhat if it is using cache,reboot server? Do you think it's RAID card that is using cache?02:47
openstackgerritMerged openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/52042302:49
Jeffrey4llxdong, if you never configure ansible.cfg, you are not using the ansible fact cache.02:52
lxdongI am stuck here.02:54
spsuryamorning guys02:54
spsuryasup Jeffrey4l duonghq02:55
Jeffrey4lmorning spsurya02:55
Jeffrey4lspsurya, could you review https://review.openstack.org/52137202:55
spsuryasure02:55
spsuryaJeffrey4l:02:55
Jeffrey4land https://review.openstack.org/508046  spsurya ;D02:55
Jeffrey4llxdong, check the environment carefully. i am still thinking you are using the wrong configuration. may the interface name is different on the compute01 and 0202:56
lxdongIs this due to update of kolla and kolla-ansible to the latest?02:56
spsuryaJeffrey4l:  https://review.openstack.org/#/c/521372/1 done02:57
Jeffrey4llxdong, do not think so.02:57
Jeffrey4lthx.02:58
lxdongexactly the same02:59
openstackgerritJeffrey Zhang proposed openstack/kolla stable/pike: Bump kafka version to 0.10.2.1  https://review.openstack.org/52140503:02
openstackgerritJeffrey Zhang proposed openstack/kolla stable/ocata: Bump kafka version to 0.10.2.1  https://review.openstack.org/52137603:03
openstackgerritJeffrey Zhang proposed openstack/kolla stable/ocata: Pin fluent-plugin-grok-parser to 2.1.4  https://review.openstack.org/52133603:04
openstackgerritJeffrey Zhang proposed openstack/kolla stable/ocata: implement deploy jobs in kolla  https://review.openstack.org/52133303:04
openstackgerritJeffrey Zhang proposed openstack/kolla stable/ocata: Dockerhub publisher job  https://review.openstack.org/52133403:04
openstackgerritChason Chan proposed openstack/kolla stable/pike: Fix the Ubuntu Fluentd image  https://review.openstack.org/52104503:05
lxdongshould #api_interface: "{{ network_interface }}" be commented out?03:06
lxdongwhich is commented out by default.03:07
duonghqhi spsurya03:15
openstackgerritDuong Ha-Quang proposed openstack/kolla master: Modify the version of skydive to v0.14.0  https://review.openstack.org/52085103:17
*** ArminderSingh has quit IRC03:18
*** Pavo has joined #openstack-kolla03:26
*** ArminderSingh has joined #openstack-kolla03:28
*** dave-mccowan has quit IRC03:39
openstackgerritHeetae Ahn proposed openstack/kolla master: Add RHEL subscription registration  https://review.openstack.org/49514803:45
*** mdnadeem has joined #openstack-kolla03:47
*** kolla-slack has quit IRC04:00
*** kolla-slack has joined #openstack-kolla04:00
lxdongcan two all-in-one nodes achieve high availability?04:10
lxdongtoke down one node and instances can't be created.04:14
*** coolsvap has joined #openstack-kolla04:22
*** yangyapeng has joined #openstack-kolla04:26
*** owalsh_ has joined #openstack-kolla04:57
*** Pavo has quit IRC05:01
*** owalsh has quit IRC05:01
*** janki has joined #openstack-kolla05:03
SamYaplelxdong: no05:04
SamYaplelxdong: but not because of openstack05:04
SamYaplefor ceph and galera, you need at least 3 nodes to do that safely05:05
SamYapleif you arent using ceph then *kind of*05:05
SamYaplea properly shutdown mariadb node thats connected to a galera cluster will *not* cause the galeracluster tofail, but an improperly shutdown or network paritioned node will cause the galera cluster to stop working05:06
SamYaplerabbitmq/memcached/openstack services are all fine with twonodes05:06
masberlxdong, If you have 2 nodes just do a hyperconverge installation cluster, that way you should be able to shut down 1 node and still work05:07
SamYaplemasber: thats not quite correct, see my post above05:07
SamYapleand is entirely incorrect for ceph05:08
masberSamYaple, yes, that is not the right way of deploying openstack/ceph, I just got the impression he was just testing openstack05:10
SamYaplesure, but even in that case itwont workwith ceph05:10
masberso I am trying to make kolla-ansible to work with external ceph... so far I got glance working, now I would like to get cinder working... I get an error when I try to create a volume 2017-11-20 15:37:46.114 7 ERROR cinder.scheduler.flows.create_volume [req-07141267-75ab-4d1b-a29d-1316ccc4a693 e07dcfd877b34a5dafb3386231ef92f1 fe558596e7e542b080e0daa89ce56152 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVol05:11
masberumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available05:11
SamYaplesounds like the configmight be off05:11
SamYaplecan you post the generated cinder.conf05:11
masberSamYaple, you mean cinder_api:/etc/cinder/cinder.conf? kolla deploys different cinder.conf depending on the container05:12
SamYapleyep05:13
*** skramaja has joined #openstack-kolla05:13
lxdongSamYaple will you post your post ? masber what is the correct way to create hyperconverged infrastrature, I meant to but seems incorrect by far.05:13
masberSamYaple, https://bpaste.net/show/c12aaf8a2f6005:14
lxdongthat't to say two nodes can never achieve HA ?05:15
masberSamYaple, rbd configuration is missing on cinder_api:/etc/cinder/cinder.conf but it does exists on cinder_volume containers. I don't know if that is expected?05:17
masberlxdong, sorry I got confused, I used 3 nodes in total, 1 nodes as controller and 2 of then hyperconverge for computing and storage05:19
lxdongIn the case I stimulate one node power outrage which should be  a improperly shutdown for galera and ceph -s timeout which suggest ceph is not functional , which is the root cause or they are independent?05:22
lxdongmasber ,can your cluster achieve instance HA using ceph backend?05:24
masberlxdong, yes, but as SamYaple said that is not the right way to do that05:25
SamYaplemasber: you arent configuring cinder ceph05:25
SamYaplelxdong: two nodes can never achieve HA for ceph.period05:25
SamYaplethey can for galera, kind of. but its veryunstable05:25
*** goutham has joined #openstack-kolla05:25
masberSamYaple, ceph configuration needs to go to cinder_api, cinder_volume and cinder_backup?05:25
SamYapleand kolla doesnt properly shutdown galera cluster05:25
lxdongMay I look at your multinode.yml masber?05:26
SamYaplemasber: just for your info, while sometimes some confis do generate differently per service, i never got around to doing that in cinder.conf. so those configs should be identical05:27
lxdongsorry ,multinode05:27
*** caowei has quit IRC05:28
*** jascott1 has joined #openstack-kolla05:29
SamYaplemasber: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/cinder/templates/cinder.conf.j2#L107-L121 you need to enable theoptions at the top of that if block05:29
gouthamHi all if at all if i want a stable openstack to be deployed can i go with master or should i go with pike??05:29
SamYaplelxdong: if you want ceph with two nodes, youll never be able touse ceph if one goes down. no need to dig into that further05:29
gouthamany idea?? after the recent fluentd issues??05:29
gouthamor is the newton a stable setup ?? if i have to deploy newton which tag should i be using?? any idea??05:30
lxdongyes ,your HCI worked well right? I am doing the same thing and tried external ceph too, reference to your configuration might be a good idea.05:32
SamYaplemasber: do you deploy 2 nodes as well?05:33
*** jascott1 has quit IRC05:33
SamYapleif so youllwant to revist you ha testing, because ceph is down when one of those nodes is down05:33
masberSamYaple, for testing yes, unfortunately I don't enough nodes to do everything :(05:34
SamYaplethats fine,im just saying, your ceph is 100% down when one node is down, its not ha05:34
masberSamYaple, even if the monitor is on the openstack controller?05:35
masberSamYaple, 2 nodes only for OSDs and the monitor running in a different node05:35
lxdongthat's to say there are four nodes in production, one contoller ,three hyperconverged nodes?05:35
SamYaplemasber: well thats not two nodes05:35
SamYaplemasber: and also not ha, since ifone node goes down(controller) everything stops05:36
masberlxdong, 3 in total, 1 nodes for controller which also runs ceph monitor and another 2 nodes for OSDs and nova05:36
lxdongso masber got a minial of05:36
masberSamYaple, yes you are right kolla-ansible deploys monitor on controller but on the inventory file you only specify 2 nodes as storage... my bad ignoring that05:37
masber(if you deploy ceph through kolla-ansible)05:38
SamYaplesure. thats shouldmakeyour data resliant. but thats farfrom "higly-available" just to clear up confusion05:38
lxdongmasber ,in your case external is the one meet your demand because you can't specify the node deploy ceph monitor to in inventory file?05:41
SamYaplelxdong: you can specify the monitor deploys in kolla-ansible too05:42
masberlxdong, yes plus I also want to learn ceph :)05:43
lxdongme too :)05:44
lxdongSamYaple ,how?05:48
lxdongchange [ceph-mon:children] ceph to[ceph-mon:children] control ?05:50
openstackgerritMerged openstack/kolla-ansible master: Remove setting of version/release from releasenotes  https://review.openstack.org/52102205:51
SamYaplelxdong: how ever you want to do it, sure. but the inventory file is 100% in your control, save for the naming of the groups05:55
duonghqJeffrey4l, are you there?06:02
Jeffrey4lduonghq, yes.06:02
*** caowei has joined #openstack-kolla06:03
duonghqcurrently, we are using ceph image which is built by our?06:03
lxdonggood point , at least my guess is right :) You mean two nodes with one controller is far from HA?06:03
Jeffrey4lduonghq, docker images? yes.06:03
*** threestrands has quit IRC06:03
duonghqJeffrey4l,  do we have plan switch to ceph one?06:03
*** caowei has quit IRC06:04
Jeffrey4lduonghq, this is talked and no any approach.06:04
Jeffrey4lduonghq, any reason why you want to ceph's?06:04
duonghqI just wonder why don't we just use image provided by ceph community, so that we do not need maintaining our image06:05
SamYapleduonghq: kolla currently builds *all* of its own images06:07
SamYapleit breaks thekolla model to use upstream images06:07
SamYapleto do that youd have to inject startup scripts at runtime (which i agree with) but others dont want to do that06:07
Jeffrey4lagree.06:08
duonghqSamYaple, can you elaborate the "model" here? is it we want to control the way image is built and work?06:08
Jeffrey4lthe way we use the image.06:09
Jeffrey4llike configuration file, how BOOTSTRAP the ceph mon or ceph osd06:09
SamYapleduonghq: kolla has extend_start.sh and other scripts like that. at the time we all felt that wasa gooddirection06:09
duonghqI got the idea, thanks Jeffrey4l and SamYaple06:10
SamYapleas time moved on, itwas clear that the upstream moved a different direction. to align kolla with upstream, we would need to inject the extend_start.sh scripts at run time06:10
Jeffrey4lthe better way, it integrate kolla with ceph official containerized deployed ceph.06:10
SamYapleim all for that if youcan getthe community to agree :) as of yet, i have not been able to06:10
duonghqJeffrey4l, I think it's one of the external ceph option?06:10
Jeffrey4lduonghq, yes. external ceph should work.06:11
Jeffrey4lso deploy ceph by using whatever you want, and kolla just use it.06:11
duonghqafaik, we completed external ceph support?06:11
Jeffrey4lyes  we are. some guys is using this feature.06:12
Jeffrey4lbut tbh, there is no test to cover this. so not sure whether it works very well now.06:12
duonghqhmm, we do not have test about external ceph or we have not had tests?06:13
Jeffrey4lwe do not have test/gate jobs about external ceph.06:13
duonghqokay06:14
Jeffrey4lbtw, could you review https://review.openstack.org/52137606:14
Jeffrey4lduonghq, ^^06:14
duonghqJeffrey4l, done06:15
Jeffrey4lthanks.06:15
duonghqJeffrey4l, do we have network isolation for external ceph?06:16
Jeffrey4lduonghq, what that mean?06:16
Jeffrey4lwhen using external ceph, the only requirement is ceph publish network is accessible by nova-compute/glance-api06:16
duonghqJeffrey4l, I mean some kind of storage interface configuration for access ceph public network06:17
*** cliles has quit IRC06:17
Jeffrey4lno. refer my previous sentence.06:18
duonghqokay, just want a little more detail on this06:18
duonghqthanks06:19
Jeffrey4lnp. ;D06:19
openstackgerritTone Zhang proposed openstack/kolla-ansible master: Doc: update swift-guide in kolla-ansible  https://review.openstack.org/52099606:21
*** janki is now known as janki|afk06:22
openstackgerritTone Zhang proposed openstack/kolla-ansible master: Doc: update swift-guide in kolla-ansible  https://review.openstack.org/52099606:23
*** lpetrut has joined #openstack-kolla06:27
*** jbadiapa has joined #openstack-kolla06:35
openstackgerritChason Chan proposed openstack/kolla-ansible stable/pike: Fix the deprecated syntax in 01-rewrite.conf  https://review.openstack.org/52143306:37
*** salv-orlando has quit IRC06:37
*** salv-orlando has joined #openstack-kolla06:38
masberfor external-ceph configuration, do I need to leave [storage] as empty in the inventory file?06:58
Jeffrey4lmasber, i think yes.07:01
*** cah_link has joined #openstack-kolla07:02
Jeffrey4lspsurya, still around? please review https://review.openstack.org/52137607:04
*** cah_link has quit IRC07:05
*** cah_link has joined #openstack-kolla07:05
spsuryaJeffrey4l: done07:05
Jeffrey4lthanks.07:06
*** xinliang has quit IRC07:06
*** lpetrut has quit IRC07:08
*** lpetrut has joined #openstack-kolla07:08
openstackgerritMerged openstack/kolla-ansible master: Doc: update swift-guide in kolla-ansible  https://review.openstack.org/52099607:09
*** lpetrut has quit IRC07:14
*** itlinux has quit IRC07:15
*** egonzalez has joined #openstack-kolla07:18
*** xinliang has joined #openstack-kolla07:18
*** lpetrut has joined #openstack-kolla07:22
*** janki|afk is now known as janki07:35
*** manheim has joined #openstack-kolla07:43
*** caowei has joined #openstack-kolla07:44
*** shardy has joined #openstack-kolla07:46
*** manheim has quit IRC07:50
*** manheim has joined #openstack-kolla07:51
*** dardelean has joined #openstack-kolla07:52
*** lpetrut has quit IRC07:53
*** manheim has quit IRC07:56
*** pcaruana has joined #openstack-kolla08:04
*** gfidente has joined #openstack-kolla08:15
openstackgerritJeffrey Zhang proposed openstack/kolla master: Add squid cache proxy to speed up build  https://review.openstack.org/52145008:17
*** dciabrin has joined #openstack-kolla08:19
*** zhangfei has quit IRC08:21
*** gkadam has joined #openstack-kolla08:26
openstackgerritMartin Mágr proposed openstack/kolla master: Clean fluentd_plugins_install block  https://review.openstack.org/51259808:26
*** magicboiz has joined #openstack-kolla08:27
openstackgerritJeffrey Zhang proposed openstack/kolla master: Add squid cache proxy to speed up build  https://review.openstack.org/52145008:33
*** zhangfei has joined #openstack-kolla08:34
*** manheim has joined #openstack-kolla08:36
*** amoralej|off is now known as amoralej08:38
*** goldyfruit has joined #openstack-kolla08:41
*** ktibi has joined #openstack-kolla08:42
*** salv-orlando has quit IRC08:42
ktibimorning all EU ;)08:42
*** salv-orlando has joined #openstack-kolla08:43
*** manheim has quit IRC08:44
*** salv-orlando has quit IRC08:47
*** numans has joined #openstack-kolla08:49
*** rmart04 has joined #openstack-kolla08:53
openstackgerritJeffrey Zhang proposed openstack/kolla master: Add squid cache proxy to speed up build  https://review.openstack.org/52145008:54
*** salv-orlando has joined #openstack-kolla08:58
*** owalsh_ is now known as owalsh08:58
*** goldyfruit has quit IRC08:58
*** numans has quit IRC09:08
*** athomas has joined #openstack-kolla09:08
*** numans has joined #openstack-kolla09:10
*** manheim has joined #openstack-kolla09:11
*** goldyfruit has joined #openstack-kolla09:11
*** manheim has quit IRC09:15
openstackgerritJeffrey Zhang proposed openstack/kolla master: Add squid cache proxy to speed up build  https://review.openstack.org/52145009:19
*** mgoddard has joined #openstack-kolla09:20
bjolomorning09:21
bjoloSamYaple: about injecting startup scripts... i (and a bunch of my colleagues) agree with you :)09:22
openstackgerritPierre Hanselmann proposed openstack/kolla-ansible master: Add horizon domain dropdown  https://review.openstack.org/51467609:23
bjoloduonghq: we use external ceph and it works just fine09:24
*** magicboiz has quit IRC09:25
*** magicboiz has joined #openstack-kolla09:27
*** egonzalez has quit IRC09:27
*** egonzalez has joined #openstack-kolla09:28
*** jascott1 has joined #openstack-kolla09:29
*** magicboiz has quit IRC09:32
*** goldyfruit has quit IRC09:33
*** jascott1 has quit IRC09:33
duonghqbjolo, thank you for your information09:35
*** yingjun has quit IRC09:35
*** magicboiz has joined #openstack-kolla09:39
lxdongSamYaple ,can three HCI nodes achieve the first level of HA?09:39
lxdongmasber ,what's your opinion?09:41
*** goldyfruit has joined #openstack-kolla09:42
gouthamHi all nova console-log and get-vnc-console log commands doesnt work when triggered continuosly09:48
gouthamhttps://hastebin.com/pixehakiko.rb09:49
gouthamhttps://hastebin.com/omifirupuj.rb but nova list shows the vm launched09:49
gouthamis this a *BUG*  or is there any issue with my deployment??09:50
*** lpetrut has joined #openstack-kolla09:50
*** lpetrut has quit IRC09:56
*** tovin07_ has quit IRC10:08
*** lpetrut has joined #openstack-kolla10:13
*** duonghq has quit IRC10:15
lxdongwhy don't you change your controller to the third HCI nodes? masber10:16
gouthamthe vm goes to error state when i reboot my vm10:17
*** caowei has quit IRC10:17
*** caowei has joined #openstack-kolla10:18
*** magicboiz has quit IRC10:18
openstackgerritKevin Zhao proposed openstack/kolla master: Creates directory /var/run/mysqld on aarch64 when building MariaDB docker image  https://review.openstack.org/52147910:21
*** magicboiz has joined #openstack-kolla10:22
openstackgerritKevin Zhao proposed openstack/kolla master: Add python-openstackclient to debian kolla-toolbox  https://review.openstack.org/52148110:22
*** jiriprox has joined #openstack-kolla10:23
*** magicboiz has quit IRC10:26
*** jaosorior has joined #openstack-kolla10:30
gouthamhttps://bugs.launchpad.net/kolla/+bug/1733300 any one please help me with this10:32
openstackLaunchpad bug 1733300 in kolla "nova service fails to find my VM regularly " [Undecided,New]10:32
*** kiennt26 has quit IRC10:32
*** magicboiz has joined #openstack-kolla10:33
gouthamhttps://bugs.launchpad.net/kolla/+bug/1733302 help on this as well10:37
openstackLaunchpad bug 1733302 in kolla "Vm fails to boot after reboot " [Undecided,New]10:37
egonzalezgoutham, marked one as duplicated, both are same issue. nova mark the vm in error because one of the hosts is not finding the vm10:52
egonzalezadd more info in the bug report, globals.yml, configs, inventory, logs, docker ps -a. etc10:52
gouthamOk sure thanks egonzalez i will add that10:53
openstackgerritMerged openstack/kolla stable/ocata: Bump kafka version to 0.10.2.1  https://review.openstack.org/52137610:59
openstackgerritVladislav Belogrudov proposed openstack/kolla-ansible master: Enable nfs driver volume snapshots by default  https://review.openstack.org/52148711:03
*** zhangfei has quit IRC11:03
*** goldyfruit has quit IRC11:15
egonzalezgoutham, qemu/nova bug https://bugs.launchpad.net/ubuntu/+source/nova/+bug/171813311:16
openstackLaunchpad bug 1718133 in nova (Ubuntu Artful) "Live block migration in Pike fails due to qemu-img" [Critical,Fix released] - Assigned to James Page (james-page)11:16
*** goldyfruit has joined #openstack-kolla11:20
egonzalezgoutham, binary images will work with kolla 5.0.0, source wont because of the pinned nova version11:22
gouthamok now i have to download binary images right??11:23
gouthamwill do it11:23
*** caowei has quit IRC11:28
*** openstackgerrit has quit IRC11:32
spsuryakrtaylor: ping11:36
*** yingjun has joined #openstack-kolla11:46
*** magicboiz has quit IRC11:47
*** magicboiz has joined #openstack-kolla11:48
*** gfidente^2nd has joined #openstack-kolla11:53
*** magicboiz has quit IRC11:55
*** magicboiz has joined #openstack-kolla11:55
*** gfidente has quit IRC11:56
*** dciabrin_ has joined #openstack-kolla12:01
*** dciabrin has quit IRC12:03
*** manheim has joined #openstack-kolla12:04
*** manheim has quit IRC12:08
*** jmccarthy has joined #openstack-kolla12:10
lxdongI am using three HCI nodes ,ceph cluster worked with a downgrade stage when taking down one node while instance on the node can't be access any more. masber ,SamYaple ,Jeffrey4l , I need to achieve instance HA.12:13
*** pbourke has quit IRC12:14
*** pbourke has joined #openstack-kolla12:16
*** unicell has quit IRC12:23
*** janki has quit IRC12:29
lxdongI tried console ,snapshot ,volume backup ,migration ,no way to recover the instance.12:31
lxdongthere are two replications, how to find and make use of another?12:33
*** Pavo has joined #openstack-kolla12:33
*** Pavo has quit IRC12:36
*** ansmith has quit IRC12:36
*** rhallisey has joined #openstack-kolla12:57
*** salv-orlando has quit IRC12:59
*** salv-orlando has joined #openstack-kolla13:00
*** salv-orlando has quit IRC13:04
*** goldyfruit has quit IRC13:12
*** goldyfruit has joined #openstack-kolla13:14
*** coolsvap has quit IRC13:20
*** goldyfruit has quit IRC13:26
sdakemorning13:31
*** ansmith has joined #openstack-kolla13:34
*** dave-mccowan has joined #openstack-kolla13:36
*** skramaja has quit IRC13:38
*** goldyfruit has joined #openstack-kolla13:43
*** dave-mcc_ has joined #openstack-kolla13:46
*** dave-mccowan has quit IRC13:48
rwellumMorning sdake13:49
*** jtriley has quit IRC13:57
*** openstackgerrit has joined #openstack-kolla13:58
openstackgerritJeffrey Zhang proposed openstack/kolla master: Bump kafka version to 0.11.0.2  https://review.openstack.org/52137213:58
*** yangyapeng has quit IRC13:59
*** salv-orlando has joined #openstack-kolla13:59
*** milan_ has joined #openstack-kolla14:00
gouthamegonzalez: i still get the same error https://hastebin.com/udaqavorug.py BTW i m just booting up VM i think it has nothing to do with the Live migration right??14:01
milan_morning folks! :) I'd like to create an ironic-inspector container; where/how do I start best?14:01
gouthamI used binary images this time egonzalez ..14:04
*** serlex has joined #openstack-kolla14:15
*** serlex has left #openstack-kolla14:15
*** jmccarthy has quit IRC14:21
*** goldyfruit has quit IRC14:22
*** rhallisey has quit IRC14:22
*** jmccarthy has joined #openstack-kolla14:24
*** yangyapeng has joined #openstack-kolla14:24
*** rhallisey has joined #openstack-kolla14:25
egonzalezgoutham, ubuntu may not packaged the fix yet. dont know14:28
*** yangyapeng has quit IRC14:29
*** benjamin_diaz has joined #openstack-kolla14:30
*** jmccarthy1 has joined #openstack-kolla14:31
*** pbourke has quit IRC14:32
*** jtriley has joined #openstack-kolla14:33
*** jmccarthy has quit IRC14:35
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Add missing hostnames to fluentd log messages  https://review.openstack.org/51459314:36
*** yangyapeng has joined #openstack-kolla14:42
*** mdnadeem has quit IRC14:44
*** hachi_ has joined #openstack-kolla14:45
*** yangyapeng has quit IRC14:46
*** pbourke has joined #openstack-kolla14:47
*** dougsz has joined #openstack-kolla15:02
*** yangyapeng has joined #openstack-kolla15:04
*** dave-mcc_ is now known as dave-mccowan15:06
*** yangyapeng has quit IRC15:08
*** zshi has quit IRC15:10
*** hongbin has joined #openstack-kolla15:11
benjamin_diazNot kolla especific, but has anyone runned the Tempest tests required por the Opnestack Powered Platform? I am only failing 2 tests, the objectstore-container-quotas ones.15:12
*** goldyfruit has joined #openstack-kolla15:12
krtaylorspsurya, pong15:15
*** yangyapeng has joined #openstack-kolla15:24
openstackgerritJames McCarthy proposed openstack/kolla-ansible master: WIP: Update swift templates for rsync_module  https://review.openstack.org/52113315:25
openstackgerritJeffrey Zhang proposed openstack/kolla master: Add squid cache proxy to speed up build  https://review.openstack.org/52145015:26
*** rhallisey has quit IRC15:28
*** yangyapeng has quit IRC15:28
*** rhallisey has joined #openstack-kolla15:29
*** ansmith has quit IRC15:30
*** hachi_ has quit IRC15:39
*** yingjun has quit IRC15:40
*** aagate has quit IRC15:46
*** jiriprox has quit IRC15:55
*** itlinux has joined #openstack-kolla15:59
*** egonzalez has quit IRC16:00
*** jtriley has quit IRC16:02
*** lrensing has joined #openstack-kolla16:02
*** jamesbenson has joined #openstack-kolla16:03
*** jtriley has joined #openstack-kolla16:04
*** gfidente^2nd is now known as gfidente16:08
*** rmart04 has quit IRC16:08
*** gfidente has quit IRC16:09
*** gfidente has joined #openstack-kolla16:09
*** lrensing has quit IRC16:21
*** lrensing has joined #openstack-kolla16:34
*** pcaruana has quit IRC16:35
*** salv-orlando has quit IRC16:38
*** salv-orlando has joined #openstack-kolla16:39
*** lebauce_ has joined #openstack-kolla16:43
*** salv-orlando has quit IRC16:43
*** jascott1 has joined #openstack-kolla16:44
*** jascott1 has quit IRC16:48
*** jtriley has quit IRC16:50
*** ntpttr_laptop has joined #openstack-kolla16:51
*** itlinux has quit IRC16:53
*** itlinux has joined #openstack-kolla16:54
*** jascott1 has joined #openstack-kolla17:00
*** jtriley has joined #openstack-kolla17:09
*** jamesbenson has quit IRC17:10
*** jamesbenson has joined #openstack-kolla17:10
*** cah_link has quit IRC17:11
jamesbensoninc0 Ping17:15
*** gkadam has quit IRC17:17
inc0good mornign17:18
inc0just woke up, what's up James17:18
*** dciabrin__ has joined #openstack-kolla17:22
*** lpetrut has quit IRC17:23
*** dciabrin_ has quit IRC17:24
inc0pbourke mandre egonzalez - https://review.openstack.org/#/c/519939/ can you review whenever get chance? this fixes gates...17:25
kfox1111morning.17:25
*** jamesbenson has quit IRC17:27
*** dougsz has quit IRC17:28
*** goldyfruit has quit IRC17:30
openstackgerritMerged openstack/kolla-kubernetes master: Specify unit (Gi) in common pvc  https://review.openstack.org/51946117:31
jascott1yay17:32
*** lrensing has quit IRC17:34
*** mgoddard has quit IRC17:38
*** mgoddard has joined #openstack-kolla17:38
*** salv-orlando has joined #openstack-kolla17:39
rwellumJeffrey4l: Is lokolla the correct repo?17:40
sdakeJeffrey4l around?17:40
*** lrensing has joined #openstack-kolla17:40
sdakerwellum yes it is17:40
sdakeanyone know how to forward a connection from 127.0.0.1 to the machine's main IP address?17:41
rwellumI know sbezverk used this for the docker registry - didn't know it was official17:41
sdakerwellum its not official - its part of the build process iirc17:41
*** ntpttr_laptop has quit IRC17:42
*** ntpttr_laptop__ has joined #openstack-kolla17:42
*** yangyapeng has joined #openstack-kolla17:43
*** salv-orlando has quit IRC17:45
*** rwsu has joined #openstack-kolla17:45
jascott1docker registry uses nodePort17:49
*** yangyapeng has quit IRC17:50
kfox1111sdake: what do you mean?17:50
kfox1111I mean, by forwarding from 127.0.0.117:50
sdakekfox1111 using kubectl port-forward17:50
sdakekfox1111 that only binds to 127.0.0.117:50
kfox1111oh. you want to expose a port-forward to a different machine?17:51
sdakesame machine17:51
sdakeI just need to be able to access it by ip17:51
kfox1111127.0.0.1 is available on the machine.17:51
sdake127.0.0.1 is not accesible from internets17:51
sdakeya sorry form a different machine17:51
kfox1111not sure the easiest way.... socat might work in a pinch.17:51
sdakedoes that go both ways?17:52
kfox1111I think so.17:52
sdakei'll check it out17:52
*** ntpttr_laptop__ has quit IRC17:52
kfox1111its a tool in the networking swiss army knife. :)17:52
*** mgoddard has quit IRC17:53
sdakehttps://www.cyberciti.biz/faq/linux-unix-tcp-port-forwarding/17:54
sdakeunfortunately it looks like greek to me ;)17:54
kfox1111time to read up on more networking. ;)17:54
kfox1111the first example looks about right:17:55
kfox1111socat TCP-LISTEN:80,fork TCP:202.54.1.5:8017:55
kfox1111might be able to do: socat TCP-LISTEN:3000,fork TCP:127.0.0.1:<proxieslocalport>17:55
* kfox1111 just spent the weekend boning up on udp and the dhcp rfc's.17:57
sdake sudo socat TCP-LISTEN:9090,reuseaddr,fork TCP:10.128.0.2:909017:57
sdake2017/11/20 17:56:37 socat[3907] E bind(5, {AF=2 0.0.0.0:9090}, 16): Address already in use17:57
sdakei tred following17:57
sdakeand get an addrinuse17:57
kfox1111cant use the same port l17:58
sdakemaybe i've got it17:58
*** pcaruana has joined #openstack-kolla18:00
sdakekfox1111 does this look correct: sudo socat TCP-LISTEN:127.0.0.9090,reuseaddr,fork TCP:10.128.0.2:900018:01
sdake2017/11/20 18:01:25 socat[3943] E bind(5, {AF=2 0.0.0.0:127}, 16): Address already in use18:01
kfox1111seems backwards18:02
sdakesudo socat TCP-LISTEN:127.0.0.1:9090 TCP:10.128.0.2:900118:03
sdake2017/11/20 18:03:13 socat[3958] E TCP-LISTEN: wrong number of parameters (2 instead of 1)18:03
sdakeso swap parameters?18:03
sdakeoh right- listen18:03
sdakegot it18:03
sdakesudo socat TCP:127.0.0.1:9090 TCP-LISTEN:10.128.0.2:900018:07
sdake2017/11/20 18:05:01 socat[3991] E TCP-LISTEN: wrong number of parameters (2 instead of 1)18:07
*** dciabrin__ has quit IRC18:07
sdakecurl 10.128.0.2:900018:07
sdakecurl: (7) Failed to connect to 10.128.0.2 port 9000: Connection refused18:07
sdakeseems like no worky18:07
*** dciabrin__ has joined #openstack-kolla18:07
sdakenot sure what that wrong number of parameters are18:07
kfox1111sudo socat TCP:127.0.0.1:9090 TCP-LISTEN:900018:07
sdakecoool that works18:09
sdakekfox1111 = brillaint :)18:09
kfox1111cool.18:09
*** manheim has joined #openstack-kolla18:12
*** yangyapeng has joined #openstack-kolla18:12
*** yangyapeng has quit IRC18:17
openstackgerritJames McCarthy proposed openstack/kolla-ansible master: Update swift templates for rsync_module  https://review.openstack.org/52113318:17
*** manheim has quit IRC18:20
*** hachi_ has joined #openstack-kolla18:26
*** jamesbenson has joined #openstack-kolla18:27
jamesbensonhey inc0, sorry multi-tasking this morning.... I had an issue but was able to figure it out.. it was related to the volume/instance creation ....  apparently something i thought was a bug isn't upon finding some bug reports about the same thing.18:28
inc0kfox1111: what do you use to deploy initial bare metal?18:28
kfox1111inc0: cobbler. but am working on a replacement.18:29
jamesbensoninc0, we use maas and like it here...18:29
inc0I'm playing with idea around something that would be effectively lightweight ironic18:29
jamesbensonfor bare metal18:29
*** lpetrut has joined #openstack-kolla18:29
inc0ipxe+mock metadata server18:29
inc0yeah I looked at it too18:30
*** yangyapeng has joined #openstack-kolla18:31
*** milan_ has quit IRC18:31
inc0so my goal is to get something that's extremally lightweight to provision seed node quickly and without big hassle18:31
inc0and everything, including maas, seems to be this giant monolith of a software which is painful in many ways18:31
inc0for cobbler - I hate kickstart scripts18:31
inc0there is coreos matchbox, that's for terraform, but I don't see any non-coreos deployment there18:32
inc0and coreos is wierd with it's ignition thingy18:33
inc0ironic/bifrost and maas are big and much more than I really need18:34
inc0you'd think it would be solved problem by now right?:)18:34
*** yangyapeng has quit IRC18:34
*** vhosakot has joined #openstack-kolla18:36
*** dciabrin__ has quit IRC18:37
*** dciabrin has joined #openstack-kolla18:37
*** salv-orlando has joined #openstack-kolla18:40
*** salv-orlando has quit IRC18:42
*** salv-orlando has joined #openstack-kolla18:42
*** lpetrut has quit IRC18:51
*** lpetrut has joined #openstack-kolla18:55
jamesbensonI agree inc018:55
jamesbensonif it helps at all, we have our maas sitting in a VM on our control plane18:56
jamesbensonbut it isn't without it's faults either...18:56
jamesbensonyou can really only deploy ubuntu nicely.. theoretically you can deploy cent, but has its own issues... and somehow, no docs mentioned anywhere you can deploy windows server, but they say it once, then nothing.18:57
*** yangyapeng has joined #openstack-kolla18:57
jamesbensonpersonally, I really liked how fuel deployed bare metal then deployed Openstack over it... wish kolla would do something similiar.18:57
*** lpetrut has quit IRC19:01
*** yangyapeng has quit IRC19:01
*** lpetrut has joined #openstack-kolla19:01
kfox1111inc0: I'm playing with a provisioning system too. very light weight.19:01
kfox1111basically using k8s to deploy k8s.19:01
kfox1111or minikube to deploy k8s.19:01
kfox1111see: https://hub.docker.com/r/kfox1111/provision-baseos-server/builds/ for the centos mirror part.19:02
kfox1111self contained container that you can kickstart centos machines from.19:03
kfox1111I'm working on some helm charts to deploy all the pieces so you can set it all up with one helm install.19:03
*** itlinux has quit IRC19:03
*** manheim has joined #openstack-kolla19:05
*** lrensing has quit IRC19:05
jascott1kfox1111 have you ever seen the problem where you cannot exec into any pods that arent on master?19:12
kfox1111yeah.19:12
jascott1whatup with dat?19:12
kfox1111tipically is a firewall issue on the compute node blocking access from the master.19:12
kfox1111or a network connectivity issue.19:12
jascott1yeah im reading about kubelet config but everything seems ok19:13
jascott1it seems like everything is working except that19:13
kfox1111for exec and logs, the master makes a http connection to the kubelet.19:13
kfox1111which is backwards from all the other stuff.19:13
kfox1111usually its the kubelet http accessing the apiserver.19:13
jascott1i cant find where kubelet logs a request auth failure19:14
kfox1111probably never saw the request?19:14
kfox1111are you using centos?19:14
*** yangyapeng has joined #openstack-kolla19:15
jascott1ubuntu19:15
kfox1111not as familior. I think they are still using ufm?19:15
jascott1so far googling suggests that it is the auth from api server to kubelet like you said but I seem to have all the flags for TLS client certs19:16
kfox1111might need to disable it temporarily to validate.19:16
jascott1ok thanks I do need to confirm that19:16
kfox1111if you arn't seeing any logs its probably never seeing the packets.19:16
*** yangyapeng has quit IRC19:20
*** rwsu has quit IRC19:30
*** itlinux has joined #openstack-kolla19:31
inc0kfox1111: what's your idea behind it? you're using pure dnsmasq+pxe or sth more?19:32
inc0jascott1: I also looked at matchbox code a bit and while ipxe+dnsmasq thing is useful there, it's all the coreos on top of it that I don't want19:35
jascott1right yeah thats unfortunate19:35
inc0my idea goes like this: install first node, docker-compose containers with dnsmasq+pxe, and another with http server for 2 things19:36
inc01. serve image contents for ipxe install19:36
inc02. serve mock metadata service for cloud images19:36
*** yangyapeng has joined #openstack-kolla19:36
inc0maybe spread ^ into 2 containers, dunno yet19:37
inc0https://www.youtube.com/watch?v=gNqQL-1gZF819:37
*** yangyapeng has quit IRC19:41
kfox1111inc0: I've seen 3 different ways to do this:19:42
kfox11111. all in one systems. like cobbler, spacewalk, etc. very not cloud native.19:42
kfox11112. single binary solutions. like digital-rebar-provision. compiles it all to a single go binary.19:43
kfox11113. microservices based. only one I really know of is kind of ironic. but very heavy and still rather monolithic like #1.19:43
inc0right19:44
kfox1111I'm trying to build something very cloud native.19:44
inc0well19:44
inc0me too in a way, but I was thinking of aio use case19:44
kfox1111dhcp conttainer, tftp container, http rpm server, rpm servers per yum repo, kickstart server, etc.19:44
inc0however way I envision it will be fairly cloud native-y19:44
kfox1111aio can be done via helm. :)19:44
inc0dnsmasq can do both tftp and dhcp19:45
kfox1111so no reason to do aio and big cluster be different.19:45
kfox1111yeah. but still mixes use cases.19:45
inc0sure19:45
kfox1111no reason not to do tftp and dhcp seperate when you can easily do so.19:45
inc0well, however to break this up, we're thinking of same thing effectively19:45
kfox1111I'm thinking pxiecore too.19:45
kfox1111yeah.19:45
kfox1111i have a prototype dhcp server that serves entries out of kube crd's too. :)19:46
*** absubram has joined #openstack-kolla19:46
kfox1111so, totally stateless. :)19:46
inc0so I wanted to do bare metal first, k8s later19:46
inc0separate these two19:46
kfox1111thats where you get forced into #2.19:46
kfox1111I want to not worry about day 2, by archetecting it as microservices.19:47
kfox1111then day 1 is just bundling it all into one vm.19:47
kfox1111k8s is great at microservices, so we just reuse that.19:47
inc0well what I'm saying is to be able to stop when you just install say ubuntu19:47
kfox1111thats a kickstart thingy.19:48
inc0rigth19:48
kfox1111just dont kubeadm join on it.19:48
inc0or lack of thereof19:48
kfox1111right.19:48
kfox1111(preseed in deb terms)19:48
kfox1111still all the same plumbing though.19:48
inc0so that can be done with metadata server19:48
inc0and cloud images19:48
kfox1111yeah, some server to do templating.19:49
kfox1111no on cloud images.19:49
inc0I think it's much more elegant than kickstarts/preseed19:49
inc0and not distro specific19:49
kfox1111I wana provision natively. much much more flexible.19:49
inc0well, image with cloud-init baked in19:49
kfox1111the big problem is, cloud images are built with the assumption of vm's.19:49
kfox1111vms are nice and pretty hardware wise.19:49
kfox1111bare metal is fugly.19:49
inc0yueah I know there are issues19:49
inc0we've seen them19:49
kfox1111very few machines work out of the box.19:49
inc0that being said, ironic will need this kind of thing19:50
kfox1111this is why coreos isn't so prtty a bare metal solution.19:50
kfox1111yeah, but tahts ironic's problem.19:50
*** amoralej is now known as amoralej|off19:50
inc0or distro problems19:50
kfox1111I want something very simple/light weight.19:50
inc0it's just adding one more pkg to distro19:50
kfox1111its just a 'cloud-init' in a kickstart file too.19:51
kfox1111image building is much harder.19:51
inc0I know, but what I'm saying is kickstart is distro specific19:51
kfox1111kind of.19:51
inc0cloud init is not19:51
kfox1111oh. our saying, template out most of the install stuff in cloud init instead of kickstart but stil have a kickstart to provision initially?19:52
kfox1111(btw, cloud init's prettty crappy for cross platform. :(  )19:52
kfox1111every distro configures it differently. :(19:52
kfox1111I'd rather kickstart it and run something like ansible standalone.19:52
kfox1111much more portable.19:53
inc0right, all I need is pubkey on machine19:53
kfox1111can drop that on with the kickstart/preseed.19:53
*** jmccarthy1 has left #openstack-kolla19:54
inc0right, that's what I was thinking of using cloud-init for19:55
kfox1111you gotta kickstart/preseed anyway. cloud init just adds complication.19:55
kfox1111as it doesn't really exist on bare metal.19:55
inc0why tho?19:56
kfox1111you have to ship a cloud init metadata file somehow just as much as pushing a key file.19:56
inc0I mean why you need kickstart anyway19:56
kfox1111how does cloud-init find data?19:56
kfox1111oh.19:56
kfox1111because I need to do stuff like:19:56
inc0169.254.169.25419:56
kfox11111. choose my partition layouts.19:56
kfox11112. choose network drivers.19:56
kfox1111169.254.169.254 doesn't exist with bare meta.19:56
kfox1111unless you are running ironic.19:56
kfox1111I'm trying to avoid it.19:56
*** yangyapeng has joined #openstack-kolla19:57
inc0or mock it on gateway in the way neutron does19:57
inc0it's one iptables rule and http server running19:57
kfox1111still need kickstart to customize partition layout, nics, storage controllers, etc.19:57
inc0yeah, true to that19:57
kfox1111so already gotta payt that cost19:58
kfox1111a metadata server is then extra cost.19:58
inc0right I guess19:59
kfox1111so one of the few parts I don't have currently is a template engine.19:59
inc0well, I just want to minimze kickstart/preseed stuff19:59
inc0as it's ugly19:59
kfox1111thinking of writing a simple go webserver thta pulls content from crd's.19:59
kfox1111me too in a way...19:59
kfox1111I mostly want to use this to provision k8s nodes.19:59
kfox1111which doesn't need very much. docker does the rest.19:59
kfox1111so like 3 repo servers.20:00
kfox11111 baseos, 2 security updates, 3 k8s repo.20:00
inc0right what I'd want is output barebone ansible inv and let ansible do the rest20:00
kfox1111if all the content were crd's in k8s, it would e easy to inventory to ansible. :)20:00
inc0anyway, let's work together on it because well, we work on it anyway;)20:01
kfox1111sure. :)20:01
*** yangyapeng has quit IRC20:01
inc0do you have any code already?20:01
kfox1111I was hoping to get a working manually installed via pixie prototype up tomorrow.20:02
kfox1111got a fair chunk. :)20:02
inc0mind sharing?20:02
kfox1111I'll try and post a non working prototype later today? its close though.20:03
inc0cool thanks20:03
inc0I was planning on focusing on this stuff this week20:03
kfox1111the dhcp stuff is crap for sure. want to get aio working before I tackle proper dhcp. got a co-worker working on a much more elegant solution.20:04
kfox1111cool. how fortunate. :)20:04
inc0yeah, that's cool coincidence:)20:04
kfox1111I spent the last 2 weeks debugging dhcp udp flows, figuring out dhcp relaying, etc.20:04
kfox1111for this kind of thing. :)20:04
inc0I promised our demo team that I'll help them figure out quick bare metal provisioning for all the demos they do20:04
kfox1111cool.20:05
kfox1111if this is done with crd's, you could even put your hardware inventory in as a helm chart.20:05
kfox1111burn your minikube down, and helm install the provisioner all the way back up. :)20:05
inc0crd?20:05
kfox1111custom resource definition.20:05
benjamin_diazIf someone has time, please take a look at https://review.openstack.org/#/c/520761/20:05
inc0ah ok20:06
kfox1111used to be called 3rd party resource.20:06
kfox1111so, you have kind=mythingy20:06
kfox1111:)20:06
kfox1111kind=dhcp-host.... ;)20:06
kfox1111kind=kickstart ...20:06
inc0benjamin_diaz: well....it shouldn't destroy images on deploy host anyway20:07
kfox1111declaratively tell k8s cluster your wishes, and the provisioning system relaizes it. :)20:07
inc0right, my first goal is quickest setup possible on aio20:07
kfox1111mine's slightly different,20:07
kfox1111in that I want to be able to start up a system, then sluff off management of the system to the nodes installed.20:08
inc0from bunck of nuc's -> working env (k8s or not) in least time20:08
kfox1111ie, minikube start a provisioner, install first node, move all the crd's/helm charrts to the spawned cluster. now can provision more nodes.20:08
inc0but however we run it, k8s or docker-compose, components are the saem20:08
kfox1111trueish.20:08
kfox1111most of the work is in the pod descriptions though I think.20:09
kfox1111and crd's.20:09
kfox1111can't exactly replicate that with docker-compose...20:09
inc0right, for me that's step 2;)20:09
inc0well you can go simple with it20:09
inc0net=host20:09
inc0and that's pretty much it20:09
kfox1111I guess what I'm saying is, where do you store your data? I want to not manage that myself so k8s crds are awesome.20:09
inc0you use containers as glorified packages, kolla-ansible style20:09
kfox1111with docker-compose you'd have to use files I guess.20:10
kfox1111might still work.20:10
inc0right20:10
kfox1111I'm mostly thinking on relying on k8s service tokens and crds.20:10
kfox1111ok.20:10
benjamin_diazinc0 It didn't in ocata, it does in pike. Suffered a lot when I discovered it. There is a bug related, where in the comments, egonzalez argued in favor of mantaining the destruction of images in deployer host. So I thought of leaving that feature, but making it so it is disabled by default.20:10
kfox1111so, the real missing piece is the template engine.20:10
kfox1111for consistency with the rest of the stack, was thinking gotl..20:11
kfox1111I guess that engine could work either from crds or from files on disk.20:11
inc0benjamin_diaz: as in, kolla-ansible destroy without --include images destroys stuff?20:11
inc0if it has include-images, I'd say it should20:12
kfox1111I guess technically though, we can just use static kickstart files too.20:12
inc0but that being said, you can just run destroyu on all nodes besides deploy20:12
kfox1111just way less good.20:12
inc0as deploy node doesn't have any running containers or volmes anyway20:12
kfox1111I could prototype it out static, and then we could make it dynamic later.20:12
inc0well I'm still more in favor of kickstart just laying down pubkey20:13
inc0and let ansible do the rest20:13
kfox1111yeah. totally should be doable.20:13
benjamin_diazOriginally, --include-images only destroyed images in target hosts, but not on the machine where you executed the command. Now, because localhost was added to the inventory file, it also destroys the image on the deployment host.20:13
kfox1111kickstart still needs templating for mac addresses, partition info sharing, etc.20:13
kfox1111ie, making classes of machiens and just pointing at a partition info template chunk.20:14
kfox1111so you don't have to duplicate the partition info per host.20:14
inc0right but you can modify ansible-playbook run20:14
kfox1111not much more then a glorified include.20:14
inc0and disable localhost20:14
kfox1111hmm?20:14
kfox1111oh. for later things, yeah.20:14
inc0sorry kfox1111 was talking to benjamin_diaz ;)20:14
kfox1111ok. :)20:14
inc0well I can quickly write piece of python for kickstart templating20:15
kfox1111bbiaf. gotta catch up with someone.20:15
kfox1111yeah. trying to use gotl though.20:15
inc0if you're hell bend on it can do that too;)20:15
kfox1111since users will be seeing much more gotl in my use case.20:15
kfox1111well, true. we could support both. :)20:15
kfox1111just lilke the deployment images, since its microservices we c an just run both. :)20:16
inc0well, whatever, I can do golang as well, python is just faster for me20:16
kfox1111gota go. something broke. bbiab.20:16
inc0sure20:16
kfox1111true. has the benifit of being more cobbler compatible too.20:16
kfox1111though they are cheetah. which is close to jinja, but a little different.20:16
kfox1111bbiab20:16
*** yangyapeng has joined #openstack-kolla20:17
*** yangyapeng has quit IRC20:22
*** salv-orlando has quit IRC20:24
*** salv-orlando has joined #openstack-kolla20:25
jascott1https://opensource.googleblog.com/2017/11/container-diff-for-comparing-container-images.html20:26
openstackgerritBenjamin Diaz proposed openstack/kolla-ansible master: Added var to disable deploy host image destruction  https://review.openstack.org/52076120:27
*** salv-orlando has quit IRC20:28
*** hachi_ has quit IRC20:29
*** lrensing has joined #openstack-kolla20:30
*** ktibi has quit IRC20:34
*** salv-orlando has joined #openstack-kolla20:36
*** pcaruana has quit IRC20:39
*** yangyapeng has joined #openstack-kolla20:41
jamesbensoninc0: I had some pretty crazy kickstart files in a straight up pxe server... not sure if I read the whole history there, but I used to put my ssh keys in the kickstart file...20:43
inc0right...;)20:44
inc0as I said, seems to be unsolved issue still;)20:44
*** yangyapeng has quit IRC20:45
kfox1111jascott1: cool. I wrote my own recently. :)20:49
jascott1yeah I thought you might be interested in that link, could be useful ;)20:49
kfox1111https://github.com/kfox1111/hubbuildtools20:49
kfox1111check this out: https://github.com/kfox1111/dhcp :)20:50
SamYapleo/ jascott120:50
SamYaplesup buddy20:50
jascott1SamYaple whatsup!20:50
*** rwsu has joined #openstack-kolla20:54
*** rwsu has quit IRC21:00
*** lrensing has quit IRC21:00
*** rwsu has joined #openstack-kolla21:01
*** retr0h has quit IRC21:02
*** retr0h has joined #openstack-kolla21:02
*** krtaylor has quit IRC21:03
*** lrensing has joined #openstack-kolla21:04
*** rwsu has quit IRC21:05
*** rwsu has joined #openstack-kolla21:05
*** rwsu has quit IRC21:06
*** jamesbenson has quit IRC21:06
*** rwsu has joined #openstack-kolla21:06
*** gfidente has quit IRC21:08
*** jamesbenson has joined #openstack-kolla21:10
sdakekfox1111 have a moment?21:10
*** rhallisey has quit IRC21:15
*** jamesbenson has quit IRC21:16
*** krtaylor has joined #openstack-kolla21:16
*** benjamin_diaz has quit IRC21:16
*** dave-mccowan has quit IRC21:18
*** jamesbenson has joined #openstack-kolla21:19
*** jamesbenson has quit IRC21:21
*** jamesbenson has joined #openstack-kolla21:25
*** jamesbenson has quit IRC21:28
*** rwsu has quit IRC21:29
rwellumhas anyone seen this? keystone container complaining: "sh: line 3: /usr/bin/openstack: No such file or directory" - Pike generated from source.21:32
rwellumSorry neutron - not keystone21:32
rwellumjascott1: ^^ ?21:33
jascott1rwellum idk. i did have an issue with from source horizon container not having symlink for actual dashbaord content but that was a known bug21:34
rwellumThanks jascott1  - yes this looks like neutron missing an install step21:35
inc0rwellum: check if you have this change https://review.openstack.org/#/c/519971/21:38
inc0dependency mgmt is hard21:39
*** mchlumsky has joined #openstack-kolla21:41
rwelluminc0: ah crud - I was a bit out of date - I'll rety21:42
rwellumretry21:42
inc0yeah we got hit by this painfully :(21:43
rwellumDefinitely missing this - thanks!21:46
*** jamesbenson has joined #openstack-kolla21:48
rwelluminc0: Am I safe to just rebuild neutron?21:49
inc0rwellum: you want to rebuild kolla-toolbox21:49
inc0but yeah you should be ok with just this one21:49
*** jamesbenson has quit IRC21:52
inc0any centos ppl around?21:57
*** jamesbenson has joined #openstack-kolla21:58
*** jamesbenson has quit IRC21:58
*** jamesbenson has joined #openstack-kolla21:59
*** itlinux has quit IRC22:01
*** itlinux has joined #openstack-kolla22:04
*** shardy has quit IRC22:06
*** jmccarthy has joined #openstack-kolla22:07
*** rwsu has joined #openstack-kolla22:10
*** yangyapeng has joined #openstack-kolla22:23
*** mikal has joined #openstack-kolla22:25
mikalI'm following the kolla setup instructions, and canal-2tckp is in a crashloop. Is that something I should talk to you about here, or should I talk to the canal people instead?22:26
mikalSorry, kolla-kubernetes that is22:26
*** yangyapeng has quit IRC22:28
rwellummikal: openstack version?22:30
mikalI haven't gotten as far as installing openstack22:30
rwellummikal: I think first debug locally, because lot of folks using canal with no issues.22:30
mikalI'm still at the step just before "verify installation" for kube at https://docs.openstack.org/kolla-kubernetes/latest/deployment-guide.html22:31
rwellumSorry kubectl version etc.22:31
mikaluser@lab8:~$ kubectl version22:31
mikalClient Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}22:31
mikalServer Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}22:31
mikalIt looks like one of the containers in that pod isn't working, but I'm still trying to learn enough kube to work out which one...22:31
rwellumHaven't tried 1.8.4 yet - 1.8.2 yes.22:32
mikalkube-system   canal-2tckp                    2/3       CrashLoopBackOff   6          10m       192.168.50.32   lab822:32
rwellumSo "kubectl describe pod canal-2tckp -n kube-system"22:32
mikalOh, its the calico node22:32
rwellumyup22:34
mikalAhhh, this is the thing I think:22:34
mikaltime="2017-11-20T22:33:17Z" level=error msg="Hit error initializing TPR" error="resource does not exist: {{ } {global-config.projectcalico.org  kube-system    %!s(int64=0) 0001-01-01 00:00:00 +0000 UTC <nil> %!s(*int64=<nil>) map[] map[] [] [] } Calico Global Configuration [{v1}]}"22:34
mikalAt least that's an error I can google22:34
rwellumAlso: kubectl log canal-2tckp -n kolla22:34
rwellumRunning from VM?22:35
mikalNo, this is a baremetal install22:35
rwellumAh tricky  - you're brave :)22:35
rwellumYou can ping google.com right?22:36
rwellumcentos or ubuntu and version?22:36
mikalhttps://github.com/samsung-cnct/kraken-lib/issues/90122:36
mikalUbuntu 16.0422:36
mikalYeah, the machine has working internet connectivity22:36
mikalI think this is a canal version incompatability22:37
mikalSo if I change the canal version in canal.yaml and then kubectl apply the new version, does that do the right thing?22:39
rwellumSo which version did you download?22:42
mikalOf canal?22:43
mikalThe install guide uses a YAML file which specifies 2.4.122:43
mikalThat bug I found implies I should use 2.6.1 instead22:43
mikalGiving it a try now22:43
*** yangyapeng has joined #openstack-kolla22:44
inc0mikal: you can try different network overlays22:45
inc0canal is quite sophiscicated one22:45
rwellumYeah I am using: "          image: quay.io/calico/node:v2.6.2"22:45
inc0and unless you want calico I'd stick to flannel or weave22:45
mikalinc0: oh, I had flannel working before, but the docs strongly suggested canal22:45
mikalinc0: so I rebuilt with canal22:45
inc0well canal is calico-in-flannel22:46
inc0so it's another layer of complexity out there22:46
inc0cool tech, but I wouldn't call it simplest22:46
*** rwsu has quit IRC22:47
mikalHmmm, do I have to reinstall the OS to make canal go away? I can tear down the kube and rebuild, but I don't know what else (i.e. network config) that might have been done22:47
rwellumWe use it in the gates, and the deployment guide though inc022:47
mikalYeah, I'm just tyring to follow the deployment guide22:47
inc0rwellum: yeah I know, it's cool, just saying that there are simpler ones22:47
inc0and it doesn't affect opesntack itself that much22:47
*** jamesbenson has quit IRC22:48
rwellumYou may be able to 'kubectl delete pod <name> -n kube-system - after you make your changes mikal22:48
mikalWhere is the gate job defined? It would be interesting to compare that with the deployment guide and see if there's a reason this isn't working22:48
rwellumBut might be safer to restart.22:48
*** jamesbenson has joined #openstack-kolla22:48
*** yangyapeng has quit IRC22:49
inc0mikal: https://github.com/openstack/kolla-kubernetes/tree/master/tests/bin22:49
*** itlinux has quit IRC22:49
rwellumThey are a little hard to read mikal22:49
rwellumYou can also try: https://github.com/openstack/kolla-kubernetes/blob/master/contrib/orchestration/ko.py22:49
rwellumFor a quick start to test your environment etc.22:49
*** itlinux has joined #openstack-kolla22:50
mikalOh, you're using a different version of canal in the gate...22:51
mikal1.7 vs 1.6 in the deploy guide22:51
inc0https://github.com/openstack/kolla-kubernetes/blob/1d1a9c2177b87866685249a5599ea95263174a21/tools/setup_gate_ceph.sh if you stat from this file you should be able to follow this22:51
inc0yeah deploy guide wasn't updated I guess22:51
rwellumDeployment guides always get out of date22:51
rwellumBut the tool above is up-todate :)22:52
mikalYep, I have to go to a meeting now but will continue to play later22:55
mikalIf I can identify the updates the deployment guide needs I'll send a patch22:55
*** lrensing has quit IRC22:55
rwellumSure mikal  - I have a few too :)22:55
mikalOh, bumping to 1.7 worked!22:59
mikalThat's a win22:59
*** kristaps_ has joined #openstack-kolla23:00
rwellumnice!23:02
mikalNow to see if busybox runs23:02
mikalOMG, it works!23:02
*** lvdombrkr has quit IRC23:02
mikalI am a genius!23:02
*** jamesbenson has quit IRC23:03
*** pc_m has quit IRC23:11
*** pc_m has joined #openstack-kolla23:12
*** jrist is now known as jrist-afk23:13
mikalOh course, then I get weird errors from the next step...23:14
mikal:~$ kubectl apply -f rbac-kolla.yaml23:14
mikalerror: error validating "rbac-kolla.yaml": error validating data: unknown object type schema.GroupVersionKind{Group:"rbac.authorization.k8s.io", Version:"v1alpha1", Kind:"ClusterRoleBinding"}; if you choose to ignore these errors, turn validation off with --validate=false23:14
masbergood morning, do I need to specify a storage node in the inventory file if I choose to use external ceph? I am using kolla-ansible23:16
inc0masber: yes23:26
*** lrensing has joined #openstack-kolla23:27
inc0you still need cinder-volume to land somewhere23:27
masberinc0, you mean cinder-volume container?23:27
inc0yes23:27
inc0cinder-volume service is deploymet on [storage]23:27
masberok, so I still need to put ceph nodes as storage nodes in the ansible inventory file to deploy cinder-volume container there? or do i only need 1 ceph node there?23:29
*** lpetrut has quit IRC23:32
masberinc0, ^^^23:33
inc0no23:33
inc0you just neeed a storage node23:33
inc0or nodes23:33
inc0doesn't need to be same as ceph nodes23:33
inc0what cinder-volume will do is it'll conenct to ceph-mons to deal with volumes23:34
masberok, thank you very much for the clarification23:34
inc0no worries23:34
masberI will give it a go :)23:34
*** lrensing has quit IRC23:34
*** lrensing has joined #openstack-kolla23:41
*** gfhellma has joined #openstack-kolla23:49
*** lrensing has quit IRC23:55
*** yingjun has joined #openstack-kolla23:56
*** Pavo has joined #openstack-kolla23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!