Tuesday, 2017-06-20

*** jascott1 has quit IRC00:05
*** yee37922 has joined #openstack-kolla00:09
*** openstack has joined #openstack-kolla00:18
*** ducttape_ has quit IRC00:19
*** sambetts_ has joined #openstack-kolla00:19
*** cmceniry has joined #openstack-kolla00:20
*** igordcard has joined #openstack-kolla00:20
*** PramodJ has quit IRC00:22
*** strigazi has joined #openstack-kolla00:24
*** yingjun has joined #openstack-kolla00:32
*** manheim has joined #openstack-kolla00:34
*** yuanying has quit IRC00:36
*** manheim has quit IRC00:39
*** jaosorior has joined #openstack-kolla00:40
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CEILOMETER CHANGES  https://review.openstack.org/47545200:42
*** rwsu has quit IRC00:48
*** srwilkers_ has joined #openstack-kolla00:48
*** srwilkers_ has quit IRC00:49
*** TxGirlGe_ has quit IRC00:51
*** cuongnv has joined #openstack-kolla00:56
*** tonyb_ is now known as tonyb00:57
*** dciabrin has joined #openstack-kolla01:01
*** caowei has joined #openstack-kolla01:02
*** harlowja has joined #openstack-kolla01:04
*** jascott1 has joined #openstack-kolla01:05
*** jtriley has joined #openstack-kolla01:09
*** tovin07_ has joined #openstack-kolla01:12
*** rwsu has joined #openstack-kolla01:12
*** caowei_ has joined #openstack-kolla01:13
*** caowei has quit IRC01:16
*** caowei_ is now known as caowei01:16
*** magicboiz has quit IRC01:16
*** ducttape_ has joined #openstack-kolla01:16
*** yangyapeng has joined #openstack-kolla01:20
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Fix the database user name for all roles  https://review.openstack.org/47525801:26
*** ducttape_ has quit IRC01:26
*** ducttap__ has joined #openstack-kolla01:31
*** xinliang has quit IRC01:31
*** xinliang has joined #openstack-kolla01:34
*** xinliang has quit IRC01:34
*** xinliang has joined #openstack-kolla01:34
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CEILOMETER CHANGES  https://review.openstack.org/47545201:34
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Revert "Fix a typo and complement doc in ceph-guide.rst"  https://review.openstack.org/47561101:36
*** ducttap__ has quit IRC01:36
*** xinliang_ has joined #openstack-kolla01:44
*** yangyape_ has joined #openstack-kolla01:48
*** yangyapeng has quit IRC01:52
*** yangyape_ has quit IRC01:56
*** yangyapeng has joined #openstack-kolla01:56
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CEILOMETER CHANGES  https://review.openstack.org/47545202:01
*** xinliang has quit IRC02:02
openstackgerritLi Yingjun proposed openstack/kolla-ansible master: Add option ha_vrrp_health_check_interval for l3  https://review.openstack.org/47561402:07
*** eanylin_ has quit IRC02:11
openstackgerritLi Yingjun proposed openstack/kolla-ansible master: Add option ha_vrrp_health_check_interval for l3  https://review.openstack.org/47561402:12
*** unicell has quit IRC02:18
openstackgerritLi Yingjun proposed openstack/kolla master: ping is needed for ha router  https://review.openstack.org/47561702:21
*** cuongnv_ has joined #openstack-kolla02:21
*** cuongnv has quit IRC02:24
*** Guest8722 is now known as sdake02:29
*** sdake has joined #openstack-kolla02:29
*** ducttape_ has joined #openstack-kolla02:32
*** masber has joined #openstack-kolla02:34
*** yuanying has joined #openstack-kolla02:35
*** masuberu has quit IRC02:37
*** masuberu has joined #openstack-kolla02:38
*** masber has quit IRC02:42
*** ducttape_ has quit IRC02:46
*** zhugaoxiao has joined #openstack-kolla02:51
*** cuongnv_ has quit IRC02:57
*** yuanying has quit IRC03:04
*** yuanying has joined #openstack-kolla03:04
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add webconsole support in ironic  https://review.openstack.org/47562803:05
*** spsurya has joined #openstack-kolla03:05
spsuryamorning03:06
*** cuongnv has joined #openstack-kolla03:08
*** yuanying has quit IRC03:09
*** kbaegis has quit IRC03:17
openstackgerritcaoyuan proposed openstack/kolla master: Update min Ansible version to 2.2.0  https://review.openstack.org/45252903:20
*** eanylin_ has joined #openstack-kolla03:39
*** ducttape_ has joined #openstack-kolla03:43
*** jamesbenson has joined #openstack-kolla03:45
*** jamesbenson has quit IRC03:49
*** ducttap__ has joined #openstack-kolla04:00
*** ducttape_ has quit IRC04:00
*** ducttap__ has quit IRC04:05
*** kbaegis has joined #openstack-kolla04:07
*** ducttape_ has joined #openstack-kolla04:08
*** ducttape_ has quit IRC04:13
*** ducttape_ has joined #openstack-kolla04:16
*** kbaegis1 has joined #openstack-kolla04:19
*** kbaegis has quit IRC04:19
*** caowei has quit IRC04:21
*** ducttape_ has quit IRC04:21
*** ducttape_ has joined #openstack-kolla04:21
*** ducttape_ has quit IRC04:26
*** jamesbenson has joined #openstack-kolla04:26
*** jamesbenson has quit IRC04:31
*** kbaegis1 has quit IRC04:35
*** ducttape_ has joined #openstack-kolla04:39
*** zhubingbing_ has joined #openstack-kolla04:42
zhubingbing_Hi  guys04:43
*** ducttape_ has quit IRC04:43
*** salv-orlando has joined #openstack-kolla04:48
*** salv-orlando has quit IRC04:49
*** salv-orlando has joined #openstack-kolla04:49
spsuryazhubingbing_: sup04:50
*** salv-orlando has quit IRC04:51
*** salv-orlando has joined #openstack-kolla04:51
*** janki has joined #openstack-kolla04:53
*** zhubingbing_ has quit IRC04:57
*** rstarmer has quit IRC04:58
*** unicell has joined #openstack-kolla05:06
*** jamesbenson has joined #openstack-kolla05:08
*** jtriley has quit IRC05:10
*** yuanying has joined #openstack-kolla05:11
*** jamesbenson has quit IRC05:12
*** pcaruana has joined #openstack-kolla05:14
*** salv-orl_ has joined #openstack-kolla05:15
*** salv-orlando has quit IRC05:18
*** skramaja has joined #openstack-kolla05:21
*** pcaruana has quit IRC05:30
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CEILOMETER CHANGES  https://review.openstack.org/47545205:34
*** zhubingbing_ has joined #openstack-kolla05:35
zhubingbing_Hi spsurya05:35
spsuryazhubingbing_: trying to help caoyuan into https://review.openstack.org/#/c/464673/05:36
spsuryawanted to find the changes related to celiometer05:37
spsuryaw.r.t event_connection05:37
spsuryadid you see anywhere ?05:37
zhubingbing_I. Don 't agree05:37
zhubingbing_I have see event_connection in ceilomete.conf05:38
spsuryame too05:38
spsuryaI was looking Into master just in case05:39
zhubingbing_If remove ,kilometer maybe not  working05:44
zhubingbing_ceilometer05:44
zhubingbing_https://github.com/openstack/ceilometer/blob/master/ceilometer/dispatcher/__init__.py#L3805:44
spsuryaagree05:47
*** jamesbenson has joined #openstack-kolla05:49
*** ducttape_ has joined #openstack-kolla05:53
*** jamesbenson has quit IRC05:53
masuberugood afternoon all,06:11
*** iniazi_ has quit IRC06:12
*** iniazi has joined #openstack-kolla06:12
*** manheim has joined #openstack-kolla06:15
*** magicboiz has joined #openstack-kolla06:22
*** unicell1 has joined #openstack-kolla06:23
*** manheim has quit IRC06:23
*** unicell has quit IRC06:24
masuberuI am running an openstack kolla-ansible environment for a few weeks now. Just realized that the compute node is running our of memory, the reason is elastic search is taking all the memory, is this a knows issue?06:26
*** caowei has joined #openstack-kolla06:26
*** magicboiz has quit IRC06:27
*** jamesbenson has joined #openstack-kolla06:30
*** magicboiz has joined #openstack-kolla06:33
*** jamesbenson has quit IRC06:35
*** pcaruana has joined #openstack-kolla06:49
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Remove ceilometer-api and ceilometer-collector services  https://review.openstack.org/47075206:53
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CLEAN CEILOMETER  https://review.openstack.org/47567006:56
*** jamesbenson has joined #openstack-kolla06:59
*** salv-orl_ has quit IRC07:01
*** salv-orlando has joined #openstack-kolla07:02
*** jamesbenson has quit IRC07:04
*** salv-orlando has quit IRC07:07
*** salv-orlando has joined #openstack-kolla07:09
*** manheim has joined #openstack-kolla07:11
*** rmart04 has joined #openstack-kolla07:15
*** manheim has quit IRC07:17
*** manheim has joined #openstack-kolla07:17
*** huzhengchuan has joined #openstack-kolla07:29
*** dave-mccowan has quit IRC07:30
*** sbezverk has quit IRC07:31
*** manheim has quit IRC07:32
*** pcaruana|afk| has joined #openstack-kolla07:35
*** pcaruana|afk| has quit IRC07:35
*** pcaruana has quit IRC07:36
*** dave-mccowan has joined #openstack-kolla07:36
spsuryamasuberu: yes there a known problem with elasticseach https://github.com/elastic/elasticsearch/issues/1220807:39
spsuryaissue is still in new state07:40
spsuryanot so sure whether your problem related to this or not07:40
spsuryalogs may help into that07:40
*** pcaruana has joined #openstack-kolla07:40
*** reidrac has joined #openstack-kolla07:42
*** egonzalez has joined #openstack-kolla07:43
openstackgerritLi Yingjun proposed openstack/kolla master: ping is needed for ha router  https://review.openstack.org/47561707:47
*** tovin07_ has quit IRC07:54
spsuryamasuberu: running openstack with kolla-asible from 4 weeks with no issue on memory07:58
spsuryahttps://www.irccloud.com/pastebin/1ZJEODga/07:59
spsuryafor your reference08:00
openstackgerritzhubingbing proposed openstack/kolla master: Alphabetical order install item  https://review.openstack.org/47520508:00
openstackgerritMathias Ewald proposed openstack/kolla master: Fix apache configuration for ceilometer_api  https://review.openstack.org/47569408:03
*** mewald has joined #openstack-kolla08:03
*** manheim has joined #openstack-kolla08:06
*** manheim has quit IRC08:08
*** manheim has joined #openstack-kolla08:09
*** Administrator_ has joined #openstack-kolla08:17
*** zhubingbing_ has quit IRC08:18
*** serlex has joined #openstack-kolla08:20
*** zhugaoxiao has quit IRC08:20
*** mgoddard has joined #openstack-kolla08:20
*** cuongnv has quit IRC08:21
openstackgerritLi Yingjun proposed openstack/kolla-ansible master: Add option ha_vrrp_health_check_interval for l3  https://review.openstack.org/47561408:24
openstackgerritLi Yingjun proposed openstack/kolla master: ping is needed for ha router  https://review.openstack.org/47561708:27
*** openstackgerrit has quit IRC08:48
reidracanyone familiar with on_keystone_name kolla ansible module?08:51
reidracoh, sorry; I meant "on_keystone_role"08:52
*** openstackgerrit has joined #openstack-kolla08:53
openstackgerritLi Yingjun proposed openstack/kolla-ansible master: Add option ha_vrrp_health_check_interval for l3  https://review.openstack.org/47561408:53
*** tvignaud has quit IRC08:55
egonzalezreidrac, with this https://docs.ansible.com/ansible/os_keystone_role_module.html ?08:55
reidracright; thanks egonzalez!08:55
reidracit is failing here randomly; I found that adding a short sleep before it fixes the issue08:56
reidracwhen deploying keysytone08:56
*** zhubingbing_ has joined #openstack-kolla08:57
*** karimb has joined #openstack-kolla08:58
reidracOK, so that ansible module is used to wait until the user role is created08:59
reidracI'll make a bug report, it may be nothing, but can't find what's going on09:00
egonzalezprobably some kind of network issue09:00
reidracegonzalez: in an all-in-one deployment on vagrant?09:00
egonzalezwe had retries before but we removed it, maybe we should recover them https://github.com/openstack/kolla-ansible/commit/38ad05a8729c00ccc012f234135885907065035d09:00
reidraclet me reproduce it so I can tell you the exact error09:02
*** gfidente has joined #openstack-kolla09:04
*** gfidente has quit IRC09:04
*** gfidente has joined #openstack-kolla09:04
*** n0isyn0i1e is now known as n0isyn0ise09:05
*** tvignaud has joined #openstack-kolla09:08
*** salv-orlando has quit IRC09:09
*** salv-orlando has joined #openstack-kolla09:09
*** magicboiz has quit IRC09:09
reidracegonzalez: OK, it fails onn os_keystone_role with "Could not determine a suitable URL for the plugin"; and if I re-run it, it will work fine (so I tried the sleep after the bootstrap)09:11
reidracthis is all-in-one, with Vagrant and libvirt09:11
*** salv-orlando has quit IRC09:13
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add webconsole support in ironic  https://review.openstack.org/47562809:18
*** sambetts_ is now known as sambetts09:19
reidracegonzalez: any suggestion would be appreciated; I can open a bug report, but I'd like to investigate a bit more first09:27
openstackgerritMathias Ewald proposed openstack/kolla master: Fix apache configuration for ceilometer_api  https://review.openstack.org/47569409:28
egonzalezreidrac, I'd try reverting the commit I posted before, I think that have sth to do. Anyway, verify haproxy is working fine and is not any issue with balancing to non working nodes09:32
egonzalezpbourke, around?09:32
reidracegonzalez: good stuff; I'll check both things, thanks!09:33
*** openstackgerrit has quit IRC09:33
*** yingjun has quit IRC09:38
*** shardy has joined #openstack-kolla09:39
*** hrw has quit IRC09:42
reidracegonzalez: haproxy reports on active server and 0 backups servers; so looks like that's OK; I'll try reverting the patch09:44
*** jascott1 has quit IRC09:45
*** jascott1 has joined #openstack-kolla09:46
reidracoh, wait... keystone logs creating def role @ 11:41:55.833 and haproxy repotrs keystone servers up @ 11:41:5709:47
reidracon_keystone_role runs just after the default role has been created, so that's the problem09:48
*** jascott1 has quit IRC09:51
reidracyes, the retries fix the problem; also name of the task "Creating default user role" is somewhat misleading as is not actually doing that but checking that the role was created10:13
*** dims has quit IRC10:16
*** yangyapeng has quit IRC10:16
*** yangyapeng has joined #openstack-kolla10:17
*** pbourke has quit IRC10:17
egonzalezreidrac, the bootstrap creates admin role, Creating default user role: creates _member_ role10:18
reidracOK; thanks10:18
reidracbut is uses the API instead of SQL directly like keystone-manage, isn't it?10:18
reidracthat's why that one fails10:18
egonzalezreidrac, yep10:19
*** pbourke has joined #openstack-kolla10:19
egonzalezis failing because ansible goes faster that keystone and haproxy to be marked as UP10:19
reidracso may be the retries aren't really required all over the place but in this one (at least010:19
reidrac)10:19
egonzalezmaybe a wait for the port to be ready10:19
reidracOK10:20
*** dims has joined #openstack-kolla10:21
*** yangyapeng has quit IRC10:21
*** myoung has quit IRC10:21
*** yangyapeng has joined #openstack-kolla10:22
*** myoung has joined #openstack-kolla10:23
*** yangyapeng has quit IRC10:27
manheimhi guys10:32
manheimthe presentation is up10:32
manheimhttps://www.youtube.com/watch?v=dujGk5FML1410:32
reidracfiled a bug: https://bugs.launchpad.net/kolla-ansible/+bug/169909610:32
openstackLaunchpad bug 1699096 in kolla-ansible "Race condition between haproxy and keystone register task" [Undecided,New]10:32
manheimenjoy and let me know what can I improve10:32
*** rhallisey has quit IRC10:33
reidracI'll work on a fix now10:33
*** caowei_ has joined #openstack-kolla10:36
*** caowei has quit IRC10:36
*** caowei_ is now known as caowei10:36
*** ansmith has quit IRC10:38
*** rstarmer has joined #openstack-kolla10:45
*** jascott1 has joined #openstack-kolla10:46
*** karimb has quit IRC10:50
*** jascott1 has quit IRC10:51
*** karimb has joined #openstack-kolla10:51
*** ansmith has joined #openstack-kolla10:51
*** mewald has left #openstack-kolla11:01
*** janki has quit IRC11:06
*** janki has joined #openstack-kolla11:07
*** ansmith has quit IRC11:11
*** jchhatbar has joined #openstack-kolla11:16
*** janki has quit IRC11:19
*** jchhatbar is now known as janki11:26
*** yangyapeng has joined #openstack-kolla11:28
*** schwicht has joined #openstack-kolla11:28
*** karimb has quit IRC11:31
*** rhallisey has joined #openstack-kolla11:40
*** schwicht has quit IRC11:42
serlexHi, I see 4.0.2 is out. Any special steps to upgrade from 4.0.1?11:49
*** ducttape_ has quit IRC11:56
*** ducttape_ has joined #openstack-kolla11:59
*** openstackgerrit has joined #openstack-kolla12:00
openstackgerritjianyi proposed openstack/kolla-ansible master: all.yml should be in alphabetical order  https://review.openstack.org/45995612:00
*** caowei_ has joined #openstack-kolla12:05
*** caowei has quit IRC12:09
*** caowei_ is now known as caowei12:09
zhubingbing_Hi guys12:10
*** tumble has joined #openstack-kolla12:12
egonzalezsup zhubingbing_12:14
*** caowei has quit IRC12:14
*** caowei has joined #openstack-kolla12:14
zhubingbing_hi12:14
*** ducttape_ has quit IRC12:15
*** salv-orlando has joined #openstack-kolla12:19
*** trozet has joined #openstack-kolla12:21
*** yingjun has joined #openstack-kolla12:30
openstackgerritLi Yingjun proposed openstack/kolla master: ping is needed for ha router  https://review.openstack.org/47561712:35
*** kbaegis has joined #openstack-kolla12:37
*** bmace has quit IRC12:40
*** salv-orlando has quit IRC12:52
*** salv-orlando has joined #openstack-kolla12:52
*** janki has quit IRC12:52
*** bmace has joined #openstack-kolla12:55
-openstackstatus- NOTICE: restarting gerrit to address slowdown issues12:56
*** rwallner has joined #openstack-kolla12:56
*** salv-orlando has quit IRC12:56
*** rwallner has quit IRC12:58
*** rwallner has joined #openstack-kolla12:58
*** lucasxu has joined #openstack-kolla13:00
*** zhubingbing_ has quit IRC13:02
*** awebber has joined #openstack-kolla13:02
sdakemanheim fwiw fuel is dead - ref mailing list13:07
*** jtriley has joined #openstack-kolla13:08
*** lpetrut has joined #openstack-kolla13:09
*** karimb has joined #openstack-kolla13:10
manheimyep13:10
manheimit was just an example13:10
manheimplease let me know what can be impproved13:11
*** schwicht has joined #openstack-kolla13:12
*** goldyfruit has joined #openstack-kolla13:12
*** salv-orlando has joined #openstack-kolla13:12
*** caowei_ has joined #openstack-kolla13:16
*** eanylin_ has quit IRC13:18
*** jtriley has quit IRC13:19
*** caowei has quit IRC13:19
*** caowei_ is now known as caowei13:19
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Fix the database user name for all roles  https://review.openstack.org/47525813:19
sdakemanheim mechanism drive exqmaple, /etc/kolla/nova/config/nova.conf13:21
sdakeis how you override rather then modifyign /etc/kolla/nofa.conf13:21
sdakelooks good tho ;)13:21
manheimty13:22
sdakeliberty was 1.1.0 as well - we killed 1.0.0 because it had a fatal issue with storage13:27
*** jrist_ is now known as jrist13:31
*** jrist has quit IRC13:31
*** jrist has joined #openstack-kolla13:31
*** caowei has quit IRC13:33
*** ducttape_ has joined #openstack-kolla13:33
*** yingjun has quit IRC13:34
*** eanylin_ has joined #openstack-kolla13:35
*** yingjun has joined #openstack-kolla13:38
*** ducttape_ has quit IRC13:38
*** kbaegis has quit IRC13:39
*** kbaegis has joined #openstack-kolla13:39
*** wahnsinn has joined #openstack-kolla13:41
*** kbaegis1 has joined #openstack-kolla13:41
openstackgerritDave McCowan proposed openstack/kolla-kubernetes master: [WIP] Add support for HTTPS on external API network  https://review.openstack.org/47199213:42
*** kbaegis1 has quit IRC13:44
*** kbaegis has quit IRC13:45
*** kbaegis has joined #openstack-kolla13:46
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add warning while upgrading with dockerhub images N->O  https://review.openstack.org/47580213:49
dmsimardsdake: do you guys have a registry inside infra yet ?13:50
sdakedmsimard no13:50
dmsimard:(13:50
dmsimardI stood up an openshift registry for RDO recently, it was a wild ride13:51
*** spsurya has quit IRC13:55
rwellumhey sdake - I'm attending kubernetes training hosted by Cisco today. :)13:55
sdakerwellum fascinating13:56
sdakewho is providing it13:56
rwellumApprenda + cisco13:56
*** karimb has quit IRC13:58
*** skramaja has quit IRC13:59
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add warning while upgrading with dockerhub images N->O  https://review.openstack.org/47580214:00
reidracegonzalez: wiating for the port doesn't work; haproxy is always bound, but the backend isn't available, I'll go with retry unless you have a better idea14:00
egonzalezreidrac, retry is fine14:00
reidracegonzalez: cheers!14:01
*** Administrator_ has quit IRC14:01
*** Administrator_ has joined #openstack-kolla14:02
*** srwilkers_ has joined #openstack-kolla14:02
*** mickt has joined #openstack-kolla14:09
*** mickt has left #openstack-kolla14:09
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add warning while upgrading with dockerhub images N->O  https://review.openstack.org/47580214:10
*** karimb has joined #openstack-kolla14:10
*** magicboiz has joined #openstack-kolla14:12
*** tumble has quit IRC14:12
wahnsinnhey, I have problems pushing images to a private registry: "ERROR:kolla.image.build.base:open /dev/mapper/docker-254:3-3605282-0d1f95af4ae76c1c535e296b64838d7a58a7700f980f7978542085cf6cb0a52a: no such file or directory"14:12
wahnsinnneither gitlab registry nor a registry deployed with the start-registry script is working14:13
wahnsinndoes somebody have an idea where I should look for the problem?14:14
sdakerwellum not familiar with apprenda14:14
wahnsinnthe weird thing is that some images were pushed sometimes14:14
sdakewahnsinn my first recommendation would be to not use devicemapper14:15
sdakedocker themselves says "not supported, deprecated"14:15
sdakewahnsinn my second recommdnation would be to hceck disk space14:16
*** magicboiz has quit IRC14:16
sdakeeven though your system may say 30-40% free, devicemapper actually uses more of the storage than df reports, resulting in all sorts of errors14:16
*** huzhengchuan has quit IRC14:17
wahnsinnthanks, I'll have a look14:17
rwellumsdake: they and cisco have a long running partnership (apparently)14:18
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add warning while upgrading with dockerhub images N->O  https://review.openstack.org/47580214:20
*** magicboiz has joined #openstack-kolla14:21
*** salv-orlando has quit IRC14:24
*** tumble has joined #openstack-kolla14:24
openstackgerritJuan J. Martínez proposed openstack/kolla-ansible master: Retry Keystone's default user role creation  https://review.openstack.org/47582014:25
egonzalezsdake, mind taking a look at https://review.openstack.org/#/c/475802/14:26
egonzalezquite important warning imo, here is the rendered release note http://docs-draft.openstack.org/02/475802/4/check/gate-kolla-ansible-releasenotes/49c598c//releasenotes/build/html/unreleased.html#id114:26
*** ducttape_ has joined #openstack-kolla14:27
*** salv-orlando has joined #openstack-kolla14:28
egonzalezsdake, note is a change directly to stable/ocata release notes14:28
*** salv-orlando has quit IRC14:30
*** yangyape_ has joined #openstack-kolla14:30
*** salv-orlando has joined #openstack-kolla14:30
*** yangyapeng has quit IRC14:33
*** jmccarthy has joined #openstack-kolla14:37
jmccarthyJust checking, any cores around who can bump these along ?14:38
jmccarthyhttps://review.openstack.org/#/c/469511Add possibility to configure tenant network types and type drivers14:38
jmccarthyhttps://review.openstack.org/#/c/464675Add possibility to configure tenant and provider VLANs14:38
openstackgerritcaoyuan proposed openstack/kolla master: Update the "prechecks" to "precheck"  https://review.openstack.org/47582714:38
*** ducttape_ has quit IRC14:39
*** ducttape_ has joined #openstack-kolla14:39
*** dciabrin has quit IRC14:40
*** yingjun has quit IRC14:40
openstackgerritcaoyuan proposed openstack/kolla master: Update the "prechecks1" to "prechecks"  https://review.openstack.org/47582714:40
*** jamesbenson has joined #openstack-kolla14:46
*** tovin07 has joined #openstack-kolla14:52
*** dciabrin has joined #openstack-kolla14:54
*** rhallisey is now known as rhallisey-brb14:57
*** rhallisey-brb is now known as rhallisey-away14:57
*** spsurya has joined #openstack-kolla14:57
jamesbensonmorning all14:58
manheimhi14:58
spsuryamorning jamesbenson manheim15:04
*** hrw has joined #openstack-kolla15:04
spsuryahrw sup15:05
jamesbensonmorning spsurya15:05
*** sbezverk has joined #openstack-kolla15:08
*** rwallner has quit IRC15:10
*** rmart04 has quit IRC15:14
*** ducttape_ has quit IRC15:20
*** schwicht has quit IRC15:21
*** ducttape_ has joined #openstack-kolla15:21
*** rstarmer has quit IRC15:22
*** rstarmer has joined #openstack-kolla15:23
*** rwallner has joined #openstack-kolla15:25
inc0good morning everyone15:30
inc0egonzalez: hey, https://review.openstack.org/#/c/466815/17 I think it's ready15:31
inc0answering your question:)15:31
egonzalezinc0, cool15:31
egonzalezmind checking  https://review.openstack.org/#/c/475802/ ? all gates are red because we have not merge any image with kolla 4.0.3 yet15:32
inc0hmm, ok yeah15:32
inc0we need to do it, there is no easy way to manually trigger15:32
inc0I'll propose fake change, merge it and rollback afterwards15:33
*** karimb has quit IRC15:33
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla stable/ocata: Fake change to kickoff image publishing  https://review.openstack.org/47584915:34
inc0done15:35
inc0I'll remove it when images are published15:35
openstackgerritVladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant network types and type drivers  https://review.openstack.org/46951115:39
openstackgerritVladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant network types and type drivers  https://review.openstack.org/46951115:39
*** rstarmer_ has joined #openstack-kolla15:40
*** rstarmer has quit IRC15:42
*** awebber has quit IRC15:43
kfox1111weird...15:46
kfox1111k8s is moving so fast now they are releasing alpha's for 1.8 while 1.7 isn't even released.15:46
jamesbensoninc0, are the gates why I can't build the docker images?15:46
inc0jamesbenson: no, building locally should just work15:47
kfox1111I wonder if that was a mistake...15:47
srwilkers_kfox1111, really? damn, thats crazy15:48
inc0kfox1111: yeah, I'd bet...15:48
inc0probably typo15:48
inc0or something as trivial15:48
inc0srwilkers_: what's up man?15:48
srwilkers_inc0, hey dude15:49
kfox1111nice. finally. https://github.com/kubernetes/kubernetes/pull/46076 node secret authorization.15:49
kfox1111that will help security significantly.15:49
kfox1111yeah. that feature is targeted for 1.7 but was in the 1.8alpha1 notes.15:51
kfox1111I bet the release should have been another 1.7 alpha but something slipped.15:51
*** manheim has quit IRC15:57
*** manheim has joined #openstack-kolla15:57
*** dciabrin has quit IRC16:00
*** egonzalez has quit IRC16:01
*** manheim has quit IRC16:02
sdakekfox1111 question - i have a cratered multlinode deployment of kubernetes - mariadb mounted via nfs16:03
kfox1111ok.16:03
sdakei suspect two copies of mariadb are running even though only one copy is reported running by kubernetes16:04
jamesbensoninc0: thanks, I figured out the issue....16:04
sdakeis there any way to validate or invalidate this theory that you are aware of?16:04
kfox1111sdake: possible.16:04
sdakei am getting an errror no 11 on read of ibdata16:04
jamesbensoninc0: I'm building the 4.0.2 images (pip install kolla==4.0.2 kolla-ansible==4.0.2) and going to try to deploy on freshly reimaged systems.  I'll let you know if I have ceph issues still... I'm hoping not..16:04
kfox1111docker ps on all hosts or ps ax | grep mariadb on all hosts.16:04
sdakeps -ax does not seem to return any results16:04
sdakelet me check out docker ps16:05
jamesbensonDo have one question though, how come a lot of the monitoring tools are default to controller and not monitoring node?16:05
kfox1111ther is only one running?16:05
kfox1111(Not sure if mariadb on nfs is really supported or not..)16:05
inc0jamesbenson: our monitoring was never finished16:05
inc0you can try it out, but can't vouch for it16:05
*** jascott1 has joined #openstack-kolla16:07
sdakekfox1111 i hear ya re support - apparently docs say it works16:07
*** JamesBenson-mobi has joined #openstack-kolla16:07
sdakekfox1111 it is definately having a problem with FCNTL on ibdata116:08
kfox1111yeah, thats a locking primitive I think.16:08
kfox1111nfs can support locking but tricky to get right.16:08
kfox1111fyi, I think your playing with fire.16:09
jamesbensonah okay, what things aren't complete?16:11
*** dciabrin has joined #openstack-kolla16:13
sdakekfox1111 i think i am plying with TNT actually16:13
sdakeknowing that - i am proceeding ahead :)16:13
inc0sdake: I have something cool to show ;) later this day16:14
kfox1111ok. fairly warned ye be says I. :)16:14
sdakekfox1111 ya - i'm gonna say I told ya so too... :)16:14
*** tovin07 has left #openstack-kolla16:15
sdakekfox1111 ya I know all about fcntl16:15
sdakekfox1111 i'm curious if you have any other suggestions for what may be the root cause (of a system you know nohting about in detail:)16:15
kfox1111if I had to guess, I'd say you need to setup nfs locking on the client/server.16:16
kfox1111lockd I think it is called.16:16
kfox1111(been a long time since I've played with it)16:16
kfox1111or, I think it may be built into v4.16:17
*** sudipto has joined #openstack-kolla16:17
*** sudipto_ has joined #openstack-kolla16:17
*** pcaruana has quit IRC16:19
*** unicell has joined #openstack-kolla16:21
*** unicell1 has quit IRC16:23
*** JamesBenson|phon has joined #openstack-kolla16:26
*** JamesBenson-mobi has quit IRC16:26
wahnsinnsdake : thanks for the tip - it's working fine now16:31
*** unicell has quit IRC16:36
*** schwicht has joined #openstack-kolla16:36
*** JamesBenson-mobi has joined #openstack-kolla16:37
*** JamesBenson|phon has quit IRC16:37
*** JamesBenson|phon has joined #openstack-kolla16:38
*** lpetrut has quit IRC16:40
*** jmccarthy has left #openstack-kolla16:41
*** JamesBenson-mobi has quit IRC16:42
*** SamYaple has quit IRC16:44
*** reidrac has quit IRC16:44
*** SamYaple has joined #openstack-kolla16:44
*** SamYaple has quit IRC16:44
*** SamYaple has joined #openstack-kolla16:44
*** sambetts is now known as sambetts|afk16:45
*** gfidente is now known as gfidente|afk16:47
*** sbezverk has quit IRC16:51
*** itlinux has joined #openstack-kolla16:52
*** itlinux_ has joined #openstack-kolla16:52
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP Dockerfile to run ansible orchiestration  https://review.openstack.org/47477016:57
inc0kfox1111: sdake ^ if we do this we don't need to create configmaps manually any more16:59
*** ducttape_ has quit IRC17:02
*** kbaegis has quit IRC17:05
*** manheim has joined #openstack-kolla17:08
kfox1111looking...17:09
*** mgoddard has quit IRC17:09
*** manheim has quit IRC17:12
*** manheim has joined #openstack-kolla17:12
kfox1111inc0: generally +1. a bunch of comments inline.17:18
inc0yeah it's wip17:19
inc0I'm not sure about having more than one distro on container side tbh17:20
*** kbaegis has joined #openstack-kolla17:21
*** unicell has joined #openstack-kolla17:25
kfox1111your favorite's ubuntu. others are centos, etc.17:25
kfox1111easier adoption if you support what peples favorites are.17:25
kfox1111kolla already has pretty much all the plumbing.17:26
inc0yes, but maknig kolla built container with that is not going to be easy17:26
inc0and not that much value17:26
inc0imho17:26
inc0I mean it's short lived17:26
inc0I'd go for alpine if I would have time for figuring it out17:27
kfox1111why is it hard? its just a couple of statically linked go binaries and ansible?17:28
kfox1111plus, you could use distro packages for ansible, making it even easier.17:28
inc0kfox1111: putting kolla-k8s inside is harder17:29
kfox1111we dont' use any fancy newfangled ansible stuff.17:29
inc0ADD ansible17:29
inc0that's what's hard17:29
kfox1111why is that hard with centos?17:29
inc0it's not harder with centos17:29
inc0it's hard with kolla - kolla assumes Dockerfile.j2 being in Kolla17:29
inc0and Kolla doesn't have access to Kolla-k8s code17:30
kfox1111ah.17:30
kfox1111is that exteran repo ps far out then?17:30
inc0this is just plain Dockerfile without templating17:30
kfox1111external17:30
inc0dockerhub you mean?17:30
kfox1111no. the exteral to kolla, kolla build support.17:30
inc0well it's coming, but this will be lots of added complication imho17:31
inc0for little benefit17:31
kfox1111its complicated when there are more then one way to do things.17:31
kfox1111"build all the things" and then build this other thing.17:31
inc0we can do busybox/alpine and just keep it native Dockerifle17:31
inc0don17:31
kfox1111I don't think this will be the only container specific to kolla-kubernetes either.17:32
inc0don't forget that we'll have kolla docker container in dockerhub17:32
kfox1111so cant be just /Dockerfile17:32
openstackgerritMerged openstack/kolla stable/ocata: Fake change to kickoff image publishing  https://review.openstack.org/47584917:32
kfox1111I'm not forgetting.17:32
inc0so far it's only one having kolla-k8s code in it17:32
kfox1111only becase we've successfully landed some other kolla-k8s specific contaienrs in kolla.17:32
kfox1111this is the first one we havent tried.17:32
kfox1111if kolla-kubernetes is to get containers, the others probably belong here too.17:33
inc0yeah I agree17:33
inc0but then we will have harder time publishing them17:33
inc0since I build publisher pipeline for kolla17:33
inc0adding kolla-k8s is going to be...trickier17:33
kfox1111yeah. but I think its going to be important.17:33
inc0agree17:34
kfox1111how far out do you think that other ps is?17:34
inc0it's orthogonal17:34
kfox1111I think its maybe a prereq. depends just how far out it is though.17:35
inc0but well, still pretty far, not sure what bradjones has locally17:35
kfox1111if its a week or two, probably best to finish it first.17:35
serlexkolla-ansible upgrade seems to be stuck on "Sighup nova-compute" anyone seen that before?17:35
inc0they can go together, after it merges we'll just add new job17:36
inc0serlex: no...17:36
kfox1111I guess I'd be ok with the dockerfile as is provided a techdebt bug is filed and the other ps is being worked on.17:36
inc0you can try to restart container17:36
serlexok.An hour in. Ctrl+C rerun upgrade?17:36
serlexok17:37
serlexrestarting container didnt do much, I will try rerunning upgrade17:38
inc0kk17:39
*** sudipto has quit IRC17:40
*** sudipto_ has quit IRC17:40
inc0serlex: all nodes hung?17:41
inc0or some succeed?17:41
*** srwilkers_ has quit IRC17:43
serlexIt hang on "Sighup nova-compute" after completing on 4 out of the 5 nodes17:51
inc0anything special on 5th node?17:51
serlexI cancelled it ran it again, finished pretty quick. should all out containers be restarted or relevant ones?17:52
serlex5th is just another compute node, 4th was compute node, seem to have ran with "changed" output17:52
inc0if it's finished it's done, restarting is built into play17:52
inc0we added sighub instead of restart to avoid well..restarts17:52
serlexi see17:53
serlexany way I can confirm successful upgrade?17:53
serlexlook through the commits i guess?17:53
inc0check images17:53
serlexsure17:53
inc0if it's tagged with 4.0.2 then it's upgraded17:53
inc0docker ps should tell you that17:53
serlexchecking...17:57
*** JamesBenson|phon has quit IRC17:59
*** JamesBenson-mobi has joined #openstack-kolla17:59
*** athomas has quit IRC17:59
*** ducttape_ has joined #openstack-kolla18:01
*** wahnsinn has quit IRC18:02
*** rwsu has quit IRC18:03
*** JamesBenson-mobi has quit IRC18:05
*** MasterOfBugs has joined #openstack-kolla18:06
openstackgerritAndy Smith proposed openstack/kolla-ansible master: Add qdrouterd role for messaging infrastructure component  https://review.openstack.org/46946218:13
*** serlex has quit IRC18:13
*** TxGirlGeek has joined #openstack-kolla18:15
*** dciabrin has quit IRC18:16
*** tonanhngo has joined #openstack-kolla18:24
*** tonanhngo has quit IRC18:25
*** tonanhngo has joined #openstack-kolla18:26
*** iniazi_ has joined #openstack-kolla18:28
*** ducttape_ has quit IRC18:29
*** iniazi has quit IRC18:29
*** dciabrin has joined #openstack-kolla18:31
*** klindgren has quit IRC18:33
*** sbezverk has joined #openstack-kolla18:33
inc0kfox1111: did you try to run built helm repo on k8s?18:35
*** ducttape_ has joined #openstack-kolla18:36
kfox1111a little ambigious18:36
kfox1111you talking about the helm repo microservice chart?18:36
inc0yes18:36
kfox1111I have not tested it. I think only sbezverk has.18:37
inc0use my (or similar) container to build and host repo as job18:37
inc0or actally even better I guess - pv18:37
kfox1111its a service.18:37
inc0kk18:37
kfox1111so deployment.18:37
inc0then a service18:37
inc0well yeah18:37
sbezverkinc0: yes I have and we have chart for it and it starts in the gate workflow18:37
*** salv-orlando has quit IRC18:38
inc0btw sbezverk https://review.openstack.org/#/c/474770/ besides distro, how do you like idea?18:38
*** salv-orlando has joined #openstack-kolla18:39
sbezverkinc0: I have huge outage to deal, I will try to find some time later today.. but currently I spend 24 hours per day on con calls18:40
inc0sure18:40
inc0good luck18:40
*** rstarmer has joined #openstack-kolla18:41
*** pradk has joined #openstack-kolla18:41
rstarmerstill trying with ceph, looks like there's something not right when I deploy with erasure coding, but replication works.18:41
jamesbensoninc0: what version of python do we need for kolla?  genpwd is yelling at me...18:42
inc02.718:42
jamesbensonodd...18:42
inc0what's the yelling about?18:42
rstarmerbigger question: my mon/osd/pools all look happy, cinder is happy, volumes "act" right, but systems keep booting against a blank volume.  Anyone gotten ceph working via kolla-ansible?18:42
inc0rstarmer: hmm I don't know anything about erasure conding unfortunately:/18:42
inc0I have18:43
inc0rstarmer: but boots work in general?18:43
jamesbensonhttps://pastebin.com/DURaZvQd18:43
inc0because glance and nova also uses ceph18:43
inc0so if no-volume ephemerals work, that means ceph is ok18:43
rstarmerI'm less worried about erasure coding, woudl like but not critical.  But no, boot doesnt/  The logs all look "right", but in the end the vms boot with "no bootable device"18:44
jamesbensonrstarmer, I've got ceph working a few times, but always seems to be a rough spot for me.18:44
inc0jamesbenson: did you copy empty passwords.yml to /etc/kolla?18:44
jamesbensonyep18:44
rstarmerinc0: yeah, I think Ceph is happy.  Cinder seems happy talking to ceph, cinder and nova talk, but ther's agap somewhere.18:44
inc0rstarmer: glance add image works?18:45
jamesbensoninc0: there and empty...18:45
inc0jamesbenson: readable too?18:45
rstarmeryes, image in glance.  cinder volume log shows cinder grabing the image from a rbd target, and asssociating it with the volume id that is then mapped to the VM18:45
jamesbenson064418:46
inc0so, if you try to boot with ephemeral it's still bad right18:46
inc0?18:46
jamesbensonreadable by the world18:46
inc0(no cinder involved)18:46
rstarmerhmm, you know, I didn't try that, let me :D18:46
inc0jamesbenson: duh...wierd indeed...try to debug python code18:46
inc0rstarmer: fyi that will also use ceph18:47
inc0ephemerals lands on ceph as well18:47
rstarmerwill try now, and let you know18:47
inc0after that try to manually mount rbd with glance image to see if it's good18:47
inc0maybe something fails at initalizing volume from image18:47
jamesbensonI am using pip 4.0.2 for both kolla and kolla-ansible.18:48
inc0duh...we test that in gates and is consistently good...18:48
rstarmerok, so ephemeral also ends up with no bootable device18:48
jamesbensonI figured, sort of a core/basic thing.18:49
rstarmeris there a pointer to the manual glance/rbd mount? I've never done that18:49
inc0http://docs.ceph.com/docs/master/start/quick-rbd/18:49
*** ducttape_ has quit IRC18:59
*** dciabrin has quit IRC19:00
rstarmerFound it.  wrong image file type.  Selected 'raw' instead of qcow2...19:00
inc0happened to me more than once19:00
rstarmerall is right with the world.  Now, to figure out erasure coding, and how to separate our "big" storage from our "little" storage.  One for glance/backups/rgw and one for volumes...19:01
*** rwallner has quit IRC19:03
*** itoperatorguy has joined #openstack-kolla19:04
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla master: WIP: Dockerfile publishing pipeline  https://review.openstack.org/47546919:11
itoperatorguyhi all. I am trying to install kolla. And during deploy I get error messeges about missing docker images. for example the kolla-ansible try to use fluentd:4.0.2 and toolbox:4.0.2 but they does not exists on hub.docker.com. What am I missing?19:13
*** itlinux_ has quit IRC19:13
*** itlinux has quit IRC19:13
inc0itoperatorguy: we haven't pushed images yet, sorry19:13
inc0it's on us19:13
inc0I encourage you to build them19:13
jamesbensonyou'll have to build them yourself itoperatorguy19:13
inc0we are working on automation for timely pushes but in general I discourage using dockerhub images before that19:14
inc0because these images are stale and updates haven't been included19:14
itoperatorguythis is the right way to do it? "./build.py -b centos --registry kolla:5000 --push"19:15
inc0well if you made pip install kolla19:15
inc0then you have kolla-build19:15
inc0but yeah that's that it means19:15
itoperatorguy"kolla" is the hostname of my deployment mashine where I previously run ./start-registry19:15
inc0yeah that works19:15
inc0then in globals.yml in "docker_registry" specify kolla:500019:16
inc0and it'll work19:16
*** rhallisey-away is now known as rhallisey19:16
itoperatorguywell then I am still in trouble because I did. I started the registry, I run the push command above. I run the kolla-ansible prechecks then I wanted to do a deploy and I got the error messeges that I have missing docker images.19:17
inc0what is exact error you have?19:18
itoperatorguyfirst for fluentd. I checked on docker hub and yes there is no 4.0.2 tag only 4.0.0 so I pulled it retagged it and pushed it to my local repo as a workaround. I run the deploy agan and now I have same issue with toolbox.19:19
itoperatorguyfatal: [network1]: FAILED! => {"changed": false, "failed": true, "msg": "The requested image does not exist: kolla:5000/kolla/centos-binary-kolla-toolbox:4.0.2"}19:19
inc0strange, did you build and push all of them?19:19
inc0ahh19:19
inc0you pulled and retagged19:19
inc0yeah, well, that won't work19:20
inc0I mena it will19:20
inc0but will require you to do all manual19:20
inc0try just doing pip install -U kolla19:20
inc0and kolla-build --registry kolla:5000 --push -b centos -t binary19:20
inc0it will build and tag images to 4.0.2 automatically19:21
inc0if you're hell bend on not building19:21
inc0just add openstack_release: "4.0.0" to globals.yml19:21
inc0and then you cna pull from dockerhub19:21
inc0but again, old, not updated images so good for exploration but not prod19:21
itoperatorguyif I run "docker images | wc -l" my results are 204 and if I run just run docker images then I see that most of them are tagged with "5.0.0"19:23
inc0if you checkout master it will be 5.x19:23
inc0as in Pike19:23
inc05.x is development now19:23
jamesbensoninc0: wierd... my passwords file was really empty.... not sure how that happened...19:23
inc0a yeah19:23
jamesbensonfixed now19:23
inc0well19:23
itoperatorguywhat should be the latest stable tag?19:24
inc04.0.219:24
inc0if you do git checkout stable/ocata19:24
inc0that will be one you'll get19:25
inc0or rather19:25
inc04.0.3 is what you get because it's always stable tag+next minor19:25
itoperatorguyso I need something like this? "./build.py -b centos --registry kolla:5000 --tag 4.0.3 --push"   + openstack_release: "4.0.3" to globals.yml ?19:27
inc0itoperatorguy: you have code from github?19:27
inc0then if you want ocata (stable19:27
inc0you need to run git checkout stable/ocata19:27
inc0to swithc to stable branch19:28
inc0after that run pip install -e -U .19:28
inc0(from kolla dir)19:28
inc0and kolla-build -b centos --registry kolla:5000 --push19:28
inc0no need to specify tag19:28
itoperatorguyok which repo? Because I see to on the official documentation (BTW what is the difference)? kolla or kolla-ansible?19:30
itoperatorguyprobably my English is the "barrier factor" but I can not really get the difference19:33
*** ducttape_ has joined #openstack-kolla19:33
*** ducttape_ has quit IRC19:34
*** ducttape_ has joined #openstack-kolla19:35
*** spsurya has quit IRC19:35
inc0itoperatorguy: kolla - building images19:36
inc0kolla-ansible - deploying images19:36
inc0when you're use this method, your deployment is split into two steps19:36
inc01. building set of images and populating your registry19:37
inc0that's Kolla's job19:37
inc0after that you run ansible-playbooks which downloads these images, prepares config and runs them in correct way19:37
inc0that's kolla-ansible19:37
*** manheim has quit IRC19:37
itoperatorguyinc0: ok it makes much more sense. Thanks. I see the light now :D19:37
inc0haha19:37
inc0yeah, let me know if you have any further questions19:38
*** mgkwill_ is now known as mgkwill19:46
Reepicheepwhat is the proper way when adding new nodes to an existing cluster to bootstrap-server on the new nodes?19:57
Reepicheepif I add it to my inventory and run bootstrap-server it is restarting docker on all my nodes which breaks mariadb19:58
*** rwallner has joined #openstack-kolla19:59
*** rstarmer has quit IRC20:03
*** rstarmer has joined #openstack-kolla20:11
*** sgrasley has joined #openstack-kolla20:12
inc0Reepicheep: you can just make temp inventory for that20:13
inc0(but it's bug)20:13
inc0temporarily specify [baremetal] group20:13
inc0or you can use ansible --limit20:13
*** kbaegis has quit IRC20:15
*** jascott1 has quit IRC20:15
*** jascott1 has joined #openstack-kolla20:16
*** kbaegis has joined #openstack-kolla20:16
*** jascott1 has quit IRC20:17
Reepicheepinc0: thanks that is what I was gonig to do then handle the host files elseware20:19
inc0deploy is idempotent so it won't break20:19
inc0however bootstrap servers seems to have issue there20:19
inc0but yeah, hostfiles20:20
inc0damn20:20
inc0well, I'll fix it quickly20:20
*** kbaegis has quit IRC20:20
Reepicheepinc0: lol, I can confirm there are issues .. it stops all the mariadb containers at once .. which leaves the galera in a unkown state with no host "safe_to_bootstrap" and all "seqno" as -120:22
*** kbaegis has joined #openstack-kolla20:23
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Fix baremetal role idempotency  https://review.openstack.org/47591620:24
inc0Reepicheep: ^20:24
inc0that patch should fix it20:24
Reepicheepinc0: thanks.. I will give it a try20:26
jamesbensoninc0: clean system and having issues with ceph : Bootstrapping Ceph OSDs.  Fails here. was this related to the ceph issues yesterday we were stuck on?20:35
inc0show me log please20:36
Reepicheepinc0: that worked on stable/ocata thanks again20:39
ReepicheepI'm guessing that patch will not be backported so I can just track it localy for now, correct?20:39
*** shardy has quit IRC20:40
inc0it actually might20:40
inc0mind submitting bug?20:40
inc0(or I can do it)20:41
ReepicheepI can do that20:41
inc0if we tag this to be backported, it will20:41
inc0and should be released in 4.0.320:41
inc0thanks20:41
*** goldyfruit has quit IRC20:43
jamesbensoninc0: here's the ansible output https://pastebin.com/TSQy9RUc20:49
jamesbensoninc0: docker logs bootstrap_osd_0 and ceph_mon20:50
jamesbensonhttps://pastebin.com/ntD7gAwe20:50
inc0jamesbenson: whenever you see "container returned non-zero code" you should check docker logs <<container>>20:50
inc0yeah20:50
jamesbensonand I'm getting the others20:50
inc0check /var/lib/docker/volumes/kolla_logs/20:50
inc0for ceph mon and ceph osd20:51
inc0logs20:51
jamesbensoninc0: https://pastebin.com/mL9Sgk1r for ceph-mon20:51
inc0yeah I saw similar thing locally:/ no idea what's happening20:52
jamesbensoninc0: ceph_client: https://pastebin.com/Qz1Tk5Rr20:52
inc0I saw something like this when ntp wasn't synchronized20:52
*** manheim has joined #openstack-kolla20:52
jamesbensonthey seem to be all synced...at least ntpd is running and pointing to a local ntp server.20:52
inc0yeah20:52
inc0I couldn't figure it out few days ago too:/20:53
jamesbensonyeah, no idea here either....20:53
jamesbensonI know it was working a few days ago (before I did some kolla/kolla-ansible updates20:53
Reepicheepinc0: https://bugs.launchpad.net/kolla/+bug/169933520:54
openstackLaunchpad bug 1699335 in kolla "running bootstrap-servers on an existing install breaks mariadb" [Undecided,New]20:54
jamesbensonbut looking at the changelogs, I couldn't find anything that should have impacted it.20:54
inc0jamesbenson: I would say this can be ceph issue20:54
inc0that wouldn't be reflected in changelogs20:54
jamesbensonis stuff pulled directly from ceph's repo?20:55
inc0no20:55
inc0apt/yum installed20:55
jamesbensonoh...20:55
jamesbensonand it's not marked to a specific version?20:55
inc0it's marked to major release (as in jewel)20:56
inc0but I don't know if it's issue20:56
inc0how many control nodes do you have?20:56
inc03?20:56
*** manheim has quit IRC20:56
jamesbensonyeah20:56
jamesbenson320:56
inc0no idea:/21:00
jamesbensonnow I understand why ceph is such a pain though, if we are just apt it from somewhere else ...21:01
jamesbensoninc0: where is that in kolla's repo?21:02
*** manheim has joined #openstack-kolla21:02
inc0ceph installation?21:02
jamesbensonyeah, the apt21:02
inc0https://github.com/openstack/kolla/blob/master/docker/ceph/ceph-base/Dockerfile.j2 here21:03
inc0and repos are added here https://github.com/openstack/kolla/blob/master/docker/base/sources.list.ubuntu21:04
inc0hmm interesting, we don't pin to jewel in ubuntu it seems21:05
jamesbensonyeah, wanted to check these things..21:05
jamesbensonthat might be the issue then, no?21:05
*** lucasxu has quit IRC21:05
inc0seems like it, I was pinning this but that was long ago21:06
inc0however still unlikely because we're using native ubuntu21:06
inc0so maintainers are supposed to provice these things21:06
inc0and also gates uses latest images and theyre just fine21:06
*** rwallner has quit IRC21:06
*** rwallner has joined #openstack-kolla21:07
jamesbensonstill worth a try...21:08
inc0yeah we removed pin as ubuntu cloud archive provices jewel21:08
*** rhallisey has quit IRC21:10
*** rwallner has quit IRC21:11
jamesbensonso do we just need to add in the ceph repo here:https://github.com/openstack/kolla/blob/stable/ocata/docker/base/sources.list.ubuntu21:13
jamesbensonI'm looking at your commit back in Sept 6 2016, removed the version lock because it was only used in cent, not ubuntu and risky if we miss a security patch on ceph.....21:14
jamesbensonhttps://github.com/openstack/kolla/commit/0a50cb8889f245bc89b49fbb2dacb9c02ded5f8821:14
*** salv-orl_ has joined #openstack-kolla21:17
*** jbadiapa_ has quit IRC21:19
*** salv-orlando has quit IRC21:20
*** gfidente|afk has quit IRC21:21
jamesbensonhas there been much luck with ceph-ansible, I remember seeing people talk about it in the past....21:25
*** dciabrin has joined #openstack-kolla21:27
*** rstarmer has quit IRC21:30
inc0well they're going to be better at ceph deploy than we will21:33
openstackgerritEric Peterson proposed openstack/kolla-kubernetes master: Add missing setting for novncproxy_base_url  https://review.openstack.org/47593321:33
*** dave-mccowan has quit IRC21:35
*** schwicht has quit IRC21:35
*** Mannheim has joined #openstack-kolla21:35
*** pramodrj07 has joined #openstack-kolla21:37
*** MasterOfBugs has quit IRC21:40
*** kong has left #openstack-kolla21:40
*** rwallner has joined #openstack-kolla21:41
*** rwallner_ has joined #openstack-kolla21:43
*** manheim has quit IRC21:43
*** Mannheim has quit IRC21:44
*** rwallner has quit IRC21:45
*** dciabrin has quit IRC21:45
*** Manheim has joined #openstack-kolla21:46
*** rwallner_ has quit IRC21:47
jamesbensongot to run inc0, chat more tomorrow :-)21:50
inc0have a good one21:50
jamesbensonyou too21:50
*** jamesbenson has quit IRC21:50
*** Pavo has quit IRC21:50
*** jamesbenson has joined #openstack-kolla21:52
*** calbers has quit IRC21:54
*** jamesbenson has quit IRC21:57
*** jamesbenson has joined #openstack-kolla21:59
*** itoperatorguy has quit IRC21:59
*** omenv has quit IRC22:03
*** TxGirlGeek has quit IRC22:03
*** schwicht has joined #openstack-kolla22:03
*** jamesbenson has quit IRC22:04
*** kolla-slack has quit IRC22:10
*** yee37923 has joined #openstack-kolla22:10
*** yee37923 has quit IRC22:10
*** ducttape_ has quit IRC22:11
*** kolla-slack has joined #openstack-kolla22:11
*** kolla-slack has quit IRC22:12
*** openstack has joined #openstack-kolla22:18
*** jrist has quit IRC22:18
*** kolla-slack1 has quit IRC22:19
*** yee379 has quit IRC22:20
*** jrist has joined #openstack-kolla22:20
*** jrist has quit IRC22:20
*** kolla-slack has joined #openstack-kolla22:20
*** dosaboy has joined #openstack-kolla22:20
*** sklgromek has joined #openstack-kolla22:20
*** yee379 has joined #openstack-kolla22:20
*** jascott1 has joined #openstack-kolla22:20
*** jrist has joined #openstack-kolla22:21
openstackgerritHongbin Lu proposed openstack/kolla-ansible master: Upgrade from docker-py to docker  https://review.openstack.org/47594122:21
*** goldyfruit has joined #openstack-kolla22:23
*** Pavo has joined #openstack-kolla22:27
*** goldyfruit has quit IRC22:27
*** manheim has joined #openstack-kolla22:30
*** jascott1 has quit IRC22:31
sdakenottzomg could it be hotter22:33
sdake120DEGREES F22:33
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla master: WIP: Dockerfile publishing pipeline  https://review.openstack.org/47546922:37
inc0sdake: global warming is real22:39
inc0it'll be close to 100 in Oregon too22:39
inc0and it's Oregon22:39
inc0I moved here because it's colder ;(22:39
*** schwicht has quit IRC22:39
inc0sdake: check this out https://review.openstack.org/#/c/474770/22:40
*** TxGirlGeek has joined #openstack-kolla22:40
inc0kubectl create -f /this_file22:40
inc0and it will create all configs and secrets22:40
inc0(when I'm dnoe with it)22:40
*** sbezverk has quit IRC22:42
*** manheim has quit IRC22:43
*** manheim has joined #openstack-kolla22:45
kolla-slack<jascott1> sdake its nor really 120 is it?22:45
*** jascott1 has joined #openstack-kolla22:46
inc0jascott1: yes, yes it is22:46
jascott1HFS!22:46
jascott1i would think I would be digging underground in that climate22:47
inc0almost 50 C22:47
inc0jascott1: if you touch ground, it'll burn you22:48
inc0also happy digging:P22:48
jascott1so basically living in hell22:48
jascott1ya if warming keeps up we will be in the primo spot ;)22:49
jascott1well we already are imo22:49
jascott1oregon tropical beachfront property22:49
*** manheim has quit IRC22:50
inc0just a tad north and I'll be good22:50
jascott1heheh22:50
jascott1hey I got your changes working for me wrt to playbooks. I had to correct something or other and also added var for helm version as they had a problem with google storage yesterday22:51
jascott12.5 release was broken most day and all night22:52
*** pramodrj07 has quit IRC22:52
*** pramodrj07 has joined #openstack-kolla22:53
*** goldyfruit has joined #openstack-kolla22:55
*** manheim has joined #openstack-kolla22:57
*** goldyfruit has quit IRC22:58
*** salv-orl_ has quit IRC22:59
*** salv-orlando has joined #openstack-kolla22:59
*** manheim has quit IRC23:01
*** salv-orlando has quit IRC23:03
*** schwicht has joined #openstack-kolla23:10
*** harlowja has quit IRC23:19
*** spiette has joined #openstack-kolla23:20
*** jamesben_ has joined #openstack-kolla23:28
openstackgerritHongbin Lu proposed openstack/kolla-ansible master: Upgrade from docker-py to docker  https://review.openstack.org/47594123:33
*** jamesben_ has quit IRC23:33
openstackgerritHongbin Lu proposed openstack/kolla master: Upgrade from docker-py to docker  https://review.openstack.org/47593923:43
*** sbezverk has joined #openstack-kolla23:44
*** tonanhngo has quit IRC23:47
*** eaguilar has joined #openstack-kolla23:50
*** tonanhngo has joined #openstack-kolla23:52
*** tonanhngo has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!