Thursday, 2017-07-06

*** eaguilar has joined #openstack-kolla00:01
*** yuanying_ has joined #openstack-kolla00:05
*** Pavo has joined #openstack-kolla00:06
*** yuanying has quit IRC00:06
*** dixiaoli has joined #openstack-kolla00:09
*** dixiaoli has quit IRC00:14
Seyongrwellum sbezverk: Hello, with k8s 1.7.0, kolla-kubernetes deploying works as well? I followed deployment-guide.rst but failed. Just want to know anybody succeed or not00:16
*** yangyape_ has quit IRC00:17
*** yangyapeng has joined #openstack-kolla00:17
sbezverkSeyong : 1.7.0 is actively tested at the gate, I check and most of gate jobs were ok00:17
Seyongjust curious, the document is guiding that k8s 1.6.3 has a bug and giving workround fix... is the doc. uptodated?00:22
*** yangyapeng has quit IRC00:22
*** manheim has quit IRC00:26
sbezverkSeyong : most likely no, that bug was fixed in 1.6.400:29
*** yuanying has joined #openstack-kolla00:31
*** yuanying_ has quit IRC00:31
duonghqinc0, ping00:34
duonghqvhosakot, still there?00:36
*** zhurong has joined #openstack-kolla00:41
*** unicell has quit IRC00:48
*** yuanying has quit IRC00:49
kolla-slack<inc0> What's up?00:53
*** xinliang has quit IRC00:53
*** emccormick has quit IRC00:53
kolla-slack<inc0> Duonghq00:53
duonghqinc0, would to have a talk about slim Kolla image beside of gate session?00:56
kolla-slack<inc0> Summit talk? I'd love to see it, but I'm going to pass on presenting01:01
kolla-slack<inc0> I'll also have kolla project update and onboarding01:01
*** yuanying has joined #openstack-kolla01:02
*** xinliang has joined #openstack-kolla01:06
*** xinliang has joined #openstack-kolla01:06
*** dixiaoli has joined #openstack-kolla01:09
*** gfidente has quit IRC01:18
*** yangyapeng has joined #openstack-kolla01:19
*** huzhengchuan has joined #openstack-kolla01:21
*** iniazi_ has joined #openstack-kolla01:22
*** Pavo has quit IRC01:23
*** yangyapeng has quit IRC01:24
*** iniazi has quit IRC01:25
*** zhubingbing has joined #openstack-kolla01:26
*** Pavo has joined #openstack-kolla01:26
*** goldyfruit has joined #openstack-kolla01:27
duonghqinc0, yes, Sydney summit, I'm thinking about this topic, it'd be better if you can be co-speaker01:27
*** MarginHu has joined #openstack-kolla01:28
*** yangyapeng has joined #openstack-kolla01:28
zhubingbinghi guys01:29
sdakesup zhubingbing01:29
zhubingbingSup sdake01:29
sdakeplanet on fire01:29
duonghqzhubingbing, sdake do you have plan go to next PTG?01:30
sdakeyes01:30
zhubingbingI want to go01:30
duonghqnice01:30
zhubingbingbut01:30
zhubingbingnit01:30
zhubingbingbut01:30
zhubingbing;(01:30
zhubingbingvisa does not necessarily come down01:31
duonghqI applied for TSP again, wait for result01:31
duonghqah, visa...01:31
*** ducttape_ has joined #openstack-kolla01:32
*** yangyapeng has quit IRC01:33
*** caowei has joined #openstack-kolla01:34
*** yangyapeng has joined #openstack-kolla01:34
*** ducttape_ has quit IRC01:37
*** yangyape_ has joined #openstack-kolla01:39
*** yangyapeng has quit IRC01:39
*** ducttape_ has joined #openstack-kolla01:42
*** caoyuan has joined #openstack-kolla01:42
*** ducttape_ has quit IRC01:46
*** schwicht has quit IRC01:49
*** ducttape_ has joined #openstack-kolla01:49
*** cuongnv has joined #openstack-kolla01:49
*** ducttape_ has quit IRC01:54
*** yuanying has quit IRC01:57
*** yuanying has joined #openstack-kolla02:00
Pavoanyone here? have a question on custom repos02:00
jascott1Pavo i am here but no idea if I can help :|02:01
*** mattmceuen has joined #openstack-kolla02:01
jascott1whats up?02:02
Pavowell my question is when using custom repos for building images, does it have to be a url in a normal repo format?02:02
Pavoor just a url likecentos/7/os/x86_64/Packages/02:02
*** caoyuan has quit IRC02:04
Pavoalso getting python errors with kolla-build02:04
duonghqPavo, I'm here,02:05
duonghqI use custom repos on day basis02:06
Pavosweet02:06
Pavoduonghq whats the url format need to be like?02:06
Pavothis is what I have02:06
Pavorpm_setup_config = http://192.168.1.36:8080/centos/7/os/x86_64/Packages/02:06
Pavowould that work?02:07
vhosakotduonghq: yes, inc0 and I discussed today that upgrades in gate would be a nice talk. we ( inc0, you and I ) can submit one if you're up for it :)02:07
Pavoalso docs https://docs.openstack.org/kolla/latest/image-building.html very beginning isn't working02:08
vhosakotPavo, rpm_setup_config per https://github.com/openstack/kolla/blob/master/doc/image-building.rst#custom-repos is for custom binary images... if you want to build custom source images, rwellum has some work and documented the steps in https://etherpad.openstack.org/p/kolla-kubernetes_with_your_own_source_images.02:08
Pavosays to Generating kolla-build.conf do pip install tox and then tox -e genconfig02:08
Pavodoesn't work02:08
duonghqvhosakot, I think Pavo is talking about CentOS custom repo?02:09
Pavoerror ERROR: toxini file 'tox.ini' not found02:09
Pavovhosakot yeah I always do binary02:09
Pavonot source02:09
Pavocan not trust source02:09
vhosakotPavo: ah, cool02:09
duonghqvhosakot, upgrade gate is very nice, and it's quite mandatory when I finish rolling upgrade ps (trying to test it), and I think rolling upgrade is a good use-case for the gate02:10
Pavoduonghq any idea on the Generating kolla-build.conf error?02:10
*** manheim has joined #openstack-kolla02:10
duonghqPavo, no, I haven't gotten this error :(02:10
Pavohmmm02:10
duonghqPavo, about the repo: you can use create_repo package02:11
duonghqit'll build the repo metadata02:11
kolla-slack<inc0> Pavo worst case scenario you can just write ini file with options02:11
kolla-slack<inc0> No need to generate conf really02:11
kolla-slack<inc0> Only to make commented defaults02:11
Pavoinc0 ini file?02:11
kolla-slack<inc0> Well build.conf02:11
Pavothe toc.ini file?02:12
kolla-slack<inc0> No, actual build.conf02:12
*** caoyuan has joined #openstack-kolla02:12
kolla-slack<inc0> It's not mandatory02:12
Pavosee I tried making the kolla-build.conf located in /etc/kolla/02:12
Pavoand kolla-build gives python error02:12
kolla-slack<inc0> What error is that?02:13
Pavoinc0 http://paste.openstack.org/show/614520/02:13
kolla-slack<inc0> Can you paste your conf?02:14
Pavoinc0 sure and this is the directory tree http://paste.openstack.org/show/614521/02:14
*** manheim has quit IRC02:14
Pavoinc0 only thing I have in the kolla-build.conf is rpm_setup_config = http://192.168.1.36:8080/centos/7/os/x86_64/Packages/02:14
vhosakotPavo: does /etc/kolla/kolla-build.conf have the [DEFAULT] section?02:14
Pavono02:15
Pavolet me add it02:15
vhosakotPavo: I think it must be under the [DEFAULT] section :)02:15
Pavook02:15
Pavook guess that worked02:15
vhosakotinc0: duonghq: should I create an enterpad for the upgrade talk for Sydney?02:15
vhosakotPavo: cool02:15
Pavonow duonghq getting rpm setup issue with the url I used02:16
duonghqvhosakot, please02:16
Pavoduonghq RPM setup must be provided as .rpm or .repo files. Attempted configuration was http://192.168.1.36:8080/centos/7/os/x86_64/Packages/02:16
vhosakotduonghq: cool.. will do02:16
Pavoduonghq got an example kolla-build.conf with rpm url for custom repo?02:17
duonghqvhosakot, it's depend on you and inc0 if both of you'd like to keep it as private or feel free to public before we submit CFP02:17
duonghqPavo, hmm, I used template override02:18
kolla-slack<inc0> I don't care if it's public02:18
Pavoduonghq hmmm02:18
vhosakotduonghq: inc0? what do you mean public vs private?  all talks will be be public after voting starts anyway :)02:18
kolla-slack<inc0> Duong if you want to co present with us, I'd be happy to have you there02:18
duonghqvhosakot, I mean the etherpad02:19
duonghqinc0, sure, I want to go with you02:19
kolla-slack<inc0> Nothing secret out there. Presentation will be about dockerhub publisher and gates02:19
vhosakotduonghq: ah, the etherpad is usually worked on by whoever is speaking, we can ask the public for feedback sure, but I think the foundation limits only 4 speakers per talk or something unless it is a panel :)02:20
kolla-slack<inc0> 3 of us would be good team;)02:20
duonghqinc0, vhosakot nice, thanks02:21
vhosakotinc0: cool.. shouldn;t the "dockerhub publisher and gates" talk be different than the "rolling upgrades in gate" talk?  are, they're the same?02:21
duonghqPavo, mins, let me gather some information for you02:21
Pavoduonghq thank, would help because listing every single .rpm file would be insane02:21
kolla-slack<inc0> So cool Duong you're in:) let's talk tomorrow about it and make abstract02:21
kolla-slack<inc0> We'll think of something02:22
duonghqvhosakot, I think rolling upgrade can be one use-case for "upgrade gate", not to put in title02:22
*** jtriley has joined #openstack-kolla02:22
duonghqinc0,  sure02:23
duonghqPavo, https://wiki.centos.org/HowTos/CreateLocalRepos02:24
Pavoduonghq I have a local repo built02:24
vhosakotduonghq: right right.. I was thinking should we combine "dockerhub publisher and gates" and "upgrades in gate" as a single talk? or, two separate talks?02:24
duonghqand you can configure an http server point to it02:24
Pavoduonghq thats not the issue02:24
vhosakotinc0: what do you think? should we combine "dockerhub publisher and gates" and "upgrades in gate" as a single talk? or, two separate talks?02:25
Pavoits the way kolla-build.conf reads the address02:25
duonghqvhosakot, it'd better if 2 topis is "dockerhub publisher" and "upgrade gate"?02:25
Pavothe docs say The repos are accepted as a list of comma separated values and can be in the form of .repo, .rpm, or a url. but there is no example of url02:25
vhosakotduonghq: that is what I think too as the dockerhub publisher+gates talk does not cover upgrades.. cool02:25
*** mattmceuen has quit IRC02:26
duonghqPava, vhosakot. sorry, I need talk a short walk around for refresh my brain02:27
duonghq*take a short walk02:27
duonghqbrb02:27
duonghq5mins02:27
vhosakotduonghq: or a beer? ;)02:27
duonghqno, it's my working time here02:27
duonghq:P02:27
duonghqbeer or whisky is a good option in other time02:28
vhosakotduonghq: inc0: I'd see a talk about upgrades as a talk that shows how kolla can be used to upgrade OpenStack (both minor release-to-minor release, and major release-to-major release) in an automated, rolling way with minimum or no service interruption, and the talk can also show how our kolla gate can be used as a sample to implement rolling upgrade in any CI/CD gate internally in any company.02:29
* duonghq back 02:36
duonghqvhosakot, it's another topic?02:36
duonghqPavo, still here?02:37
vhosakotduonghq: that is what I'm thkinking.  the "upgrade talk" must be separate from "dockerhub publisher+gate" talk..02:37
vhosakotinc0: what do you think? should we combine "dockerhub publisher and gates" and "upgrades in gate" as a single talk? or, two separate talks?02:38
Pavoduonghq yeap02:38
duonghqvhosakot, hmm, it'll be very interesting topic02:38
*** caoyuan has quit IRC02:38
duonghqPavo, after run createrepo, notice that it'll generate repodata dir02:38
duonghqthe URL should be point to the dir contains the repodata02:39
Pavook02:39
duonghqi.e. the repodata's path is /long/long/long/path/repodata -> point URL to /long/long/long/path02:39
Pavoduonghq will try that but I know my local repo works because I use it for normal repo on my centos machine02:40
Pavoduonghq  so.... http://192.168.1.36:8080/centos/7/os/x86_64/repodata/02:40
PavoI tried only pointing to http://192.168.1.36:8080/centos/7/os/x86_6402:41
Pavothat doesn't work either02:41
Pavoduonghq same error RPM setup must be provided as .rpm or .repo files. Attempted configuration was http://192.168.1.36:8080/centos/7/os/x86_6402:41
*** MarginHu has quit IRC02:43
duonghqwhat is the error?02:43
duonghqyou can try to configure repo manual inside an container to test the repo02:43
*** MarginHu has joined #openstack-kolla02:43
*** cuongnv has left #openstack-kolla02:44
Pavoduonghq that is the error02:45
Pavoduonghq http://paste.openstack.org/show/614523/02:46
duonghqRPM setup must be provided as .rpm or .repo files.02:47
duonghqseem that it doesn't accept url directly02:47
Pavowell another thing in the docs that need to be changed02:48
Pavoduonghq docs say "The build method allows the operator to build containers from custom repos. The repos are accepted as a list of comma separated values and can be in the form of .repo, .rpm, or a url. See examples below."02:48
Pavoduonghq but the example only shows .repo example02:49
duonghqyeah, you can put it in repo file02:49
duonghqI use my repo file02:49
*** ducttape_ has joined #openstack-kolla02:49
Pavoyeah I could but that would also require to put that .repo file in kolla/docker/base directory correct?02:52
*** caoyuan has joined #openstack-kolla02:53
*** dave-mccowan has quit IRC02:53
*** ducttape_ has quit IRC02:54
Pavook that worked02:55
Pavobuilding now02:55
duonghqPavo, yeah, quite frustrate02:59
Pavowow there is a ton of 3 party repos03:00
Pavothat aren't part of normal centos repo03:00
duonghqyup, many 3rd party repos already here03:00
duonghqI must create mirror for all of this03:00
Pavoyeah /etc/yum.repos.d/grafana.repo03:00
Pavoand others03:00
Pavoduonghq you have a rysnc script you use?03:01
Pavowould love to get a list of them all03:01
Pavoso I can just make a rsync script03:01
duonghqPavo, I use reposync03:02
Pavobut you still have a list of all the repos right?03:02
duonghqI grad trouble repo and get its url03:02
duonghqYou mean you want URL list?03:02
duonghqminutes03:02
Pavoyes please03:02
*** caowei_ has joined #openstack-kolla03:08
*** mmehan has quit IRC03:10
*** caowei has quit IRC03:12
*** caowei_ is now known as caowei03:12
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: [WIP] Implement Keystone zero-downtime upgrade  https://review.openstack.org/48008103:13
Pavoduonghq you getting that list of URLs?03:13
duonghqPavo, mins, I quite busy03:13
Pavoduonghq oh ok, mybad03:13
Pavoduonghq can I PM you?03:14
duonghqPavo, http://paste.openstack.org/show/614525/03:15
duonghqyes03:15
duonghqPavo, http://paste.openstack.org/show/614526/ _> I cat * my repo dir03:16
Pavooh nvm was gonna PM you with my email address so you could just sent it to me later when you had time03:16
Pavobut that works03:16
Pavothanks03:16
duonghqhope that it fine, with one exception03:17
duonghqthe delorean repo03:17
Pavono need for that03:17
Pavoso... most of it looks like its just centos repo03:18
Pavowith a few other repos03:18
duonghqya03:18
Pavoceph, docker, mariadb, elastic and kubernetes03:19
duonghqmainly delorean, epel, treasuredata (for fluentd)03:19
duonghqdocker is for my own purpose03:19
duonghqnot related to kolla03:19
Pavoand this allows you to completely build all kolla images that are centos based?03:20
duonghqmariadb is most painful repo when create mirror03:20
Pavowithout having any internet03:20
duonghqiirc, it should be enough03:20
Pavook03:20
duonghqnot sure, I still use Internet as backup plan when I forget synchronize my mirror03:20
Pavoyeah guess I'll findout what doesn't build03:21
duonghqand please let me know if you found one,03:21
openstackgerritMerged openstack/kolla-ansible master: Add designate dashboard to horizon  https://review.openstack.org/47815803:22
*** MarginHu has quit IRC03:23
*** MarginHu has joined #openstack-kolla03:24
*** goldyfruit has quit IRC03:24
*** goldyfruit has joined #openstack-kolla03:25
*** vhosakot has quit IRC03:26
openstackgerritMerged openstack/kolla-ansible master: Fix keystone-paste.ini custom file management  https://review.openstack.org/48060803:27
*** vhosakot has joined #openstack-kolla03:27
openstackgerritMerged openstack/kolla-ansible stable/ocata: Use proper variable in horizon copy policy task  https://review.openstack.org/48064903:27
*** vhosakot has quit IRC03:27
spsuryamorning03:27
*** vhosakot has joined #openstack-kolla03:27
*** vhosakot has quit IRC03:27
openstackgerritMerged openstack/kolla-ansible master: Move mDNS to network nodes  https://review.openstack.org/46853403:34
openstackgerritMerged openstack/kolla-ansible master: Fix skydive analyzer "etcd" config  https://review.openstack.org/47372303:40
*** eaguilar has quit IRC03:40
*** Pavo has quit IRC03:46
*** mdnadeem has joined #openstack-kolla03:47
*** itlinux has joined #openstack-kolla03:56
openstackgerritMerged openstack/kolla-ansible stable/ocata: Fix service_type of Octavia  https://review.openstack.org/47417404:04
duonghqmorning spsurya04:09
openstackgerritMerged openstack/kolla-ansible master: Modify the configuration file of telegraf  https://review.openstack.org/46544604:11
openstackgerritMerged openstack/kolla-ansible stable/ocata: Fix Grafana config file template to use variables  https://review.openstack.org/46216104:12
*** ipsecguy_ has joined #openstack-kolla04:17
*** janki has joined #openstack-kolla04:18
*** ipsecguy has quit IRC04:18
*** jtriley has quit IRC04:25
*** goldyfruit has quit IRC04:45
openstackgerritNoboru Iwamatsu proposed openstack/kolla master: change fernet-key repository permissions to 750  https://review.openstack.org/48083004:47
*** caowei_ has joined #openstack-kolla04:58
*** dixiaoli has quit IRC04:58
*** caowei has quit IRC05:02
*** caowei_ is now known as caowei05:02
*** yeyon has joined #openstack-kolla05:05
spsuryasup duonghq05:07
*** dixiaoli has joined #openstack-kolla05:11
*** cargonza_ has joined #openstack-kolla05:13
*** unicell has joined #openstack-kolla05:17
*** dosaboy_ has joined #openstack-kolla05:20
*** skramaja has joined #openstack-kolla05:21
*** harbie_ has joined #openstack-kolla05:21
*** harlowja has quit IRC05:22
*** cargonza has quit IRC05:22
*** harbie has quit IRC05:22
*** devananda has quit IRC05:22
*** dosaboy has quit IRC05:22
*** harbie_ is now known as harbie05:22
*** harbie has quit IRC05:22
*** harbie has joined #openstack-kolla05:22
*** cargonza_ is now known as cargonza05:22
*** unicell1 has joined #openstack-kolla05:23
*** unicell has quit IRC05:23
*** prateek has joined #openstack-kolla05:24
*** mnaser has quit IRC05:24
*** pc_m has quit IRC05:25
*** calbers has quit IRC05:25
yeyonsystemctl docker start docker05:27
yeyonerror message:(code =exited,status=1/FAILURE)05:27
*** pc_m has joined #openstack-kolla05:27
*** calbers has joined #openstack-kolla05:27
yeyonwhy??05:27
*** prateek has quit IRC05:28
*** prateek_ has joined #openstack-kolla05:30
*** prateek_ is now known as 7GHABESXC05:30
*** prateek_ has joined #openstack-kolla05:30
*** prateek_ has quit IRC05:30
*** mnaser has joined #openstack-kolla05:34
*** sgordon has quit IRC05:41
*** devananda has joined #openstack-kolla05:44
*** dgonzalez has quit IRC05:44
*** dgonzalez has joined #openstack-kolla05:46
*** sgordon has joined #openstack-kolla05:50
*** itlinux has quit IRC05:58
*** mcnanci has quit IRC06:01
*** calbers has quit IRC06:01
*** calbers has joined #openstack-kolla06:04
*** tovin07_ has joined #openstack-kolla06:17
*** zhubingbing_ has joined #openstack-kolla06:18
*** jascott1_ has joined #openstack-kolla06:19
*** ddyer2 has joined #openstack-kolla06:20
*** zhubingbing has quit IRC06:21
*** jascott1 has quit IRC06:21
*** ddyer has quit IRC06:21
*** manheim has joined #openstack-kolla06:37
*** manheim has quit IRC06:43
*** gfidente has joined #openstack-kolla06:44
*** gfidente has quit IRC06:44
*** gfidente has joined #openstack-kolla06:44
openstackgerritNoboru Iwamatsu proposed openstack/kolla-ansible master: delete delimiter from init fernet condition check  https://review.openstack.org/48085306:52
*** shardy has joined #openstack-kolla07:06
*** rwsu has quit IRC07:06
*** janki has quit IRC07:12
*** janki has joined #openstack-kolla07:12
*** mgoddard_ has joined #openstack-kolla07:14
*** mandre has quit IRC07:18
*** mandre has joined #openstack-kolla07:30
*** reidrac has joined #openstack-kolla07:35
*** manheim has joined #openstack-kolla07:43
*** mgoddard_ has quit IRC07:48
openstackgerritzhubingbing proposed openstack/kolla-ansible stable/ocata: Add option ha_vrrp_health_check_interval for l3  https://review.openstack.org/48088807:49
*** egonzalez has joined #openstack-kolla07:50
openstackgerritBertrand Lallau proposed openstack/kolla master: Fix searchlight_api container starting issue  https://review.openstack.org/48074307:54
openstackgerritMerged openstack/kolla master: Migrate cinder-api under apache  https://review.openstack.org/46353507:55
openstackgerritMerged openstack/kolla-ansible master: Add option ha_vrrp_health_check_interval for l3  https://review.openstack.org/47561407:56
zhubingbing_sup egonzalez07:57
egonzalezzhubingbing_, sup07:57
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Migrate cinder-api under apache  https://review.openstack.org/47175307:59
iniazi_is there kolla documents reorg going on?  kolla-kubernetes and links from kolla docs are not working (or docs moving out of openstack.org?)08:00
iniazi_i'm new to kolla so it may have been old news08:01
egonzaleziniazi_, which links? all openstack docs are being worked atm08:03
openstackgerritMerged openstack/kolla-ansible master: Remove elasticsearch_protocol unused variable  https://review.openstack.org/48018108:03
openstackgerritjangseon ryu proposed openstack/kolla-ansible master: deploy failed when didn't defined node_config_directory  https://review.openstack.org/48089108:07
*** jascott1_ has quit IRC08:10
*** jascott1 has joined #openstack-kolla08:10
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: [WIP] Implement Keystone zero-downtime upgrade  https://review.openstack.org/48008108:10
*** jascott1 has quit IRC08:10
duonghqegonzalez, ping08:11
*** jascott1 has joined #openstack-kolla08:11
egonzalezduonghq, pong08:12
duonghqegonzalez, https://review.openstack.org/#/c/480081/ <<< I draft one strategy in the ps08:13
duonghqfor zero-downtime upgrade08:13
duonghqbut in my testbeds, the restart keystone* handler isn't triggered08:13
duonghqdo you have any idea?08:13
*** blallau has joined #openstack-kolla08:17
iniazi_egonzalez, oh docs.openstack.org/developer/kolla-kubernetes/ and https://docs.openstack.org/developer/kolla/mitaka/    but yeah, if its actively being worked on, than nevermind.  was just starting to look into kubernetes install today, but got 404s08:18
iniazi_well k8s was 40408:19
iniazi_actually didn't read the 404 message too carefully.  so they are doing a reorg... i guess i'll have to wait a few days08:20
iniazi_actually that may not even be relevant to me...08:20
iniazi_well i don't know... anyway08:20
*** mgoddard_ has joined #openstack-kolla08:21
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: [WIP] Implement Keystone zero-downtime upgrade  https://review.openstack.org/48008108:23
*** yangyapeng has joined #openstack-kolla08:26
openstackgerritMerged openstack/kolla-ansible master: Switch from oslosphinx to openstackdocstheme  https://review.openstack.org/47736308:31
*** zhurong has quit IRC08:32
reidraciniazi_: is not the same, but GH renders the RST files: https://github.com/openstack/kolla-kubernetes/blob/master/doc/source/deployment-guide.rst08:37
egonzalezduonghq, i'm not sure whats happening there08:37
reidracfar from perfect, but readable08:37
egonzalezI dont fully understand the strategy plugins yet08:37
duonghqegonzalez, even without the strategy plugin, in my testbed, the handler isn't invoked08:38
duonghqI'm trying to revert and run upgrade again08:38
*** zhurong has joined #openstack-kolla08:39
*** yangyapeng has quit IRC08:50
mandreegonzalez, duonghq: hi! can I get some eyes on https://review.openstack.org/480200?08:56
*** tvignaud has quit IRC08:57
iniazi_reidrac, thanks, i'll look at that.09:01
mandrethx egonzalez!09:05
*** caoyuan has quit IRC09:08
*** tvignaud has joined #openstack-kolla09:11
*** blallau has quit IRC09:14
*** blallau has joined #openstack-kolla09:15
*** magicboiz has quit IRC09:16
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Migrate cinder-api under apache  https://review.openstack.org/47175309:16
openstackgerritMartin André proposed openstack/kolla master: Make set_configs.py work with symlinks  https://review.openstack.org/48020009:20
*** bmotz has joined #openstack-kolla09:22
*** lyang__ has quit IRC09:28
openstackgerritPavel Gluschak (scsnow) proposed openstack/kolla-ansible master: Make cinder-backup service optional  https://review.openstack.org/46462009:29
*** lyang__ has joined #openstack-kolla09:30
*** dciabrin has quit IRC09:35
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Migrate cinder-api under apache  https://review.openstack.org/47175309:42
*** shardy is now known as working_shardy09:44
*** igordcard has quit IRC09:48
openstackgerritMerged openstack/kolla-ansible master: Fix field data is too large  https://review.openstack.org/47636709:48
*** igordcard has joined #openstack-kolla09:48
*** dciabrin has joined #openstack-kolla09:49
*** dixiaoli has quit IRC09:51
*** duonghq has quit IRC09:53
*** duonghq has joined #openstack-kolla09:55
openstackgerritMerged openstack/kolla stable/ocata: Fix apache configuration for ceilometer_api  https://review.openstack.org/48008809:57
*** zhubingbing__ has joined #openstack-kolla09:59
*** tovin07_ has quit IRC10:05
*** caowei has quit IRC10:08
*** caowei has joined #openstack-kolla10:09
*** zhubingbing__ has quit IRC10:10
openstackgerritMerged openstack/kolla master: Fix searchlight_api container starting issue  https://review.openstack.org/48074310:10
*** MarginHu has quit IRC10:11
*** jascott1 has quit IRC10:12
*** sambetts|afk is now known as sambetts10:12
*** jascott1 has joined #openstack-kolla10:12
*** jascott1 has quit IRC10:17
*** working_shardy is now known as shardy10:21
openstackgerritPavel Gluschak (scsnow) proposed openstack/kolla-ansible master: Don't enable ceilometer-api when database type is gnocchi  https://review.openstack.org/48096010:24
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Fix zun log dir permissions  https://review.openstack.org/48096110:24
*** japestinho has joined #openstack-kolla10:26
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Fix zun-api logging  https://review.openstack.org/48096310:27
*** manheim_ has joined #openstack-kolla10:30
*** manheim has quit IRC10:33
*** yangyape_ has quit IRC10:35
*** rwallner has joined #openstack-kolla10:49
*** caowei has quit IRC10:49
*** pbourke has quit IRC10:50
*** lpetrut has joined #openstack-kolla10:50
*** pbourke has joined #openstack-kolla10:51
*** retr0h has quit IRC10:57
*** retr0h has joined #openstack-kolla10:57
openstackgerritTaeha Kim proposed openstack/kolla-ansible master: Correct a syntax error in the tools/cleanup-images  https://review.openstack.org/48097710:57
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Remove pep8 exceptions  https://review.openstack.org/48097811:06
sdakemorning peeps11:10
sdakeiniazi_ the docs need a 1 patch march to make them uptodate on docs.oo as a result of recent changes to infrastructure11:12
sdakeiniazi_ as of yet nobody has published any update to trigger the new docs job bulid11:12
*** rwsu has joined #openstack-kolla11:33
manheim_hi11:34
manheim_is it possible not to give any interface for neutron_external_interface?11:34
manheim_coz I'll want to do a NAT from the management network11:35
manheim_i'm NIC limited11:35
*** jtriley has joined #openstack-kolla11:38
*** schwicht has joined #openstack-kolla11:39
*** ducttape_ has joined #openstack-kolla11:41
*** ansmith has quit IRC11:44
*** ducttape_ has quit IRC11:46
*** jtriley has quit IRC11:51
*** mattmceuen has joined #openstack-kolla11:55
*** tumble has joined #openstack-kolla11:57
*** yangyapeng has joined #openstack-kolla12:02
rwellumSeyong: Any more luck with kolla-k8s? I know the deployment guide needs a couple of updates.12:06
rwellummanheim_: my understanding is 'no'. Neutron needs a NIC with either an unused ip address or no ip address. Can you use the second scenario?12:09
*** emccormick has joined #openstack-kolla12:12
openstackgerritBradley Jones proposed openstack/kolla-kubernetes master: (WIP) Deploy playbook testing with nova PV  https://review.openstack.org/47982612:14
*** oanson has quit IRC12:15
*** oanson has joined #openstack-kolla12:20
*** goldyfruit has joined #openstack-kolla12:21
openstackgerritDennis Mueller proposed openstack/kolla-ansible master: Added neutron metering-agent support  https://review.openstack.org/44585812:22
*** jbadiapa has quit IRC12:25
*** rhallisey has joined #openstack-kolla12:28
*** hrw has quit IRC12:30
*** hrw has joined #openstack-kolla12:32
*** ansmith has joined #openstack-kolla12:35
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Fix bandit tests directory  https://review.openstack.org/48101612:42
*** mdnadeem has quit IRC12:42
spsuryamanheim_: you can create one more NIC12:48
spsuryafor neutron12:48
*** mattmceuen has quit IRC12:48
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Fix bandit tests directory  https://review.openstack.org/48101612:51
manheim_i've added one more NIC, yes12:58
*** yeyon has quit IRC13:03
openstackgerritMerged openstack/kolla-ansible master: Fixed vmware typos  https://review.openstack.org/48029713:08
*** skramaja has quit IRC13:09
*** 7GHABESXC has quit IRC13:10
openstackgerritNoboru Iwamatsu proposed openstack/kolla-ansible master: delete delimiter from init fernet condition check  https://review.openstack.org/48085313:11
*** rwsu has quit IRC13:14
openstackgerritNoboru Iwamatsu proposed openstack/kolla-ansible master: delete delimiter from init fernet condition check  https://review.openstack.org/48085313:15
*** janki has quit IRC13:16
*** janki has joined #openstack-kolla13:16
*** iniazi has joined #openstack-kolla13:16
*** lucasxu has joined #openstack-kolla13:19
*** iniazi_ has quit IRC13:20
*** zhurong has quit IRC13:22
*** ducttape_ has joined #openstack-kolla13:23
*** ducttape_ has quit IRC13:27
*** igordcard has quit IRC13:28
*** igordcard has joined #openstack-kolla13:29
openstackgerritAndy Smith proposed openstack/kolla-ansible master: Add support for hybrid messaging backends  https://review.openstack.org/46896613:31
*** huzhengchuan has quit IRC13:34
egonzalezJeffrey4l, around? mind if change this dir to /etc/kolla/version_check/releases or any other dir? https://github.com/openstack/kolla/blob/master/tools/version-check.py#L4113:36
egonzalezi'm working on adding bandit tests to all code and complains about being in /tmp13:36
*** ducttape_ has joined #openstack-kolla13:37
Jeffrey4legonzalez, no. but which folder is better?13:37
*** ducttape_ has quit IRC13:37
egonzalezhrm, not sure, I can add just a # nosec to ignore the warning13:38
Jeffrey4lnp, i mean* :)13:38
*** ducttape_ has joined #openstack-kolla13:38
egonzalezbecause i dont think a git repo belongs to /etc/kolla13:38
Jeffrey4lhrm, another solution is use tox, then we can put the file into .tox folder ;)13:42
*** krtaylor has quit IRC13:42
*** bmotz_ has joined #openstack-kolla13:45
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Fix bandit tests directory  https://review.openstack.org/48101613:47
egonzalezbut when running version_check without tox will fail13:48
Jeffrey4lyes.13:49
*** bmotz has quit IRC13:49
manheim_olla nova waiting for nova-compute to come up13:50
manheim_sorry13:50
manheim_https://bugs.launchpad.net/kolla/+bug/166848913:50
openstackLaunchpad bug 1668489 in kolla "failed to deploy because haproxy Waiting for virtual IP to appear" [Medium,Confirmed] - Assigned to Eduardo Gonzalez (egonzalez90)13:50
manheim_was this fixed?13:50
manheim_i have this issue13:50
manheim_the interface is up and has an IP13:50
*** blallau has quit IRC13:50
Jeffrey4legonzalez, how about use # nosec before we have a better solution?13:51
Jeffrey4lor just use ".release" folder and add .release into gitignore file?13:52
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Add bandit check to all python code  https://review.openstack.org/48108713:52
*** bmotz_ has quit IRC13:55
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Add bandit check to all python code  https://review.openstack.org/48108713:55
egonzalezJeffrey4l, I like that one13:55
egonzalezbetter .releases13:55
*** krtaylor has joined #openstack-kolla13:56
Jeffrey4legonzalez, ok.13:56
egonzalezthanks!13:56
egonzalezmanheim_, which release? is rabbitmq running, are there compute nodes actives?13:58
manheim_it's an AIO13:59
manheim_ubuntu 16.04 host w latest upgrades, ubuntu source images, stable/ocata repo with 4.0.0 images14:00
*** hongbin has joined #openstack-kolla14:01
*** bmotz_ has joined #openstack-kolla14:01
egonzalezmanheim_, ensure nova service-list returns nova-compute binary active, if not check if is actually running or connected to rabbitmq14:03
manheim_i took out the haproxy for the moment14:03
manheim_I have another issue now14:03
manheim_libvirt is not starting14:03
*** ducttape_ has quit IRC14:11
*** ducttape_ has joined #openstack-kolla14:11
*** jbadiapa has joined #openstack-kolla14:18
*** jtriley has joined #openstack-kolla14:22
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Standardize Keystone domain variables  https://review.openstack.org/47928714:34
*** eaguilar has joined #openstack-kolla14:36
jamesbensonmanheim_: regarding the eth's.  I use veth here and that seems to have worked... check out my interfaces file here for an idea: https://github.com/JamesOBenson/Kolla-misc/blob/master/tasks/Generate_interfaces.yml14:42
manheim_ok, ty14:43
jamesbensonwe use only 2 ports... one thing I did notice is that eno1:1 for example would fail because ansible has issues with it... but this way worked.... now I just have ceph issues to work out...14:44
*** ducttape_ has quit IRC14:46
*** zhubingbing_ has quit IRC14:53
*** eaguilar has quit IRC15:01
*** ducttape_ has joined #openstack-kolla15:08
openstackgerritMerged openstack/kolla master: Make set_configs.py work with symlinks  https://review.openstack.org/48020015:09
*** ducttape_ has quit IRC15:10
*** ducttape_ has joined #openstack-kolla15:10
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Implement blazar Docker images  https://review.openstack.org/45474515:18
*** unicell has joined #openstack-kolla15:22
*** unicell1 has quit IRC15:23
jamesbensoninc0: ping15:25
*** eaguilar has joined #openstack-kolla15:27
sdakekfox1111 quick quesiton about the neutron not restartble problem15:30
sdakekfox1111 wouodl you mind expanding on which pods cannot be restarted - local testing shows neutron-server restarts just fine15:31
kfox1111morning.15:31
kfox1111anything that makes a network namespace.15:31
kfox1111neutron-dhcp-agent, neutron-l3-agent, neutron-openvswitch-agent.15:32
*** janki has quit IRC15:32
*** mandre is now known as mandre_away15:33
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Use monasca master tarballs  https://review.openstack.org/48111815:33
inc0good morning15:36
*** bmotz_ has quit IRC15:37
inc0duonghq so, summit talks15:37
inc0if you want me to co-present with you, I'll be happy to do so15:38
inc0only if I happen to have 2 full presentations and kolla project update + onboarding, I'll need a lot of your help guys with preparation15:39
jamesbensoninc0, mind helping pushing this bug? https://review.openstack.org/#/c/477605/615:41
inc0yeah hold on15:41
jamesbensonthanks15:41
inc0so, I'll explain how you should debug gates15:41
inc0first of all, click on red gate, you'll be redirected to page like this http://logs.openstack.org/05/477605/6/check/gate-kolla-ansible-dsvm-deploy-multinode-ubuntu-source-ubuntu-xenial-2-node-nv/969ec62/15:42
inc0console is stdout of gate, that's first thing to look at15:42
jamesbensonk15:43
jamesbensonlooks good there from what I can see15:43
jamesbensonit had some sudo: unable to resolve host ubuntu-xenial-2-node-osic-cloud1-s3700-9582718: Connection timed out15:44
jamesbensonissues15:44
inc0I see the issue15:45
inc0so go to logs/ansible15:45
inc0and click through various logs15:45
inc0http://logs.openstack.org/05/477605/6/check/gate-kolla-ansible-dsvm-deploy-multinode-ubuntu-source-ubuntu-xenial-2-node-nv/969ec62/logs/ansible/get-logs this one15:45
inc0failed15:45
inc0and it failed because one of containers is in Exited state15:46
inc0http://logs.openstack.org/05/477605/6/check/gate-kolla-ansible-dsvm-deploy-multinode-ubuntu-source-ubuntu-xenial-2-node-nv/969ec62/logs/docker_logs/ this one15:46
inc0http://logs.openstack.org/05/477605/6/check/gate-kolla-ansible-dsvm-deploy-multinode-ubuntu-source-ubuntu-xenial-2-node-nv/969ec62/logs/kolla/mariadb/mariadb.log and this is actual log15:47
inc0connection time out15:47
inc0totally not helpful15:47
jamesbensonyeah, story of my life :-p15:47
jamesbensonthat's why I did a recheck the first time.... because a I think 2-3 failed for timeout issues, after recheck, half failed.15:48
*** hieulq_ has joined #openstack-kolla15:48
jamesbensonjust issued another recheck this morning, hasn't gone yet.15:49
inc0maybe sth with infra, not sure, I'll keep looking15:49
inc0I have meeting now so I'll be back in hour or so15:49
jamesbensonk15:49
sdakekfox1111 thanks for the answer earlier - i missed it  - appreciate it15:51
kfox1111np15:53
*** dixiaoli has joined #openstack-kolla15:53
*** itlinux has joined #openstack-kolla15:55
*** dixiaoli has quit IRC15:57
*** mgiles has joined #openstack-kolla15:59
*** david-lyle has joined #openstack-kolla16:00
openstackgerritRich Wellum proposed openstack/kolla master: Add a section on building custom kolla images from source.  https://review.openstack.org/48112716:03
*** mgiles has left #openstack-kolla16:05
*** egonzalez has quit IRC16:05
*** manheim_ has quit IRC16:05
*** jbadiapa has quit IRC16:07
*** rwsu has joined #openstack-kolla16:12
*** eaguilar has quit IRC16:15
*** reidrac has quit IRC16:33
sdakerwellum can you do me a solid and find something simple to fix in the kolla-kubernetes docs to fix this prooblem: https://docs.openstack.org/kolla-kubernetes16:34
sdakecould be a typo - could be a junk commit with a revert afterwards - anything will do :)16:34
*** sambetts is now known as sambetts|afk16:35
*** jtriley has quit IRC16:36
rwellumsdake: yes - actually I have a couple of fixups in the bare-metal guide - will that do?16:37
*** jbadiapa has joined #openstack-kolla16:38
sdakerwellum is that in kolla-kubernetes deliverable?16:39
sdakeif so yes16:39
sdake(we have a couple bre metal guide)16:39
sdakes16:39
*** jtriley has joined #openstack-kolla16:40
rwellumsdake: kolla-kubernetes/doc/source/deployment-guide.rst16:43
*** hieulq_ has quit IRC16:43
sdakerwellum please subit - would like to fix asap16:44
sdakerwellum even a one liner16:44
rwellumYes - gimme a minute and will :)16:44
sdakecool :)16:44
*** unicell has quit IRC16:45
*** gfidente has quit IRC16:53
openstackgerritRich Wellum proposed openstack/kolla-kubernetes master: Update the deployment guide with a few corrections.  https://review.openstack.org/48115016:54
rwellumsdake: ^^^16:55
sdakeinc0 kfox1111 sbezverk can one of you rubber stmap this docs change - all docs publishing are broken on docs.oo as a result of a change to infra on july 3rd16:59
kfox1111already done.16:59
sdakethanks16:59
kfox1111np.16:59
sdakerwellum note i left a note in there recommedning further followup work17:00
*** mgoddard_ has quit IRC17:02
*** inc0 has quit IRC17:02
*** inc0 has joined #openstack-kolla17:03
krtaylordoes anyone know of any talk to use kolla builds and maybe kolla-ansible instead of dsvm for infra CI check/gate testing? like a zuul/nodepool container alternative?17:04
inc0krtaylor: there were whispers here and there17:04
inc0also to use kolla as devstack alternative (patches for that are already in)17:04
openstackgerritMerged openstack/kolla-kubernetes master: Update the deployment guide with a few corrections.  https://review.openstack.org/48115017:05
inc0but nothing concrete17:05
*** shardy has quit IRC17:07
*** rwsu has quit IRC17:07
*** rwsu has joined #openstack-kolla17:07
krtaylorinc0, thanks, I would be interested17:07
krtaylorI've searched mail and irc logs17:07
krtaylorI'll also post to infra and see what happens17:07
inc0once we have readily available images17:08
inc0dockerhub + cache17:08
inc0deployment of gate would be couple times faster17:08
sdakekrtaylor we have typically steered clear of that area to avoid the hassle of duplication of effort, parallel development, forcing people to learn ansible+docker to do development work when in many cases devstack works, and finally the shitshow that could errupt as a result :)17:09
krtayloryes, I have been following that, and the discussion has included gate/check deployment17:09
krtaylorlol, yes!17:09
kfox1111yeah, but then you get the opposite issue...17:09
sdakekrtaylor that said, i think the timing is right to make kolla-ansible a viable development environment for openstack17:09
kfox1111devs that only work against something users will never use, devstack.17:09
krtayloryes, the puppet vs ansible discussion...17:09
sdakekrtaylor go forth and conquer :)17:10
kfox1111devs being in their own world causes lots of other issues. :/17:10
sdakekfox1111 agreed devs view the world through a different lens then opertors17:10
kfox1111yeah.17:10
sdakekfox1111 one of those lenses is kolla-ansible :)17:10
krtaylorit is really only meant to be a test environment, and kolla-ansible could be so much more flexible17:10
inc0so now we have wiring in Kolla to make it work like devstack17:11
kfox1111so if they don't share common deployment tooling, the devs make changes and thinks the world's all rosy, and the ops suffer. :/17:11
inc0https://docs.openstack.org/kolla-ansible/latest/kolla-for-openstack-development.html17:11
sdakekfox1111 the only problem with that argument is there is more then one deployment tool :)17:11
kfox1111yeah. but thats not reson to add yet another deployment tool only for devs.17:12
krtaylorhm, kolla-puppet  :)17:12
* krtaylor ducks17:12
inc0krtaylor: if someone writes that17:12
sdakeinc0 that looks good dude17:12
sdakeinc0 is that implemented?17:12
inc0sdake: at least some of services17:12
inc0and all the wiring17:12
sdakewiring hardest part17:12
inc0so adding other services is quite easy by now17:12
*** harlowja has joined #openstack-kolla17:13
sdakeadding the 50 or so services mostly C&P?17:13
inc0yeah17:13
rwellumsdake: thanks for the note. I will do that. Should I add the 1.6.3 DOA and remove the work-around - or stagger it?17:13
inc0and you get devstack which will not ruin your machine17:13
krtaylor++17:14
inc0and is easy to make HA + used on prod17:14
inc0lots of benefits17:14
inc0but culture of devstack is pretty entrenched and changing that would take a lot of effort17:14
sdakerwellum i'd add 1.6.3 = DOA an remove workaround in one go17:15
rwellumsdake: ack17:15
sdakeinc0 well its not like we force it upon people17:15
sdakeinc0 its more like "bulid it they will come" :)17:16
*** lpetrut has quit IRC17:16
sdakewhat position are we to say that kolla-ansible dev mode is better than devstack17:16
inc0yes and no, it takes effort to show them this option17:16
sdakeok..  I know devstack destroys your system - but still :)17:16
inc0it's prod ready17:16
inc0it's easy to test on multinode scenario17:16
inc0it's much faster to deploy:)17:17
inc0there are lots of benefits17:17
sdakesounds like a great summit talk to me17:17
krtayloragreed, it would be interesting to present a functional comparison...sydney?17:17
krtaylorhehheh17:17
sdakei'd present it but haven't the faintest idea how it works17:17
inc0indeed it does, pbourke should be one on stage tho, it's his child17:17
sdakekrtaylor i thiink presenting comparisons needlessly creates conflict17:17
sdakeinc0 cool - hope he will make it to syndey17:18
sdakewe should probably see if that is the case so we can et a talk submitted17:18
krtaylorwell, I was thinking showing how similar it could be would help the "argument"17:18
sdakekrtaylor you used   the word right there - argument17:18
sdake;)17:18
sdakeno reason to argue over it17:19
krtaylorI'd implement it in our CI and show as a PoC..alas, if I had 3 more me17:19
rwellumgit clone krtaylor17:20
rwellumdammit - didn't work..17:20
krtaylorlol, I wish17:20
krtaylorwell, thanks for the discussion, I'll see if I can drum up a set of folks interested in kicking in a bit of time via email17:20
inc0would be awesome krtaylor17:21
inc0I think changing culture of devs to use more opsy prod-like env would be beneficial for OpenStack as whole17:21
rwellum+1 inc017:21
inc0if we figure out how to easily do multinode code editing (nfs mounted to /opt/stack?) that would be even better17:22
krtayloryeah, thats whats really driving my interest, that and the time factor, it would really speed up our testing17:22
inc0"here run this 3 commands and you'll have 3 node HA openstack for your development pleasure"17:22
rwellumI've been using my tool to do the same for kolla-k8s environment, and there's a couple of folks from Cinder looking at it now too.17:22
rwellumContainers invalidate the need for dev-stack imo.17:23
sdakewell lets not go on an all out offensive against devstackc plz :)17:23
sdakethey are our friends17:23
inc0also as I said, lots of devs uses devstack daily and switching to something else will slow down their dev process17:24
rwellumTBF - containers are doing that to a lot of technologies.17:24
inc0at least initially17:24
rwellumIt's one step backwards to go two steps forward inc0 :)17:25
*** jtriley has quit IRC17:25
inc0also, while we have it written, I'm sure when we actually be used for dev lots of issues with dev workflow will appear, it'll take us time to get it all right17:26
inc0so what we need today is volunteers to try it out and develop openstack on kolla-ansible instead of devstack17:26
inc0and see how successful they are17:26
sbezverkkfox1111: With help of Intel developer, we got multi networks POD in kubernetes running :) has 3 interfaces,  1 - management (flannel), 1 - mapped to Linux bridge, 1 - SRIOV VF interrace17:26
kfox1111sbezverk: nice. :)17:26
kfox1111hows ip allocation done on the secondary interfaces?17:27
inc0sbezverk: who was that?17:27
sbezverkinc0: dude from Ireland Kural17:27
inc0interesting, I don't know him17:27
sbezverkkfox1111: each interface has its own ipam17:28
inc0Intel is big...17:28
kfox1111that would allow for the elimination of yet another use case of vm's over containers. :)17:28
kfox1111sbezverk: cool.17:28
sbezverkfor sriov ipam is optional, basically it is layer 2 pipe17:28
kfox1111optional would be cool.17:28
kfox1111is it optional for bridge too?17:28
inc0bbiaf17:29
sbezverkkfox1111: not sure, will need to test it17:29
rwellumsbezverk and sdake - in the deployment guide, should I bump the k8s version recommended to 1.6.4 or 1.7.0?17:31
kfox1111has anyone pushed the gate to 1.7.0 yet?17:31
*** hieulq has quit IRC17:33
*** daidv has quit IRC17:33
*** duonghq has quit IRC17:33
rwellumsbezverk: mentioned it was working in the gate17:33
*** unicell has joined #openstack-kolla17:34
sdakerwellum 1.6.1 and 1.6.2 work17:34
sdakerwellum i'd just != 1.6.317:34
*** ducttape_ has quit IRC17:34
sdakekfox1111 been recovering from a couple falls - so bsically hands off the laptop for aout 3 weeks17:35
sdakeresponding to mail and doing the minimum :)17:35
jamesbensoninc0, sorry I was out for a bit, did you check the recheck?  all but one17:35
sdakekfox1111 i dont have any idea the status of the k8s gate17:35
*** krtaylor has quit IRC17:35
kfox1111ok. we shoudl submit a patch that bumps the version to 1.7 and see what happens.17:35
jamesbensoninc0, failed with a no space left on device in the gates?17:36
sbezverkkfox1111: I check gate logs and it uses 1.7.017:36
*** vhosakot has joined #openstack-kolla17:37
*** jtriley has joined #openstack-kolla17:38
*** manheim has joined #openstack-kolla17:41
rwellumsdake: I think it's better to bump the min. revision past 1.6.3 (it's currently 1.6.2) - that way we avoid mentioning 1.6.3 at all. If we are DOA/EOL 1.6.3,17:42
rwellumsbezverk: yes because yum install just pulls in latest17:42
sdake1.6..1 and 1.6.0 also work i think17:42
rwellumsdake: is there any advantage to using it though?17:43
sdakerwellum if you want to state 1.6.4 as a minimum that wfm - bt pleaes indicate 1.6.3 is DOA17:43
*** itlinux has quit IRC17:43
rwellumok sdake17:43
sdakei know it is duplicative, however, we shouold be extremely clear 1.6.3 is not to be used17:43
sdakeinc0 i'm going to have a difficult day ahead17:45
sdakeand i had wanted to fix docs.oo17:45
sdakebut i dont have time17:45
sdakecan you check out the recent comments on openstack-infra17:45
sdakeand take it on?  it looks like less then 30 mins of work17:45
*** manheim has quit IRC17:45
sbezverkrwellum: I would say 1.6.6 or 1.7.0 is good for the doc17:47
*** daidv has joined #openstack-kolla17:48
*** duonghq has joined #openstack-kolla17:48
*** hieulq has joined #openstack-kolla17:48
sdakesbezverk i think we want to set the minimums we know work17:49
sdakewe know 1.6.4 works correct?17:49
rwellumSo right now I have 1.6.4, I explicitly mention 1.6.3 is DOA and added a note saying current version 1.7.0 is stable.17:50
sdakeduonghq can you check out the openstack infra logs rom about 20 mins ago (search for sdake) and fix docs.oo plz17:50
sdakeI'd do so, however, have meetings17:50
sdakeduonghq and our docs are not published properly17:50
sdakewhich is a big problem requiring immedite attention17:50
sdakeduonghq hi btw :)17:51
sdakeor nayone else out there :)17:52
sdakeplz17:52
*** eaguilar has joined #openstack-kolla17:56
*** ducttape_ has joined #openstack-kolla18:01
*** ducttape_ has quit IRC18:02
*** ducttape_ has joined #openstack-kolla18:03
*** dave-mccowan has joined #openstack-kolla18:03
*** krtaylor has joined #openstack-kolla18:12
*** jascott1 has joined #openstack-kolla18:15
jascott1sdake what up!18:15
jascott1114F yikes18:16
*** lpetrut has joined #openstack-kolla18:20
*** lpetrut has quit IRC18:22
*** krtaylor has quit IRC18:28
*** lpetrut has joined #openstack-kolla18:29
*** krtaylor has joined #openstack-kolla18:33
*** eaguilar has quit IRC18:34
*** jbadiapa has quit IRC18:45
kfox1111jascott1: sorry, I havent responded to you yet. been super busy trying to fix some broken servers for the physics mc right now.18:45
kfox1111hopefully will be able to get back to you later today.18:45
jascott1kfox1111 not a problem, i know you are busy busy18:47
*** tonanhngo has joined #openstack-kolla18:49
*** eaguilar has joined #openstack-kolla18:51
*** hieulq has quit IRC18:51
*** daidv has quit IRC18:51
*** duonghq has quit IRC18:51
*** cliles_ has quit IRC18:52
*** cliles has joined #openstack-kolla18:52
jamesbensonmanheim_: ping18:55
*** itlinux has joined #openstack-kolla18:58
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/47465219:01
*** Pavo has joined #openstack-kolla19:03
*** daidv has joined #openstack-kolla19:06
*** hieulq has joined #openstack-kolla19:06
*** duonghq has joined #openstack-kolla19:07
*** pcaruana has quit IRC19:12
*** ducttape_ has quit IRC19:21
*** ducttape_ has joined #openstack-kolla19:22
*** ducttape_ has quit IRC19:24
*** ducttape_ has joined #openstack-kolla19:24
*** ducttape_ has quit IRC19:31
*** vhosakot has quit IRC19:32
*** Pavo has quit IRC19:34
*** rwallner has quit IRC19:35
dmsimardsdake: what do you guys need epel for on el7 ?19:36
sdakedmsimard iscsi package of some sort19:36
sdakedmsimard there may be a few other packages19:36
sdakedmsimard i'd really like to remove epel19:36
sdakedmsimard i don't have the fortitude to get the rpm packaging done in rdo19:37
dmsimardsdake: fwiw RDO has successful deployments with zero dependencies on EPEL19:37
dmsimardsdake: and I believe openstack-ansible has done away with their EPEL dependencies as well19:37
sdakedmsimard i hear y - perhaps our iscsi dependency can be changed19:37
dmsimardour dependencies have been self contained since at most liberty19:37
sdakedmsimard i am not a cinder expert19:37
sdakei can barely get it to wor19:37
sdakek19:37
sdakeunless using ceph19:37
sdakeceph cinder is a nobrainer19:37
sdakeceph + lvm = painful19:37
sdakerather vm+cinder = painful19:38
sdakeits the lvm+cinder that require the scsi package iirc19:38
sdakedmsimard thanks for the note that openstack-ansible moved to centos19:38
sdakedmsimard I didn't know that :)19:38
dmsimardsdake: they didn't move, they just support centos as well as ubuntu19:38
*** rwallner has joined #openstack-kolla19:38
sdakesorry i mispooke thats what i meant19:39
dmsimardThey still deploy from source as usual, and they use RDO for dependencies19:39
sdakea few years ago i asked them to do that19:39
sdakethe response cn't be repeated :)19:39
dmsimardSomeone showed up to do the work :p19:39
sdakealways the key factor19:39
dmsimardI think they're happy about it19:39
sdakesomeone has to show up to do the work19:39
sdakewhich begs  the question, are you showing up to do the epel removal work :)19:40
sdakethe actual removal of epel is trivial - 10 mins of work - the dependencies are a different matter19:40
dmsimardsdake: I don't even have the bandwidth to do the things I need to do :P19:41
*** ducttape_ has joined #openstack-kolla19:41
dmsimardsdake: but I do have the time to point it out19:42
sdakedmsimard so what can I hep you with19:42
dmsimardsdake: there actually is something now that I think of it19:42
sdakethe other thing i want to do is move to mariadb inside rdo19:42
dmsimardsdake: it already is19:42
sdakenot in kolla19:42
dmsimardsdake: I think you mean xtradb or percona19:42
sdakewe are using mariadb.org19:42
sdakextradb is integrated in rdo already19:42
dmsimardsdake: mariadb is already in RDO mirrors, not sure what else you need19:43
*** rwallner has quit IRC19:43
dmsimardhttp://mirror.centos.org/centos/7/cloud/x86_64/openstack-ocata/common/mariadb-server-galera-10.1.20-1.el7.x86_64.rpm19:44
sdakedmsimard i was just telling you the two changes i want to make to the structure of the kolla containers19:48
sdakeremove epel - integrate rdo's version of mariadb19:48
dmsimardsdake: oh19:48
sdakei cannot remove epel without either a) help on cinder implementation without iscsi or b) addition of deps to rdo19:49
sdakemariadb different matter - that is https://review.openstack.org/#/c/468632/19:50
larskssdake: which iscsi tools are in epel that aren't in centos?20:00
sdakelarsks there is one way to find out20:01
sdakelarsks you will have to give me a bit to build the images20:01
dmsimardlarsks: scsi-target-utils ?20:01
dmsimardfrom looking at the cinder-volume dockerfile..20:01
dmsimardhttps://github.com/openstack/kolla/blob/master/docker/cinder/cinder-volume/Dockerfile.j220:02
larsksdmsimard: ah, thanks, I was looking for iscsi-related packages and wasn't spotting any.20:02
larsksTricky package naming :)20:02
sdakei'm not sure if that is the only missing dependency we have20:02
sdakeand i'm not sure if that dependency is actually necessary for correct operation20:02
sdakelast i looked at it was june20:03
larskssdake: scsi-target-utils is what provides "tgtd", so that looks pretty necessary.20:03
sdaketgtd required for cinder?20:04
larskssdake: I'm looking at the cinder-volume Dockerfile, which appears to be configuring tgtd.20:04
sbezverkfor iscsi-lvm to work we need tgtd, iscsid, targetcli20:04
sbezverkand iscsiadm20:04
*** manheim has joined #openstack-kolla20:10
dmsimardsdake: btw would love to land https://review.openstack.org/#/c/479881/ - it'd be super useful for us.20:11
openstackgerritRich Wellum proposed openstack/kolla-kubernetes master: Update deployment guide to use 1.6.4.  https://review.openstack.org/48121820:11
dmsimardlarsks: my knowledge of cinder w/ iscsi+lvm is not exactly awesome but we don't require epel in puppet integration tests where it's setup20:12
sdakelarsks i am now building all of kolla (with source) to see what fails to build20:13
sdakenote source shuld build entirely20:13
dmsimardlarsks: https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/cinder.pp#L1720:13
sdakealthough i took out epel in the proess20:13
sdakeso we can see what deps are missing20:14
sdakedmsimard enjoy20:16
dmsimard\o/20:16
*** ducttape_ has quit IRC20:19
sdakedmsimard either that manfiest doesn't work with rdo or kolla is using the wrong or outdated dependencies20:19
sdakedmsimard i had a question related to the skip thing20:20
sdakedmsimard you said it would be useful - I am struggling to understand the use case20:20
sdakedmsimard mind sharing20:20
*** daidv has quit IRC20:21
*** hieulq has quit IRC20:21
*** duonghq has quit IRC20:21
larsksdmsimard: it looks like the puppet integration tests are using 'targetcli', which is in base centos.20:21
larsksSo that's probably the solution for configuring the cinder iscsi backend without requiring epel.20:22
*** iniazi_ has joined #openstack-kolla20:23
sdake@lalarINFO:kolla.image.build.base:No package scsi-target-utils available.20:23
larskssdake: as expected...20:23
jamesbensonfor cherry picking, you just mention 4.0.2 or stable/ocata?20:24
sdakehere is the mitaka install guide: https://docs.openstack.org/mitaka/install-guide-rdo/cinder-storage-install.html20:24
openstackgerritJames Benson proposed openstack/kolla-ansible stable/ocata: Adds NTP sync process to account for ceph time sync requirements. Removes precheck portion of NTP. Corrects for redhat Fixes typo  https://review.openstack.org/48122420:26
jamesbensonstable/ocata, got it20:26
sdakerdo instal guide: https://docs.openstack.org/ocata/install-guide-rdo/cinder-storage-install.html#install-and-configure-components20:26
*** iniazi has quit IRC20:27
*** ducttape_ has joined #openstack-kolla20:27
sdakelarsks i just asked in #openstack-cinder what the correct solution from their POV is20:28
larsksI am pretty sure, based on https://github.com/openstack/puppet-cinder/blob/master/manifests/backend/iscsi.pp#L105, that the answer is "use targetcli".20:28
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Allow tmp path to download registry  https://review.openstack.org/47594020:31
sdakelarsks sbezverk https://review.openstack.org/#/c/474999/20:31
*** mandre_away is now known as mandre20:32
sdakelarsks the next question is how is targetcli configured20:35
sdakeatm we hve a tgtd.conf file which works (or worked) correctly20:35
sdakelarsks keep in mind i dont parse puppet20:36
*** daidv has joined #openstack-kolla20:36
sbezverksdake: if this ps committed I think it will break kolla, as some binaries will be missing..20:36
larskssdake: there's no time like the present to learn :)20:36
*** duonghq has joined #openstack-kolla20:36
sdakelarsks i've got about 10000 things to leran20:37
*** hieulq has joined #openstack-kolla20:37
sdakelarsks puppet is about 9999 on that list20:37
larsksTrue dat.20:37
sbezverkhere the test is only to build container, but no tests confirming sucessful deployment20:37
sdakethe gates deploy - however - they dont test lvm20:38
sbezverksdake: Above I listed binaries which must be present for iscsi-lvm to work20:38
sdakesbezverk let me read scrollback20:38
sdakelarsks does tripleo deploy iscsi-lvm?20:40
sdakesbezverk iscsi-lvm is used as a dep for cinder20:40
sdake?20:40
larskssdake: I don't know for certain, but the default is probably ceph.20:40
sdakelarsks is there any way to look at   that puppet wizardlly and find out?20:41
sdakedefualt is ceph in kolla as well20:41
sdakewe lack kolla-ansible gates for it20:41
larsksI mean, tripleo probably consumes that puppet-cinder module, which would deploy iscsi using targetcli if passed the right options.20:42
larsksI'll see if I can figure that out over the next few days.20:42
sdakelarsks the question is  - does that deployment work :)20:42
*** ansmith has quit IRC20:42
sdakesbezverk is iscsi-lvm gated in kola-kuberntes20:43
sbezverksdake: if scsi-target-utils is removed then we will not have tgtd and tgtdadm binaries and it will break lvm20:44
*** vhosakot has joined #openstack-kolla20:44
*** jbadiapa has joined #openstack-kolla20:44
sbezverksdake: yes, we test lvm, but we use 3.0.0 and 4.0.0 images20:44
*** eharney has joined #openstack-kolla20:46
eharneysdake: hi20:46
*** jtriley has quit IRC20:46
sdakesbezverk eharney can you fellas discuss iscsi + lvm + cinder or whatever problem we  are trying to solve here:)20:46
sdakeeharney just to catch you up - end goal is to remove epel20:47
sdakeiscsi-target-utils is not in rdo (nor rhel7)20:47
eharneygreat20:47
sbezverksdake: we do not have problem, I see iscsi-lvm is working perfectly fine in kolla-k8s20:47
sdakesbezverk has indicated binaries from these packages are required in order to operate lvm20:47
eharneyso always use lio when on el7 and then scsi-target-utils is not needed20:47
sdakesbezverk if you take out epel - the containers do not build20:47
sbezverksdake: then do not take it out20:48
sdakesbezverk itt really must be done - epel is sloppy and error prone20:48
sdakesbezverk the alternative is to add iscsi-target-utils to epel20:48
sdakesbezverk what id really like to understand is if there is a htird solution20:48
sbezverksdake: the goal is to have functional software and not some compliancy to some rules ;)20:48
sdakethat involves using lio exclusviely whatever that is20:48
sdakesbezverk this is about cross-community collaboration20:49
sbezverksdake: we cannot use 'lio'20:49
sdakesbezverk cool - expand plz?20:49
sbezverkout code does not handle this helper at all20:49
sbezverksdake: at the time when I put together iscsi support for kolla, the only option avaialbe for centos was tgtdadm as helper20:50
sdakepardon ignorance, could it be made to work with lio20:50
sbezverklio was purely rh thing20:50
sdakelio is kernel module right?20:50
eharneyyes20:50
sdakeso not a rht thing?20:50
sbezverksdake: possibly, but somebody need to implement it before removing tgtdadm20:50
sdakesbezverk agree on order of operations20:51
eharneyit works on Ubuntu -- the Cinder LIO check job runs it on Ubuntu20:51
sbezverksdake: if folks who wants to remove tgtdadm is ready to implement it, then great!!20:51
inc0rwellum: around?20:52
sdakesbezverk you have any idea where to begin that process20:52
sbezverksdake: for now I would ask inc0 to put -2 on this change20:52
rwelluminc0: yes20:52
sdakesbezverk (file link)20:52
inc0sbezverk: which change?20:52
sdakesbezverk the change isn't merging if it breaks deployment20:52
sdakeno need to -2 it20:52
inc0yeah please don't break deployment20:52
sbezverksdake: if it is not tested it can get in20:52
sbezverkkolla-ansible does not test lvm20:53
inc0rwellum: so, custom image talk, maybe dockerhub publisher + custom images can be done in same talk20:53
sbezverkso nothing to break20:53
sdakeit can be manually verified20:53
inc0sbezverk: gates are not only deployment20:53
rwelluminc0: +1 don't see why not20:53
inc0also I'd be happy if we could add iscsi to gates too20:53
inc0it should be fairly easy now20:54
*** manheim has quit IRC20:54
*** krtaylor has quit IRC20:54
sbezverkinc0: there is ps which most likely will break iscsi-lvm deployment, I wanted to make sure it is not merged until proven not breaking actual deployment20:55
sdakei think the expectaton of someone adding a gate to kolla-ansible is a bit high for someone new to the code20:55
sdakesbezverk i will leave a big note in the review stating it shoudl not merge without confirmation20:55
sdakethat will do teh rick20:55
inc0ok, who volunteers to check patch? I don't have Centos around20:55
vhosakotinc0 rwellum I like the idea to combine.. so, can we cover dockerhub publisher from gate, users pulling tested images from dockerhub and deploying kolla, with custom images - all in 45 mins to 1 hour?20:55
inc0can someone *please* test it out and find whether or not it breaks deploy?20:56
inc0and after that we'll focus on writing iscsi gates20:56
vhosakotinc0 sbezverk: which patch, I have a Centos box.20:56
*** lucasxu has quit IRC20:57
inc0we don't release stable today, we have time to fix it in case something breaks and if skips our initial tests20:58
sdakeinc0 i know for certain it wont work - it requries both ansible and kuberntes implementation changes20:58
inc0sbezverk sdake dmsimard here's my plan, we remove iscsi-target-utils and move it to alternative (whatever that is) now20:58
inc0well, if we'll lose epel, then we need this change20:59
inc0also having full repo just for single package which can be replaced20:59
inc0is bad imho20:59
sdakewe arent losing epel20:59
sdakewe want to lose epel :)20:59
sbezverkinc0: -1 it will break 50% of kolla-k8s gates20:59
sdakewe can't blindly merge the patch as it20:59
sdakeas is20:59
sdakeit ust be tested20:59
sdakekolla-kubernetes implementation must be changed21:00
inc0sbezverk: then we need to fix it there too, if epel is going to be less maintained it's ticking bomb21:00
sdakekolla-ansible implmeentation needs changing21:00
*** vhosakot has quit IRC21:00
sdakeand THEN it can be tested21:00
inc0I'd rather fix it before it's critical21:00
rwellumvhosakot: I agree. Now will it include a demo? If it does then demoing both ansible and k8s might be overkill - but the process will be different. How should we do that in 45m?21:00
sbezverkinc0: it is wrong approach, to break something before a tested alternative solution is implemented21:00
*** vhosakot has joined #openstack-kolla21:00
sdakesbezverk i'm petty sure nobody said to do that?21:01
inc0sbezverk: I assume it will be tested when implemented...21:01
inc0before21:01
inc0I mean21:01
sbezverkinc0: if somebody implements lio support, great let's drop tgtd, but before that it should be there21:01
inc0we can't block this change because we don't have gate for it in ansible21:01
sdakeinc0 you miss sbezverk 'z point21:01
sdakekolla-kubernetes gates extensively test this code path21:01
sbezverkinc0: I can guarantee you it will break21:01
sbezverkas important binaries will be missing21:02
sdakeso even if it breaks lvm - which we may or may not cre about - it will break the gates dramatially21:02
sdakewe do care about breaking gates imo :)21:02
inc0ok, but as far as I understand we need to get rid of this package because epel has issues right?21:02
sdakeyes and step 1 in getting rid of the package is implementing support  for the otehr "heper" whatever that may be :)21:02
jgriffithsdake what do you mean you "may not care about"21:02
jgriffiththat's pretty lame21:03
inc0is there any way to have both at the same time21:03
inc0we care about iscsi jgriffith21:03
inc0don't worry21:03
jgriffith:)21:03
inc0we do care and because we care we're having this discussion21:03
jgriffithsdake sbezverk I can help you all with the LIO stuff if you want but I don't think it's even remotely critical for you to worry about or deal with21:03
jgriffithand the reality is that most are probably using another backend anyway in production with Kolla21:04
inc0jgriffith: I think what's critical is that scsi-target-utils in epel is going to be issue in future21:04
sdakejgriffith poor choice of wording - its a matter of priorities -when our gate break - all development stops - if lvm is broken for a little while - its not critial immediately21:04
sbezverkjgriffith : not so sure, last time I tried using lio as iscsi helper nothing worked21:04
jgriffithinc0 yeah, I get that for sure.  But that's not like in the next month or something is it?21:04
jgriffithinc0 on the bright side it's almost a direct replacment of how tgt works and is setup21:05
inc0well I don't want to wait till it happens, I'd rather have it fixed, but we're not on fire21:05
eharneyLIO has been the only iscsi helper used on RHEL with LVM for years, i think it works21:05
jgriffithso it shouldn't be a terrible effort, although Ubuntu can be troublesome21:05
eharneywe also have a gate job that tests it on Ubuntu...21:05
jgriffitheharney I certainly am not disputing that, it works great!21:05
inc0is there any way to have both in image for transition time?21:05
sbezverkinc0: all I ask is to let that PS merge only after lio is tested and proven to be working..21:05
sdakesbezverk that is non-negotiable21:06
jgriffitheharney I just point out Ubuntu because I do run into weirdness sometimes with the lib intiializing properly on install21:06
inc0sbezverk: ofcourse. That should go without saying21:06
jgriffithbut I'm agreeing with you21:06
jgriffitheharney ^^21:06
inc0I think we're in violent agreement21:06
jgriffithLIO is certainly a better option in many respects21:06
eharney:)21:06
inc0yes we want to replace package and ultimatelyu remove epel21:06
sdakeok - well i think we were in agreement when this conversation started21:07
jgriffithperformance, flexibility etc21:07
inc0and yes we want to keep both gates and regular deployments happy21:07
sdake the question is how to implement the support (say starting from kolla-ansible)21:07
inc0my question is, rolling upgrade while switching these libs will work right?21:07
jgriffithsdake provide an option?21:07
jgriffithsdake and add a second gate job?21:07
inc0it won't affect communitaction protocol in any way?21:07
sdakejgriffith i'm thinking more along the lines of "how is it configured"21:07
jgriffiththen switch the default next cycle?21:07
inc0yeah jgriffith we want iscsi gate, shouldn't be hard to do21:08
jgriffithsame way you do drivers in Kolla today21:08
eharneyrolling upgrade from scsi-target-utils to LIO on a live deployment will not work21:08
jgriffithcinder drivers that is21:08
inc0eharney: ok that's bigger issue then21:08
sbezverkjgriffith : it is not as simple as you try to show ;) we have two containers which helps with what tgtdadm does21:09
inc0we need to figure out upgrades before making switch21:09
jgriffitheharney BTW, remind me to ask you back in Cinder how RHEL is going to deal with that problem21:09
sdakelarsks - can o worms dude :)21:09
sbezverkit would require reworking these containers to support lio21:09
jgriffithinc0 the good news is it's easier in Kolla than it is in traditional BM deployments21:09
jgriffithstill not trivial, but not as difficult21:09
inc0jgriffith: correction, it's possible with kolla and it's often not for BM21:10
sdakeinc0 is there any way to turn off the "it didn't build abort" feature of kolla-build21:10
inc0that being said, we're keeping upgrades.21:10
*** mandre is now known as mandre_away21:10
jgriffithinc0 tomateo/tomato :)21:10
inc0yeah, well, if that breaks upgrades, I'll hold on till we figure out how to not break upgrades21:10
sdakedo we really expect people will be running with iscsi lvm in production21:11
inc0lvm no, iscsi definetly21:11
sdakei dont know21:11
sdakeits an honest question21:11
inc0netapp and others use iscsi21:11
inc0many enterprise properitary storage solutions uses iscsi protocol21:11
sdakei got it21:11
inc0so it defintely has to stay working21:12
sdakeis the problem here LIO or lvm or a combo thereof21:12
inc0well by default I'll assume yes21:12
*** rhallisey has quit IRC21:12
inc0unless proven that it's not importnant21:12
inc0also there is intermediate fix21:13
inc0dockerfile customizations21:13
sbezverksdake: lio is alternative iscsi helper21:13
inc0if someone *can't* for some reason use scsi-target-utils, they can build images without it21:13
sdakeok - so this has nothing to do with lvm then?21:13
sbezverkcurrently we use tgtdadm helper21:13
inc0while it's not long term, it's possible until we figure out good upgrade strategy and gate migration21:14
sbezverksdake: the only implementation we have for lvm is with iscsi21:14
inc0sbezverk: I think question was, netapp+iscsi will be affected right?21:15
eharneysdake: it's just the context of the lvm driver -- which uses one of these iSCSI helpers to handle serving the LVM storage21:15
eharneythese helpers aren't used with other cinder iscsi-based drivers21:15
inc0also jgriffith feel free to correct me if I'm wrong about protocols, I'm storage ignorant21:15
sbezverkinc0: I have no idea, I do not think we even implemented netapp support21:15
sdakeok - so this is lvm only21:15
sdakedo people deploy with iscsi ilo with lvm in production21:16
inc0if it's lvm only then I'm less concerned21:16
inc0still wouldn't like to break it if possible21:16
sdakeinc0 right - my queestion was more around upgrades rather than functionality21:16
sdakei htink you throw upgrades into the mix on a feature virtually nobody uses - pointless waste of effort21:16
inc0yeah, same here, "break it during upgrade" is what I meant21:16
sdakeof course i make some assumptions21:16
sdakei have no ide aif fpeople use this at all21:17
sbezverkI can look into it from kolla-k8s perspective when I understand the difference between tgtd and li approaches21:17
inc0sdake: I'd be very careful in saying who does and who doesn't use it21:17
sdakeinc0 i hve no idea21:17
*** manheim has joined #openstack-kolla21:17
inc0sbezverk: would you plz? I mean same problem will be with ansible21:17
inc0you know by far most about our implementation of iscsi21:17
sbezverkinc0: sure but I will need lots of help from jgriffith21:18
inc0I'd be much happier person if I'd know you're looking at this21:18
inc0jgriffith: will you help your fellow stackers please?21:18
inc0sbezverk: let me know what do you find. If we can seamlessly get rid of epel that's great21:19
sdakei would d othe work yself if I had any idea hwo to setup kolla-ansible with lvm with iscsiadm21:20
sdakeor whatever magic incantation of stuff is needed to see how it works21:20
inc0we have docs sdake ;)21:22
sdakecool21:22
mgkwillCongrats sean-k-mooney on https://review.openstack.org/#/c/342354/ merge! Long time coming :)21:22
sdakethat woudl be great if the docs were on docs.oo :)21:22
inc0yeah year without few days21:22
inc0in queue21:22
inc0;)21:22
inc0and merged21:22
mgkwillinc0: '21:23
mgkwillinc0:  ;)21:23
mgkwillwhoops21:23
inc0hmm I thought we did have these docs21:23
inc0https://docs.openstack.org/kolla-ansible/latest/cinder-guide.html yes yes we did21:24
vhosakotwasn't there a nova patch like 2 years or something and finally got merged.. lol21:25
inc0I think that's most of nova patches21:25
*** ducttape_ has quit IRC21:28
*** leseb has left #openstack-kolla21:28
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role docs  https://review.openstack.org/48128921:29
*** lpetrut has quit IRC21:30
*** ducttape_ has joined #openstack-kolla21:33
*** krtaylor has joined #openstack-kolla21:35
*** japestinho has quit IRC21:36
*** itlinux has quit IRC21:40
*** eharney has left #openstack-kolla21:40
sdakere the iscsi packaging and EPEL - if nobody else is goingto work on it I guess I'll take a crack21:41
sdakeit would be hepful if someone could throw me the slightest bone on how to configure the thing21:41
*** itlinux has joined #openstack-kolla21:42
sdake(with kolla-ansible)21:42
sdakeso I have something to test against21:42
*** eaguilar has quit IRC21:48
*** jamesbenson has quit IRC21:50
*** jamesbenson has joined #openstack-kolla21:52
*** hieulq has quit IRC21:53
*** daidv has quit IRC21:53
*** duonghq has quit IRC21:54
*** jamesbenson has quit IRC21:57
*** dave-mccowan has quit IRC22:05
*** schwicht has quit IRC22:06
*** ducttap__ has joined #openstack-kolla22:08
*** daidv has joined #openstack-kolla22:08
*** duonghq has joined #openstack-kolla22:08
*** hieulq has joined #openstack-kolla22:08
*** ductta___ has joined #openstack-kolla22:11
*** ducttape_ has quit IRC22:11
*** ducttap__ has quit IRC22:13
*** iniazi_ is now known as iniazi22:16
*** jascott1 has quit IRC22:18
*** jascott1 has joined #openstack-kolla22:18
*** tonanhngo has quit IRC22:19
*** goldyfruit has quit IRC22:21
*** jascott1 has quit IRC22:22
*** goldyfruit has joined #openstack-kolla22:24
*** schwicht has joined #openstack-kolla22:28
*** tonanhngo has joined #openstack-kolla22:32
*** tonanhngo has quit IRC22:37
openstackgerritMerged openstack/kolla-ansible master: Reinstate [DEFAULT] firewall_driver option  https://review.openstack.org/47931322:44
openstackgerritMerged openstack/kolla-ansible master: Add VMware NSXV support to neutron  https://review.openstack.org/45009222:46
openstackgerritMerged openstack/kolla-ansible master: Add lbaas support to vsphere by vmware-nsx plugin  https://review.openstack.org/46284422:46
*** schwicht has quit IRC22:56
inc0sdake: in change with scsi package I described upgrade scenario that needs testing too23:00
sdakei really think dealing with upgrades or something nobody usess is overkilll23:00
inc0I think assuming nobody uses this is premature23:00
inc0while I dislike lvm myself, I know for a fact that people use it23:01
sdakeisnt' it single node23:01
inc0well data doesn't have HA on it's own23:01
sdakelvm is hosted on the node that runs openstack23:02
sdakesingle node23:02
inc0but there are cases when it's not big issue23:02
sdakethe only real use csae i see is development23:02
inc0negative, you can have many lvm nodes23:02
inc0just data won't get replicated23:02
sdakehow is that useful?23:02
inc0if you for example have database23:02
inc0and database has replication built into it23:02
inc0then you don't care that much23:02
inc0about replication on storage level23:02
inc0as in, if your app handles HA, you don't need storage HA23:03
sdakehow is lvm + iscsi used in a production deployment23:03
sdakeyou sid people use it23:03
sdakei'd like ot know how23:03
inc0I just pointed you to one use case23:04
inc0put db on it23:04
sdakeok let me translate so i understand23:04
inc0and handle replication within db23:04
sdakeyou put lvm on 20 nodes23:04
sdakeyou have those 20 nodes managed by cinder23:04
sdakethe 20 nodes provide block storage23:04
sdakeis that how its used23:04
sdake?23:04
inc0yes23:04
inc0when you say "I want 20gig of volume"23:04
inc0cinder scheduler will pick one of 20 nodes and make lvm volume with 20gig on it23:05
inc0and connect it with iscsi23:05
inc0if this node dies, your volume is dead23:05
inc0and data is lost23:05
inc0but if you have 3 volumes on 3 different nodes and database cluster uses it23:05
inc0one dead volume will not hurt you because database has 2 more replicas23:06
inc0so yeah, if database handle replication, iscsi+lvm makes sense23:06
inc0I'm nto saying it's best of ideas, but it's not horrible idea too23:06
inc0certinaly less complex than running ceph23:07
*** Pavo has joined #openstack-kolla23:08
inc0makes sense sdake ?23:10
sdakebest practice in this situation is to ping the operator list and ask if they really care for upgrades for this scenario23:11
sdakeupgrade sounds impossible23:11
inc0I'm sure we can think of something23:11
kfox1111oh, cool. k8s 1.7 statefulset has   podManagementPolicy: "Parallel"23:12
kfox1111makes it much more like a deployment.23:13
kfox1111but with the nice naming / optional volume support stuff.23:13
*** kbyrne has quit IRC23:14
kfox1111heh. I'd say its a lot less complex to deploy ceph then to try and make it such that cinder volumes with iscsi lands in just the right way to keep a cluster stable.23:14
inc0kfox1111: agree, I'd run ceph too, but there are people who don't23:15
kfox1111you only need a non clustered mysql and a ceph. the other you need to setup the iscsi volume pools in such a way as users can ask for volumes on particular ones,23:15
kfox1111setup a mysql cluster, carefully allocating a volume per, etc.23:15
kfox1111yeah, thats fine, but just saying, they cant make the "its simpler" argument.23:15
kfox1111cause, yeah, no. its not. :)23:15
inc0it's simpler to understand and dig through23:16
inc0it gets worse to handle later23:16
vhosakotI always visualize LVM as a software-based way to _spread_ data on multiple physical hard disks, replicated or not, the replication part is added by the storage backend being used with LVM.23:16
kfox1111heh. digging through lvm/iscsi/tgtd/etc,23:16
inc0but still yeah, people do that23:16
kfox1111its not simpler then ceph. ;)23:16
inc0ok ok we both are ceph fans, I'm poor devils advocate here:(23:16
*** manheim has quit IRC23:17
vhosakotwell, ceph has given me many lovable sleepless nights :)23:17
kfox1111vhosakot: yeah. and ceph is just the same, but across multiple hosts too. :)23:17
*** manheim has joined #openstack-kolla23:17
kfox1111yeah? I've had the opposite experience.23:17
vhosakotkfox1111: true, I see ceph as the modern cloud-ready way to spread data on disks compared to LVM.23:17
kfox1111cephs been pretty rock solid. lvm, not as much. :/23:17
kfox1111we've built lustres on top of dual attached fiber to try and get ha lustres.23:18
kfox1111it works, but a lot of work to get to work right.23:18
kfox1111ceph by comparison's really simple.23:18
inc0well that being said ceph is not simple to tune in too. It's all rainbows and unicorns until it's rebalancing itself23:19
vhosakotyeah, I did ceph scaling for 5 months, it is awesome! :)  no sarcasm there23:19
inc0can eat a lot of resources23:19
kfox1111inc0: yeah. we experienced some of that. till we got a script from somewhere that is a little smarter about rebalancing.23:19
kfox1111bluestore is looking sweet too.23:19
kfox1111that is the main drawback.23:19
kfox1111lots of resources.23:20
inc0nothing's perfect, still would run ceph23:20
kfox1111we saw something like a 20% performance loss years ago between ceph and iscsi.23:20
inc0wow that's significant23:20
kfox1111we deamed it then to not be worth the overhead of having an iscsi storage class too.23:20
kfox1111the performance hit was fine for our use case and ceph was rock solid.23:21
inc0well probably due to replication23:21
inc0copying data to replicas takes time23:21
*** goldyfruit has quit IRC23:21
kfox1111we didn't do much tuning on it, so the gap could have been lessoned too.23:21
kfox1111yeah.23:21
vhosakotI saw ceph gets angry when we trigger rebalance of near-full OSDs.  Empty OSDs and half-empty OSDs, ceph love to rebalance them.23:21
inc0yeah, cache tiering can help23:21
kfox1111we're doing ssd tiers now, and that really has helped.23:21
inc0that ^23:21
kfox1111the things really fast. :)23:22
*** manheim has quit IRC23:22
kfox1111yeah. the built in balancing needs a little help to not take too much resource.23:22
inc0I need to experiment with erasure coding23:22
kfox1111it was basically unusable until recently.23:22
kfox1111I think luminus has a huge performance tweak in it.23:23
*** goldyfruit has joined #openstack-kolla23:23
kfox1111so ssd tier with ec behind it works well finally23:23
inc0anyway, I gtg23:23
inc0talk to y'all later23:24
inc0o/23:24
vhosakoto/23:24
*** daidv has quit IRC23:24
*** hieulq has quit IRC23:24
*** duonghq has quit IRC23:24
kfox1111l8r23:26
*** manheim has joined #openstack-kolla23:27
*** manheim has quit IRC23:27
*** manheim has joined #openstack-kolla23:30
*** lyang__ has quit IRC23:33
Seyongrwellum: No, still I am stucked in kolla-k8s..23:36
*** hongbin has quit IRC23:38
*** duonghq has joined #openstack-kolla23:40
*** hieulq has joined #openstack-kolla23:40
*** daidv has joined #openstack-kolla23:40
*** bmace has quit IRC23:40
*** bmace has joined #openstack-kolla23:40
*** itlinux has quit IRC23:43
-openstackstatus- NOTICE: nb03.openstack.org has been cleaned up and rebooted, and should return to building rotation23:44
*** awiddersheim has joined #openstack-kolla23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!