Monday, 2019-05-06

*** whoami-rajat has quit IRC00:20
*** k_mouza has joined #openstack-kolla00:48
*** k_mouza has quit IRC00:50
*** bmace_ has quit IRC01:03
*** bmace_ has joined #openstack-kolla01:08
*** radeks has joined #openstack-kolla01:18
*** zhangfei has joined #openstack-kolla01:18
openstackgerritpangliye proposed openstack/kolla-ansible master: modify variable monasca_grafana_server_port  https://review.opendev.org/65729501:25
*** radeks_ has joined #openstack-kolla01:26
*** radeks has quit IRC01:28
*** radeks__ has joined #openstack-kolla01:30
*** radeks_ has quit IRC01:30
*** gmann has quit IRC01:43
*** zhangfei has quit IRC01:59
*** radeks__ has quit IRC02:15
*** hongbin has joined #openstack-kolla02:23
*** kplant has quit IRC02:32
*** JamesBenson has joined #openstack-kolla02:36
*** altlogbot_1 has quit IRC02:41
*** altlogbot_0 has joined #openstack-kolla02:43
*** hongbin has quit IRC03:13
*** whoami-rajat has joined #openstack-kolla03:24
*** altlogbot_0 has quit IRC03:31
*** altlogbot_3 has joined #openstack-kolla03:32
*** gkadam has joined #openstack-kolla03:42
*** gkadam has quit IRC03:42
*** iniazi has joined #openstack-kolla03:51
*** iniazi_ has quit IRC03:52
*** ivve has quit IRC04:12
*** JamesBenson has quit IRC04:17
*** dannins has joined #openstack-kolla04:27
*** Talion has quit IRC04:31
*** igordc has quit IRC04:37
*** iniazi has quit IRC04:55
*** iniazi has joined #openstack-kolla05:07
*** unicell has joined #openstack-kolla05:10
*** ivve has joined #openstack-kolla05:15
*** janki has joined #openstack-kolla05:23
openstackgerritChandan Kumar (raukadah) proposed openstack/kolla stable/stein: Reuse /var/lib/tempest symlink for rpm also  https://review.opendev.org/65731405:26
*** radeks__ has joined #openstack-kolla05:48
*** radeks__ has quit IRC05:56
*** cah_link has joined #openstack-kolla05:57
*** radeks has joined #openstack-kolla06:04
*** skramaja has joined #openstack-kolla06:22
openstackgerritpangliye proposed openstack/kolla-ansible master: modify variable monasca_grafana_server_port  https://review.opendev.org/65729506:34
*** JamesBenson has joined #openstack-kolla06:54
*** JamesBenson has quit IRC06:58
*** iniazi_ has joined #openstack-kolla07:09
*** iniazi has quit IRC07:12
*** altlogbot_3 has quit IRC07:13
*** unicell has quit IRC07:14
*** altlogbot_2 has joined #openstack-kolla07:18
*** yan0s has joined #openstack-kolla07:26
*** tosky has joined #openstack-kolla07:28
*** dmellado has quit IRC07:44
*** dmellado has joined #openstack-kolla07:44
*** _Cyclone_ has quit IRC07:45
openstackgerritpangliye proposed openstack/kolla-ansible master: add masakari_log  https://review.opendev.org/65732907:46
*** tolisbar has joined #openstack-kolla07:50
*** _Cyclone_ has joined #openstack-kolla07:50
hrwmorning07:51
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add kolla-build-debian-source-arm64 experimental job  https://review.opendev.org/55765907:56
hrwmgoddard: my plan is to limit what is built there to fit in 3h time limit.07:59
hrwmgoddard: will look also into enlarging VM it runs on08:00
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add kolla-build-debian-source-arm64 experimental job  https://review.opendev.org/55765908:19
*** pcaruana has joined #openstack-kolla08:24
*** hamzaachi has joined #openstack-kolla08:27
*** ivve has quit IRC08:40
openstackgerritpangliye proposed openstack/kolla-ansible master: add masakari_log  https://review.opendev.org/65732908:51
*** ivve has joined #openstack-kolla08:55
*** altlogbot_2 has quit IRC09:11
*** altlogbot_0 has joined #openstack-kolla09:15
*** altlogbot_0 has quit IRC09:15
*** altlogbot_0 has joined #openstack-kolla09:16
*** zbr is now known as zbr|rover09:18
*** gmann has joined #openstack-kolla09:41
*** tolisbar1 has joined #openstack-kolla09:44
*** tolisbar has quit IRC09:46
*** tolisbar1 is now known as tolisbar09:46
openstackgerritzhufl proposed openstack/kolla master: [Trivial fix]Remove unnecessary slash  https://review.opendev.org/65736609:49
*** gmann is now known as gmann_pto09:57
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: in Debian qemu-kvm is x86 only  https://review.opendev.org/65737010:37
openstackgerritMerged openstack/kolla stable/stein: Reuse /var/lib/tempest symlink for rpm also  https://review.opendev.org/65731410:39
*** skramaja has quit IRC10:40
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add kolla-build-debian-source-arm64 experimental job  https://review.opendev.org/55765910:43
*** tolisbar has quit IRC10:58
*** tolisbar has joined #openstack-kolla10:58
*** dpawlik has joined #openstack-kolla11:03
*** legend96th has joined #openstack-kolla11:07
*** altlogbot_0 has quit IRC11:19
*** altlogbot_0 has joined #openstack-kolla11:20
*** henriqueof has joined #openstack-kolla11:44
*** rafaelweingartne has joined #openstack-kolla12:12
rafaelweingartneHello Guys, I seem to be getting some errors with "kolla-ansible-ubuntu-source-ceph" in "https://review.opendev.org/#/c/656981/", which I am not able to understand12:13
rafaelweingartnecould somebody give me a hand here?12:13
*** iniazi has joined #openstack-kolla12:13
*** kplant has joined #openstack-kolla12:14
*** altlogbot_0 has quit IRC12:15
*** iniazi_ has quit IRC12:17
*** altlogbot_1 has joined #openstack-kolla12:20
*** pcaruana has quit IRC12:29
*** happyhemant has joined #openstack-kolla12:41
*** Talion has joined #openstack-kolla12:45
*** mchlumsky has joined #openstack-kolla12:48
emccormickrafaelweingartne: The bug you referenced talks about updating Ceilometer settings but you're having trouble with a ceph container. Wrong bug reference?13:01
rafaelweingartneactually the right one13:02
rafaelweingartneI wrote that "extension" to add the ability to overwrite the default metrics in ceilometer via Kolla-ansible13:03
rafaelweingartnehowever, when the CI tests were triggered this ceph test keeps failing13:03
rafaelweingartneI checked the logs and it seems a dead lock issue13:03
rafaelweingartnehowever, I have not changed anything in Keystone to cause that13:03
rafaelweingartne2019-05-06 11:29:43.242794 mod_wsgi (pid=23): Exception occurred processing WSGI script '/var/lib/kolla/venv/bin/keystone-wsgi-admin'.13:04
rafaelweingartne (pymysql.err.OperationalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/e3q8)13:04
*** mchlumsky has quit IRC13:05
rafaelweingartnemight be some environment issue in the CI system?13:05
*** mchlumsky has joined #openstack-kolla13:08
*** tolisbar has quit IRC13:13
*** altlogbot_1 has quit IRC13:15
*** altlogbot_1 has joined #openstack-kolla13:16
*** pcaruana has joined #openstack-kolla13:16
*** altlogbot_1 has quit IRC13:21
*** tolisbar has joined #openstack-kolla13:23
*** altlogbot_2 has joined #openstack-kolla13:24
*** altlogbot_2 has quit IRC13:25
*** altlogbot_2 has joined #openstack-kolla13:26
*** legend96th has quit IRC13:47
hrwZuul experimental15:1314:04
hrwkolla-build-debian-source-arm64SUCCESS in 1h 57m 00s (non-voting)14:04
hrwYES!!!14:04
*** janki has quit IRC14:07
*** egonzalez has quit IRC14:22
*** egonzalez has joined #openstack-kolla14:24
kplanti'm trying to deploy ceph with kolla and i'm getting random failures when building OSDs: http://paste.openstack.org/show/750716/14:26
kplantit's a different set of disks each time but the error message looks more like a success message14:27
*** tolisbar has quit IRC14:27
*** Chaserjim has joined #openstack-kolla14:29
*** ykarel|away has joined #openstack-kolla14:30
hrwcan not help - never deployed with ceph14:37
kplantit'd be nice if it just used ceph-ansible :\14:37
hrwkplant: there was idea to go that way14:39
*** ivve has quit IRC14:40
hrw"[WARNING]: Could not match supplied host pattern, ignoring:" - anyone remember/know what it mean and why ansible jobs fail due to that?14:41
kplanti'm trying to make them coexist peacefully, not trivial14:41
kplantif you re-run with -vvv does it tell you which host pattern?14:42
hrwit is CI job - no way to alter14:53
*** tolisbar has joined #openstack-kolla14:57
*** tolisbar has quit IRC15:01
ykarel|awayany cores around? please review:- https://review.opendev.org/#/c/625261/15:02
ykarel|awayso: Could not match supplied host pattern, ignoring happens when ansible inventory don't have hosts against which playbook is run15:04
hrwykarel|away: which looks weird on ci15:06
ykarel|awayhrw, new job?15:07
ykarel|awayor already existing job15:07
hrwykarel|away: https://review.opendev.org/#/c/557659/43 -> http://logs.openstack.org/59/557659/43/check/kolla-ansible-ubuntu-source/d2676c5/15:07
* ykarel|away looks15:08
hrwthx ykarel15:09
ykarel|awayhrw, i could not locate that Warning15:09
ykarel|awayme looking http://logs.openstack.org/59/557659/43/check/kolla-ansible-ubuntu-source/d2676c5/job-output.txt.gz15:10
ykarel|awaycan you link me at what place u see warning15:10
ykarel|awayCould not match supplied host pattern, ignoring15:10
hrwykarel|away: I got for ara-raport then to entry with red, next page and then last entry is error one15:10
hrwhttp://logs.openstack.org/59/557659/43/check/kolla-ansible-ubuntu-source/d2676c5/ara-report/15:11
hrwhm. "Shared connection to 149.202.188.69 closed."15:11
ykarel|awayack15:11
hrwprobably machine in inventory died and whole process went down15:12
hrwShared connection to 158.69.66.25 closed.15:12
hrwShared connection to 149.202.173.231 closed.15:13
ykarel|awayhrw, main issue seems msg": "Timeout when waiting for 149.202.188.69:61313"15:16
ykarel|awayhttp://logs.openstack.org/59/557659/43/check/kolla-ansible-ubuntu-source/d2676c5/primary/logs/ansible/deploy15:17
* ykarel|away looking at kolla logs first time15:17
hrwykarel|away: thanks15:17
hrwlet me ask infra team15:19
ykarel|awaydoesn't seems infra though15:20
hrwlet then wait for someone who understand kolla's CI15:21
ykarel|awayhrw, ack, seems related to python versiondistro_python_version15:22
ykarel|awayhrw, ack, seems related to python version distro_python_version15:22
ykarel|awayhttp://logs.openstack.org/59/557659/43/check/kolla-ansible-ubuntu-source/d2676c5/primary/logs/docker_logs/haproxy.txt.gz15:23
hrwykarel|away: 3>15:23
ykarel|awayhmm just saw Depends-On, and it's python3 related15:27
hrwit's a pile of issues15:27
*** igordc has joined #openstack-kolla15:28
ykarel|awayhmm it will take good efforts for transition15:29
* ykarel|away leaving15:29
hrwykarel|away: simple fix in testing. thanks again15:33
*** rafaelweingartne has quit IRC15:44
*** hamzaachi has quit IRC15:51
*** tolisbar has joined #openstack-kolla15:51
*** henriqueof has quit IRC15:53
*** ykarel|away has quit IRC15:53
*** tolisbar has quit IRC15:54
*** tolisbar has joined #openstack-kolla15:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Debian/Ubuntu: move to Python 3 for binary/source images  https://review.opendev.org/64237515:58
*** yan0s has quit IRC15:59
*** rafaelweingartne has joined #openstack-kolla15:59
rafaelweingartneHello guys, I am working on a new "feature" in Kolla-ansible, which is to enabe Cloudkitty InfluxDB Storage backend via Kolla-ansible configuration16:00
rafaelweingartneI have written a spec for it16:00
rafaelweingartnehow is the process from this point on?16:00
hrwrafaelweingartne: send patch with spec, patch with code?16:01
rafaelweingartneDo I just create the PR in Gerrit with this spec as the commit body?16:01
rafaelweingartneit will be a patch with code16:01
hrwand remember about release notes, ok?16:01
rafaelweingartneI mean, a patch with a spec and the code that we changed16:01
rafaelweingartnehmm16:01
rafaelweingartneI am not sure I follow16:02
rafaelweingartneI have been proposing some PRs lately, but I have not complete the cycle (PRs getting merged). What do you mean by release notes?16:02
*** cah_link has quit IRC16:02
hrwok, first send code.16:03
rafaelweingartnehrw: I will do that. Thanks for the guidance16:11
*** k_mouza has joined #openstack-kolla16:15
*** rafaelweingartne has quit IRC16:15
*** tolisbar has quit IRC16:16
*** iniazi_ has joined #openstack-kolla16:16
*** k_mouza has quit IRC16:17
*** ykarel has joined #openstack-kolla16:17
*** ivve has joined #openstack-kolla16:20
*** iniazi has quit IRC16:20
*** ivve has quit IRC16:35
hrwhttp://logs.openstack.org/75/642375/19/check/kolla-ansible-ubuntu-source-zun/4b074a6/primary/logs/ansible/deploy - one step closer. but new failure16:45
*** JamesBenson has joined #openstack-kolla16:46
hrwand http://logs.openstack.org/75/642375/19/check/kolla-ansible-ubuntu-source-upgrade/dadddfa/primary/logs/ansible/upgrade one16:51
*** igordc has quit IRC16:53
*** qqmber has joined #openstack-kolla16:56
qqmberHi.. is anybody else facing this on /var/log/messages? systemd: Failed to start Session c102725 of user root.16:56
qqmbera lot... like... every 1 minute16:56
*** gmann_pto has quit IRC16:56
qqmberIf I run "systemctl | grep 'of user'| grep 'abandoned'" there is a lot of abandoned sessiones16:56
qqmberMay be related to https://bugzilla.redhat.com/show_bug.cgi?id=143998916:56
openstackbugzilla.redhat.com bug 1439989 in systemd "why so many sessions scope file didn't been deleted?" [Medium,New] - Assigned to dtardon16:56
ykarelAny core around? please review:- https://review.opendev.org/#/c/625261/16:59
*** Sravan has joined #openstack-kolla17:01
*** Talion has quit IRC17:04
*** henriqueof has joined #openstack-kolla17:07
*** k_mouza has joined #openstack-kolla17:17
*** ivve has joined #openstack-kolla17:18
*** k_mouza has quit IRC17:22
hrwykarel: https://review.opendev.org/642375 got fix and CI jobs look better. thanks17:23
hrwykarel: still fail but that's other issue now17:23
*** whoami-rajat has quit IRC17:24
* hrw off17:24
*** ykarel has quit IRC17:25
*** ykarel has joined #openstack-kolla17:48
*** Sravan has quit IRC17:57
*** Sravan has joined #openstack-kolla17:59
*** Sravan has quit IRC18:02
*** Sravan has joined #openstack-kolla18:03
*** radeks_ has joined #openstack-kolla18:07
*** radeks has quit IRC18:09
*** Sravan has quit IRC18:16
*** Sravan has joined #openstack-kolla18:30
qqmberquestion: If I configure "storage_interface: "em2"" in globals.yml, em2 has to be with IP BEFORE the deploy, or Kolla-Ansible/Openstack will put an IP there? I'm asking because I put a fixed IP and I get this error all the time: iquique NetworkManager[17922]: <info>  [1557167748.0055] device (em2): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')18:38
qqmberso.. Do I have to let em2 configuration as "BOOTPROTO=dhcp"?18:38
qqmber(instead of none18:38
qqmber)18:38
*** Sravan has quit IRC18:38
*** KeithMnemonic has joined #openstack-kolla18:39
*** Sravan has joined #openstack-kolla18:39
*** Sravan has quit IRC18:44
*** Sravan has joined #openstack-kolla18:47
*** happyhemant has quit IRC18:48
*** ykarel is now known as ykarel|away18:49
*** Sravan has quit IRC19:03
*** Sravan has joined #openstack-kolla19:09
*** Sukhdev has joined #openstack-kolla19:11
*** cah_link has joined #openstack-kolla19:17
*** ykarel|away has quit IRC19:22
*** Sravan has quit IRC19:25
*** Sravan has joined #openstack-kolla19:25
*** iniazi has joined #openstack-kolla19:33
*** bmace_ has quit IRC19:39
*** iniazi_ has quit IRC19:39
*** Sravan has quit IRC19:48
*** radeks_ has quit IRC19:50
*** spiette has quit IRC19:52
*** spiette has joined #openstack-kolla19:55
*** Sukhdev has quit IRC20:00
*** Sravan has joined #openstack-kolla20:05
JamesBensonqqmber, I haven't played with the storage_interface explicitly (I tried briefly and had unresolved issues and abandoned for now), but since it's default is network_interface. I believe it has to be a preconfigured IP.20:14
*** unicell has joined #openstack-kolla20:22
qqmberI thought the same... but the host tries and tries to configure an IP at that interface20:22
kplant^ that's what i do in my deployments20:22
qqmberJamesBenson:20:22
kplanthave you tried killing NetworkManager?20:22
kplanti have a 'yum remove -y NetworkManager\*' in my kickstart and haven't had any such problems20:23
qqmberkplant: and there is no issue removing it?20:29
*** mchlumsky has quit IRC20:31
*** k_mouza has joined #openstack-kolla20:32
qqmberkplant: by the way..when I put that option and em2 had an IP, nothing happened... all storage traffic went trhru de default network_interface (em1 in my case)20:33
*** k_mouza has quit IRC20:33
qqmbernow would be a grat moment to solve that, because I'll reinstall everything in the next hour :)20:33
JamesBensonqqmber, I typically do ubuntu 16 deploys although that might change in the future.  But our server MaaS is coninical based, so ubuntu is easier to deploy for us.20:45
JamesBensonqqmber, If I remember I had the same issue, traffic still going through a different interface.20:45
JamesBensonIf you solve that, ::please:: let me know!20:45
JamesBensonIt would be nice for kolla to have a few use case examples in their docs just to verify that.20:46
qqmberJamesBenson: if with Ubuntu I won't have this problem, I don't have any problem to install with it.20:47
qqmberJamesBenson: but you had my problem with Ubuntu? or with Centos= problem and Ubuntu =ok?20:48
JamesBensonqqmber, you don't have issues with the network/storage_interface issue?  or something else?20:48
qqmberbtw, Maas is cool20:48
JamesBensonboth for the interfaces I think.20:48
JamesBensonMaaS has been helpful certainly, but I also have my issues with it.20:48
*** unicell has quit IRC20:49
qqmberit would be awesome to have that solved.. because I have problems with volumes (Cinder) when I transfer a lot of small files (120GB of 12KB files). The volume (or something) crashes and re mount as read only.20:49
qqmberso, havve a independant interface is important in that case20:50
JamesBensonNot sure if you are familiar with it but FYI, rule of thumb is: Ubuntu-source, centos-binary, those are the two most tested lines in kolla from what I understand20:50
qqmberoh, good to know. I use Centos binary20:50
JamesBensonyes, for us, we were trying to use a new 10G nic for all of our servers, so I couldn't figure out if it was a nic issue or kolla issue.20:50
JamesBensonhence why I paused because I didn20:50
*** pcaruana has quit IRC20:51
JamesBensonI didn't want to spend time on chasing that bug... I have a lot of other urgent things...20:51
JamesBensonright now I'm trying to get xen working with kolla20:51
*** unicell has joined #openstack-kolla20:54
*** unicell has left #openstack-kolla20:54
qqmberoh, cool21:06
qqmberJamesBenson: so.. It's a known bug that you cannot have storage_interface different than network_interface (I'm asking to change that in my configuration now and forget about that for now21:15
*** dpawlik has quit IRC21:15
*** henriqueof has quit IRC21:16
qqmberat least neutron_external_interface works fine.21:17
*** dpawlik has joined #openstack-kolla21:20
*** dpawlik has quit IRC21:24
*** Sravan has quit IRC21:26
JamesBensonqqmber, not sure if it's a reported bug or not. I didn't report it because wasn't sure if it was my new hardware or a bug.  Typically the interfaces I use are the kolla_external_vip_interface, neutron_external_interface, and network_interface21:33
JamesBensonoutside of that not sure.21:33
qqmberok, thx21:33
kplantso my storage_interface is different from everything else and it works fine21:33
kplanthttp://paste.openstack.org/show/750839/21:34
qqmberkplant: really? and you configure an IP address on that interface (the one that is configured on storage_interface in globals.yml)21:34
qqmber?21:34
kplantyes21:34
kplantevery interface except the neutron external interface21:35
qqmberdo you use Cinder or Ceph?21:35
kplantboth21:35
kplantceph is the backend for cinder21:35
qqmberok21:35
qqmberin my case, I use Cinder, not Ceph and all the storage traffic goes thru the network_interface (em1), even though I have configured em2 as the storage_interface21:36
qqmber:(21:36
qqmberI have network_interface: "em1" and storage_interface: "em2" and nothing in the inventory file that can overwrite this (which is an option)21:37
kplantand em2 is IPed?21:38
*** Talion has joined #openstack-kolla21:38
qqmberyes21:38
kplantis em1 also your api_interface?21:38
qqmberand I have neutron_external_interface: "p2p2" without IP21:38
qqmberapi_interface: "{{ network_interface }}"21:38
qqmberyes21:38
qqmbercan you show me (pastebin) your globals.yml?21:39
kplanthonestly all i've changed is what i showed you, and i enabled a few things like ceph21:43
kplantand the vips21:43
JamesBensonkplant, what version of kolla-ansible are you using?21:48
JamesBensonI was trying on rocky, 7.0.121:48
JamesBensonI also use a kolla deployed version of ceph21:49
qqmberkplant: ok21:52
qqmberI asked about ceph because kolla-ansible documentation only mention Ceph in storage_interface configuration: "storage_interface - This is the interface that is used by virtual machines to communicate to Ceph. This can be heavily utilized so it’s recommended to put this network on 10Gig networking. Defaults to network_interface."21:55
qqmberIt doesn't say that works with Cinder, but it doesn't say the opposite either...21:56
*** JamesBenson has quit IRC21:58
*** Sravan has joined #openstack-kolla22:00
*** hamzaachi has joined #openstack-kolla22:03
*** JamesBenson has joined #openstack-kolla22:06
*** JamesBenson has quit IRC22:10
*** kplant has quit IRC22:19
*** hamzaachi has quit IRC22:28
*** kplant has joined #openstack-kolla22:38
*** cah_link has quit IRC22:58
*** zbr|rover has quit IRC23:01
*** pomac has joined #openstack-kolla23:02
*** qqmber has quit IRC23:07
*** JamesBenson has joined #openstack-kolla23:11
*** JamesBenson has quit IRC23:15
*** Sravan has quit IRC23:19
*** Sravan has joined #openstack-kolla23:20
*** Sravan has quit IRC23:24
*** Sravan has joined #openstack-kolla23:31
*** tosky has quit IRC23:32
*** Sravan has quit IRC23:37

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!