Tuesday, 2019-08-13

*** macz has quit IRC00:00
*** macz has joined #openstack-ansible00:01
*** macz has quit IRC00:04
*** macz has joined #openstack-ansible00:15
*** strattao has quit IRC00:17
*** macz has quit IRC00:18
*** spatel has joined #openstack-ansible00:27
*** markvoelker has quit IRC00:39
*** gyee has quit IRC00:40
*** markvoelker has joined #openstack-ansible00:44
*** dave-mccowan has quit IRC00:58
*** zirpu has joined #openstack-ansible01:30
*** spsurya has joined #openstack-ansible02:12
*** spatel has quit IRC03:02
*** gkadam has joined #openstack-ansible03:39
*** gkadam has quit IRC03:40
*** macz has joined #openstack-ansible03:51
*** macz has quit IRC03:55
*** macz has joined #openstack-ansible03:56
*** macz has quit IRC03:57
*** macz has joined #openstack-ansible03:57
*** macz has quit IRC04:01
*** macz has joined #openstack-ansible04:02
*** raukadah is now known as chkumar|ruck04:03
*** macz has quit IRC04:06
*** goldyfruit_ has quit IRC04:11
*** goldyfruit_ has joined #openstack-ansible04:11
*** dave-mccowan has joined #openstack-ansible04:56
*** dave-mccowan has quit IRC05:01
*** gkadam has joined #openstack-ansible05:30
*** udesale has joined #openstack-ansible05:33
*** pcaruana has joined #openstack-ansible05:40
*** pcaruana has quit IRC05:49
*** ksdean has quit IRC06:10
*** ivve has joined #openstack-ansible06:43
*** luksky has joined #openstack-ansible07:01
openstackgerritMerged openstack/openstack-ansible master: Revert "Change haproxy http-check expect for nova_console"  https://review.opendev.org/67392907:15
*** kopecmartin|off is now known as kopecmartin07:29
*** rpittau|afk is now known as rpittau07:34
*** tosky has joined #openstack-ansible07:37
*** pcaruana has joined #openstack-ansible07:45
*** macz has joined #openstack-ansible07:57
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/67572507:59
*** macz has quit IRC07:59
*** yolanda has quit IRC08:07
*** klamath has joined #openstack-ansible08:14
admin0hi all .. I am getting this error on cinder from  18.1.3 -> 18.1.9 upgrade ->volume_service_uuids_online_data_migration  nSome migrations failed unexpectedly. Check log for details. -- which log is it referring to ?08:26
ioniadmin0, https://bugs.launchpad.net/cinder/+bug/180615608:35
openstackLaunchpad bug 1806156 in Cinder "shared_targets_online_data_migration fails when cinder-volume service not running" [Undecided,Confirmed]08:35
ioniadmin0, maybe this?08:35
admin0ioni, many thanks .. i will try to apply the workarounds08:37
noonedeadpunkadmin0: please share your experience regarding this08:44
admin0i will08:44
admin0i am running rest of the playbooks ( to see if my heat is fixed ) :)08:44
admin0heat decided to die yesterday with: error while loading shared libraries: libpython2.7.so.1.0   -- out of the blue08:45
*** trident has quit IRC08:54
admin0heat-engine inside conainer is running, heat-api fails with: /openstack/venvs/heat-18.1.9/bin/uwsgi: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared .. how do I solve this ?08:57
admin0noonedeadpunk, my error is different:  https://gist.github.com/a1git/9abf881940097b607b880ed3d905e81609:03
noonedeadpunkI think I was facing this as well... But all cinder-volumes were online in the meanwhile...09:06
noonedeadpunkwhat dou you have in "openstack volume service list"?09:06
admin0noonedeadpunk,   updated -- in the comment09:08
*** trident has joined #openstack-ansible09:08
admin0opensack volume list seems to work09:08
admin0trying to create a new volume09:08
noonedeadpunkSo yes, you have volume services down for some reason...09:08
noonedeadpunklist is retrieved from database09:08
ionii think it moved it's services at some point from volume to metal09:09
ionisince it has iscsi09:09
ionithey are down since march09:09
*** rpittau is now known as rpittau|bbl09:09
noonedeadpunkBut it won't allow you to do online migrations in case at least one ciner-volume service is down...09:11
noonedeadpunkI guess you at least supposed to disable them09:12
admin0how do I disable a down cinder service ( never done it before)09:13
noonedeadpunkopenstack volume service set --disable --disable-reason "Some reason" c1-cinder-volumes-container-a96c1c74@ssd_iscsi  cinder-volume09:16
admin0disabed . running the command again to see what it does this time09:21
admin0anyone knows how to fix this issue: libpython2.7.so.1.0: cannot open shared   -- its only apparing on the heat api vnev09:22
admin0which was just upgraded to 18.1.909:22
ioniyou don't have python2 installed?09:23
noonedeadpunkI'd say that python-dev is missing...09:23
ionidpkg -l | grep python09:23
ionipython lib09:23
ioni-dev is just the headers if i recall09:23
admin0same issue noonedeadpunk  .. i got a timeout for rabbitmq .. but from the GUI, i was able to create  a volume09:24
noonedeadpunknot really. There libraries that are required for building some packages09:24
noonedeadpunkor maybe I'm mixed up things09:24
*** trident has quit IRC09:25
admin0 dpkg -l | grep python  -- inside the venv ?09:25
admin0my thought was all things are provided by the venv itself09:26
ionion the container09:26
*** ksdean has joined #openstack-ansible09:26
ionidpkg -L libpython2.7:amd6409:27
ionithis has the file09:27
ioni /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.009:27
admin0https://gist.github.com/a1git/81e1e25e73039186f99f8d06e99e2b88  -- this is inside the container09:27
ioniyou are missing the package  libpython2.709:27
admin0only have libpython 3.509:28
ioniapt-get install libpython2.709:28
admin0\o/ -- that worked ..   is this a new bug report or QA . how could the playbooks miss this library  /?09:31
admin0the package i mean09:31
*** trident has joined #openstack-ansible09:32
ionimaybe it had some problems apt and it forced to remove it09:33
ionicheck history.log in /var/log/apt/09:33
admin0ioni, before running the upgrade to 18.1.9, i did a apt-get upgrade on the host only ( not inside the containers, as i know containers are OSA managed) ..09:38
jrosseri wonder if this is another issue with only running metal jobs for role tests09:52
jrosserthings like a dependancy moving slightly that is accidentally provided by the host or some other role won't be caught09:52
noonedeadpunkjrosser: may be, since that package was apparently installed by another role.09:52
noonedeadpunkfor metal09:52
*** namrata has joined #openstack-ansible09:55
admin0ioni, when this failed, i deleted the old containers and re-created them .. wouldn't that have caught/fixed this ?10:00
*** ksdean has quit IRC10:13
admin0is there a filed in the cinder DB i can check to see what the status is and if a db migration is required ?10:19
admin0my issue seems to be due to some rabbitmq timeout which is strange10:20
CeeMacmorning10:20
admin0\o10:20
*** namrata has quit IRC10:24
noonedeadpunkjrosser: can you please check this patch https://review.opendev.org/#/c/669870/ and https://review.opendev.org/#/c/672719/4 ?10:44
*** udesale has quit IRC11:03
*** goldyfruit_ has quit IRC11:13
*** miloa has joined #openstack-ansible11:16
*** kplant has joined #openstack-ansible11:17
*** trident has quit IRC11:34
*** trident has joined #openstack-ansible11:40
*** ansmith has quit IRC11:40
*** namrata has joined #openstack-ansible11:45
*** ivve has quit IRC12:09
*** rpittau|bbl is now known as rpittau12:11
*** ansmith has joined #openstack-ansible12:23
*** electrofelix has joined #openstack-ansible12:32
*** goldyfruit has joined #openstack-ansible13:13
*** gkadam has quit IRC13:17
*** pcaruana has quit IRC13:23
*** tinwood has quit IRC13:30
*** udesale has joined #openstack-ansible13:30
*** tinwood has joined #openstack-ansible13:31
*** lbragstad has joined #openstack-ansible13:32
*** BjoernT has joined #openstack-ansible13:36
*** BjoernT_ has joined #openstack-ansible13:39
*** BjoernT has quit IRC13:39
jrossernoonedeadpunk: done13:41
*** BjoernT has joined #openstack-ansible13:44
*** BjoernT_ has quit IRC13:46
*** pcaruana has joined #openstack-ansible14:00
*** ansmith_ has joined #openstack-ansible14:01
*** lbragstad has quit IRC14:02
*** ansmith has quit IRC14:03
*** chkumar|ruck is now known as raukadah14:57
*** dave-mccowan has joined #openstack-ansible14:58
*** dave-mccowan has quit IRC15:04
*** ansmith has joined #openstack-ansible15:04
*** ansmith_ has quit IRC15:06
*** namrata has quit IRC15:21
openstackgerritMerged openstack/openstack-ansible-os_octavia master: iptables save fix for centos 7  https://review.opendev.org/66987015:31
*** jbadiapa has joined #openstack-ansible15:35
*** gyee has joined #openstack-ansible15:44
*** macz has joined #openstack-ansible15:52
*** KeithMnemonic has quit IRC15:52
*** shyamb has joined #openstack-ansible15:57
*** vesper11 has quit IRC15:58
*** vesper11 has joined #openstack-ansible16:00
tiffanie#startmeeting openstack_ansible_meeting16:00
openstackMeeting started Tue Aug 13 16:00:20 2019 UTC and is due to finish in 60 minutes.  The chair is tiffanie. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:00
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:00
tiffanie#topic office hours16:00
*** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)"16:00
mnaserhi16:01
mnasereveryone16:01
mnaser:>16:01
*** markvoelker has quit IRC16:02
*** spsurya has quit IRC16:04
openstackgerritMerged openstack/openstack-ansible stable/stein: Update docs & upgrade scripts for Stein  https://review.opendev.org/67271916:08
jamesdentonhi16:09
noonedeadpunko/16:09
* mnaser is getting tired of seeing warnings on CentOS16:09
mnaserwhen sourcing openrc16:09
mnasernoonedeadpunk: seems to have traced it back to https://github.com/openstack/openstack-ansible-openstack_openrc/commit/ef3fe9966fff2570ce2b40094921abe17fa3f2a116:09
mnaseri was thinking maybe we should set LC_ALL to some other value? or does centos not have a utf8 locale?16:10
noonedeadpunkI'm not really sure that all systems will have some locale by default (except C, but not for centos)16:11
noonedeadpunkSo I saw ubuntu without en_us locale some time ago.16:11
*** markvoelker has joined #openstack-ansible16:12
noonedeadpunkprobably we should install one (or verify that it's there)16:12
noonedeadpunkor kinda detect avaiable ones16:12
mnaserit looks like ubuntu is ok with that locale16:13
mnaserhmm let me find a centos machine16:13
noonedeadpunkYeah, I guess ubuntu has it by default16:14
*** shyamb has quit IRC16:15
mnaserhttps://termbin.com/692k16:15
mnaserthis is all the centos locales16:16
mnaserisnt there a way for us jsut to say "use utf8"16:16
mnaserand not the specific lang16:16
noonedeadpunkI guess no... That's why C was created I guess16:17
*** BjoernT has quit IRC16:17
noonedeadpunk(but I's just a guess)16:17
mnaseri wonder what would happen if we dropped that line entirely16:19
noonedeadpunkProbably I'd suggested adding some en_us.UTF8 generation for centos and defining it instead of C16:20
noonedeadpunkat least for centos 7. And hope appearing C in centos 816:20
noonedeadpunkand add posibility to override it:)16:24
mnaseryeah to not mess up users who have specific locales i guess16:24
mnaseris C.UTF8 the same as en_us.UTF816:24
mnaseror mostly same16:25
noonedeadpunkI'm not a big expert but out of what I've found for such pressed time it looks pretty common16:25
mnaseryeah i dont think having C.UTF8 will keep your local16:26
noonedeadpunkdespite en_us may be missing things16:26
mnaserso it will revert you to english anyways16:26
mnaserso if we default to en_us its not like people will get confused16:26
*** macz has quit IRC16:26
noonedeadpunkmnaser: So we need to ensure that this locale is present for sure16:28
guilhermespupps late16:29
guilhermespo/16:29
noonedeadpunkso the question is should be do this for both ubuntu and centos or centos only...16:29
mnasernoonedeadpunk: oh i wonder if there is an ansible fact that can say what is the current locale?16:29
noonedeadpunkansible_env.LANG16:30
noonedeadpunkhttps://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#variables-discovered-from-systems-facts16:30
mnaserso maybe we can use that?16:30
*** miloa has quit IRC16:30
noonedeadpunkyeah, let me try to make a patch16:31
*** rpittau is now known as rpittau|afk16:32
mnaserother than that, the uwsgi repo thing seems to have been approved16:37
mnaserso we can start on that clean up16:37
noonedeadpunkwow, that was fast16:38
mnasernoonedeadpunk: ptl +1 counted as +2 :P16:38
noonedeadpunkI'd say yes:))16:38
jrosserhello16:38
mnaserhttps://opendev.org/openstack/ansible-role-uwsgi16:38
mnaserits up and ready16:39
mnaserso we can start iterating16:39
mnaserhi jrosser16:39
noonedeadpunkThe bad thing is that https://review.opendev.org/#/c/676193/ failed...16:39
mnasernoonedeadpunk: thats normal because of how openstack/project-config works16:40
noonedeadpunkok, seems that depends on doesn't work16:40
mnasernoonedeadpunk: its a 'trusted' repo so you cannot do depends-on it16:40
noonedeadpunkOk, great. I've already started work on uwsgi16:41
mnaser"A list of projects to be treated as config projects in this tenant. The jobs in a config project are trusted, which means they run with extra privileges, do not have their configuration dynamically loaded for proposed changes, and Zuul config files are only searched for in the master branch."16:41
noonedeadpunkso repo shouldn't be empty for a long time16:41
mnaserthey're called config projects now i guess :)16:41
mnasernoonedeadpunk: awesome16:41
*** macz has joined #openstack-ansible16:43
*** udesale has quit IRC16:43
*** kopecmartin is now known as kopecmartin|off16:44
jrosserthe nova-lxd jobs on stein are failing16:46
jrosserand i didnt really understand why, but problem is it seems to try to use the master version of the repo when it builds the nova venv16:47
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_openrc master: Add option to redefine locale  https://review.opendev.org/67624416:47
noonedeadpunkjrosser: so nova-lxd has moved to x/nova-lxd - can't this be the problem?16:47
jrosserah yes thats the otehr thing it it using the wrong url16:48
jrosserbut i could not see how or where that came from16:48
*** weifan has joined #openstack-ansible16:49
noonedeadpunkI guess that they've just move repo to the new url as a part of obsolate process16:49
noonedeadpunkmnaser: what do you think about approach in  https://review.opendev.org/676244 ?16:50
mnasernoonedeadpunk: that seemes reasonable actaully16:51
mnaserjrosser: i think i remember looking into that16:52
*** weifan has quit IRC16:54
mnaserjrosser: i think the step #1 issue is this16:54
mnaserhttps://review.opendev.org/#/c/675155/16:54
mnaserits using the old functional jobs16:54
jrosseroh right yes16:55
mnasermainly https://review.opendev.org/#/c/658521/16:55
mnaserlogan-: is that safe/good to backport?16:55
*** mgariepy has quit IRC16:58
tiffanie#endmeeting17:00
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2xA1eZC"17:00
openstackMeeting ended Tue Aug 13 17:00:28 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-08-13-16.00.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-08-13-16.00.txt17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-08-13-16.00.log.html17:00
*** jbadiapa has quit IRC17:12
*** weifan has joined #openstack-ansible17:16
*** mgariepy has joined #openstack-ansible17:20
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Add mixed setup for debian based OS  https://review.opendev.org/67535317:47
*** hamzy has quit IRC17:54
openstackgerritMerged openstack/openstack-ansible-os_sahara master: Update tests git source to opendev  https://review.opendev.org/65732218:04
*** spatel has joined #openstack-ansible18:07
spatelquick question, does nova-compute directly talk to neutron using REST-API call or it use RabbitMQ rpc.call ?18:07
*** markvoelker has quit IRC18:12
*** markvoelker has joined #openstack-ansible18:15
*** BjoernT has joined #openstack-ansible18:15
openstackgerritMerged openstack/openstack-ansible-os_horizon stable/stein: Add encoding to local_settings.py  https://review.opendev.org/67588518:30
*** hamzy has joined #openstack-ansible18:30
*** hamzy has quit IRC18:51
*** hamzy has joined #openstack-ansible18:51
openstackgerritMerged openstack/openstack-ansible-ops master: Config updates for elk 7.x  https://review.opendev.org/66985218:52
openstackgerritMerged openstack/openstack-ansible master: Bump SHAs for master  https://review.opendev.org/67578118:58
logan-mnaser: yes should be fine, however that patch is part of a series of patches reworking tests, so there may be patches below/above it in the stack that apply.19:05
mnaserspatel: it does not talk over rabbitmq19:06
*** electrofelix has quit IRC19:06
spatelmnaser: thanks so it use REST-API to talk right?19:06
mnaseryes19:06
spatelperfect!19:07
*** mgariepy has quit IRC19:08
*** ivve has joined #openstack-ansible19:14
spatelmnaser: Doing documentation stuff let me know if you think this is wrong or anything wrong here: https://imgur.com/a/H4oQ5xo19:15
mnaserspatel: maybe you can push that up for a review on gerrit to add to osa :D19:22
spatelsure, why not :)19:23
spatelI don't know how to do that :( in worst case i will email that diagram to you and you can push that.19:24
*** mgariepy has joined #openstack-ansible19:34
*** kplant has quit IRC19:48
*** weifan has quit IRC19:59
*** weifan has joined #openstack-ansible20:00
*** weifan has quit IRC20:01
logan-mnaser: want me to work on weeding thru that stack and propose backports?20:08
mnaserPlease!20:10
*** markvoelker has quit IRC20:10
*** markvoelker has joined #openstack-ansible20:28
*** ansmith has quit IRC20:32
*** spatel has quit IRC20:32
*** strattao has joined #openstack-ansible20:39
*** weifan has joined #openstack-ansible20:40
*** weifan has quit IRC20:42
openstackgerritMerged openstack/openstack-ansible master: Fix scripts/inventory-manage.py help invocation  https://review.opendev.org/66386420:54
*** markvoelker has quit IRC21:07
*** allanb has joined #openstack-ansible21:15
allanbanyone using the debian9 container images?  i just figure out why they couldn't resolve deb.debian.org: broken symlink instead of a file due to systemd not creating the resolv.conf.21:16
allanbi.e. resolv.conf -> /var/run/systemd/resolve/resolv.conf21:16
openstackgerritMerged openstack/openstack-ansible-os_sahara master: Add integrated coverage  https://review.opendev.org/66033721:23
openstackgerritMerged openstack/openstack-ansible-os_sahara master: Clarify supported SUSE versions  https://review.opendev.org/66033821:23
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: Add variable causing error on rerun for redhat OS  https://review.opendev.org/66071321:26
mnaserallanb: oh awesome to see someone trying it out!21:41
mnaserwhat os are you trying this on?21:41
allanbdebian921:42
allanbcurrent $job wants to use debian9. *shrug*21:43
openstackgerritMerged openstack/openstack-ansible stable/pike: Bump SHAs for stable/pike  https://review.opendev.org/67578421:53
*** hamzy has quit IRC21:58
*** markvoelker has joined #openstack-ansible22:10
*** markvoelker has quit IRC22:14
allanbjan 1st, 2020 not too far away, is openstack getting close to switching to python3?22:19
allanbjust found a debian9/python2.7 bug running setup-infrastructure.yml.  still trying to figure it to submit a bug report.22:20
strattaohey jamesdenton are you in? We're seeing a weird issue where some of our vms can't reach out through dns - they can ping all the ips listed in /etc/resolv.conf, but when we check the packets, we're seeing a bunch of bad checksums in the dhcp udp requests... This is only affecting some vms in a project which leads me to thinking that maybe there is a funky cache or something on the hypervisor...22:20
strattaogot any suggestions for things that we could look at?22:21
*** ansmith has joined #openstack-ansible22:24
*** tosky has quit IRC22:27
*** spatel has joined #openstack-ansible22:30
allanbhttps://bugs.launchpad.net/openstack-ansible/+bug/184008922:35
openstackLaunchpad bug 1840089 in openstack-ansible "debian9 container setup during setup-infrastructure.yml run failes in retries" [Undecided,New]22:35
allanbgah. s/failes/fails/  :-\22:35
*** ivve has quit IRC22:36
*** luksky has quit IRC22:37
*** spatel has quit IRC22:51
*** markvoelker has joined #openstack-ansible23:25
*** macz has quit IRC23:32
*** markvoelker has quit IRC23:36
*** weifan has joined #openstack-ansible23:51
*** weifan has quit IRC23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!