Thursday, 2019-01-24

*** bgmccollum has joined #openstack-ansible00:05
*** macza has quit IRC00:13
*** TxGirlGeek has quit IRC00:29
*** TxGirlGeek has joined #openstack-ansible00:40
*** ThiagoCMC has joined #openstack-ansible00:55
ThiagoCMCGuys, I'm OSA with Ceph and it's from here: TASK: ceph-mon : test if initial mon keyring is in mon kv store00:58
ThiagoCMCThe command `ceph --cluster ceph config-key get initial_mon_keyring` enters in a infinity loop... I checked with `strace`00:58
*** nurdie has joined #openstack-ansible01:04
*** nurdie has quit IRC01:08
*** ThiagoCMC has left #openstack-ansible01:08
*** ThiagoCMC has joined #openstack-ansible01:10
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update delegated setup hosts to support IP delegation  https://review.openstack.org/63220101:28
cloudnullThiagoCMC oh that sounds all bad01:28
cloudnulldid you get it figured out ?01:28
ThiagoCMCNo...01:29
ThiagoCMCI'm seeing that the ceph.conf inside of the container, have its "mon host = IPs" pointing to the host itself, NOT to the ceph-mon container's IP!01:30
ThiagoCMCMight be it, right?01:30
cloudnullderks https://review.openstack.org/#/c/632142/ - maybe related - regarding the missing ceph role ?01:30
cloudnullThiagoCMC sadly I'm garbage w/ ceph .01:30
cloudnulli really have no idea what the issue could be :'(01:31
cloudnullhowever if it works on one node and not another, i'd say it makes sense some config is busted.01:31
ThiagoCMCHmm... ok01:32
ThiagoCMCI'll double check everything01:32
ThiagoCMCAnyway, looks like that OSA is missing a more complete doc about how to install Ceph via "openstack-ansible".01:33
cloudnullI know there's a  bunch of docs for ceph, though, because I don't use ceph, I'm not really sure what the state of those are .01:34
cloudnullA lot of folks seem to deploy ceph as a stand a lone thing using ceph ansible , then integrate openstack with that. maybe that would be more successful?01:34
cloudnullthat said, I dont really know if that would be better, as I've not done it myself.01:35
ThiagoCMCI would prefer to use ceph-ansible with openstack-ansible, because I'm curious about what it does, then, learn from it.01:35
ThiagoCMCA friend of mine that knows more about Ceph, told me that the ceph.conf must have the container's IPs... Not the IP of the host running the container!01:36
ThiagoCMCNot sure about how that IP ended up there... Maybe it will work if I change ceph-mon to is_metal = true01:36
ThiagoCMCsomething like that01:36
*** nurdie has joined #openstack-ansible01:37
cloudnullI'm not sure, logan- jrosser odyssey4me cjloader theintern would be the folks I'd think would know?01:38
*** TxGirlGeek has quit IRC01:39
ThiagoCMCHey, I used "monitor_interface: br-storage", so, it WILL take the ip of that interface! LOLOL01:39
cloudnullthat does make some sense :)01:40
ThiagoCMCMight be "eth1", -> br-mgmt at host (172.29.236.0/22)01:40
ThiagoCMC:-P01:40
ThiagoCMCIt's only locigal! lol01:40
ThiagoCMClogical! ha! ^_^01:40
ThiagoCMCNop... :-/01:44
ThiagoCMCMaybe "mon_group_name: ceph-mon_hosts" is wrong instead...01:45
ThiagoCMCcloudnull, BTW thanks for calling more ppl to help!01:46
*** nurdie has quit IRC01:51
*** nurdie has joined #openstack-ansible01:52
ThiagoCMCNo idea about how to tell ceph-ansible (via user_variables.yml of OSA), to use the container's IPs... =\01:53
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update delegated setup hosts to support IP delegation  https://review.openstack.org/63220101:54
*** nurdie has quit IRC01:56
cloudnullThiagoCMC im just not sure how that's all wired up01:58
cloudnullprobably someone in the morning, I suspect most folks who would know are sleeping01:59
ThiagoCMCI guess that I just fixed it!02:01
ThiagoCMCNow the error is at: "please provide lvm_volumes to your osd scenario" :-D02:01
ThiagoCMClol02:01
ThiagoCMCReading: http://docs.ceph.com/ceph-ansible/master/osds/scenarios.html#osd-scenario-lvm :)02:16
ThiagoCMCceph-install.yml, worked! Wheee!02:21
ThiagoCMCcloudnull, just curious, how is the nspawn dev evolving?! I can't wait to use it!  :-P02:37
ThiagoCMCAnd also the "install_method: distro" on Ubuntu LTS02:38
openstackgerritMarc Gariépy (mgariepy) proposed openstack/ansible-hardening stable/rocky: Switch to rtcsync for chrony  https://review.openstack.org/63290703:29
openstackgerritMarc Gariépy (mgariepy) proposed openstack/ansible-hardening stable/queens: Switch to rtcsync for chrony  https://review.openstack.org/63290803:29
*** udesale has joined #openstack-ansible03:49
*** mino_ has joined #openstack-ansible04:07
*** spsurya has joined #openstack-ansible04:33
*** gyee has quit IRC05:01
*** udesale has quit IRC05:27
*** udesale has joined #openstack-ansible05:27
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update delegated setup hosts to support IP delegation  https://review.openstack.org/63220105:28
*** udesale has quit IRC05:29
*** udesale has joined #openstack-ansible05:29
*** udesale has quit IRC05:30
*** mkuf has quit IRC05:31
*** shyamb has joined #openstack-ansible05:36
chandankumarodyssey4me: Hello06:00
chandankumarodyssey4me: http://paste.openstack.org/show/743206/ import requests still exists in virtualbox centos vm06:00
*** shyamb has quit IRC06:07
*** hwoarang has quit IRC06:08
*** hwoarang has joined #openstack-ansible06:09
*** udesale has joined #openstack-ansible06:09
*** shyamb has joined #openstack-ansible06:11
*** jbadiapa has joined #openstack-ansible06:13
*** markvoelker has joined #openstack-ansible06:16
*** nurdie has joined #openstack-ansible06:26
*** markvoelker has quit IRC06:45
jrosserchandankumar: I don’t know what’s wrong there but you could try issuing a similar virtualenv command at the cli and see what happens06:58
jrosserFind out if it’s something wrong generally on the host or it only goes wrong via the ansible06:58
*** shyamb has quit IRC07:12
*** shyamb has joined #openstack-ansible07:13
*** eumel8 has joined #openstack-ansible07:16
*** kopecmartin|off is now known as kopecmartin07:19
*** mkuf has joined #openstack-ansible07:19
*** udesale has quit IRC07:22
*** markvoelker has joined #openstack-ansible07:42
*** DanyC has joined #openstack-ansible07:44
*** DanyC has quit IRC07:45
*** udesale has joined #openstack-ansible07:46
*** DanyC has joined #openstack-ansible07:48
*** shyamb has quit IRC07:48
*** shyamb has joined #openstack-ansible07:49
*** DanyC has quit IRC07:49
*** nurdie has quit IRC07:55
*** nurdie has joined #openstack-ansible07:55
*** ostackz has quit IRC07:58
*** shyamb has quit IRC07:59
*** nurdie has quit IRC08:00
chandankumarjrosser: odyssey4me it is easily reproducable08:10
chandankumarjrosser: odyssey4me virtualenv --no-download --python=python2 /openstack/venvs/neutron-testing08:10
*** markvoelker has quit IRC08:15
*** rgogunskiy has joined #openstack-ansible08:17
*** fnpanic has joined #openstack-ansible08:33
openstackgerritFilippo Inzaghi proposed openstack/openstack-ansible-os_keystone master: fix tox python3 overrides  https://review.openstack.org/60682008:34
*** udesale has quit IRC08:37
*** udesale has joined #openstack-ansible08:42
*** udesale has quit IRC08:42
*** tosky has joined #openstack-ansible08:48
*** eumel8 has quit IRC08:49
*** shyamb has joined #openstack-ansible08:52
*** ianychoi has quit IRC08:55
*** nurdie has joined #openstack-ansible09:01
*** hamzaachi has joined #openstack-ansible09:04
*** nurdie has quit IRC09:05
chandankumarjrosser: Here is the hack I applied I enabled delorean-deps.repo from rdo install virtualenv and pip from there it is working09:18
chandankumarworking pip version is 8.009:18
*** CeeMac has joined #openstack-ansible09:18
*** udesale has joined #openstack-ansible09:20
*** nurdie has joined #openstack-ansible09:21
jrosserchandankumar: remind me which patch you hit this trouble with?09:25
*** nurdie has quit IRC09:26
*** shardy has joined #openstack-ansible09:27
*** errr has quit IRC09:30
*** errr has joined #openstack-ansible09:35
chandankumarjrosser: there is no patch involved09:41
chandankumarjrosser: I am trying to reproduce os_Tempest scenario tests failure09:41
*** nurdie has joined #openstack-ansible09:42
chandankumarjrosser: I think the problem is here https://github.com/openstack/openstack-ansible-tests/blob/master/run_tests_common.sh#L7609:42
chandankumarjrosser: it installs pip <1809:42
chandankumarbut we need pip == 8.009:42
jrosseri am confused - iirc we took out the get-pip stuff becasue it was so troublesome and tried to rely entirely on the tools on the host certinaly in the openstack-ansible repo09:46
*** nurdie has quit IRC09:46
jrosserthis really is odyssey4me area of expertise though09:46
jrosserchandankumar: it would be worth taking a read through this https://github.com/openstack/openstack-ansible/blob/master/scripts/bootstrap-ansible.sh, lots of careful setup of all these tools happens in there for the integrated builds09:49
chandankumarjrosser: sure09:50
*** electrofelix has joined #openstack-ansible09:51
odyssey4meoh dear, I have the feeling my replies got list - or did they? I wrote three lines in reply to chandankumar09:56
jrosserlost in the intertubes i think09:57
chandankumarodyssey4me: the vagrant steps I tried does not worked09:58
chandankumarit stuck with the same error09:58
odyssey4mechandankumar that's odd - I had a perfectly good test with my virtualbox test10:01
odyssey4mebut yes, we should probably no longer install pip with get-pip.py any more, and rather prep the host like we do for the integrated build10:01
odyssey4meit's a bit messy in that tests repo though, and I was hoping to just ditch its use entirely :/10:01
*** nurdie has joined #openstack-ansible10:03
chandankumarodyssey4me: RDO provides https://trunk.rdoproject.org/centos7-master/delorean-deps.repo to provides other dependencies which are not built by delorean10:04
chandankumarit is used in puppet-openstack-integration tripleo and packstack in the gates10:04
chandankumarhttp://codesearch.openstack.org/?q=delorean-deps.repo&i=nope&files=&repos=10:04
*** hamzaachi has quit IRC10:05
*** nurdie has quit IRC10:06
odyssey4mechandankumar ok, taking a step back - is this an issue for tripleo gates, our gates, or just for human testing?10:07
chandankumarodyssey4me: human testing10:07
chandankumarodyssey4me: the venv issue i fixed by installing python-virtualenv and pip from rpm10:09
chandankumarfrom delorean-deps repo10:09
odyssey4mechandankumar yeah, let me take a peek for a bit and see - I don't think it'll be an issue to convert it, I just need to spend a little time re-familiarising myself with it10:10
*** udesale has quit IRC10:12
*** markvoelker has joined #openstack-ansible10:12
*** udesale has joined #openstack-ansible10:13
*** asettle has joined #openstack-ansible10:14
*** shyamb has quit IRC10:15
odyssey4mechandankumar jrosser ah, I remember now why we couldn't switch this like we did for the integrated repo10:15
odyssey4methe issue is that all role tests use tox, and for human testing to get tox there we need pip on the host, and there is no native pip package for centos... for that we have to add the RDO repositories, and that's where things get complicated because we're duplicating somethign already done in the playbooks10:17
odyssey4memy plan all along was to get rid of our use of tox for all functional testing by switching all tests to use the integrated repo10:17
odyssey4mechandankumar I'm curious though - with the vagrant test you did, did you use the vagrant file in the repo - or did you use something else?10:18
chandankumarodyssey4me: I used the same vagrant file10:18
chandankumarodyssey4me: I downloaded the latest vagrant images10:18
chandankumarof centos10:18
odyssey4mevery strange that it worked for me10:18
odyssey4meyeah, I got my virtualbox and vagrant up to date, then did http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2019-01-23.log.html#t2019-01-23T14:22:4310:19
odyssey4mechandankumar ok, so there is a bonus here - it will only try to install pip if it's not already on the host10:20
chandankumarodyssey4me: yes,10:20
chandankumarodyssey4me: are we using openstack-ansible-pip_install role somewhere?10:22
odyssey4mechandankumar yes10:22
odyssey4mechandankumar for rocky, onwards, though we mainly use it for laying down pip.conf, not to actually install pip any more10:23
odyssey4mewhy do you ask?10:23
*** nurdie has joined #openstack-ansible10:23
chandankumarodyssey4me: I was just checking the version of pip getting installed10:23
odyssey4meit should be whatever version is pinned here: https://github.com/openstack/openstack-ansible/blob/master/global-requirement-pins.txt10:24
odyssey4mealthough for role tests it's plausible that it's whatever is latest instead10:24
chandankumarodyssey4me: https://github.com/openstack/openstack-ansible-tests/blob/master/run_tests_common.sh#L76 is installing pip 18.1 so asked10:24
chandankumarsorry pip <1810:25
odyssey4meyeah, this is all a mess :/10:25
odyssey4mechandankumar please feel free to change https://github.com/openstack/openstack-ansible-tests/blob/master/run_tests_common.sh#L73-L77 with using packages instead, I would really prefer that anyway10:26
chandankumarodyssey4me: sure10:26
chandankumarodyssey4me: I will push a patch there10:26
odyssey4meit should then do https://github.com/openstack/openstack-ansible/blob/master/scripts/bootstrap-ansible.sh#L132-L143 so that we're using the same pin as the integrated repo, and all distributions are using the same version of pip, setuptools, wheel10:27
odyssey4meotherwise things go weird10:27
*** nurdie has quit IRC10:27
odyssey4mechandankumar another method could actually be to install virtualenv, then install tox & bindep into a virtualenv and go from there - but I'm not sure how much change that wuld need to work10:29
*** mkuf has quit IRC10:39
*** mkuf_ has joined #openstack-ansible10:39
*** nurdie has joined #openstack-ansible10:44
*** markvoelker has quit IRC10:46
*** nurdie has quit IRC10:48
*** mino_ has quit IRC11:07
*** noonedeadpunk has joined #openstack-ansible11:09
*** udesale has quit IRC11:25
*** rgogunskiy has quit IRC11:35
*** markvoelker has joined #openstack-ansible11:43
*** nurdie has joined #openstack-ansible11:45
*** nurdie has quit IRC11:49
*** shyamb has joined #openstack-ansible11:57
*** nurdie has joined #openstack-ansible12:06
*** nurdie has quit IRC12:10
*** markvoelker has quit IRC12:16
*** nurdie has joined #openstack-ansible12:26
*** mkuf has joined #openstack-ansible12:28
*** nurdie has quit IRC12:31
*** mkuf_ has quit IRC12:31
*** mkuf_ has joined #openstack-ansible12:32
*** mkuf has quit IRC12:33
*** mkuf has joined #openstack-ansible12:34
*** mkuf_ has quit IRC12:37
*** bgmccollum has quit IRC12:39
*** pcaruana has quit IRC12:40
*** ansmith has quit IRC12:41
*** shyamb has quit IRC12:44
*** nurdie has joined #openstack-ansible12:47
*** shyamb has joined #openstack-ansible12:51
*** nurdie has quit IRC12:51
*** udesale has joined #openstack-ansible13:01
*** nurdie has joined #openstack-ansible13:07
chandankumarodyssey4me: I have a setup with os_Tempest instaleld buit failed at libselinux issue,13:08
chandankumarodyssey4me: I had made some changes in the role13:08
chandankumarodyssey4me: But I need to rekick the stuff for tempest13:09
chandankumarodyssey4me: I am not sure where the tempest playbook exists? Do I write a new playbook and run it?13:09
chandankumaragainst tempest1?13:09
odyssey4mechandankumar the tempest playbook is in tests/common/test-install-tempest.yml13:10
*** nurdie has quit IRC13:12
*** markvoelker has joined #openstack-ansible13:13
*** mkuf has quit IRC13:16
*** hwoarang has quit IRC13:18
*** hwoarang has joined #openstack-ansible13:19
*** pcaruana has joined #openstack-ansible13:20
*** jrosser has quit IRC13:20
*** shyamb has quit IRC13:23
*** markvoelker has quit IRC13:23
*** jrosser has joined #openstack-ansible13:23
*** shyamb has joined #openstack-ansible13:23
*** markvoelker has joined #openstack-ansible13:23
*** mkuf has joined #openstack-ansible13:24
*** ygk_12345 has joined #openstack-ansible13:26
ygk_12345odyssey4me: Hi. Is there any difference in  horizon versions between 18.1.1  and 18.1.2 ?13:26
ygk_12345odyssey4me: especially  the folder structure ?13:26
odyssey4meygk_12345 probably - we bump sha's for each release13:27
odyssey4meygk_12345 unlikely major changes - usually just bugfixes13:27
ygk_12345odyssey4me: actually I have made a custom hack with three .py files and applied it to horizon in both version. it is only working for 18.1.2 and not 18.1.113:28
*** nurdie has joined #openstack-ansible13:28
odyssey4meygk_12345 this is the horizon SHA for the next release: https://github.com/openstack/openstack-ansible/blob/stable/rocky/playbooks/defaults/repo_packages/openstack_services.yml#L7513:28
odyssey4mecheck the SHA differences in your test env, and work out what changed in horizon between the two13:29
jrosserygk_12345: you can override the variables in openstack_services.yml and point to a modified repo of your own if that helps you develop/test some changes13:31
*** nurdie has quit IRC13:32
CeeMacanyone come across an issue with dashboard not loading with "ERR_SSL_PROTOCOL_ERROR" error after stable/rocky install on bionic?13:33
jamesdentonmornin13:34
CeeMachi13:35
*** shyamb has quit IRC13:36
jamesdentonhi CeeMac13:37
*** ygk_12345 has quit IRC13:39
*** ansmith has joined #openstack-ansible13:43
*** shyamb has joined #openstack-ansible13:43
*** shyamb has quit IRC13:46
*** shyamb has joined #openstack-ansible13:47
*** nurdie has joined #openstack-ansible13:49
*** nurdie has quit IRC13:53
*** nurdie has joined #openstack-ansible13:56
CeeMacjamesdenton, i don't suppose you've come across any issues with horizon dashboard not loading because of "ERR_SSL_PROTOCOL_ERROR"14:00
openstackgerritMichael Vollman proposed openstack/openstack-ansible-os_manila master: Basic working os_manila role  https://review.openstack.org/61193014:01
jamesdentoni have not, sorry14:01
jamesdentonwhat version?14:02
jamesdentonoh n/m14:02
CeeMacstable/rocky on bionic14:02
CeeMacif i use curl on the command line I dont get any errors, but then I don't get any response either, I'm not curl expert though14:02
jamesdentondo you see any errors in the logs in horizon container?14:03
CeeMaccant' load the page from IE or Chrome on remote (routed) network.  Just about to attach a OC to the same lan14:03
openstackgerritAndy Smith proposed openstack/openstack-ansible master: Add qdrouterd role for rpc messaging backend deployment  https://review.openstack.org/62418414:03
CeeMacgood question14:03
CeeMaclet me look again14:03
*** strattao has joined #openstack-ansible14:12
CeeMacjamesdenton, in the other_vhosts_access.log i see this http://paste.openstack.org/show/743224/14:13
CeeMacnothing in access log of note14:14
*** mgariepy has joined #openstack-ansible14:15
CeeMacnothing in journal of note14:15
CeeMacif i try and connect directly to the container IP on port 80, it redirects to https, with /auth/login/?next=/ then throws the same error14:19
jamesdentonok - are the backends up for haproxy?14:20
jamesdentonhatop should be able to reveal that14:20
CeeMacthey were, let me double check as I've restared the container14:21
CeeMacyep14:21
CeeMacfront-ends open, back-end is up14:22
CeeMacincidentally, it has the same issue going through the LB address and direct to container IP14:22
CeeMaclast time I saw something like this it was a ssl/tls issue14:22
jamesdentonso when you curl https://<container>/ you get an error too?14:23
CeeMaci get nothing i think14:23
CeeMaci'll try again14:23
CeeMaci did this time http://paste.openstack.org/show/743226/14:24
CeeMacthink i tested against LB address originally14:24
jamesdentonHmm. did you deviate from the deploy guide at all? meaning, did you have overrides related to SSL/certs or anything?14:25
jrosserit would be a surprise if you could https://<container>/ for a couple of reasons, isnt the ssl termination done at the LB14:26
jrosserand if you can reach the containers from the "outside" then I think there are bigger problems :)14:26
CeeMacjamesdenton, i did not14:27
CeeMacjrosser, its the 'internal' network i'm connecting to from a routed management network on the 'inside'14:27
jrosserCeeMac: I have a rocky/bionic AIO running right now, so i'll be able to see how that turns out later when it's cooked14:28
CeeMacif i http://<container>/ it performs a redirect to https://<container>/auth/login/?next=/14:28
jamesdentonCeeMac you might also try hitting the horizon container on port 80 with 'X-Forwarded-Proto: https' and see if it doesn't redirect you14:28
*** nurdie has quit IRC14:28
*** nurdie has joined #openstack-ansible14:29
jamesdentonhaproxy frontend should offload ssl and send that header to the backend to avoid the redirect. default connection to the backend is 8014:29
CeeMacthen I am confused.  Saying that, the logs do show the connection on port 8014:32
CeeMacjamesdenton, how do i do the thing with the header?14:33
*** nurdie has quit IRC14:34
jamesdentonExample to my horizon container directly: curl -v http://172.29.236.167/auth/login/ -H 'X-Forwarded-Proto: https'14:34
nowsterjamesdenton: Do you mind if I pick your brains about networking?14:34
CeeMacah, ok, one sec14:34
jamesdentonCeeMac This is to the VIP: curl -k -v https://172.29.236.100/auth/login/14:34
jamesdentonIn both cases, I got a 200 OK and a page14:35
CeeMacjamesdenton, that returns xml14:35
jamesdentonthat's html14:35
jamesdentonor should be, anyway14:35
jamesdentonnowster what's up?14:36
*** shyamb has quit IRC14:36
CeeMacsorry, it is html14:36
CeeMacso, interesting.  If I use curl -k -v https://<internal-vip> then it returns valid code HTTP/1.1 200 OK14:37
jamesdentonCeeMac If the curl is working to the VIP and the container directly using those examples (with your IPs), then I think the OSA side is OK. Maybe the browser doesn't like the cert?14:37
CeeMacthats from my deployment node14:38
jamesdentonk14:38
CeeMaci'll try hooking up another VM to the network with a different browser14:38
nowsterjamesdenton: All the documentation assumes you're able to allocate IPs from the provider network interface. I'm trying to set up a routed network which exists on a VXLAN, but with the neutron host routing public traffic out to the upstream router.  (For extra difficulty, I'm trying to do this with IPv6.)14:38
jamesdentonCan you elaborate on this? >> All the documentation assumes you're able to allocate IPs from the provider network interface14:39
nowstereg. https://docs.openstack.org/newton/install-guide-ubuntu/launch-instance-networks-provider.html14:39
nowsterIn my case the provider gives you a single IP, but allows net blocks to be routed to it.14:40
jamesdentonOh i see.14:41
jamesdentonSo in that circumstance, normally i would expect some kind of hardware router or something that can take that IP on its WAN interface and have multiple LAN networks, any of which could be those routed nets14:42
nowsterwe don't have that14:42
jamesdentonThe normal Neutron router scenario does not really support that model14:42
nowsterWould using OVS work better?14:43
*** fnpanic has quit IRC14:43
jamesdentonNo, it's not really an OVS vs LB issue or anything like that. You need some sort of demarc where you can configure that address and provide routing for those add'l networks14:44
jrosserthis sounds like the IP on the host is the 'next hop' for this netblock then14:45
jamesdentonit's possible, i suppose, to have a neutron "provider" network with a subnet consisting of a single IP, and then you can configure the add'l networks to hang off that router. But it's sorta limiting14:45
jamesdentonvery possible. while it can be made to work, i'm not entirely sure how to make it work within the bounds of the "reference arch", if you know what i mean14:46
nowsterHow about setting up a virtual bridge network with that address range, then having the kernel know how to route for it?14:47
jamesdentonis this just for a PoC? prod?14:47
CeeMacjamesdenton, must be an internal routing issue, i've just tested access to the external LB VIP and that seems to be working fine14:47
CeeMacthanks anyway :)14:47
jamesdentonglad you eliminated OSA, anyway.14:47
jamesdentongood luck14:47
CeeMacthanks14:48
nowsterIt would eventually be production config.  It appears that my colleague's config uses kolla and OVS.14:48
*** nurdie has joined #openstack-ansible14:50
*** mgariepy has quit IRC14:50
jamesdentonWell, whatever that IP is configured on would be a SPOF unless you also implemented something like keepalived between the hosts, i would think14:51
nowsterThat's a later concern. ;-)14:52
jamesdentonAnd the vxlan between the hosts is part of this, too? A non-neutron vxlan mesh?14:52
nowsterforget the vxlan14:52
jamesdentonwell, how are the hosts connecting to one another for the various networks (br-mgmt, br-vlan, storage, etc)?14:53
nowsterthose exist14:53
nowsterexcept we can't do br-vlan, because the provider filters tagged traffic14:53
jamesdentonDid they give you vlans for those networks?14:56
jamesdentonor actual interfaces per network (untagged)?14:57
nowstervlans are allowed, but provider assigned (max 12 per project)14:57
jamesdentonk14:57
nowsterI'm thinking to bring up a "provider" vlan, and have that be "flat" type.14:58
*** nurdie has quit IRC14:59
jrosserthat sounds like a plan - you need to leave the door open to adding another controller15:01
nowsteryep15:02
nowsterOff to try to set that up...15:02
jamesdentonYou ought to be able to configure infra01 as the 'gateway' for one or more of those routed networks by configuring, say, x.x.x.1, on br-vlan. As long as the other hosts have connectivity to infra01 via whatever provider interface you've chosen, it should work (as a flat net)15:02
nowsteryep15:04
nowsterI think I'll call it br-provider to avoid confusion.15:04
jamesdentongood idea15:05
*** hwoarang has quit IRC15:07
*** hwoarang has joined #openstack-ansible15:08
*** strattao has quit IRC15:10
*** ztr has joined #openstack-ansible15:10
nowsterjamesdenton: I notice that sysctl net.ipv4.conf.*.forwarding is all 1, but the ipv6 equivalents are all 0. Should I be enabling them manually, or will neutron do that for me?15:16
jamesdentonIt might do it for you, but there's no harm in doing it, too15:16
nowsterta15:18
*** kukacz_ has quit IRC15:20
CeeMachmm15:21
CeeMaclooks like i might have spoken too soon15:21
openstackgerritJames Denton proposed openstack/openstack-ansible-os_neutron master: [WIP] Deploy Vector Packet Processing (VPP) Platform for Neutron  https://review.openstack.org/63164415:22
CeeMacok, so if i try and connect to the internal VIP from a machine on the same network, I get ERR_CONNECTION_REFUSED15:31
*** kukacz has joined #openstack-ansible15:31
*** nurdie has joined #openstack-ansible15:31
CeeMacIf I connect in through the external VIP I can log in, but get little popup errors for things not being able to be retrieved15:31
*** pcaruana has quit IRC15:35
*** kukacz has quit IRC15:42
*** ianychoi has joined #openstack-ansible15:44
*** kukacz has joined #openstack-ansible15:46
openstackgerritFrank Kloeker proposed openstack/openstack-ansible master: Change docs build to openstackdocstheme logic  https://review.openstack.org/62342915:46
*** udesale has quit IRC15:53
CeeMacok, so i've now got an ubunto desktop VM connecting on the br-mgmt network, it can log in to the internal VIP but also gets Error: unable to retrieve the <stuff> messages15:53
*** udesale has joined #openstack-ansible15:54
CeeMacany horizon experts in the house?15:54
openstackgerritJames Denton proposed openstack/openstack-ansible-os_neutron master: [WIP] Deploy Vector Packet Processing (VPP) Platform for Neutron  https://review.openstack.org/63164415:54
*** pcaruana has joined #openstack-ansible15:55
jamesdentonCeeMac Do those errors only occur on certain tabs?15:55
*** kukacz has quit IRC15:58
*** TxGirlGeek has joined #openstack-ansible15:58
CeeMacjamesdenton, I'm logged in as the 'admin' user, and it defaults to the 'service' project. If I switch to the 'admin' project and browse through the project | Compute tabs, every one throws an error15:59
*** kukacz has joined #openstack-ansible16:00
CeeMacsame for volumes16:00
CeeMacnetwork topology threw a "Something went wrong!" page16:00
jamesdentonYou know, it may be worth re-running setup-openstack playbook just incase something went wonky along the way16:01
CeeMacmaybe.  i'll refresh my snapshots and do that. Can't harm16:02
*** gyee has joined #openstack-ansible16:06
CeeMacsomething definitely looks wonky. openstack cli isn't working just now either16:11
CeeMacodd16:12
CeeMacgetting an error about role os_blazar not being found16:12
CeeMaci'm just going to comment that out for the moment16:14
*** gyee has quit IRC16:14
ztrchandankumar: I'm told that you are working on an issue affecting openstack-ansible-os_tempest on centOS16:15
CeeMacah, ok, cloudnull had me check out a patch for the remote-journal issue i was having16:17
CeeMacso i'm in that not in stable/rocky16:17
CeeMacif i checkout stable/rocky will i lose the patch?16:17
jamesdentonmore than likely, yes16:17
CeeMaci'm a total git n00b, sorry16:17
prometheanfiresomeone mind testing https://review.openstack.org/#/c/608393/44/tasks/lxc_pre_install.yml16:18
CeeMacso if i check out stable/rocky16:18
prometheanfirevia command line I get the correct thing, but via ansible it is truncating the url16:18
CeeMacis there an alternate way I can pulldown the patched playbook for this run so don't undo the fix?16:18
*** eumel8 has joined #openstack-ansible16:19
jamesdentonCeeMac I'll have to defer, sorry.16:20
eumel8o/16:21
chandankumarztr: yes16:21
ztrIs there a bug for this issue?16:21
CeeMacno problem, i'll do this anyway because i need the dashboard to work, then I'll catch cloudnull to see if i can get a cherry pick of his fix instead for afterwards16:22
chandankumarztr: if you are free, feel free to take a look at this issue16:22
chandankumarztr: http://logs.openstack.org/08/631708/4/check/openstack-ansible-functional-centos-7/cc0ab67/logs/openstack/tempest1/stestr_results.html16:22
chandankumarztr: it might be related to network cidr issue16:22
*** kukacz has quit IRC16:24
*** gyee has joined #openstack-ansible16:25
eumel8we have now OSA docs with the new openstackdocstheme release 1.29 in the pipeline. sneak preview: http://logs.openstack.org/29/623429/3/check/openstack-tox-docs/0795a7f/html/de/index.html16:25
eumel8new is that language menu on each page after sphinx build with the script in openstackdocstheme repo16:26
*** kukacz has joined #openstack-ansible16:26
ztrchandankumar: I think that is the same failure that is blocking me, if it is I'll file a bug for tracking purposes16:27
chandankumarztr: sure16:31
*** macza has joined #openstack-ansible16:33
FrankZhanganyone has idea why centOS gating consistently failed at tempest 'test_server_basic_ops'?  2019-01-23 18:42:09,306 4664 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@10.1.3.24 (timed out).16:33
chandankumarFrankZhang: We are still debugging why it is happening16:34
FrankZhangchandankumar: thanks, can you share the card we're tracking it?16:34
chandankumarztr: is creating a ticket for that16:35
FrankZhangCool thank you16:36
*** pcaruana has quit IRC16:48
*** asettle has quit IRC16:50
odyssey4meeumel8 lol @ english vs english(uk)16:51
odyssey4meBut that's awesome. :)16:51
ztrchandankumar: https://bugs.launchpad.net/openstack-ansible/+bug/181318716:52
openstackLaunchpad bug 1813187 in openstack-ansible "CentOS tempest test_server_basic_ops failure" [Undecided,New]16:52
eumel8odyssey4me: I'm not the super specialist for such kinds of differents ;)16:52
odyssey4mechandankumar was cloudnull trying to help you isolate the issue there?16:52
*** antonym has quit IRC16:53
CeeMaclooks like i done broke it16:57
CeeMachttp://paste.openstack.org/show/743236/16:57
CeeMacany help would be appreciated16:57
*** eumel8 has quit IRC16:57
CeeMacall front/back open/up in haproxy16:57
CeeMacopenstack cli wont connect now, dashboard won't work16:58
*** strattao has joined #openstack-ansible16:58
*** bgmccollum has joined #openstack-ansible17:00
odyssey4meCeeMac what exactly was done that resulted in requiring os_blazar?17:08
*** antonym has joined #openstack-ansible17:11
*** kopecmartin is now known as kopecmartin|off17:18
CeeMacodyssey4me, i think that came down from the git checkout cloudnull had me do for his journal patch17:22
odyssey4meCeeMac what exactly did you do, because that patch doesn't include any changes that would makr blazar required17:22
CeeMacodyssey4me, git fetch https://git.openstack.org/openstack/openstack-ansible refs/changes/05/632505/2 && git checkout FETCH_HEAD17:22
odyssey4meright, that's why - you did the 'check out' option, not the 'cherry-pick' option17:23
odyssey4meso basically you changed from rocky to master17:23
CeeMac*shrugs* i just did what I was told :/17:23
prometheanfiremachinectl import-image is timing out, it seems to have completed17:23
prometheanfireJan 24 17:05:37 os-infra-test systemd-importd[29126]: (transfer1) Operation completed successfully.17:23
CeeMacso, i've checked out stable/rocky again17:23
prometheanfireJan 24 17:05:37 os-infra-test systemd-importd[29126]: (transfer1) Exiting.17:23
odyssey4meCeeMac from the patch in question - https://review.openstack.org/#/c/632505/17:24
odyssey4medownload, o the top right17:24
odyssey4meclick on the clip-board next to 'cherry-pick' - not checkout17:24
CeeMacah, gotcha17:24
CeeMacok, i'l make a note of that for when things are working again17:24
CeeMacright now something is hinky and dashboard/keystone don't seem to work17:25
odyssey4meCeeMac well, if you're in some weird mix of rocky and stein then it's not surprising17:25
FrankZhangchandankumar: it seems centOS gating error became github accessing failure http://logs.openstack.org/19/617719/2/check/openstack-ansible-functional-centos-7/332b944/job-output.txt.gz#_2019-01-24_17_15_17_915497 after you recheck https://review.openstack.org/#/c/617719/17:25
CeeMaci only ran the infra-journal-remote.yml playbook from that checkout17:26
CeeMacand openstack cli has been working today since then17:26
CeeMacjust appears to have stopped recently17:26
CeeMacwhile I've been trying to troubleshoot dashboard access17:26
odyssey4meCeeMac that's always nice :/17:27
CeeMacyeah17:27
CeeMacthe only thing I can remember doing is restart apache2 in the horizon container, then restart the horizon container17:27
CeeMacand I'm sure it was working after that17:27
CeeMacthen it wasn't17:27
CeeMacdid you see the error I pasted?17:28
prometheanfireodyssey4me: cloudnull (I think both of you worked on the machinectl stuff most), import-tar is timing out because it expects uesr input...17:34
odyssey4meodyssey4me Is that for a new patch or what, because it hasn't required input before? Also, I did no work on anything relating to machinectl except to fix some ansible bugs here and there.17:35
ThiagoCMCHey guys, I just installed OSA Rocky on Ubuntu, everything looks good, except Heat API, there is an error on heat.log:17:36
ThiagoCMCERROR oslo.messaging._drivers.impl_rabbit [-] Unable to connect to AMQP server on 172.29.238.12:5671 after None tries: Connection.open: (530) NOT_ALLOWED - access to vhost '/heat' refused for user 'heat': NotAllowed: Connection.open: (530) NOT_ALLOWED - access to vhost '/heat' refused for user 'heat'17:36
ThiagoCMCAny idea?17:36
prometheanfireodyssey4me: ah, ok, not for a new patch17:37
prometheanfire:q17:37
prometheanfirefound the error though (it's in systemd itself, and fixed already)17:40
*** Darcidride has joined #openstack-ansible17:48
*** udesale has quit IRC18:01
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_tempest master: Adds tempest run command with --test-list option  https://review.openstack.org/63135118:03
chandankumarcloudnull: once we have env ready using run-test.sh functional18:12
chandankumarwhen i sourced .tox/functional env18:13
chandankumarcloudnull: ansible-playbook does not able to find hosts18:13
*** Darcidride has quit IRC18:17
*** Darcidride has joined #openstack-ansible18:18
odyssey4mechandankumar run_tests sets out some env vars and also provide CLI options which you need to add to make that work18:21
odyssey4meIIRC you need to set TESTING_DIR, and you need to set where the inventory is, and you need to supply the override file18:22
*** Darcidride has quit IRC18:26
chandankumarodyssey4me: got it18:26
*** Darcidride has joined #openstack-ansible18:26
*** Darcidride has quit IRC18:28
chandankumarodyssey4me: it worked thanks, https://github.com/openstack/openstack-ansible-tests/blob/master/test-ansible-syntax.sh#L3118:28
chandankumari ecported first 3 vars18:29
*** Darcidride has joined #openstack-ansible18:29
chandankumarrun ansible playbook18:29
odyssey4mechandankumar ok, then it should be something like: ./tox/functional/bin/ansible-playbook -i tests/inventory -e@tests/????_overrides.yml tests/test.yml18:30
odyssey4meoh, with ANSIBLE_INVENTORY set, you don't need the -i18:30
*** Darcidride has quit IRC18:31
*** Darcidride has joined #openstack-ansible18:31
*** Darcidride has quit IRC18:34
*** Darcidride has joined #openstack-ansible18:34
*** shardy has quit IRC18:43
*** TxGirlGe_ has joined #openstack-ansible18:59
*** TxGirlGeek has quit IRC19:00
*** ztr has quit IRC19:01
*** nurdie has quit IRC19:02
*** nurdie has joined #openstack-ansible19:02
chandankumarodyssey4me: there is a bug here http://git.openstack.org/cgit/openstack/openstack-ansible-tests/tree/test-vars.yml#n44919:04
odyssey4mechandankumar how so?19:04
chandankumarodyssey4me: when we use run run_tests.sh it create this file in /root/src19:04
chandankumarnot in development directoru19:05
odyssey4methat will only be if TESTING_HOME=/root19:05
*** nurdie has quit IRC19:06
chandankumarodyssey4me: yes correct, I exported Testing home it worked19:07
broken_onehey guys,  you happen to have an legalese for contributors and commiters?  we have reach the sharks and they want to look at docs19:08
chandankumarodyssey4me: I think we need a additonal script where we export folder so that one can run a playbook19:08
chandankumarodyssey4me: I will put up these things tomorrw in docs19:09
odyssey4mechandankumar I'll wait for those docs, or an etherpad or something. I'm not sure what you mean, but it can wait for tomorrow.19:09
chandankumarodyssey4me: sure19:09
odyssey4mebroken_one I think you might mean the Contributor License Agreement?19:09
odyssey4mebroken_one https://review.openstack.org/static/cla.html19:11
odyssey4mebroken_one note that https://docs.openstack.org/contributors/common/introduction.html has multiple ways of contributing, and they don't all ask for the CLA to be agreed to19:11
broken_oneodyssey4me: thank you19:14
broken_onedoes the CLA also cover corps?19:20
broken_oneor are we working as individuals within a corp?19:21
odyssey4mebroken_one oh, I dunno - it's perhaps best to ask in #openstack-dev, which has a broader set of people watching19:21
*** nurdie has joined #openstack-ansible19:22
broken_onewill do.  thank you again19:23
*** noonedeadpunk has quit IRC19:24
openstackgerritMatthew Thode proposed openstack/openstack-ansible-lxc_hosts master: add gentoo support  https://review.openstack.org/60839319:28
ThiagoCMCGuys, I'm trying to upload a raw Ubuntu image to Glance+Ceph, like this: `openstack image create --disk-format raw --container-format bare  --public --file ubuntu-18.04-server-cloudimg-amd64.raw ubuntu1804` - But there is an error on glance-api.log: "since image size is zero we will be doing resize-before-write for each chunk which will be considerably slower than normal", any tips?  =P19:29
ThiagoCMCI converted the ubuntu "img" to "raw" using `qemu-img convert`...19:30
ThiagoCMC`openstack image list` is empty19:30
*** TxGirlGe_ has quit IRC19:32
prometheanfirecloudnull: looks like networkd isn't used within containers?19:33
prometheanfirelxc_container_create that is19:33
*** Darcidride has quit IRC19:40
*** tosky has quit IRC19:44
*** Darcidride has joined #openstack-ansible19:54
*** strattao has quit IRC20:02
*** strattao has joined #openstack-ansible20:06
*** TxGirlGeek has joined #openstack-ansible20:26
ThiagoCMCSome ceph vars where wrong on cinder config (rbd_secret_uuid and the user)... I'm re-deploying everything from scratch again, including Ceph (all under MaaS).20:34
*** bgmccollum has quit IRC21:02
*** Darcidride has quit IRC21:10
*** hamzaachi has joined #openstack-ansible21:12
*** bgmccollum has joined #openstack-ansible21:24
*** ansmith has quit IRC21:28
*** hamzaachi has quit IRC21:31
*** cmart has joined #openstack-ansible22:15
cmartGood afternoon OSA22:16
*** cmart has left #openstack-ansible22:17
*** cmart has joined #openstack-ansible22:17
cmartI'm upgrading from Queens to Rocky, and the Rocky upgrade process is installing things in containers that should have been deprecated / removed in Queens, for example the cinder_scheduler_container and nova_api_os_compute container. If I destroy these containers and remove them from inventory, then the upgrade process re-creates them.22:17
cmartAny idea what may be going on?22:17
*** ansmith has joined #openstack-ansible22:18
*** nurdie has quit IRC22:18
*** nurdie has joined #openstack-ansible22:19
*** nurdie has quit IRC22:23
jrossercmart: looks like cinder_scheduler_container should have been removed for going from P->Q, which is probably why Q->R upgrade tooling doesnt touch it22:27
cmartjrosser yeah :(22:27
cmartI try to remove them from my inventory JSON manually, but apparently the inventory script is adding them back22:27
jrosserare they defined in /etc/openstack_deploy/env.d at all?22:28
ThiagoCMCQuick question: Why there is a btrfs volume mounted under /var/lib/machines (a loop0 dev) if the OSA containers are being stored under /var/lib/lxc ?22:29
jrosserThiagoCMC: that contains the container base image22:29
cmartjrosser I see this sort of thing in my /etc/openstack_deploy/env.d/cinder.yml: http://paste.openstack.org/show/743433/22:30
cmartso cinder_scheduler_container still "contains" "cinder_scheduler". Does that control how the inventory JSON gets built?22:31
jrosserThiagoCMC: you have many choices of lxc storage backend, dir/overlayfs/btrfs/zfs so depeding on which you use slightly different things happen to the base image22:31
jrossercmart: yeah, you should see that map to a physical host group there, which you'll then have defined in your openstack_user_config22:32
jrosserthat is deciding which containers get mapped to which hosts22:32
jrosserso by fiddling with the env.d contents you can add/delete which containers should exist22:33
cmartOK cool. so in that case my `/etc/openstack_deploy/env.d`is still defining stuff that should have been deprecated in Queens.22:33
cmartI don't think I've changed anything in there manually, so maybe i'll just replace the contents of `/etc/openstack_deploy/env.d` with what comes stock with OSA, and try again.22:33
cmartthanks very much for the breadcrumb jrosser :)22:34
openstackgerritMerged openstack/openstack-ansible-ops master: Update delegated setup hosts to support IP delegation  https://review.openstack.org/63220122:34
jrosserthere is stuff in the upgrade tooling which looks for duplicate /etc/openstack_deploy/env.d/.... and /opt/openstack-ansible/inventory/env.d/..... and tries to remove anything from /etc/.... that isnt changed from the dfault22:35
jrosserhowever as you seem to have a P version of that file it will be different from the R version in /opt/openstack-ansible so will not be touched22:35
jrosseronce you have resolved what is going in in /etc/openstack_deploy/env.d use scripts/inventory_manage.py to delete the inventory elements that you do not need any more22:36
*** bgmccollum has quit IRC22:36
cmartOH this is super helpful. "/opt/openstack-ansible/inventory/env.d/cinder.yml" looks like what I want.22:37
jrosserand take a backup first :)22:37
ThiagoCMCjrosser, where can I check if I have too many choices of str back? Also, is it possible to store / run the OSA LXC containers only from btrfs?22:37
cmartjrosser of course. already have 'em22:37
jrosserThiagoCMC: you can see the combinations of backing store that get tested for the lxc_hosts role here https://review.openstack.org/#/c/632118/22:40
jrosserThiagoCMC: but probably stick with the default unless you have a good reason to change it?22:41
*** bgmccollum has joined #openstack-ansible22:46
ThiagoCMCjrosser, interesting... I didn't knew that this (not run the containers from within the btrfs) is the default, I thought that I was doing something wrong. lol22:49
*** spsurya has quit IRC23:04
ThiagoCMCJust curious, will OSA have LXD support for its containers, someday?  :-P23:10
*** TxGirlGeek has quit IRC23:15

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!