Thursday, 2018-02-22

*** pbandark has quit IRC00:13
*** lbragstad has quit IRC00:18
*** chyka has quit IRC00:37
*** Miouge has quit IRC00:43
*** acormier has joined #openstack-ansible01:29
*** acormier has quit IRC01:30
*** acormier has joined #openstack-ansible01:30
*** esberglu has quit IRC01:35
*** lbragstad has joined #openstack-ansible01:56
*** woodard_ has joined #openstack-ansible01:57
*** woodard has quit IRC01:57
*** acormier has quit IRC01:58
*** chyka has joined #openstack-ansible02:10
*** chyka has quit IRC02:14
*** dave-mccowan has joined #openstack-ansible02:28
*** dave-mccowan has quit IRC02:58
*** andymccr has quit IRC03:00
*** andymccr has joined #openstack-ansible03:03
*** lbragstad has quit IRC03:04
*** acormier has joined #openstack-ansible03:12
*** acormier has quit IRC03:27
*** dave-mccowan has joined #openstack-ansible03:38
*** gkadam has quit IRC04:01
*** gkadam has joined #openstack-ansible04:02
*** udesale has joined #openstack-ansible04:04
*** udesale_ has joined #openstack-ansible04:07
*** dave-mccowan has quit IRC04:09
*** nikm has quit IRC04:14
*** udesale_ has quit IRC04:31
*** udesale_ has joined #openstack-ansible04:31
*** udesale has quit IRC04:32
openstackgerritMerged openstack/openstack-ansible master: [Docs] Include test scenario as a new user story
*** gkadam has quit IRC04:41
*** gkadam has joined #openstack-ansible04:51
*** poopcat has quit IRC05:08
gokhanhi folks, I destroyed and recreated rabbitmq containers bacause of some reasons and then now rabbitmq users are missing. is there a better way to create rabbitmq users instead of running setup-openstack.yml ?05:34
openstackgerritMerged openstack/openstack-ansible-os_ceilometer stable/ocata: Deprecate auth_plugin option
openstackgerritMerged openstack/openstack-ansible-os_gnocchi stable/queens: Zuul: Remove project name
openstackgerritMerged openstack/openstack-ansible-os_gnocchi master: Change default gnocchi ceph pool name to metrics
openstackgerritMerged openstack/openstack-ansible-os_gnocchi stable/pike: Zuul: Remove project name
cloudnullgokhan: yes. you still need to run setup-openstack.yml but you can do so with a tag05:43
cloudnullto recreate the rabbitmq users05:43
cloudnullthe tag is common-rabbitmq05:43
cloudnullto see all of the tags you can use the --list-tags option05:43
gokhancloudnull, I see that we can run with  --step tag is it right tag ?05:44
*** bhujay has joined #openstack-ansible05:45
cloudnullthe command would be `openstack-ansible setup-openstack.yml --tags common-rabbitmq`05:46
*** hybridpollo has quit IRC05:47
gokhancloudnull, ok thanks. also I see that on rabbitmq containers dbus package is missing05:55
gokhancloudnull, because of it when run systemctl status rabbitmq-server we see warning like beware of timeouts05:56
*** bhujay has quit IRC06:00
*** bhujay has joined #openstack-ansible06:04
*** aruns has joined #openstack-ansible06:04
*** aruns__ has joined #openstack-ansible06:05
*** nikm has joined #openstack-ansible06:08
nikmevrardjp: hi06:08
*** aruns has quit IRC06:09
nikmevrardjp: there is a diff between cert files of repo container and cert file of controller host in HA env deployed using openstack-ansible ocata06:10
*** dariko has quit IRC06:10
nikmevrardjp: thats why we were getting the curl issue
nikmwhen we copied manually the cert file of host to the container, the issue is getting solved06:12
nikmhow do we automate the copying of correct cert from host to containers06:12
cloudnullgokhan: what release are you running. Ive seen that error in early pike06:14
cloudnullyou can install the dbus package06:15
gokhancloudnull, yes on pike 16.0.4 release. I installed it. of if it is already solved , there is no problem thanks :)06:16
cloudnullI was looking for the patch06:17
cloudnullbut i cant quickly find it06:17
cloudnulli do however know we fixed it06:17
cloudnullbut im glad you got it orted06:18
nikmdo any one else know ?06:19
nikmhow do we copy the correct certs inside /etc/ssl/certs of container from controller host06:19
nikmwe are using openstack-ansible ocata on centos 706:20
nikm  are the issues which came inside repo container and was not coming on the host hosting containers06:21
nikmwhen we copied manually the certs from /etc/ssl/certs of the host to the container, the issue gets resolved06:21
cloudnullnikm: there are overrides for the various cert files however if you're using haproxy you should only need to override the one.06:21
cloudnullall of the ssl is normally terminated at the lb06:21
cloudnullare you seeing an ssl issue when hitting the repo containers directly ?06:22
gokhancloudnull, when run with common-rabbitmq tags, it gives error on ceph client.  ceph_mon_host is undefined. I ignored this error.06:24
nikmcloudnull: curl is failing inside repo container and there is no issue of host06:25
nikmso we took a diff on the cert files of host and container06:26
nikmso we found one subject line is missing in container cert06:26
nikmso when we manually copied cert from host to container06:26
nikmcurl command is started working06:27
*** acormier has joined #openstack-ansible06:27
nikmand also openstack-ansible setup-infrastructure.yml not failed on
nikmcloudnull : can u give the link for this "if you're using haproxy you should only need to override the one."06:29
*** acormier has quit IRC06:31
nikmcloudnull : can u tell the overrirides06:38
nikmhaproxy is not configured till now06:39
ThomasSmoring guys06:44
*** bhujay has quit IRC06:57
*** ianychoi has quit IRC07:01
*** arbrandes1 has joined #openstack-ansible07:01
*** ianychoi has joined #openstack-ansible07:01
*** mardim has joined #openstack-ansible07:03
cloudnullnikm: the overrides for haproxy can all be found here
*** arbrandes has quit IRC07:03
nikmcloudnull: will it copy the host certs inside repo container07:05
*** SmearedBeard has joined #openstack-ansible07:06
nikmif we overrides haproxy07:06
*** bhujay has joined #openstack-ansible07:07
*** vedin has joined #openstack-ansible07:11
*** bhujay has quit IRC07:13
vedinHi Everyone, I am facing while running setup-infrastructure playbook in galara container, it is giving the following issue >>>
vedinthis issue looks like HAproxy, I tried to re run the HAproxy playbook, but I am not able to ping internal LB ip07:15
nikmhi guys07:23
nikmhow do we get log file in
nikmhaproxy log file   /var/log/haproxy07:24
*** aruns__ has quit IRC07:34
*** aruns__ has joined #openstack-ansible07:36
*** pcaruana has joined #openstack-ansible07:37
*** john51 has quit IRC07:39
sarvedin: I've also had that error. Tt was a haproxy/keepalived issue. What is the status of the keepalived and haproxy services on the haproxy hosts?07:43
*** armaan has quit IRC07:48
*** armaan has joined #openstack-ansible07:48
*** SmearedBeard has quit IRC07:53
*** Miouge has joined #openstack-ansible07:54
*** chyka has joined #openstack-ansible07:56
*** lvdombrkr has joined #openstack-ansible07:59
*** lvdombrkr has quit IRC07:59
*** lvdombrkr has joined #openstack-ansible07:59
*** chyka has quit IRC08:01
*** epalper has joined #openstack-ansible08:10
*** threestrands_ has joined #openstack-ansible08:15
*** admin0 has joined #openstack-ansible08:16
admin0morning \o08:16
*** threestrands has quit IRC08:18
*** mbuil has joined #openstack-ansible08:20
*** jwitko_ has quit IRC08:25
sarmorning evrardjp08:28
lvdombrkrmorning is possible connect linux bridge to ovs bridge?08:28
evrardjpnikm: hey -- to modify containers you have two choices08:28
evrardjpeither modify the base container that gets used when creating new containers08:29
evrardjpOr you modify the containers at their creation08:29
evrardjp(you have a third option that could be modifying the container later, but that sounds bad)08:29
admin0i have  strange issue .. greenfield new deployment:
admin0lxc_hosts : Retrieve base image08:30
evrardjplvdombrkr: while it would be technically possible, I doubt it's a great idea to mix and match on the same host08:30
evrardjpadmin0: there is a thing going on about that08:30
admin0aha ..08:30
evrardjpwe did something -- then we changed it -- then it broke the upgrades -- so we add a compatibility layer08:31
*** sxc731 has joined #openstack-ansible08:31
evrardjpadmin0: you might want to check in this commit:
evrardjpit has links to changes08:32
evrardjpdepending on your version you might be hitting something.08:32
evrardjpvedin: there are many causes that can lead to "no route to host" It depends on your config.08:33
evrardjpsar: good morning :)08:33
vedinsar : haproxy service is running but keepalived is not08:33
evrardjplvdombrkr: what are you trying to achieve?08:33
evrardjpvedin: how many hosts do you have for haproxy hosts?08:33
admin0hmm. 1TB space in the variables08:33
evrardjpadmin0: check also the history of the bug, it might be something that's wrongly reported/need reboot or fancy stuff.08:34
evrardjpvedin: if you have only one node, that's normal to not have keepalived.08:34
evrardjpvedin: if you have more than one, your keepalived configuration is busted. Please give us your internal_lb_vip_address and external one + your user_*.yml variables.08:35
vedinevrardjp: I am using internal_lb_vip_address:, and external_lb_vip_address:
cmorellimorning guys. thanks to cloudnull and evrardjp for the help yesterday08:38
cmorelliI finally managed to finish the deployment08:38
vedinevrardjp: internal_lb_vip_address is from managment network and external_lb_vip_address form external network08:39
admin0lvdombrkr, don't :D08:39
cmorellibut now if I try to connect to Horizon dashboard I get this error "CSRF verification failed. Request aborted." right after the Horizon admin login08:39
lvdombrkrevrardjp: nothing yet, i just want know it is possible.. cant find any information around08:39
admin0err ..08:40
admin0i setup that variable and retrying again08:40
lvdombrkradmin0 evrardjp: so yes or no? )))08:40
*** electrofelix has joined #openstack-ansible08:40
admin0lvdombrkr, i am setting up a new env today .. it uses ovs .. but needs some creative thinking08:41
admin0like only vlan and vxlan on ovs .. and only on computes . not on controllers as the script fails . . and network is on metal on one of the compute nodes and not on controllers08:41
admin0doing that, can work with ovs and get things done08:41
sarvedin: do you have multiple haproxy hosts, or just one?08:42
cmorellimore or less the same as shown here
vedinsar: I am using 3 nodes for haproxy08:43
sarand you haven't specifically told it not to use keepalived? i think it defaults to installing keepalived if you have more than one haproxy host08:43
sarif so, i guess keepalived should be running, and one of the hosts should have the vip's assigned08:44
vedinsar : I have this configuration for keepalived >>
vedinIn all nodes keepalived service is down08:47
*** gillesMo has joined #openstack-ansible08:48
sarok, then that is the problem08:48
sarwhat is the output of systemctl status keepalived.service ?08:48
admin0evrardjp, this space requirement, that is on which node08:52
evrardjplvdombrkr: ovs is getting more traction lately and is tested in opnfv. We plan to introduce a scenario soon08:52
gillesMoHello, I have duplicate hypervisors and also a non-fonctionning overview page in horizon. I read that bug : and it is my problem, I have 3 cells (cell0 and 2 cell1). I dont't know if it's may upgrade from Ocata to Pike or a change my internal_lb_vip_address (impacts endpoints URLs).08:53
openstackLaunchpad bug 1736731 in openstack-ansible "os_nova might create a duplicate cell1" [High,Confirmed] - Assigned to Jean-Philippe Evrard (jean-philippe-evrard)08:53
evrardjpbut right now it's not as tested as lxb08:53
evrardjpunless you really need it, I'd say use lxb08:53
admin0lvdombrkr, i can share my configs on my hybrid ovs setup08:53
gillesMoHow many cells do we really need ? cell0, cell1, both ?08:53
evrardjpadmin0: container hosts08:53
evrardjpgillesMo: at least cell0 cell108:53
armaanevrardjp: Hello, how are you today?08:53
evrardjparmaan: hello!08:53
admin0i have 734G   gb free08:53
armaanlogan-: Hello, are you around?08:53
evrardjpgood and you ?08:54
evrardjpadmin0: then you're hitting the bug08:54
armaanevrardjp: same as usual, struggling with something in the Ceph land :)08:54
evrardjphahaha who doesn't :)08:54
evrardjpandymccr: and logan- are the best for your case08:54
admin0setting up that var in user_variables did not helped08:54
evrardjpvedin: hmmmm08:55
armaanI am just wondering if anyone know about the stable-3.0-good branch in ceph-ansible08:55
evrardjpit should be up in all nodes08:55
lvdombrkradmin0: yes it will be useful08:55
vedinsar: keepalived service output >>
evrardjpvedin: the interfaces exist on the host?08:55
gillesMoevrardjp: OK, thanks. After deleting one of my 2 cell1, it works, but after updating cell0 (to correct the endpoint URL), I'm now getting duplilcate hypervisors/compute services...08:55
vedinevrardjp: yes it is ans associated with br-flat bridge08:56
evrardjpvedin: ahah08:57
evrardjpfound the issue -- read your log08:57
evrardjpvs L5-6 on
evrardjpmisconfiguration -- did you re-run the haproxy playbook after changing vars?08:58
admin0evrardjp, totally lost on how to move ahead :D08:58
evrardjpgillesMo: wow.08:58
vedinevrardjp:  yes, but that interface is for external network for openstack08:58
admin0lvdombrkr, i am going to create a quick writeup on the current state of  ansible + ovs08:58
evrardjpgillesMo: why don't you use our playbooks?08:59
vedinit is saying to provide IP08:59
*** pbandark has joined #openstack-ansible08:59
evrardjpit should modify the endpoints08:59
vedinbr-flat is associated with that interface08:59
evrardjpvedin: drop your keepalived config08:59
evrardjpthis way I can understand08:59
evrardjpnetwork config and keepalived config and I have a full view :)08:59
evrardjpadmin0: you might be hitting a bug that's severe.09:00
evrardjpwhich branch/version?09:00
evrardjpchanging the variable wouldn't do things without the update of the roles09:00
vedinevrardjp: How to do that ?09:00
gillesMoevrardjp: which playbook ? Of course I ran multiple times the os-* playbooks or setup-openstack.yml, haproxy-install.yml to chnage the endpoints URL, but I think that's what leads to my duplicate cell09:00
evrardjpvedin: cat /etc/keepalived/keepalived.conf | pastebinit09:01
evrardjpvedin: cat /etc/network/interfaces | pastebinit09:01
evrardjpI love myuseless cat09:01
evrardjpyou can ofc use pastebinit /etc/network/interfaces09:01
vedinI am using centOS09:01
evrardjpvedin: ... you get the idea :)09:02
evrardjpgillesMo: that's the issue we have to solve then09:02
evrardjpif you can redo-it, and file a bug that would be great.09:03
evrardjp The idea is that we should adapt automatically.09:03
evrardjpif it's dangerous, maybe throw a warning or something.09:03
*** pbandark has quit IRC09:03
gillesMoevrardjp: OK but I think this bug describe it well :
openstackLaunchpad bug 1736731 in openstack-ansible "os_nova might create a duplicate cell1" [High,Confirmed] - Assigned to Jean-Philippe Evrard (jean-philippe-evrard)09:03
*** Sha000000 has joined #openstack-ansible09:03
gillesMoSorry !09:04
evrardjpit's alright, I just didn't get any cycles to do it. Imagine!09:04
evrardjpthat's a high bug.09:04
evrardjpany help is welcomed.09:04
vedinevrardjp :  keepalive configuration >> >>> intrfaces >>
evrardjpvedin: is haproxy running?09:05
evrardjpvedin: that's good. So next question -- why eno209:06
vedinbecause that interface is related to that IP
evrardjpit seems like the interface that will carry traffic is br-flat right?09:06
*** pbandark has joined #openstack-ansible09:07
evrardjpso why not having br-flat in the config?09:07
vedinbefore it was br-flat only, then also same issue was throwing09:07
admin0evrardjp, is there a known tag that works ?09:07
evrardjpmmm let's be consistent there09:07
vedinok, I will replace with br-flat now09:08
evrardjpvedin: could you show me your br- information ?09:08
evrardjpbecause I don't see bridges IPs there09:08
evrardjpvedin: also: why is the ping IP instead of 192.168.64.<external router ip>  ?09:09
evrardjpit would make more sense to shut down if you can't reach the network09:10
evrardjpif a node is disconnected from the network at least09:10
vedinevrardjp: bridge interfaces >>
evrardjpit can work on the internal side, it just loses business value imo09:10
evrardjp(test clustering is already what vrrp is doing)09:11
cmorelliI found out probably what my problem is. Browsing the OS documentation it seems that I need to change a parameter in /etc/openstack_dashboard/, because i'm using HaProxy without SSL and I have to change a boolean flag over there. Howto do this with openstack-ansible though?09:11
evrardjpvedin: br-flat doesn't have any networking?09:11
evrardjpsorry wrongly said09:11
cmorelliI mean, what is the correct procedure?09:11
evrardjpno IP?09:11
vedinthis bridge I want to use for openstack provider network09:12
evrardjpoh ok09:12
evrardjpso it's dedicated to that09:12
evrardjpok let's do something different then09:12
evrardjpvedin: please configure your external lb vip address to an ip (or a dns name that points to an ip) in the range of your internal lb vip address. but a different one than the internal vip. Configure your user_* to this address, and use the nic as br-mgmt.09:13
admin0vedin, if your router/NAT can see mgmt, you can use the same IP range in both09:14
evrardjpbecause that's effectively what you're doing: all your api traffic will flow through that network.09:14
admin0one will be for internal haproxy .. one will be external09:14
evrardjpyour tenant traffic would still be isolated on your flat network, and no overlap. Problem solved!09:15
evrardjpgillesMo: I am very sorry you have that issue right now.09:15
evrardjpIn the meantime, I guess CLI calls are your best choice.09:15
evrardjpadmin0: to my knowledge they all work but start to fail after the first reboot, and that's what I think you're hitting. And that's what the bug is fixing.09:16
admin0evrardjp, is there a known case of a windows95 style reboot of the controller nodes (which is also the container hosts) and retry script to fix ?09:18
*** hamzy_ has quit IRC09:20
*** hamzy_ has joined #openstack-ansible09:20
evrardjpadmin0: I am not sure to understand what you mean -- retry script to fix?09:20
admin0i mean i cannot figure out how to fix and move ahead with the setup-hosts09:21
admin0so was asking if a reboot of the controllers fix it09:22
evrardjpadmin0: I didn't get the chance to work on that -- you should probably contact other ppl, like cloudnull09:24
evrardjpmaybe jrosser has seen that09:24
nikmevrardjp : how do I change base container or containers at creation09:30
vedinevrardjp: if I want to use external network ip for external lb vip, then what are the change I need to do in my environment ??09:30
evrardjpnikm: it's in lxc_container_create role (for at creation) or lxc_host (for base)09:31
evrardjpnikm: these are cache prep commands for example:
evrardjpyou can override some for your use case09:32
nikmevrardjp: ok thanks09:33
evrardjpvedin: not sure what you mean -- you mean using an ip from the same public range for provider network/tenant network AND api ?09:33
evrardjpvedin: it's technically doable, but I'd put that into a separate nic. separation of concerns.09:34
vedinyes, form same external network09:34
evrardjpYou can do veth pairs and stuff like that to overcome the problem.09:34
evrardjpthat's up to you09:34
evrardjpbut you can't use the carrying interface for assigning an IP -- you must assign to the bridge. And then the bridge cannot be passed to neutron09:35
evrardjpso you could have a br-ext that's wired to br-flat09:35
evrardjpand br-ext having an IP09:35
evrardjpvedin: we are using this kind of trickery in
evrardjpnot exactly the same for your use case though09:37
*** threestrands_ has quit IRC09:37
admin0that is my config09:38
admin0on osa+ovs09:38
admin0this is working on PoC .. now trying to do a real deplyment and stuck on the setup-hosts  lxc stuff09:38
evrardjpvedin: oh you're on centos -- maybe you don't need both09:39
evrardjpyou can maybe just have a veth that's plugged into the host and into the br-flat.09:40
evrardjp(yeah that's on the same host)09:40
admin0i am going to put my hosts mapping as well09:40
admin0so that its more clear09:40
evrardjpbut on the host you could use that veth end for assigning an ip09:40
evrardjpthat should do the trick without br-ext.09:41
evrardjpplease note your traffic would still overlap -- You'd be on the same bridge and all.09:41
evrardjpso tenant traffic can crush your apis09:41
evrardjpbut you can implement tc on outbound. which is not fully helpful but a good start.09:42
admin0lvdombrkr, updated:
admin0hopefully it will help09:42
evrardjpvedin: does that help?09:43
*** gkadam_ has joined #openstack-ansible09:43
*** gkadam has quit IRC09:44
*** gkadam_ has quit IRC09:44
*** gkadam_ has joined #openstack-ansible09:44
lvdombrkradmin0: thanks i will look into09:45
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Fix references
nikmevrardjp: what about giving here /etc/ssl/certs
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Move more examples to user guide
nikmevrardjp: will it copy the certs from host09:49
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Move Ceph example to user guides
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Move network architecture into reference
*** aruns has joined #openstack-ansible09:50
*** aruns__ has quit IRC09:50
evrardjpnikm: you therefore have better:
*** aruns__ has joined #openstack-ansible09:51
evrardjpthis work is great right? :p09:51
*** SmearedBeard has joined #openstack-ansible09:52
*** aruns has quit IRC09:54
admin0i set my variable lxc_host_machine_volume_size: 500G and will retry09:55
admin0or i need to cherrypick that patch evrardjp ?09:56
evrardjpadmin0: check on your existing code , and maybe look for the previous days conversations.09:56
evrardjpI can't help you there.09:56
admin0when will it go into stable/pike :D ?09:57
admin0so that nothing "fancy/extra" needs to be done and documented09:57
evrardjpneeds backporting first, then need bump10:03
evrardjpI can't say but at least 2 weeks.10:03
*** aruns__ has quit IRC10:03
openstackgerritPeriyasamy Palanisamy proposed openstack/openstack-ansible master: Make Opendaylight as the BGP speaker using Quagga
*** aruns__ has joined #openstack-ansible10:04
admin0evrardjp, is there also a switch feature to move from new lxc to old lxc behaviour .. ?10:10
admin0that might fix right10:10
admin0nah .. manually bump up the quota :D10:12
openstackgerritMerged openstack/openstack-ansible master: [Docs] Fix references
admin0 machinectl set-limit infinity10:13
admin0  :D10:13
openstackgerritMerged openstack/openstack-ansible master: [Docs] Move more examples to user guide
openstackgerritMerged openstack/openstack-ansible master: [Docs] Move Ceph example to user guides
openstackgerritMerged openstack/openstack-ansible master: [Docs] Move network architecture into reference
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: Remove periodic translations job
vedinevrardjp : You mean, we should have IP assigned interface in all controller nodes which we want to use for external lb10:35
Taseerevrardjp: is there anything I can do to mitigate =>
admin0evrardjp,  so I had to set-limit 500G and then systemctl restart /var/lib/machines to fix it and move forward10:37
evrardjpvedin: I think it's advised to separate tenant traffic than API traffic, so it would be wise indeed to have a external_lb_vip_address on an ip/dns that's different. But like I said it's possible to do both.10:39
evrardjpadmin0: does it work?10:39
admin0ansible worked fine in one machine i used limt and test10:39
admin0now running the playbook like normal10:40
admin0if it worked on 1, should work on the other 2 controllers as well10:40
admin0will update10:40
evrardjpTaseer: you'll have to debug it yourself I am afraid.10:40
*** nattanon has joined #openstack-ansible10:40
nattanonHello !!! guys. Really need a help !!!10:41
evrardjpnattanon: hello10:41
evrardjpwe'll try to do our best with the available resources.10:41
admin0hello nattanon10:41
gokhanhi evrardjp odyssey4me, I have problem on rabbitmq clusters. I have 2 different environments with same configs. but on one of them rabbitmq gives error report about timeout. this is some logs: also get errors on nova compute logs like that: . What can be reason of this ? it is weird on my second environment there is no timeout error. only difference between my environme10:42
gokhannts, servers are different brand. what is your thoughts ?10:42
nattanonI'm running OSA pike version with tag 16.0.810:42
admin0gokhan,  different brand = ?10:42
admin0whats a brand ?10:42
admin0different os, different tags, different hardware ?10:42
gokhanadmin0, I mean hp and dell servers10:43
admin0what does tcpcump show ?10:43
nattanonThen i face through the problems with evacuate VM provision using volume10:43
admin0some hp's have crappy network cards :D10:43
gokhanadmin0, you are right. problem is on hp servers10:43
*** gunix has left #openstack-ansible10:43
admin0dump hp .. move to dell10:44
admin0speaking from experience ( might not be true in your case ) .. change to a different network card then what comes in HP and issue solved10:45
nattanonDo you guys have any idea for that ?10:45
admin0nattanon,  logs ?10:45
admin0 problems is a very broad term as well .. like you had pain in the finger when typing evacuate :D10:46
admin0nattanon, the vm's don't want to move to new hypervisors ?10:47
gokhanadmin0, yep move to dell :) but now it is very diffucult to change network card :(10:49
admin0you unscrew the old one out and screw the new one in10:49
nattanon@admin0 , VM moved to new hypervisor but cant boot. Note that i'm using volume be a boot disk.10:50
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible stable/pike: Update existing container_networks
openstackgerritMaxime Guyot proposed openstack/openstack-ansible master: Ceph RadosGW integration
*** indistylo has joined #openstack-ansible10:54
admin0nattanon,  what is in the console ?10:54
*** gkadam__ has joined #openstack-ansible10:54
*** aruns__ has quit IRC10:55
nattanon@admin0 Wait a sec let me give you all data about us, I'm preparing.10:55
*** gkadam_ has quit IRC10:57
gokhanadmin0, yep I asked boss and we sold melanox 25gb cards but we can use them 2 months later :( and is there otherway to solve which you advice ? maybe increase timeout time ? and also there is abug rabbitmq itself. one process beam.smp consumes more cpu10:58
*** manuelbuil has joined #openstack-ansible10:58
gokhan*sold took10:58
cmorellihi guys, any help on this?
*** EmilienM has quit IRC10:59
*** mbuil has quit IRC10:59
*** nwonknu has quit IRC10:59
*** mcarden has quit IRC10:59
*** mattoliverau has quit IRC10:59
cmorelliI get this error using 15.0.15 playbooks10:59
nattanon@admin0 Ordering event after compute1 is going down we evacuate all vm to compute2 all vm been moved fine but can't boot OS.10:59
cmorellitrying to login to the dashboard for the first time10:59
nattanon ---------- this is error log from console.10:59
admin0cmorelli, not using SSL is a crime :D10:59
*** mattoliverau has joined #openstack-ansible11:00
*** mcarden has joined #openstack-ansible11:00
*** EmilienM has joined #openstack-ansible11:00
admin0nattanon,  i think it happens if the disk was busy during the crash . use the nova rescue, boot to the new image, mount this disk and fsck11:00
cmorelliadmin0: am I doing something wrong? I'm setting `haproxy_ssl: false` in user-variables before the deployment, otherwise the setup-openstack.yml playbook fails11:00
cmorelliI appreciate the help ;)11:01
cmorelliI know not using ssl is a crime, but I would like to see it working first11:01
admin0cmorelli, not even using proper hostnames ?11:02
*** nwonknu has joined #openstack-ansible11:03
nikmevrardjp : while creating container which file of can we use for copying files from host11:03
nikmsince we do not want to recreate container base image11:03
cmorelli@admin0, what do you mean?11:04
nattanonThen we trying to delete instance cant be completely deleted volume can't be delete and my compute is going down then can't going forever with  this error.11:04
nikmevrardjp: will container base image will be recreated with # openstack-ansible setup-infrastructure.yml --syntax-check11:05
nikmopenstack-ansible setup-infrastructure.yml11:05
nattanon@admin0 i will try your advice first.11:06
*** pcaruana has quit IRC11:07
admin0nattanon, first in the compute node the vms are migrated, use the nova , do the fsck and fix the volumes so that they boot .. client happy11:07
gokhanevrardjp, how can we reach rabbitmq on browser ? I mean which user I must use ?11:07
admin0then work on the other one which gives this or that error to fix11:07
admin0evrardjp,  how do I increase: Ensure that the LXC cache has been prepared (X retries left).11:08
admin0is there a var for it11:08
admin0so it works in 1 contrller11:08
admin0but in the other 2 it failed .. but not due to that limit error11:08
admin0i think the disks on raid  ae a bit slow11:09
admin0so they are not getting enough time to expand as the retry is already reached11:09
*** alex____ has joined #openstack-ansible11:09
*** alex____ has quit IRC11:09
*** fusmu has joined #openstack-ansible11:10
evrardjpadmin0: I don't think we have a var for that let me double check11:11
evrardjplxc_cache_prep_timeout: 600011:12
nattanon@admin0 I'm curious that instance with volume can be evacuate as a standard feature ??11:12
nikmevrardjp: hi11:13
admin0nattanon,  it is a standard feature11:13
admin0did you evacuate after or before a crash11:14
admin0like pre-maintenance migration or post-crash evacuation11:14
cmorelli@admin:  this is exactly what I did: in /etc/openstack-deploy.yml I set `openstack_service_publicuri_proto: http` and `haproxy_ssl: false`; then I had to modify CSRF_COOKIE_SECURE=False and SESSION_COOKIE_SECURE=Falise in ansible conf `/etc/ansible/roles/os_horizon/templates/`11:14
nikmevrardjp: where can we copy files in
nattanonbefore it crash .11:14
cmorellistill, I get the CSRF error after login11:14
nattanon@admin0 before it crash.11:14
admin0how many moved, how many worked fine , how may not worked @ all ?11:15
nattanonadmin0: All failed with boot. but locate new hypervisor is fine.11:15
admin0does nova rescue helped ?11:16
admin0used it before ?11:16
admin0also what is the backend ? nfs, ceph, iscsi ?11:16
admin0@cmorelli, never tried that use case .. so no idea :)11:17
admin0mine is ssl, proper certificate and hostname mapping for both ext and int = must haves even before touching ansible11:17
evrardjpnikm: don't copy the files, just pass the variables of the path on the host of the files you want to copy to container cache11:18
nattanonadmin0: Need 5 min my compute is down so need move it out and then plug it in a gain like i said delete evacuated vm make my compute gone away.11:18
*** Jack_Iv has joined #openstack-ansible11:19
nattanonadmin0: More information cinder volume backend using ceph.11:21
admin0nattanon,  what command u used ?11:22
admin0to evacuate ?11:22
admin0evrardjp, retrying with value set ot 1200011:22
*** epalper has quit IRC11:23
nikmevrardjp: what is the variable name in
nattanonadmin0: button on horizon11:24
nikmto be used11:24
*** portante has quit IRC11:24
admin0nikm, what exactly are you trying to do ?11:24
nattanonadmin0: Oooopss !!!!! nova rescue is work.11:24
nikmevrardjp : like you told for base container image
admin0is working ? or more work for you ?11:25
nikmevrardjp: i want to copy host cert files to an existing container11:25
nikmwhile running openstack-ansible setup-infrastructure.yml11:25
admin0nikm, isn't that controlled using a variable and setup during haproxy run ?11:25
nattanonadmin0: sad it more work for me. T_T11:25
admin0oh .. well, there is no magic button :)11:26
*** mattoliverau_ has joined #openstack-ansible11:27
nattanonadmin0: Actually, evacuate instance with volume should be work fine right with no more operation.11:27
nikmadmin0 :  do u mean variables11:28
nattanonadmin0: Should i back to check nova,cinder,ceph configuration ?11:28
admin0nikm, haproxy_user_ssl_cert: and haproxy_user_ssl_key:11:32
admin0point to the location where you key and cert is in the deoploy11:32
admin0and re-run haproxy-setup.yml11:32
admin0nattanon, should be = yes, is it always = no11:32
*** Faster-Fanboi_ has joined #openstack-ansible11:32
*** epalper has joined #openstack-ansible11:33
admin0you said more work .. so here is how it goes .. you have to do a nova resuce to every uuid .. and then boot into rescue .. but the rescue pass shown always does not work .. so you have to catch grub of the resuce system, and rescue itself ( single user mode ) and then fsck the vdb ( vda is the rescue itself ) and reboot11:33
admin0for each machines11:33
admin0do 1 verify that you see vdb and fsck works and that you can boot up the machine11:34
admin0checking conf has nothing to do with this11:34
cmorelli@admin, to use SSL do I need to generate certificates beforehand? I read somewhere that the playbooks would generate self-signed certs automatically?11:34
*** mattoliverau has quit IRC11:34
*** bradm has quit IRC11:34
*** Faster-Fanboi has quit IRC11:34
admin0cmorelli, it will generate itself11:34
admin0use haproxy_ssl_self_signed_regen: true11:34
*** portante has joined #openstack-ansible11:35
openstackgerritMerged openstack/openstack-ansible-galera_client master: Fix cache update after initial apt_repository fail
admin0cmorelli, you can easily map the ip and domain and get one from letsencrypt :) good enough for 3 months before you have to repeat that again11:35
cmorelliI'm following the ufficial gude here...
cmorelliin particular, the botom of the page says to just put `openstack_service_publicuri_proto: http` in the user config11:36
admin0csrf will come if horizon is accessed using a different ip then what you set it in the config11:36
admin0you can go to the haproxy container and check what ip works fine with11:37
cmorelliI will try to redeploy the playbooks with these two vars11:38
cmorelliopenstack_service_publicuri_proto: http11:38
cmorellihaproxy_ssl_self_signed_regen: true11:38
cmorellihopefully it will be enough to make it work11:38
*** stuartgr has joined #openstack-ansible11:38
*** armaan has quit IRC11:39
cmorelli@admin, this is what I have in the config global_overrides:11:39
cmorelli  internal_lb_vip_address:
cmorelli  external_lb_vip_address:
cmorelli  tunnel_bridge: "br-vxlan"11:39
cmorelli  management_bridge: "br-mgmt"11:39
cmorellietc ...11:40
*** armaan has joined #openstack-ansible11:40
cmorellithat ip address is the same that I try to access from the browser to login11:40
*** shardy has quit IRC11:42
*** bradm has joined #openstack-ansible11:44
*** armaan has quit IRC11:46
*** Jack_Iv has quit IRC11:47
*** udesale_ has quit IRC11:50
MiougeSupport for letsencrypt would be kind of cool, if not already possible.11:51
MiougeIt might be a little tricky because one has to share the certs & key as well as a state between for the HTTP-01 validation11:54
admin0evrardjp, that fixed it11:56
*** armaan has joined #openstack-ansible11:57
admin0my disks are slow, but the patch and solution was tldr for me .. so setting up the limit manually and sytsemctl restart /var/lib/machines solved for me11:57
admin0\o/ :D11:58
*** Sha000000 has quit IRC12:04
*** pcaruana has joined #openstack-ansible12:09
*** nattanon has quit IRC12:10
vedinevrardjp : now this new problem we are facing
admin0vedin, cannot ubuntu ?12:18
vedinno we have to stick with centos 7 :(12:19
*** gunix has joined #openstack-ansible12:23
gunixthis returns 40412:24
gunixis this the official link ?12:24
odyssey4mevedin did the repo build complete properly - can you confirm two things: 1. that haproxy is up and running with its configuration; 2. that the repo container has content in /var/www/repo/os-releases/<tag number>/12:24
*** aruns has joined #openstack-ansible12:25
odyssey4megunix it looks like evrardjp has been moving some stuff around, so that link's not working any more12:25
odyssey4methe pike one is still there:
gunixbe careful with that. google has a delay when upgrading12:26
odyssey4meand the process for an AIO has not changed since kilo, so you're safe to use it ;)12:26
*** dave-mccowan has joined #openstack-ansible12:27
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Install common packages into container cache
*** indistylo has quit IRC12:28
evrardjpgunix: where did you find that link ?12:29
gunixevrardjp: on google12:29
evrardjpyup, maybe we should do redirections12:30
vedinodyssey4me: haproxy is working fine and /var/www/repo/os-releases/<tag number>/ have the files, i found ./mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl file available in directory but  it is throwing error for this python package only12:30
cmorelliso, if I don't put `haproxy_ssl: false` in user config, I stumble in this error on the setup-openstack.yml playbook:12:31
cmorellifatal: [infra1_glance_container-57f5f85d]: FAILED! => {"attempts": 5, "changed": false, "failed": true, "module_stderr": "mesg: ttyname failed: Inappropriate ioctl for device\nTraceback (most recent call last):\n  File \"/tmp/ansible_TLcg0A/\", line 1459, in <module>\n    main()\n  File \"/tmp/ansible_TLcg0A/\", line 1453, in main\n12:31
cmorellikm.command_router()\n  File \"/tmp/ansible_TLcg0A/\", line 484, in command_router\n    facts = action(variables=action_command['variables'])\n  File \"/tmp/ansible_TLcg0A/\", line 1030, in ensure_service\n    self._authenticate()\n  File \"/tmp/ansible_TLcg0A/\", line 606, in _authenticate\n    self.keystone =12:31
cmorelliclient.Client(**client_args)\n  File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/v3/\", line 238, in __init__\n    self.authenticate()\n  File \"/usr/local/lib/python2.7/dist-packages/positional/\", line 101, in inner\n    return wrapped(*args, **kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/\", line 581, in authenticate\n12:31
cmorelliresp = self.get_raw_token_from_identity_service(**kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/v3/\", line 324, in get_raw_token_from_identity_service\n    _('Authorization failed: %s') % e)\nkeystoneauth1.exceptions.auth.AuthorizationFailure: Authorization failed: Unable to establish connection to\n", "module_stdout": "",12:31
cmorelli"msg": "MODULE FAILURE"}12:31
cmorellithis time I also used `haproxy_ssl_self_signed_regen: true` as admin0 suggested12:31
cmorelliI'm lost guys...12:32
evrardjpcmorelli: what are you trying to achieve?12:32
*** Guy has joined #openstack-ansible12:32
evrardjpyou want full http no https?12:32
evrardjpcmorelli: what is your openstack_user_config ?12:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Update documentation index to include Queens
evrardjpcmorelli: because you can have two different IPs12:33
cmorelli@evrardjp I'm just following the user guide , I wanted to do the test-environment setup (here: , but I stumbled in this error first. Then I googled a little bit and found that people suggested to set `haproxy_ssl: false` in the user config12:34
evrardjpmaybe the text here is more clear:
evrardjpit's latest branch text12:34
*** Jack_Iv has joined #openstack-ansible12:34
cmorelliso, if I do that I get to the WebUI working, but then I get that CSRF error a12:34
*** Jack_Iv has quit IRC12:34
evrardjpcmorelli: which branch?12:35
evrardjpyou want to do ocata only?12:35
cmorellii'm using tag 15.1.1512:35
evrardjpPike is the current latest released branch12:35
evrardjpwhy no Pike?12:35
evrardjphe deserves love too.12:35
evrardjpadmin0: do that for next version?12:36
admin0if it works flawless :)12:36
evrardjpcmorelli: anyway it should work in both cases.12:36
cmorelliI don't know, I would prefer to use not the bleeding edge version in general, it's nothing agains Pike release in particular :)12:36
evrardjpcmorelli: the bleeding edge is master12:37
admin0cmorelli, if for prod, i think pike sets a good foundtaion for cells12:37
cmorelliyes true12:37
evrardjpin two weeks or so we'll release queens12:37
evrardjpjust FYI :p12:37
cmorelliit's not prod... i'm just trying to setup openstack to demo a in-permises cloud in my workpace12:37
evrardjpcmorelli: oh.12:38
cmorellithat's why I don't really need SSL working immediately :)12:38
evrardjpcmorelli: start with an empty machine, and do scripts/gate-check-commit.sh12:38
evrardjpwell that's bad advice.12:38
vedinodyssey4me : what should i do now, i pasted what u asked12:38
cmorellibut I have three machines, i don't want to do AIO12:38
evrardjpif it's for very quick PoC that's what you want to show. Bleeding edge fast delivered!12:39
*** chyka has joined #openstack-ansible12:39
evrardjpor do it with more stable:12:39
cmorelliI already setup Vlans, networking, everything that is needed to run the playbooks12:39
evrardjpup to you12:39
evrardjpI'd say the easiest though is not to mess up with ssl and stuff at the beginning. Just do the standard procedure.12:40
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Update Queens doc index
evrardjpthat's for test in Pike cmorelli :
evrardjphaha you beat me to it12:40
cmorelliI understand that, but I also have to demo terraform/packer/ansible IaaS on top of Openstack, so the AIO solution could be too 'clogged', i think,.... am I wrong?12:41
cmorellimaybe if I paste my openstack_user_config.yml in a pastebin, could you guys rewiev it?12:41
odyssey4mecmorelli pike is a currently stable release, ocata is getting a bit old and crusty by now... queens is a release candidate right now... so if you want n-1 then pike is what you want, not ocata12:42
odyssey4mebut meh, ocata works too if you want12:42
*** chyka has quit IRC12:43
evrardjpodyssey4me: I will update the deploy_guide system to make sure it's pointing to queens deploy guide12:43
odyssey4mecmorelli and doing haproxy_ssl: no is not the answer for disabling the public endpoints - if you want to do that, you should only need to do
odyssey4meevrardjp My patches are a stop-gap to make sure our indexes expose the docs properly. Nothing more. You're welcome to make changes beyond that if you wish, but I think it's important that we expose the docs correctly ASAP.12:44
cmorellihere is my openstack_user_config :
evrardjpodyssey4me: ok12:45
*** sxc731 has quit IRC12:45
cmorelliok I wikk move to Pike then.... maybe my problem will be fixed12:45
cmorelliin any case I appreciate if you can review it :)12:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Update documentation index to include Queens
odyssey4mevedin looking through backscroll12:47
odyssey4mevedin ok, then if the container is unable to download the package, and haproxy and the repo look complete, then you'll have to troubleshoot connectivity issues from the source container to the LB, and from the LB to the repo12:48
odyssey4meit could be bad network config, bad MTU, bad routing... all sorts12:48
evrardjpodyssey4me: done12:51
*** zul has quit IRC12:51
cmorelli@odyssey4me: if I only put `openstack_service_publicuri_proto: http` in user variables without `haproxy_ssl: false`, then the playbook setup-openstack.yml does not finish correctly12:52
*** Chealion has quit IRC12:53
cmorellifew minutes ago I posted the problem I get : ...Authorization failed: Unable to │ chason12:53
cmorelli                         | establish connection to
odyssey4mecmorelli hmm, that's odd - if that is the case then there must be some sort of regression somewhere, because the endpoint using http does not require haproxy not to be using certs, they're independent of each other12:54
evrardjpI think it's the way we structured docs in ocata that's the problem12:54
*** Chealion has joined #openstack-ansible12:54
*** zul has joined #openstack-ansible12:55
evrardjpcmorelli: please use two different IPs for internal and external_lb_vip_address, and re-run your playbooks. You can then safely remove these overrides.12:55
evrardjpplease use two different IPs for internal and external_lb_vip_address, remove these overrides, and re-run your playbooks :)12:55
evrardjpand give the CA to your users. Problem solved.12:56
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Update Queens doc index
odyssey4meyeah, realistically you should implement two different IP's regardless12:56
cmorelli@evrardjp, sorry for my noobness: any suggestions on what ip to use as internal lb and external? Currently these three machines are running in a private lan12:56
cmorelli10.13.0.11-12-13 are my ips12:56
cmorelliand .11 is the 'infra1' machine12:56
evrardjpthat's their mgmt network?12:57
evrardjpbr-mgmt I mean12:57
cmorelliand i also have a vlan20 and vlan30 setup12:57
cmorellifor br-storage and br-vxlan12:57
evrardjpok you can have I guess for internal lb vip address (and put that into the reserved IPs)12:57
evrardjpif 10 is not taken12:57
cmorelliah so for virtualip I can use any IP which is not already physically taken?12:58
evrardjpand you can have for external lb vip address for example (and put that into the reserved ips)12:58
evrardjpcmorelli: you have to configure keepalived12:58
evrardjpyou must12:58
evrardjpelse you'd have conflicts12:58
cmorelligonna try immediately, thanks for the tip12:58
evrardjpplease check the prod guide on how to configure your user_variables.yml for keepalived12:59
evrardjpplease use a dns name for external lb vip address.12:59
evrardjpyou'll thank me later.12:59
evrardjpher emails?12:59
evrardjpcmorelli: ?13:05
*** aruns has quit IRC13:07
evrardjpsorry odyssey4me13:09
*** wagner has joined #openstack-ansible13:09
cmorellii am just confused about the DNS name that you suggested. idon't have any dns resolution setup , it is a private lan13:10
*** aruns has joined #openstack-ansible13:11
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Update Queens doc index
*** niraj_singh has joined #openstack-ansible13:13
*** electrofelix has quit IRC13:14
odyssey4mecmorelli using a DNS name makes it easier to use certs if you want to later, and also makes it easier to implement other things to make access to the system easier - for example if you use an IP, only that IP will work... but if you use a name, then even if you NAT/PAT access to the system it will work as long as you have a way to resolve the name back to the IP used for access internally/externally13:14
odyssey4meso, given that the systems themselves use the internal endpoints to talk to each other, keep that as an IP, but generally it's advised to use a DNS name for the public endpoints - changing them after the deployment is a bit of a pain13:15
*** hamzy_ is now known as hamzy13:16
*** JohnnyOSA has quit IRC13:17
cmorellifatal: [infra1_galera_container-b3750f45]: FAILED! => {"attempts": 5, "changed": false, "cmd": "/usr/local/bin/pip2 install -U --constraint  pyasn1 pyOpenSSL requests urllib3", "failed": true, "msg": "\n:stderr: Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection13:18
cmorellibroken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x7f1db9b42590>:13:18
*** JohnnyOSA has joined #openstack-ansible13:18
cmorelli10.13.0.100 is the internal lb vip13:18
cmorellithat is set13:18
*** berendt has quit IRC13:18
*** berendt has joined #openstack-ansible13:20
odyssey4mecmorelli trace the path of the data to find the failure: galera_container -> haproxy -> repo container13:21
odyssey4meif the packets are making their way from one to the next just fine, then validate whether the repo container has the content13:21
*** woodard_ has quit IRC13:29
openstackgerritMerged openstack/openstack-ansible stable/newton: Update all SHAs for 14.2.16
*** udesale has joined #openstack-ansible13:32
*** MikeW has joined #openstack-ansible13:36
*** acormier has joined #openstack-ansible13:39
openstackgerritMerged openstack/openstack-ansible-galera_server master: Fix cache update after initial apt_repository fail
*** acormier has quit IRC13:42
*** woodard has joined #openstack-ansible13:44
*** shardy has joined #openstack-ansible13:48
admin0phew, .. setup hosts finished finally  without a hiccup13:58
*** sxc731 has joined #openstack-ansible14:01
admin0lvdombrkr, so the ovs thing i pasted,  ironic still wants linuxbridge, so under those guidelines, for ironic, ironic is also linuxbridge14:03
*** lbragstad has joined #openstack-ansible14:10
admin0hmm.. evrardjp my new also ,keeplive is running in all, but no ip is being added/seen in any interface ..  to move on, i manually did a ifconfig in one of the controller and proceed agahed14:13
admin0will check later why its failing/not working14:13
admin0quick question .. does stable/pike allow live migrations by default ( non shared storage ) ?14:20
admin0or do i need certain overrides to do14:20
*** manuelbuil has quit IRC14:20
*** Sha000000 has joined #openstack-ansible14:21
*** Sha000000 has quit IRC14:23
*** Sha000000 has joined #openstack-ansible14:23
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ceph_client master: Fix cache update after initial apt_repository fail
*** sxc731 has quit IRC14:25
TahvokAnyone got promo code for ptg?14:25
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-openstack_hosts master: Fix cache update after initial apt_repository fail
odyssey4meTahvok do you mean the hotel code? it's on
*** jwitko_ has joined #openstack-ansible14:30
*** esberglu has joined #openstack-ansible14:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Fix cache update after initial apt_repository fail
Tahvokodyssey4me: the code if for the 155 euro price? or is there another discount?14:35
odyssey4meTahvok that's the only discount I'm aware of14:38
*** ansmith has joined #openstack-ansible14:38
admin0strange .. keepalive is running ..but tcpdump registeres no activity on vrrp14:40
*** Sha000000 has quit IRC14:48
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-rabbitmq_server master: Fix cache update after initial apt_repository fail
evrardjpadmin0: check the interfaces config and your network connectivity. But you already know that :D14:48
evrardjpalso check you are listening to everything14:48
evrardjp(multicast address)14:49
*** sxc731 has joined #openstack-ansible14:50
admin0i had to do tcpdump  -i any -n vrrp and then re-run the haprxy setup playbook 3 times ( coz i find nothing wrong ) so .. try try until die .. tryhard mode14:51
admin0and it worked :D14:51
*** kstev has joined #openstack-ansible14:52
*** epalper has quit IRC14:52
*** gkadam__ has quit IRC14:53
ansmithjmccrory: hello14:54
*** epalper has joined #openstack-ansible14:55
*** kstev has quit IRC14:56
*** kstev has joined #openstack-ansible14:56
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-pip_install stable/pike: Fix cache update after initial apt_repository fail
admin0 \o/  setup infra also passed ... hiccups on setup-hosts = that bug which was done using manual quota limit ( thank god i did not have to cherry pick that fix ) and went relatively smooth .. .. and then to increase the container build wait .. i had to double from 6000 to 1200015:15
admin0vip worked when i override the internal and external ids to 120  17215:15
odyssey4meadmin0 so the default value is too small?15:15
admin0in this specif case yes . i am on a 4 disk raid 10 setup .. but what i noticed was the extraction took a long time and by the time it countdown to 6000 retries, it was timing out15:17
admin0maybe my disks are bad .. will be reviewing them15:18
admin0quota bug, i was able to move forward by doing  machinectl limit-set 500G ; sytsemctl restart /var/lib/machines15:19
*** udesale has quit IRC15:19
admin0cherry picking that code was scary for me15:19
admin0i meant that patch15:19
admin0odyssey4me, i think it was on a  todo somehwere .. osa+ovs .. i have it up quick here:  --- that is how it works now that i have been able to make it work15:20
*** Guy has quit IRC15:22
admin0now to setup-openstack :)15:22
admin0setup infra also passed on good15:22
TahvokDo we know what time ptg will end on Friday?15:27
odyssey4meTahvok Friday is typically more of a social day. We chat for the morning, but most people start leaving after lunch.15:28
TahvokThen it's fine. Trying to arrange my flight, saw one at 20:50, and was not sure if I would make it15:29
*** acormier has joined #openstack-ansible15:32
*** mardim has quit IRC15:34
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-pip_install stable/pike: Fix cache update after initial apt_repository fail
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-galera_server master: tasks: Fix use_percona_upstream variable usage
hwoarangodyssey4me: evrardjp^ would you be able to get this in :) conflicts conficts conflicts15:37
hwoaranghello btw15:37
* hwoarang is super busy today15:37
*** aruns has quit IRC15:38
evrardjphwoarang: yup.15:38
evrardjpodyssey4me: could you vote on hwoarang 's patch too?15:40
*** Sha000000 has joined #openstack-ansible15:41
evrardjpTahvok: my flight is at 6pm on Friday , FYI.15:41
odyssey4meevrardjp busy working through it15:43
*** SerenaFeng has joined #openstack-ansible15:45
evrardjpodyssey4me: thanks.15:46
*** mardim has joined #openstack-ansible15:53
*** sxc731 has quit IRC15:57
openstackgerritgit-harry proposed openstack/openstack-ansible-galera_client stable/queens: Fix cache update after initial apt_repository fail
openstackgerritgit-harry proposed openstack/openstack-ansible-galera_client stable/pike: Fix cache update after initial apt_repository fail
*** mrch has joined #openstack-ansible15:59
*** lvdombrkr has quit IRC16:03
*** dariko has joined #openstack-ansible16:04
openstackgerritgit-harry proposed openstack/openstack-ansible-galera_client stable/ocata: Fix cache update after initial apt_repository fail
*** wagner has quit IRC16:11
*** mbuil has joined #openstack-ansible16:12
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-pip_install stable/pike: Fix cache update after initial apt_repository fail
openstackgerritgit-harry proposed openstack/openstack-ansible-galera_client stable/newton: Fix cache update after initial apt_repository fail
*** kstev has quit IRC16:16
MikeWevrardjp I linted my json file and promise to never touch it again lol :)16:18
openstackgerritgit-harry proposed openstack/openstack-ansible-galera_server stable/queens: Fix cache update after initial apt_repository fail
evrardjpMikeW: good :)16:20
evrardjpMikeW: what was the thing you wanted to do with it? Change ips?16:20
MikeWevrardjp Still doesn't work I think you were right in renaming being an issue. Yeah changing IPs16:20
mhaydenodyssey4me: on -- i replied about the http mirror16:26
openstackgerritgit-harry proposed openstack/openstack-ansible-galera_server stable/pike: Fix cache update after initial apt_repository fail
*** SerenaFeng has quit IRC16:30
*** yolanda has quit IRC16:30
*** weezS has joined #openstack-ansible16:30
*** yolanda has joined #openstack-ansible16:30
*** armaan has quit IRC16:33
*** armaan has joined #openstack-ansible16:33
*** sxc731 has joined #openstack-ansible16:35
openstackgerritgit-harry proposed openstack/openstack-ansible-galera_server stable/ocata: Fix cache update after initial apt_repository fail
*** fusmu has quit IRC16:37
*** chyka has joined #openstack-ansible16:48
*** Smeared_Beard has joined #openstack-ansible16:49
*** SmearedBeard has quit IRC16:50
mhaydenodyssey4me: many thanks good sir16:54
evrardjpmhayden: do you know why it's not caching with https?16:56
mhaydenit bypasses the cache16:56
mhaydenthe cache only works with http16:56
evrardjpwe might want to use gpg checking16:56
mhaydenyum has that enabled by default16:57
evrardjpwe disable it in the package install16:57
mhaydenand it's enabled by ansible-hardening too16:57
odyssey4meI think that's pretty standard. I'm not aware of a proxy cache that is able to cache HTTPS content. It can proxy it, but not cache it.16:57
mhaydenwell we could fix that16:57
evrardjp(apt key)16:57
evrardjpnot apt yum key16:57
mhaydenodyssey4me: yeah, it would require a trusted cert with a mitm :)16:57
odyssey4meI could be wrong. It has been a while.16:57
mhaydenevrardjp: if we aren't checking GPG keys on pkgs, i'd like to fix that too16:57
evrardjpyeah I am fine with that too16:57
mhaydenthe mariadb fix should help a lot since those packages are big16:58
evrardjpwell yeah but that takes a lot of time :D16:58
evrardjplet me double check16:58
evrardjpit seems I was wrong17:00
mhaydenGPG checking on packages should be really fast17:01
mhaydensince you're just verifying the sig17:01
mhaydenbut i saw a 70-80% reduction in time when i switched the mariadb repo to http17:01
mhaydenat least for yum17:01
mhaydenwhich is good17:01
mhaydeni think the server pkg is ~ 90MB by itdelf17:01
evrardjpthat's a big pkg17:02
evrardjpall of that for DATABASES!17:02
evrardjplet put our data into images.17:02
evrardjpwhere should I put "Securing services with SSL certificates": A new section under user guide, or reference?17:04
evrardjpI don't think putting it into reference/architecture/security.17:05
mhayden1 HR 53 M17:05
* mhayden REJOICES17:05
mhaydenit is a basekit (thanks andymccr) but it is exciting nonetheless17:05
evrardjp#success mhayden got centos OSA gate under 2h today17:05
openstackstatusevrardjp: Added success to Success page (
evrardjpyou're the deal!17:06
openstackgerritgit-harry proposed openstack/openstack-ansible-galera_server stable/ocata: Fix cache update after initial apt_repository fail
evrardjpwhere is the patch?17:06
mhaydenthat depends on andymccr's
evrardjpif that's a replacement of playbooks with debug: msg="i did it" it doesnt count17:06
odyssey4mewhat is a good word to describe achitectures other than x86_64/i38617:06
evrardjpnon intel17:07
mhaydenevrardjp: also the other galera patch is here ->
mhaydenodyssey4me: alternative architectures?17:07
mhaydensecondary architectures?17:07
odyssey4mealt_arch will do I guess17:07
evrardjpnon intel seem more likely17:07
odyssey4mewell, x86_64 covers AMD too ;)17:07
mhaydenwe could use something like adreznec_stuff17:07
evrardjpnot IA64, not IA3217:07
* mhayden winks at adreznec17:07
evrardjpthat's definitely not supported17:08
adreznecI'm sure Itanium and SPARC support is coming any day now, right mhayden?17:10
mhaydenalso, those new galera patches take 60-90 seconds on initial install without caching, and 20-30 sec after17:10
mhaydenadreznec: but of course17:10
* mhayden will be back shortly17:11
*** kstev has joined #openstack-ansible17:11
*** armaan has quit IRC17:12
*** armaan has joined #openstack-ansible17:13
evrardjpI guess I will move that to user guide17:13
openstackgerritJimmy McCrory proposed openstack/openstack-ansible-os_nova master: Rearrange cell mapping tasks
*** admin0 has quit IRC17:16
andymccrcells :/17:16
evrardjpa nightmare in dragonball, a nightmare in openstack!17:17
jmccroryheh yeah...finally moving to ocata. that was my face, andymccr17:18
andymccrjmccrory: i thought we had fixed up the ordering stuff17:18
logan-watch out for also17:19
openstackLaunchpad bug 1729661 in openstack-ansible "Map instances to new Cell1 takes excessive amounts of time to run on upgraded cloud" [Wishlist,Confirmed]17:19
andymccri literally sat with a nova core and fleshed out the deploy path, and then the upgrade path (which are different and really hard to automate)17:19
jmccrorypushing stateless hypervisors at the same time, so the discover were never actually running in our environments17:19
openstackgerritgit-harry proposed openstack/openstack-ansible-galera_server stable/newton: Fix Apt cache update due to adding Galera repo
jmccroryor instance mapping rather17:20
odyssey4melogan- I think that jmccrory's patch may actually help with that bug17:20
odyssey4mebut to be honest, I'm in the weeds of something else right now, so I'm probably not thinking straight17:20
andymccrocata was a nasty release for nova17:21
andymccrthat + the placement bits17:21
andymccrfrom a deployment perspective made for not-fun17:21
jmccroryyeah, should help there. this will make instance mapping and discover_hosts commands both only run one time, instead of once per compute17:21
*** gillesMo has quit IRC17:24
odyssey4mejmccrory perhaps it'd be good to add 'Related-Bug' or 'Closes-Bug' to the review then ;)17:25
openstackgerritJimmy McCrory proposed openstack/openstack-ansible-os_nova master: Rearrange cell mapping tasks
evrardjpodyssey4me: +117:27
evrardjpthanks jmccrory17:28
logan-jmccrory: great17:29
openstackgerritMerged openstack/openstack-ansible-repo_build stable/pike: SUSE: Fix MariaDB development package
openstackgerritMerged openstack/openstack-ansible-galera_client master: Allow Galera package downloads over HTTP
openstackgerritMerged openstack/openstack-ansible-galera_server master: tasks: Fix use_percona_upstream variable usage
ansmithjmccrory: hi, had a question regarding
jmccroryansmith sure, but haven't been able to get around to working on that unfortunately17:37
ansmithjmccrory: would like to help if possible, might it be discussed at ptg?17:37
jmccroryyeah definitely17:38
ansmithshould i propose in the etherpad or just plan to swing by mtg room17:39
jmccroryyeah i think it'd be good to have it on etherpad. not sure what the time slots look like already, but if not it's own dedicated topic, there should be some free time to go over that in the room17:42
*** mrch has quit IRC17:43
ansmithsounds like a plan, will add it to epad17:44
*** idlemind has quit IRC17:50
*** armaan has quit IRC17:51
*** armaan has joined #openstack-ansible17:52
openstackgerritMerged openstack/openstack-ansible-galera_server master: Allow Galera package downloads over HTTP
*** pbandark has quit IRC18:07
*** admin0 has joined #openstack-ansible18:10
*** mbuil has quit IRC18:16
*** d3n14l has joined #openstack-ansible18:20
*** sxc731 has quit IRC18:23
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_server stable/queens: Allow Galera package downloads over HTTP
openstackgerritMajor Hayden proposed openstack/openstack-ansible-galera_client stable/queens: Allow Galera package downloads over HTTP
admin0how does this error come: oslo_config.cfg.DefaultValueError: Error processing default value c3_cinder_api_container-57ae96e9 for Opt type of HostAddress.18:30
*** openstackgerrit has quit IRC18:33
d3n14lHey there, I am looking for a best practices guide for writing ansible roles. Is there some guide by the osa project?18:33
mhaydencloudnull: if you have a moment ->
*** openstackgerrit has joined #openstack-ansible18:36
openstackgerritMajor Hayden proposed openstack/openstack-ansible master: CentOS 7 integrated gate optimization
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server master: Restore support for percona packages when using ppc64le
odyssey4meadreznec not sure when/if this will touch you guys, but I've proposed - will add a reno to it shortly18:39
openstackgerritMerged openstack/openstack-ansible stable/ocata: Update all SHAs for 15.1.17
sarYesterday i got this error message in cinder-volume.log all the time when trying to migrate attached volumes: "Could not determine a suitable URL".18:41
sarToday i added os_privileged_user_auth_url, os_privileged_user_name and os_privileged_user_password in cinder.conf, and now i get a new message: "BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error."18:41
sarDoes anyone know where i can specify domain in cinder.conf, or know otherwise how i can fix this?18:42
openstackgerritMajor Hayden proposed openstack/openstack-ansible-repo_build master: [WIP] Remove unneeded clients from heat
*** epalper has quit IRC18:45
openstackgerritMajor Hayden proposed openstack/openstack-ansible master: [WIP] Test repo build w/trimmed heat requirements
openstackgerritMajor Hayden proposed openstack/openstack-ansible master: [WIP] Test repo build w/trimmed heat requirements
armaanfolks, have you released 15.1.17?18:50
shananiganssar: Not sure it will help, but it looks like most of this might be set under the keystone_authtoken section.
sarI already have that in the keystone_authtoken section. I was looking at this:
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server master: Restore support for percona packages when using ppc64le
sarWhere it describes os_privileged_user_name like this: OpenStack privileged account username. Used for requests to other services (such as Nova) that require an account with special rights.18:55
sar(under the DEFAULT section)18:55
*** poopcat has joined #openstack-ansible18:55
sarThe problem is, when cinder is trying to migrate an attached volume, it calls the nova api18:55
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server master: Restore support for percona packages when using ppc64le
*** d3n14l has left #openstack-ansible19:00
*** stuartgr has quit IRC19:01
*** Sha000000 has quit IRC19:01
odyssey4mearmaan nope, that's the next proposed release - the sha bumps only just merged:
mnaserdebhelper : Depends: dh-strip-nondeterminism (>= 0.028~) but it is not going to be installed19:04
mnaseranyone else run into this?19:04
armaanodyssey4me: ahh, that makes sense. we were just going to upgrade Ocata to Pike. Using 15.1.6 for now19:05
odyssey4memnaser yeah, I just saw it in the panko role test19:05
mnaserodyssey4me: looks like master went through though?19:05
mnaserbut this is for the stable branches19:05
mnaserand infra said the ubuntu mirrors were fixed last night19:05
odyssey4meI'm not sure why that package is actually even needed, but that sort of failure happens when the packages on the host are newer than those available in the apt sources configured... or there's a conflict.19:05
mnaserlooks like stable/pike passsed19:06
odyssey4memnaser master failed too - same issue:
mnaserand andreas did the rechecks at the same time19:06
mnaseri wonder if maybe one provider got older nodepool images or something19:07
odyssey4memnaser FYI the original content was pike, so anything after that has the same code base and has not been adjusted other than to change the branches...19:07
mnaserodyssey4me: yeah i was jsut going to ask if there was any differences but maybe this failure is an os_keystone one19:07
odyssey4mebut I did test master on a cloud image of my own yesterday, and it passed functional tests... so this is new19:07
mnasercause that's where it is failing19:07
odyssey4meah? that's new then19:08
mnaserits failing during keystone distro pkg installs19:08
*** sxc731 has joined #openstack-ansible19:08
mnaseryep, its not pankos fault19:08
odyssey4meif it's in a container where the fail happens, then bear in mind that our container image is downloaded from, so it might have newer packages in it than those in the infra mirror19:08
mnaserodyssey4me: recent os_nova change, failed with the same issue19:08
odyssey4mesounds to me like this might be the case19:08
odyssey4mewe can wait it out, or ask infra to update the mirror again19:09
odyssey4methis is very unusual, because the base image package list is very small19:09
odyssey4mebut with gcc releasing yesterday, and whatever this is today, I guess it happens sometimes19:09
mnaserlatest ubuntu xenial image19:09
mnaserso that can explain it19:09
odyssey4meI've been meaning to convert our base image prep into d-i-b to replace what we have in lxc_hosts today.19:10
odyssey4meIt just hasn't been that much of a priority.19:10
mnaserodyssey4me: i'll pretty much be at all the osa ptg stuff and i can bring up some stuff that we learned to stabilize ci in puppet19:10
mnaserex: it looks like there is no usage of mirrors right now?  and is not mirrored either so that can't be good19:11
odyssey4mecool bananas - I'm out for the night, cheerio!19:11
mnasero/ later19:11
mnaseri'll try to ask infra to force an update19:11
admin0anyone knows how to fix this error. or what might I be missing:  "oslo_config.cfg.DefaultValueError: Error processing default value c3_cinder_api_container-57ae96e9 for Opt type of HostAddress"19:14
admin0hmm.. nova works, neutron got the same error: oslo_config.cfg.DefaultValueError: Error processing default value c3_neutron_server_container-4e8ae1a8 for Opt type of HostAddress19:32
admin0what am I missing/doing wrong ?19:32
openstackgerritMerged openstack/openstack-ansible-os_panko stable/pike: Zuul: Remove project name
admin0hostname, hostname -f returns fine19:33
logan-mnaser re: the linuxcontainers stuff above and in infra... it is downloaded thru a reverse proxy (see
logan-i think that setting was implemented in both osa and role test gates19:34
admin0logan-, not my error right ?19:35
admin0aah . was for mnaser19:36
logan-admin0: maybe the underscores are breaking it. does 'hostname' show '_' or '-'19:36
admin0infra nodes = c1 c2 and c3,  compute = b4 .. b1019:37
logan-inside the container though19:37
logan-the container hostnames should have '-'19:37
admin0it generated all on _19:40
admin0my inventory list output:
admin0i am using stable/pike19:41
logan-the container name is not the same as the hostname of the container19:41
logan-the container name is expected to contain '_'19:41
logan-the hostname is not19:41
admin0err.. its auto generated by the script19:41
admin0sorry .. me confused19:42
logan-lxc-attach -n `lxc-ls -1 | grep cinder_api` -- hostname19:42
logan-what is the output19:42
admin0root@c3:~# lxc-attach -n `lxc-ls -1 | grep cinder_api` -- hostname19:42
admin0oops .. sorry for multi-line19:42
admin0returned: c3_cinder_api_container-57ae96e919:43
logan-as a quick test try changing the /etc/hostname and /etc/hosts in the container to set -19:43
logan-and see if cinder still complains19:43
logan-(reboot the container after changing the hostname files)19:43
admin0before i do this test.. assuming this change fixes it, what is the solution ? i have to redo the whole patform again ?19:44
admin0and this is a greenfield deployment .. why would the setup-hosts generate _ when we know it breaks in setup-openstack ?19:45
MikeWMy repo build isn't pulling down the magnum checksum and virtualenv... can I do this manually?19:45
logan-admin0: i'm not sure if there is a patch proposed yet. I just recall seeing someone else with this issue from bug triage. looking for the bug report19:47
openstackLaunchpad bug 1743805 in openstack-ansible "neutron-db-manage fails on hostname with underscore" [Undecided,Incomplete]19:47
mnaserlogan-: i see the bootstrap_host_ubuntu_repo but i dont see anything that sets it19:49
logan-user_variables_aio.yml template i think19:49
mnaserlogan-: ok ill double check but indeed its here
mnaserlogan-: but still can't find bootstrap_host_ubuntu_repo (at least with github search which isn't always reliable)19:52
logan-ohh gotcha19:52
logan-github search is the worst19:53
logan-looks like it tries to figure it out here
*** weezS_ has joined #openstack-ansible20:03
odyssey4meo/ logan- mnaser trying to figure out how repositories are configured? I can help20:10
mnaserodyssey4me: trying to figure out why this error is happening20:10
mnaserinfra recently started mirorring bionic20:10
mnasercould it be related?20:10
* mnaser shrugs20:10
odyssey4melooks like went through, which is odd20:11
odyssey4meI wonder if some regions are out of date, and some are up to date20:11
mnaserodyssey4me: thats not possible because of how afs works (afaik)20:11
odyssey4merole tests and integrated build tests get set differently, so let's focus on one of them20:11
odyssey4memnaser yep, that's why I think it's odd20:11
odyssey4meok, so the basic overview of what happens is this20:12
mnasernothing bionic related here for the container20:12
odyssey4melxc_hosts preps an image on the host which forms the basis of all containers20:12
odyssey4methe base cache is downloaded from - inside infra it's downloaded through a reverse proxy20:13
odyssey4methe download is initiated here:
odyssey4meand finalised here:
odyssey4meit's done via an async task because it takes some time, so we let the host do some other work while that's happening20:15
odyssey4meinside nodepool/openstack-ci, the right mirror var is set in this task:
odyssey4meso if we look in at the failure - which is a role test20:16
odyssey4mewe can see when that var is set for the test by looking for 'Discover the lxc_image_cache_server value when in nodepool' in the ARA report in
odyssey4mewe can see it did the right thing in the async result:
odyssey4memnaser with me so far?20:18
mnaserodyssey4me: yeah, also some interesting discussion from #openstack-infra too20:18
odyssey4meit looks like the host prep worked just fine:
mnasercombining what you're saying with what they're saying, we're ending up with either bionic images with xenial repos OR xenial images with bionic repos20:19
odyssey4mewhat that means is that the base image was downloaded, and the prep was done using infra mirrors without a hitch - so that means the problem happens later20:19
admin0odyssey4me, this bug affects all new installs .20:20
openstackLaunchpad bug 1743805 in openstack-ansible "neutron-db-manage fails on hostname with underscore" [Undecided,Incomplete]20:20
odyssey4mea key thing to understand here is that all containers do not use the infra images - only the host does20:20
admin0i am doing a new greenfield install today ..  cinder and neutron fails .. cinder i can live without for a few days, neutron cannot20:20
odyssey4mealso, the container apt sources are pristine - we only copy /etc/apt/sources.list from the host - everything else is laid down by our ansible tasks20:21
mnaserodyssey4me: so download image, 'prep' it locally, build containers from it, right?20:21
odyssey4memnaser yep20:21
mnaseris it possible that we're downloading a bionic image and prepping it with xenial repos20:21
odyssey4mewe would like to change the download part to something prepped by diskimage-builder so that we have *full* control of it, but it's not work anyone's picked up yet20:21
mnaserthat gets a list of all images20:22
mnaserand grabs the latest one20:22
mnaserwhich might happen to be bionic20:22
odyssey4meok, so what is a 'bionic' image?20:22
mnaserodyssey4me: latest release of ubuntu20:22
odyssey4meoh, that'd be weird20:23
mnaserthat codebase doesnt seem to filter20:23
mnaserit seeems to grab this file
mnaserthen matches against cache_index_item20:23
odyssey4mebased on it looks like it's using xenial sources - and that file came from the host20:24
odyssey4memnaser would it be helpful to continue the discussion in #openstack-infra?20:24
mnaserodyssey4me: nono, i'm saying the lxc image we download is bionic (18.04)20:24
openstackgerritMajor Hayden proposed openstack/openstack-ansible master: [WIP] Install Python 3.5 on CentOS 7
mnaserodyssey4me: for now, i think this is an OSA issue20:24
mnaserone tiny thing i need to check20:25
mnaserwhat "Set image index fact" is doing20:25
odyssey4melooks to me like it's right?
odyssey4mealso see the actual image downloaded's path:
odyssey4meso, if it is bionic, then something has gone wrong upstream20:26
mnaser{{ lxc_images[0].split(';')[-1] }}20:26
mnaserok let me download this real quick to check20:26
odyssey4meok - let's take a step back for a bit - do that, that might be useful20:27
odyssey4melemme take a peek at the actual fail and look at it from that end20:27
mnaserodyssey4me: interestingly enough xenial has the same exact update timestamp as bionic20:28
odyssey4medebhelper happens to be in xenial-backports:
odyssey4memnaser ah yes, but the CI that builds those images runs regularly, and all the time20:29
mnaserodyssey4me: but that release has no pinned dh-strip-nondeterminism dependency20:29
mnaserodyssey4me: where as look at .. exact dependency it's trying to pull20:29
odyssey4meit does dep on it:
openstackgerritMerged openstack/openstack-ansible-tests master: Use ARA instead of profile_tasks callback
mnaserodyssey4me: but it's not a pinned dependency.  if you notice, the issue is that it's trying to install >= 0.028~ specficially in the error20:32
odyssey4meyeah, that is odd20:32
admin0is there a way to force creation of containers with - instead of _ ?20:32
mnaserso with the xenial-backports or xenial package, it'll install any release happily.  but if it's trying to install debhelper from bionic, then it absolutely needs >= 0.028~ .. which doesn't exist in xenial20:33
mnaseradmin0: i think you're using an old release of osa20:33
mnaserall new ones use dashes20:33
odyssey4meadmin0 the inventory uses _, but lxc-container-create should translate that to -20:34
odyssey4mefor the hostnames and dns entries, I mean20:34
admin0mnaser, checked out like 12 hours ago20:34
mnaseradmin0: not sure, my deployment def has dashes20:34
mnaserid double check configs/vars/etc20:34
admin0mnaser, this is what i got
odyssey4methat's been the case since mitaka... but you're not the first to report a problem, so maybe something new has crept into LXC which is breaking the mechanism20:34
mnaserodyssey4me: thats not the same thing20:34
odyssey4menot sure if cloudnull is around, 'cos he was looking into it20:35
mnaserread what odyssey4me just mentioned re: inventory and hostname20:35
mnaserlxc-attach -n c1_neutron_server_container-efcf9281 on c120:35
mnaserand type hostname20:35
mnaserodyssey4me: maybe we can talk a bit with infra some more, i'm a bit at a loss20:35
odyssey4meyep, if you do ^ then you should see '-' instead20:35
admin0returns: c1_neutron_server_container-efcf928120:35
odyssey4memnaser okie dokey20:36
admin0i do not see -20:36
mnaserodyssey4me: is queens supported on xenial by uca?20:40
admin0mnaser, whichi file is responsible for this ... maybe i can compare/check this20:42
mnaseradmin0: no idea, it just works for me :X20:42
mnaserid retry the deployment i think something went wrong20:42
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-nspawn_container_create master: Updated from OpenStack Ansible Tests
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-nspawn_hosts master: Updated from OpenStack Ansible Tests
odyssey4memnaser yup:
openstackgerritMerged openstack/openstack-ansible stable/queens: Update Queens doc index
openstackgerritMerged openstack/openstack-ansible master: Update documentation index to include Queens
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_panko master: Updated from OpenStack Ansible Tests
mnaserodyssey4me: now to raise the question20:46
mnaserwhy do we even install debhelper20:46
mhaydenit's a helper20:47
openstackgerritMerged openstack/openstack-ansible-galera_client stable/ocata: Fix cache update after initial apt_repository fail
mhaydenfor debs20:47
openstackgerritMerged openstack/openstack-ansible-galera_client stable/newton: Fix cache update after initial apt_repository fail
odyssey4methat is a very good question20:47
openstackgerritMerged openstack/openstack-ansible-galera_client stable/queens: Fix cache update after initial apt_repository fail
mnasermhayden with the 🔥 answers20:47
mhaydeni doubt we need it unless we're building debs or converting a python pkg to deb20:47
mnasergit blame time20:47
mhaydenOH EMOJI20:47
mnaserhey cloudnull -- wanna try to remember a decision you took 2 years ago? :P20:48
admin0magic of git blame :)20:48
mnaserim back to 3 years and still see debhelper20:49
mnaseractually, it looks like this has existed since the start of os_keystone so before things got split20:49
odyssey4meit might just be leftover cruft from days of yore20:50
openstackgerritMerged openstack/openstack-ansible-galera_client stable/pike: Fix cache update after initial apt_repository fail
odyssey4memnaser mhayden yep, I found it in our icehouse keystone dep list20:51
mnaserthats waaay back20:52
odyssey4meyep, no mention of it before that20:53
odyssey4melemme check something else, to see if we can go back even further ;)20:53
mnaserdont think i can go back beyond that :p20:53
mnaserbut while you do that i'll propose a patch to drop it for now20:53
odyssey4meyep, looks to me like it's old cruft anyway20:54
odyssey4meit's curious to me that there's an issue though20:54
odyssey4meit's plausible that UCA was added, but the indexes weren't updated after that20:54
mnaserdoes OSA have someone to reach out for UCA issues like this20:55
odyssey4meI could see if jamespage is around.20:55
mnaserdo we need dh-apparmor too?20:55
odyssey4meHeh, he's in channel - so if he's available, he'll pop up.20:55
mnaserdoes mhayden respond to selinux only or apparmor questions work too :P20:55
odyssey4meI doubt it, if it's a related package.20:55
* mhayden has little apparmor experience20:56
mnaser"dh-apparmor provides the debhelper tools used to install and migrate AppArmor profiles. This is normally used from package maintainer scripts during install and removal."20:56
odyssey4memhayden put selinux in permissive mode... I'm not sure if we can trust him any more20:56
odyssey4memnaser sounds lie we can remove it20:56
admin0"<odyssey4me> mhayden put selinux in permissive mode... I'm not sure if we can trust him any more"  :D20:56
odyssey4memnaser what I'm suggesting is that a possible cause could be the issue which is actually trying to solve20:58
mnaserodyssey4me: i mean that change did pass20:58
odyssey4methe UCA repo is configured, but the cache does not get updated - then the install tries to install and finds a dep it can't resolve20:58
openstackgerritMerged openstack/openstack-ansible-plugins master: Make connection plugin compatible with Ansible 2.5
openstackgerritMerged openstack/openstack-ansible-galera_server stable/newton: Fix Apt cache update due to adding Galera repo
mnaserbut it didn't try to install the os_keystone stuff20:58
odyssey4methat said, how does it know about the new dep20:59
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_keystone master: Drop unnecessary dependencies from role
odyssey4mewe can try a test patch to see if we get a pass20:59
mhaydenodyssey4me: TEMPORARILY20:59
odyssey4melemme push a test patch with a depends-on20:59
mhaydenodyssey4me: ಠ_ಠ20:59
odyssey4memhayden sure, sure... I bet you say that to all the auditors20:59
admin0i am going to destroy the containers, pull the 16.0.9 i see now and hope the new containers will be using - and not _21:00
openstackgerritMohammed Naser proposed openstack/openstack-ansible-repo_build master: Drop unnecessary dependencies from role
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: [TEST] Try using updated openstack_hosts
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build master: Drop unnecessary dependencies from role
odyssey4memnaser just added the depends-on - that repo build change should not merge unless the other does... although that said, the repo build patch data is not used, so meh21:04
odyssey4meit's just better to link them :)21:04
mnaserodyssey4me: no worries, i just used codesearch to find all debhelper references21:04
odyssey4meyep, codesearch is *way* better than github search :) cc logan-21:05
logan-oh neat. never even heard of codesearch21:05
odyssey4meI'm no expert, but it's far more useful than github search as it searches substrings.21:06
odyssey4meI only wish you could specify the branch to search too.21:06
odyssey4meas far as I've seen - it does master only21:07
odyssey4me(same as github)21:07
logan-i gave up on github search long ago, always just clone+grep. will have to give codesearch a try :)21:07
mhaydenripgrep is quite fast too21:14
mhaydenrust implementation of grep21:15
odyssey4meneat, but still requires local clones21:15
mhaydeni always have updated local clones thanks to gertty21:16
odyssey4meof course I know mhayden has all the OSA repositories cloned locally - every. single. one.21:16
odyssey4meespecially those news ones we imported yesterday when we broke zuul again21:16
mhaydenyou betcha21:16
mhaydenhot damn21:16
mhaydenwait, you guys were breaking stuff and you didn't invite me?21:16
mhaydenthat's, like, my specialty21:17
mhaydenol logan- should be thankful nothing's broken in his datacenter yet21:17
odyssey4meyes, the breakdown was spectacular - but lacked the pazezz it has when you're around21:17
* mhayden will try harder21:17
odyssey4meso, welcome to our new repositories:21:17
* mhayden is hungry for breadcrumbs21:18
* odyssey4me gives up trying to spell pazazz (whatever) and switches to the word 'flare' instead21:18
odyssey4meheh, although that's probably the wrong word - 'flair' ?21:19
* odyssey4me gives up on words21:19
odyssey4mespeaking of which, we should probably add some functional tests to those repositories :p21:19
mhaydenapparently pizzazz doesn't directly translate into afrikaans21:20
mhayden... the more you know ...21:20
odyssey4memhayden flair doesn't either:
odyssey4melogan- got a minute to look through ?21:23
logan-on it21:23
mhaydenodyssey4me: dang, how do people in south africa enjoy office space?!21:25
jamespageodyssey4me: ok here for a bit21:27
odyssey4mejamespage thanks :)21:27
odyssey4meso some time this afternoon we started getting fails from our keystone role, which tries to install debhlper21:27
jamespagelemme check what time I promoted everything21:27
odyssey4methe error was that it couldn't find its dependency, which was pinned21:28
openstackgerritMerged openstack/openstack-ansible-galera_server stable/ocata: Fix cache update after initial apt_repository fail
openstackgerritMerged openstack/openstack-ansible-galera_server stable/pike: Fix cache update after initial apt_repository fail
openstackgerritMerged openstack/openstack-ansible-galera_server stable/queens: Fix cache update after initial apt_repository fail
jamespageodyssey4me: 15:0321:28
odyssey4meit took a bit of spelunking, but we found that pinned requirement in UCA21:28
odyssey4meok, let's see if we can find the first failures in logstash21:29
odyssey4mehere's an example:
jamespagethat included fresh backports of debhelper + associated misc tooling bumps needed to support that including pkgbinarymangler, strip-nondeterminism and cmake (yah I know)21:29
jamespageodyssey4me: have you managed to unstick that held package issue?21:30
*** deadnull has quit IRC21:30
jamespageI just did  quick check on fresh xenial with queens-updates and its installable  afaict21:30
jamespagewell it must be otherwise nothing would have built for the last three weeks :-)21:31
odyssey4methis appears to be the earliest failure in logstash:
odyssey4meit was using the mirror:
odyssey4meall the way:
odyssey4meso it's a bit odd21:33
odyssey4meI guess it's plausible that there was an infra mirror update process running at around the same time21:34
jamespageodyssey4me: lots of moving parts21:34
odyssey4meand perhaps the update you did wasn't done yet21:34
odyssey4meso when the mirror happened, the source mirror was incomplete21:34
jamespagebinary package copies between -proposed and -updates PPA's in launchpad21:34
jamespagesync of PPA -> UCA (via reprepro)21:34
*** acormier has quit IRC21:36
jamespageand then sync from UCA to infra mirrors as well21:36
*** acormier has joined #openstack-ansible21:37
*** sxc731 has quit IRC21:37
*** pcaruana has quit IRC21:37
odyssey4meyeah, none of which are event based - especially the last part21:38
jamespageodyssey4me: urgh so21:38
jamespageodyssey4me: debehlper when in at 15:0621:38
jamespagestrip-nondt at 15:3921:38
jamespagethat's either side of a sync from the source PPA's on LP to the actual UCA21:38
odyssey4meheh, aha21:39
jamespageso from 15:30 until 16:30 there would have been an installability issue21:39
odyssey4meok, so these are all scheduled, not event driven21:39
odyssey4meand we happened to find a perfect storm - I guess partly due to the volume of updates21:39
jamespageunfortunately that does appear to be the case here21:40
evrardjpgood morning everyone21:40
*** acormier has quit IRC21:41
odyssey4mejamespage so this is when the last infra update was:
jamespageshould be OK now then21:41
evrardjpwhat's the issue?21:42
odyssey4mejamespage aha, that would then explain why managed to get past the issue, but which ran before that time did not21:43
odyssey4merechecked some more patches now to see what happens21:43
odyssey4mejamespage ok, curiosity satisfied - mnaser does that all make more sense now?21:44
*** ansmith has quit IRC21:44
odyssey4methanks jamespage :)21:44
jamespageyou're welcome - I'll have a think about how we can make these type of promotion a bit more transactional to keep things consistent21:45
mnaserjamespage, odyssey4me: thanks for the info, what an odd set of coincidences21:45
odyssey4memnaser rather :) it happens21:46
odyssey4meit would seem that it was an issue for not too long - 90 mins at most, although that's my speculative guess and I'm too lazy to dig up a more accurate guess :p21:47
*** dave-mcc_ has joined #openstack-ansible21:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Centralize Inventory documentation
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Move limited connectivity to user guide
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Migrate security into user guide
cloudnullmnaser: whats up ?21:47
odyssey4meevrardjp the symptom was
mnasercloudnull: we're all good now but it looks like in a history far far away debhelper was added as a os_keystone dependency21:48
mnaserand with some combination of issues, we couldn't install it21:48
odyssey4methe root cause turned out to be a few things happening at the wrong place at the wrong time21:49
*** dave-mccowan has quit IRC21:49
cloudnullok, so nothing to see here ?21:50
odyssey4mejamespage pushed a bunch of updates to the PPA for UCA, which then somewhere while it wasn't done got pulled into UCA, which then also got pulled in by infra's mirror process21:50
odyssey4meso packages that UCA wanted weren't there21:50
cloudnulladmin0: odyssey4me: are we seeing hostnames with _ instead of - ?21:50
odyssey4methe syncs are all done now, and everythihg is back to working21:50
evrardjpcloudnull: :)21:50
admin0i spent 10 hours to fail at cinder and neutron in the end :(21:50
admin0just now deleted all the containers and pulled tag 16.0.9 hoping it might fix21:51
evrardjpodyssey4me: thanks for taking care of it21:51
evrardjpand thanks jamespage and infra :)21:51
admin0so confident with OSA that i directly used it on  production :D so took this long time to find out21:51
* admin0 feels cheated21:52
odyssey4meadmin0 was this a fresh install, or an upgrade from a previous version?21:52
admin0super fresh21:52
cloudnulladmin0: if it happens again I'd love to work though the issue and see what falls out21:53
cloudnullwe've had two different reports of this in the last couple of weeks but on different releases but no means to reproduce it21:53
admin0i nuked everything , pulled 16.0.9 , even deleted the /etc/ansible from deploy and redoing the setup ansible part21:53
cloudnullso I'm a little bit at a loss on what went wrong21:53
admin0since this is new install, i can add your keys for you to poke around if required21:54
odyssey4mecloudnull what's useful in this case is that admin0 is quite meticulous in documenting all his config ;)21:54
admin0i also faced the machinectl bug and your patche scared me to cherry pick .. so i had to manually do  machinectl limt to 500g, systemctl restart /var/lib/machines and then i do not have to do the patches or do cherry pick21:55
admin0that 16GB limit reached bug21:55
cloudnulllet me know what you find out, I'm happy to go dig-in21:55
admin0cloudnull,  unrelated to the errors, this is my hybrid setup .. osa with ovs -
cloudnullthat's awesome!21:56
admin0i have one running fine on okata .. so this new greenfield is based on the same hybrid setup to use osa but on pike21:56
admin0here in the new one, we plan to run a platform that uses nfv .. so i will be able to document those as well21:57
cloudnulladmin0: might want to update the user variables for pike so that you can use firewall_v221:58
cloudnullother than that the configs look great!21:59
admin0this setup is going to be big ..  i have 6 machines now .. but i want to migrate workload from an old cluster to here which has like 60 machines .. and we have one tenant that needs dedicated hardware where heat will create like 1000 machines every 3hours, and then destroy .. again 1000.. agin destroy21:59
cloudnullalso nice touch w/ jumbo frames22:00
admin0so if you have a good example on using cells, i will incorporate from start22:00
admin0cloudnull, i experiment with OSA a lot :D22:00
cloudnullI've not run cells in any meaningful way and have done VERY little with cells v222:00
odyssey4meadmin0 doing a cells implementation would be a small stretch from an ansible standpoint, but a bit impractical from an inventory standpoint right now22:01
odyssey4meif we start working on multi-cell implementations, I personally would rather see that we work out how to scale the inventory better22:01
odyssey4mefor now we recommend having multiple regions instead, each with their own inventory22:02
cloudnullthat said, 1000 vm workload in a single cloud of <500 compute hosts should be fine with the general configs22:02
mnaserfyi at 60 machines you're going to have more issues dealing with cells v222:02
mnaserhow do i rephrase this22:02
mnasermultiple cells at cells v2 scale is too much hassle22:02
mnaseri think host aggregates with scheduler configuration would be much easier for you22:03
cloudnull^ that's what I've done more with22:03
mnaseryou'll obviously need a single cell because cells v2 has an api and 'default' cell but yeah22:03
admin01000 vm is just for training guys who destroy and bring it up every 4 hours .22:03
odyssey4meyup, using host aggregates/availability zones for localised scale  is far more useful22:03
admin0for their case, i disable ceilometer agents .22:04
admin0so firewall needs to change to firewall_v2 .. done22:05
admin0day gone .. here goes my night :D22:05
cloudnullI recently had to dig that info up22:06
cloudnullwhich came from the osic cloud at 252 compute hosts, 3 infra nodes.22:07
cloudnullwhich should give you an idea of what a single cell should be capable of22:08
odyssey4me(with awesome hardware) ;)22:09
cloudnull^ this is true22:09
admin0so this hostname thing .. how do I validate if 16.0.9  does it correctly .. it should be visible after lxc-hosts-setup.yml right ?22:09
cloudnullafter lxc container create22:09
admin0i have to login and do hostname -f22:10
admin0running !22:10
cloudnullping me if you figure anything out.22:10
cloudnullor generally see the issue22:10
*** armaan has quit IRC22:13
odyssey4meadmin0 if you just do setup-hosts.yml, then check, that will do22:14
admin0its running22:14
odyssey4mecloudnull could you push up some patches to get the functional tests back into the nspawn roles?22:15
odyssey4mewe broke infra with what was in the seeded repositories last night, so they got force-removed from the repo22:15
odyssey4meif you could push up the configs to get tests back in, that'd be awesome22:15
spotzodyssey4me: OSA broke infra?22:15
odyssey4meif you don't manage, then I'll likely pick that up in the morning22:16
odyssey4mespotz it was a team effort ;)22:16
odyssey4mecloudnull wrote the code, I inspected it and got it imported - boom... we discovered that infra had no tests on importing repositories which checked whether the zuul stuff worked, and when the repo imported, zuul broke22:17
cloudnulloh ?22:17
admin0does the /etc/hosts also get cleaned of old stuff in cases like mine when i have to delete the old hostname_ip mapping to start fresh22:17
admin0right now, been clearing out that by hand22:17
cloudnullodyssey4me: that's funny :)22:18
cloudnullso what do i need to do, maybe just push up the roles without the zuul things?22:18
odyssey4mecloudnull the roles are imported22:19
cloudnulloh ok22:19
admin0ok: [c3_nova_api_os_compute_container-92064114 ->]22:19
admin0 - not a good sign right ?22:19
odyssey4mecloudnull so we just need the functional test config (ie the zuul.d directory and content) pushed up in a review22:20
admin0new format should be c3-nova-api-os-compute-container-xxx ?22:20
cloudnullill get on that22:20
cloudnulladmin0: yes,22:20
cloudnullcan you cat /etc/hosts22:21
odyssey4mecloudnull thanks22:21
admin0i will checkout only 16.0.9 ( not master and then checkout ) and give it 1 more try22:21
odyssey4mewhat you had in the source broken zuul badly :) that said, if you're now pushing it up for review, zuul will tell you that it's no good and you can figure it out until it's right22:21
odyssey4meI do suspect that part of the issue was that the roles were pulled in together, not one after the other - and the jobs depended on each other22:22
odyssey4meanyway, I'll leave it to you22:22
cloudnullodyssey4me: interesting that it accepted it given that it was busted.22:22
odyssey4metime for me to go back to !computering22:22
cloudnullodyssey4me: take care.22:22
cloudnulladmin0: both the name with _ and name with - should work22:23
odyssey4mecloudnull the fault was that it imported the repo, without verifying that the repo contents would not break it... so that's fixed now in a new test :)22:23
odyssey4menew system, new lessons ;)22:23
cloudnullodyssey4me: so we're all welcome :)22:23
admin0cloudnull, is therea  way to just force/change all hostname generation with -22:23
cloudnullthe hostname should be with a -22:23
admin0so that i will know it will just work for cinder/neutron on setup-openstack playbook22:24
cloudnullhowever an alias will exist with an _22:24
cloudnullodyssey4me: exactly22:24
admin0is there a way i can git rid of _ and use only -?22:24
cloudnulladmin0: so /etc/hostname should be only with a -22:24
cloudnullhowever a line in /etc/hosts will contain both22:24
cloudnulli don't think we have any way to disable the alias, but we could make one.22:25
admin0if we all see _ gives an issue, maybe make a variable so that for new installs, only use -22:25
cloudnulli'd be happy to ssh in and poke around. see if something is off22:25
cloudnullanything with an _ should just be an alias, which shouldn't impact hostname resoution22:26
cloudnullbut if it is, then there's a bug we need to fix22:26
odyssey4meadmin0 we have a feature request for that already:
openstackLaunchpad bug 1643680 in openstack-ansible "Shift to using dashes instead of underscores for container names" [Wishlist,Confirmed]22:26
odyssey4methe answer for now is, not yet22:27
odyssey4meanywa, night night!22:27
cloudnullsee you tomorrow22:27
admin0see ya !22:27
openstackgerritMerged openstack/openstack-ansible-openstack_hosts master: Fix cache update after initial apt_repository fail
openstackgerritMerged openstack/openstack-ansible-nspawn_container_create master: Updated from OpenStack Ansible Tests
openstackgerritMerged openstack/openstack-ansible-ceph_client master: Fix cache update after initial apt_repository fail
openstackgerritMerged openstack/openstack-ansible-pip_install stable/pike: Fix cache update after initial apt_repository fail
*** esberglu has quit IRC22:32
*** ansmith has joined #openstack-ansible22:37
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: add minimal functional tests
admin0containers have  names like  c3_nova_scheduler_container-c46a84fd  . waiting for it to complete so that i can login and check its hostname22:41
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Add minimal functional gate
cloudnullyes the name should always have an _22:43
cloudnullhowever the hostname should be tranlated to -22:43
admin0cloudnull, c3_cinder_api_container-9182f2dd22:44
*** dave-mcc_ has quit IRC22:44
admin0is the hostname22:44
admin0it did not translate22:44
cloudnullthat's what's been written into the /etc/hostname file ?22:45
cloudnullmind pasting /etc/hostname and /etc/hosts?22:45
admin0cat /etc/hostname => c3_cinder_api_container-9182f2dd      cat /etc/hosts =>   c3_cinder_api_container-9182f2dd22:46
cloudnullwell that's not good.22:47
openstackgerritMerged openstack/openstack-ansible-os_panko master: Updated from OpenStack Ansible Tests
cloudnullyou should see a domain name22:49
cloudnullin etc/hosts22:49
cloudnulland etc/hostname should have been rewritten
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: Fix cache update after initial apt_repository fail
admin0i am adding your keys22:49
admin0uno momento22:49
*** jwitko__ has joined #openstack-ansible22:50
*** jwitko_ has quit IRC22:54
*** jwitko__ has quit IRC22:56
*** acormier has joined #openstack-ansible22:57
*** acormier has quit IRC23:01
*** idlemind has joined #openstack-ansible23:02
openstackgerritMerged openstack/openstack-ansible-tests master: Set SELinux to permissive mode for tests
admin0is there an example of magnum, designate, octavia setup and usage ?23:10
admin0and does live-migration (without shared storage) work out by default ?23:10
admin0or need to override some vars for it ?23:10
*** jwitko_ has joined #openstack-ansible23:14
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: add minimal functional tests
*** dariko has quit IRC23:30
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Migrate security into user guide
*** acormier has joined #openstack-ansible23:38
*** acormier has quit IRC23:38
*** acormier has joined #openstack-ansible23:39
*** weezS__ has joined #openstack-ansible23:44
*** weezS has quit IRC23:45
*** weezS_ is now known as weezS23:45
*** weezS is now known as 7JTADKMG923:45
*** weezS__ is now known as 7GHAB538623:45
*** abelur has quit IRC23:45
*** 7JTADKMG9 has left #openstack-ansible23:46
*** 7GHAB5386 has left #openstack-ansible23:46
admin0ok .. suppose if you have 50 lxc containers to create . say 5 on each 10 machines, and if only 1 of that fails .. so you see 49 green and 1 red ..  guess what ? the hostname conversion does not apply to the rest 49 shown in green23:51
admin0silent bug23:51
admin0so while i see all green, just 1 red on container X that fails .. i do not realize that the hostname thing is not done23:52
admin0and it hits back @ the end23:52
admin0after you have done all23:52
admin0was able to reproduce this23:52
admin0so say while i had a container failed on say c1 and everything on c3 looked green,  hostname still remained c3_utility_container-d12ce6d9 .. i had to rerun the container create -- bypassing everything that could have failed and then it updated the hostname23:54
admin0so cloudnull , if people complain of this again and we do not know why it happens, this is the case23:55
admin0ask if out of XXX if any container failed due to whatever reasons and rest all  = green .. this happens23:55
cloudnullah. that's kinda a bummer23:56
admin0to reproduce i put one host under iroic with OSA which does not work .. so the ironic container will fail while the whole infrastructure will show red and ok .. but the hostname will not change23:56
admin0so when the cinder was run, it fixed cinder only23:56
admin0not others23:56
cloudnullyea. that makes sense.23:56
admin0i did a debug and when i bypass containers i know will fail, it updated all23:56
admin0so green != green :D23:57
cloudnulli wonder if this is related to
admin0no idea :)23:57
cloudnullIE there was less than 20% failure23:58
cloudnullI think we should just dump those lines23:58
admin0the hostname thing should still execute on a per vm basis23:58
admin0maybe the trigger is somewhere else when it does when ALL passes without issues23:58

Generated by 2.15.3 by Marius Gedminas - find it at!