Friday, 2020-02-14

*** dave-mccowan has joined #openstack-ansible00:18
*** tosky has quit IRC00:51
*** dave-mccowan has quit IRC01:32
*** gyee has quit IRC01:36
*** jawad_axd has joined #openstack-ansible02:22
*** jawad_axd has quit IRC02:27
*** jawad_axd has joined #openstack-ansible02:43
*** jawad_axd has quit IRC02:47
openstackgerritMerged openstack/openstack-ansible-ops master: Allow user override of beat install locations  https://review.opendev.org/70652803:42
*** Neurognostic_ has joined #openstack-ansible04:15
*** Neurognostic has quit IRC04:17
*** Neurognostic_ has quit IRC04:31
*** udesale has joined #openstack-ansible05:04
*** nicolasbock has quit IRC05:09
*** DanyC has joined #openstack-ansible05:21
*** DanyC has quit IRC05:23
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-ansible05:34
*** raukadah is now known as chandankumar05:36
*** CeeMac has quit IRC05:50
*** goldyfruit has quit IRC06:00
*** goldyfruit has joined #openstack-ansible06:00
*** elenalindq has joined #openstack-ansible06:17
*** jhesketh has quit IRC06:48
*** jhesketh has joined #openstack-ansible06:50
*** mcarden has quit IRC06:51
*** rgogunskiy has joined #openstack-ansible07:00
*** jhesketh has quit IRC07:15
*** jawad_axd has joined #openstack-ansible07:22
*** jhesketh has joined #openstack-ansible07:25
*** sluna has joined #openstack-ansible07:27
*** miloa has joined #openstack-ansible07:30
*** shyamb has joined #openstack-ansible07:41
miloaMorning07:41
*** cshen has joined #openstack-ansible07:45
*** shyamb has quit IRC07:47
*** ivve has joined #openstack-ansible07:56
*** blue_asni has joined #openstack-ansible08:03
blue_asnihello fork, nice to join you. i have a problem with deploying "train" AIO version using openstack-ansible in opensuse leap  but "failed retrying ensure image has been pre-staged08:08
blue_asnianyone help appreciated08:08
*** sluna has quit IRC08:08
*** sluna has joined #openstack-ansible08:09
blue_asniopemstack-ansible setup-hosts.yml08:22
blue_asnifailer08:22
cshenblue_asni: put your error logs in http://paste.openstack.org/08:30
cshenwill help people investigate your issue.08:30
jrossernotably we don't test containerised deployments at all as far as i can see on opensuse08:33
*** tosky has joined #openstack-ansible08:36
*** shyamb has joined #openstack-ansible08:51
*** xakaitetoia has joined #openstack-ansible08:59
*** masterpe has quit IRC09:00
*** blue_asni has quit IRC09:26
*** fridtjof[m] has joined #openstack-ansible09:31
*** shyamb has quit IRC09:33
*** shyamb has joined #openstack-ansible09:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/70779409:42
admin0good morning .. what playbook adds the VIPs to the controllers ?09:44
admin0 i am trying for the first time with a single controller only .. the vip is not coming up ..  on multi controller env, i recall it used to be the haproxy playbook09:44
admin0or i am mistaken .. it was ages(rocky) ago09:45
admin0i am trying 20.0.1 now09:45
*** ysastri has joined #openstack-ansible09:45
*** shyamb has quit IRC09:52
jrosseradmin0: on a single controller there is no VIP09:54
admin0so i need to add the vip addresses manually myself ?09:54
jrosserjust the IP for the external interface that haproxy is listening on09:54
jrosserit is very much like how an AIO is setup09:55
jrosserthe haproxy role only deploys keepalived when there are >1 controllers09:56
admin0ok ..09:56
admin0thanks09:56
admin0i have either done aio or multi controllers.. but never a single node multi setup ..09:56
admin0thanks.. i will add the ips myself then09:56
jrosserafaik it should look kindof like one controller from your existing multnode deploys, minus keepalived09:57
*** masterpe has joined #openstack-ansible10:00
*** andrewbonney has joined #openstack-ansible10:06
*** L_SR has joined #openstack-ansible10:33
L_SRHello guys, some people in #openstack-infra get error 500 from opendev.org/openstack; do you? (it fail in the middle of an OSA deployment I was doing)10:34
*** shyamb has joined #openstack-ansible10:36
*** ysastri has quit IRC11:05
*** L_SR has quit IRC11:14
openstackgerritMerged openstack/openstack-ansible master: Drop virtualenv pip package for CI  https://review.opendev.org/70721211:27
*** udesale_ has joined #openstack-ansible11:56
*** shyamb has quit IRC11:59
*** udesale has quit IRC11:59
*** shyamb has joined #openstack-ansible12:01
admin0stuck in a strange error: Create openstack client bash_completion script /bin/bash: openstack: command not found .. on the utility container in the setup-infra playbook12:07
admin0rerunning hosts and infra with limit to this container fixed it :)12:11
cshenweird :-)12:11
*** nicolasbock has joined #openstack-ansible12:13
admin0hi .. i am getting this error during setup-infra ..   http://paste.openstack.org/show/789569/  .. set_fact _current_monitor_address\n  ^ here\n" .. what might i be missing ..  in the config, ceph-mon_hosts: *infrastructure_hosts is set12:18
noonedeadpunkadmin0: I'm sure I know what it is12:25
admin0current_monitor_address does not look like a var12:25
admin0\o/12:25
*** shyamb has quit IRC12:25
admin0my savior12:25
noonedeadpunklet me find patch fixing this12:25
noonedeadpunkadmin0: https://review.opendev.org/#/c/702853/12:27
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/train: Allow ceph metal CI deployments  https://review.opendev.org/70782912:27
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/stein: Allow ceph metal CI deployments  https://review.opendev.org/70783012:27
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/rocky: Allow ceph metal CI deployments  https://review.opendev.org/70783112:28
noonedeadpunkadmin0: you do aio deployment?12:28
admin0no ..12:28
admin0its a multi node .. but with a single controller12:28
noonedeadpunkhm.. with lxc?12:29
admin0kvm12:29
noonedeadpunkok, anyway I think you need to set valid monitor_interface12:30
admin0 but how .. i did not find a variable for this12:30
noonedeadpunkjust "monitor_interface: bond0" or whatever in user_variables.yml12:31
*** cshen has quit IRC12:31
noonedeadpunkI was catching exact the same error in aio builds when monitor_interface was set to eth1, which didin't exist on the node12:31
noonedeadpunkIt's used to retrieve monitor address by interface name12:32
admin0i put monitor_interface: br-mgmt in variables,  .. still got the same error12:32
noonedeadpunkwhat interface is used on ceph monitor hosts as ceph public network?12:33
admin0br-storage12:34
admin0public_network: 172.29.244.0/2212:34
admin0monitor_address_block: 172.29.244.0/2212:34
noonedeadpunkdoes br-storage exist on ceph-mon hosts?12:35
admin0yep12:35
admin0all my hosts have the 4 bridges setup12:35
noonedeadpunkok, so then set monitor_interface: br-storage12:35
admin0they are blades, so they come with identical interfaces12:35
admin0same error -- error output and relevant lines from user_variables.yml http://paste.openstack.org/show/xvYLFCrAdLyv89QVPjdg/12:38
admin0tag 20.0.112:38
noonedeadpunkah, ok, so you set monitor adress not by interface, but by address...12:38
noonedeadpunkSo everything I said before is unrelated^(12:39
admin0i just copied over what i was doing in a rocky install12:39
admin0i should remove this ? monitor_address_block: 172.29.244.0/2212:41
admin012:41
noonedeadpunkyou may try... but not sure what's better option.12:42
admin0the user_variables.yml.prod-ceph.example also users monitor_address_block: "{{ cidr_networks.container }}"12:42
admin0there is no mention of monitor_address anywhere12:42
noonedeadpunkwe use it in CI actually12:43
admin0the default is set to 0.0.0.012:43
admin0from the /etc/ansible/roles/ceph-ansible/roles/ceph-defaults/defaults/main.yml12:44
noonedeadpunkadmin0: do you have IP assigned on br-storage?12:44
admin0i do .. 172.29.244.1512:44
noonedeadpunkhum... what if you try to run some simple task with the only debug of the var hostvars[inventory_hostname]['ansible_all_ipv4_addresses'] against ceph-mon?12:46
noonedeadpunkThe thing is that while setting fact https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-facts/tasks/set_monitor_address.yml#L4 you have undefined addr key12:46
noonedeadpunkthat means that most likely one of the used vars there are not defined12:47
admin0how do I debug of that hostvar ?12:47
admin0i am not so great in debugging ansible12:47
noonedeadpunkso it's either hostvars[item]['ansible_all_ipv4_addresses'] or hostvars[item]['monitor_address_block']. Since I think that monitor_address_block is defined, I have doubts regarding the first one12:47
admin0big output .. finding the right key12:49
noonedeadpunkyou can try just runnign that playbook http://paste.openstack.org/show/789573/12:49
admin0172.29.238.104 -- it gave me the IP of the container ceph-mon12:50
admin0http://paste.openstack.org/show/789574/12:51
noonedeadpunkhum12:51
noonedeadpunkthis looks like your monitor do not have appropriate storage isnbtreface it should have12:52
noonedeadpunk*interface12:52
admin0ceph-mon_hosts: *infrastructure_hosts12:52
admin0right .. it only has mgmt as extra and not storage12:52
noonedeadpunkok, so probably double check your provider_networks12:53
jrossernoonedeadpunk: I think we may have discussed previously how the osa/ceph networking could be better12:53
jrosserthe default example is exactly like you are finding12:53
jrosserwith many things on mgmt which should be on storage12:53
noonedeadpunkhm, I think we have CI on mgmt for somplicity, by I was pretty sure that in examples we have storage networks mapping for containers....12:55
fridtjof[m]Hi! I'm trying to migrate my openstack (rocky) environment from Ubuntu 16.04 to 18.04. First thing I attempted to do was set up a second infra host, to make migrating the db etc possible. I based this infra node on 18.04, but the last step (setup-everything.yml limited to the new host) is failing, because the repo (on the existing infra host) only has packages etc for 16.04.12:55
fridtjof[m]Is there an easy way to extend the repo for 18.04?12:55
admin0noonedeadpunk,  jrosser  my config and user_variable http://paste.openstack.org/show/789576/12:55
noonedeadpunkfridtjof[m]: I think that once you have at least one ubuntu 18.04 host running repo-build should generate stuff for 18.0412:56
admin0based on what jrosser if i change my monitor_interface to br-mgmt and monitor_address_block to  .236.0/22  , it might work with the current playbooks ?12:57
noonedeadpunkadmin0: I think youve missed ceph-mon for br-storage12:57
noonedeadpunkin terms of group_binds12:57
noonedeadpunkactually it's probably worth doing ceph_all (or what group do we have for all ceph?)12:58
admin0its not in this file: openstack_user_config.yml.prod-ceph.example12:58
admin0i see only ceph-osd listed12:58
admin0i thought this file passed QA, so i did not read line by line or compared with my rocky one 012:58
noonedeadpunkwe might eventually have errors in examples :p12:58
admin0now i realize12:59
noonedeadpunkand, you can offer it's change if you find one :p12:59
admin0i checked from the working one done using rocky .. it does have ceph-mon on the group binds12:59
jrosserthere are two Ceph networks, and I think we confuse the cluster and public networks12:59
admin0i will submit a patch12:59
noonedeadpunkjrosser: oh... not sure about that actually - never deployed ceph with cluster net with my own...13:00
noonedeadpunkbut what I know that we don't make use of storage network at all in CI13:00
noonedeadpunkit's totally broken there13:01
fridtjof[m]noonedeadpunk: that seems kind of circular to me. That 18.04 infra host is my first 18.04 host, and repo-build does not seem to have generated anything new13:01
fridtjof[m]ls /var/www/repo/pools/ on the repo container only gives me one dir, ubuntu-16.04-x86_6413:01
*** dave-mccowan has joined #openstack-ansible13:03
noonedeadpunkfridtjof[m]: have you runned setup-hosts against this new ubuntu host?13:03
noonedeadpunkiirc repo build relies on cahced host facts13:04
admin0now failing at ceph-mon : waiting for the monitor(s) to form the quorum...]13:06
admin0monitor_name' is undefined13:07
noonedeadpunkhum, it should be set by ceph-ansible ceph-facts role here https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-facts/tasks/facts.yml#L27-L2913:08
*** ysastri has joined #openstack-ansible13:09
admin0-vvvv gave me more info : http://paste.openstack.org/show/ARDgEi51sv9RPVPgD4iP/13:10
admin0line 29 has the error13:11
admin0thisip .. 244.51 is the ip of the ceph-mon_container13:11
*** rh-jelabarre has joined #openstack-ansible13:12
admin0i think i have to drop the idea of a single controller and just go to either multinode or aio13:24
*** ivve has quit IRC13:28
admin0is everything that has .aio tested in CI and working ?13:33
admin0is there a scenario/command that will activate/install all of the things that aio can do13:33
admin0is this valid: export SCENARIO='aio_swift_barbican_cinder_designate_heat_magnum_masakari_murano_octavia_manila_trove' ?13:43
*** sshnaidm|afk has quit IRC13:50
*** sshnaidm has joined #openstack-ansible13:50
*** rgogunskiy has quit IRC13:52
*** rgogunskiy has joined #openstack-ansible13:53
noonedeadpunkadmin0: if you want to deploy all of these components, than yes. Actually for aio you can run the folowing ./scripts/gate-check-commit.sh aio_swift_barbican_designate_magnum_masakari_murano_octavia_manila_trove - but kinda manila may probably work only on master13:55
noonedeadpunk*"./scripts/gate-check-commit.sh aio_swift_barbican_designate_magnum_masakari_murano_octavia_manila_trove deploy source"13:56
noonedeadpunkand you probably need to add ceph to the scenariio as well13:58
*** cshen has joined #openstack-ansible14:05
*** rgogunskiy has quit IRC14:05
admin0noonedeadpunk, what does this gate_check command does ?14:05
*** sshnaidm is now known as sshnaidm|off14:05
admin0its downloading and doing a bunch of stuff :D14:05
noonedeadpunkit does the same as CI do14:06
noonedeadpunkactually deploys aio with provided scenario14:06
admin0hmm.. so no need to run the playbooks etc individually ?14:07
noonedeadpunkno - it does everything needed from blank host with only openstack-ansible cloned14:07
*** ysastri has quit IRC14:08
noonedeadpunkso make aio takes 3 steps - 1. create vm, 2. clone openstack-ansible 3. run gate-check-commit.sh $SCENARIO deploy|upgrade source|distro14:09
admin0i had already ran setup hosts .. does it conflict ?14:09
noonedeadpunkhm14:10
admin0i will try the regular way first . if it fails .. will nuke, recreate and try this way14:10
admin0noonedeadpunk, using your way :)14:28
admin0waiting for this "magic" to finish14:28
admin0so question .. say this gate-check works, and i want to make little changes .. like change the haproxy ip etc , can i use the regular playbooks for that ?14:29
*** jawad_axd has quit IRC14:38
*** jawad_axd has joined #openstack-ansible14:39
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: [WIP] Optional subdivision of elastic-logstash group  https://review.opendev.org/70784914:40
*** jawad_axd has quit IRC14:41
admin0i see .. it just calls the playbooks aferwards :)15:02
*** cshen has quit IRC15:39
openstackgerritMerged openstack/openstack-ansible-ops master: Allow specified http ports to be ignored by packetbeat  https://review.opendev.org/70712415:52
taccohey guys.. does anyone of you also have problems with creating a stack with os_stack ansible module? If i try to do a stack create it always fails but in the output you see stack state "create in progress" even with wait: True15:57
taccobut. if i run again with state presend and it does a update to the stack everything works as expected.15:57
taccothis is just confusing.. and maybe a bug in the os_stack module right?15:58
*** udesale_ has quit IRC16:17
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/train: Drop virtualenv pip package for CI  https://review.opendev.org/70787116:18
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/train: Drop virtualenv pip package for CI  https://review.opendev.org/70787116:20
admin0does the containers copy the apt sources from the aio machine ?16:20
admin0hmm.. the aip containers cannot ping outside16:22
admin0somewhere, some nat rule is missing16:22
noonedeadpunktacco: I'd write either to #openstack-ansible-sig or to #openstack-sdks (I think you may find there ppl who look after ansible modules as well)16:25
bjoerntHello does OSA is considering full multi cell support at some point which localized cell controllers etc?16:35
bjoerntRight now we do it with custom env.d groups which is near impossible to manage16:36
bjoerntSo the inventory manager would have to be rewritten to support ironc or other types to allow for a more flexible role based management16:37
bjoerntnot to mention changes in roles themselves to limit services to roles16:37
fridtjof[m]noonedeadpunk: sorry for coming back on this so late, but I have not done that yet (the docs didn't say so). I'll try that now.16:39
fridtjof[m]okay, I ran setup-hosts against the new infrastructure host. The repo is still not updated, but i see that repo-build is not called from setup-hosts anyway. Running setup-everything now.16:46
fridtjof[m]oh. weird, setup-everything should actually have run setup-hosts already16:47
noonedeadpunkbjoernt: that is pretty nice feature request but not sure if we have capacity for realization... So I'd say if someone would offer a solution that would be great :p16:47
fridtjof[m]i'll report back16:47
fridtjof[m]oh.16:52
fridtjof[m]https://imgur.com/a/7aHUxKW16:54
fridtjof[m]noonedeadpunk: i think this is the root cause16:54
fridtjof[m]my limit group only contains localhost (the deployment host) and the new infra host16:54
fridtjof[m]should I run repo-build with all hosts?16:54
fridtjof[m](or maybe just all of setup-hosts?)16:55
noonedeadpunkoh, sure you shouldn't use any limits with repo-build16:55
noonedeadpunkactually for setup-infrastructure you also probably shouldn't use limit only new controller16:56
noonedeadpunk(as rabbit, galera, etc may not join cluster correctly)16:56
fridtjof[m]yeaaah, i just scrolled back and it completely skipped even installing memcached, galera, and rabbitmq16:57
fridtjof[m]is it safe to also run setup-hosts without a limiter?17:03
fridtjof[m]I think the containers on my new hosts did not get set up properly17:03
*** feichh has quit IRC17:04
admin0fridtjof[m], yes17:11
admin0you can safely run the setup-hosts, infra and openstack anytime17:11
*** xakaitetoia has quit IRC17:11
fridtjof[m]great, thanks :)17:11
*** pcaruana has quit IRC17:19
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-ansible17:34
*** errr has quit IRC17:55
*** errr has joined #openstack-ansible17:56
*** miloa has quit IRC18:13
*** macz_ has joined #openstack-ansible18:29
*** macz_ has quit IRC18:30
fridtjof[m]ahhhhhhhhh18:34
fridtjof[m]galera broke18:34
fridtjof[m]it somehow skipped over setting up galera on the new infra host, which made replication fail because the old galera couldn't reach the new one18:35
fridtjof[m]okay, it seems that i fixed it18:48
*** andrewbonney has quit IRC18:55
*** tosky has quit IRC19:36
admin0on aio, manila fails with 'manila_default_store' is undefined ....   when i do lvs in the aio, i do see that it created the manila-shares vg ..19:51
admin0i added manila_default_store: lvm in variables and checking if that fixes it19:53
*** hwoarang has quit IRC20:43
*** spatel has joined #openstack-ansible20:51
*** hwoarang has joined #openstack-ansible20:51
*** spatel has quit IRC21:08
*** dave-mccowan has quit IRC21:23
*** Neurognostic has joined #openstack-ansible21:28
*** nicolasbock has quit IRC21:54
*** macz_ has joined #openstack-ansible22:09
openstackgerritMerged openstack/openstack-ansible stable/train: Drop virtualenv pip package for CI  https://review.opendev.org/70787122:31
*** rh-jelabarre has quit IRC22:31
*** elenalindq has quit IRC22:51
*** mcarden has joined #openstack-ansible22:54
*** spatel has joined #openstack-ansible23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!