Tuesday, 2019-11-12

*** nurdie has joined #openstack-ansible00:07
*** zbr has quit IRC00:09
*** nicolasbock has joined #openstack-ansible00:10
*** nurdie has quit IRC00:16
*** DanyC has quit IRC00:30
*** ckonstanski has quit IRC00:46
*** yoctozepto has quit IRC00:53
*** yoctozepto has joined #openstack-ansible00:54
*** yoctozepto has quit IRC01:05
*** yoctozepto has joined #openstack-ansible01:05
*** nicolasbock has quit IRC02:27
*** udesale has joined #openstack-ansible03:41
*** udesale has quit IRC03:45
*** udesale has joined #openstack-ansible03:46
*** ivve has joined #openstack-ansible03:46
*** rohit02 has joined #openstack-ansible03:49
*** idlemind has joined #openstack-ansible04:01
*** rohit02 has quit IRC05:12
*** rohit02 has joined #openstack-ansible05:16
*** shyamb has joined #openstack-ansible05:33
*** shyamb has quit IRC05:38
*** udesale has quit IRC05:43
*** udesale has joined #openstack-ansible05:43
*** udesale has quit IRC05:44
*** udesale has joined #openstack-ansible05:44
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible-os_tempest master: Make smoke tests as a default whitelist tests  https://review.opendev.org/65206005:51
*** ivve has quit IRC05:59
*** shyamb has joined #openstack-ansible06:03
*** ygk_12345 has joined #openstack-ansible06:08
*** shyamb has quit IRC06:31
*** shyamb has joined #openstack-ansible06:31
*** rohit02 has quit IRC06:48
*** lixingxing has joined #openstack-ansible06:50
*** rohit02 has joined #openstack-ansible06:59
*** ivve has joined #openstack-ansible07:03
*** udesale has quit IRC07:03
*** udesale has joined #openstack-ansible07:04
ygk_12345https://bugs.launchpad.net/oslo.messaging/+bug/185217507:22
openstackLaunchpad bug 1852175 in oslo.messaging "MessagingTimeout" [Undecided,New]07:22
*** rohit02 has quit IRC07:33
*** rohit02 has joined #openstack-ansible07:36
*** trident has quit IRC07:37
*** shyamb has quit IRC07:40
*** trident has joined #openstack-ansible07:48
*** tosky has joined #openstack-ansible07:52
*** cshen has joined #openstack-ansible08:05
*** rohit02 has quit IRC08:30
*** zbr has joined #openstack-ansible08:30
*** tosky has quit IRC08:32
*** zbr has quit IRC08:35
*** lixingxing has quit IRC08:35
*** tosky has joined #openstack-ansible08:37
*** zbr has joined #openstack-ansible08:37
*** shyamb has joined #openstack-ansible08:39
chandankumarHello is suse jobs broken? https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_5ee/652060/7/check/openstack-ansible-deploy-aio_metal-opensuse-15/5eed25d/job-output.txt08:41
*** rohit02 has joined #openstack-ansible08:47
*** lixingxing has joined #openstack-ansible08:48
noonedeadpunkmornings08:49
noonedeadpunkchandankumar: yeah, looks like they are:(08:50
*** rohit02 has quit IRC08:55
*** rohit02 has joined #openstack-ansible08:56
*** DanyC has joined #openstack-ansible08:58
*** DanyC_ has joined #openstack-ansible08:59
*** DanyC has quit IRC08:59
*** rpittau|afk is now known as rpittau08:59
*** DanyC_ has quit IRC09:03
*** DanyC has joined #openstack-ansible09:04
*** hamzaachi has joined #openstack-ansible09:07
*** sshnaidm|off is now known as sshnaidm|rover09:30
*** rohit02 has quit IRC09:33
*** rohit02 has joined #openstack-ansible09:33
CobHeadHi everyone :)09:39
*** irclogbot_1 has quit IRC09:39
CobHeadAnyone frequently working with the Cinder role in OSA here?09:39
chandankumarnoonedeadpunk: let me do a recheck09:39
*** irclogbot_1 has joined #openstack-ansible09:40
*** tosky has quit IRC09:41
*** tosky has joined #openstack-ansible09:42
*** shyamb has quit IRC09:49
*** tosky_ has joined #openstack-ansible09:52
*** tosky has quit IRC09:54
*** tosky has joined #openstack-ansible09:58
*** tosky_ has quit IRC10:01
ygk_12345can anyone tell me how to start a multinode galera cluster10:03
ygk_12345i have rebooted the servers10:03
ygk_12345now the mariadb is not starting10:03
*** tosky_ has joined #openstack-ansible10:05
noonedeadpunkygk_12345: if you've restarted them all at once, most likely you'll need to recover galera manually10:06
ygk_12345how to do that ?10:06
noonedeadpunkyou can start here https://docs.openstack.org/openstack-ansible/rocky/admin/maintenance-tasks/galera.html and continue on galera docs10:07
ygk_12345it didn't help10:08
*** tosky has quit IRC10:08
noonedeadpunkso generaly spoeaking you need to detect which node stopped the last and has better state, and re-init cluster making that node as a master to which rest will connect and refer10:11
ygk_12345i tried but the master is not starting10:11
*** tosky has joined #openstack-ansible10:14
gokhaniygk_12345, firstly stop mariadb services on nodes and then check seqno from /var/lib/mysql/grastate.dat. on a node which has bigger seqno edit /var/lib/mysql/grastate.dat as  safe_to_bootstrap: 1 after that kill all myql processes as 'killall -KILL mysql mysqld_safe mysqld mysql-systemd' then start a new cluster with running galera_new_cluster command and on other nodes run 'systemctl restart mariadb ' I hope it will work10:16
*** tosky_ has quit IRC10:17
ygk_12345gokhani from where I should run the galera_new_cluster command ?10:20
ygk_12345the seq no is -1 in both the containers10:21
*** tosky_ has joined #openstack-ansible10:21
*** tosky has quit IRC10:24
jrossernoonedeadpunk: i think i know whats going on with out upgrade10:26
*** shyamb has joined #openstack-ansible10:26
jrosserthe documentation says to run this openstack-ansible "${SCRIPTS_PATH}/upgrade-utilities/deploy-config-changes.yml"10:26
jrosserbut not to pass placement_migrate_flag to that10:27
noonedeadpunkI think it should be included into upgrade script10:27
noonedeadpunkah....10:27
noonedeadpunkhttps://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/run-upgrade.sh#L17210:28
admin0after a power failure , when i try to start mysql, i am gettin this: The galera_cluster_name variable does not match what is set in mysql. Check your galera_cluster_name setting in your user_*.yml files in \"/etc/openstack_deploy\" and compare to the current value in \"/etc/mysql/conf.d/cluster.cnf\" on the host, and the \"wsrep_cluster_name\" value set in your running galera cluster. To ignore the cluster state set '-e galera_ignore_cluster_sta10:28
admin0te=true10:28
jrosserok, i never use the upgrade script10:28
jrosseralways the manual step-by-step10:28
noonedeadpunkah, I see:)10:28
admin0it simply fails to start manually also10:29
gokhaniygk_12345, if seq no is -1 on all container nodes , To find the sequence number of the last committed transaction, run mysqld with the --wsrep-recover option. and then check seqno in galera log10:29
*** hamzaachi has quit IRC10:29
noonedeadpunkso if we need just to adjust docs - that's great10:29
jrosserthe manual docs also say openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e 'placement_migrate_flag=true'10:29
jrosserand i'm not sure that flag goes anywhere inside the infrastructure playbooks10:29
*** vesper has quit IRC10:30
noonedeadpunkit may affect haproxy10:30
noonedeadpunkby dropping out nova placement containers10:30
*** vesper11 has joined #openstack-ansible10:31
noonedeadpunkbut looks like it's not10:32
*** hamzaachi has joined #openstack-ansible10:33
*** tosky has joined #openstack-ansible10:34
jrosserhttp://codesearch.openstack.org/?q=placement_migrate_flag&i=nope&files=&repos=10:34
jrosserdirk: are you around - need some advice on where we should be getting our erlang packages from for suse10:36
*** tosky_ has quit IRC10:36
jrossernoonedeadpunk: ^ did you see the version of these packages changed already10:36
noonedeadpunkjrosser: https://review.opendev.org/#/c/664867/17/inventory/group_vars/haproxy/haproxy.yml10:36
noonedeadpunkyeah, that's annoying10:37
noonedeadpunkwe can drop lock on erlang, but dunno how good this idea is10:37
CobHeadI will just pitch this out in general then: When OSA runs the Cinder role, it seems to me that it is creating a backend with <host>+volumtype name for each volume type over n hosts. The only caveat when doing this is that if you want to change the volume type name, you first change it in OpenStack, and then you change it in the OSA config and redeploy. What happens then is that the backend for the old volume type is set to "Down" by OSA,10:38
CobHeadand this ultimately leads to users being unable to delete, snapshot and resize their volumes.10:38
noonedeadpunkI mean that I'd prefer using some more stable repo if it exist10:38
noonedeadpunkexternal repo ofc10:39
jrosserwhy don't we have a placement whitelist networks?10:39
noonedeadpunknot sure I understood...10:40
jrossershould it be in here https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/haproxy/haproxy.yml#L30-L3710:40
jrosseroh hmm seems placement has a public endpoint10:41
jrosserwhich isnt really used for anything right now10:41
noonedeadpunkyou mean to restrict access to placement only to internal services?10:41
jrosseryeah i'd assumed it was internal only10:41
jrosserbut that may be a bad assumption10:41
noonedeadpunkYep, it probably shouldn't be used right now by anything, except openstack client10:42
*** usr2033 has joined #openstack-ansible10:42
usr2033Hi, where should i start for multi node osa install on virtual machines10:43
noonedeadpunkbut not sure if non-admin cad do anything to placement10:43
*** tosky_ has joined #openstack-ansible10:43
jrosserusr2033: we have a thing here https://github.com/openstack/openstack-ansible-ops/tree/master/multi-node-aio10:43
noonedeadpunkor https://docs.openstack.org/openstack-ansible/train/user/prod/example.html10:44
jrosserthat makes a bunch of VM on one giant server that makes a multinode dev environment10:44
noonedeadpunkoh, we have that? :P10:44
jrosseror alternatively you can use individual VM from an existing cloud, just make sure to disable port security on the vm interfaces10:45
*** tosky has quit IRC10:45
usr2033jrosser, thank you10:45
usr2033noonedeadpunk, thank you10:45
usr2033i want to use multiple vms as nodes of installation10:45
*** rohit02 has quit IRC10:46
noonedeadpunkYeah, that's what I thought about10:46
*** shyamb has quit IRC10:46
usr2033i am looking for may be one controller, one compute installation ike10:46
*** shyam89 has joined #openstack-ansible10:46
*** tosky_ has quit IRC10:46
*** tosky has joined #openstack-ansible10:46
jrosserusr2033: have you started by trying the all-in-one?10:47
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible stable/train: Collect etcd logs  https://review.opendev.org/69380610:48
*** ygk_12345 has quit IRC10:48
jrosserusr2033: this is the best place to start https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html10:49
jrosserthat will build you everything in a single VM, and is a good way to get familiar with how openstack-ansible works10:49
jrosserthen you can move on to building something larger / more complex10:50
usr2033jrosser, i installed aio once, inspected some. i looked at containers, service isolation but no networking parts.10:51
jrosserok so next thing to do is study the example that noonedeadpunk gave here https://docs.openstack.org/openstack-ansible/train/user/prod/example.html10:52
jrosserbut that is for a 3 controller HA setup10:52
jrosserand the AIO does all of the config for you, when you move beyond the AIO it is up to you to generate openstack_user_config and any user_variables you need10:53
*** gokhani has quit IRC10:54
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible stable/stein: Collect etcd logs  https://review.opendev.org/69380910:54
usr2033jrosser, i will check https://docs.openstack.org/openstack-ansible/train/user/prod/example.html. thank you10:54
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-os_neutron stable/stein: l3 agent on network_hosts do dvr_snat, anywhere else dvr  https://review.opendev.org/69303110:56
*** gshippey has joined #openstack-ansible11:00
*** tosky has quit IRC11:01
*** tosky has joined #openstack-ansible11:01
*** shyam89 has quit IRC11:03
*** cshen has quit IRC11:05
*** tosky_ has joined #openstack-ansible11:05
*** tosky has quit IRC11:06
*** tosky_ has quit IRC11:09
*** tosky has joined #openstack-ansible11:09
*** lixingxing has quit IRC11:10
*** rohit02 has joined #openstack-ansible11:13
*** udesale has quit IRC11:17
*** tosky has quit IRC11:17
*** cshen has joined #openstack-ansible11:20
*** tosky has joined #openstack-ansible11:23
*** shyamb has joined #openstack-ansible11:31
*** rohit02 has quit IRC11:32
*** aedc_ has joined #openstack-ansible11:46
*** aedc has quit IRC11:47
*** tosky_ has joined #openstack-ansible11:47
*** tosky has quit IRC11:49
*** tosky has joined #openstack-ansible11:50
*** tosky_ has quit IRC11:53
*** aedc_ has quit IRC11:53
*** rohit02 has joined #openstack-ansible12:12
*** nicolasbock has joined #openstack-ansible12:13
*** dave-mccowan has joined #openstack-ansible12:19
*** shyamb has quit IRC12:26
*** shyamb has joined #openstack-ansible12:30
*** masterpe has quit IRC12:39
*** csmart has quit IRC12:39
*** openstackgerrit has quit IRC12:41
*** weshay|rover is now known as weshay12:54
*** sshnaidm|rover is now known as sshnaidm|ruck12:55
*** shyam89 has joined #openstack-ansible12:56
*** shyamb has quit IRC12:56
*** shyam89 has quit IRC13:00
*** shyamb has joined #openstack-ansible13:00
*** shyamb has quit IRC13:05
*** rohit02 has quit IRC13:09
*** masterpe has joined #openstack-ansible13:10
*** csmart has joined #openstack-ansible13:10
admin0quick question .. logs container can be anywhere right ? it just need layer3 and not layer213:12
noonedeadpunkI think so. It needs connection to mgmt network though13:13
noonedeadpunkbut yes, you're right13:14
*** schwicht has joined #openstack-ansible13:22
*** schwicht has quit IRC13:28
*** openstackgerrit has joined #openstack-ansible13:34
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Revert "Add nova placement to placement migration"  https://review.opendev.org/69374013:34
*** ansmith has joined #openstack-ansible13:35
*** lixingxing has joined #openstack-ansible14:19
*** lixingxing has quit IRC14:21
*** ivve has quit IRC14:22
*** schwicht has joined #openstack-ansible14:22
*** schwicht has quit IRC14:23
*** schwicht has joined #openstack-ansible14:23
*** ansmith_ has joined #openstack-ansible14:31
*** ansmith has quit IRC14:33
*** ansmith has joined #openstack-ansible14:34
*** ansmith_ has quit IRC14:36
*** DanyC has quit IRC14:40
*** goldyfruit has joined #openstack-ansible14:44
*** nurdie has joined #openstack-ansible14:45
*** gokhani has joined #openstack-ansible14:53
*** tosky_ has joined #openstack-ansible15:00
*** tosky has quit IRC15:00
*** tosky_ is now known as tosky15:01
*** DanyC has joined #openstack-ansible15:08
jrossernoonedeadpunk: maybe we need to delegate these to the db setup host? https://github.com/openstack/openstack-ansible-os_placement/blob/stable/train/tasks/placement_db_sync.yml#L16-L3315:25
jrosserjust testing that now15:25
*** hamzaachi has quit IRC15:25
noonedeadpunkno15:26
noonedeadpunkwe need to install placement on db_srtup_host then...15:26
*** ansmith has quit IRC15:26
noonedeadpunkas mysql-migrate-db.sh is shipped in pip/distro packages15:26
*** pcaruana has joined #openstack-ansible15:28
*** rgogunskiy has joined #openstack-ansible15:28
jrosserit seems to need mysql cli which we don't have in the placement container15:30
*** hamzaachi has joined #openstack-ansible15:30
*** tosky has quit IRC15:31
*** tosky has joined #openstack-ansible15:32
jrosserso it seems either placement container needs the mysql cli tools and a suitable credential15:34
jrosseror the placement_db_tools could go on the utility container that already has db access15:35
*** hamzaachi_ has joined #openstack-ansible15:37
*** aedc has joined #openstack-ansible15:39
*** hamzaachi has quit IRC15:40
*** rgogunskiy has quit IRC15:40
*** rgogunskiy has joined #openstack-ansible15:43
*** rgogunskiy has quit IRC15:48
*** also_stingrayza is now known as stingrayza15:54
*** cshen has quit IRC16:01
*** jhesketh has quit IRC16:05
*** ansmith has joined #openstack-ansible16:10
*** gyee has joined #openstack-ansible16:15
*** jhesketh has joined #openstack-ansible16:21
*** spatel has joined #openstack-ansible16:33
spatelMorning folks16:33
spatelwhat is the best way to calculate capacity in openstack cloud?16:33
spatelI know its hard question and can have many variables in it.16:34
spatelTrying to do capacity planning..16:34
*** macz has joined #openstack-ansible16:55
*** ivve has joined #openstack-ansible16:57
*** rpittau is now known as rpittau|afk17:00
*** pcaruana has quit IRC17:03
noonedeadpunkoh my, I mixed up meeting time.. AGAIN:(((((17:07
* noonedeadpunk installing second alarm for the meeting17:07
noonedeadpunkjrosser: oh... I thought I tested it in lxc, but maybe I'm wrong:(17:09
noonedeadpunkI guess installing mysql-client is easier, but not good...17:10
*** ansmith has quit IRC17:14
*** ansmith has joined #openstack-ansible17:18
*** hamzaachi_ has quit IRC17:21
*** chandankumar is now known as rrrrrrrrrrrr17:29
*** asettle has quit IRC17:30
*** rrrrrrrrrrrr is now known as chandankumar17:34
*** chandankumar is now known as raukadah17:38
*** rgogunskiy has joined #openstack-ansible17:45
*** spatel has quit IRC17:45
jrossernoonedeadpunk: we are just looking at that now17:47
jrossernoonedeadpunk: so we seem to have success with running galera_client against the placement containers, and making sure a .my.cnf gets created with the placement galera credentials in it17:49
jrosserand also it needs mysqdump which is in galera-client package rather than galera-client-core17:50
*** rgogunskiy has quit IRC17:50
noonedeadpunkthat sounds like rather good news17:51
jrosserit needs a bit more work becasue it would be easy to break the metal case here17:52
jrosserby overwriting /root/.my.cnf with the placement one17:52
jrosserso we need to re-test with that in /home/placement/.my.cnf17:53
noonedeadpunkI don't think placement needs it's credentials in my.cnf17:53
noonedeadpunkmigrate script should take creds from https://opendev.org/openstack/openstack-ansible-os_placement/src/branch/stable/train/templates/placement-migrate-db.rc.j217:54
jrosseroh hmm well thats not working then :)17:54
noonedeadpunkhm...17:54
noonedeadpunkit should be executed that way https://opendev.org/openstack/openstack-ansible-os_placement/src/branch/stable/train/tasks/placement_db_sync.yml#L2517:55
jrosseri didnt know about that file, and we had it all failing not being able to make a db connection17:55
jrosserso just went ahead and made a .my.cnf17:55
noonedeadpunkI believe it worked for me passing that config17:56
jrosserbut we will go back and check why it does not seem to use the .rc file17:56
noonedeadpunkbut yes, it will still need mysqlclient...17:56
noonedeadpunkSo migration should fail I guess when no rc file provided https://opendev.org/openstack/placement/src/branch/stable/train/placement_db_tools/mysql-migrate-db.sh#L168-L18717:58
jrosserwe just checked and the rc file is there and looks reasonable18:00
*** spatel has joined #openstack-ansible18:01
*** idlemind has quit IRC18:07
*** DanyC has quit IRC18:07
*** hamzaachi has joined #openstack-ansible18:20
*** hamzaachi has quit IRC18:21
*** cshen has joined #openstack-ansible18:24
*** aedc has quit IRC18:25
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_zun master: Drop commented out values from configs  https://review.opendev.org/69354318:35
*** cshen has quit IRC18:40
openstackgerritDave Wilde (d34dh0r53) proposed openstack/openstack-ansible-os_keystone stable/train: Standardize on nginx-extras  https://review.opendev.org/69390318:41
*** cshen has joined #openstack-ansible18:51
*** cshen has quit IRC18:56
*** sshnaidm|ruck is now known as sshnaidm|afk19:42
*** andrea15 has joined #openstack-ansible20:10
*** munimeha1 has joined #openstack-ansible20:21
*** nurdie has quit IRC20:30
*** DanyC has joined #openstack-ansible20:41
*** nurdie has joined #openstack-ansible21:15
*** nurdie has quit IRC21:21
*** DanyC has quit IRC21:23
*** DanyC has joined #openstack-ansible21:24
*** spatel has quit IRC21:44
*** schwicht has quit IRC21:53
admin0is there a way to add affinity or setup preferred master in haproxy for a certain controller ?21:54
admin0for 1 deployment, i have 2 physical servers and 1 virtual server for controller .. i want to say i prefer to have the haproxy be master on the physical controller and not be the virtual one21:55
*** schwicht has joined #openstack-ansible21:59
*** samuelbernardo has joined #openstack-ansible22:00
openstackgerritDirk Mueller proposed openstack/openstack-ansible-os_horizon master: opensuse: adjust for Python 3.x switch  https://review.opendev.org/69183322:16
*** ansmith has quit IRC22:17
admin0while adding a new mysql node, i get this: The galera_cluster_name variable does not match what is set in mysql. Check your galera_cluster_name setting in your user_*.yml files in \"/etc/openstack_deploy\" and compare to the current value in \"/etc/mysql/conf.d/cluster.cnf\" on the host, and the \"wsrep_cluster_name\" value set in your running galera cluster. To ignore the cluster state set '-e galera_ignore_cluster_state=true'.\n" -- what is22:45
admin0the recommended way to address this ?22:45
admin0runing galera has nothing22:46
admin0do i have to manually setup this value ?22:47
admin0cluster.conf has "openstack_galera_cluster"  -- these have been running for months now .. not sure if this was ever set22:48
admin0oh .. its set .. its on variable and i was checking status22:50
admin0but again, i am stuck due to that issue22:50
*** schwicht has quit IRC23:20
*** DanyC has quit IRC23:24
*** DanyC has joined #openstack-ansible23:24
*** ivve has quit IRC23:25
*** goldyfruit has quit IRC23:28
*** DanyC has quit IRC23:43
*** munimeha1 has quit IRC23:49
*** schwicht has joined #openstack-ansible23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!