Tuesday, 2020-12-01

*** tosky has quit IRC00:02
*** macz_ has quit IRC00:40
openstackgerritMerged openstack/openstack-ansible master: Added Openstack Adjutant role deployment  https://review.opendev.org/c/openstack/openstack-ansible/+/75631000:58
*** cshen has quit IRC01:07
*** rfolco has joined #openstack-ansible02:27
*** rfolco has quit IRC02:32
*** spatel has joined #openstack-ansible04:24
spatelnoonedeadpunk: hey04:24
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-ansible05:33
*** gyee has quit IRC05:55
*** spatel has quit IRC06:42
*** klamath_atx has quit IRC06:47
*** bjoernt has quit IRC06:58
*** pto has joined #openstack-ansible07:18
*** pto_ has joined #openstack-ansible07:20
*** pto has quit IRC07:23
*** SiavashSardari has joined #openstack-ansible07:34
*** pcaruana has joined #openstack-ansible07:46
*** luksky has joined #openstack-ansible07:56
*** rpittau|afk is now known as rpittau07:59
*** cshen has joined #openstack-ansible08:01
*** pto_ has quit IRC08:03
*** andrewbonney has joined #openstack-ansible08:03
*** pto has joined #openstack-ansible08:04
*** ChosSimbaOne has left #openstack-ansible08:10
jrossernoonedeadpunk: did you see we have trouble with the new pip?08:11
*** pto has quit IRC08:13
*** pto has joined #openstack-ansible08:14
noonedeadpunkmornings08:20
noonedeadpunkhaven't yet, but this is expected?08:20
noonedeadpunkas they're having new resolver?08:20
noonedeadpunkBut we're supposed to have it bumped and fixed?08:21
jrosserfunctional tests seem to pick up the new pip08:22
noonedeadpunkuh...08:22
noonedeadpunkI see08:22
jrosseri took a fairly big guess at a fix but it's not worked https://review.opendev.org/c/openstack/openstack-ansible-tests/+/76482408:23
*** pto has quit IRC08:29
*** ygk_12345 has joined #openstack-ansible08:29
*** pto has joined #openstack-ansible08:32
SiavashSardarimorning08:37
SiavashSardariI have a qq08:37
noonedeadpunkjrosser: maybe it's a chance to test out new resolver as well :)08:38
noonedeadpunkhey SiavashSardari08:38
jrosseryeah, i'm just trying to debug my patch, it's not right08:38
SiavashSardariis handler stuff manage centrally somewhere? because they all kinda looks like the same08:38
jrosserit splits the global pins file into characters rather than lines i think08:39
noonedeadpunkout of error I see it seems it does08:39
noonedeadpunkSiavashSardari: no, handlers are not managed centrally08:39
*** tosky has joined #openstack-ansible08:40
SiavashSardarioh ok, I found some issue on handlers, almost all handlers of 'Restart * services' will check if  the service is started. but in update config case services should restart08:42
*** spatel has joined #openstack-ansible08:43
SiavashSardarifor  example in https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/handlers/main.yml08:43
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_zun master: DNM: Update zun role to match current requirements  https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/76314108:45
noonedeadpunkSiavashSardari: but they frist stop and then start for `Restart * services`08:45
noonedeadpunkfor config update I can recall only issue with uwsgi which did not get restarted on config change but it has been fixed08:46
SiavashSardarihm... you're right. I didn't see that. but why when I update my default quotas and run the playbook with e.g. nova-config tag, it needs a service restart afterwards?08:47
*** spatel has quit IRC08:47
SiavashSardarioh because the service won't stop first08:48
noonedeadpunkwell, regarding tags there are bugs in terms that we might miss some task that is required08:48
noonedeadpunklike setting some facts or registering variable we're referencing later08:48
SiavashSardarinope i was wrong they stop nicely08:48
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Apply OSA global-requirements-pins during functional tests  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/76482408:50
SiavashSardarinoonedeadpunk I had an experience like that in keystone and uploaded a patch for it. but in nova I think the problem is the playbooks will never restart nova-api service08:52
noonedeadpunkSiavashSardari: yeah and we had a fix for that specific issue09:01
SiavashSardarifor nova?09:02
noonedeadpunkincluding nova09:02
noonedeadpunklet me search for the topic09:02
noonedeadpunkit was backported as well09:02
noonedeadpunkhttps://review.opendev.org/q/topic:%22restart-uwsgi%22+(status:open%20OR%20status:merged)09:03
SiavashSardariThanks,09:05
*** shyamb has joined #openstack-ansible09:07
*** pto has quit IRC09:07
SiavashSardarinoonedeadpunk I can't find any handler for 'Restart uwsgi services' in nova role09:10
noonedeadpunkbecause it's not in nova role :p09:10
*** pto has joined #openstack-ansible09:10
noonedeadpunkhttps://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/handlers/main.yml#L16-L2909:10
*** pto has quit IRC09:10
*** pto has joined #openstack-ansible09:11
SiavashSardariOops, didn't think of that '=D09:11
noonedeadpunkand we include role here https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tasks/main.yml#L183-L19109:12
SiavashSardarithank you, my role are a bit outdated.09:15
SiavashSardarijust a question. how often you think it is good to do a minor upgrade? I mean OSA wise09:17
noonedeadpunkI usually didn't do any :(09:19
noonedeadpunkunless smth is really needs fixing09:20
*** jbadiapa has joined #openstack-ansible09:24
jrosserSiavashSardari: you should probably do one right before you do a major upgrade09:28
jrosseras a minimum09:28
*** ygk_12345 has quit IRC09:30
SiavashSardarithank you guys. we have kind of mirrored repo of OSA on our gitlab and we tried to adopt upstream branching for that so we can take advantage of new commits of upstream and also scratch our own itches before we can contribute if possible.09:35
SiavashSardariso that's why I'd like to do minor upgrades as often as I could.09:37
jrosserSiavashSardari: if you need to carry your own patches for a while you should look at the file /etc/openstack_deploy/user-role-requirements.yml, that lets you override what comes in user-role-requirements09:37
jrosseryou can point to your own repos for that in a version controlled way09:38
jrosser*override what comes in ansible-role-requirements, sorry09:38
SiavashSardariwe use that for roles that we updated roles that we used before it gets to upstream.09:39
jrosserawesome09:39
jrosserif theres anything difficult with these hooks let us know, it's good to provide places you can plumb in your local repos09:40
SiavashSardariwe will certainly do, thanks for your support09:40
*** spatel has joined #openstack-ansible09:44
*** fanfi has quit IRC09:44
*** spatel has quit IRC09:48
MickyMan77jrosser: you last tips did not work, "openstack-ansible -e venv_rebuild=yes os-cinder-install.yml", samme error, "Error during database migration: \"Database schema file with version 140 doesn't exist.09:52
MickyMan77do you have any other idea/suggestions ?09:53
jrosserMickyMan77: can you show me your hash of cinder from this file https://github.com/openstack/openstack-ansible/blob/master/playbooks/defaults/repo_packages/openstack_services.yml#L6609:54
jrosserand can you check which hash/tag you have checked out in /opt/openstack-ansible on your deploy host09:56
MickyMan77the checkout file look like this, http://paste.openstack.org/show/800583/10:07
MickyMan77but what do you mean with, "show me your hash of cinder from this file"10:08
*** shyamb has quit IRC10:41
*** shyamb has joined #openstack-ansible10:57
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-galera_server master: Use mysql user instead of root  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/76444911:03
*** kleini has quit IRC11:04
*** kleini has joined #openstack-ansible11:07
*** waverider has joined #openstack-ansible11:09
jrosserMickyMan77: sorry been in a meeting11:12
jrosserMickyMan77: i wanted to know what version of the cinder service was being installed, thats what you put in your paste11:12
jrosseri still think that somehow you are not deploying the ussuri version of cinder11:13
jrosserbecasue on the stable/ussuri branch version 140 of the DB schema does not exist https://github.com/openstack/cinder/tree/stable/ussuri/cinder/db/sqlalchemy/migrate_repo/versions11:14
jrosserbut it is there from victoria and onward https://github.com/openstack/cinder/tree/stable/victoria/cinder/db/sqlalchemy/migrate_repo/versions11:14
MickyMan77is victoria stable ?11:14
jrosseropenstack-ansible has not made a release for victoria yet, but the openstack components like nova/cinder are released as stable for victoria11:15
jrosserMickyMan77: on the host that has failed, can you find the file /openstack/venvs/cinder-21.x.y/constraints.txt and find a line that looks like cinder==16.1.1.dev4611:19
MickyMan77# grep cinder /openstack/venvs/cinder-21.1.0/constraints.txt --> cinder==16.2.011:22
MickyMan77# grep cinder /openstack/venvs/cinder-21.2.0/constraints.txt --> cinder==16.2.1.dev311:22
MickyMan77 # grep cinder /openstack/venvs/cinder-21.1.0.dev119/constraints.txt --> cinder==16.1.0.dev27511:24
jrosserMickyMan77: i am running out of ideas really, i don't know enough about the db migrations to know why it is looking for version 140 when it should not be11:32
jrosserimho this is not really an openstack-ansible problem11:32
MickyMan77is there any timeline when openstack-ansible will have a stable version for deploying victoria?11:34
MickyMan77I did run this command in the "cinder-api-container", /openstack/venvs/cinder-21.2.0/bin/cinder-manage db sync11:40
MickyMan77Error during database migration: "Database schema file with version 140 doesn't exist."11:41
MickyMan77so if I solve that issue, the openstack-ansible should work.11:41
jrosseryes, that is the exact command that the playbook is running11:42
MickyMan77# /openstack/venvs/cinder-21.2.0/bin/cinder-manage db version11:42
MickyMan7714011:42
jrosserwould this be one of the original environments that you tested many different tags of openstack-ansible on?11:43
MickyMan77so I need to downgrade the db... if that is even possible...11:44
MickyMan77"Make a backup of your database. Cinder does not support downgrading of the database. Hence, in case of upgrade failure, restoring database from backup is the only choice"11:45
jrosseri have a ussuri cloud here and it is running db version 13611:46
jrosserand that is consistent with the schema numbering on the cinder ussuri branch11:46
MickyMan77can I use openstack-ansible just to deploy victoria version of os-cinder-install.yml ?11:50
jrosserif you update the sha in the file i had you look at earlier https://github.com/openstack/openstack-ansible/blob/master/playbooks/defaults/repo_packages/openstack_services.yml#L66 then you can update the version of cinder that is installed11:53
jrosseryou will also have to take care of ensuring that the victoria version of openstack constraints is used for the venv build11:53
*** shyam89 has joined #openstack-ansible11:54
*** shyam89 has quit IRC11:56
*** shyamb has quit IRC11:58
*** rfolco has joined #openstack-ansible12:06
*** mgariepy has quit IRC12:10
jrossertheres a bunch of patches for os_tempest which i think the tripleo folk would like reviews on13:05
openstackgerritMerged openstack/openstack-ansible-tests master: Apply OSA global-requirements-pins during functional tests  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/76482413:05
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_adjutant master: Updated from OpenStack Ansible Tests  https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/73672313:21
*** mgariepy has joined #openstack-ansible13:27
*** sshnaidm has quit IRC13:38
*** spatel has joined #openstack-ansible13:45
*** sshnaidm has joined #openstack-ansible13:46
*** sshnaidm has quit IRC13:48
*** spatel has quit IRC13:50
*** pcaruana has quit IRC14:03
*** admin0 has quit IRC14:03
*** spatel has joined #openstack-ansible14:05
spatelnoonedeadpunk: morning14:10
noonedeadpunko/14:11
spatelYesterday i was trying to deploy senlin on ussuri and found its still in master :(14:11
spatelso i did cherry pick patch and manually add senlin required files. and when trying to deploy i got this error http://paste.openstack.org/show/800588/14:12
noonedeadpunkI don't think we can now create ussuri branch for senlin14:12
noonedeadpunkbut I mean role won't change, so you can use working SHA from master and overwrite senlin sha itself from ussuri14:14
noonedeadpunkas eventually role can deploy senlin from U without any issues14:14
spatelI did deployed but i have noticed some changes in senlin roles and i got stuck here http://paste.openstack.org/show/800588/14:14
*** pcaruana has joined #openstack-ansible14:14
spatelwhere is this error coming from? "msg": "The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required."14:15
spatellook like python interpreter issue14:15
noonedeadpunksec14:15
spatelThis module is very different then other roles - /etc/ansible/roles/os_senlin/tasks/db_setup.yml14:16
noonedeadpunkyou should set `openstack_db_setup_host: "{{ groups['utility_all'][0] }}"` in user_variavbles14:16
noonedeadpunkit will be default afterwards14:16
spatelhmm14:16
noonedeadpunkor you can checkout os_senlin to fb8f6966e260fb5508f4290e31966a8b25b1f150 which does not include this change14:17
*** cshen has quit IRC14:18
spatelare we going to use openstack_db_setup_host: "{{ groups['utility_all'][0] }}  for final release or this is just hack for meantime?14:19
spatelalso i have noticed we are using community.mysql.mysql_db: module - https://opendev.org/openstack/openstack-ansible-os_senlin/src/branch/master/tasks/db_setup.yml#L3114:20
spatelis that something new we are going to use for other roles?14:20
jrosserspatel: thats how everything is for master/victoria because we have switched to the new ansible collections14:21
spateljrosser: oh!14:21
jrossersenlin hasnt been included in a release yet so the first point at which it will get a stable branch is stable/victoria14:22
spatelhow much work left for senlin?14:23
jrossernot very much really, you can see the outstanding patches here https://review.opendev.org/q/project:openstack/openstack-ansible-os_senlin14:25
spatelcool, let me know if need help for testing. i want to deploy that on production soon14:27
spatelnoonedeadpunk: Hey i have add "openstack_db_setup_host: "{{ groups['utility_all'][0] }}"" in user_variables and still seeing same error :(14:28
spatel"msg": "The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required."14:28
spateldid i miss something when i did cherry pick this patch?14:29
spatelor may be we are missing that module on utility container because i build utility container using stable/ussuri branch14:31
jrosserspatel: you still somehow have the delegation wrong for that task, from your paste "failed: [infra-lxb-1_senlin_container-f31c6b85 -> localhost]"14:32
jrosserit's delegating to localhost14:32
spatelits saying <infra-lxb-1_senlin_container-f31c6b85> Task is delegated to infra-lxb-1_utility_container-085107e1.14:33
spateli have added "openstack_db_setup_host: "{{ groups['utility_all'][0] }}"  in var so now its using utility14:34
jrosseralso openstack_db_setup_python_interpreter: "/openstack/venvs/utility-{{ openstack_release }}/bin/python"14:36
spatelthat fix my issue :)14:37
spatelso what was the deal?14:37
spateluse venvs python instead of default os one right?14:38
openstackgerritMerged openstack/openstack-ansible stable/train: Bump SHAs for stable/train  https://review.opendev.org/c/openstack/openstack-ansible/+/76459114:40
jrosserspatel: yes, the db setup used to be done on one of the galera nodes but thats been moved to the utility container14:44
jrosserand so now it has to be specified which host and which python to use to do that14:45
spatel+114:45
jrosserthose will be new vars in victoria https://opendev.org/openstack/openstack-ansible/blame/branch/master/playbooks/defaults/source_install.yml#L35-L3614:45
spateljrosser: that was the one i was missing in my cherry pick14:46
spatelwhen do you think first stable/victoria coming out?14:46
spatelfor stability i deployed ussuri on production.14:47
noonedeadpunkI'm really aimed to branch this week14:48
spatelwow! that would be awesome!14:48
*** admin0 has joined #openstack-ansible14:48
spatelhope upgrade would be easy also :)14:49
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant master: Define condition for the first play host one time  https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/76465614:49
ThiagoCMCI can't wait to upgrade Ussuri to Victoria! It's going to be my first OSA upgrade ever.  lol14:49
spateli have successfully deployed senlin, now i am feeling good14:49
spatelnoonedeadpunk: i did review those changes also and they all looks good https://review.opendev.org/q/project:openstack/openstack-ansible-os_senlin14:50
spatelThiagoCMC: same here14:50
spatelQuestion: I deployed my cloud last week but forgot to change region: RegionOne :(14:51
spatelhow hard its to change it now ?14:51
ThiagoCMCSenlin is awesome! I wanna try it.14:52
spateloh yes.. i want to test it for my SRIOV instances. (its working for normal instances)14:53
ThiagoCMCIs your branch for master/victoria or ussuri ?14:54
noonedeadpunkspatel: I think pretty much. you need to re-run all roles and drop RegionOne endpoints afterwards14:54
noonedeadpunkspatel: yeah, awesome, I was a bit worried about https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/764661 and wanted to ping you but you wasn't online at a time14:55
SiavashSardariThiagoCMC spatel major upgrade works fine. keep an eye on Ceph though for me something went wrong with ceph, even though Ceph upgrade worked like a charm, after a week an horror story happend and we lost part of our data14:55
noonedeadpunko_O14:55
noonedeadpunkthat's frightening... what has happened there?14:55
ThiagoCMCWow, that's bad Siavash!14:55
SiavashSardariI'm not sure what went wrong but after upgrade our io_wait raised. we didn't think it was a big deal but on a pg_num change three osds (each on a different host) got corrupted somehow14:58
SiavashSardariwe extract moset of pgs but 3 pgs was corrupted14:59
SiavashSardarito this day that wakes me up at nights. we didn't find the root cause14:59
noonedeadpunkwere you using raid cards for doing smth like raid0 with each drive in it? (in case there was no support for jbod it's common practise)14:59
SiavashSardariwe even tried to reproduce is but no luck there too15:00
SiavashSardarino osds was raw disks15:01
noonedeadpunkor drives are directly connected to sas/sata/pci/whatever?15:01
noonedeadpunkI see.15:01
*** pcaruana has quit IRC15:01
*** pcaruana has joined #openstack-ansible15:02
noonedeadpunkjust was seing huge latency and util increase for SSD drives connected through raid card after L->N update15:02
noonedeadpunkwhich was a result of ceph algo changes for bluestore, where it become reflecting on how osd are reported by kernel - as rotational drives or not15:03
SiavashSardariI found that on mailing lists I think. but that wasn't our case15:04
SiavashSardariwe upgraded from L -> O15:04
SiavashSardariand basically everything was the same except the versions.15:05
*** cshen has joined #openstack-ansible15:05
ThiagoCMCHave you guys (OSA) ever considered a HA MySQL master/master instead of the Galera Cluster? I have it done for places where I only have 2 servers and it works great! And it's a lot simpler than Galera!15:13
*** SiavashSardari has quit IRC15:13
openstackgerritMerged openstack/openstack-ansible-openstack_openrc stable/ussuri: Adding support of system scoped openrc and clouds.yaml  https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/76350815:14
*** nurdie has joined #openstack-ansible15:15
ThiagoCMCI mean, I have HA MariaDB but not for OSA... I was using it for small things like Kea DHCP and other network services15:16
dmsimardbtw it is Ansible PR day in #ansible-community if you'd like to get attention on one of your PRs15:16
*** waverider has quit IRC15:16
admin0ThiagoCMC, how do you handle split brain ?15:20
spatelI have tried Galera Master/Master and it didn't work for my application (i think its debatable )15:21
noonedeadpunkdmsimard: uh, I should have written test for one of my PRs but I haven't so :(15:22
noonedeadpunkThiagoCMC: personally I haven't considered MySQL as even to download it you need oracle account.15:23
dmsimardif you need help or pointers it's a good opportunity to ask15:23
noonedeadpunkI had great ones, but had no time :(15:23
noonedeadpunkbut thatnks for letting know!15:23
noonedeadpunkhow frequently are they going to be?15:24
ThiagoCMCadmin0, That's a very good question!  The setup I've built has VIP and, while both can receive writes, the applications could only see the VIP. Then, the binary logs did the magic behind the scenes15:24
ThiagoCMCnoonedeadpunk, I actually used MariaDB lol  =P15:24
ThiagoCMCBut from Ubuntu repos15:24
*** pto has quit IRC15:25
*** pto_ has joined #openstack-ansible15:25
spatelI tried Master/Master with Zabbix and got some lock issue. (then i changed zabbix to single MySQL db and all good)15:26
noonedeadpunkisn't galera is master-mastrer implementation for mariadb? https://mariadb.com/kb/en/what-is-mariadb-galera-cluster/15:26
admin0ThiagoCMC, with 2 nodes only, split brain is a possibility . and having faced it before, i don't like it :)15:27
noonedeadpunkyeah, there's no pure master-master eventually15:27
admin0either single with a good backup, or 315:27
spatelnoonedeadpunk: does OSA has option to point write to A node and Read to B node?15:27
noonedeadpunkit might be great to use maxscale for balancing instead of haproxy, but yeah15:27
admin0with write going to 1 only, you have making that particular disk prone to failure sooner15:28
admin0a load balanced approach .. should be a load balanced approach :)15:28
jrosserThiagoCMC: we had bad-times with 2 galera instances15:28
jrosser*mariadb15:28
ThiagoCMCInteresting points! I'll share with you guys later on. I don't think I used Galera. It was last year...15:28
noonedeadpunkadmin0: anyway galera proxies all writes to master iirc15:29
jrosserif you have any kind of issue, like restart one node or some network trouble then a master cant be elected with just two15:29
spatelYes osa/haproxy point all request to only 1 node (not round-robin)15:29
admin0btw, i had a question .. how does ceph handle failure .. suppose due to power failure, all my 3 nodes are down .. is there a specific way to bring it up ?15:29
ThiagoCMCpower it on  =)15:30
admin0or is it same as mariadb where i power 1, then 2 and wait for it to sync and then third one15:30
ThiagoCMCMons first, I believe15:30
noonedeadpunkno, it's better to power on at the same time as for me15:30
noonedeadpunkor mons first, set noout,norebalance and osd then15:30
noonedeadpunk(was just having outages where mons were on osd nodes - super bad solution)15:31
spatelall mon down can create disaster also15:32
jrossernoonedeadpunk: we just are looking at how to implement https://securitytxt.org/15:34
jrossernot very obvious what to do though - hack in haproxy to serve a static file, or some way to use keystone or horizon instead.....15:35
jrosserall kind of ugly15:35
MickyMan77jrosser: I have change the file, openstack_services.yml but what about ansible-role-requirements.yml.      http://paste.openstack.org/show/800590/15:35
noonedeadpunkI think keystone is the closest option in terms that it has just web server that can be configured for a separate path for this specific file15:36
*** pcaruana has quit IRC15:36
noonedeadpunkbut yeah...15:36
noonedeadpunknot clean for sure15:37
jrosseryes i was thinking acl in haproxy to catch that specific path then use the keystone backend15:37
jrossera bit like we do for LE15:37
noonedeadpunkyeah thought about the same actually15:41
*** pcaruana has joined #openstack-ansible15:49
openstackgerritMerged openstack/openstack-ansible-os_ceilometer stable/ussuri: Add event_pipeline support  https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/76274315:50
openstackgerritMerged openstack/openstack-ansible-os_placement master: Reduce number of processes on small systems  https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/76465815:51
*** irclogbot_1 has quit IRC15:53
*** irclogbot_2 has joined #openstack-ansible15:55
openstackgerritMerged openstack/openstack-ansible-os_swift master: Reduce number of processes on small systems  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/76465216:00
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests stable/ussuri: Apply OSA global-requirements-pins during functional tests  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/76489516:04
*** sshnaidm has joined #openstack-ansible16:05
openstackgerritMerged openstack/openstack-ansible-os_trove master: Reduce number of processes on small systems  https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/76465316:08
noonedeadpunk#startmeeting openstack_ansible_meeting16:09
openstackMeeting started Tue Dec  1 16:09:05 2020 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.16:09
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:09
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:09
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:09
noonedeadpunk#topic bug triage16:09
*** openstack changes topic to "bug triage (Meeting topic: openstack_ansible_meeting)"16:09
noonedeadpunksorry for the delay with meeting :(16:09
jrossero/ hello16:09
openstackgerritMerged openstack/openstack-ansible-os_senlin master: Reduce number of processes on small systems  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/76465016:10
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests stable/train: Apply OSA global-requirements-pins during functional tests  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/76497616:10
noonedeadpunkI haven't done anything from the last week regarding fixing bugs unfortunatelly - had really a lot of internal things16:10
openstackgerritMerged openstack/openstack-ansible-os_senlin master: Define condition for the first play host one time  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/76465916:10
noonedeadpunkI think the only new bug was https://bugs.launchpad.net/openstack-ansible/+bug/1906108 and there's already a fix for it16:10
openstackLaunchpad bug 1906108 in openstack-ansible "os-keystone-install with keystone-config and keystone-install tags failure" [Undecided,New] - Assigned to Siavash Sardari (siavash.sardari)16:10
openstackgerritMerged openstack/openstack-ansible-os_adjutant master: Trigger uwsgi restart  https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/76465516:11
*** irclogbot_2 has quit IRC16:11
noonedeadpunkwhich is https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/76455216:11
*** irclogbot_2 has joined #openstack-ansible16:11
noonedeadpunkit's really interesting why launchpad does not get updated after gerrit upgrade, but I think it's not the only issue here...16:12
jrosseri think the new gerrit broke the integration with LP and storyboard16:12
*** gshippey has joined #openstack-ansible16:13
* jrosser waves to gshippey 16:13
gshippey👋16:14
noonedeadpunkwell, I think we don't have much discussions regarding bugs this week16:14
noonedeadpunk#topic office hours16:14
*** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)"16:14
noonedeadpunkso regarding overall plans. thanks everyone for doing lots of reviews today16:15
*** macz_ has joined #openstack-ansible16:15
jrosserseems like a few gate failures, need some rechecks16:16
jrosserand i think that the functional test jobs should be better now16:16
MickyMan77jrosser: did you see my msg..   http://paste.openstack.org/show/800590/16:16
noonedeadpunkYeah functional looks goot atm16:16
noonedeadpunkI'm going to do branching this week, hopefuly tomorrow once rechecks will pass16:16
noonedeadpunkand was thinking about doing stable release in 2 weeks after that16:17
noonedeadpunkI don't think we have much to merge, except fixing out octavia16:17
noonedeadpunk(for debian)16:17
*** macz_ has quit IRC16:17
jrosseris that something understood or needs more investigating?16:17
*** macz_ has joined #openstack-ansible16:18
noonedeadpunkwell I haven't spawn aio yet16:18
noonedeadpunkI hope I will have time on my hands during this week for investigation. I'm worried that this might be upstream thing, as before the latest bump things were good16:18
noonedeadpunkand debian uses py3.7 which is weird overall16:19
noonedeadpunkgood in terms that it was failing for uprade jobs but passing for debian ones16:19
noonedeadpunkbtw have you seen https://lists.ceph.io/hyperkitty/list/ceph-announce@ceph.io/thread/Y267KT2TQJ3VT7UQCC2ES4ZZV2OTL46P/ ?16:20
jrosseromg16:21
noonedeadpunkI'm wondering if that's what Saivash was talking about half an hour ago...16:21
jrosserso branch will mean victoria rc this week?16:24
noonedeadpunkwell, I think I don't have much more things to discuss, but love to hear if anything needs attention from your prespective?16:24
noonedeadpunkyep, exactly16:25
jrosserok good16:25
jrosserwell it is a bit unfortunate to see how much trouble folk have with magnum16:25
noonedeadpunkoh, yes...16:25
jrosserthough our experience was the same and i'm not totally sure what can be done16:25
* noonedeadpunk has the same16:26
jrosserthe AIO is too small to start a cluster really16:26
noonedeadpunkwell, we can probably do job with 2 vms....16:26
noonedeadpunksecond one as computing node...16:26
noonedeadpunkbut again - we don't run magnum jobs with bumps...16:28
noonedeadpunkso we won't know that we just broke it16:28
noonedeadpunkand not sure we can really add task for each service during bumps...16:29
jrosserno, that would be very difficult16:30
jrosseris multinode a lot of work?16:30
noonedeadpunknot sure.. I think not much.16:32
noonedeadpunkbut it seems we do test cluster creation....16:32
noonedeadpunkhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_483/410681/3/gate/openstack-ansible-deploy-aio_metal-ubuntu-focal/4836457/logs/openstack/aio1-utility/stestr_results.html16:33
noonedeadpunkhttps://github.com/openstack/magnum-tempest-plugin/blob/master/magnum_tempest_plugin/tests/api/v1/test_cluster.py16:33
noonedeadpunkmaybe issue is there somewhere in networking16:33
noonedeadpunksince in aio we really do everything through same mgmt?16:34
noonedeadpunkah, well16:36
noonedeadpunkthese are all negative tests16:36
noonedeadpunkand we've blacklisted the only positive one which is test_create_list_sign_delete_clusters16:37
jrosserhmm16:37
jrosserwe really need a contributor for magnum stuff16:38
jrosserit is a project of it's own somehow16:38
noonedeadpunkwell we're using it here but tend not to touch it at the moment16:38
noonedeadpunkI hope I will get hands on it in some future, but now in the nearest :(16:39
noonedeadpunk*but not in the nearest16:39
jrosserok - i don't think i have anything much more16:41
*** mgariepy has quit IRC16:41
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Remove magnum tempest blacklists  https://review.opendev.org/c/openstack/openstack-ansible/+/76498616:42
noonedeadpunkHowever w're about to implement troove support, so it will get better docs at least (I hope)16:42
jrosserthere are a couple of small patches toward zun from andrewbonney which could do with review to keep that moving16:42
jrosserwe are close to some basic tempest tests passing for that and the kuryr people are helpful with fixing the networking16:43
noonedeadpunkyeah, I realizied I even submited some bug to them one day and andrewbonney had fixed it lately :)16:43
openstackgerritMerged openstack/openstack-ansible stable/train: Remove git repo haproxy backend  https://review.opendev.org/c/openstack/openstack-ansible/+/76220916:45
openstackgerritMerged openstack/openstack-ansible stable/stein: Switch to stable/stein for EM  https://review.opendev.org/c/openstack/openstack-ansible/+/76193716:45
jrosserah yes, merging the magnum docs is good16:46
jrosseri think we should just continue to improve those as people get their stuff working16:46
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: [DNM] Test CI  https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/76498716:49
openstackgerritMerged openstack/openstack-ansible-os_magnum master: Add docs for suggested cluster template and debugging hints  https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/76462716:49
noonedeadpunkYeah, it seems by far the best approach atm...16:49
noonedeadpunkwell, maybe tests will pass now for magnum with without blacklists16:49
noonedeadpunkwhich will kind of test clusters at least somehow...16:50
jrosserdmsimard: have you seen this before? http://paste.openstack.org/show/800592/16:50
noonedeadpunkas otherwise even having several vms for testing we would need to write some tests as well...16:50
* dmsimard looks16:50
dmsimardjrosser: rarely, but yes16:51
dmsimardwhen I've seen it happen it was due to a prior playbook crashed or was interrupted16:51
dmsimardrunning another playbook after that seemed to be fine16:52
dmsimardhaven't seen it reproduce with mysql16:52
jrosseri just grabbed that off here https://zuul.openstack.org/stream/8e41cd8233c249a798ccafcf8e69ad46?logfile=console.log16:52
noonedeadpunkoh, and execution stuck16:52
jrosseras the console does seem to have jammed up and i'm not sure if it's related16:52
jrosseri took a look at that becasue i've seen more timeouts than usual16:53
noonedeadpunkuh, and need to find out where all these deprecations come from...16:55
jrosserheres another from earlier the same https://zuul.opendev.org/t/openstack/build/51374ae7438040d3bc00078b77c03174/log/job-output.txt#842816:56
noonedeadpunklast one is even more weird16:59
noonedeadpunkit has timeouted in 1.5 hours?17:00
noonedeadpunkjob has started at 12:30 and timeouted at 14:0017:00
noonedeadpunkwhile we should have 3h timeout....17:00
noonedeadpunk#endmeeting17:01
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/osa-review-board-v3"17:01
openstackMeeting ended Tue Dec  1 17:01:10 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-12-01-16.09.html17:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-12-01-16.09.txt17:01
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-12-01-16.09.log.html17:01
*** rpittau is now known as rpittau|afk17:01
openstackgerritMerged openstack/openstack-ansible-os_adjutant master: Updated from OpenStack Ansible Tests  https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/73672317:04
openstackgerritMerged openstack/openstack-ansible-os_nova master: Reduce number of processes on small systems  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/76464617:10
*** mgariepy has joined #openstack-ansible17:19
*** gyee has joined #openstack-ansible17:23
spatelnoonedeadpunk: i deployed senlin and everything works except health manager and what i found is, it missing17:42
-spatel- [root@infra-lxb-1-senlin-container-f31c6b85 /]# systemctl status senlin_health-manager17:42
-spatel- Unit senlin_health-manager.service could not be found.17:42
noonedeadpunkand service name should be senlin-health-manager17:46
spatelyes17:47
spatel /openstack/venvs/senlin-21.1.1.dev4/bin/senlin-health-manager17:47
spatelhttp://paste.openstack.org/show/800594/17:47
noonedeadpunk`systemctl status senlin-health-manager` not `systemctl status senlin_health-manager`17:47
noonedeadpunkaha....17:48
spateldamn!17:48
spatelwhere is that typo?17:48
noonedeadpunkwell according to output of systemctl it seems that it's really missing. I got typo from your first paste17:50
noonedeadpunkok and group name is that   https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/env.d/senlin.yml#L1117:51
spatelservice name should be senlin-health-manager17:51
spatelgroup name is correct17:51
ThiagoCMCspatel, just curious, do you think that Senlin will be included in "stable/victoria"?17:52
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Fix senlin health manager group name  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/76498917:52
noonedeadpunkspatel: can you check this out? ^17:52
noonedeadpunkThiagoCMC: it will17:52
spatelnoonedeadpunk: i have group: senlin_health-manager17:53
noonedeadpunkhm, and how does your env.d look like?17:53
ThiagoCMCAmazing!!!!17:53
noonedeadpunkregarding senlin17:54
spatelThiagoCMC: i have created blog for senlin here, working example - https://satishdotpatel.github.io//openstack-senlin-autoscaling/17:54
ThiagoCMCspatel, thank you! This is so cool!17:54
ThiagoCMCI've used Senlin before, I really like it. But, was back in 2017 for testing and evaluation.17:55
spatelnoonedeadpunk: i have senlin_health_manager in env.d17:55
spatelthat could be my issue..17:56
noonedeadpunkso you should not have senlin_health-manager group name then17:56
spatellet me edit my role and fix  group: senlin_health-manager17:56
noonedeadpunkas upstream we have senlin_health_manager in env.d as well17:56
noonedeadpunkso i think the issue is in role17:56
noonedeadpunkyeah17:56
spatelyes.. let me check quickly and if need patch it17:57
noonedeadpunkhttps://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/764989/1/defaults/main.yml17:57
spatelyou are awesome!17:57
noonedeadpunkthat's what I asked to check out :p17:58
spatelDidn't realized that part because its hard to catch - vs _18:00
spatelre-running playbook18:01
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Adding tags to federated openid support using auth_mod_openidc  https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/76455218:02
spatelnoonedeadpunk: its back now senlin-health-manager.service  thank you :)18:02
noonedeadpunkdon't forget to vote on patch :p18:03
spateldone!18:04
*** cshen has quit IRC18:08
*** d34dh0r53 has quit IRC18:13
spatelnoonedeadpunk: i think all senlin bugs has been fixed if you ask me..18:20
spateleverything looks good so far, i didn't find any issue.18:20
*** cshen has joined #openstack-ansible18:21
noonedeadpunknice!18:23
ThiagoCMCQuick question! I have "debug: false" but, my logs shows things like: `neutron-keepalived-state-change: 2020-12-01 13:23:54.033 9979 DEBUG neutron.agent.l3.keepalived_state_change` - Why?18:24
*** cshen has quit IRC18:33
*** cshen has joined #openstack-ansible18:34
*** andrewbonney has quit IRC18:40
noonedeadpunkI think this wirth adressing to neutron folks...18:42
noonedeadpunk*worth18:42
ThiagoCMCok, thanks  =)18:46
*** d34dh0r53 has joined #openstack-ansible18:54
*** sshnaidm has quit IRC19:15
*** sshnaidm has joined #openstack-ansible19:18
*** cshen has quit IRC19:30
*** cshen has joined #openstack-ansible19:35
*** luksky has quit IRC19:43
*** luksky has joined #openstack-ansible19:43
openstackgerritMerged openstack/openstack-ansible-os_nova stable/stein: Enable notifications when Designate is enabled  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/75841319:49
*** klamath_atx has joined #openstack-ansible19:54
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Rename nova_metadata_* variables  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/76014919:55
*** spatel has quit IRC20:02
openstackgerritMerged openstack/openstack-ansible-os_tempest stable/ussuri: Switch tripleo jobs to content provider  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/76101920:14
openstackgerritMerged openstack/openstack-ansible-os_panko master: Reduce number of processes on small systems  https://review.opendev.org/c/openstack/openstack-ansible-os_panko/+/76464820:18
*** pcaruana has quit IRC20:27
*** cshen has quit IRC20:55
ThiagoCMCThere is a first time for everything! Right?  lol21:02
ThiagoCMCI'm running openstack-ansible a second time and I'm seeing the following TASK: "os_nova : Grant access to the database for the service" failing with:21:02
ThiagoCMCfailed: [vosctrl-1_nova_api_container-1b9485de -> 10.232.199.74] (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}21:03
ThiagoCMCfatal: [vosctrl-1_nova_api_container-1b9485de]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}21:03
ThiagoCMCAny clue?21:03
*** rfolco is now known as rfolcOUT21:06
*** spatel has joined #openstack-ansible21:12
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_neutron stable/train: Add neutron_keepalived_no_track variable  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/76333721:48
*** rfolcOUT has quit IRC21:48
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_neutron stable/ussuri: Add neutron_keepalived_no_track variable  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/76333621:49
*** cshen has joined #openstack-ansible21:59
ThiagoCMCHow can I "--limit" the playbook run (os-nova-install.yml, for example) to go only against the "compute nodes" host group?22:03
*** gyee has quit IRC22:04
*** gyee has joined #openstack-ansible22:05
openstackgerritMerged openstack/openstack-ansible-os_senlin master: Simplify service creation  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/76466122:15
spotzThiagoCMC: I kinda remember seeing something about nova DB credentials. Check the discuss list?22:25
ThiagoCMCspotz, ok, cool... I'll google about it  =)22:36
spotzThiagoCMC: NP:)22:36
openstackgerritMerged openstack/openstack-ansible master: Cover nova metadata with SSL  https://review.opendev.org/c/openstack/openstack-ansible/+/76016522:40
*** luksky has quit IRC22:42
*** cshen has quit IRC22:46
*** jbadiapa has quit IRC22:53
*** luksky has joined #openstack-ansible22:56
*** nurdie has quit IRC23:05
*** spatel has quit IRC23:08
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Reduce number of processes on small systems  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/76464423:13
*** nurdie has joined #openstack-ansible23:21
*** nurdie has quit IRC23:25
*** luksky has quit IRC23:26
*** klamath_atx has quit IRC23:39
*** cshen has joined #openstack-ansible23:42
*** klamath_atx has joined #openstack-ansible23:47
*** cshen has quit IRC23:47
*** mcarden has joined #openstack-ansible23:54
*** tosky has quit IRC23:57
openstackgerritMerged openstack/openstack-ansible-tests stable/train: Apply OSA global-requirements-pins during functional tests  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/76497623:57
openstackgerritMerged openstack/openstack-ansible-tests stable/ussuri: Apply OSA global-requirements-pins during functional tests  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/76489523:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!