Tuesday, 2020-10-06

*** cshen has joined #openstack-ansible00:25
*** cshen has quit IRC00:30
*** ChiTo has quit IRC00:43
*** MickyMan77 has joined #openstack-ansible01:03
*** MickyMan77 has quit IRC01:11
*** gyee has quit IRC01:25
*** MickyMan77 has joined #openstack-ansible01:42
*** MickyMan77 has quit IRC01:50
*** maharg101 has joined #openstack-ansible01:53
openstackgerritMerged openstack/openstack-ansible stable/stein: Bump SHAs for stable/stein  https://review.opendev.org/75597601:54
*** maharg101 has quit IRC01:58
*** MickyMan77 has joined #openstack-ansible02:17
*** MickyMan77 has quit IRC02:25
*** cshen has joined #openstack-ansible02:26
*** cshen has quit IRC02:30
*** MickyMan77 has joined #openstack-ansible03:02
*** MickyMan77 has quit IRC03:10
*** recyclehero has quit IRC03:14
*** MickyMan77 has joined #openstack-ansible03:15
*** recyclehero has joined #openstack-ansible03:21
*** shyamb has joined #openstack-ansible03:30
*** shyamb has quit IRC03:40
*** shyamb has joined #openstack-ansible03:40
*** shyamb has quit IRC03:52
*** maharg101 has joined #openstack-ansible03:54
*** maharg101 has quit IRC03:58
*** yolanda has quit IRC04:07
*** MickyMan77 has quit IRC04:19
*** cshen has joined #openstack-ansible04:26
*** cshen has quit IRC04:30
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-ansible04:33
*** mmethot_ has quit IRC04:54
*** mmethot_ has joined #openstack-ansible04:55
*** shyamb has joined #openstack-ansible05:05
*** MickyMan77 has joined #openstack-ansible05:16
*** MickyMan77 has quit IRC05:25
*** cshen has joined #openstack-ansible05:32
*** shyam89 has joined #openstack-ansible05:36
*** cshen has quit IRC05:37
*** shyamb has quit IRC05:38
*** viks____ has joined #openstack-ansible05:43
*** miloa has joined #openstack-ansible05:48
*** maharg101 has joined #openstack-ansible05:55
*** MickyMan77 has joined #openstack-ansible05:57
*** maharg101 has quit IRC06:00
*** cloudnull2 has joined #openstack-ansible06:03
*** fresta_ has joined #openstack-ansible06:04
*** gouthamr has quit IRC06:05
*** fyx has quit IRC06:05
*** alanmeadows has quit IRC06:05
*** weshay|ruck has quit IRC06:05
*** fresta has quit IRC06:05
*** cloudnull has quit IRC06:05
*** openstackgerrit has quit IRC06:05
*** cloudnull2 is now known as cloudnull06:05
*** gouthamr has joined #openstack-ansible06:05
*** MickyMan77 has quit IRC06:05
*** alanmeadows has joined #openstack-ansible06:05
*** fyx has joined #openstack-ansible06:05
*** weshay has joined #openstack-ansible06:06
*** macz_ has joined #openstack-ansible06:13
*** macz_ has quit IRC06:18
*** MickyMan77 has joined #openstack-ansible06:36
*** MickyMan77 has quit IRC06:45
jrossermorning07:00
*** pto has joined #openstack-ansible07:09
*** spatel has joined #openstack-ansible07:15
*** shyam89 has quit IRC07:18
*** shyamb has joined #openstack-ansible07:18
*** spatel has quit IRC07:20
*** cshen has joined #openstack-ansible07:20
*** andrewbonney has joined #openstack-ansible07:21
noonedeadpunkmornings07:28
*** maharg101 has joined #openstack-ansible07:29
jrossergood morning07:30
* jrosser tries to unblock the haproxy/metal patch, not sure quite whats happening with the merge conflict07:31
noonedeadpunkoh, linters have merged, finally07:33
* noonedeadpunk catching in07:34
jrosserseems like the gerritbot has gone away07:34
*** pto has quit IRC07:38
noonedeadpunkit really is07:41
*** tosky has joined #openstack-ansible07:44
*** rpittau|afk is now known as rpittau07:45
*** pto has joined #openstack-ansible07:53
masterpeMorning, I try to upgrade from Stein to Train, with that do that by running the playbooks from osa 20.1.6. With that I need to upgrade ceph from Mimic to Nautilus.07:57
masterpeThis step is not described in https://docs.openstack.org/openstack-ansible/train/admin/upgrades/major-upgrades.html, but I figured out to upgrade the mons I need to use playbooks/ceph-install.yml with the -e 'rolling_update=true' option. After that I can do an apt-get upgrade on the osd nodes.08:01
masterpeIf I after that run just run playbooks/ceph-install.yml I get the following error ansible error:08:02
masterpeceph-mon : fetch ceph initial keys08:02
masterpe2020-10-06 07:52:13,455 p=15811 u=root |  fatal: [ceph02_ceph-mon_container-917cd0ee]: FAILED! => {"changed": false, "msg": "initial keyring does not contain keys: client.bootstrap-rbd-mirror", "rc": 1}08:02
ptoDuring installation of os-cinder-install.yml, config_template breaks when it are to generate cinder.conf: n exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'AnsibleUnicode' object has no attribute 'items'. Any suggestion whats missing/wrong?08:07
masterpepto: what do you see when you run it with the -vvv option?08:08
ptoOutput: https://pastebin.com/sKRcqfdg08:10
ptoIt appears to me that some variable is undefined in the jinja2 template08:10
jrosserwhich branch is this?08:10
*** shyamb has quit IRC08:18
ptoqueens 17.1.1208:19
ptoLatest "stable"08:22
*** pto has quit IRC08:30
*** openstackgerrit has joined #openstack-ansible08:34
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/rocky: Remove periodic CI jobs for rocky branch  https://review.opendev.org/75622708:34
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: Remove support for Centos-7  https://review.opendev.org/74212008:37
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: Remove support for LXC2 configuration keys  https://review.opendev.org/74212108:37
*** danpawlik has quit IRC08:38
*** danpawlik has joined #openstack-ansible08:39
jrossernoonedeadpunk: i am trying to clean up the periodic CI jobs, do you know how this works? https://zuul.openstack.org/builds?project=openstack%2Fopenstack-ansible-os_cinder&pipeline=periodic08:45
jrosserwe are running periodic jobs for the integrated repo and all the roles, just for opensuse on stable/train08:45
jrosserthe integrated repo i think i see how that is defined https://review.opendev.org/#/c/756221/1/zuul.d/project-templates.yaml08:46
jrosserbut really a bit stuck on what is making the role jobs run08:46
noonedeadpunkIt's cron kind of https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L186-L19808:51
jrosserhmm, right08:52
jrosserthere must be some config too08:52
noonedeadpunkah I got the question08:54
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: Explicitly use rabbitmq collection  https://review.opendev.org/75543908:54
noonedeadpunkwait, pipeline itself makes jobs run08:57
openstackgerritMerged openstack/openstack-ansible-galera_server master: Explicitly use mysql collection  https://review.opendev.org/75559408:57
openstackgerritMerged openstack/openstack-ansible-os_nova master: Make possible to set nova_libvirt_images_rbd_pool to empty string  https://review.opendev.org/75414708:57
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Require the use of community.rabbitmq ansible collection  https://review.opendev.org/75465708:58
noonedeadpunkso like setting job for periodic is enough to get it run on daily basis08:58
*** jbadiapa has joined #openstack-ansible09:02
jrosseroh right maybe the role jobs are using the periodic job defined in the project tempalate in the integrated repo09:02
noonedeadpunkyep09:02
*** shyamb has joined #openstack-ansible09:09
*** miloa has quit IRC09:10
openstackgerritMerged openstack/openstack-ansible-os_blazar master: Define condition for the first play host one time  https://review.opendev.org/75125409:28
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Fix linter errors  https://review.opendev.org/75569709:36
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add configuration example for Ceph configuration from files.  https://review.opendev.org/75605109:37
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Allow to proceed with role if ceph_conf_file is set  https://review.opendev.org/75582009:37
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: Add centos-8/focal/bionic zuul jobs for the different storage backends  https://review.opendev.org/75599609:40
openstackgerritJonathan Rosser proposed openstack/ansible-role-systemd_networkd master: Fix linter errors  https://review.opendev.org/75623409:43
snadgeremember i had that problem where neutron-backend would fall over every minute or so?09:43
jrosseryes!09:43
snadgei turned on all of the compute nodes.. one at a time, and enabled them individually, spaced out, excluding one which i thought might be suspicious09:43
snadgeand now its all just working and the problem has gone away09:44
snadgein short the problem was between the controller and the neutron container.. the br-mgmt network just went awol for a minute or so09:44
snadgeduring this time, you could ping it from other hosts on that network, and you could connect to the port within the container itself09:44
snadgeso it was some kind of weird lxc bridge issue that I never was able to pinpoint09:45
snadgeim going to call that a win for now ;)09:45
snadgeonce they migrate to that production cluster.. it will free up resources that I can do more testing with09:46
jrosserwrap the node in hazard tape and move on :)09:47
snadgeim hoping to move to centos 8 and ussuri is it?09:47
snadgeits centos 7 and train for the moment09:47
jrosserussuri is the release for OSA that supports both09:48
snadgei guess by the time i move to that.. it will be 6 months away09:48
snadgeso it could be victoria09:48
snadgewhatever works and makes sense09:49
jrosserin victoria there is no centos-7 support, so depends if you need to upgrade something in-place or just start again09:49
recycleherohey guys, I have finally deployed.09:52
recycleheroone thing that I noticed is that the performance and resource usage is much higher comparing to my manual installaion of queens09:53
recycleherosecond thing is when I first logined to horizon I got 504 error timeouts after two more tries I was in09:53
recycleherothen in admin/compute of horizon I see not hypervisor/compute host09:54
*** shyamb has quit IRC09:56
openstackgerritMerged openstack/openstack-ansible master: Remove python2 packages from bindep.txt  https://review.opendev.org/75498709:57
openstackgerritMerged openstack/openstack-ansible master: [doc] Adjust deployment docs  https://review.opendev.org/75534509:57
jrossernoonedeadpunk: do we have a good reason for this any more? https://github.com/openstack/openstack-ansible-tests/blob/master/run_tests_common.sh#L74-L7810:11
jrosseri think that for focal we are installing virtualenv from pypi with that ^ on the host10:11
jrosserthen openstack-hosts comes along and instally python3-virtualenv on the top10:12
jrosserand it's broken10:12
*** yolanda has joined #openstack-ansible10:13
*** sum12 has quit IRC10:15
*** pto has joined #openstack-ansible10:16
ptoSorry. Lost my internet10:16
*** shyamb has joined #openstack-ansible10:16
noonedeadpunkjrosser: I think we can install pip from packages indeed there10:23
jrosseryes i think so too10:23
jrosserbtw i think i just broke a *lot* of stuff /o\10:24
noonedeadpunkrly?10:24
jrosserlxc config keys trouble in the integrated repo10:24
jrosserwe have legacy lxc2 keys in group vars10:24
noonedeadpunkuh, it's not lot of things)10:25
noonedeadpunkin terms that it's smth easy to fix, but will need tons of rechecks indeed10:25
jrosseryeah indeed10:26
* jrosser fixes10:26
noonedeadpunkrecyclehero: in terms of resource usage we probably use too many processes by default for each service, but it's easily tunable10:27
* jrosser has a patch for that10:28
noonedeadpunkthat's true10:28
jrosserhttps://review.opendev.org/#/c/705680/10:28
noonedeadpunkand for compute part you shoukd check nova-compute logs10:28
*** miloa has joined #openstack-ansible10:28
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance master: Reduce number of processes on small systems  https://review.opendev.org/70568010:30
*** shyamb has quit IRC10:33
*** PrinzElvis has quit IRC10:33
*** fridtjof[m] has quit IRC10:33
*** csmart has quit IRC10:33
*** scanepa has quit IRC10:33
*** shyamb has joined #openstack-ansible10:34
*** PrinzElvis has joined #openstack-ansible10:34
*** fridtjof[m] has joined #openstack-ansible10:34
*** csmart has joined #openstack-ansible10:34
*** scanepa has joined #openstack-ansible10:34
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Convert lxc2 config keys to lxc3 format  https://review.opendev.org/75624410:37
snadgeluckily we have enough hardware where we can alternate between production and staging.. ie, promote staging to production10:38
snadgeand create a new staging out of old production.. so im very happy to move to centos 8 in the new staging environment10:38
snadgelooking forward to it even10:38
openstackgerritJonathan Rosser proposed openstack/ansible-role-python_venv_build master: Remove support for lxc2 config keys  https://review.opendev.org/75624710:39
snadgeim also looking forward to covid being beaten, so I can actually travel from australia to the UK, and learn a few things from openstack people in that part of the world10:40
snadgeim a dual citizen but currently its illegal to travel to and from australia except for under extreme circumstances10:40
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_glance master: Remove support for lxc2 config keys  https://review.opendev.org/75624810:41
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_mistral master: Remove support for lxc2 config keys  https://review.opendev.org/75624910:43
recycleheroguys how does this /etc/network/interfaces looks? http://paste.openstack.org/show/798738/10:44
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Remove support for lxc2 config keys  https://review.opendev.org/75625110:45
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_placement master: Remove support for lxc2 config keys  https://review.opendev.org/75625210:46
recycleherois there any preferred facility I can share files with this chanell, I am lost. I cant get it to work. I want to show u the openstack user config and interfaes to see whats wrong10:48
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Remove support for lxc2 config keys  https://review.opendev.org/75625510:49
recycleheroif my method works we could share the config for people with no vlans. labs want to use macvlan10:52
jrosserrecyclehero: it would be good to check that from the controller you can ping all the mgmt ip on your compute node10:55
jrosserand that the compute node can ping the infra host and also the containers10:55
jrosservalidate that all the expected paths that should work are actually working with macvlan10:55
jrosserpaste.openstack.org is the best place to put config to share10:56
recycleheroon the tests11:00
noonedeadpunkprobably worth sharing my setup with sr-iov for storage as well...11:06
*** mgariepy has quit IRC11:08
*** shyamb has quit IRC11:08
*** sum12 has joined #openstack-ansible11:10
jrossershould we revert this https://review.opendev.org/#/c/742121/ i wonder11:11
jrosserthe lxc2 config keys are in all sorts of repos and it might not be possible to merge them11:11
ptoThere are MANY bugs in the latest queens release. I would like to help fix them, if someone could help me get started?11:13
noonedeadpunkjrosser: I see in functional tests we use lxc2_container_config_list but I'm not sure it's valid variable11:14
jrosserah that gets used in the lines just below to form lxc_container_config_list with a conditional11:14
noonedeadpunkah indeed11:15
masterpepto: is this usefull there it is unmaintained?11:15
noonedeadpunkand it looks safe11:15
noonedeadpunkso just integrated repo and ops?11:15
jrosserhopefully i've found all of those and patched them out anyway just becase it's not needed any more11:15
jrosserpython_venv_build :?11:16
jrosser:(11:16
ptomasterpe: Queens: Extended Maintenance11:16
ptoAccording to the website11:16
openstackgerritMerged openstack/ansible-role-systemd_mount master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75419411:16
noonedeadpunkbut python_venv_build has it in functional tests only?11:16
noonedeadpunkwhich check lxc version themselves11:16
jrosserok so maybe actaully this is one should get things working again https://review.opendev.org/#/c/756244/11:18
masterpepto: I was looking in "Next Phase" collum.11:18
noonedeadpunkI think you missed that? https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/cinder_volume.yml#L2111:18
jrosserah11:18
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Convert lxc2 config keys to lxc3 format  https://review.opendev.org/75624411:19
openstackgerritMerged openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75554511:20
*** shyamb has joined #openstack-ansible11:22
openstackgerritMerged openstack/openstack-ansible-os_masakari master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75553811:40
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Fix octavia tempest tests  https://review.opendev.org/75573711:42
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Update integrated ansible-lint rules  https://review.opendev.org/75612111:42
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Remove "when" statement from vars_prompt  https://review.opendev.org/75582411:42
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Use nodepool epel mirror in CI for systemd-networkd package  https://review.opendev.org/75470611:43
*** CeeMac has quit IRC11:48
*** rfolco has joined #openstack-ansible11:59
*** shyamb has quit IRC12:01
jrosserpto: if you are interested in fixing the queens branch because it is important to you then it still exists for OSA12:09
jrosserpto: the queens branch is extended-maintainance https://docs.openstack.org/project-team-guide/stable-branches.html#extended-maintenance12:12
jrosserpto: and i would say that "reduced CI commitment" maybe underestimates the amount of work that will be required to get the CI jobs passing again12:12
ptojrosser: Well, my interest is to get a test deployment up, so i can do an upgrade to latest version12:13
jrosseryou may well also find yourself in a similar position for rocky, i made a patch for rocky today and the CI has completely bit-rot12:14
openstackgerritMerged openstack/openstack-ansible-os_trove master: Define condition for the first play host one time  https://review.opendev.org/75443112:16
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Switch integrated linters to focal  https://review.opendev.org/75575912:18
openstackgerritJonathan Rosser proposed openstack/ansible-role-python_venv_build master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75419312:19
*** mgariepy has joined #openstack-ansible12:22
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Use pip and virtualenv from host packages rather than get-pip.py  https://review.opendev.org/75627912:28
*** pto has quit IRC12:38
*** jamesdenton has quit IRC12:46
openstackgerritMerged openstack/ansible-role-systemd_mount master: Install required packages for NFS/CephFS mounts  https://review.opendev.org/75497812:46
*** jamesdenton has joined #openstack-ansible12:47
openstackgerritErik Berg proposed openstack/openstack-ansible-lxc_container_create master: Allow user to override UseRoutes in template  https://review.opendev.org/75608812:48
jrossernoonedeadpunk: that python log collection turns out to be tricky12:48
openstackgerritErik Berg proposed openstack/openstack-ansible-lxc_container_create master: Allow user to override UseDNS in template  https://review.opendev.org/75609612:48
jrosserit needs the lxc python module which the only sensible install is from a distro package12:48
noonedeadpunkoh:(12:49
noonedeadpunknot so good :(12:49
noonedeadpunkbtw, wont https://review.opendev.org/#/c/752059/ result in that we will get all journals directly on the host?12:50
jrossercanonical don't push to pypi12:50
noonedeadpunkas we actually do bind mount to the hosts, but shadowing it afterwards12:51
jrosservery good point12:51
jrosserperhaps this can be simpler12:51
jrosserif we didnt need the lxc module it can be moved to osa_toolkit and would be installed in the ansible-runtime venv12:51
jrosserthen no messing with host python things12:51
noonedeadpunkalso another thing, is that considering python script failed, it didn't influence on the execution time....12:52
noonedeadpunkso it might be either the same or even slower that current nasty bash?12:52
jrossercould be12:54
jrosserstill we have collcetion of /var/log/<service> for everyting as weill which is surely redundant for most12:54
jrosserwe can leave it, was nice experiment at the least :)12:56
noonedeadpunkEventually I like it but depends on the effort that needs to be spent to finish...12:57
noonedeadpunkand it will be slower - that would be dissapointment12:57
noonedeadpunk*if it be slower12:57
*** rh-jelabarre has joined #openstack-ansible13:00
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Fix linter errors  https://review.opendev.org/75557313:08
openstackgerritMerged openstack/openstack-ansible stable/ussuri: Run tempest install when tempest_run is False  https://review.opendev.org/75446513:10
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Remove support for lxc2 config keys  https://review.opendev.org/75625113:10
*** pcaruana has quit IRC13:11
*** sum12 has quit IRC13:12
*** carlosmss has joined #openstack-ansible13:14
*** cshen has quit IRC13:15
*** pto has joined #openstack-ansible13:17
ptojrosser: Is the upgrade path from queens -> rocky -> stein -> train or can I jump from queens -> train?13:17
jrosserif you are on ubuntu then rocky is the release which supports both xenial and bionic13:18
carlosmssHi guys, I started deployment of the Octavia as LBaaS. Someone use it? In the documentation shows it needs a flat network, but when playbook runs an error occurs because the octavia container requires a bridge type network.13:18
jrosserpto: so at some point you are needing an operating system upgrade as well as an openstack upgrade13:19
jrossercarlosmss: the documenation really just shows what happens for the AIO test/CI environment13:19
jrossercarlosmss: there is a nice blog post by an OSA user here which gives an example production deployment of ovtavia https://satishdotpatel.github.io//openstack-ansible-octavia/13:20
ptojrosser: I guess that needs to be done between rocky and stein.13:20
ptojrosser: Is it impossible / too hard - to roll up the software or will a fresh install and migration afterwards be better?13:21
jrosserpto: yes, you'd kind of pause at rocky and sort the OS out13:21
jrosserpto: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005089.html13:22
carlosmssRight jrosser, thanks for share it13:23
jrosserpto: there are two choices, you can totally re-install each node as you go, or you can do an in-place OS upgrade but there will certainly be in depth fixing up to do13:23
jrosserxenial -> bionic is LXC2 -> LXC3 just as one example of that13:24
ptoouch...13:24
ptoSo I guess the best approach would be to bump openstack sw to rocky. Take down an infra node, reinstall the node fresh with 18.04 and redeploy the infra node13:24
jrosserthere is an etherpad linked from that ML post https://etherpad.opendev.org/p/osa-rocky-bionic-upgrade13:25
ptoIf you were to roll up an old queens cluster - would you upgrade it or install a new cluster in parallel and migrate all workloads to it?13:25
jrosseralso more notes here https://review.opendev.org/#/c/710803/13:25
jrosserpersonally i've done upgrades, but i've never been more than 1 release behind13:26
jrosserwe have other users like snadge who run two environments and upgrade/migrate between them13:27
ptoThanks for the docs btw13:27
ptoI have managed to install queens on a small stage env, and i will try to bump the version to rocky, do the os upgrade, and proceed in majors13:28
ptoBut so far, its has been a challenge to get queens running13:28
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Bump SHAs for master  https://review.opendev.org/75597313:29
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Deprecate os_congress role  https://review.opendev.org/74252113:29
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Bump SHAs for master  https://review.opendev.org/75597313:29
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Use pip and virtualenv from host packages rather than get-pip.py  https://review.opendev.org/75627913:30
noonedeadpunkpto: yeah, it's not supported for a while, as well as rocky... eventually things might be challenging because of py2 being unsupported as well13:30
jrosseryou want to get to at least train ASAP13:31
jrosserthen you should have python3 everywhere (unless centos)13:31
ptoIm working 100% of my time to do the upgrade13:31
ptoI am considering building new nodes with train, and the migrate all instances to it. This will have a number of drawbacks (e.g new IPs for instances etc), but the rather complex upgrade can be skipped. Any suggestion on which part to take?13:33
noonedeadpunkeventually R->T upgrade should be pretty good - never was an issue for me at least13:34
noonedeadpunkCan't say it's really too complex13:35
ptoI have done a queens -> rocky a year+ a go. That went pretty smooth13:36
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Remove support for lxc2 config keys  https://review.opendev.org/75625513:43
ptoThank you all for clarifying the upgrade problem im facing13:45
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Fix linter errors  https://review.opendev.org/75569713:45
openstackgerritJonathan Rosser proposed openstack/ansible-role-systemd_networkd master: Fix linter errors  https://review.opendev.org/75623413:48
mgariepyhttps://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_04a/755508/1/check/openstack-ansible-deploy-aio_ovn_lxc-ubuntu-focal/04a6363/logs/ara-report/results/3073.html#stdout13:48
mgariepyanyone seen that before?13:48
jrossermgariepy: yes that is happening quite frequently on focal jobs13:49
jrosseri have no idea why13:49
*** pcaruana has joined #openstack-ansible13:51
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-nspawn_container_create master: Fix linter errors and add lint skip list  https://review.opendev.org/75532013:55
openstackgerritJonathan Rosser proposed openstack/ansible-role-systemd_networkd master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75419513:56
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-nspawn_container_create master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75415713:57
*** d34dh0r53 has joined #openstack-ansible13:58
*** cshen has joined #openstack-ansible14:01
mgariepyjrosser, https://zuul.opendev.org/t/openstack/build/04a63639459f4fe3860e18647221e090/log/logs/openstack/aio1_nova_api_container-1b015e6b/nova-scheduler.service.journal-21-29-18.log.txt#147014:03
mgariepyrace.14:03
ebbexjrosser: https://review.opendev.org/#/c/756244/2/inventory/group_vars/all_containers.yml why is only debian10 running unconfined here?14:03
mgariepyhost mapping fails at : 2020-10-05 21:28:55.434916 and the aio host is discovery at Oct 05 21:29:0214:04
jrosserebbex: there is something weird on debian 10 related to apparmor which prevents the normal profile being used14:05
ebbexah, ok then :)14:05
jrosserhttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=91663914:06
openstackDebian bug 916639 in lxc "LXC AppArmor confinement breaks systemd v240" [Normal,Fixed]14:06
jrosseri don't know if this is sufficiently fixed that we could undo that now....14:06
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Added Openstack Adjutant role deployment  https://review.opendev.org/75631014:07
*** spatel has joined #openstack-ansible14:07
jrossermgariepy: if the timestamps are correct thats happening *after* the error we got?14:08
admin0how do I specify an additonal NFS backend to glance when it already has CEPH as the main backend ?14:08
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_adjutant master: Make role fit to the OSA standards  https://review.opendev.org/75631314:08
mgariepyyep.14:09
mgariepyit seems. i'll try adjust the attemp..14:09
jrossermgariepy: looks like there is stuff already to avoid a race https://github.com/openstack/openstack-ansible-os_nova/blob/ab9642f95ca870f2616dfab3f9e22937646cc648/tasks/nova_compute_wait.yml14:11
jrossermaybe that needs extending to also check that the cell there14:11
noonedeadpunkadmin0: I think you need to specify glance_available_stores and add to glance_nfs_client14:13
admin0ok14:13
admin0thanks14:13
noonedeadpunkin the format https://docs.openstack.org/glance/rocky/admin/multistores.html (regarding glance_available_stores)14:14
openstackgerritMerged openstack/openstack-ansible-openstack_hosts master: Ensure SSL certificate and key directories and symlinks are present  https://review.opendev.org/75409514:19
mgariepydiscover_hosts_in_cells_interval is set to 60 sec. changing the number of attempt just to be over that 60 sec would be quite a simple fix14:21
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Remove glance_registry from inventory  https://review.opendev.org/75631814:23
openstackgerritMarc GariĆ©py (mgariepy) proposed openstack/openstack-ansible-os_nova master: Adjust the retry for the nova-status upgrade check  https://review.opendev.org/75632114:24
mgariepyi'll be back in an hour or so14:27
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-nspawn_container_create master: Fix linter errors and add lint skip list  https://review.opendev.org/75532014:28
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-galera_server master: Update galera to 10.5.5  https://review.opendev.org/74210514:29
*** mgariepy has quit IRC14:32
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Use pip and virtualenv from host packages rather than get-pip.py  https://review.opendev.org/75627914:33
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ceph_client master: Add configuration example for Ceph configuration from files.  https://review.opendev.org/75605114:33
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-ceph_client master: Allow to proceed with role if ceph_conf_file is set  https://review.opendev.org/75582014:33
*** spatel has quit IRC14:33
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-nspawn_container_create master: Fix linter errors and add lint skip list  https://review.opendev.org/75532014:34
*** spatel has joined #openstack-ansible14:41
openstackgerritMerged openstack/openstack-ansible-os_placement master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75554214:44
*** carlosmss has quit IRC14:46
recycleheroguys please take a look at my drawing. I cant ping compute1 from infra1 containers. I can trace the packet ICMP reply until compute 1 NIC1. after that it disappears14:51
recycleherohttp://a.uguu.se/ybpizp.jpg14:52
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_nova master: Simplify scheduler filter additions  https://review.opendev.org/65014414:56
*** macz_ has joined #openstack-ansible14:56
*** mgariepy has joined #openstack-ansible14:56
mgariepyi'm back.15:03
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_swift master: Define condition for the first play host one time  https://review.opendev.org/75442815:04
jrosserlooks like not just us seeing issues in CI with collection installation https://github.com/ansible/galaxy/issues/230215:08
*** alvinstarr has joined #openstack-ansible15:09
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_nova master: Define condition for the first play host one time  https://review.opendev.org/75633215:14
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ironic master: Define condition for the first play host one time  https://review.opendev.org/75634015:22
noonedeadpunknot sure if it's good or not:)15:22
noonedeadpunkpleasant to hear thet not only we are suffering...15:22
noonedeadpunk*that15:23
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-nspawn_hosts master: Add vars file for ubuntu focal  https://review.opendev.org/75634115:24
jrosserrecyclehero: am i still not really sure about the macvlan stuff15:27
jrosseringress traffic will be matched by the macvlan interface for it's own mac address and everything else discarded?15:28
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Minimise distro packages installed into the lxc image  https://review.opendev.org/61307815:28
openstackgerritMerged openstack/openstack-ansible-os_placement master: Remove support for lxc2 config keys  https://review.opendev.org/75625215:28
recycleheroyou mean macvlan interface is only capable of receiving its own mac hence not reciveing the container traffic inside its subnet15:29
recycleherojrosser:15:29
recycleheroI am getting to ur point. I ordered something capable of vlan15:29
jrosserhttps://superuser.com/questions/1205346/macvtap-interface-created-on-top-of-macvlan-interface-of-a-docker-container-cann15:30
jrosser^ similar issue15:30
recycleheroI would love to here other guys idea to so I will paste the URL to my network again15:31
recycleherohttp://a.uguu.se/ybpizp.jpg15:31
recycleherojrosser: I get it its only capable of receiving its on mac so I cant make a bridge behind it15:32
recycleherothank u15:32
openstackgerritMerged openstack/openstack-ansible-os_glance master: Remove support for lxc2 config keys  https://review.opendev.org/75624815:35
jrosserrecyclehero: it would be worth reading this https://cloudnull.io/2017/06/nspawning-openstack-ansible/15:35
cloudnullo/15:35
cloudnullhappy to answer questions as needed15:35
jrosserhello there!15:35
cloudnullohai :D15:35
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-nspawn_container_create master: Fix linter errors and add lint skip list  https://review.opendev.org/75532015:47
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-nspawn_container_create master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75415715:47
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-nspawn_container_create master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75415715:48
*** rpittau is now known as rpittau|afk15:48
*** shyamb has joined #openstack-ansible15:53
openstackgerritMerged openstack/openstack-ansible stable/train: Remove opensuse periodic jobs from train branch  https://review.opendev.org/75622115:55
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-nspawn_hosts master: Add vars file for ubuntu focal  https://review.opendev.org/75634116:00
noonedeadpunk#startmeeting openstack_ansible_meeting16:00
openstackMeeting started Tue Oct  6 16:00:51 2020 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:00
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:00
noonedeadpunk#topic office hours16:01
*** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)"16:01
*** miloa has quit IRC16:01
noonedeadpunkso, we have a lot of ongoing things that are slowly merging, which makes me think that we're pretty good at the moment16:02
noonedeadpunkand it feels that we will have pretty early release16:03
noonedeadpunkand having mariadb 10.5, ansible 2.10 and 2 new roles (senlin and adjutant) sounds as a nice things16:04
noonedeadpunkI guess once we will merge most of the things we have at the moment we shall do milestone release (beta) before doing rc16:05
noonedeadpunkas we have time on our hands16:05
jrossero/ hello16:06
jrosseryes it feels like good for an early release, seems we catch some interesting things already but get fixes in16:07
spotzo/ hey!16:07
jrossergood find from mgariepy on the focal job nova cell thing16:08
jrosserhello!16:08
jrossersenlin stuff is blocked on this https://review.opendev.org/#/c/755759/16:10
*** shyamb has quit IRC16:10
jrosserthat role is using the integrated linter already16:10
jrosserwhere are we with ceph stuff - i didnt get chance today to understand whats breaking ceph_client role16:11
noonedeadpunkI thought it was more blocked by this https://review.opendev.org/#/c/756121/16:11
noonedeadpunkI think ceph should be kind of passing?16:12
*** shyamb has joined #openstack-ansible16:12
jrosseroh wrong patch yes you are right16:12
noonedeadpunkand https://review.opendev.org/#/c/756051/ just passed in terms of linters16:12
noonedeadpunkhttps://review.opendev.org/#/c/753376/ is passing for integrated as well16:13
jrosseri think ceph_client may just be victim of the various things that broke / need fixes16:13
jrosserso that should come good soon16:13
jrosserwe should decide about the very old branch periodic jobs16:14
noonedeadpunkyeah, I think that it's ok by itself16:14
jrosseri make patches today to drop the periodics but a lot of the CI there seems totally broken16:15
noonedeadpunkwe can drop those branches, but seems ppl still deploying queens and stuff..16:15
jrosserat least the changes to the CI image python setup will be causing trouble16:15
noonedeadpunkI can ask infra to force merge them16:15
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-galera_server master: Ability to take mariadb backups using mariabackup  https://review.opendev.org/75526116:16
noonedeadpunk(I guess I can?:)16:16
noonedeadpunkOh, that is good ^ :)16:16
jrosserthey may just ask us to make the jobs NV, or even delete them16:16
noonedeadpunkI think I already have a scenario where I will be using built-in backups16:16
noonedeadpunkwell...16:17
noonedeadpunknot sure we should really spend much time on it16:17
*** shyamb has quit IRC16:17
noonedeadpunkI don't feel good about dropping branches tbh, (maybe we can drop ocata ofc)16:18
noonedeadpunkbut rocky and queens probably worth staying for a while?16:18
jrosseri can modify the patches i made today just to make all the jobs noon-voting, that would be a starting point16:18
*** shyamb has joined #openstack-ansible16:18
noonedeadpunkyeah, I think that's the best option at the moment16:19
noonedeadpunkas for sure we shouldn't be trying to fix them16:19
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/pike: Remove periodic CI jobs for pike branch  https://review.opendev.org/75622616:21
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/queens: Remove periodic CI jobs for queens branch  https://review.opendev.org/75622316:23
*** shyamb has quit IRC16:26
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/rocky: Remove periodic CI jobs for rocky branch  https://review.opendev.org/75622716:26
*** d34dh0r53 has quit IRC16:26
jrosser^ rocky is completley broken16:27
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/ocata: Remove periodic CI jobs for ocata branch  https://review.opendev.org/75622516:29
noonedeadpunkyeah, it is:(16:29
jrosseri expect thats mostly python/CI image stuff16:30
noonedeadpunkFeels like stein might be broken pretty soon as well16:30
*** d34dh0r53 has joined #openstack-ansible16:30
jrosseri think there is still hope for stein https://zuul.openstack.org/builds?project=openstack%2Fopenstack-ansible&branch=stable%2Fstein&pipeline=periodic16:31
noonedeadpunkoh, hm, why translations fail instantly...16:36
*** rh-jlabarre has joined #openstack-ansible16:44
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Use pip and virtualenv from host packages rather than get-pip.py  https://review.opendev.org/75627916:44
*** rh-jelabarre has quit IRC16:45
noonedeadpunk#endmeeting16:58
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://bit.ly/2SAcGAn"16:58
openstackMeeting ended Tue Oct  6 16:58:05 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:58
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-10-06-16.00.html16:58
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-10-06-16.00.txt16:58
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-10-06-16.00.log.html16:58
mgariepyjrosser, can i have your vote also ? https://review.opendev.org/#/c/756321/16:58
jrossermgariepy: done!17:09
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Fix boolean in config file.  https://review.opendev.org/75550817:16
mgariepynoonedeadpunk, this will will need 75632117:17
arxcruzjrosser: https://review.opendev.org/#/c/733408/ needs rebase :/17:26
jrosserarxcruz: yes, but i have no idea where the mege conflict is17:27
arxcruzjrosser: i'll try take a look tomorrow17:27
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Fix keystone nginx behaviour  https://review.opendev.org/75438217:33
*** maharg101 has quit IRC17:52
recycleheroI dont have openstack cli on my infra host to verify the operation17:56
recycleheroshould I install the cli with pip17:56
recyclehero?17:56
jrosserthe cli is in the utlilty container17:57
recycleherogreat, I want to see why I dont have glance. but most seems ok this time at least I see hypervisor. yes17:58
recycleherosetup with macvlan was bogus17:59
recycleherountil my vlan switch arrive I did like this to have the 3 bridges setup. create 1 bridge connected to only nic name dup(duplicating bridge.) create 3 VETH s and connected one end of them all to br-dup18:00
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Do not manage /etc/ssl or /etc/pki directories or symlinks  https://review.opendev.org/75409218:00
recycleheroand the other end to br-mgmt br-storage and br-vxlan18:00
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Move openstack-ansible-uw_apache centos job to centos-8  https://review.opendev.org/75412218:00
recycleherojrosser: thanks18:02
recycleherojrosser: I am getting lots of errors about neutron metering agent, how should I discard it? I want to learn how to things with ansible now18:04
recycleheroI mean I dont want the metering agent18:04
*** gyee has joined #openstack-ansible18:16
*** andrewbonney has quit IRC18:22
openstackgerritMerged openstack/openstack-ansible-os_gnocchi master: Use the utility host for db setup tasks  https://review.opendev.org/75472218:28
openstackgerritMerged openstack/openstack-ansible-os_nova master: Adjust the retry for the nova-status upgrade check  https://review.opendev.org/75632118:34
*** pto has quit IRC18:35
*** viks____ has quit IRC18:35
*** maharg101 has joined #openstack-ansible18:35
recycleherodoes the os-infra_hosts even work? I see it in only one line18:41
*** maharg101 has quit IRC18:44
*** d34dh0r53 has quit IRC18:53
*** d34dh0r53 has joined #openstack-ansible18:53
recycleherohttps://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/app-custom-layouts.html this is helping18:55
*** klamath_atx has quit IRC19:00
*** cshen has quit IRC19:04
*** cshen has joined #openstack-ansible19:10
*** sum12 has joined #openstack-ansible19:11
*** cshen has quit IRC19:14
*** yolanda has quit IRC19:20
*** yolanda has joined #openstack-ansible19:20
*** cshen has joined #openstack-ansible19:42
*** cshen has quit IRC19:47
*** tosky has quit IRC19:52
*** klamath_atx has joined #openstack-ansible20:04
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Simplify path for letsencrypt usage  https://review.opendev.org/75132720:16
*** cshen has joined #openstack-ansible20:17
*** cshen has quit IRC20:22
*** maharg101 has joined #openstack-ansible20:41
*** maharg101 has quit IRC20:45
*** cshen has joined #openstack-ansible21:01
*** cshen has quit IRC21:05
*** rfolco has quit IRC21:14
*** maharg101 has joined #openstack-ansible21:25
*** maharg101 has quit IRC21:29
*** spatel has quit IRC21:35
*** rfolco has joined #openstack-ansible21:36
*** rfolco has quit IRC21:36
*** macz_ has quit IRC22:33
*** recyclehero has quit IRC22:39
*** MickyMan77 has joined #openstack-ansible22:49
*** recyclehero has joined #openstack-ansible22:51
*** MickyMan77 has quit IRC22:58
*** cshen has joined #openstack-ansible23:01
*** cshen has quit IRC23:05
*** cshen has joined #openstack-ansible23:06
*** cshen has quit IRC23:10
*** maharg101 has joined #openstack-ansible23:26
*** MickyMan77 has joined #openstack-ansible23:27
*** maharg101 has quit IRC23:31
*** MickyMan77 has quit IRC23:35

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!