Tuesday, 2019-01-22

*** macza has joined #openstack-ansible00:00
*** tosky has quit IRC00:06
*** DanyC has quit IRC00:23
*** macza has quit IRC00:24
*** DanyC has joined #openstack-ansible00:39
*** hwoarang has quit IRC00:40
*** hwoarang has joined #openstack-ansible00:42
*** DanyC has quit IRC00:43
cloudnullevenings all01:04
cloudnull++ https://review.openstack.org/#/c/632142 if folks are around this would be great to get merged, it should unblock that branch for folks still using it01:05
*** markvoelker has quit IRC01:08
*** markvoelker has joined #openstack-ansible02:09
*** markvoelker has quit IRC02:43
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin  https://review.openstack.org/63202102:46
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use the correct heat tests  https://review.openstack.org/63069502:46
*** shyamb has joined #openstack-ansible03:17
*** gkadam has joined #openstack-ansible03:19
*** shyamb has quit IRC03:28
*** udesale has joined #openstack-ansible03:30
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update delegated setup hosts to support IP delegation  https://review.openstack.org/63220103:32
*** udesale has quit IRC03:35
*** udesale has joined #openstack-ansible03:36
*** shyamb has joined #openstack-ansible03:36
*** markvoelker has joined #openstack-ansible03:41
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update delegated setup hosts to support IP delegation  https://review.openstack.org/63220103:59
*** shyamb has quit IRC04:03
openstackgerritChandan Kumar proposed openstack/openstack-ansible-tests master: [DNM]Revert "Setup clouds.yaml on tempest node"  https://review.openstack.org/63221204:04
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Always generate stackviz irrespective of tests pass or fail  https://review.openstack.org/63196704:05
*** markvoelker has quit IRC04:13
*** gkadam has quit IRC04:21
*** sm806 has quit IRC04:22
*** sm806 has joined #openstack-ansible04:22
*** shyamb has joined #openstack-ansible05:04
*** shyamb has quit IRC05:05
*** shyamb has joined #openstack-ansible05:05
*** markvoelker has joined #openstack-ansible05:10
*** macza has joined #openstack-ansible05:24
*** macza has quit IRC05:29
*** shyamb has quit IRC05:38
*** markvoelker has quit IRC05:43
openstackgerritMatthew Thode proposed openstack/openstack-ansible-lxc_hosts master: add gentoo support  https://review.openstack.org/60839305:49
*** radeks has joined #openstack-ansible05:51
openstackgerritMatthew Thode proposed openstack/openstack-ansible-tests master: add Gentoo jobs as non-voting  https://review.openstack.org/60810205:54
prometheanfirewell, time to get started again05:56
*** shyamb has joined #openstack-ansible06:14
*** spsurya has joined #openstack-ansible06:21
*** Cjloader_ has joined #openstack-ansible06:21
Cjloader_Hi all06:23
Cjloader_I'm looking for a second review on https://review.openstack.org/#/c/632142/06:24
Cjloader_Fixes the ceph issues on pike06:24
chandankumarjrosser: odyssey4me \o/06:25
Cjloader_I'm on webchat but my account is cjloader06:26
*** faizy98 has joined #openstack-ansible06:37
*** markvoelker has joined #openstack-ansible06:40
*** Cjloader_ has quit IRC06:49
*** shyamb has quit IRC06:51
*** shyamb has joined #openstack-ansible06:54
cjloaderthx jrosser06:55
cjloaderAppreciate it06:55
cjloaderAny reviews I can help with?06:56
chandankumarodyssey4me: I think I found the issue why centos7 tempest scenario tests are failing06:58
chandankumarodyssey4me: in passing jobs http://logs.openstack.org/12/632212/1/check/openstack-ansible-functional-centos-7/574e83b/logs/ara-report/result/61697164-67a9-4097-a27a-bae17caad052/  Ensure public /private network exists tasks skipped but in failing one06:59
chandankumarhttp://logs.openstack.org/67/631967/5/check/openstack-ansible-functional-distro_install-centos-7/b4fc1b2/logs/ara-report/result/d622d6da-8be0-45f4-b452-d831486f4432/ it is getting created07:00
chandankumarmay be a mismatch in public network config leading to that07:00
chandankumarodyssey4me: http://git.openstack.org/cgit/openstack/openstack-ansible-tests/tree/test-vars.yml#n224 is working but I think this might be the reason http://git.openstack.org/cgit/openstack/openstack-ansible-os_tempest/tree/tests/os_tempest-overrides.yml#n2907:11
*** markvoelker has quit IRC07:14
*** faizy_ has joined #openstack-ansible07:22
cjloaderMaking Ocata changes as well... chandankumar jrosser cloudnull -- look for similar PR07:24
*** faizy98 has quit IRC07:25
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218207:25
*** faizy98 has joined #openstack-ansible07:25
cjloaderodyssey4me: ^07:26
cjloaderPike/Ocata should be fixed with these two PRs07:26
*** faizy_ has quit IRC07:27
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218207:27
*** faizy_ has joined #openstack-ansible07:28
*** faizy98 has quit IRC07:30
*** faizy98 has joined #openstack-ansible07:31
*** faizy_ has quit IRC07:32
jrossercjloader: nice, will keep an eye on those07:38
*** pcaruana has joined #openstack-ansible07:41
*** DanyC has joined #openstack-ansible07:48
cjloaderlinter failed on Ocata07:48
cjloaderjrosser07:49
cjloadernot sure why07:49
cjloaderERROR: InvocationError for command '/home/zuul/src/git.openstack.org/openstack/openstack-ansible/.tox/linters/bin/ansible-galaxy install --role-file=/home/zuul/src/git.openstack.org/openstack/openstack-ansible/ansible-role-requirements.yml --force' (exited with code 250)07:51
*** gkadam has joined #openstack-ansible07:51
openstackgerritzhouxinyong proposed openstack/ansible-role-python_venv_build master: Fix author-email in setup.cfg  https://review.openstack.org/63227207:53
*** shyamb has quit IRC07:58
*** macza has joined #openstack-ansible08:05
*** kopecmartin|off is now known as kopecmartin08:08
*** macza has quit IRC08:09
*** markvoelker has joined #openstack-ansible08:10
*** DanyC has quit IRC08:18
*** Dantalion has quit IRC08:23
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218208:33
*** shyamb has joined #openstack-ansible08:40
*** faizy98 has quit IRC08:41
*** tosky has joined #openstack-ansible08:43
*** markvoelker has quit IRC08:43
*** electrofelix has joined #openstack-ansible08:56
*** priteau has quit IRC08:56
*** priteau has joined #openstack-ansible08:57
openstackgerritMerged openstack/openstack-ansible stable/pike: Fix ceph-ansible role-requirements  https://review.openstack.org/63214208:59
cjloaderjrosser: ^09:01
cjloaderPike Ceph issues are fixed09:01
*** DanyC has joined #openstack-ansible09:29
*** DanyC has quit IRC09:29
*** DanyC has joined #openstack-ansible09:29
admin0morning all . has anyone done a ubuntu 18 + rocky + ovs ? care to share configs ?09:38
*** markvoelker has joined #openstack-ansible09:40
*** sum12 has quit IRC09:43
*** sum12 has joined #openstack-ansible09:44
*** aedc has quit IRC10:00
*** shyamb has quit IRC10:05
openstackgerritzhouxinyong proposed openstack/monitorstack master: Change openstack-dev to openstack-discuss in setup.cfg  https://review.openstack.org/63237810:07
*** markvoelker has quit IRC10:13
odyssey4mecjloader I think the ocata one is failing because I don't think that tag exists10:19
odyssey4meoh no, it's because the test is using ansible-galaxy to pull down the roles, and that is not a galaxy role10:20
odyssey4melet me see if I can figure out a simple solution for that - unfortunately the pike changes for that aren't back portable :/10:24
*** udesale has quit IRC10:28
*** udesale has joined #openstack-ansible10:28
*** shardy has joined #openstack-ansible10:32
*** udesale has quit IRC10:33
*** shyamb has joined #openstack-ansible10:38
*** shyamb has quit IRC10:39
*** shyamb has joined #openstack-ansible10:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218210:45
*** markvoelker has joined #openstack-ansible11:10
evrardjpodyssey4me: hey11:15
evrardjpodyssey4me: I see this: https://review.openstack.org/#/c/632182/11:15
evrardjpis that the plan to switch everywhere to use git only?11:15
odyssey4meevrardjp ocata is the only branch that uses ansible-galaxy at this time11:15
evrardjpyeah, but that changes things11:16
odyssey4mealthough in theory ansible-galaxy works... although I'm finding that using it with ceph-ansible results in something rather odd11:16
evrardjpif we say that we use git by default and don't support ansible-galaxy anymore, then a problem for tracking could be solved11:16
evrardjpwe could add the track branch there11:16
odyssey4meit might make sense for us to switch to git exclusively and remove all pretense of being able to use ansible-galaxy11:16
evrardjpthat would be lovely11:16
evrardjpbecause we could say "track branch x for role y"11:17
odyssey4meyep, you could always propose it with a release note ;)11:17
evrardjpI am fine with that11:17
*** udesale has joined #openstack-ansible11:19
openstackgerritzhouxinyong proposed openstack/openstack-ansible-nspawn_container_create master: Change openstack-dev to openstack-discuss in setup.cfg  https://review.openstack.org/63242911:40
openstackgerritzhouxinyong proposed openstack/openstack-ansible-os_manila master: Change openstack-dev to openstack-discuss in setup.cfg  https://review.openstack.org/63243011:42
openstackgerritzhouxinyong proposed openstack/openstack-ansible-os_masakari master: Change openstack-dev to openstack-discuss in setup.cfg  https://review.openstack.org/63243211:43
*** markvoelker has quit IRC11:44
openstackgerritzhouxinyong proposed openstack/openstack-ansible-os_placement master: Fix author-email in setup.cfg  https://review.openstack.org/63243311:45
*** aedc has joined #openstack-ansible11:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218211:50
chandankumarodyssey4me: Hello11:51
chandankumarodyssey4me: regarding centos-7 jobs failure I think there is some problem with network cidr passed to the job11:52
chandankumarodyssey4me: the same issue was coming on tripleo side, I added the correct cidr gateway and it started working there11:52
*** radeks_ has joined #openstack-ansible11:52
odyssey4mechandankumar ok11:53
*** radeks has quit IRC11:55
*** dave-mccowan has joined #openstack-ansible11:56
*** shyamb has quit IRC12:00
*** shyamb has joined #openstack-ansible12:02
chandankumarodyssey4me: does the job configuration openstack-ansible-functional-centos-7 running against os_tempest and openstack-ansible-tests are different?12:07
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218212:10
odyssey4mechandankumar there are differences in the inventory provided to it, and might also be overrides - look in the tests/ folder of each repo to see the inventory and overrides given12:11
odyssey4mechandankumar you could also fire up a test vm and use 'run_tests.sh functional' to replicate the test in the vm and inspect it12:15
chandankumarodyssey4me: I am going to do that12:15
*** macza has joined #openstack-ansible12:21
*** shyamb has quit IRC12:23
*** shyamb has joined #openstack-ansible12:23
*** gary_perkins has quit IRC12:24
*** macza has quit IRC12:26
*** gary_perkins has joined #openstack-ansible12:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218212:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218212:37
*** gary_perkins has quit IRC12:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218212:38
*** markvoelker has joined #openstack-ansible12:41
*** gary_perkins has joined #openstack-ansible12:41
*** gary_perkins has quit IRC12:44
*** gary_perkins has joined #openstack-ansible12:46
*** shyamb has quit IRC12:50
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Apply constraints when building the venv  https://review.openstack.org/62033112:51
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Use the repo if available  https://review.openstack.org/62033912:52
*** shyamb has joined #openstack-ansible12:52
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218213:03
*** shyamb has quit IRC13:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Update ceph galaxy cleanup for monolithic repo  https://review.openstack.org/63246013:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Update ceph galaxy cleanup for monolithic repo  https://review.openstack.org/63246013:05
*** shyamb has joined #openstack-ansible13:06
*** radeks_ has quit IRC13:06
*** radeks_ has joined #openstack-ansible13:06
*** markvoelker has quit IRC13:09
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218213:09
odyssey4meguilhermesp mnaser jrosser well, that's kinda exciting - the queens->rocky upgrade test passed: http://zuul.openstack.org/build/86ccfb00a47046b3a7b31fd2cfcc3a4513:14
jrosserodyssey4me: did that *just* scrape in the 3hours13:15
odyssey4meyep, it seems so13:15
odyssey4mealthough periodics get 4 I think13:15
*** shyamb has quit IRC13:15
*** shyamb has joined #openstack-ansible13:15
jrosseri was thinking about the that - if we are ever to get an arm64 job to pass it's going to need to run for a really really long time13:16
odyssey4mejrosser I would think we'd be better of doing multi-node, multi-arch for that... instead of an AIO.13:16
odyssey4meOtherwise yeah, it will take *ages*.13:16
jrossertrouble is there is every python package needs building13:16
jrosserthere are no binaries for anything at all13:17
jrosserso i'm not sure more nodes will help13:17
odyssey4meOh? There's no infra wheel set for them?13:17
jrosserunless i'm missing something they're not upstream on pypi13:17
jrosserit spends forever building ancialliary stuff like PyNaCl which takes ages13:18
odyssey4meno, but infra builds wheels of everything in upper-constraints and hosts them on their own pypi mirror13:18
jrosseroh i see - that would help hugely if it is available13:18
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Use the repo if available  https://review.openstack.org/62033913:23
jrosserodyssey4me: looks like all x86_64? http://mirror.dfw.rax.openstack.org/wheel/13:24
odyssey4mejrosser yeah, perhaps they've still got the others in the works13:24
odyssey4meor noone has volunteered to do it yet13:25
jrosserit's a bit chicken/egg i guess13:25
jrosseri think we've got nearly enough patches in/proposed to make arm64 go13:25
*** shyamb has quit IRC13:26
odyssey4mejrosser of course another approach we could take is to use a x86_64 VM with an arm64 vm on top of it - that may go faster13:27
jrosseri can try this out on arm64 vm - getting a mirror would be a massive bonus because that really is taking a huge time13:28
jrosserinstalling and configuring stuff is quite lightweight compared to the amount of compiling it is doing, and to make matters worse most wheel builds don't use any form of parallelism13:29
*** oanson has quit IRC13:31
*** mbuil_ has quit IRC13:33
*** sum12 has quit IRC13:33
*** flaviosr has quit IRC13:33
*** ansmith has joined #openstack-ansible13:35
*** devx has joined #openstack-ansible13:37
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: [WIP] Add integrated deploy/upgrade jobs  https://review.openstack.org/61976513:42
*** priteau has quit IRC13:46
openstackgerritMerged openstack/openstack-ansible-ops master: Add virtualenv to packages downladed  https://review.openstack.org/60734413:49
*** ostackz has joined #openstack-ansible13:53
openstackgerritMerged openstack/ansible-role-python_venv_build stable/rocky: Mark build task changed only when wheels are built  https://review.openstack.org/61296613:59
openstackgerritMerged openstack/ansible-role-python_venv_build stable/rocky: Only install the appropriate distro packages  https://review.openstack.org/61337213:59
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zun master: Improve docker API version retrieval  https://review.openstack.org/63213114:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zun master: Enable overriding the service setup host python interpreter  https://review.openstack.org/62123914:03
*** flaviosr has joined #openstack-ansible14:05
openstackgerritMerged openstack/openstack-ansible-os_gnocchi stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62253414:08
*** sum12 has joined #openstack-ansible14:11
openstackgerritMerged openstack/openstack-ansible-os_designate master: Enable overriding the service setup host python interpreter  https://review.openstack.org/62119714:12
openstackgerritMerged openstack/openstack-ansible-os_ceilometer stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62253014:12
ostackzodyssey4me hi, could you be aware of existing cherry-pickable patch for horizon(rocky) that solves issue that user cannot see volume details for volume it has just created?14:14
ostackzGetting this error in horizon: Recoverable error: Invalid filters volume_id are found in query options. (HTTP 400)14:15
ostackzseems that this is definitely bug in horizon as same user can see volume details if using openstack cli client14:15
odyssey4meostackz I've never heard of that issue. Is it perhaps some sort of policy issue?14:15
ostackzusing default policies14:15
odyssey4meostackz is there an /etc/cinder/policy.json file in your cinder containers?14:16
ostackzodyssey4me yes14:17
odyssey4meostackz is that from overrides you configured?14:18
odyssey4meif not, then see if https://github.com/openstack/openstack-ansible-os_cinder/commit/7c90daa9097f816aac797a0d93cabff23bdd7edd is present in your /etc/ansible/roles/os_cinder14:18
*** TxGirlGeek has joined #openstack-ansible14:18
odyssey4meif not, do the following in that folder: git fetch --all; git checkout 7c90daa9097f816aac797a0d93cabff23bdd7edd14:19
odyssey4methen re-run your os-cinder-install.yml playbook14:19
ostackzodyssey4me thanks! will try14:20
*** priteau has joined #openstack-ansible14:21
chandankumarodyssey4me: need some help here http://logs.openstack.org/00/627500/66/check/tripleo-ci-centos-7-standalone-os-tempest/0042800/job-output.txt.gz#_2019-01-22_13_02_56_36383414:29
chandankumarodyssey4me: I am not sure what I am doing wrong here https://review.openstack.org/#/c/628415/34/playbooks/multinode-standalone.yml@7114:30
chandankumarodyssey4me: basically I need to replace the last IPV4 digit with those values14:30
odyssey4mechandankumar what's the nthhost filter? I've not seen that before.14:32
odyssey4mechandankumar https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters_ipaddr.html shows various ways of getting particular addresses from a cidr14:33
chandankumarodyssey4me: It is used in tripelos side to replce https://github.com/ansible/ansible/issues/3638814:34
*** hwoarang has quit IRC14:36
odyssey4meinteresting - I'm afraid that's a bit much of a context switch for me right now, I'm tied up in something else - maybe someone else can help14:36
*** hwoarang has joined #openstack-ansible14:37
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/63247814:38
*** TxGirlGeek has quit IRC14:39
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-ops master: Update delegated setup hosts to support IP delegation  https://review.openstack.org/63220114:42
jamesdentonchandankumar maybe {{ tempest_cidr|nthhost(100) ~ "-" ~ tempest_cidr|nthhost(120) }}?14:44
*** DanyC has quit IRC14:51
*** DanyC has joined #openstack-ansible14:52
CeeMacquick question, I've just run rsyslog-install.yml and it completed with no output effectively. Not "no nosts matched" just blank under PLAY RECAP14:53
CeeMacin my openstack_user_config.yml I have an log_hosts entry14:54
CeeMacam I missing something?14:54
CeeMacoh, and hi :)14:55
*** DanyC has quit IRC14:56
ostackzodyssey4me your advice regarding horizon helped! Thanks a lot!  Wondering how user of osa can know about these issues? I believed if I checkout stable 18.1.2 all fixes should be there :)15:01
odyssey4meostackz nope, that patch just missed 18.1.2, but should be in 18.1.315:17
odyssey4mehmm, actually it missed the 18.1.3 role sha bump too... so I guess we need to bump that to get it into 18.1.315:18
odyssey4meostackz hanging in here and taking a peek at the chat now and again helps to keep up to speed, monitoring the proposed patches also helps - otherwise the release notes... although sometimes patches make it in without a note15:19
odyssey4meostackz I actually put that patch in because I found out we were carrying out of date policies - I didn't realise there were actual issues as a result :p15:20
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Update os_cinder role for policy-in-code fix  https://review.openstack.org/63248515:23
openstackgerritMerged openstack/ansible-hardening master: Switch to rtcsync for chrony  https://review.openstack.org/63099615:23
jrosserodyssey4me: this policy stuff feels like a meeting topic?15:23
odyssey4mejrosser we can discuss it if you like - it's tech-debt really, which we're well behind on cleaning up15:24
jrosserodyssey4me: it's something i understand relatively small amount of - so just all being on the same page about what needs doing would be great15:25
odyssey4mejrosser ok sure, can you add a topic to the meeting15:25
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Update roles for policy-in-code fixes  https://review.openstack.org/63248515:28
ostackzodyssey4me Nice this horison patch went to next tag! There is one more bothering issue in rocky that could be solved already - I do not get any logs in log_hosts - not under /var/log neither in journalctl -fa15:30
CeeMacostackz, wonder if thats related to the issue i'm seeing15:30
ostackzI have searched docs but cannot get the mechanism how logs should be forwarded to those hosts15:30
odyssey4meostackz I dunno anything about that.15:30
odyssey4meIt uses journal-remote IIRC.15:31
CeeMacostackz, did your rsylog playbook run with results?15:31
cloudnullmornings15:31
odyssey4meCeeMac for rocky onwards, rsyslog isn't used any more for any platform except suse15:31
odyssey4meit's all systemd-journal15:31
CeeMacooooooh, right15:31
spotzHey!15:31
ostackzCeeMac my rsyslog containers are created, but inside I cannot find where logs are gathered15:32
CeeMacjust seemed weird that it didn't give a "no matching hosts" like the other ones that aren't relevant15:32
odyssey4meostackz journals will now publish to the hosts, and the hosts are configured to upload the journals to the log container as far as I know15:32
cloudnullo/ spotz!15:32
CeeMacodyssey4me, so its still using the rsyslog container for the journal uploads?15:33
cloudnullodyssey4me ostackz the journals from remote hosts should be under /var/log/journal/${JOURNAL_ID}15:33
odyssey4meCeeMac I think so, but I'm not familiar enough with that work to confirm. cloudnull is the SME ;)15:33
CeeMacah, ok15:34
spotz:)15:34
CeeMachi cloudnull :)15:34
cloudnullif its publishing you should see systemd-journal-remote and systemd-journal-upload running on the various host15:34
prometheanfirecloudnull: would next step for me to change gentoo to machinectl only? http://logs.openstack.org/02/608102/9/check/openstack-ansible-functional-gentoo-17-0-systemd/586f741/job-output.txt.gz#_2019-01-22_07_59_36_04648715:36
CeeMaccloudnull, so all of the remote journals will aggregate under the /var/log/journal/${JOURNAL_ID} folder? or there would be entries for each host/container?15:36
cloudnullCeeMac they aggregate under the journal directory and all hosts will have their own journal ID15:37
cloudnullprometheanfire ++15:37
CeeMaccloudnull, is there an easy way to correlate journal ID to host?15:38
prometheanfirecloudnull: we don't have a lxc base image, though I guess the systemd stage4 would work15:41
cloudnullCeeMac you should be able to run something like `journalctl --directory=/var/log/journal/remote` to stream the journals15:41
cloudnulland then filter based on a given host15:41
openstackgerritMerged openstack/openstack-ansible master: Bump pins  https://review.openstack.org/63042615:41
CeeMaccool thanks cloudnull, i'll give that a go15:42
ostackzcloudnull CeeMac I see multiple folders under /var/log/journal/ in my log_hosts, but how can you actually read those logs so you can understand where those come from?15:42
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Remove glance_developer_mode and glance_venv_download  https://review.openstack.org/62034015:43
ostackzjournalctl --directory=/var/log/journal/remote  returns "No journal files were found."15:43
cloudnullIE: if you  were looking for log entries from nova-api-os-compute you could run something like `journalctl --directory=/var/log/journal/remote -u "nova-api-os-compute"`15:43
ostackzcloudnull is it intended so that you can see logs from all containers/hosts in log_hosts machines centrally?15:44
CeeMaccloudnull, you'd be running that from within the rsyslog container?15:45
cloudnullI would run that on the log host directly15:45
CeeMacright15:45
* cloudnull is operating from memory. not in a real system at the moment 15:46
cloudnullmy syntax might be a little wrong15:46
CeeMacunderstood15:46
CeeMacso, on my log host i have /var/log/journal15:46
CeeMacwith a bunch of IDs15:46
CeeMacand remote15:46
CeeMacfolders15:46
CeeMacthe remote folder is empty15:46
CeeMacbut that could be because I haven't finished my deployment yet >_>15:47
CeeMaci'll get back to you :D15:47
cloudnullok.15:47
cloudnullgotta run to get coffee and then meetings, but I will be back in a n hour or so15:47
CeeMacsure15:47
cloudnulllet me know how it goes :)15:47
CeeMacostackz, if i run 'journalctl --directory=/var/log/journal' on my log host I get a bunch of records showing15:48
ostackzCeeMac I did the same, but I get logs regarding only localhost15:50
CeeMacright15:50
CeeMacdo you see a bunch of folders under /var/log/journal?15:51
ostackzI also tried journalctl -fa --directory=/var/log/journal and then creating some volume, hoping to see cinder logs - but nothing pops up15:51
*** openstackgerrit has quit IRC15:51
ostackzCeeMac yes + "remote" folder15:51
CeeMacis your remote folder empty?15:51
ostackzyes15:52
*** TxGirlGeek has joined #openstack-ansible15:52
CeeMacmaybe try referencing the ID folders one at a time see which host they belong to?15:52
spotzGonna head out for a bit, if we do triage and there's anything for me just assign it;)15:53
odyssey4mecores, we could do with some reviews for https://review.openstack.org/632182 - it's ocata, but it seems to work according to the tests15:53
odyssey4merelated: https://review.openstack.org/63246015:54
*** TxGirlGe_ has joined #openstack-ansible15:55
*** DanyC has joined #openstack-ansible15:56
*** TxGirlGeek has quit IRC15:57
CeeMacostackz, try 'journalctl --directory=/var/log/journal/ -m' and filter for what you're looking for15:59
*** DanyC has quit IRC16:00
*** TxGirlGe_ has quit IRC16:01
evrardjpis it weekly meeting time?16:01
redrobotwondering the same ^^16:01
*** udesale has quit IRC16:02
spotzWe didn't meet last week, but if we're meeting now I'll stick around a bit before I go ride:)16:02
evrardjpwell I think it's time for it16:02
evrardjpmnaser: are you there/available?16:03
*** DanyC has joined #openstack-ansible16:03
*** DanyC has quit IRC16:03
*** DanyC has joined #openstack-ansible16:04
evrardjpI am in two meetings, so I can't really chair the meeting16:04
evrardjpcould someone help there?16:05
evrardjpI have generated the bug triage page16:05
evrardjpare the topics up to date on the wiki?16:05
redrobotI had a topic, but did not add it to the wiki16:06
ostackzCeeMac -m for journalctl works, but those folder IDs point just to containers on local host, I guess remote logs should be under that "remote" folder, but it is empty.16:06
redrobots/had/have16:06
evrardjpit is not up to date16:06
jrosserthey look old bar the last one which i added earlier16:06
CeeMacostackz, ok that looks the same as me then16:06
evrardjphttps://wiki.openstack.org/wiki/Meetings/openstack-ansible16:06
evrardjpredrobot: ^16:06
CeeMacout of interest, do you see systemd-journal-remote and systemd-journal-upload running on the hosts? (referencing earlier comment by cloudnull)16:07
redrobotevrardjp, added16:07
* odyssey4me is also in another meeting, so can't chair16:08
chandankumarjamesdenton: thanks!16:08
evrardjpredrobot: I added it in the open discussion instead16:08
jamesdentonchandankumar did that work?16:08
CeeMacostackz: i see systemd-journal-remote running on the log host, but not elsewhere16:08
evrardjpfirst topic16:08
evrardjpwho can chain the meeting?16:08
evrardjpchair*16:09
*** TxGirlGeek has joined #openstack-ansible16:09
chandankumarjamesdenton: locally working16:09
ostackzCeeMac mee too, but seems that systemd-journal-remote.service is receiving part and it is ok16:10
chandankumarjamesdenton: but this one also working {{ tempest_cidr|nthhost(100)}}-{{ tempest_cidr|nthhost(120) }}16:10
chandankumarcurrently runned in ci16:10
jamesdentonahh yeah. cool16:10
redrobotevrardjp, oh whoops, sorry about that.16:10
spotzodyssey4me: cloudnull jrosser? I don't know the bugs well enough for the discussions16:10
spotzWhich is why I used to lead our community meetings:)16:11
CeeMacostackz, do you see systemd-journal-upload anywhere on your OS hosts?16:11
jrosserwho knows how to work the meeting bot? i have no idea16:11
guilhermespooooh sorry, late o/16:11
spotzlookign for a chair guilhermesp16:11
guilhermespspotz: :D16:12
odyssey4mejrosser https://wiki.openstack.org/wiki/Meetings/openstack-ansible has the startmeeting command, and the boot tells you other commands16:12
*** openstackgerrit has joined #openstack-ansible16:12
openstackgerritMatthew Thode proposed openstack/openstack-ansible-lxc_hosts master: add gentoo support  https://review.openstack.org/60839316:12
evrardjpjrosser: I have updated the page for meeting bot16:12
ostackzCeeMac systemctl status systemd-journal-remote.service shows "...systemd-journal-remote --listen-http=-3 --split-mode=host --compress --seal --output=/var/log/journal/remote/"16:12
CeeMacostackz, yep same here16:12
evrardjpit's now a real story16:12
evrardjp:)16:12
ostackzCeeMac but I do not know how all containers are supposed to push logs to these log_hosts16:13
CeeMacbut the way i understand it, systemd-journal-upload should be running on the other hosts and configured to send to the log_hosts16:13
openstackgerritMatthew Thode proposed openstack/openstack-ansible-lxc_hosts master: add gentoo support  https://review.openstack.org/60839316:13
jrosser#startmeeting openstack_ansible_meeting16:13
openstackMeeting started Tue Jan 22 16:13:40 2019 UTC and is due to finish in 60 minutes.  The chair is jrosser. Information about MeetBot at http://wiki.debian.org/MeetBot.16:13
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:13
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:13
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:13
spotzWoot:)16:13
evrardjpthanks jrosser16:13
jrossero/ everyone16:13
odyssey4me:) thanks jrosser16:13
spotzThen #topic roll call:)16:14
chandankumar\o/16:14
spotzI'm helping:)16:14
jrosser#topic roll call16:14
*** openstack changes topic to "roll call (Meeting topic: openstack_ansible_meeting)"16:14
spotzo/16:14
jrossersorry i have no clue here16:14
spotzNo prob, the bot I know:)16:14
evrardjpjrosser: you're doing great :)16:15
redroboto/16:15
evrardjpo/16:15
chandankumar\o/16:15
jrossercloudnull, DimGR, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__, colinmcnamara, thorst, adreznec, eil397,16:15
jrosserqwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking, mnaser, nicolasbock, jrosser, cjloader, antonym, dcdamien16:15
guilhermesp\o16:15
spotzSo whenever you want to put a bug up do #link <bug> change a topic is #topic <topic>16:15
odyssey4meo/16:15
admin0\o16:15
ostackzCeeMac https://www.freedesktop.org/software/systemd/man/systemd-journal-remote.service.html  systemd-journal-remote is just receiver, pusher part should be something other16:15
spotzAnd we all ran our first meeting, you've got an easy crowd here:)16:15
guilhermespI need to submit PRs and bug reports, but there are deployments making me mad16:15
jrosser#topic highlights from last week16:16
*** openstack changes topic to "highlights from last week (Meeting topic: openstack_ansible_meeting)"16:16
evrardjpSadly I don't much highlights from last week16:16
evrardjpin the wiki16:16
evrardjpanything big which happened last week that is worth mentioning?16:16
prometheanfireo/16:16
spotzPretty quiet week I believe16:17
jrosserodyssey4me: an upgrade job passed? thats pretty cool16:17
CeeMacostackz, yep. the way i understand it, systemd-journal-upload should be running on the other hosts and configured to send to the log_hosts (systemd-journal-remote)16:17
chandankumarI am not sure it is shareable tempest runned using os_tempest in tripleo job http://logs.openstack.org/00/627500/65/check/tripleo-ci-centos-7-standalone-os-tempest/198ae77/logs/undercloud/var/log/tempest/stestr_results.html.gz16:17
CeeMacostackz, think it should be configured by this https://github.com/openstack/openstack-ansible/blob/master/playbooks/infra-journal-remote.yml16:17
guilhermespthere is one still needs to be fixed https://review.openstack.org/#/c/627782/4 not sure if it is an upgrade issue or a tempest issue16:17
evrardjpchandankumar: CeeMac guilhermesp could we keep all those conversations for open discussion please?16:18
jrosserok - onward if theres nothing else?16:18
evrardjpjrosser: is trying to have a meeting in here :p16:18
evrardjpjrosser: onwards!16:18
jrosser#topic bug triage16:18
*** openstack changes topic to "bug triage (Meeting topic: openstack_ansible_meeting)"16:18
jrosser#link16:18
evrardjpyou can just paste the first line of each bug there16:19
evrardjplike this:16:19
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/181254316:19
openstackLaunchpad bug 1812543 in openstack-ansible "Ceph production example in openstack-ansible: devices definition missing" [Undecided,New]16:19
jrosser#link https://bugs.launchpad.net/openstack-ansible/+bug/181254316:19
evrardjpok on this one...16:20
CeeMacevrardjp, sorry, i'll pipe down16:20
jrosserpart of the issue here is that the ceph example config is really just indicative16:21
jrosserand also doesnt get tested16:21
evrardjpCeeMac: hehe, its fine to talk, but wait for the appropriate topic during meeting time :)16:21
evrardjpjrosser: agreed16:21
evrardjpis that a hard requirement?16:21
*** asettle has joined #openstack-ansible16:21
evrardjpit's also up to the deployer to read ceph-ansible bits I guess16:22
evrardjpbut it's maybe worth pointing out?16:22
jrosseralso there is a choice between external vs. OSA deployed ceph and that config probably addresses neither of those cases16:22
evrardjpyup16:22
guilhermespin the past I used to define a different user_variables...like a user_ceph_variables.yml. But I think worth it to document and offer the option for the user to add the configs in user_variables16:22
evrardjpI think it's worth mentioning this conversation in the bug16:23
evrardjpand I'dmark the bug as confirmed16:23
guilhermespagreed16:23
evrardjpfor severity I'd say it prevents ppl from deploying and is not really user friendly so I'd put that in medium16:23
evrardjpwe need to refactor this page16:23
odyssey4meguilhermesp for simplicity in documentation, we stick to referring to user_variables.yml - it helps to be consistent16:24
odyssey4meand it's less confusing16:24
jrosseri cant set severity it seems16:25
odyssey4meadvanced users will understand that more can be used16:25
*** KeithMnemonic has joined #openstack-ansible16:25
guilhermespmakes sense odyssey4me16:25
jrosserwe need to signpost what the options are16:25
jrosserhttps://bugs.launchpad.net/openstack-ansible/+bug/181254116:25
openstackLaunchpad bug 1812541 in openstack-ansible "Ceph production example in openstack-ansible: cidr_networks not found" [Undecided,New]16:25
jrosser^ this is the same thing16:25
guilhermespyup16:26
odyssey4mejrosser what's your launchpad ID?16:26
*** ianychoi has joined #openstack-ansible16:26
spotzSo close as duplicate16:26
odyssey4mejrosser I see jrosser & jonathan-rosser16:26
jrosserodyssey4me: "Jonathan Rosser (jrosser)" in the top right of launchpad16:26
*** strattao has joined #openstack-ansible16:27
odyssey4mejrosser ok, you should now be able to change bugs16:28
jrosseris there a proper way to make duplicate?16:28
jrosserhttps://bugs.launchpad.net/openstack-ansible/+bug/181224516:29
openstackLaunchpad bug 1812245 in openstack-ansible "Config Template doesn't respect variable names sections" [Undecided,New]16:29
odyssey4me'Mark as duplicate' on the top right16:30
evrardjpthat sounds like a nasty templating bug16:30
evrardjpI would not be surprised of the issue there16:30
jrosserodyssey4me: cheers got it16:31
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Fix journal remote playbook  https://review.openstack.org/63250516:32
jrosserok, we should probably assign this one if it's 'nasty bug'16:32
cloudnullCeeMac ostackz ^ - noticed https://pasted.tech/pastes/ebbdb7ea6b5f48b125b2bfa01830e703a3d68057 - looks like the service was not running as expected.16:32
* cloudnull goes back to lurking 16:33
jrosserany takers?16:33
cloudnullhttps://pasted.tech/pastes/0049908c08a574a116b54aa714ed769672350538.raw - now i see my remote journals16:34
evrardjpI don't have the time to take it jrosser16:34
evrardjpbut it sounds like high16:34
* cloudnull reading the bug16:34
cloudnulljrosser assign it to me, ill have a crack at that16:35
cloudnullmaybe this weekend16:35
jrossercloudnull: cheers16:36
jrosserhttps://bugs.launchpad.net/openstack-ansible/+bug/181218516:36
openstackLaunchpad bug 1812185 in openstack-ansible "cannot create volume from image" [Undecided,New]16:36
evrardjpjrosser: don't forget #link :)16:36
evrardjpyou can copy paste the whole line, super easy :p16:36
jrosserok :)16:36
spotz#link https://bugs.launchpad.net/openstack-ansible/+bug/181218516:37
openstackLaunchpad bug 1812185 in openstack-ansible "cannot create volume from image" [Undecided,New]16:37
spotz:)16:37
evrardjpinteresting, would that be following a sha bump?16:37
evrardjpI am curious why it would start to appear, would it have passed upstream testing without noticing?16:38
evrardjpI think this needs further triage16:38
jrosser"To enable the Block Storage services to create a new volume by cloning Image- Volume, set the following options in the DEFAULT section of the cinder.conf file"16:38
guilhermesp^16:39
evrardjpjrosser: where did you see that?16:39
jrosserhttps://docs.openstack.org/cinder/latest/admin/blockstorage-volume-backed-image.html16:39
guilhermespI did have a """"problem"""" like this16:39
guilhermesp"""""fixed"""" reading the docs hahah16:39
jrosserguilhermesp: do you have a patch for this?16:40
guilhermespnot yet =/16:40
guilhermespI have a bunch to submit16:40
evrardjpguilhermesp: that sounds like a valid patch then16:40
guilhermespthis bug can be assigned to me16:40
jrossergreat, thanks16:40
evrardjpjrosser: should we mark this as confirmed, medium, and assign guilhermesp?16:40
evrardjpguilhermesp: thanks!16:40
guilhermespnp!16:40
spotzYep16:41
jrosser#link https://bugs.launchpad.net/openstack-ansible/+bug/181182616:41
openstackLaunchpad bug 1811826 in openstack-ansible "Octavia api fails to start because libpython2.7 is missing" [Undecided,New]16:41
jrosserslightly surprised we pass tests if that is missing16:42
evrardjpjrosser: could that have appeared after our relatively recent cleanups?16:42
evrardjpjrosser: yup worrying16:43
jrosserthe patch which removes a ton from the lxc base image is not merged16:43
evrardjpmaybe it's not properly tested in the role16:43
evrardjpoh I see16:43
evrardjpI thought it was16:43
jrosserno there is work to do in tempest before that can happen16:44
odyssey4meit's plausable that changes in the openstack_hosts role removed things which osoctavia relied on being there16:44
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: [DNM] public subnet allocation pool  https://review.openstack.org/63250816:45
evrardjpodyssey4me: oh I see16:45
jrossertheres nothing merged in octavia for quite a long time, something could easily have gone awry here16:46
evrardjpopenstack-hosts is an important enough role to maybe try to fanout or periodic test of other roles would be good16:46
jrosseri'll poke at this16:47
evrardjpI guess it needs further testing/validation ?16:47
odyssey4meevrardjp I'm working on that, yeah - once we've moved over to using the integrated build for all tests, that'll happen automatically.16:48
evrardjpit's only 12 minutes remaining, should we just to open discussion?16:48
evrardjpjust move*16:48
evrardjpor simply move16:48
jrosseryes16:48
jrosser#topic open discussion16:48
*** openstack changes topic to "open discussion (Meeting topic: openstack_ansible_meeting)"16:48
jrosserfirst (redrobot) New roles for HSM Client Software management16:49
redroboto/16:49
jrosserhello o/16:49
redrobotHeya!  So I talked to spotz and mnaser about this not too long ago.16:49
evrardjpis that linked to ansible-hardening?16:50
redrobotI'm part of Barbican core, and we've been working on automating the Barbican deployment using HSMs as the backend.16:50
evrardjpk16:50
* evrardjp listens carefully 16:50
redrobotWe've got a couple of ansible roles that we would like to add to the openstack-ansible umbrella16:50
redroboteach role targets a specific HSM16:50
* cloudnull is also listening carefully :)16:50
redrobot#link https://github.com/dmend/ansible-role-atos-hsm16:51
redrobot#link https://github.com/dmend/ansible-role-thales-hsm16:51
redrobotWe're using those to install the required HSM Client software on the nodes running barbican16:51
redrobotWe've already merged code into TripleO to make use of the roles16:51
redrobotand we'd prefer them to be hosted in openstack-ansible instead of my personal github16:52
evrardjpredrobot: is it only for software,or also configuration of barbican? We'd like to separate those16:52
redrobotthe roles only do the client software configuration16:52
redrobotthe barbican bits are part of THT16:52
redrobot(tripleo heat templates)16:52
evrardjpok16:53
redrobotI submitted the required changes to governance and openstack-infra16:53
redrobot#link https://review.openstack.org/#/c/631324/16:53
evrardjpoh great16:53
redrobot#link https://review.openstack.org/#/c/631326/16:53
evrardjpI guess mnaser should vote there16:53
redrobotwe're just waiting on +1s from mnaser16:53
redrobotyep, I _think_ I've done all the legwork16:53
redrobotAlso, I'd welcome any feedback on the roles to get them up to your standards if they're not there yet.16:54
evrardjpI think it would make sense if you were core on those roles16:54
evrardjpas you have knowledge we don't have16:54
redrobotagreed.  I think the openstack-infra patch adds a new review team16:54
evrardjpif you need more ppl to help you you should tell mnaser16:54
redrobotthe idea was to add myself and probably Ade Lee (barbican PTL) to the team as well as your core team16:55
spotzI go over and review there when I have time but yeah they were looking for more folks in Vancouver16:55
chandankumarredrobot: both the roles will be discovered by tripleo quickstart as setup.cfg confirms that16:55
spotzon infra as a whole that is16:56
redrobotI see16:56
redrobotAnyway, I mainly wanted to get these patches on your radar16:57
evrardjpredrobot: cool, thanks16:57
evrardjpthank you!16:57
spotzthanks redrobot16:57
jrosserok, lets see whats next....16:57
cloudnullsecond (anyone) how are things going with mitogen? folks using it?16:57
evrardjpIt seems a good idea to collaborate on those redrobot16:57
* cloudnull wants to use it 16:57
pabelangercloudnull: I tried, but had an issue, cannot remember something with lxc I think16:58
jrosseri'm using it all the time outside OSA, and seems OK16:58
evrardjpcloudnull: I don't have cycle to continue to work with dw16:58
jrosserbut you might have to fiddle a bit16:58
evrardjpjrosser: is now more experienced there: )16:58
cloudnullreloated - https://review.openstack.org/#/c/591236/16:58
jrosserthere was a whole heap of bugfixes commited this week16:58
cloudnull** related16:58
jrosserand it feels *exceedingly* close to being able to to end-end OSA deploy with it16:59
jrosserthe only thing missing is wiring OSA physical_host into the mitogen_via setting16:59
jrosserchandankumar: tempest things are still on the 'open discussion' list - is there anything we need to cover?17:00
cloudnullok.17:00
evrardjpI think it's time to wrap up for today17:00
cloudnullpabelanger I wonder if the issue with lxc was something fixed in the recent patch release ?17:01
* cloudnull needs to find more time to tinker 17:01
evrardjpjrosser: you can do that with the static inventory that's shipped by default in osa17:01
evrardjpjrosser: https://github.com/openstack/openstack-ansible/blob/master/inventory/inventory.ini17:01
chandankumarjrosser: openstack-ansible-functional-centos-7 is broken currnetly may be something wrong with public network cidr or allocation pool17:01
evrardjpI have to run17:02
chandankumarfor example17:02
evrardjpttyl everyone, and thanks jrosser for chairing17:02
jrosserok - any more meeting things?17:02
*** pcaruana has quit IRC17:02
jrosser#endmeeting17:02
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2xA1eZC"17:02
evrardjpI guess chandankumar's point is worth caring :)17:02
openstackMeeting ended Tue Jan 22 17:02:27 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:02
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-01-22-16.13.html17:02
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-01-22-16.13.txt17:02
pabelangercloudnull: possible, I can update my ansible-network deployment to try again17:02
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-01-22-16.13.log.html17:02
pabelangerhaven't had much time to work on it recently17:02
*** macza has joined #openstack-ansible17:02
evrardjpchandankumar: I can't help you , but I am sure you're in good hands17:02
chandankumarI tried to reproducer locally but got blocked here on dependency issue on my box http://paste.openstack.org/show/743097/17:03
cloudnullpabelanger me neither :'(17:04
*** macza_ has joined #openstack-ansible17:06
*** macza has quit IRC17:07
chandankumarcan someone please have a look when free thanks :-)17:07
*** radeks__ has joined #openstack-ansible17:08
*** radeks_ has quit IRC17:10
chandankumarodyssey4me: regarding this failure http://logs.openstack.org/00/627500/67/check/tripleo-ci-centos-7-standalone-os-tempest/a076a67/logs/stackviz/#/testrepository.subunit having old venv is it good to use here venv_rebuild: yes?17:20
chandankumarodyssey4me: sorry this link http://logs.openstack.org/82/627782/4/check/openstack-ansible-upgrade-aio_lxc-ubuntu-bionic/5430fd4/logs/ara-report/result/93303b7a-2400-409a-96cc-090af9623497/17:20
odyssey4mechandankumar if that venv was built by the role, then that won't help I don't think, because it's fresh at the start17:22
chandankumarbut old tempest packages are conflicting with the new one17:22
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Remove glance_developer_mode and glance_venv_download  https://review.openstack.org/62034017:24
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: [TEST] Test using integrated build  https://review.openstack.org/61976517:24
odyssey4mechandankumar how? is that an upgrade test? or are you re-using a venv path for different things?17:24
chandankumarodyssey4me: based on this error ImportError: 'test_agents' module incorrectly imported from '/openstack/venvs/tempest-19.0.0.0b1/lib/python2.7/site-packages/tempest/api/compute/admin'. Expected '/openstack/venvs/tempest-18.1.3/lib/python2.7/site-packages/tempest/api/compute/admin'. Is this module globally installed?17:26
odyssey4mechandankumar clearly not, given the path17:26
odyssey4mechandankumar although it looks like somehow there are two paths there17:27
odyssey4mechandankumar I'm confused though, I thought you were providing your own path in that test?17:27
chandankumarodyssey4me: it is related to this upgrade job getting added https://review.openstack.org/#/c/62778217:28
odyssey4mechandankumar oh - this isn't the tripleo test...17:30
chandankumarodyssey4me: tripleo one is working fine now17:30
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_cinder master: Ensure create a volume from image  https://review.openstack.org/63251917:30
chandankumarodyssey4me: for tempest test failure, my local setup failed due to venv request depenecy issue on centos17:31
guilhermespchandankumar: odyssey4me just as reference http://logs.openstack.org/82/627782/4/check/openstack-ansible-upgrade-aio_lxc-ubuntu-bionic/5430fd4/logs/ara-report/result/93303b7a-2400-409a-96cc-090af9623497/17:32
*** radeks__ has quit IRC17:33
odyssey4mechandankumar guilhermesp I suspect that the workspace move might be causing that? Perhaps the workspace somehow has references to the old path?17:33
guilhermespI though this was caused by the workspace path that we fixed...but I tried so hard to find an explanation that I ended up excluding this possibility17:33
guilhermespthe thing is that we move the workspace and then we do an init. Not sure if there was "thrash " in there17:34
chandankumarodyssey4me: but workspace is just a seperate directory there is no python dependency installed there17:34
odyssey4meguilhermesp chandankumar oh hang on, the result of that workspace move task was 'OK', nnot 'Changed'17:34
odyssey4mewell, I guess the only way to unwind that would be to build an AIO the same way and look at it17:36
odyssey4mebuild a vm with 8vcpu/8gb ram and enough disk space - then run: ./scripts/gate-check-commit.sh aio_lxc upgrade17:36
guilhermespcould be an option odyssey4me I'm not able to do it right now, but if it is not urgent I can find some spare time to do it17:39
guilhermespjust don't know when...I'm already outdated with some production deployments17:39
guilhermespmaybe a fix in the workspace patch? maybe another pr? not sure but the workspace fix is really something that we needed to do...the tests I did was only moving workspaces and stuff17:40
odyssey4mejrosser any chance I could get your review for https://review.openstack.org/624773 this week?17:40
jrosserodyssey4me: oh sorry :/ i started looking at this so many times.......17:41
jrosseri shall do it right now17:41
odyssey4mejrosser it's quite a bit of stuff, so perhaps first thing in the morning with fresh eyes... but hey, I'll take any eyes right now ;)17:42
jrosseryeah - it's heavy enough that time-to-interruption > time-to-review17:42
jrosseror <17:42
jrosserperhaps that means look at it in the morning :)17:43
odyssey4memost of it is just junk removal, but yeah - it's not an easy review... sorry :/17:44
openstackgerritMerged openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218217:46
*** strattao has quit IRC17:46
*** asettle has quit IRC17:46
*** radeks has joined #openstack-ansible17:47
*** nurdie has joined #openstack-ansible17:47
nurdieController cluster with containerized services, neutron-l3-agent shows this: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-keepalived-state-change" failed: No such file or directory17:48
nurdieThat executable definitely exists in the /openstack/venvs/.../bin directory17:48
nurdieIn the trace, "File "/openstack/venvs/neutron-16.0.23/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 374, in spawn_state_change_monitor"17:49
nurdieWhat can I edit to let python see that in its path?17:49
*** radeks has quit IRC17:50
nurdieControllers are running Pike17:54
openstackgerritMerged openstack/openstack-ansible stable/pike: Update ceph galaxy cleanup for monolithic repo  https://review.openstack.org/63246018:00
*** pcaruana has joined #openstack-ansible18:01
odyssey4menurdie it looks like this may be related? https://review.openstack.org/#/c/630374/18:04
odyssey4mealthough I'm not sure that solution exactly will work for pike18:04
jrosserodyssey4me: i also see that on rocky - trying to find time to road test that patch18:05
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: [DNM] public subnet allocation pool  https://review.openstack.org/63250818:05
*** priteau has quit IRC18:05
nurdieodyssey4me: I added /openstack/venvs/neutron-16.0.23/bin to root's ENV, just in case it was a sudo ran command or something, but that didn't help18:06
odyssey4menurdie it won't if it's a systemd-unit, which it is18:08
nurdieI added Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/openstack/venvs/neutron-16.0.23/bin" to /etc/systemd/system/neutron-l3-agent.service18:16
*** aedc has quit IRC18:16
nurdieStill no dice. Same thing in the l3-agent log18:16
*** aedc has joined #openstack-ansible18:21
jenglisch_nurdie: you could add it to /etc/environment18:21
jenglisch_my proposed fix https://review.openstack.org/#/c/630374/ should work as well however i haven't yet time to fully test it, hence it is a bit stalled, and i initially also added it to /etc/environment18:23
*** pcaruana has quit IRC18:23
jenglisch_imo the systemd unit should be fine, but probably the venv loading overrides it18:24
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [TEST] Remove lock on repo for source installs  https://review.openstack.org/63252618:25
*** gkadam has quit IRC18:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: [TEST] Test using integrated build  https://review.openstack.org/61976518:26
nurdiewooowwww jenglisch_ thank you so much! The /etc/environment update fixed my whole networking issue. Services were flapping and pretty unreliable until just now18:32
*** aedc has quit IRC18:32
nurdieNope nevermind I'm still seeing flapping lol18:33
nurdieBut the logs sure got a lot quieter18:33
*** shardy has quit IRC18:34
*** kopecmartin is now known as kopecmartin|off18:38
*** TxGirlGeek has quit IRC18:48
*** aedc has joined #openstack-ansible18:49
nsmedsIs someone familiar with how/where `/etc/openstack_deploy/openstack_hostnames_ips.yml` is referenced?18:51
nsmedsI understand it keeps a mapping of names->IPs, so Ansible SSH's into the container's host and then runs lxc-attach commands.18:52
nsmedsbut unsure where Ansible is being told to do so18:52
jenglisch_nurdie: i'd suggest you to restart your whole networking node18:57
jenglisch_there may be "orphan" netns18:57
nurdieJust the container or the whole controller?18:57
jenglisch_l3-agent is not running in a container18:57
nurdieYessir it is18:57
jenglisch_you can also stop the l3-agent and cleanup all netns namespaces18:58
jenglisch_and if you have more then one controller, you need to fix all of them18:58
jenglisch_otherwise you will get duplicate ip problems18:58
jenglisch_since all keepalive instaces may think they're the only one18:58
nurdieYeap. A soon as I made the /etc/environment change, all of the neutron log files went quiet. Right now we have 1 of 3 controllers completely dead in the water, which is probably causing some of the issues18:58
*** TxGirlGeek has joined #openstack-ansible18:59
nurdieOur controllers are swapping hard right now though. Mostly used up by nova-api-os-compute, mysqld, cinder, and cinder-api19:00
*** DanyC_ has joined #openstack-ansible19:01
jrosser nsmeds the openstack-ansible wrapper calls openstack-ansibke.rc which sets a ton of environment that brings in a special connection plugin to connect to containers via their host19:01
jenglisch_it's also not that long and just a shell script, so easily readable, i'd suggest you to read (and understand it)19:02
jenglisch_if you have ansible exprience19:02
jrosserThere is data in the dynamic inventory (inventory/dynamic_inventory.py iirc) which includes a physical_host hostvar for containers so the connection plugin knows which host to got to for a particular container19:03
jenglisch_and OSA connects to the LXC container via sshd (running in the container) directly19:04
jrosserI think it actually uses nsenter or similar approach rather than ssh19:04
*** DanyC has quit IRC19:05
jenglisch_lxc-attach is only used for bootstrapping (and also there it has been replaced by nsenter)19:05
nsmedsah, I think I see my mistake - so for context, I'm working on https://github.com/openstack/openstack-ansible-ops/tree/master/elk_metrics_6x#deploying--manually-resolving-the-dependencies19:05
nsmedswhat is the "config_template action directory" ?19:05
jrosserRight - the elk deployment relies on an ansible version that is potentially newer than your osa ansible version19:06
nsmedsyep, used embedded ansible for that reeason19:07
jrosserAnd a corresponding set of ansible plugins19:07
jrosserThe embedded ansible setup should download the right version of config_tenplate and set up env vars so that it is used when the embedded ansible is activated19:08
jrosserCheck the output from ‘env’19:08
nsmedsyep, found that snippet from bootstrap_embedded, and env variable appears correctly set19:09
nsmedsANSIBLE_ACTION_PLUGINS=/root/ansible_venv/repositories/roles/config_template/action19:09
nsmedsi'll look there19:09
jrossernsmeds: you should take a look at this too https://review.openstack.org/#/c/631668/ the way the inventory integration for elk works is likely to get a refactor like that very soon19:10
*** TxGirlGeek has quit IRC19:11
*** DanyC_ has quit IRC19:13
*** DanyC has joined #openstack-ansible19:14
*** TxGirlGeek has joined #openstack-ansible19:14
*** sreejithp has joined #openstack-ansible19:14
*** DanyC has quit IRC19:18
nsmedsok, appreciate it guys - will browse through these files some more19:20
*** strattao has joined #openstack-ansible19:22
*** Darcidride has joined #openstack-ansible19:27
cjloaderthx jrosser for your help this morning19:31
jrossercjloader: you’re welcome :)19:33
cjloaderseems like pike/ocata are fixed now19:33
jamesdentonQQ - for the repo server, what dictates the checked out branch for a given project?19:35
jamesdentonlike, /var/www/repo/openstackgit/nova detached at a particular commit19:38
cjloaderjamesdenton: I think ansible-role-reqs19:39
cjloaderhttps://github.com/openstack/openstack-ansible/blob/master/ansible-role-requirements.yml#L113-L11619:40
jamesdentoni think that applies to the OSA roles themselves19:40
*** bgmccollum has quit IRC19:40
cjloaderyes, but it points to a specific sha within a role19:41
*** aedc has quit IRC19:42
jamesdentonwell, what's interesting, is that on this AIO master deploy, nova is detached at https://github.com/openstack/nova/commit/c52c2caf97a90de8770e9296d553e0e4a65ac946, from Aug 201819:45
jrosserjamesdenton: maybe this? https://review.openstack.org/#/c/630430/2/playbooks/defaults/repo_packages/openstack_services.yml19:45
jamesdentonjust trying to figure out if i'm doing something wrong somewhere, or what the deal is19:45
*** TxGirlGeek has quit IRC19:46
jamesdentonyep that's it jrosser. thank you19:46
jamesdentonand cjloader19:46
jrosserjamesdenton: see that it's all set to master version for the services on the osa master branch https://github.com/openstack/openstack-ansible/blob/master/playbooks/defaults/repo_packages/openstack_services.yml19:48
jrosserthey all get pinned to specific sha on the release branches19:49
jamesdentonbut the SHA is what's used, even on an OSA 'master' deploy, right?19:50
*** pcaruana has joined #openstack-ansible19:50
jrossernow you come to ask, i'm not sure :)19:51
*** ztr has joined #openstack-ansible19:51
jamesdentonwell, that's what i'm seeing, whether that's intended or not :P19:52
jamesdentoni'm sure i've asked it before. lol. thanks again19:52
cjloaderwhen the branch is pulled19:53
cjloaderthe latest commit19:53
*** TxGirlGeek has joined #openstack-ansible19:58
*** Darcidride has quit IRC20:08
*** pcaruana has quit IRC20:11
*** bgmccollum has joined #openstack-ansible20:12
*** ztr has quit IRC20:15
*** Darcidride has joined #openstack-ansible20:17
*** pcaruana has joined #openstack-ansible20:24
*** bgmccollum has quit IRC20:26
*** bgmccollum has joined #openstack-ansible20:27
FrankZhangcloudnull: hey man, do we have any card tracking down centOS gating ssh failing?20:33
*** strattao has quit IRC20:38
jrosserlogan-: regarding this http://logs.openstack.org/31/632131/2/check/openstack-ansible-functional-ubuntu-xenial/9addd27/job-output.txt.gz#_2019-01-22_14_37_40_350931 - etcd.default.j2 tries to look etcd_advertise_address up in hostvars, where it will be found if it is overridden, but not if the role default should be used?20:40
*** aedc has joined #openstack-ansible20:44
*** dave-mccowan has quit IRC20:48
*** dave-mccowan has joined #openstack-ansible20:51
jamesdentonjrosser https://github.com/logan2211/ansible-etcd/tree/template-fix21:03
jamesdentonhe just needs some +1s on that to merge it21:03
jrosseri wasnt totally sure why it doesnt use etcd_advertise_address directly? what am i missing?21:04
jamesdentoni dunno. that's what i ended up doing to fix my local instance21:04
*** mgariepy has quit IRC21:06
*** strattao has joined #openstack-ansible21:06
jamesdentonusing etcd_advertise_address directly, that is21:06
*** bgmccollum has quit IRC21:16
*** pcaruana has quit IRC21:19
*** hwoarang has quit IRC21:21
*** hwoarang has joined #openstack-ansible21:22
*** ztr has joined #openstack-ansible21:40
*** priteau has joined #openstack-ansible21:41
*** ansmith has quit IRC21:42
*** strattao has quit IRC21:42
nurdieIn 16.0.23, in task TASK [repo_build : Clone git repositories] of "repo-build.yml -e repo_build_git_reclone=yes -e repo_build_wheel_rebuild=yes -e repo_build_venv_rebuild=yes", /openstackgit/designate/requirements.txt requires keystoneauth1>3.4.0 which breaks the repo-build every time?21:52
nurdiekeystoneauth1>=3.4.0 *21:52
nurdieEverything else is keystoneauth1==3.1.0 or even keystoneauth1<=3.1.022:03
*** strattao has joined #openstack-ansible22:04
*** priteau has quit IRC22:08
*** DanyC has joined #openstack-ansible22:19
*** DanyC has quit IRC22:22
*** DanyC has joined #openstack-ansible22:23
*** TxGirlGeek has quit IRC22:24
*** nurdie has quit IRC22:25
*** TxGirlGe_ has joined #openstack-ansible22:25
*** nurdie has joined #openstack-ansible22:25
*** DanyC has quit IRC22:27
*** DanyC_ has joined #openstack-ansible22:27
*** nurdie has quit IRC22:30
*** TxGirlGe_ has quit IRC22:31
*** Darcidride has quit IRC22:32
*** sreejithp has quit IRC22:35
logan-jrosser: some ansible quirk with how vars are resolved causes that weirdness I think22:38
*** TxGirlGeek has joined #openstack-ansible22:40
*** DanyC_ has quit IRC22:48
*** mmercer has quit IRC22:50
*** DanyC has joined #openstack-ansible22:50
*** bgmccollum has joined #openstack-ansible22:51
*** DanyC has quit IRC22:55
*** DanyC has joined #openstack-ansible22:56
*** TxGirlGeek has quit IRC22:59
*** TxGirlGeek has joined #openstack-ansible23:04
*** TxGirlGeek has quit IRC23:35
*** strattao has quit IRC23:42
*** mmercer has joined #openstack-ansible23:46

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!