Wednesday, 2020-08-26

*** slaweq_ has joined #openstack-qa00:10
*** slaweq has quit IRC00:12
*** yamamoto has quit IRC00:14
*** rfolco has quit IRC00:19
*** jparker_ has quit IRC00:24
*** jparker_ has joined #openstack-qa00:24
*** paras333 has joined #openstack-qa00:35
*** paras333 has quit IRC01:02
*** paras333 has joined #openstack-qa01:02
openstackgerritDoug Schveninger proposed openstack/patrole master: Clean up tox.ini and document  zuul.yaml settings  https://review.opendev.org/74780601:39
*** Liang__ has joined #openstack-qa01:41
openstackgerritDoug Schveninger proposed openstack/patrole master: Clean up tox.ini and document  zuul.yaml settings  https://review.opendev.org/74780601:46
openstackgerritDoug Schveninger proposed openstack/patrole master: Clean up tox.ini and document  zuul.yaml settings  https://review.opendev.org/74780601:59
*** Liang__ has quit IRC02:36
*** Liang__ has joined #openstack-qa02:37
*** yamamoto has joined #openstack-qa02:43
*** paras333 has quit IRC02:46
*** rh-jelabarre has quit IRC03:08
*** rcernin has quit IRC03:25
*** rcernin_ has joined #openstack-qa03:25
*** psahoo has joined #openstack-qa03:31
*** gcheresh has joined #openstack-qa03:49
*** Liang__ has left #openstack-qa03:56
*** Liang__ has quit IRC03:57
*** dave-mccowan has quit IRC04:07
*** gcheresh has quit IRC04:14
*** dmellado has quit IRC04:31
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-qa04:33
*** psahoo has quit IRC04:35
*** psahoo has joined #openstack-qa05:00
*** psahoo has quit IRC05:11
*** sridharg has joined #openstack-qa05:21
*** sridharg has quit IRC05:22
*** sridharg has joined #openstack-qa05:23
*** psahoo has joined #openstack-qa05:27
*** raukadah is now known as chandankumar05:44
*** redrobot has quit IRC05:53
*** psahoo has quit IRC05:55
*** psahoo has joined #openstack-qa06:12
*** lpiwowar has joined #openstack-qa06:14
*** slaweq_ has quit IRC06:14
*** slaweq_ has joined #openstack-qa06:35
*** ralonsoh has joined #openstack-qa06:42
*** slaweq_ is now known as slaweq06:48
*** vishalmanchanda has joined #openstack-qa06:49
openstackgerritVishal Manchanda proposed openstack/stackviz master: migrate to victoria testing runtime  https://review.opendev.org/74036906:57
*** brinzhang_ has quit IRC06:59
*** brinzhang has joined #openstack-qa07:03
*** lpiwowar has quit IRC07:11
*** lpiwowar has joined #openstack-qa07:12
*** lpiwowar has quit IRC07:29
*** lpiwowar has joined #openstack-qa07:29
*** lpiwowar has quit IRC07:34
*** lpiwowar has joined #openstack-qa07:34
*** lpiwowar has quit IRC07:39
*** tosky has joined #openstack-qa07:40
*** lpiwowar has joined #openstack-qa07:40
*** lpiwowar has quit IRC07:40
*** lpiwowar has joined #openstack-qa07:40
*** lpiwowar has quit IRC07:44
*** lpiwowar has joined #openstack-qa07:44
*** lpiwowar has quit IRC07:49
*** lpiwowar has joined #openstack-qa07:49
*** Luzi has joined #openstack-qa07:55
*** lpiwowar_ has joined #openstack-qa08:08
*** lpiwowar has quit IRC08:11
*** lpiwowar_ has quit IRC08:11
*** lpiwowar has joined #openstack-qa08:11
*** lpiwowar has quit IRC08:12
*** lpiwowar has joined #openstack-qa08:13
*** lucasagomes has joined #openstack-qa08:16
*** dosaboy has quit IRC08:17
*** dosaboy has joined #openstack-qa08:17
*** tosky has quit IRC08:32
*** simondodsley has quit IRC08:32
*** tosky has joined #openstack-qa08:33
openstackgerritTom Stappaerts proposed openstack/devstack stable/train: Fix uwsgi install from source  https://review.opendev.org/74798708:34
*** simondodsley has joined #openstack-qa08:38
*** adriant has quit IRC08:48
*** adriant has joined #openstack-qa08:48
*** gfidente has joined #openstack-qa08:50
openstackgerritLucas Alvares Gomes proposed openstack/devstack master: [OVN] Follow up of OVN module migration to DevStack  https://review.opendev.org/74814008:52
*** TomStappaerts has quit IRC08:52
openstackgerritLucas Alvares Gomes proposed openstack/devstack master: DO NOT REVIEW YET: [OVN] Change Neutron's default ML2 driver to OVN  https://review.opendev.org/73509708:53
*** TomStappaerts has joined #openstack-qa08:58
*** TomStappaerts has quit IRC08:58
*** dtantsur|afk is now known as dtantsur09:09
*** yamamoto has quit IRC09:27
*** rcernin_ has quit IRC09:39
*** yamamoto has joined #openstack-qa09:49
*** sridharg has quit IRC09:54
*** cgoncalves has quit IRC10:14
*** sridharg has joined #openstack-qa10:26
*** cgoncalves has joined #openstack-qa10:31
openstackgerritMichał Dulko proposed openstack/devstack master: Allow extra servicefile options in run_process  https://review.opendev.org/74815810:41
*** Luzi has quit IRC10:42
openstackgerritVishal Manchanda proposed openstack/stackviz master: migrate to victoria testing runtime  https://review.opendev.org/74036910:42
*** sridharg has quit IRC10:53
*** yamamoto has quit IRC11:10
*** yamamoto has joined #openstack-qa11:44
*** raildo has joined #openstack-qa11:44
*** rh-jelabarre has joined #openstack-qa11:50
*** yamamoto has quit IRC11:51
openstackgerritTom Stappaerts proposed openstack/devstack stable/train: Fix uwsgi install from source  https://review.opendev.org/74798712:13
openstackgerritSoniya Murlidhar Vyas proposed openstack/tempest master: Use of single interface for creating image  https://review.opendev.org/74649212:26
openstackgerritLee Yarwood proposed openstack/devstack-plugin-ceph master: zuul: Add devstack-plugin-ceph-compute-qcow2 job  https://review.opendev.org/74821212:33
openstackgerritDoug Schveninger proposed openstack/patrole master: fix patrole-multinode-member non-voting gate by skip_because a bug  https://review.opendev.org/74752812:47
openstackgerritLee Yarwood proposed openstack/devstack-plugin-ceph master: zuul: Add devstack-plugin-ceph-compute-qcow2 job  https://review.opendev.org/74821212:50
*** icey has joined #openstack-qa12:53
iceyhey - I wondered if I could get a review on https://review.opendev.org/#/c/747690/ ? In essence, it would be great to be able to install tempest with Ubuntu Xenial's system python3 :)12:54
*** paras333_ has joined #openstack-qa12:59
openstackgerritStephen Finucane proposed openstack/tempest master: Remove deprecated nova scheduler filters  https://review.opendev.org/74822113:03
*** bhagyashris is now known as bhagyashris|away13:06
svyaskopecmartin, I guess you have reviewed earlier patchset, can you please review https://review.opendev.org/#/c/724631/ this once again?13:11
openstackgerritLee Yarwood proposed openstack/devstack-plugin-ceph master: zuul: Add devstack-plugin-ceph-compute-qcow2 job  https://review.opendev.org/74821213:12
*** dave-mccowan has joined #openstack-qa13:17
openstackgerritMichał Dulko proposed openstack/devstack master: Allow extra servicefile options in run_process  https://review.opendev.org/74815813:19
*** nweinber has joined #openstack-qa13:23
*** rfolco has joined #openstack-qa13:34
openstackgerritLukas Piwowarski proposed openstack/tempest master: Fix import line location  https://review.opendev.org/74490413:34
*** gfidente has quit IRC13:35
*** paras333_ has quit IRC13:38
slaweqhi devstack cores13:40
slaweqcan You check https://review.opendev.org/#/c/748140/ as this is blocking neutron gate currently :)13:40
slaweqthx in advance13:40
*** brinzhang has quit IRC13:41
*** brinzhang has joined #openstack-qa13:41
openstackgerritSoniya Murlidhar Vyas proposed openstack/tempest master: Need to have stable tempest scenario manager  https://review.opendev.org/72463113:48
*** yamamoto has joined #openstack-qa13:48
*** brinzhang has quit IRC13:48
*** brinzhang has joined #openstack-qa13:48
*** Liang__ has joined #openstack-qa13:49
*** Liang__ is now known as LiangFang13:50
svyaskopecmartin, I have revised the patch https://review.opendev.org/#/c/724631/13:52
kopecmartinsvyas: thanks, 724631 looks good .. please don't put there any more changes - the biggest change the hardest to review - let's use a new review per every next method which requires a modification13:52
svyaskopecmartin sure :)13:53
*** andrebeltrami has joined #openstack-qa13:56
*** yamamoto has quit IRC13:58
*** LiangFang has left #openstack-qa14:01
*** gfidente has joined #openstack-qa14:10
openstackgerritEric Harney proposed openstack/devstack-plugin-ceph master: DNM: Octopus test run  https://review.opendev.org/74824514:25
fricklerslaweq: approved, but I'm worried as to why this wasn't noticed earlier, I though there had been a depending patch for neutron to test things14:25
slaweqfrickler: thx a lot for quick review14:26
*** lpiwowar_ has joined #openstack-qa14:30
*** lpiwowar has quit IRC14:32
*** lpiwowar_ has quit IRC14:34
*** lpiwowar has joined #openstack-qa14:35
openstackgerritParas Babbar proposed openstack/whitebox-tempest-plugin master: Test selinux labels in QEMU/KVM environment  https://review.opendev.org/74596114:35
*** lpiwowar has joined #openstack-qa14:36
*** dustinc has joined #openstack-qa14:37
*** tkajinam has quit IRC14:57
*** lpiwowar has quit IRC15:05
*** psahoo has quit IRC15:05
openstackgerritMichał Dulko proposed openstack/devstack master: Allow extra servicefile options in run_process  https://review.opendev.org/74815815:09
*** TomStappaerts has joined #openstack-qa15:30
TomStappaertsHi everyone, I've been trying to fix centos7 bring up in CI but it seems like larger changes are needed. eg. An outdated version of setuptools is used? https://review.opendev.org/#/c/747987/4 Anyone any ideas?15:31
clarkbTomStappaerts: newer setuptools will fix 'error in setup command: Error parsing /opt/stack/keystone/setup.cfg: SyntaxError: '>=' operator not allowed in environment markers' ?15:35
TomStappaertsclarkb: Yes I believe so, python-setuptools version is only 0.915:36
clarkbTomStappaerts: looks like your change is making it so that centos stops uninstalling pip and setuptools and only does that on fedora?15:36
clarkbif you keep that in won't install_pip.sh update the versions to latest for you?15:36
TomStappaertsclarkb: Am I misreading my change? The purpose was that on fedore as before it does not execute the uninstall but on centos/rhel it does?15:37
clarkbI was looking at logs (where it definitely doesn't seem to upgrade) and may have misparsed the intent of the chang ethere due to the observed behavior15:38
clarkbI think the actual issue may be pip for python2 vs python3?15:40
TomStappaertsThat is possible, I believe on centos7 everything still happens under python2 right?15:40
clarkb`sudo -H -E python${PYTHON3_VERSION} $LOCAL_PIP` is how get-pip.py is run and that is explicitly with python315:41
clarkboh except this is for train15:42
clarkbTomStappaerts: https://zuul.opendev.org/t/openstack/build/9130517dc777458c957fc119f32de76b/log/job-output.txt#407315:42
clarkband now I remember the reason for the pip cap is pip 10 doesn't know how to uninstall distutils installed packages on the root system which wasn't worked around until newer branches15:43
clarkbhttps://zuul.opendev.org/t/openstack/build/9130517dc777458c957fc119f32de76b/log/job-output.txt#4238 and that is why setuptools doesn't update15:44
clarkbI think you need to update global-requirements to have setuptools install. Or uninstall setuptools explicitly15:44
clarkbTomStappaerts: add an uninstall for the setuptools packages to https://review.opendev.org/#/c/747987/4/tools/install_pip.sh too?15:45
TomStappaertsPatchset 5 uninstalls setuptools15:45
TomStappaertsclarkb: But it also uninstalls virtualenv then15:45
clarkbprobably because virtualenv depends on setuptools15:46
TomStappaertsindeed15:46
clarkbin that case updated global-requirements may be the best bet15:46
clarkbyou can use a depends-on to test it all pre merge15:46
clarkb(add a >$version to the global-requirements line15:46
TomStappaertsSo which version of setuptools should we require for train?15:46
clarkbor change keystone's env markers15:46
TomStappaertsI believe those env markers are very common15:47
clarkbyes env markers are fine. The problem is the >= in the env marker according to the error message15:47
clarkbI would use the first version of setuptools that supports the >= in an env marker15:47
openstackgerritMerged openstack/devstack master: [OVN] Follow up of OVN module migration to DevStack  https://review.opendev.org/74814015:47
clarkb(the alternative is to update the env marker to remove the >=)15:47
TomStappaertsOk I will find that out, thx for the input :)15:47
clarkbTomStappaerts: I think train 2.7 was also tested on ubuntu bionic. The version of setuptools there may serve as a hint15:48
clarkb(though I think you can sort of binary search tree on some keystone installs to find where it changed)15:48
*** yamamoto has joined #openstack-qa15:55
rpittauhello everyone! Any chance I can get the size of the files/dir under /opt at the end of a devstack run in CI? I tried converting a job from bionic to focal and it's taking 40GB more!16:00
clarkbrpittau: yes, just add a du -s /path1 /path2 /path3 to your change and log that16:00
*** lucasagomes has quit IRC16:01
rpittauclarkb: thanks, uhmmm, where to add that? :)16:03
clarkbrpittau: at the end of the run playbooks for your job probably16:04
*** yamamoto has quit IRC16:05
rpittauclarkb: ok, so I will have probably have to add a playbook like we do for tftpd logs, thanks!16:06
*** artom has quit IRC16:10
*** dosaboy has quit IRC16:19
*** dosaboy has joined #openstack-qa16:19
openstackgerritBrian Haley proposed openstack/devstack master: Add support for IPv6 tunnel endpoints  https://review.opendev.org/71051916:25
*** paras33__ has joined #openstack-qa16:26
*** paras33__ is now known as paras33316:26
*** yamamoto has joined #openstack-qa16:38
*** yamamoto has quit IRC16:47
*** ralonsoh has quit IRC16:51
*** artom has joined #openstack-qa16:56
*** dtantsur is now known as dtantsur|afk16:57
*** gfidente has quit IRC17:03
*** icey has quit IRC17:19
*** icey has joined #openstack-qa17:24
*** artom has quit IRC17:45
*** brinzhang_ has joined #openstack-qa18:21
*** brinzhang has quit IRC18:25
Open10K8SHi team. The tempest is failing today. It was ok yesterday. test_image_signing.ImageSigningTest.test_signed_image_upload_boot_failure|test_volume_encryption.VolumeEncryptionTest.test_encrypted_cinder_volumes_cryptsetup18:33
Open10K8SI am facing "The image cannot be deleted because it is in use through the backend store outside of Glance"18:34
Open10K8Shttps://zuul.opendev.org/t/vexxhost/build/f800c44a785f4cfb981d19969775b7ac18:34
Open10K8Sseems any upstream repo(devstack or tempest) updated?18:34
*** gcheresh has joined #openstack-qa18:52
clarkbOpen10K8S: thats normal for ceph backed images with boot from volume18:58
clarkbOpen10K8S: if you boot an instance off the image it cannot be deleted until the instance is deleted if using ceph and boot from volume18:58
Open10K8Sclarkb: so you mean that tempest must be in blacklist?18:59
Open10K8Sclarkb: the thing is that happened from today18:59
clarkbno, you need to ensure the instance is deleted first18:59
clarkbit could be a race?18:59
Open10K8Ssorry?19:00
clarkbI don't think this is a new issue19:00
clarkbwe've had this problem with nodepool and vexxhost for years. It is an expected behavior19:00
clarkbI don't know why the test has suddenly started failing, perhaps it is a code race with resource cleanup?19:01
Open10K8Shm..19:01
Open10K8Sclarkb: https://zuul.opendev.org/t/vexxhost/build/f800c44a785f4cfb981d19969775b7ac/log/job-output.txt#22077419:07
Open10K8Scan you check this?19:07
Open10K8Shttps://zuul.opendev.org/t/vexxhost/build/f800c44a785f4cfb981d19969775b7ac/log/job-output.txt#22078419:08
Open10K8Sclarkb: seems instance already deleted19:08
clarkbthat link crashed my desktop. I think firefox started swapping andoomkiller mustbe running19:09
clarkbwill be a bit while I let it recover19:09
*** vhari has joined #openstack-qa19:11
*** vhari has quit IRC19:13
clarkbOpen10K8S: can you paste the relevant bits?19:14
*** vhari has joined #openstack-qa19:15
openstackgerritParas Babbar proposed openstack/hacking master: Move the most common hacking rules from other projects  https://review.opendev.org/72669619:16
mnaserclarkb: the issue we're talking about here is an issue inside tempest failing19:23
mnaserso not hitting our cloud, but actually hitting the a cloud inside a vm in nodepool19:23
mnaserso that failure is a tempest test failure that's talking to an openstack-cloud-in-a-vm and not a failure talking to the vexxhost cloud itself19:24
clarkbmnaser: right but is that cloud doing ceph with boot from volume?19:24
clarkbfrom the test name it seems it is likely doing at least boot from volume19:25
mnaserclarkb: correct, but the tests themselves were passing for weeks now and started failing mid-day today (and we gate against tempest so i doubt we did something..)19:25
mnaserso i think Open10K8S was curious if glance/tempest/etc had a change that might have affected that failure19:25
clarkbright, I'm just pointing out that that behavior has existed for years19:26
clarkbit would not surprise me if it is just a timing issue19:26
mnaseri think the nodepool failures are probably because when nodepool tries to rm an image that a nodepool vm is still using19:27
clarkbbut I can't look at the logs as my machine locks up while oomkiller finds a sacrifice19:27
mnaserwe have a patch to reduce log output but unfortunately19:29
mnaserthat cant land bc of this19:30
mnaserhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f80/748222/5/gate/openstack-operator:functional/f800c44/job-output.txt is a little easier on the browser19:30
mnaserour failing test is: barbican_tempest_plugin.tests.scenario.test_certificate_validation.CertificateValidationTest.test_signed_image_invalid_cert_boot_failure[compute,id-6d354881-35a6-4568-94b8-2204bbf67b29,image]19:30
mnaserOpen10K8S: that test comes in from https://opendev.org/openstack/barbican-tempest-plugin19:31
mnaserwhich seems to have gotten a commit yesterday19:31
clarkbdo multiple tests use the same image?19:33
mnaserclarkb: https://opendev.org/openstack/barbican-tempest-plugin/commit/b8437a2f41a36da410c68157078ce89cd2e8e3cc seems to be the commit19:33
mnaseri think maybe the create_server is not cleaning up by deleting the vm19:34
clarkbya looks like each test does a separate image upload whihc should result in unique images19:36
clarkband that one test only asserts the create fails19:36
*** tosky has quit IRC19:38
mnaserclarkb: filtered out hte logs - http://paste.openstack.org/show/797186/19:42
mnaserlooks like the instance is not found.. somehow?19:42
mnasernvm, it does a delete that returns a 204 ok19:43
openstackgerritTom Stappaerts proposed openstack/devstack stable/train: Fix uwsgi install from source  https://review.opendev.org/74798719:44
clarkbmnaser: any idea where the rbd logs go?19:47
clarkbhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f80/748222/5/gate/openstack-operator:functional/f800c44/controller/logs/ceph/ is the ceph logs but nothing looks like rbd specific (and grepping the image uuid shows nothing in those files)19:47
clarkbbut if hte instance is being deleted that does make me think it could still be a race19:48
mnaserclarkb: yeah rbd doesnt really have logs for create/delete but we can check n-cpu logs to verify this theory19:48
mnaserhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f80/748222/5/gate/openstack-operator:functional/f800c44/controller/logs/screen-n-cpu.txt19:49
mnaserterminating request @ Aug 26 18:01:15.39316719:49
mnaserimage delete was at 2020-08-26 18:01:20,12319:51
clarkbalso is the request to rbd async maybe?19:51
mnaserbut i dont actually see it deleting an rbd image in the neutron cpu logs19:51
clarkbya I wonder if that is it, nova batches them up ro something19:52
mnaserthe terimation is done at Aug 26 18:01:18.15052519:52
mnaserso i think this is nova actually not cleaning up after itself19:52
*** artom has joined #openstack-qa19:55
*** nweinber has quit IRC20:01
*** paras333 has quit IRC20:32
*** lbragstad has quit IRC20:35
*** lbragstad has joined #openstack-qa20:35
*** vishalmanchanda has quit IRC20:37
*** yamamoto has joined #openstack-qa20:44
*** paras333 has joined #openstack-qa20:45
*** yamamoto has quit IRC20:51
*** paras333 has quit IRC20:52
*** paras333_ has joined #openstack-qa21:07
*** gcheresh has quit IRC21:10
*** paras333_ has quit IRC21:12
*** slaweq has quit IRC21:18
*** yamamoto has joined #openstack-qa21:45
*** yamamoto has quit IRC22:00
*** brinzhang0 has joined #openstack-qa22:00
*** raildo_ has joined #openstack-qa22:01
*** raildo has quit IRC22:04
*** brinzhang_ has quit IRC22:04
*** rcernin has joined #openstack-qa22:40
*** tkajinam has joined #openstack-qa23:01
*** yamamoto has joined #openstack-qa23:16
*** brinzhang0 has quit IRC23:22
*** brinzhang0 has joined #openstack-qa23:23

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!