| *** artom has quit IRC | 00:15 | |
| *** rfolco has quit IRC | 00:29 | |
| *** paras333 has quit IRC | 00:38 | |
| *** paras333 has joined #openstack-qa | 01:21 | |
| openstackgerrit | Merged openstack/tempest master: Add response schema validation for volume group snapshots https://review.opendev.org/606772 | 02:03 |
|---|---|---|
| *** paras333 has quit IRC | 02:32 | |
| *** dave-mccowan has quit IRC | 03:14 | |
| *** rcernin has quit IRC | 03:29 | |
| openstackgerrit | Colleen Murphy proposed openstack/tempest master: Add keystone v3 EC2 credential tests and client https://review.opendev.org/744087 | 03:32 |
| *** rcernin has joined #openstack-qa | 03:40 | |
| *** gcheresh has joined #openstack-qa | 03:51 | |
| *** bhagyashris|away is now known as bhagyashris | 03:52 | |
| *** rh-jelabarre has quit IRC | 03:52 | |
| *** rcernin has quit IRC | 03:58 | |
| *** gcheresh has quit IRC | 04:14 | |
| *** rcernin has joined #openstack-qa | 04:15 | |
| *** psahoo has joined #openstack-qa | 04:30 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-qa | 04:33 | |
| *** vishalmanchanda has joined #openstack-qa | 04:41 | |
| *** redrobot has quit IRC | 05:17 | |
| *** markvoelker has joined #openstack-qa | 05:19 | |
| *** markvoelker has quit IRC | 05:23 | |
| *** markvoelker has joined #openstack-qa | 05:26 | |
| *** udesale has joined #openstack-qa | 05:27 | |
| *** markvoelker has quit IRC | 05:30 | |
| *** psahoo has quit IRC | 05:41 | |
| *** udesale has quit IRC | 05:50 | |
| *** psahoo has joined #openstack-qa | 05:54 | |
| *** gcheresh has joined #openstack-qa | 05:59 | |
| *** psahoo has quit IRC | 06:03 | |
| *** udesale has joined #openstack-qa | 06:04 | |
| *** psahoo has joined #openstack-qa | 06:17 | |
| *** gcheresh has quit IRC | 06:28 | |
| *** slaweq has joined #openstack-qa | 07:02 | |
| *** nightmare_unreal has joined #openstack-qa | 07:16 | |
| *** tosky has joined #openstack-qa | 07:19 | |
| *** chaconpiza has joined #openstack-qa | 07:19 | |
| *** ralonsoh has joined #openstack-qa | 07:33 | |
| *** jpena|off is now known as jpena | 07:56 | |
| *** ianw has quit IRC | 08:01 | |
| *** ianw has joined #openstack-qa | 08:02 | |
| *** ccamel has joined #openstack-qa | 08:02 | |
| *** camelCaser has quit IRC | 08:03 | |
| openstackgerrit | Merged openstack/tempest master: Add glance image import web-download tests https://review.opendev.org/742295 | 08:05 |
| *** lucasagomes has joined #openstack-qa | 08:10 | |
| *** dtantsur|afk is now known as dtantsur | 09:19 | |
| *** gcheresh has joined #openstack-qa | 09:58 | |
| *** gcheresh has quit IRC | 10:22 | |
| *** rcernin has quit IRC | 10:33 | |
| *** markvoelker has joined #openstack-qa | 10:41 | |
| *** gcheresh has joined #openstack-qa | 10:43 | |
| *** markvoelker has quit IRC | 10:46 | |
| *** brinzhang_ has joined #openstack-qa | 10:48 | |
| *** brinzhang0 has quit IRC | 10:51 | |
| *** gcheresh has quit IRC | 10:52 | |
| *** jpena is now known as jpena|afk | 10:56 | |
| *** raildo has joined #openstack-qa | 11:11 | |
| *** jpena|afk is now known as jpena | 11:20 | |
| *** brinzhang_ has quit IRC | 11:32 | |
| *** tkajinam has quit IRC | 11:36 | |
| *** jpena is now known as jpena|lunch | 11:48 | |
| *** artom has joined #openstack-qa | 11:55 | |
| *** lbragstad_ has joined #openstack-qa | 12:00 | |
| *** lbragstad has quit IRC | 12:03 | |
| *** rh-jelabarre has joined #openstack-qa | 12:14 | |
| *** rfolco has joined #openstack-qa | 12:24 | |
| *** yamamoto has joined #openstack-qa | 12:24 | |
| *** nightmare_unreal has quit IRC | 12:48 | |
| *** jpena|lunch is now known as jpena | 12:59 | |
| *** dave-mccowan has joined #openstack-qa | 12:59 | |
| svyas | kopecmartin, gmann, we don't use publickey as keypair while creating keypair in scenario manager right? So does it mean, its the private key we are using for creating keypair | 13:02 |
| svyas | masayukig, ^^ | 13:02 |
| *** paras333 has joined #openstack-qa | 13:06 | |
| kopecmartin | svyas: seems not: https://github.com/openstack/tempest/blob/master/tempest/scenario/manager.py#L162 | 13:07 |
| gmann | svyas: yeah we do only in API tests | 13:08 |
| svyas | kopecmartin, gmann, so, is it privatekey then? | 13:08 |
| *** paras333 has quit IRC | 13:09 | |
| *** paras333 has joined #openstack-qa | 13:09 | |
| gmann | svyas: if you pass it in request that will be imported by nova otherwise nova will create a new keypair and return the keys in response | 13:11 |
| kopecmartin | svyas: as i understand is, that in scenario we tell the client to create a key pair with a specific name, but we don't care about the public/private part - so it's randomly generated | 13:12 |
| gmann | yeah | 13:12 |
| *** nweinber has joined #openstack-qa | 13:13 | |
| svyas | kopecmartin,gmann, thanks. I wanted to update the docstring for create_keypair() method and there i got confused with public/private pair | 13:15 |
| gmann | and further, tests fetch the private key by keypair['private_key']. but note, private key is used only when validation.auth_method == 'keypair' otherwise not | 13:16 |
| gmann | svyas: also api-ref can help you to get details about all those API and its parameter. this has most of the projects api-ref https://docs.openstack.org/api-ref/ | 13:17 |
| gmann | these might not be 100% correct :) but if something you find wrong or bug compare to how code behave then feel free to fix also. During Tempest service client or response schema writing, we have found lot of issue in api-ref and fixed also | 13:19 |
| svyas | gmann, thanks. I will go through it :) | 13:21 |
| gmann | svyas: and do not worry or wait to push patches to have 100% perfect /correct doc string. you can even push the draft version and we can input/improve on gerrit if something you stuck at. | 13:22 |
| svyas | gmann, thanks. Next time, I will push the draft version if I come across same thing again | 13:24 |
| gmann | +1 | 13:25 |
| gmann | because it will be easy to review the patch series at same time than with intervals, that's how i did for test desc and volume schema cases. | 13:26 |
| openstackgerrit | Lajos Katona proposed openstack/tempest master: Test for changing QoS policy with min bw of bound port https://review.opendev.org/743695 | 13:33 |
| Open10K8S | Hi team. I have been suffering from unstable infra from last morning. This is the typical example, https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fa4/743943/3/check/openstack-operator:functional/fa4ea09/testr_results.html. Is there any scheduled downtime these days? | 13:36 |
| gmann | Open10K8S: you are facing ssh timeout issues in failed tests there is no schedule downtime. | 13:38 |
| Open10K8S | gmann: is that general one? | 13:39 |
| gmann | Open10K8S: yeah, it happen in many test case and in gate. that can be triggered by multiple reason and it is not 100% fixed in upstream CI/CD. you can get more info from compute logs about failure if that help | 13:40 |
| Open10K8S | gmann: ok, how about this one? https://zuul.opendev.org/t/vexxhost/build/bdf146a0637742a59abd68b91510f435/log/controller/logs/devstacklog.txt#27627 | 13:43 |
| Open10K8S | happened frequently | 13:44 |
| gmann | Open10K8S: its not frequent in openstack gate. it seems keystone not responding. | 13:48 |
| Open10K8S | gmann: keystone was ok, anyway, just wanted to check the infra downtime schedule | 13:50 |
| Open10K8S | gmann: thank you | 13:50 |
| mnaser | gmann: do you know if there's much we can do about ssh timeout or this is something more long standing? :( | 13:53 |
| Open10K8S | gmann: faced again such ssh timeout in tempest. seems becoming more frequent | 13:56 |
| gmann | mnaser: its long standing issue and there was no exact root cause so that we can solve it - https://bugs.launchpad.net/tempest/+bug/1298472 | 13:59 |
| openstack | Launchpad bug 1298472 in OpenStack Compute (nova) "SSHTimeout in tempest trying to verify that computes are actually functioning" [Critical,Fix released] | 13:59 |
| gmann | we did few improvement(and few adjustment) 1. test modified to ping before they ssh the test VM (not all but few of tests were modified) 2. run scenario tests serially | 14:01 |
| gmann | 2nd one helped to minimize it at some extend where lot of ssh parallel request from scenario tests. | 14:01 |
| gmann | mnaser: Open10K8S: do you run scenario tests parallel or serially ? | 14:02 |
| mnaser | gmann: right now we actually just have devstack run it, so whatever upstream devstack jobs do | 14:03 |
| mnaser | (we override specific devstack functions to make them noop as we move that elsewhere so we can have a hybrid devstack+our stuff until we fully finish) | 14:03 |
| Open10K8S | mnaser: serially, ofc | 14:04 |
| *** udesale_ has joined #openstack-qa | 14:05 | |
| gmann | yeah its smoke + heat tests so https://opendev.org/vexxhost/openstack-operator/src/branch/master/zuul.d/functional-jobs.yaml#L32 | 14:05 |
| gmann | test_reboot_server_hard being a smoke tests is another question, this is debatable whether we should remove it from smoke tests or not. | 14:07 |
| mnaser | gmann: we actually had to blacklist one that failed pretty often | 14:07 |
| mnaser | the network basic ops + floating ip was failing quite often | 14:07 |
| mnaser | we were very stable until a few days ago | 14:07 |
| *** udesale has quit IRC | 14:08 | |
| gmann | Open10K8S: it is run in parallel which is default in tempest - https://zuul.opendev.org/t/vexxhost/build/2caf1de31eca485c81d38ae408c38bb8/log/job-output.txt#37209 | 14:13 |
| gmann | mnaser: may be it will be good if you can report the network basic ops + floating ip bugs and we can check we have fixed many FIP bugs and those were reported from 3rd party CI/CD as upstream testing with specific configuration does not catch those always | 14:14 |
| gmann | you are running with --concurrency=4 so that is also good | 14:16 |
| mnaser | gmann: i wonder if we're starved on resources and that's causing the slow down.. | 14:26 |
| gmann | mnaser: that could be one possible issue. | 14:28 |
| mnaser | gmann: it is a much more different architecture in our case where we're doing things like rabbitmq-per-service | 14:29 |
| mnaser | so i guess that's possible | 14:29 |
| clarkb | swapping is also a concern | 14:34 |
| clarkb | itsbeen a while since I looked at abasedevstack job byt for a while they were all swapping whoch has noisy neighbor affects on other jobs too | 14:35 |
| mnaser | yeah we have a lot in an 8gb vm right now | 14:36 |
| mnaser | ceph, multiple 'small' 2 replica db clusters, multiple services (like heat+magnum) | 14:36 |
| mnaser | rabbitmq for every single service (at least 3-4 instances now) | 14:36 |
| kopecmartin | gmann: i wanted to create a new tag for python-tempestconf, but i can't find it in openstack/releases under deliverables/_independent/ where it used to be .. did something change in tag creation process after tempestconf was moved under osf org? | 14:54 |
| gmann | kopecmartin: yeah, we moved that under osf org as that is interop repo | 14:55 |
| kopecmartin | gmann: and how should I create a new tag / where to send a review? | 14:57 |
| *** lbragstad_ is now known as lbragstad | 14:58 | |
| gmann | kopecmartin: you can check with release team on #openstack-release if they can support it otherwise you can push new tag in git itself | 15:00 |
| gmann | sean can help you on that | 15:00 |
| tosky | kopecmartin: iirc you can just push a tag | 15:01 |
| gmann | kopecmartin: tosky do you release it on pypi also ? | 15:02 |
| kopecmartin | really? no review process? | 15:02 |
| kopecmartin | gmann: before it was synced with pypi somehow | 15:02 |
| kopecmartin | we just sent a review to openstack/releases and when it got merged it created a new tag in the repo and in pypi too | 15:03 |
| gmann | kopecmartin: yeah that included the pypi release | 15:05 |
| gmann | kopecmartin: so if you want to continue that, you need to do that also along with new tag on git | 15:05 |
| tosky | kopecmartin: http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-release.2020-02-18.log.html#t2020-02-18T10:46:32 | 15:05 |
| gmann | release team do all those magic for all openstack repo with a simple request to them on gerrit | 15:06 |
| gmann | https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#tagging-a-release | 15:08 |
| gmann | kopecmartin: ^^ that is all you need | 15:09 |
| gmann | kopecmartin: btw where you use python-tempestconf ? refstack is one place, tripleO testing also ? | 15:11 |
| *** chkumar|rover is now known as raukadah | 15:12 | |
| kopecmartin | gmann: yest, tripleo too .. and in downstream | 15:12 |
| kopecmartin | gmann: tosky: so based on this, python-tempestconf-release group is able to push new tags - https://review.opendev.org/#/c/706479/ .. but that group is empty https://review.opendev.org/#/admin/groups/1843,members o.O | 15:13 |
| gmann | ok, did you try this to move under TripleO? i know we in QA has objected to ake under QA projects :) | 15:14 |
| tosky | gmann: I would argue it's more generic than that | 15:14 |
| gmann | kopecmartin: i think you can ask clarkb or someone from infa to add you in python-tempestconf-release as this is not in openstack namespace now | 15:15 |
| gmann | tosky: yeah but who use it own it, that is where we can find long term maintainer | 15:16 |
| tosky | refstack situation should be clarified, that was a strong user too | 15:16 |
| tosky | also I've heard in the past about people using it in their clouds (we had a few patches iirc) | 15:16 |
| gmann | or fungi ^^ (to add kopecmartin tosky in https://review.opendev.org/#/admin/groups/1843,members ) | 15:17 |
| *** yamamoto has quit IRC | 15:18 | |
| gmann | tosky: refstack situation is clear :). its under interop with no maintainer. | 15:18 |
| gmann | situation is communicated to BoD many times. | 15:18 |
| tosky | BoD? | 15:19 |
| tosky | oh, board of directors | 15:19 |
| gmann | Board of Diredctors | 15:19 |
| gmann | directors | 15:19 |
| gmann | tosky: kopecmartin that is why my worry and having python-tempestconf under some OpenStack projects can give it long term maintenance thanbeing under interop (osf) | 15:20 |
| tosky | uhm, but the amount of contributors hasn't changed despite the projects being moved around | 15:21 |
| gmann | tosky: yeah but you need to do all these extra things like releasa etc | 15:21 |
| tosky | right now, with the simplified maintenance (push a tag) it doesn't sound too complicated | 15:22 |
| fungi | tosky: since you're in python-tempestconf-core already i can take that as authority to add you to python-tempestconf-release or i can just set that group to include python-tempestconf-core if you'd prefer | 15:23 |
| fungi | or kopecmartin, you're both in the core group yeah | 15:24 |
| kopecmartin | tosky: it shouldn't be a problem if all cores will have ability to create a tag, right? | 15:25 |
| tosky | kopecmartin: not at all | 15:26 |
| kopecmartin | tosky: good, so fungi please could you give all cores ability to create tags? | 15:26 |
| fungi | mainly they just all need to be aware not to accidentally `git push --tags` or similar | 15:26 |
| clarkb | or push a tag of content not in gerrit | 15:27 |
| fungi | yeah, which is also totally possible | 15:27 |
| clarkb | or push a bad tag and want to delete it | 15:27 |
| kopecmartin | :D | 15:27 |
| fungi | #status log seeded python-tempestconf-release gerrit group by including python-tempestconf-core | 15:28 |
| openstackstatus | fungi: finished logging | 15:28 |
| tosky | thanks! | 15:29 |
| fungi | my pleasure | 15:29 |
| * fungi gets back to doing a terrible job of taking the day off | 15:29 | |
| kopecmartin | thanks | 15:29 |
| fungi | you're welcome! | 15:30 |
| * kopecmartin is gonna study tag creation so that he doesn't break anything :P | 15:31 | |
| *** Guest82019 has joined #openstack-qa | 15:35 | |
| *** Guest82019 is now known as redrobot | 15:37 | |
| *** udesale_ has quit IRC | 15:47 | |
| *** psahoo has quit IRC | 15:56 | |
| *** irclogbot_3 has quit IRC | 15:56 | |
| *** irclogbot_3 has joined #openstack-qa | 15:58 | |
| *** lucasagomes has quit IRC | 16:02 | |
| *** yamamoto has joined #openstack-qa | 16:06 | |
| *** irclogbot_3 has quit IRC | 16:15 | |
| *** irclogbot_0 has joined #openstack-qa | 16:16 | |
| *** yamamoto has quit IRC | 16:20 | |
| openstackgerrit | Brian Rosmaita proposed openstack/devstack stable/pike: Change cinder default branch to pike-eol tag https://review.opendev.org/742952 | 16:24 |
| openstackgerrit | Brian Rosmaita proposed openstack/devstack stable/ocata: Change cinder default branch to ocata-eol tag https://review.opendev.org/742953 | 16:29 |
| *** gcheresh has joined #openstack-qa | 16:45 | |
| *** jpena is now known as jpena|off | 17:01 | |
| *** gcheresh has quit IRC | 17:01 | |
| *** dtantsur is now known as dtantsur|afk | 17:33 | |
| *** ralonsoh has quit IRC | 17:52 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/coverage2sql master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/740363 | 18:03 |
| openstackgerrit | Ghanshyam Mann proposed openstack/eslint-config-openstack master: migrate to victoria testing runtime https://review.opendev.org/740373 | 18:08 |
| *** yamamoto has joined #openstack-qa | 18:20 | |
| *** yamamoto has quit IRC | 18:31 | |
| clarkb | kopecmartin: probably the biggest thing is to just check the commit you're tagging is present on https://opendev.org | 18:36 |
| clarkb | if thats the caes then you won't accidentally tag and push unmerged code | 18:36 |
| clarkb | other than that double check the commit is what you want and the version number is correct (basically its a lot of double checking as deleting tags is somewhat painful) | 18:36 |
| *** dave-mccowan has quit IRC | 18:46 | |
| *** vishalmanchanda has quit IRC | 18:50 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-testr master: Use unittest.mock instead of third party mock https://review.opendev.org/744241 | 19:08 |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-testr master: DNM: testing focal https://review.opendev.org/740367 | 19:09 |
| *** ajitha has joined #openstack-qa | 19:43 | |
| ajitha | Hi all, We are running a third party driver. currently testing multiattach scenario in cinder.. After enabling [compute] min_microversion=2.60 | 19:45 |
| ajitha | i am facing tempest.lib.exceptions.NotFound: Object not found | 19:45 |
| ajitha | exception in tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario | 19:46 |
| clarkb | ajitha: a link to the logs would probably be helpful in order to hvae context for that exception | 19:46 |
| gmann | ajitha: what is your volume microversion | 19:47 |
| ajitha | gmann: how to check that.. i changed min_microversion for compute since it throwed error as Multi attach requires 2.60 | 19:51 |
| gmann | ajitha: you need 3.44 (if i am not wrong) as min cinder microversion | 19:52 |
| ajitha | clarkb: sure.. i will paste in pastebin | 19:52 |
| ajitha | gmann: I should put in [volume] section in tempest.conf? | 19:52 |
| gmann | ajitha: i see then it is config issue | 19:53 |
| gmann | ajitha: please paste the log, test should have mention the min version for volume too and skip accordingly instead of API error | 19:54 |
| ajitha | gmann, clarkb: https://pastebin.com/ZQrVRiWd | 20:01 |
| ajitha | i have enabled [volume] min_microversion = 3.44.. Still same error | 20:02 |
| ajitha | if i remove min_microversion = 2.60 , test case it getting passed | 20:03 |
| gmann | ajitha: ok, this is not multiattach issue. issue is floating ip compute API are no longer available after microversion 2.35 | 20:05 |
| gmann | ajitha: I will fix the tests to switch to neutron API after 2.35. or now you can fix this in your env by removing min_microversion setting in config | 20:06 |
| ajitha | gmann: yes but multiattach needs this 2.60 microversion | 20:06 |
| gmann | make sure you have max_microversion any value > 2.60 or 'latest' which is deafult | 20:06 |
| ajitha | yes its default one 'latest' | 20:07 |
| gmann | ajitha: yeah tempest will request the multiattach request with 2.60 with setting of min_microversion = None and max_microversion = <any value greater than 2.60> | 20:08 |
| gmann | ajitha: then it will work if you remove the min_microversion | 20:08 |
| gmann | ajitha: only thing you need to make sure that your env has 2.60 compute microversion | 20:08 |
| gmann | which is Queens or later | 20:09 |
| gmann | ajitha: Tempest will select the max of (config's min_microversion, test's min_microversion)to request to API | 20:11 |
| ajitha | gmann: ok | 20:11 |
| ajitha | but if i give None to min_microversion. Its throwing error? | 20:12 |
| gmann | what error | 20:12 |
| ajitha | tempest.lib.exceptions.InvalidAPIVersionString: API Version String None is of invalid format. Must be of format MajorNum.MinorNum or string 'latest'. | 20:12 |
| ajitha | within quotes?? | 20:13 |
| ajitha | [compute] | 20:13 |
| gmann | ajitha: ok, just remove it form config, default is None. | 20:13 |
| ajitha | ok | 20:14 |
| gmann | None for string opt are not best handling in oslo | 20:14 |
| gmann | btw '2.1' or None in default is same | 20:14 |
| ajitha | but if i remore the min_microversion i am getting error as Multiattach volumes are only supported starting with compute API version 2.60 | 20:16 |
| ajitha | for some other tempest test case | 20:16 |
| gmann | ajitha: oh, i min | 20:17 |
| gmann | 1 min | 20:17 |
| ajitha | ok | 20:17 |
| gmann | ajitha: this test right? AttachVolumeMultiAttachTest | 20:17 |
| ajitha | tempest.api.compute.admin.test_volume_swap.TestVolumeSwap.test_volume_swap or tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment | 20:18 |
| ajitha | one of the above.. i didnt remember | 20:19 |
| ajitha | i shall confirm again | 20:19 |
| gmann | ajitha: test_volume_swap has min version capped so it should not be issue, can you paste the traceback in case of no min_microversion setting | 20:22 |
| ajitha | ok i will check again | 20:23 |
| *** nweinber has quit IRC | 20:29 | |
| *** paras333 has quit IRC | 20:46 | |
| *** gcheresh has joined #openstack-qa | 20:50 | |
| *** ajitha has quit IRC | 20:50 | |
| *** raildo has quit IRC | 21:01 | |
| *** rfolco has quit IRC | 21:01 | |
| *** yamamoto has joined #openstack-qa | 21:03 | |
| *** yamamoto has quit IRC | 21:07 | |
| *** jparker has quit IRC | 21:16 | |
| *** gcheresh has quit IRC | 21:23 | |
| *** tosky has quit IRC | 22:10 | |
| *** slaweq has quit IRC | 22:17 | |
| *** slaweq has joined #openstack-qa | 22:30 | |
| *** yamamoto has joined #openstack-qa | 22:46 | |
| *** yamamoto has quit IRC | 23:18 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-testr master: Use unittest.mock instead of third party mock https://review.opendev.org/744241 | 23:22 |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-testr master: DNM: testing focal https://review.opendev.org/740367 | 23:22 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!