| gmann | masayukig: can you check this also. though I need to fix the patrole gate for glance test failure before release - https://review.opendev.org/#/c/721305/ | 00:01 |
|---|---|---|
| masayukig | gmann: thanks. And https://review.opendev.org/#/c/721305/ can be improved. | 00:11 |
| masayukig | gmann: And I found that I missed the same thing on the tempest release note patch.. :p | 00:12 |
| gmann | masayukig: ohk, let me modify the patrole one. | 00:15 |
| openstackgerrit | Ghanshyam Mann proposed openstack/patrole master: Add releasenote to tag the Patrole for Ussuri release https://review.opendev.org/721305 | 00:16 |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest-plugin-cookiecutter master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721372 | 00:23 |
| openstackgerrit | Ghanshyam Mann proposed openstack/qa-specs master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721373 | 00:23 |
| openstackgerrit | Ghanshyam Mann proposed openstack/karma-subunit-reporter master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721374 | 00:23 |
| openstackgerrit | Ghanshyam Mann proposed openstack/devstack-plugin-cookiecutter master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721375 | 00:23 |
| gmann | masayukig: ^^ few easy one | 00:23 |
| masayukig | gmann: thanks. but for patrole, needed to make it like this https://opendev.org/openstack/patrole/src/branch/master/releasenotes/notes/intermediate-ussuri-release-8f7bb2140bca827c.yaml | 00:37 |
| openstackgerrit | Merged openstack/tempest master: Add releasenote to tag the Tempest for Ussuri release https://review.opendev.org/721300 | 00:42 |
| *** Liang__ has joined #openstack-qa | 01:03 | |
| openstackgerrit | Merged openstack/patrole master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721109 | 01:07 |
| *** rh-jelabarre has quit IRC | 01:53 | |
| *** paras333 has quit IRC | 02:42 | |
| *** hongbin has joined #openstack-qa | 03:00 | |
| *** ricolin has joined #openstack-qa | 04:31 | |
| *** evrardjp has quit IRC | 04:35 | |
| *** evrardjp has joined #openstack-qa | 04:35 | |
| *** hongbin has quit IRC | 04:35 | |
| *** ricolin has quit IRC | 04:52 | |
| *** ricolin has joined #openstack-qa | 04:56 | |
| *** ricolin_ has joined #openstack-qa | 05:06 | |
| *** ricolin has quit IRC | 05:09 | |
| *** vishalmanchanda has joined #openstack-qa | 05:09 | |
| *** udesale has joined #openstack-qa | 05:10 | |
| *** udesale_ has joined #openstack-qa | 05:21 | |
| *** udesale has quit IRC | 05:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack/patrole master: Cleanup py27 support https://review.opendev.org/721525 | 05:44 |
| *** gcheresh has joined #openstack-qa | 05:50 | |
| *** yamamoto has quit IRC | 06:08 | |
| *** udesale_ has quit IRC | 06:14 | |
| soniya29 | gmann: done | 06:36 |
| *** yamamoto has joined #openstack-qa | 06:36 | |
| *** udesale has joined #openstack-qa | 06:37 | |
| *** udesale has quit IRC | 06:43 | |
| *** gfidente|afk is now known as gfidente | 06:46 | |
| *** Aditi20 has joined #openstack-qa | 06:49 | |
| Aditi20 | Hi, I am seeing recent failures in IBM Power CI arising because of new tests being added to tempest in https://review.opendev.org/#/c/700813 | 06:50 |
| Aditi20 | These are the test cases failing - {3} tempest.api.compute.servers.test_server_rescue.ServerStableDeviceRescueTest.test_stable_device_rescue_cdrom_ide [19.418171s] ... FAILED | 06:50 |
| Aditi20 | tearDownClass (tempest.api.compute.servers.test_server_rescue.ServerStableDeviceRescueTest) [0.000000s] ... FAILED | 06:50 |
| Aditi20 | Detail logs - https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/41/712741/7/check/tempest-dsvm-full-bionic-py3/dfc495f/job-output.txt | 06:51 |
| Aditi20 | Is there anything that needs to be fixed in the CI to make the rescue test cases work? | 06:51 |
| frickler | Aditi20: I would assume that you need to skip these tests in your environment: "Instance fa6bec08-78d1-4a30-ab05-e86d6d506db0 cannot be rescued: Driver Error: unsupported configuration: IDE controllers are unsupported for this QEMU binary or machine type" | 07:00 |
| frickler | or maybe tempest should be better and check and skip by itself | 07:00 |
| *** rpittau|afk is now known as rpittau | 07:03 | |
| *** nightmare_unreal has joined #openstack-qa | 07:03 | |
| frickler | lyarwood: ^^ that's your patch | 07:04 |
| frickler | Aditi20: otoh, iiuc those tests should be off by default, did you explicitly enable them? https://review.opendev.org/#/c/700813/11/tempest/config.py,unified | 07:06 |
| openstackgerrit | zhufl proposed openstack/tempest master: Adding description for testcases - image part1 https://review.opendev.org/721536 | 07:08 |
| *** slaweq_ has joined #openstack-qa | 07:11 | |
| Aditi20 | frickler Thanks for checking! I am currently testing all tempest api test cases - tempest run --regex '^tempest\.api' --blacklist-file=/home/zuul/pkvm_blacklist.txt --concurrency=4 --black-regex= | 07:12 |
| Aditi20 | Guess will have to add these tests in blacklist | 07:12 |
| *** yamamoto has quit IRC | 07:20 | |
| *** yamamoto has joined #openstack-qa | 07:25 | |
| *** ccamposr has joined #openstack-qa | 07:26 | |
| *** ccamposr__ has quit IRC | 07:28 | |
| *** tosky has joined #openstack-qa | 07:44 | |
| *** ricolin_ has quit IRC | 07:47 | |
| *** ricolin has joined #openstack-qa | 07:47 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack/tempest master: Fix test_get_console_output_server_id_in_shutoff_status https://review.opendev.org/710206 | 07:53 |
| *** ralonsoh has joined #openstack-qa | 07:57 | |
| *** jpena|off is now known as jpena | 07:57 | |
| lyarwood | frickler / Aditi20 ; Morning, https://review.opendev.org/#/c/700814/7/lib/tempest@573 controls this for most jobs upstream FWIW, we can extend this to cover power. | 08:01 |
| *** lennyb has joined #openstack-qa | 08:04 | |
| openstackgerrit | zhufl proposed openstack/tempest master: Adding description for testcases - image part2 https://review.opendev.org/721544 | 08:07 |
| *** lucasagomes has joined #openstack-qa | 08:07 | |
| lyarwood | Aditi20: actually maybe just blacklisting this is the way to go given this is also a LIBVIRT_TYPE=kvm based job. | 08:08 |
| Aditi20 | yes thanks Iyarwood | 08:14 |
| *** udesale has joined #openstack-qa | 08:16 | |
| *** udesale_ has joined #openstack-qa | 08:22 | |
| *** udesale has quit IRC | 08:22 | |
| lyarwood | Aditi20: sorry for breaking your job :) | 08:22 |
| *** udesale has joined #openstack-qa | 08:22 | |
| Aditi20 | haha no issues :D | 08:29 |
| *** Aditi20 has quit IRC | 08:45 | |
| *** dtantsur|afk is now known as dtantsur | 09:00 | |
| *** vishalmanchanda has quit IRC | 09:09 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack/tempest master: WIP: Use SATA bus for CDROM https://review.opendev.org/721555 | 09:09 |
| *** Liang__ has quit IRC | 10:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack/patrole master: Cleanup py27 support https://review.opendev.org/721525 | 10:16 |
| *** rpittau is now known as rpittau|bbl | 10:31 | |
| *** slaweq_ is now known as slaweq | 10:43 | |
| *** ricolin has quit IRC | 11:00 | |
| *** ricolin has joined #openstack-qa | 11:00 | |
| *** udesale has quit IRC | 11:09 | |
| brtknr | clarkb: rather than mirroring, is it possible to download and unxz in advance? | 11:32 |
| brtknr | clarkb: the list of images we want to download are here: https://builds.coreos.fedoraproject.org/streams/stable.json | 11:33 |
| brtknr | clarkb: at the moment, the unxz is burning 16 minutes of CPU for every Zuul job | 11:34 |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** nweinber has joined #openstack-qa | 11:52 | |
| *** rh-jelabarre has joined #openstack-qa | 11:53 | |
| *** raildo has joined #openstack-qa | 12:07 | |
| brtknr | core, please review this, we need it for magnum CI: https://review.opendev.org/721213 | 12:15 |
| brtknr | Cores, please review this, we need it for magnum CI: https://review.opendev.org/721213 | 12:15 |
| *** soniya29 is now known as soniya29|afk | 12:15 | |
| brtknr | AJaeger: gmann ^ | 12:16 |
| tosky | brtknr: in ~70 minutes from now the QA team office hour starts, and there is a point about urgent reviews | 12:19 |
| *** brinzhang has quit IRC | 12:30 | |
| *** dtantsur is now known as dtantsur|brb | 12:30 | |
| *** rpittau|bbl is now known as rpittau | 12:31 | |
| *** jpena|lunch is now known as jpena | 12:35 | |
| *** paras333 has joined #openstack-qa | 12:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack/tempest master: Set TOX_CONSTRAINTS_FILE https://review.opendev.org/721606 | 13:02 |
| *** soniya29|afk is now known as soniya29 | 13:07 | |
| AJaeger | brtknr: I'm not a core here... | 13:08 |
| *** rcernin has quit IRC | 13:14 | |
| *** toabctl has quit IRC | 13:25 | |
| brtknr | hmm who is? | 13:25 |
| *** toabctl has joined #openstack-qa | 13:27 | |
| *** gfidente has quit IRC | 13:28 | |
| gmann | office hour time | 13:31 |
| gmann | #startmeeting qa | 13:31 |
| openstack | Meeting started Tue Apr 21 13:31:11 2020 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:31 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:31 |
| *** openstack changes topic to " (Meeting topic: qa)" | 13:31 | |
| openstack | The meeting name has been set to 'qa' | 13:31 |
| gmann | who all here today | 13:31 |
| masayukig | hi | 13:31 |
| kopecmartin | hello | 13:31 |
| tosky | hi | 13:31 |
| soniya29 | hello | 13:31 |
| gmann | let's start | 13:33 |
| gmann | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours | 13:33 |
| gmann | today agenda ^^ | 13:33 |
| gmann | #topic Announcement and Action Item | 13:33 |
| *** openstack changes topic to "Announcement and Action Item (Meeting topic: qa)" | 13:33 | |
| gmann | one action item i had about cleaning up the few of the QA projects core list. | 13:34 |
| gmann | I did that. | 13:34 |
| gmann | and added masayukig in qa-release team as PTL. | 13:34 |
| masayukig | gmann: ++ thanks :) | 13:35 |
| gmann | if anyone would like to help in hacking side, please let us know | 13:35 |
| gmann | we still looking for few more members there | 13:35 |
| gmann | announcement is about wiki pages updates | 13:35 |
| gmann | I have updated the QA wiki to reflect all the projects and their link for doc, source code etc | 13:36 |
| gmann | #link https://wiki.openstack.org/wiki/QA | 13:36 |
| gmann | ^^ this is up to date now | 13:36 |
| masayukig | cool | 13:36 |
| gmann | next was release page | 13:36 |
| gmann | #link https://wiki.openstack.org/wiki/QA/releases | 13:36 |
| gmann | I have added the release things and steps with example for all the projects under QA | 13:37 |
| gmann | with project releases and extra infra setup we need to do for every release in devstack and grenade | 13:37 |
| gmann | masayukig: it will help you | 13:38 |
| paras333 | gmann: I can help but I am not much aware of hacking plugins so might need some time to get up to the speed | 13:38 |
| gmann | paras333: thanks, this is the project https://github.com/openstack/hacking | 13:38 |
| masayukig | gmann: thanks, that must be helpful | 13:38 |
| tosky | (technically https://opendev.org/openstack/hacking ) | 13:38 |
| gmann | masayukig: i am doing the ussuri release things and will add you in review. and then ball is in your court :) | 13:39 |
| masayukig | gmann: sure, I see | 13:39 |
| paras333 | gmann: This is good, I will take a look. Is there any doc as well that help me getting up to the speed to understand this plugin? | 13:39 |
| gmann | paras333: you can look at that and do some review etc not so many incoming request though | 13:39 |
| paras333 | gmann: sure | 13:39 |
| gmann | paras333: https://docs.openstack.org/hacking/latest/ | 13:40 |
| gmann | #link https://docs.openstack.org/hacking/latest/ | 13:40 |
| gmann | this one ^^ | 13:40 |
| paras333 | gmann: awesome, thanks! | 13:40 |
| gmann | thanks | 13:40 |
| gmann | if you find something missing on those wiki page please let me or masayukig know. | 13:40 |
| gmann | at the end we wanted to move this to QA level doc which is not available yet but let's discuss it in PTG | 13:41 |
| paras333 | gmann: sure | 13:41 |
| gmann | #topic Ussuri Priority Items progress | 13:41 |
| *** openstack changes topic to "Ussuri Priority Items progress (Meeting topic: qa)" | 13:41 | |
| gmann | #link https://etherpad.opendev.org/p/qa-ussuri-priority | 13:42 |
| gmann | Remove/migrate the .testr.conf to .stestr | 13:42 |
| gmann | or sanity blacklist removal | 13:42 |
| masayukig | I've listed up the candidates for the migration/removing #link https://etherpad.opendev.org/p/enkM4eeDHObSloTjPGAu | 13:43 |
| gmann | masayukig: any update on these ^^ | 13:43 |
| gmann | masayukig: nice. I think doing those in V cycle will be good to avoid any issue in Ussuri release | 13:43 |
| masayukig | I pushed one patch already, and preparing pushing the others for patches of removing .testr.conf | 13:43 |
| masayukig | yeah, | 13:43 |
| gmann | ok. +1 | 13:44 |
| masayukig | I don't want to bother the Ussuri release | 13:44 |
| gmann | thanks. can you add the link on etherpad also | 13:44 |
| masayukig | which etherpad? | 13:44 |
| gmann | #link https://etherpad.opendev.org/p/qa-ussuri-priority | 13:44 |
| gmann | this one | 13:44 |
| gmann | RBAC testing strategy (reader/member/admin X project/domain/system) | 13:45 |
| masayukig | I already put the link https://etherpad.opendev.org/p/enkM4eeDHObSloTjPGAu to the priority page | 13:45 |
| gmann | ohk, then link of this etherpad | 13:45 |
| gmann | its there, thanks | 13:45 |
| gmann | no progress on this, this was pending on review. I will be able to spend some time now | 13:45 |
| gmann | Adding New glance tests to tempest | 13:46 |
| *** gfidente has joined #openstack-qa | 13:46 | |
| gmann | not sure if glance team need help on this. devstack side it was all good for multi store feauture. | 13:46 |
| gmann | feature | 13:46 |
| gmann | i will check with abhishek and close this | 13:46 |
| gmann | mproving Tempest cleanup | 13:46 |
| gmann | kopecmartin: ^^ | 13:46 |
| kopecmartin | waiting for some reviews there | 13:47 |
| kopecmartin | no other updates | 13:47 |
| gmann | ok, I think that was close.if i remember correctly did you update the patch for two separate var for init saved and actual cleanup ? | 13:48 |
| kopecmartin | yeah, i think so, i added some variables there .. i answered your questions and suggestions, so maybe you can have a look after the meeting | 13:49 |
| kopecmartin | if it's enough or should i change anything else | 13:49 |
| gmann | sure, thanks | 13:49 |
| gmann | let's move | 13:49 |
| gmann | #topic OpenStack Events Updates and Planning | 13:49 |
| *** openstack changes topic to "OpenStack Events Updates and Planning (Meeting topic: qa)" | 13:49 | |
| gmann | #link https://etherpad.opendev.org/p/qa-victoria-ptg | 13:49 |
| gmann | this is etherpad to collect the topic ^^ | 13:49 |
| gmann | we need to signup for the virtual PTG, I think deadline is 28th April. | 13:51 |
| gmann | let me find the email | 13:51 |
| masayukig | oh, I missed that.. :p | 13:51 |
| gmann | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014126.html | 13:51 |
| openstackgerrit | Lee Yarwood proposed openstack/tempest master: remote_client: Use blkid -L to find config drive by filesystem label https://review.opendev.org/710864 | 13:51 |
| gmann | masayukig: can you prepare for that? like topic and time required and accordingly signup. | 13:52 |
| masayukig | gmann: ok, I'll try that | 13:52 |
| gmann | masayukig: thanks. | 13:52 |
| gmann | #topic Sub Teams highlights | 13:53 |
| *** openstack changes topic to "Sub Teams highlights (Meeting topic: qa)" | 13:53 | |
| gmann | Tempest | 13:53 |
| gmann | Tempest is going to be released for ussuri as final release. I will update the hash to release patch. | 13:53 |
| gmann | any other things to updates on Tempest ? | 13:53 |
| soniya29 | gmann: With regards to scenario manager, I found certain methods are already well defined but still the tempest plugins have duplicated the code in the method. So in this usecase, do we require to work at plugin side or is there something we can do at tempest as well? | 13:54 |
| gmann | Patrole | 13:54 |
| gmann | soniya29: thanks. we did copy everything on plugins side to avoid breaking them if we change anything on tempest scenario manager | 13:55 |
| gmann | soniya29: so plan is 1. once we declare the scenario manager stable after doing some work to make each method more consistent etc 2. remove the copy of scenario manager from plugins side and replace them with import tempest.scenario.manager | 13:56 |
| gmann | soniya29: so whatever cleanup is required on tempest scenario manager side, feel free to propose without worry about plugins copy | 13:56 |
| soniya29 | gmann: okay, I wasn't aware of this concept | 13:57 |
| gmann | soniya29: sorry for not giving that background. | 13:58 |
| gmann | soniya29: also as you noticed about mixin or single base class idea among scenario or api tests to avoid duplicate code. | 13:58 |
| soniya29 | gmann: that's okay :) | 13:59 |
| gmann | anything else on Tempest side ? | 13:59 |
| gmann | Patrole | 14:00 |
| gmann | i am in process of final release of patrole for ussuri but gate is blocked and failure happening for glance tests | 14:00 |
| gmann | there are some change in glance policy and the config option admin_role which might have caused it | 14:00 |
| gmann | I need to debug that and see if we can fix it. almost all the glance tests are failing | 14:01 |
| masayukig | oh.. :( | 14:01 |
| gmann | as per glance releasenotes, default policy user should not effect but seems patrole is being effected | 14:02 |
| gmann | let me debug more on that and find what happened and how it can be fixed | 14:02 |
| gmann | Grenade | 14:03 |
| gmann | zuulv3 job | 14:03 |
| gmann | #link https://review.opendev.org/#/q/topic:native-zuulv3-migration+(status:open+OR+status:merged) | 14:03 |
| tosky | yep, patches ready for review | 14:03 |
| gmann | i was half way on the re-review and could not finish | 14:03 |
| gmann | tosky: thanks. give me today or tomorrow time. I should be able to finish that | 14:03 |
| tosky | anything relevant that you think should be changed so far? | 14:03 |
| gmann | i am trying to check on target and base branch var | 14:04 |
| tosky | but I can wait for the review | 14:04 |
| gmann | but could not finish so nothing as of now. | 14:04 |
| gmann | main goal is we should have a easy way to change them over the releases. | 14:05 |
| tosky | and we do | 14:05 |
| gmann | yeah, i need to revisit that. | 14:05 |
| gmann | target is to get this in before V cycle when tosky will start the zuulv3 migration goal. | 14:06 |
| gmann | let's move next | 14:07 |
| gmann | Community goal tracking | 14:07 |
| gmann | #link https://etherpad.openstack.org/p/qa-community-wide-goals-tracking | 14:07 |
| gmann | py2 drop goal is done for QA | 14:08 |
| gmann | contributor guide is in progress and should be finished before ussuri release | 14:08 |
| masayukig | great | 14:08 |
| brtknr | when its a good time, i'd like to raise a couple of points if thats okay | 14:08 |
| gmann | there are few devstack plugins repo where I need to add some doc place or use devstack contribution guide. I think 4 of them are left | 14:08 |
| gmann | brtknr: we can discuss in open discussion | 14:09 |
| gmann | #topic Bug Triage | 14:09 |
| *** openstack changes topic to "Bug Triage (Meeting topic: qa)" | 14:09 | |
| gmann | kopecmartin: go ahead | 14:09 |
| brtknr | gmann: thanks | 14:09 |
| kopecmartin | i don't have many updates, i didn't focu on bug triaging much this week, anyway, the numbers still look ok , or are decreasing | 14:10 |
| kopecmartin | and the etherpad is up to date, so have a look anyone for reviews, or bugs to dicscuss | 14:10 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-bug-triage-ussuri | 14:10 |
| gmann | kopecmartin: ok thanks. | 14:10 |
| masayukig | ++ | 14:11 |
| gmann | everyone please review the bug fixes mentioned in etherpad ^^ | 14:11 |
| gmann | #topic Critical Reviews | 14:11 |
| *** openstack changes topic to "Critical Reviews (Meeting topic: qa)" | 14:11 | |
| gmann | any critical review ? | 14:11 |
| kopecmartin | gmann: we can +w this, we added a note to the scenario manager work as you asked: https://review.opendev.org/#/c/701434/ | 14:12 |
| gmann | kopecmartin: i was waiting for the test result which was not present in last gate result | 14:12 |
| gmann | it should be ready now, I will look after meeting | 14:13 |
| kopecmartin | gmann: and one question regarding this, where should we make a note regarding technical dept? https://review.opendev.org/#/c/701352/ there is one thing that should be done as a follow up | 14:13 |
| gmann | kopecmartin: i have not checked that yet. let me review and reply on review | 14:14 |
| kopecmartin | gmann: ok,thanks | 14:14 |
| gmann | #topic Open Discussions | 14:14 |
| *** openstack changes topic to "Open Discussions (Meeting topic: qa)" | 14:14 | |
| gmann | brtknr: go ahead | 14:14 |
| brtknr | okay so i wanted to mention the patch to support upload of qcow2.xz | 14:15 |
| openstackgerrit | Ghanshyam Mann proposed openstack/patrole master: Add releasenote to tag the Patrole for Ussuri release https://review.opendev.org/721305 | 14:15 |
| gmann | brtknr: link? | 14:15 |
| brtknr | https://review.opendev.org/721213 | 14:16 |
| brtknr | we are migrating migrating magnum CI to Zuulv3 | 14:16 |
| gmann | #link https://review.opendev.org/721213 | 14:16 |
| brtknr | but also moving towards fedora coreos image because fedora atomic is EOL | 14:17 |
| brtknr | the second point i wanted to make was about mirroring fedora coreos locally | 14:17 |
| gmann | i think we are on latest fedora image right? | 14:18 |
| brtknr | but as well as mirroring, running an unxz operation when the image is downloaded so that we save 16 minutes of CI time | 14:18 |
| gmann | may be ianw or lyarwood knows | 14:18 |
| gmann | i see. is it very heavy image? size | 14:18 |
| brtknr | gmann: i dont think it includes fedora-coreos | 14:18 |
| brtknr | yes, 460M before unxz, 1.7G after | 14:19 |
| lyarwood | hey | 14:19 |
| lyarwood | no that's not me | 14:19 |
| lyarwood | maybe the TripleO folks? | 14:19 |
| masayukig | brtknr: yeah, I think "fedora" and "fedora core" are different. So, maybe, more description in the commit message is preferable. | 14:19 |
| brtknr | https://getfedora.org/en/coreos/ | 14:20 |
| lyarwood | gmann: oh sorry I see your actual question now | 14:20 |
| lyarwood | gmann: fedora-31 is fedora-latest now | 14:20 |
| gmann | yeah i also find fedora-31 as latest | 14:20 |
| brtknr | fedora-coreos uses fedora as base layer but adds additional bits on top | 14:21 |
| brtknr | so not the same | 14:21 |
| gmann | brtknr: let 's wait for ianw or yoctozepto review on https://review.opendev.org/721213 | 14:21 |
| brtknr | gmann: okaay thanks, 4 PS depend on this, 2 in magnum and 2 in magnum-tempest-plugin :) | 14:22 |
| gmann | brtknr: not sure about fedora-coreos support as per current bandwidth, I would like to know ianw yoctozepto opinion on that | 14:23 |
| *** ricolin has quit IRC | 14:23 | |
| *** ricolin has joined #openstack-qa | 14:23 | |
| *** yamamoto has quit IRC | 14:23 | |
| brtknr | ok thanks | 14:24 |
| gmann | in devstack we have been supporting a single fedora version which is latest(as we move to new one) | 14:24 |
| * yoctozepto reporting for duty | 14:24 | |
| gmann | yoctozepto: is here :) | 14:24 |
| yoctozepto | is not that patch related just to image upload? | 14:24 |
| gmann | we can discuss after office hour on that | 14:24 |
| brtknr | yoctozepto: for your attention : https://review.opendev.org/#/c/721213/ | 14:24 |
| gmann | anything else to discuss today ? | 14:25 |
| brtknr | yoctozepto: i am discussing the possiblity of mirror+unxz, no patches yet | 14:25 |
| brtknr | happy to talk after office hour | 14:25 |
| brtknr | happy to talk after "office hour" | 14:25 |
| gmann | let's close the office hour and continue the discussion. | 14:25 |
| gmann | thanks all for joining | 14:25 |
| masayukig | ++ | 14:25 |
| gmann | #endmeeting | 14:25 |
| *** openstack changes topic to "Discussion of OpenStack Quality Assurance Projects (tempest, devstack, grenade, and friends) | docs https://wiki.openstack.org/wiki/QA | specs https://review.openstack.org/#/q/project:openstack/qa-specs | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-qa/" | 14:25 | |
| masayukig | thanks! | 14:25 |
| openstack | Meeting ended Tue Apr 21 14:25:56 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:25 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/qa/2020/qa.2020-04-21-13.31.html | 14:26 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/qa/2020/qa.2020-04-21-13.31.txt | 14:26 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/qa/2020/qa.2020-04-21-13.31.log.html | 14:26 |
| gmann | yoctozepto: apart from that patch brtknr is asking about fedora-coreos support ? | 14:26 |
| gmann | frickler: ^^ | 14:26 |
| brtknr | gmann: is right | 14:26 |
| *** gcheresh has quit IRC | 14:26 | |
| brtknr | because downloading the images is quick (460M) but unxz operation costs 16 mins in the CI | 14:26 |
| yoctozepto | like instead of cirros? | 14:27 |
| brtknr | we use fedora-coreos as the supported image in magnum | 14:27 |
| yoctozepto | ah, this usecase | 14:27 |
| brtknr | since fedora atomic is EOL | 14:27 |
| yoctozepto | well, it's huge if it's 460M xzipped | 14:27 |
| brtknr | there was a fedora atomic mirror which is now no longer being updated | 14:28 |
| brtknr | yoctozepto: yep, its 1.7G unxzipped | 14:28 |
| yoctozepto | oh my, then it's a no-no to carry this in infra images | 14:28 |
| yoctozepto | so the only solution is to put it on mirror | 14:29 |
| clarkb | brtknr: generally producing CI specific artifacts like that shpuld be avoided because now when you go to production you either need extra steps or you rely on CI resources | 14:30 |
| brtknr | yoctozepto: mirror zipped or unzipped version? | 14:30 |
| yoctozepto | well, that's tricky, either should help here | 14:31 |
| brtknr | only the zipped version is distributed upstream | 14:31 |
| yoctozepto | but if unxzipping takes a lot of time, it might be interesting to consider unxzipped too | 14:31 |
| yoctozepto | and then clarkb's point applies | 14:31 |
| yoctozepto | how much time does the whole scenario take? | 14:32 |
| brtknr | unxz takes 16 mins | 14:32 |
| brtknr | the whole job takes 56 mins | 14:32 |
| brtknr | so 25% of the time | 14:33 |
| brtknr | slightly more in fact | 14:33 |
| brtknr | 28.6% | 14:33 |
| yoctozepto | is only xz version available there? | 14:34 |
| brtknr | yoctozepto: thats right | 14:35 |
| brtknr | :( | 14:35 |
| yoctozepto | oh my, I wonder if upstream could not just use gz | 14:35 |
| yoctozepto | this compression ratio is not that great and having like 10x decompression time is absurd | 14:36 |
| clarkb | yoctozepto: I think qcow2 is already gzipped | 14:37 |
| brtknr | yes i put that questions forward in #fedora-coreos channel | 14:37 |
| brtknr | they said they are working to improve the image size | 14:37 |
| yoctozepto | clarkb: I thought it's optional? | 14:37 |
| yoctozepto | qcow2 has zeroes removed for sure but that's it I guess | 14:38 |
| brtknr | yoctozepto: gziping it reduces the image down to 698M | 14:38 |
| brtknr | not sure why they didnt just go for that option | 14:39 |
| yoctozepto | that's what I thought | 14:40 |
| clarkb | qcow2 uses zlib | 14:40 |
| yoctozepto | and decompression takes like... tens of seconds? | 14:40 |
| clarkb | soaybe they arent compressing the qcow2 | 14:40 |
| yoctozepto | clarkb: yeah, they likely don't | 14:40 |
| clarkb | since zlib is basically gzip | 14:40 |
| yoctozepto | I think most qcow2 in the wild have 0 compression | 14:41 |
| clarkb | thats my suggestion then. They should stop xz'ing qcow2 and have it zlib compress as supported by qcow2 | 14:41 |
| yoctozepto | ++ | 14:41 |
| *** yamamoto has joined #openstack-qa | 14:41 | |
| yoctozepto | plan minimum is they just do gzip instead of xz | 14:41 |
| *** yamamoto has quit IRC | 14:42 | |
| *** yamamoto has joined #openstack-qa | 14:42 | |
| *** yamamoto has quit IRC | 14:42 | |
| *** yamamoto has joined #openstack-qa | 14:43 | |
| *** gcheresh has joined #openstack-qa | 14:44 | |
| *** yamamoto has quit IRC | 14:44 | |
| *** yamamoto has joined #openstack-qa | 14:44 | |
| *** yamamoto has quit IRC | 14:44 | |
| yoctozepto | and we obviously can mirror this compressed so it's more reliable | 14:44 |
| *** yamamoto has joined #openstack-qa | 14:44 | |
| clarkb | another potential option is to reduce the xz settings | 14:46 |
| clarkb | xz is far more cpu intensive than gzip as you bump compression levels | 14:46 |
| clarkb | you may still get better than gzip compression at reasonable cpu cost if lowering the level | 14:47 |
| *** dtantsur|brb is now known as dtantsur | 14:47 | |
| yoctozepto | ^ but it could also happen that similar performance earns worse compression | 14:47 |
| yoctozepto | but this decompression time is ridiculuosly high compared to saved space | 14:48 |
| yoctozepto | far from eco solution ;p | 14:48 |
| *** dmellado has quit IRC | 14:49 | |
| *** yamamoto has quit IRC | 14:49 | |
| brtknr | while we work out what to do, can we take this in in the meantime: https://review.opendev.org/#/c/721213/ | 15:02 |
| yoctozepto | yeah, works for me | 15:03 |
| clarkb | brtknr: isn't the inline unxz the source of the swift errors? | 15:03 |
| clarkb | we should unxz then upload | 15:04 |
| clarkb | (this was my comment yseterday) | 15:04 |
| yoctozepto | clarkb: swift timeouts? | 15:04 |
| yoctozepto | or what happens? | 15:04 |
| clarkb | yoctozepto: yes glance connection to swift times out | 15:05 |
| yoctozepto | ah, imaginable | 15:05 |
| clarkb | I think beacuse glance starts the connection when the push starts but then xz doesn't write bytes in a consistent stream | 15:05 |
| yoctozepto | still, writing this locally may be painful | 15:05 |
| yoctozepto | yeah, it likely does not | 15:05 |
| clarkb | eh mktemp, unxz, upload rm tmp | 15:06 |
| clarkb | its not the worst things | 15:06 |
| yoctozepto | well, it's still 1.7G | 15:06 |
| yoctozepto | once in tmp, once in swift | 15:06 |
| yoctozepto | I guess we *could* run out of space in some cases | 15:07 |
| yoctozepto | but maybe I'm worrying too much | 15:07 |
| yoctozepto | I guess this is one of the things we would discuss in fungi's proposed sig | 15:08 |
| fungi | no need to wait for a sig to have discussions ;) | 15:08 |
| yoctozepto | fungi: sure, I'm just examining the potential scope | 15:08 |
| clarkb | yoctozepto: ya but its also self testing | 15:09 |
| clarkb | so if that is a problem we'll know | 15:09 |
| yoctozepto | well, sure; there is still no better idea if streaming times out | 15:10 |
| brtknr | yoctozepto: could the default timeout be increased? | 15:10 |
| brtknr | to infinite? | 15:10 |
| brtknr | or 1 hour? | 15:10 |
| yoctozepto | I don't think infinite is fine :-) | 15:10 |
| yoctozepto | 15 mins I guess would do | 15:10 |
| yoctozepto | it likely streams it in portions | 15:11 |
| clarkb | we should ask swift what the side effects of that are | 15:11 |
| yoctozepto | glanceclient <-> glance could time out too | 15:11 |
| yoctozepto | also, I think glance does this inefficient way if it does not know the file size ahead of time | 15:12 |
| yoctozepto | so maybe the 16 minutes come from that too? | 15:12 |
| yoctozepto | as it has to resize the object size | 15:12 |
| *** dmellado has joined #openstack-qa | 15:17 | |
| clarkb | basically if we are adding that feature to devstack we shouldn't require swift not be running without checking for that | 15:18 |
| clarkb | and ideally since this is a client side interaction we would make it work with and without swift | 15:18 |
| clarkb | (so I Think we should try for that before anything else) | 15:18 |
| openstackgerrit | Bharat Kunwar proposed openstack/devstack master: Support upload of qcow2.xz image format https://review.opendev.org/721213 | 15:31 |
| brtknr | clarkb: yoctozepto: I have addressed your comment^ | 15:31 |
| *** dmellado has quit IRC | 15:35 | |
| *** dmellado has joined #openstack-qa | 15:37 | |
| *** udesale_ has quit IRC | 15:41 | |
| yoctozepto | would not it make sense to put time around these invocations (unxz and upload) - it should be easier for profiling (could use seeing the user and system time) | 15:42 |
| clarkb | yoctozepto: devstack will trace with log timestamps | 15:44 |
| clarkb | to start you can probably go off of that | 15:44 |
| yoctozepto | clarkb: yeah, but this is only *real* time | 15:46 |
| yoctozepto | I wonder where it spends it | 15:47 |
| yoctozepto | brtknr: ^^^ | 15:47 |
| brtknr | yoctozepto: the -v arg means verbose | 15:47 |
| brtknr | this prints out the amount of time + MB/s in the devstack logs | 15:48 |
| brtknr | i have added -v arg to unxz | 15:48 |
| yoctozepto | yeah, I'm just interested in whether this is properly threaded, io-bound etc. | 15:49 |
| yoctozepto | you don't get either with any of suggested sources | 15:49 |
| *** ricolin has quit IRC | 15:49 | |
| brtknr | yoctozepto: are you suggesting adding "time unxz" into devstack codebase? | 15:52 |
| *** yamamoto has joined #openstack-qa | 15:54 | |
| yoctozepto | and time for upload | 15:58 |
| brtknr | yoctozepto: ubuntu@devstack-master:~/magnum-terraform$ time unxz -vkfT0 fedora-coreos-31.20200323.3.2-openstack.x86_64.qcow2.xz | 15:59 |
| brtknr | fedora-coreos-31.20200323.3.2-openstack.x86_64.qcow2.xz (1/1) | 15:59 |
| brtknr | 100 % 459.2 MiB / 1657.1 MiB = 0.277 55 MiB/s 0:30 | 15:59 |
| brtknr | real 0m30.363s | 15:59 |
| brtknr | user 0m28.679s | 15:59 |
| brtknr | sys 0m1.531s | 15:59 |
| brtknr | thats my local result | 15:59 |
| *** gcheresh has quit IRC | 15:59 | |
| brtknr | not sure why the CI takes 16 mins | 16:00 |
| brtknr | upload doesnt take very long | 16:00 |
| brtknr | this is in the lgs | 16:01 |
| brtknr | there^ | 16:02 |
| openstackgerrit | Bharat Kunwar proposed openstack/devstack master: Support upload of qcow2.xz image format https://review.opendev.org/721213 | 16:02 |
| brtknr | g2g now, childcare duties but i will be back tomorrow | 16:03 |
| *** lucasagomes has quit IRC | 16:07 | |
| *** kklimonda has quit IRC | 16:28 | |
| *** kklimonda has joined #openstack-qa | 16:30 | |
| *** evrardjp has quit IRC | 16:35 | |
| *** evrardjp has joined #openstack-qa | 16:35 | |
| brtknr | yoctozepto: do you want it there permanently or just for one off check? | 16:36 |
| *** kklimonda has quit IRC | 16:38 | |
| *** lseki has quit IRC | 16:38 | |
| *** dviroel has quit IRC | 16:38 | |
| *** rpittau is now known as rpittau|afk | 16:39 | |
| *** jberg-dev has quit IRC | 16:39 | |
| *** masayukig has quit IRC | 16:39 | |
| *** kklimonda has joined #openstack-qa | 16:40 | |
| *** simondodsley has quit IRC | 16:40 | |
| *** jamespage has quit IRC | 16:40 | |
| *** dviroel has joined #openstack-qa | 16:40 | |
| *** nightmare_unreal has quit IRC | 16:40 | |
| *** simondodsley has joined #openstack-qa | 16:41 | |
| *** mnaser has quit IRC | 16:41 | |
| *** lseki has joined #openstack-qa | 16:42 | |
| *** jamespage has joined #openstack-qa | 16:42 | |
| *** TheJulia has quit IRC | 16:42 | |
| *** csatari has quit IRC | 16:43 | |
| *** masayukig has joined #openstack-qa | 16:43 | |
| *** jberg-dev has joined #openstack-qa | 16:43 | |
| *** nightmare_unreal has joined #openstack-qa | 16:44 | |
| *** TheJulia has joined #openstack-qa | 16:44 | |
| *** mnaser has joined #openstack-qa | 16:44 | |
| *** csatari has joined #openstack-qa | 16:45 | |
| *** vdrok has quit IRC | 16:45 | |
| *** mnaser has quit IRC | 16:46 | |
| *** vdrok has joined #openstack-qa | 16:47 | |
| *** mnaser has joined #openstack-qa | 16:47 | |
| *** mnaser has quit IRC | 16:49 | |
| *** mnaser has joined #openstack-qa | 16:50 | |
| *** dtantsur is now known as dtantsur|afk | 16:59 | |
| yoctozepto | brtknr: perma is fine | 17:03 |
| yoctozepto | I wonder is your machine single-threaded? as it looks like it used just one core; still it's much much faster than in CI so maybe it was glance/swift doing it mega ineffectively | 17:05 |
| AJaeger | patrole fails the admin,member, reader tests for me on https://review.opendev.org/#/c/721525/ - that should be a simple py27 cleanup. If anybody can give me a pointer, i would appreciate it. | 17:09 |
| gmann | AJaeger: it is failing due to glance change. all the glance tests are failing. it is hard to debug and also blocking the release. | 17:10 |
| gmann | I need to check actual issue from patrole or glance side | 17:10 |
| AJaeger | gmann: thanks for confirmation - then I'll wait. | 17:10 |
| gmann | default policy which is what tested by patrole should keep working as it is but somehow they are failing | 17:10 |
| gmann | fixing all the nova tests is already up but that is blocked by glance failure- https://review.opendev.org/#/c/720601/ | 17:11 |
| AJaeger | ack | 17:12 |
| *** jpena is now known as jpena|off | 17:13 | |
| *** nightmare_unreal has quit IRC | 17:13 | |
| *** ajitha has joined #openstack-qa | 17:41 | |
| ajitha | hi all, we have been running a openstack CI for cinder driver.. with ubuntu 18.04 and py3.6... After this commit https://opendev.org/openstack/nova/commit/f1e448d6729f674101b34b08e3c5e95fd8f75770 , our CI/CD pipeline is getting break in devstack with following error "pip._vendor.pkg_resources.ContextualVersionConflict: (oslo.policy 3.0.3 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('oslo.policy>=3.1.0'), {'nova'}).." | 17:42 |
| ajitha | http://paste.openstack.org/show/792492/ -- detailed log..manual devstack execution works fine with an error throwing as ERROR: nova 20.1.0.dev1681 has requirement oslo.policy>=3.1.0, you'll have oslo-policy 3.0.3 which is incompatible... but it skips that and devstack gets completed. and while executing devstack-gate jenkins job, devstack gets error at the above point | 17:42 |
| ajitha | The error is throwing in the manual devstack execution(./stack.sh) but it skips and complete the devstack.. In case of devstack-gate, it just stops at that point | 17:42 |
| openstackgerrit | Radosław Piliszek proposed openstack/devstack master: Support upload of qcow2.xz image format https://review.opendev.org/721213 | 17:43 |
| openstackgerrit | Andreas Jaeger proposed openstack/tempest master: Set TOX_CONSTRAINTS_FILE https://review.opendev.org/721606 | 17:43 |
| gmann | ajitha: which job is this ? on master and py3 ? | 17:44 |
| ajitha | yes | 17:45 |
| ajitha | py3.6 and master | 17:45 |
| clarkb | links to the job logs are helpful | 17:45 |
| clarkb | https://opendev.org/openstack/requirements/src/branch/master/upper-constraints.txt#L824 py3.8 is the only one I would expect to try and install the old version | 17:46 |
| gmann | ajitha: nova need oslo.policy >3.1.0 now as i am using the new flag introduced in oslo policy 3.1.0 which should be available in your CI also. | 17:46 |
| gmann | yeha more log is needed here to check | 17:46 |
| gmann | yeah | 17:47 |
| ajitha | sure | 17:47 |
| ajitha | more log of devstack right? | 17:47 |
| clarkb | looks like that was updated recently https://review.opendev.org/#/c/719594/ | 17:47 |
| gmann | yeah | 17:47 |
| clarkb | ajitha: specifically the link to your zuul jobs | 17:47 |
| clarkb | or I guess it doesn't have to be zuul | 17:48 |
| clarkb | but third party ci are supposed to publish logs (maybe this isn't third party ci?) | 17:48 |
| ajitha | actually we wont populate logs if it failed. i shall send the recent success link.. and current devstack log | 17:49 |
| ajitha | its thirdparty nimble CI | 17:49 |
| gmann | ajitha: devstack log should give more info on what constrained are used and installed | 17:50 |
| ajitha | sure will paste in paste.openstack.org | 17:51 |
| clarkb | ajitha: note there is a size limite on paste.openstack.org and your devstack logs are likely to go over it | 17:51 |
| ajitha | yes.. what other options? | 17:51 |
| clarkb | ajitha: I think gist on github may not have size limits. Not sure about other paste bins. You could also run the job and publish the logs | 17:52 |
| clarkb | you are supposed to do that anyway aiui | 17:52 |
| ajitha | sure.. thanks | 17:52 |
| gmann | try to paste where you find 'oslo.policy ' but you said error is skipped in devstack ? | 17:53 |
| gmann | ajitha: note its all working in upstream CI | 17:54 |
| ajitha | yes error is skipped in devstack(./stack.sh).. But stopped while running gate | 18:00 |
| *** gfidente is now known as gfidente|afk | 18:00 | |
| ajitha | Requirement already satisfied: pyperclip===1.8.0 in /usr/local/lib/python3.6/dist-packages (from -c /opt/stack/new/requirements/upper-constraints.txt (line 536)) (1.8.0) | 18:00 |
| ajitha | ERROR: nova 20.1.0.dev1681 has requirement oslo.policy>=3.1.0, but you'll have oslo-policy 3.0.3 which is incompatible. | 18:00 |
| ajitha | Installing collected packages: nova | 18:00 |
| ajitha | Attempting uninstall: nova | 18:00 |
| ajitha | above is the skipped one | 18:01 |
| ajitha | shall send you the devstack log file clarkb and gamn ? | 18:01 |
| gmann | ajitha: this is not skipped, this show it failed- http://paste.openstack.org/show/792492/ | 18:01 |
| ajitha | this is devstack-early.txt | 18:02 |
| clarkb | ajitha: are you reusing test nodes? | 18:02 |
| gmann | ajitha: not the success one but the one failing. if you can paste the 'oslo.policy' logs so that i can check on what version it was installed and what constrained was used | 18:02 |
| ajitha | skipped is when i execute devstack alone | 18:02 |
| clarkb | I wonder if you've got an older package installed and pip isn't wanting to upgrade it for some reason | 18:02 |
| gmann | i am suspecting you are using old constraint from old requirement repo not master | 18:02 |
| clarkb | gmann: oh ya that could do it too | 18:03 |
| ajitha | yes i reused first.. even in fresh machine i got the same error.. | 18:03 |
| ajitha | i doubt whether it will work with py3.6? | 18:05 |
| ajitha | and 18.04? | 18:06 |
| gmann | check what constraint used for oslo.policy. there should be this log - "Collecting oslo.policy===3.1.0" | 18:07 |
| gmann | or other version if old constraint | 18:07 |
| gmann | if it is 3.1.0 then we can go to next step like clarkb mentioned on pip did upgrade/installed or not ? | 18:08 |
| ajitha | no its 3.0.3 | 18:11 |
| ajitha | http://paste.openstack.org/show/792497/ | 18:11 |
| gmann | ajitha: ok then your requirement repo is not up to date where constraint has been updated | 18:12 |
| ajitha | root@openstack-ubuntu-iscsi-backup:/opt/stack/new# ls -alht requirements/ | 18:12 |
| ajitha | total 160K | 18:12 |
| ajitha | drwxr-xr-x 43 stack stack 4.0K Apr 21 08:13 .. | 18:12 |
| ajitha | drwxr-xr-x 10 stack stack 4.0K Apr 21 07:44 . | 18:12 |
| ajitha | -rw-r--r-- 1 stack stack 26K Apr 21 07:44 upper-constraints.txt | 18:12 |
| ajitha | oslo.policy===3.0.3;python_version=='3.7' | 18:14 |
| ajitha | they have given | 18:14 |
| gmann | yeah these are old constraint and they got updated ~7 days ago. is /opt/stack/new/requirements latest ? | 18:15 |
| *** ralonsoh has quit IRC | 18:16 | |
| ajitha | is | 18:17 |
| ajitha | yes | 18:17 |
| ajitha | even in other CI logs i have seen as collecting 3.0.3 | 18:17 |
| ajitha | http://openstack-logs.purestorage.com/26/721626/1/check/PureISCSIDriver-tempest-dsvm-bionic-aio-multipath-chap/a6fa41d/logs/ | 18:18 |
| gmann | ajitha: humm. this should be in if latest requirements repo https://review.opendev.org/#/c/719594/1 | 18:18 |
| ajitha | ok | 18:18 |
| ajitha | for every gate execution it will get a fresh clone right? | 18:19 |
| ajitha | of devstack | 18:19 |
| gmann | http://openstack-logs.purestorage.com/26/721626/1/check/PureISCSIDriver-tempest-dsvm-bionic-aio-multipath-chap/a6fa41d/logs/devstacklog.txt.gz#_2020-04-21_14_14_47_283 | 18:20 |
| gmann | it checkout the old one, older than oslo.policy constarint update | 18:21 |
| ajitha | ok | 18:21 |
| gmann | devstack is not cloning the fresh because /opt/stack/new/requirements already exists | 18:23 |
| gmann | but you said it is fresh node/installation with no /opt/stack/new/requirements ? | 18:24 |
| ajitha | yes.. i do remove /opt/stack/new/* | 18:25 |
| ajitha | for every job | 18:26 |
| ajitha | when this oslo.policy constarint update happened today right? | 18:27 |
| gmann | not today, 8 days ago - https://review.opendev.org/#/c/719594/1 | 18:28 |
| ajitha | ok i will check once, after removing /opt/stack | 18:28 |
| gmann | even pre cloned requirement repo should be latest one which devstack detected and did not clone it again | 18:28 |
| gmann | going for lunch, will be back | 18:29 |
| clarkb | ajitha: gmann if ERROR_ON_CLONE is true then it won't update the repos | 18:29 |
| clarkb | I would double check that isn't set | 18:29 |
| clarkb | if it is set then your ci system is expected to provide the correct repo state | 18:29 |
| ajitha | sure.. that is not set | 18:29 |
| gmann | yeah, that is true | 18:29 |
| gmann | ajitha: is it? - http://openstack-logs.purestorage.com/26/721626/1/check/PureISCSIDriver-tempest-dsvm-bionic-aio-multipath-chap/a6fa41d/console.html | 18:30 |
| gmann | if you keep the latest requirement repo or let devstack clone it fresh will work | 18:30 |
| *** gmann is now known as gmann_lunch | 18:31 | |
| ajitha | clarkb: should i set that flag ERROR_ON_CLONE? | 18:33 |
| clarkb | ajitha: no, not necessarily. Its just common in CI systems so wanted to rule it out | 18:34 |
| clarkb | do you have a specific ref set? also where are you cloning from? | 18:34 |
| clarkb | would be good to confirm those values even if using the defaults | 18:34 |
| ajitha | clarkb https://ftp.nimblestorage.com/openstack_ci/iscsi/58/718758/2/check/nimble-iscsi-driver-dsvm-volume/e602519/logs/local.conf | 18:39 |
| clarkb | error on clone is set there | 18:40 |
| clarkb | so devstack won't update your git repos | 18:40 |
| ajitha | yes sorry.. | 18:41 |
| ajitha | i should remove and execute? | 18:41 |
| ajitha | clarkb i should remove and execute? | 18:42 |
| clarkb | not necessarily | 18:42 |
| clarkb | what that means is your CI system is currently setting up the repo state | 18:42 |
| clarkb | and that is where you need to look for why requriements isn't updated | 18:43 |
| clarkb | are you using zuul? if so which version? | 18:43 |
| ajitha | yes zuul 2 | 18:43 |
| clarkb | ok with zuulv2 devstack-gate was responsible for updating the repos to the correct version. Are you using devstack-gate? | 18:44 |
| ajitha | yes | 18:45 |
| ajitha | devstack-gate | 18:45 |
| clarkb | ok so now what you need to do is look at devstack-gate logs and see how it is updating openstack/requirements | 18:45 |
| clarkb | and work from there to find out why it isn't pulling the latest version | 18:45 |
| ajitha | https://ftp.nimblestorage.com/openstack_ci/iscsi/58/718758/2/check/nimble-iscsi-driver-dsvm-volume/e602519/ | 18:46 |
| ajitha | this is like the logs will be | 18:46 |
| clarkb | ajitha: that shows that particular job was for stable/train (whcih won't be as up to date as master | 18:47 |
| ajitha | this is 10 days older.. we have some issues with ftp.. thatsy couldnt update the logs after that | 18:47 |
| clarkb | but also it uses review.openstack.org/p/ which ended up out of sync for a bit | 18:48 |
| clarkb | I think we thoguht that was fixed yesterday | 18:48 |
| clarkb | can you rerun the job today and see if it still fails with the oslo.policy problem | 18:48 |
| *** gmann_lunch is now known as gmann | 18:51 | |
| ajitha | sorry clarkb.. that is a train commit | 18:52 |
| ajitha | https://ftp.nimblestorage.com/openstack_ci/iscsi/58/721058/1/check/nimble-iscsi-driver-dsvm-volume/f6c5929/ | 18:52 |
| ajitha | sure.. i will rerun it now | 18:53 |
| ajitha | clarkb oslo.policy===2.4.1;python_version=='2.7' | 19:03 |
| ajitha | oslo.policy===3.0.3;python_version=='3.6' | 19:03 |
| ajitha | oslo.policy===3.0.3;python_version=='3.7' | 19:03 |
| ajitha | in requirements.txt.. may be is it like this for ubuntu 18.04?? | 19:03 |
| clarkb | ajitha: no it isn't distro specific its global | 19:03 |
| ajitha | sorry i am new to this.. thats y the trouble | 19:04 |
| ajitha | ok | 19:04 |
| clarkb | I have meetings now but I think the next step is for someone to clone from review.openstack.org/p/openstack/requirements and check it outside of your ci system | 19:04 |
| clarkb | gmann: ^ fyi | 19:04 |
| gmann | ok | 19:04 |
| clarkb | if that is still otu of date mordred was the person debugging it | 19:05 |
| clarkb | I've just cloned it and it is out of date | 19:10 |
| clarkb | so I think we still haven't solved that entirely | 19:11 |
| clarkb | I think I may know why | 19:11 |
| clarkb | going to try a thing | 19:11 |
| gmann | you mean this - review.openstack.org/p ? | 19:11 |
| clarkb | yes | 19:11 |
| gmann | ahh | 19:11 |
| ajitha | clarkb you are saying about my issue? | 19:12 |
| clarkb | ajitha: yes, I'm testing something really quick | 19:13 |
| ajitha | clarkb: thanks | 19:13 |
| clarkb | ok that didn't fix it. I'll bring it up with mordred as he was driving this | 19:13 |
| clarkb | gmann: we should have d-g stop cloning from review.openstack.org/p/ | 19:14 |
| *** slaweq has quit IRC | 19:14 | |
| clarkb | maybe lets see what mordred says | 19:14 |
| ajitha | So it will be oslo.policy==3.0.3 for everyone right?.. i spend hours on analyzing | 19:15 |
| gmann | that whats i was thinking whether to fix d-g or wait for users to move to zuulv3 in V cycle. even we move all upstream CI to zuulv3 but still third party CI need it ? | 19:15 |
| clarkb | gmann: at some point we have to retire d-g | 19:16 |
| clarkb | it isn't being maintained properly anymore | 19:16 |
| gmann | true | 19:16 |
| clarkb | ajitha: no, the issue is specific to devstack-gate | 19:16 |
| clarkb | ajitha: and very little uses devstack-gate anymore as it hasn't been maintained for a few years | 19:16 |
| ajitha | ok | 19:16 |
| gmann | ajitha: you are not planning to move to zuulv3 ? just making sure that devstack-gate is not you are expecting to be maintained for ling term. | 19:17 |
| gmann | long | 19:18 |
| *** slaweq has joined #openstack-qa | 19:20 | |
| ajitha | yes.. we thought to move recently.. but it was working fine.. so didnt think off later.. will plan | 19:22 |
| gmann | thanks. | 19:26 |
| ajitha | will the devstack error will be solved today? | 19:26 |
| clarkb | ajitha: I don't know | 19:28 |
| ajitha | clarkb | 19:30 |
| ajitha | ok | 19:30 |
| AJaeger | gmann: could you review https://review.opendev.org/721606 later, please? I just noticed that we might need this... | 19:30 |
| ajitha | clarkb and gmann: Thanks for the help in figuring out the issue | 19:32 |
| gmann | ajitha: np! sorry for not solving it for you. | 19:37 |
| gmann | AJaeger: commented. | 19:37 |
| AJaeger | gmann: if we call tox, we need both since both are used currently. If we use it internally only, we can use TOX_CONSTRAINTS_FILE directly. | 19:40 |
| AJaeger | tools/tempest-plugin-sanity.sh looks like internal use - but please confirm | 19:40 |
| AJaeger | while roles/run-tempest/tasks/main.yaml is calling out-of-repo files, so needs both | 19:40 |
| gmann | AJaeger: yeah, almost all tox.ini use UPPER_CONSTRAINTS_FILE not sure what all moved to TOX_CONSTRAINTS_FILE | 19:43 |
| AJaeger | gmann: it's a slow move, I think... | 19:43 |
| gmann | yeah, and for tempest tox, i remember it is set from different places too devstack, grenade, d-g, tempest plugins etc | 19:44 |
| AJaeger | devstack does not set UPPER_CONSTRAINT, I checked - or what is set? | 19:45 |
| gmann | it is set on stable branch. on those using master tempest also but let me check | 19:46 |
| AJaeger | let me check other repos... | 19:46 |
| AJaeger | we should always set both | 19:46 |
| gmann | its on devstack rocky which use master run-tempest role | 19:48 |
| AJaeger | will check, thanks | 19:48 |
| gmann | but my concern is of tox.ini still use UPPER_CONSTRAINTS_FILE then why to set both. or we should plan for complete move soem time ?/ | 19:49 |
| gmann | sometime | 19:49 |
| openstackgerrit | Andreas Jaeger proposed openstack/devstack-gate master: Set TOX_CONSTRAINTS_FILE https://review.opendev.org/721718 | 19:52 |
| AJaeger | we need to plan for a move, first repos use TOX_CONSTRAINTS_FILE... | 19:55 |
| openstackgerrit | Andreas Jaeger proposed openstack/tempest master: Set TOX_CONSTRAINTS_FILE https://review.opendev.org/721606 | 20:00 |
| *** ajitha has quit IRC | 20:02 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest master: Use stable constraint in tox to release new tag for Ussuri https://review.opendev.org/720578 | 20:08 |
| clarkb | ajitha is gone now? | 20:09 |
| clarkb | fwiw I think the thing I did earlier may have actually addressed this for requirements | 20:09 |
| clarkb | it seems up to date now | 20:09 |
| gmann | kopecmartin: masayukig let's do this also before tempest release so that we can keep the tox constraint compatible with the source code of tag -https://review.opendev.org/#/c/720578/ | 20:09 |
| clarkb | so I think rerunning the jobs at this point might work as long as no other stale repos cause it problems | 20:09 |
| clarkb | gmann: ^ fyi | 20:09 |
| gmann | kopecmartin: masayukig and as soon as release is done we will revert it to master constraint | 20:10 |
| gmann | clarkb: i think ajitha did re-run ? | 20:10 |
| gmann | or no ? | 20:10 |
| kopecmartin | gmann: ok | 20:12 |
| gmann | kopecmartin: masayukig this can avoid setting the env var for constraint which i hacked during rocky testing on lot of places, devstack, grenade, d-g, heat plugin etc | 20:12 |
| *** paras333 has quit IRC | 20:14 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest master: Use the master constraint again in tox.ini https://review.opendev.org/721724 | 20:17 |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/devstack-plugin-ceph master: Updates script to use latest version of Ceph https://review.opendev.org/676722 | 20:17 |
| gmann | kopecmartin: masayukig FYI. this is kind of revert of that once Tempest 24.0.0 is released - https://review.opendev.org/#/c/721724/1 | 20:18 |
| clarkb | gmann: rerun again | 20:19 |
| gmann | ok. | 20:19 |
| AJaeger | gmann: I answered your comment on 721606 - I don't know what syntax you propose ;( Could you help, me please? | 20:29 |
| gmann | AJaeger: ah, I have not tried that actually or do not know if that is possible :). I was thinking if you know some hacking.... | 20:31 |
| AJaeger | gmann: I don't - sorry ;( | 20:32 |
| *** paras333 has joined #openstack-qa | 20:37 | |
| *** nweinber has quit IRC | 20:39 | |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/devstack-plugin-ceph master: Updates script to use latest version of Ceph https://review.opendev.org/676722 | 20:43 |
| *** paras333 has quit IRC | 21:38 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/patrole master: Update compute tests to adopt new policies https://review.opendev.org/720601 | 21:41 |
| *** slaweq has quit IRC | 21:42 | |
| *** paras333 has joined #openstack-qa | 21:52 | |
| *** raildo has quit IRC | 22:15 | |
| *** rcernin has joined #openstack-qa | 22:17 | |
| *** paras333 has joined #openstack-qa | 22:25 | |
| openstackgerrit | Merged openstack/devstack-plugin-cookiecutter master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721375 | 22:41 |
| openstackgerrit | Ghanshyam Mann proposed openstack/qa-specs master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721373 | 22:45 |
| openstackgerrit | Merged openstack/tempest-plugin-cookiecutter master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721372 | 22:46 |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-testr master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721112 | 22:47 |
| openstackgerrit | Merged openstack/karma-subunit-reporter master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721374 | 22:52 |
| *** paras333 has quit IRC | 22:55 | |
| openstackgerrit | Merged openstack/openstack-health master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721118 | 23:01 |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest-stress master: [ussuri][goal] Update contributor documentation https://review.opendev.org/721120 | 23:02 |
| *** tosky has quit IRC | 23:03 | |
| masayukig | gmann: +A for https://review.opendev.org/#/c/720578/ . | 23:12 |
| gmann | masayukig: thanks | 23:27 |
| openstackgerrit | Ghanshyam Mann proposed openstack/patrole master: Update compute tests to adopt new policies https://review.opendev.org/720601 | 23:34 |
| gmann | masayukig: can you review these also - https://review.opendev.org/#/q/topic:project-ptl-and-contrib-docs+status:open+owner:self | 23:34 |
| gmann | https://review.opendev.org/#/q/topic:project-ptl-and-contrib-docs+status:open | 23:35 |
| masayukig | gmann: sure, will do | 23:35 |
| masayukig | gmann: And, I'm now filling the form for the PTG registration, https://openstackfoundation.formstack.com/forms/june2020_virtual_ptg_survey And, I'm wondering the number of attendees, and "Are there other groups we need to avoid scheduling conflicts with?" | 23:36 |
| gmann | masayukig: as per last counted I think we are around 7-8 people but we can consider 10 or 15 max | 23:37 |
| gmann | masayukig: for me, I would like to avoid conflcit with nova, TC | 23:38 |
| masayukig | gmann: thanks! OK, I'll put 10 as attendees, and Nova and TC to avoid conflict | 23:40 |
| gmann | thanks. | 23:40 |
| gmann | masayukig: also sending that etherpad again to ML to ask for topic to discuss will be good. Previously it was with physical PTG so people might not have added the topic. | 23:41 |
| gmann | considering their absent for physical PTG | 23:42 |
| openstackgerrit | Ghanshyam Mann proposed openstack/patrole master: Add releasenote to tag the Patrole for Ussuri release https://review.opendev.org/721305 | 23:43 |
| *** paras333 has joined #openstack-qa | 23:46 | |
| *** paras333 has quit IRC | 23:51 | |
| *** paras333 has joined #openstack-qa | 23:52 | |
| *** paras333 has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!