| liuc49_ | openstack server add floating ip opins 172.24.4.34 | 02:20 |
|---|---|---|
| liuc49_ | ResourceNotFound: 404: Client Error for url: http://10.225.110.7:9696/networking/v2.0/floatingips/1b69a719-8d89-4427-be63-4704e27d4926, External network af849ff4-7d4a-45c4-b31e-07739ac98404 is not reachable from subnet adc5f1d5-8db0-4111-831d-529e0a98d75f. Therefore, cannot associate Port 06c0f9c7-98b9-4a80-bbc4-ecece81ca6ab with a Floating IP. | 02:20 |
| liuc49_ | What's the reason of above issue? I have reboot my VM. | 02:20 |
| ykarel | frickler, kopecmartin please have a look at https://review.opendev.org/c/openstack/devstack/+/892848 | 03:57 |
| opendevreview | Merged openstack/devstack master: [neutron] Rely on PATH env set by devstack https://review.opendev.org/c/openstack/devstack/+/892848 | 08:17 |
| liuc49 | I reboot my vm. Then create an instance and assign a floatingip, but cannot ping successfully. | 08:55 |
| ykarel | liuc49, check ip a show br-ex | 08:57 |
| frickler | liuc49: did you see my response yesterday? | 08:58 |
| frickler | liuc49: the "External network ... is not reachable" error indicates that there is no router connecting public and private network | 08:59 |
| ykarel | thx frickler i see you already pointed https://bugs.launchpad.net/devstack/+bug/2032774 yesterday, so it could be just that | 08:59 |
| * ykarel afk | 09:00 | |
| frickler | liuc49: if you loose your connection, you can check the channel logs at https://meetings.opendev.org/irclogs/%23openstack-qa/index.html | 09:01 |
| liuc49 | sorry, I miss your message yesterday. thanks. | 09:03 |
| liuc49 | Thanks for your answer. Resolved!!! | 09:20 |
| *** ykarel_ is now known as ykarel | 14:01 | |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Aug 29 15:00:14 2023 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours | 15:00 |
| kopecmartin | agenda ^^^ | 15:00 |
| frickler | \o brb | 15:00 |
| kopecmartin | o/ | 15:01 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:04 |
| kopecmartin | PTL and TC nominations are ongoing | 15:04 |
| kopecmartin | #link https://governance.openstack.org/election/ | 15:04 |
| kopecmartin | as there were no other nominees for PTL of QA team, I nominated myself again | 15:04 |
| kopecmartin | #topic Bobcat Priority Items progress | 15:05 |
| frickler | \o/ | 15:05 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-bobcat-priority | 15:05 |
| * kopecmartin checking for any news | 15:05 | |
| frickler | this fixed a remaining global_venv issue for neutron https://review.opendev.org/c/openstack/devstack/+/892848 | 15:06 |
| kopecmartin | great | 15:07 |
| kopecmartin | so now only this LP remains | 15:07 |
| kopecmartin | #link https://bugs.launchpad.net/devstack/+bug/2031639 | 15:07 |
| kopecmartin | i'll try to find a volunteer | 15:07 |
| kopecmartin | btw, we got our new intern to take a look at 'unit test coverage' - a good task for the beginning | 15:08 |
| frickler | in tempest or for which project? | 15:09 |
| kopecmartin | oh, the name doesn't mention, it's tempest | 15:09 |
| frickler | ok, nice | 15:10 |
| kopecmartin | that's about it regarding the priority items | 15:11 |
| kopecmartin | #topic Gate Status Checks | 15:11 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:11 |
| kopecmartin | no patches there | 15:12 |
| kopecmartin | anything urgent to review? | 15:12 |
| kopecmartin | #topic Bare rechecks | 15:13 |
| kopecmartin | #link https://etherpad.opendev.org/p/recheck-weekly-summary | 15:13 |
| kopecmartin | only 4 rechecks (2 bare) over the last 7 days, that's fine | 15:14 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:16 |
| kopecmartin | periodic stable full | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?pipeline=periodic-stable&job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-zed&job_name=tempest-full-2023-1 | 15:16 |
| kopecmartin | periodic stable slow | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-slow-2023-1&job_name=tempest-slow-zed&job_name=tempest-slow-yoga&job_name=tempest-slow-xena | 15:16 |
| kopecmartin | periodic extra tests | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-2023-1-extra-tests&job_name=tempest-full-zed-extra-tests&job_name=tempest-full-yoga-extra-tests&job_name=tempest-full-xena-extra-tests | 15:16 |
| kopecmartin | periodic master | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:16 |
| kopecmartin | all seems fine, apart known tempest-full-test-account-no-admin-py3 failures | 15:17 |
| kopecmartin | however, we finally have some patches that should address taht | 15:17 |
| kopecmartin | at least partially | 15:17 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/890798 | 15:17 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/892663 | 15:17 |
| kopecmartin | gmann: i commented the last one - 663 - there is a weird assert, i don't get how it could work this long ^ | 15:18 |
| kopecmartin | #topic Distros check | 15:19 |
| kopecmartin | cs-9 | 15:19 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:19 |
| kopecmartin | debian | 15:19 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&job_name=devstack-platform-debian-bookworm&skip=0 | 15:19 |
| kopecmartin | focal | 15:19 |
| kopecmartin | #link https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-ubuntu-focal&skip=0 | 15:19 |
| kopecmartin | rocky | 15:19 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx | 15:19 |
| kopecmartin | openEuler | 15:19 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-22.03-ovn-source&job_name=devstack-platform-openEuler-22.03-ovs&skip=0 | 15:19 |
| kopecmartin | don't see any unusual patterns, all good \o/ | 15:20 |
| kopecmartin | #topic Sub Teams highlights | 15:21 |
| kopecmartin | Changes with Review-Priority == +1 | 15:21 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:21 |
| kopecmartin | no patches | 15:21 |
| opendevreview | Katarina Strenkova proposed openstack/tempest master: Add unit tests to increade coverage https://review.opendev.org/c/openstack/tempest/+/893080 | 15:21 |
| kopecmartin | #topic Open Discussion | 15:21 |
| kopecmartin | anything for the open discussion? | 15:21 |
| frickler | seems not | 15:23 |
| kopecmartin | #topic Bug Triage | 15:24 |
| kopecmartin | #link https://etherpad.openstack.org/p/qa-bug-triage-bobcat | 15:24 |
| kopecmartin | #link https://bugs.launchpad.net/devstack/+bug/2007287 | 15:24 |
| kopecmartin | since the patches are merged, can we consider that done? | 15:25 |
| kopecmartin | #link https://review.opendev.org/q/Ic1c9cb2e04259aaf78342b4bb2161235666abb49 | 15:25 |
| frickler | I guess yes | 15:26 |
| kopecmartin | cool, iv'e just closed that | 15:28 |
| kopecmartin | that's all from my side :) | 15:29 |
| kopecmartin | if there isn't anything else, we can call it an hour | 15:29 |
| kopecmartin | thanks frickler | 15:29 |
| frickler | thx kopecmartin | 15:29 |
| kopecmartin | #endmeeting | 15:30 |
| opendevmeet | Meeting ended Tue Aug 29 15:30:05 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:30 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-29-15.00.html | 15:30 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-29-15.00.txt | 15:30 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-29-15.00.log.html | 15:30 |
| gmann | kopecmartin: maybe yes we need to modify that assert as they are in same project. one option is to check the existence of the one that test create or delete | 17:14 |
| gmann | commented in review also | 17:16 |
| *** melwitt_ is now known as melwitt | 18:34 | |
| sean-k-mooney | looks like we might have some issue with pbr | 18:35 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/c5d49b4d4359429ea1c4887559184f12/log/job-output.txt#1431 | 18:35 |
| clarkb | that failrue appears to occur in virtualenv? | 18:37 |
| clarkb | oh in tox's virtualenv implemetnation | 18:37 |
| sean-k-mooney | that was a tox job | 18:37 |
| sean-k-mooney | i just notice a lot of read on a gate patch and all the tox jobs seam ot have failed | 18:38 |
| clarkb | ya so tox reimplementats a bunch of packaging stuff and this si explicitly why I stopped wanting to use tox | 18:38 |
| clarkb | their interpretation of related peps is very strict and they ignore a lot of MAY statements that improve compatibility with existing tooling | 18:38 |
| sean-k-mooney | ack well it looks like this is causing a gate outage in many repos | 18:38 |
| sean-k-mooney | i would assume a tox release just happend | 18:39 |
| sean-k-mooney | or virtualenve release | 18:39 |
| sean-k-mooney | and we might need to pin until we can figure this out | 18:39 |
| clarkb | sean-k-mooney: note the line you linked is it reporting exit 0 for successful install of pbr and wheel ebcause they are already isntalled | 18:39 |
| clarkb | then whatever it does next breaks | 18:39 |
| sean-k-mooney | right i was assuming the next line was related to it bu i guess its not | 18:40 |
| sean-k-mooney | tox/session/cmd/run/single.py:59 | 18:40 |
| clarkb | I don't think so. I think whatever the next thing tox is doing is breaking but it doesn't log what it is doing as clearly | 18:40 |
| sean-k-mooney | tox had a release on the 21st | 18:40 |
| clarkb | I think it is trying to build the project itself to isntall it | 18:41 |
| clarkb | in this case nova | 18:41 |
| sean-k-mooney | perhaps its failing in neutron cind an dkobe i think too | 18:41 |
| sean-k-mooney | i think this might still be related to pbr | 18:42 |
| clarkb | it could be related to pbr. It mentions pep517 here https://zuul.opendev.org/t/openstack/build/c5d49b4d4359429ea1c4887559184f12/log/job-output.txt#1467 and setuptools.build_meta here https://zuul.opendev.org/t/openstack/build/c5d49b4d4359429ea1c4887559184f12/log/job-output.txt#1465 | 18:43 |
| clarkb | weird that it wouldn't be a problem for a whole week if it was due to a tox update though. setuptools updated even less recently | 18:43 |
| sean-k-mooney | well i just checked and i dont see tox in uc | 18:44 |
| sean-k-mooney | so do we have a way to pin it? | 18:44 |
| clarkb | I don't think it is since you have to have tox before you apply uc | 18:44 |
| clarkb | its a chicken and egg problem for local developers in particular | 18:44 |
| sean-k-mooney | oh right | 18:44 |
| clarkb | however you can instruct tox to install an older or newer version of tox in a separate env and then run from there | 18:44 |
| sean-k-mooney | but doesing tox supprot installing version of itself into the venvs | 18:44 |
| clarkb | yes | 18:45 |
| sean-k-mooney | but that would require modifyign the tox.ini in the proejcts | 18:45 |
| clarkb | yup | 18:45 |
| clarkb | probably the best first step here is to try and replicate locally and then start digging into what tox is actually doing and take it from there | 18:45 |
| clarkb | if you can't reproduce locally then go for a node hodl. but I suspect this will be reproduceable locally since it appears to be tox complaining about something | 18:46 |
| sean-k-mooney | we might want to send a status notice | 18:46 |
| sean-k-mooney | since this looks like its going to break all tox jobs to me | 18:46 |
| clarkb | I was going to mention it in the tc meeting as it winds down | 18:46 |
| sean-k-mooney | at least of the openstack tenant | 18:46 |
| sean-k-mooney | ack | 18:46 |
| sean-k-mooney | ok i was jus tlookign at the gate to see if a patch merged before i finsihsed for today but i might try and look at this tomorow | 18:47 |
| sean-k-mooney | it woudl proably be beter to have a systematic approch to fixing this rahter then doint it diffently in each project however | 18:48 |
| clarkb | ya need to understand it first though | 18:48 |
| sean-k-mooney | https://github.com/openstack/requirements/blob/master/tox.ini#L8 | 18:50 |
| sean-k-mooney | we are clamping the verison of virtual evne we are using in the uc jobs | 18:51 |
| sean-k-mooney | so maybe this is realted | 18:51 |
| sean-k-mooney | we are not doing that on nova and likely not on other repos that are failing | 18:51 |
| clarkb | note the failure seems to be in tox's tox_env/python/virtual_env code | 18:54 |
| clarkb | which may or may not be related to virtualenv proper | 18:54 |
| haleyb | neutron is just as broken, and no obvious change there | 18:56 |
| clarkb | https://github.com/tox-dev/tox/commit/e66e346d49229899b172a5d32f602001149e9d51 possibly related? though the old side code appears to be what we have in the traceback not the new side | 18:58 |
| clarkb | tox 4.10.0 is what is used according to the job log | 18:59 |
| clarkb | which is the new release a week ago | 18:59 |
| haleyb | pep8 runs fine locally with tox 4.10.0 | 19:01 |
| haleyb | and i see successful runs in the gate using it right before breakage | 19:05 |
| clarkb | oh wait no the failed line is 324 on the new side of pyproject.py | 19:06 |
| clarkb | so it is failing in the new code | 19:06 |
| haleyb | one package i see different is pyproject-api, pyproject-api-1.5.4 good, pyproject-api-1.6.0 bad | 19:11 |
| haleyb | if that's the trigger | 19:12 |
| sean-k-mooney | ya that sounds proable | 19:15 |
| clarkb | thats this https://github.com/tox-dev/pyproject-api its tox's reimplemtnation of pep517 aiui | 19:15 |
| sean-k-mooney | it just had another release | 19:15 |
| sean-k-mooney | https://pypi.org/project/pyproject-api/ | 19:16 |
| clarkb | and as mentioned before they have a very strict read of the pep compatibility be damned | 19:16 |
| clarkb | if this ends up ebing the problem a very similar issue back in december is why I started pushing towards nox for stuff I have a hand in | 19:16 |
| sean-k-mooney | so 1.6.0 relased 2 hours ago and they released 1.6.1 an hour ago | 19:16 |
| sean-k-mooney | https://github.com/tox-dev/pyproject-api/releases/tag/1.6.1 | 19:17 |
| clarkb | https://github.com/tox-dev/pyproject-api/commit/c793c2c6e636a344a8b11698333fd588e5159435 is the only commit in 1.6.1 | 19:18 |
| haleyb | i just forced pyproject-api==1.6.0 here and pep8 ran fine, but don't know how different from the gate i am | 19:23 |
| clarkb | the failed job used 1.6.1 | 19:24 |
| clarkb | https://zuul.opendev.org/t/openstack/build/c5d49b4d4359429ea1c4887559184f12/log/job-output.txt#718 | 19:25 |
| haleyb | pyproject-api-1.6.0 was in one of the neutron failed jobs | 19:25 |
| haleyb | https://f75b41bb61072b844b31-b8d0e37c8407321b7ba0b13922acea87.ssl.cf1.rackcdn.com/893097/1/check/openstack-tox-pep8/75d8340/job-output.txt | 19:26 |
| sean-k-mooney | ok i can repoduce it locally | 19:34 |
| clarkb | haleyb: if you use the zuul log browser you can link to specific lines :) | 19:34 |
| sean-k-mooney | it depned on both the tox and pyproject-api version | 19:34 |
| sean-k-mooney | clarkb: on some filed.... | 19:34 |
| clarkb | sean-k-mooney: ya text logs in particular | 19:35 |
| sean-k-mooney | clarkb: i have add the pretty logs crash both firefox and chrome | 19:35 |
| sean-k-mooney | text logs are fine but not if they are 30mb | 19:35 |
| clarkb | I haven't run into that except for when they end up being huge | 19:35 |
| sean-k-mooney | with each line marked up as a div | 19:35 |
| sean-k-mooney | on longer jobs i have hadd issues with both nova-compute and neutron server logs | 19:36 |
| sean-k-mooney | anywyay if you use tox 4.10.0 and pyproject-api 1.6.0 | 19:36 |
| clarkb | in a lot of cases the uinderlying issue there is spammy warnings that should just be refactored. I think openstack did a bit of cleanup on those | 19:36 |
| sean-k-mooney | then it crashses | 19:36 |
| sean-k-mooney | tehy released tox 4.11.0 and made it require 1.6.1 | 19:36 |
| clarkb | sean-k-mooney: does tox 4.11.0 fix it? | 19:37 |
| sean-k-mooney | yep | 19:37 |
| sean-k-mooney | well tox 4.11.0 required pyproject-api>=1.6.1 | 19:37 |
| fungi | and doesn't error? | 19:38 |
| sean-k-mooney | but that combination works | 19:38 |
| clarkb | no release notes published about 4.11 yet which will hopefully give more insight | 19:38 |
| sean-k-mooney | they did the release 17 minutes ago | 19:38 |
| sean-k-mooney | https://github.com/tox-dev/pyproject-api/issues/101 i just filed this | 19:40 |
| haleyb | clarkb: if i knew what the zuul log browser was i'd use it :) i'm a luddite | 19:41 |
| clarkb | haleyb: instead of clicking on the (raw) log button click on the filename in under logs in the zuul result page | 19:42 |
| sean-k-mooney | also added repoducer steps | 19:43 |
| clarkb | this should naturally fix itself in the gate then if latest tox fixes it | 19:44 |
| sean-k-mooney | yep | 19:44 |
| sean-k-mooney | i mainly filed the issue just so they were aware of the breakage | 19:45 |
| sean-k-mooney | i expect them to close it | 19:45 |
| sean-k-mooney | "python3 -m pip install tox===4.11.0" | 19:45 |
| sean-k-mooney | that fixes it in teh venv | 19:45 |
| sean-k-mooney | so we just need to recheck the failed thigns tomorrow | 19:45 |
| sean-k-mooney | speaking of tomorrow caht to you then o/ | 19:46 |
| clarkb | goodnight! (I believe I got the timezone conversion correct) | 19:46 |
| opendevreview | Martin Kopec proposed openstack/tempest master: Fix clean up for test_access_rules tests https://review.opendev.org/c/openstack/tempest/+/892663 | 19:50 |
| haleyb | neutron gate seems happy now :) | 20:15 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!