| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Add rocky job to gate https://review.opendev.org/c/openstack/devstack/+/885469 | 05:02 |
|---|---|---|
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Add option to install everything in global venvs https://review.opendev.org/c/openstack/devstack/+/558930 | 05:09 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Add debian-bookworm job https://review.opendev.org/c/openstack/devstack/+/887547 | 05:11 |
| ykarel | kopecmartin, the patches i proposed yesterday were for different issue | 06:50 |
| ykarel | for current CentOS 9-stream issue reported https://bugs.launchpad.net/neutron/+bug/2029335 | 06:51 |
| ykarel | last night patches were just backport to fix C9-stream jobs in stable branches https://review.opendev.org/#/q/Icd99f467d47aaafaaf3ee8f2a3c4da08842cb672 | 06:52 |
| kopecmartin | ykarel: i see, thanks | 07:10 |
| opendevreview | yatin proposed openstack/devstack master: Woraround systemd issue on CentOS 9-stream https://review.opendev.org/c/openstack/devstack/+/890280 | 07:43 |
| kevko | hi, second day i'm debugging tempest.scenario.test_server_volume_attachment.TestServerVolumeAttachmentScenario.test_server_detach_rules and still failing ... i have configured service_user on both sides cinder and nova ...but still getting unathorized | 07:51 |
| kevko | https://paste.openstack.org/show/bYVAmLV2BQqoxE9LuRGa/ | 07:52 |
| kevko | can anyone help ? | 07:59 |
| frickler | ykarel: did you consider switching neutron CI to rocky instead? devstack is gating on that platform now, seems much more stable | 10:13 |
| opendevreview | Jakub Skunda proposed openstack/devstack stable/2023.1: git: git checkout for a commit hash combinated with depth argument https://review.opendev.org/c/openstack/devstack/+/890332 | 13:41 |
| opendevreview | Jakub Skunda proposed openstack/devstack stable/zed: git: git checkout for a commit hash combinated with depth argument https://review.opendev.org/c/openstack/devstack/+/890333 | 13:43 |
| opendevreview | Jakub Skunda proposed openstack/devstack stable/xena: git: git checkout for a commit hash combinated with depth argument https://review.opendev.org/c/openstack/devstack/+/890334 | 13:44 |
| opendevreview | Jakub Skunda proposed openstack/devstack stable/wallaby: git: git checkout for a commit hash combinated with depth argument https://review.opendev.org/c/openstack/devstack/+/890337 | 13:53 |
| opendevreview | Jakub Skunda proposed openstack/devstack stable/victoria: git: git checkout for a commit hash combinated with depth argument https://review.opendev.org/c/openstack/devstack/+/890338 | 13:54 |
| dansmith | wow, the top thing in gate right now has been running jobs for 2 hours and one job is still in the middle of devstack, running incredibly slowly, like one line of output at a time | 16:01 |
| dansmith | so definitely some sort of very slow worker | 16:01 |
| dansmith | slaweq: When we see a test that has a server that gets no active interface, can't set an IP and can't be SSH'd to, what should we be looking for? | 16:09 |
| dansmith | https://2e881fa3fb67c342c77f-bc5fbdd8c1ab00cec2f92d8667e378c5.ssl.cf2.rackcdn.com/888470/1/check/nova-multi-cell/75ab1ec/testr_results.html | 16:10 |
| dansmith | then: no interfaces have a carrier | 16:10 |
| opendevreview | Dan Smith proposed openstack/tempest master: Make stamp_pattern wait for SSHABLE https://review.opendev.org/c/openstack/tempest/+/890348 | 16:22 |
| opendevreview | Dan Smith proposed openstack/tempest master: Make stamp_pattern wait for SSHABLE https://review.opendev.org/c/openstack/tempest/+/890348 | 16:31 |
| opendevreview | Dan Smith proposed openstack/tempest master: Use vfat for timestamp https://review.opendev.org/c/openstack/tempest/+/890350 | 16:43 |
| dansmith | melwitt: this ^ is the change that is potentially helpful for the mkfs problem | 16:43 |
| melwitt | cool, it's worth a shot | 16:44 |
| dansmith | melwitt: did you have a test you depends-on'd your discard change that you used to determine that the discard thing wasn't helping? | 16:44 |
| melwitt | dansmith: no but the discard patch itself failed with the mke2fs timeout 🫤 | 16:45 |
| melwitt | so I gave up on it immediately | 16:45 |
| dansmith | oh I didn't see because it's +1 now, but I guess because it was rebased | 16:45 |
| melwitt | oh it looks ilke sean rebased it yeah | 16:47 |
| dansmith | unfortunately the giant nova-manila set has swamped the gate and nothing submitted in the last 3h is even running | 16:47 |
| dansmith | so won't get any data on that any time soon | 16:48 |
| dansmith | melwitt: not sure if you saw my reply on the sshable one, but my link above about unable to be sshable is why I made that change | 16:57 |
| dansmith | even though it may be futile, I think that console log means we decided about 1/4 of the way through it waiting for reachable metadata that it was un-ssh-able because the regular client call just tries once and bails | 16:58 |
| melwitt | ok, I guess I saw "2023-08-02 15:35:05,684 97347 ERROR [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@172.24.5.193 after 15 attempts. Proxy client: no proxy client" and the prior attempts. those are logged before the console log though | 17:01 |
| dansmith | um, okay maybe I'm missing something about where that's getting retried | 17:02 |
| dansmith | ah, in _get_ssh_connection() I see | 17:03 |
| melwitt | yeah.. there's at least a few layers but I saw the retry stuff here https://github.com/openstack/tempest/blob/4025fa3c8e5ba8de3f2cb60f4a225f1bc550b5e4/tempest/lib/common/ssh.py#L129 | 17:03 |
| dansmith | aight, well, we can drop that then if you want | 17:05 |
| melwitt | there's still a chance in might change something bc it would be waiting for ssh before adding the floating ip. it would be interesting to see if it does | 17:12 |
| dansmith | it also just makes that part of the server create like all the other things | 17:14 |
| dansmith | not sure that really matters much, but the consistency seems good | 17:14 |
| dansmith | we were initially going to set everything to SSHABLE by default but couldn't for a few reasons, so I think this is at least in support of that consistency goal if nothing else | 17:15 |
| melwitt | yeah, I had been thinking that but convinced myself it would be considered wasteful to add another ssh-able in, so I abandoned the idea. I think consistency is always a good thing | 17:17 |
| opendevreview | Merged openstack/tempest master: create default network for few more server tests https://review.opendev.org/c/openstack/tempest/+/890242 | 18:09 |
| opendevreview | melanie witt proposed openstack/tempest master: DNM test rbd retype https://review.opendev.org/c/openstack/tempest/+/890360 | 19:58 |
| dansmith | gmann: really seems like the increased concurrency is pushing us to more OOMs | 21:11 |
| dansmith | when I look at dumps, mysql isn't even using the most memory, but the six qemus are bigger of course, but with more pinned memory | 21:11 |
| melwitt | gmann: I'm trying to test something by adding an assert to a scenario test but I want it to run against wallaby.. I used "override-checkout: stable/wallaby" in my job definition and that checks out stuff for wallaby as expected, but the change I made to the tempest scenario test is not being used. is there a different way to do such a thing? | 21:32 |
| dansmith | melwitt: hmm I just thought about something.. what if you add required-projects: tempest? | 21:37 |
| dansmith | that normally says "get this from source" like for libraries | 21:38 |
| dansmith | although that might cause it to just checkout master, I dunno | 21:38 |
| gmann | melwitt: dansmith: on wallaby we use old tempest so it would not pick up the master change | 21:38 |
| dansmith | gmann: yeah I figured.. so there's no way to arrange it? | 21:39 |
| gmann | melwitt: dansmith: set these variable in job and it will use master tempest TEMPEST_BRANCH:master TEMPEST_VENV_UPPER_CONSTRAINTS:master | 21:40 |
| gmann | whether it will raise any requirement constraints issue or not that is something difficult to solve but you can try this way and see if master tempest/constarints work against wallaby or not | 21:40 |
| melwitt | gmann: sweet, thanks | 21:41 |
| dansmith | presumably it might also fail a bunch of other tests as a result of being on master now too | 21:41 |
| gmann | dansmith: on concurrency, is oom issues everytime ? and which job you are seeing it more? | 21:42 |
| dansmith | gmann: not everytime, but a bunch today | 21:42 |
| dansmith | all jobs I think | 21:42 |
| dansmith | I mean, a bunch of different ones, not any specific ones | 21:42 |
| dansmith | not on ceph though, but probably because it's limited to 3 | 21:42 |
| gmann | dansmith: more than test I think constraints will create issue as installing tempest with master constraints might conflict with what we have on wallaby but let's see if that work as wallaby is not so old | 21:42 |
| opendevreview | melanie witt proposed openstack/tempest master: DNM test rbd retype https://review.opendev.org/c/openstack/tempest/+/890360 | 21:46 |
| opendevreview | Ghanshyam proposed openstack/tempest master: DNM test rbd retype https://review.opendev.org/c/openstack/tempest/+/890360 | 23:18 |
| gmann | melwitt: ^^ that was still using stable constraints due to those setting in role. it should use master constraints now | 23:19 |
| melwitt | thank you gmann 🙏 | 23:20 |
| gmann | np! | 23:22 |
| opendevreview | Ghanshyam proposed openstack/tempest master: DNM test rbd retype https://review.opendev.org/c/openstack/tempest/+/890360 | 23:57 |
| gmann | melwitt: that worked for wallaby but not for train due to oslo.concurrency constraints. trying if ^^ this will work | 23:57 |
| melwitt | gmann: really cool, thank you :) | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!