| brinzhang | bauzas, gibi, gmann, stenphenfin, sean-k-mooney: I have reprosed the remove_tenant_id spec, do you have time to review? | 02:57 |
|---|---|---|
| *** redrobot2 is now known as redrobot | 05:58 | |
| gibi | brinzhang: ack | 06:11 |
| *** gibi is now known as gibi_pto_back_thu | 06:11 | |
| brinzhang | gibi: thanks | 06:11 |
| opendevreview | Elod Illes proposed openstack/nova stable/rocky: [stable-only] Pin virtualenv and setuptools https://review.opendev.org/c/openstack/nova/+/816436 | 07:56 |
| kashyap | frickler: Morning, I also filed this one: https://gitlab.com/libvirt/libvirt/-/issues/233 | 09:07 |
| kashyap | (It also links to the RFE you posted.) | 09:08 |
| frickler | kashyap: thx for the link, I'll watch that issue, too | 09:19 |
| lyarwood | frickler: do we have an openstack bug tracking this? | 09:21 |
| lyarwood | I seem to recall something for the centos issues etc but I can't find it | 09:22 |
| frickler | lyarwood: actually https://bugs.launchpad.net/devstack/+bug/1938914 might have been the same thing in retrospect | 09:26 |
| frickler | I was wondering early why centos jobs didn't seem to be affected and found the swap setting that likely mitigates it | 09:26 |
| frickler | s/early/earlier/, like yesterday | 09:27 |
| lyarwood | hmm is that setting also used by centos-stream jobs? | 09:27 |
| lyarwood | because they were still failing 3 weeks ago | 09:27 |
| * lyarwood checks | 09:27 | |
| lyarwood | yeah they are centos stream | 09:28 |
| frickler | https://review.opendev.org/c/openstack/devstack/+/803706/2/.zuul.yaml | 09:28 |
| lyarwood | okay weird, https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-centos-8-stream is green but I'm sure things were borked the last time I looked | 09:28 |
| lyarwood | ah I think it was tempest-full-py3-centos-8-stream that I was having issues with | 09:30 |
| lyarwood | in https://review.opendev.org/c/openstack/tempest/+/797614 | 09:30 |
| lyarwood | and that doesn't have the swap setting | 09:30 |
| lyarwood | cool | 09:31 |
| frickler | lyarwood: looks likely related. maybe to verify you'll want to build a custom qemu like I did for bullseye | 09:32 |
| lyarwood | frickler: yeah, I'll add the swap for now. FWIW we are likely going to escalate this to the virt folks downstream so we can work something out asap as Nova isn't even able to workaround this | 09:33 |
| lyarwood | and we really don't want to use custom builds upstream | 09:33 |
| kashyap | frickler: lyarwood: So, even if libvirt upstream provides a patch right away, next upstream release is only a month away | 09:34 |
| lyarwood | right and then it needs to be backported | 09:34 |
| lyarwood | so it's going to take time | 09:34 |
| frickler | lyarwood: kashyap: I was about to say the same thing. and getting it into Ubuntu 22.04 and/or Bullseye will even be more difficult | 09:34 |
| opendevreview | Merged openstack/nova master: db: Enable auto-generation of API DB migrations https://review.opendev.org/c/openstack/nova/+/812144 | 09:35 |
| lyarwood | I guess the swap workaround might be enough as well? | 09:35 |
| opendevreview | Merged openstack/nova master: db: De-duplicate list of removed table columns https://review.opendev.org/c/openstack/nova/+/814489 | 09:36 |
| kashyap | frickler: So, are _all_ CI jobs failing here? | 09:36 |
| kashyap | lyarwood: Another idea to explore is to temporarily pin to a lower QEMU version? | 09:37 |
| lyarwood | kashyap: I don't think that's an option with centos-8-stream? | 09:37 |
| frickler | kashyap: it depends on tempest scheduling, so it is not 100%. with -c2, which lyarwood set for the tempest patch, we had around 50% failure rate for Bullseye | 09:39 |
| kashyap | lyarwood: I don't know top off my head what's possible w/ -8-stream | 09:39 |
| kashyap | frickler: Hmm | 09:44 |
| lyarwood | frickler: have you tried the Bullseye jobs with increased swap? | 09:45 |
| frickler | lyarwood: not yet, for now it seemed o.k. with -c1, but I can give that a spin | 09:53 |
| lyarwood | ack that might be a better short term workaround | 09:54 |
| kashyap | lyarwood: What is -c1? | 09:59 |
| lyarwood | concurrency 1 , a tempest configurable that controls how many tests run at the same time | 10:00 |
| lyarwood | so -c1 meaning run one test at a time | 10:00 |
| kashyap | Ah, thanks | 10:00 |
| kashyap | lyarwood: Also, BTW: I was just chatting to the QEMU TCG maint - | 10:00 |
| lyarwood | -c1 is just shorthand FWIW it isn't the actual option | 10:01 |
| kashyap | lyarwood: They won't revert it, because ramping up that value makes TCG much faster (which also benefits us, as frickler also found it from his tests) | 10:01 |
| lyarwood | kk | 10:01 |
| kashyap | frickler: lyarwood: Can you point to the sepcs of the VM images? I'm being asked about it | 10:02 |
| kashyap | Is it 8G RAM per env? | 10:03 |
| kashyap | s/sepcs/specs/ | 10:04 |
| kashyap | lyarwood: BTW: on version pinning I just found this one: https://dnf-plugins-core.readthedocs.io/en/latest/versionlock.html | 10:06 |
| kashyap | lyarwood: It _is_ possible to lock to `dnf versionlock` | 10:06 |
| lyarwood | right my point was if we had an older version available in the repos | 10:06 |
| kashyap | lyarwood: frickler: Also, can the concurrency variable limit the no. of VMs that can run? | 10:07 |
| lyarwood | that's essentially what it's doing yes | 10:07 |
| lyarwood | some tests have multiple vms | 10:07 |
| lyarwood | but overall it should limit it to roughly twice the concurrency value | 10:08 |
| kashyap | lyarwood: *Twice* the concurrency value? Or half? | 10:08 |
| * kashyap goes to read - https://docs.openstack.org/tempest/latest/run.html | 10:08 | |
| kashyap | Okay, there's also a --serial/-t -- but that'll be slow, I guess. How slow, only one way to figure out | 10:09 |
| lyarwood | kashyap: twice, concurrency 2 will run 2 tests at the same time and both of these could have 2 instances each (at most IIRC, I can't think of anything with more than 2) | 10:09 |
| kashyap | Ah, I see. Thanks | 10:09 |
| lyarwood | so we are already running -c1 that's essentially -s | 10:11 |
| lyarwood | and my centos jobs are using -c2 and now with an increased amount of swap | 10:11 |
| lyarwood | hopefully that's enough to get things stable in the meantime | 10:11 |
| kashyap | Ah, that's good to know. You only mention CentOS. But Debian "Bullseye" jobs also seems to be impacted w/ it, yeah? | 10:13 |
| lyarwood | yeah correct, as above frickler is going to play around with adding the extra swap now I believe | 10:19 |
| kashyap | Thanks! So extra swap + serial should get out of the ditch temporarily? | 10:20 |
| kashyap | By "serial" I mean "-c1" / --serial | 10:21 |
| lyarwood | well as high as we can go | 10:21 |
| lyarwood | so hopefully -c2 | 10:21 |
| lyarwood | with swap | 10:21 |
| lyarwood | and failing that -c1 / serial | 10:21 |
| lyarwood | assuming we don't hit the job timeout of 2 hours | 10:21 |
| kashyap | Noted, thanks! | 10:29 |
| frickler | lyarwood: the job timeout for both platform jobs in devstack has been raised to 3 hours | 10:35 |
| frickler | kashyap: 8G is the memory size for default CI instances, we have that documented somewhere ... | 10:36 |
| frickler | https://docs.opendev.org/opendev/infra-manual/latest/testing.html#unprivileged-single-use-vms | 10:36 |
| frickler | does that answer your question or what specs were you referring to? | 10:37 |
| * kashyap clicks | 10:38 | |
| kashyap | frickler: Great link; yes, it does. One of the QE devs was asking of it | 10:38 |
| stephenfin | lyarwood: when I do 'openstack server add volume $SERVER $VOLUME --tag $TAG', where should I expect to see the tag appear? | 11:09 |
| lyarwood | stephenfin: https://docs.openstack.org/api-ref/compute/?expanded=attach-a-volume-to-an-instance-detail#id190 - it's in the response from 2.70 | 11:10 |
| lyarwood | stephenfin: if osc can handle that of course | 11:10 |
| lyarwood | stephenfin: looks like we don't dump anything on attach, it should also be there with a list of the attached volumes using 2.70 | 11:13 |
| lyarwood | stephenfin: you added support for it | 11:13 |
| stephenfin | yeah, I'm not sure why. Seems we just dispose of the result from the API | 11:15 |
| opendevreview | Lee Yarwood proposed openstack/nova-specs master: libvirt: Allow Manila shares to be directly attached to instances https://review.opendev.org/c/openstack/nova-specs/+/813180 | 12:00 |
| stephenfin | lyarwood: FYI https://review.opendev.org/c/openstack/python-openstackclient/+/816491 | 12:24 |
| stephenfin | also, I rediscovered 'openstack server volume list'...which it turns out I also wrote /o\ That confirms the tag gets passed through | 12:27 |
| stephenfin | With the benefit of hindsight, it should probably have been called 'openstack server volume attachment list' (and 'openstack server add volume' should be 'openstack server volume attachment create') but 🤷♂️ | 12:27 |
| opendevreview | Julia Kreger proposed openstack/nova master: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/813263 | 12:57 |
| lyarwood | stephenfin: thanks, looking now and yeah with hindsight adding attachment in there would make sense but meh | 14:45 |
| opendevreview | Julia Kreger proposed openstack/nova master: WIP Ironic - Handle instance host on rebalance https://review.opendev.org/c/openstack/nova/+/813897 | 15:05 |
| johnthetubaguy[m] | dansmith: for those policy patches, I should be availabe to help review those when they are ready | 16:02 |
| dansmith | johnthetubaguy[m]: ack, I'll continue working on getting tests passed and then will poke you | 16:03 |
| johnthetubaguy[m] | cool, awesome | 16:03 |
| gmann | johnthetubaguy[m]: dansmith i can too help in code/review. do you want me to create xls/wiki listing all APIs policy and we weight on what is system scope and project then code. so that we can visualize overall pic instead of going one by in code or separate patches ? | 16:05 |
| gmann | that was what we missed last time and mixed up scope in many APIs | 16:05 |
| gmann | i saw many projects doing it and it was helpful | 16:06 |
| dansmith | gmann: sure, I've just been focusing on servers for the moment, but that might be good to have a burndown chart | 16:07 |
| dansmith | abhi did that for policy refactor in glance and it was nice | 16:08 |
| gmann | ok, i will do that and then we can keep reviewing those | 16:08 |
| opendevreview | Merged openstack/os-vif master: Use TCP keepalives for ovsdb connections https://review.opendev.org/c/openstack/os-vif/+/805223 | 17:00 |
| opendevreview | Merged openstack/os-vif master: only register tables used by os-vif https://review.opendev.org/c/openstack/os-vif/+/805625 | 17:03 |
| opendevreview | Jan Hartkopf proposed openstack/nova-specs master: update userdata https://review.opendev.org/c/openstack/nova-specs/+/816542 | 17:27 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove models that were moved to the API database https://review.opendev.org/c/openstack/nova/+/812149 | 17:34 |
| opendevreview | Stephen Finucane proposed openstack/nova master: objects: Remove 'bandwidth' fields from notifications https://review.opendev.org/c/openstack/nova/+/812148 | 17:34 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove models for removed services, features https://review.opendev.org/c/openstack/nova/+/812150 | 17:34 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove nova-network models https://review.opendev.org/c/openstack/nova/+/812151 | 17:34 |
| opendevreview | Julia Kreger proposed openstack/nova master: WIP Ironic - Handle instance host on rebalance https://review.opendev.org/c/openstack/nova/+/813897 | 17:56 |
| sean-k-mooney | stephenfin: no you got the name right the resouce is the volumen not the attachments | 18:46 |
| sean-k-mooney | stephenfin: we shoudl not have the concept of port bidnings, volume attachments, cyborg arqs or manilla shares in our apis | 18:47 |
| sean-k-mooney | the can live in the other service or internally in our dbs but as a use i really should discib what resouces are assocaited with the server instace not how or where | 18:48 |
| sean-k-mooney | the how and where are all internal implemeation detail that users shoudl not need to reason about if we did things correctly | 18:48 |
| opendevreview | Merged openstack/nova-specs master: Re-propose Remove tenant_id https://review.opendev.org/c/openstack/nova-specs/+/809939 | 19:24 |
| opendevreview | Merged openstack/nova master: Avoid unbound instance_uuid var during delete https://review.opendev.org/c/openstack/nova/+/805605 | 19:51 |
| opendevreview | Merged openstack/nova master: Reno for qos-minimum-guaranteed-packet-rate https://review.opendev.org/c/openstack/nova/+/805046 | 19:51 |
| opendevreview | Merged openstack/nova master: db: Remove unused build_requests columns https://review.opendev.org/c/openstack/nova/+/812145 | 19:52 |
| opendevreview | Merged openstack/nova master: Prevent leaked eventlets to send notifications https://review.opendev.org/c/openstack/nova/+/814036 | 20:49 |
| opendevreview | Ghanshyam proposed openstack/nova master: Remove broken tempest-full-py3-opensuse15 job https://review.opendev.org/c/openstack/nova/+/816574 | 20:57 |
| opendevreview | Merged openstack/nova master: api: enable oslo.reports when using uWSGI https://review.opendev.org/c/openstack/nova/+/810922 | 21:57 |
| opendevreview | Merged openstack/nova stable/wallaby: Parse alias from domain hostdev https://review.opendev.org/c/openstack/nova/+/807767 | 22:10 |
| opendevreview | Merged openstack/nova stable/ussuri: Fix error that cannot overwrite policy rule for 'forced_host' https://review.opendev.org/c/openstack/nova/+/802238 | 22:42 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!