mriedem | besides the API service is registered in cell0 http://paste.openstack.org/show/744985/ | 00:01 |
---|---|---|
sean-k-mooney | http://paste.openstack.org/show/744986/ | 00:02 |
mriedem | this should be the route handler https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/versionsV21.py#L23 | 00:02 |
dansmith | yeah I thought we surveyed the service version of the computes somewhere early like that | 00:02 |
dansmith | have to do some debug printing | 00:02 |
*** wolverineav has joined #openstack-nova | 00:03 | |
mriedem | so we have the recentish change where on service startup we try to get the minimum compute service version, | 00:03 |
sean-k-mooney | so if you http://192.168.100.19/compute/ it has the microverion in the responce. isnt that how it fitures out the version to use | 00:03 |
mriedem | sean-k-mooney: the services table version is not the same as microversions | 00:03 |
sean-k-mooney | ok | 00:03 |
*** markvoelker has quit IRC | 00:06 | |
mriedem | hmm, i think it's the compute.API code trying to create an RPC API client | 00:10 |
mriedem | Feb 13 00:09:57 downcell devstack@n-api.service[14623]: DEBUG nova.compute.rpcapi [None req-53ebccae-d210-4b14-af5c-02775f3d36e8 None None] Not caching compute RPC version_cap, because min service_version is 0. Please ensure a nova-compute service has been started. Defaulting to current version. {{(pid=14625) _determine_version_cap /opt/stack/nova/nova/compute/rpcapi.py:410}} | 00:10 |
*** _alastor_ has quit IRC | 00:11 | |
dansmith | ah could be | 00:11 |
sean-k-mooney | mriedem could this be related https://github.com/openstack/nova/blob/master/nova/cmd/api.py#L47 | 00:12 |
mriedem | n-api service reports as active | 00:12 |
mriedem | sean-k-mooney: this isn't using that code | 00:12 |
mriedem | with default devstack n-api is running under uwsgi so https://github.com/openstack/nova/blob/master/nova/api/openstack/wsgi_app.py#L76 | 00:13 |
sean-k-mooney | its also done in the standalone api https://github.com/openstack/nova/blob/master/nova/cmd/api_os_compute.py#L42 | 00:13 |
mriedem | which now that i look at that i guess we don't cache the service versions | 00:13 |
mriedem | sean-k-mooney: NOT USING nova/cmd/* | 00:13 |
mriedem | dansmith: ^ aside that n-api under uwsgi isn't caching service versions | 00:13 |
sean-k-mooney | on right because the api is under uwsig in stead of python webserver | 00:14 |
dansmith | mriedem: ack | 00:14 |
*** ade_lee_ has joined #openstack-nova | 00:14 | |
mriedem | https://bugs.launchpad.net/nova/+bug/1815692 | 00:16 |
openstack | Launchpad bug 1815692 in OpenStack Compute (nova) "minimum nova-compute service version is not cached in nova-api when running under wsgi" [Low,Triaged] | 00:16 |
mriedem | so i think the hang is on | 00:17 |
mriedem | https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/api.py#L261 | 00:17 |
mriedem | https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L373 | 00:18 |
*** david-lyle has joined #openstack-nova | 00:18 | |
mriedem | https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/compute/rpcapi.py#L395 | 00:18 |
*** manjeets has quit IRC | 00:18 | |
*** manjeets has joined #openstack-nova | 00:19 | |
mriedem | and i suppose i'm seeing that for every controller that is initializing a compute.API object | 00:19 |
mriedem | which is most of them.... | 00:19 |
mriedem | but doesn't really explain why GET / hangs | 00:20 |
sean-k-mooney | look like that was intoduced by dan to fix https://bugs.launchpad.net/nova/+bug/1807044 | 00:20 |
openstack | Launchpad bug 1807044 in OpenStack Compute (nova) queens "nova-api startup does not scan cells looking for minimum nova-compute service version" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | 00:20 |
*** ade_lee_ has quit IRC | 00:20 | |
mriedem | yes i know, i approved it | 00:21 |
*** ade_lee_ has joined #openstack-nova | 00:21 | |
*** dklyle has quit IRC | 00:21 | |
sean-k-mooney | sorry just loading context for myself that said im going to sleep soon | 00:21 |
mriedem | what i need to figure out is why that impacts the versions controller route handler which doesn't use the compute API object | 00:22 |
*** sdake has quit IRC | 00:26 | |
*** tetsuro has joined #openstack-nova | 00:26 | |
*** ade_lee_ has quit IRC | 00:30 | |
*** ade_lee_ has joined #openstack-nova | 00:30 | |
*** sdake has joined #openstack-nova | 00:32 | |
*** sdake has quit IRC | 00:32 | |
mriedem | ok stopped the crazy startup noise by setting upgrade_levels.compute=rocky in nova.conf | 00:34 |
mriedem | dansmith: huh and this time nova list worked | 00:34 |
mriedem | with 2.69, i got the partial results back | 00:35 |
dansmith | well, because no =auto right? | 00:35 |
mriedem | still get tracebacks in the api logs but didn't hang | 00:35 |
mriedem | yes | 00:35 |
mriedem | Feb 13 00:34:43 downcell devstack@n-api.service[15785]: WARNING nova.compute.multi_cell_list [None req-76014247-0c3c-4758-9014-5c69f653891f demo demo] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 is not responding and hence is being omitted from the results | 00:35 |
mriedem | Feb 13 00:34:43 downcell devstack@n-api.service[15785]: WARNING nova.compute.api [None req-76014247-0c3c-4758-9014-5c69f653891f demo demo] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 is not responding and hence only partial results are available from this cell if any. | 00:35 |
dansmith | nice | 00:35 |
mriedem | i also turned those retry/interval db options to 1 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move binding ips to privsep. https://review.openstack.org/622151 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: create_veth_pair is unused, remove it. https://review.openstack.org/624226 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Create specialist set_macaddr_and_vlan helper. https://review.openstack.org/624227 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move create_tap_dev into privsep. https://review.openstack.org/624228 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move set_vf_interface_vlan to be with its only caller. https://review.openstack.org/624229 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move DHCP releasing to privsep. https://review.openstack.org/624230 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Convert additional IP management calls to privsep. https://review.openstack.org/624231 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move route management to privsep. https://review.openstack.org/624232 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move additional IP address management to privsep. https://review.openstack.org/624591 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move slight bonkers IP management to privsep. https://review.openstack.org/624592 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move arping to privsep. https://review.openstack.org/624593 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move cleaning conntrack to privsep. https://review.openstack.org/624594 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move killing processes to privsep. https://review.openstack.org/635432 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move FP device creation and deletion to privsep. https://review.openstack.org/635433 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Cleanup the exec_ebtables code a little. https://review.openstack.org/635434 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435 | 00:36 |
openstackgerrit | Michael Still proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436 | 00:36 |
mriedem | http://paste.openstack.org/show/744987/ | 00:37 |
* dansmith golf claps | 00:38 | |
dansmith | also, eye mowt | 00:38 |
sean-k-mooney | mriedem: so that is effectivly returning the instance ids which we no because they are in the instance mapping table in the api db but everything else is the blank/the default value since the cell is down | 00:41 |
*** _alastor_ has joined #openstack-nova | 00:41 | |
mriedem | yes | 00:42 |
*** mlavalle has quit IRC | 00:43 | |
*** _alastor_ has quit IRC | 00:45 | |
*** sdake has joined #openstack-nova | 00:47 | |
*** macza has quit IRC | 00:49 | |
*** openstackgerrit has quit IRC | 00:52 | |
*** bhagyashris has joined #openstack-nova | 00:55 | |
mriedem | dansmith: ok, tried to capture the startup "hang" https://bugs.launchpad.net/nova/+bug/1815697 | 00:56 |
openstack | Launchpad bug 1815697 in OpenStack Compute (nova) "[upgrade_levels]compute=auto grinds the API response times when a cell is down" [Medium,Confirmed] | 00:56 |
mriedem | by my calculation by default each API worker will take ~52 minutes to start... | 00:56 |
* mriedem goes to stop neglecting his family | 00:57 | |
*** mriedem is now known as mriedem_away | 00:57 | |
*** mriedem_away has quit IRC | 01:00 | |
*** markvoelker has joined #openstack-nova | 01:03 | |
*** gyee has quit IRC | 01:07 | |
*** yankcrime has quit IRC | 01:08 | |
*** gyee has joined #openstack-nova | 01:10 | |
*** sdake has quit IRC | 01:14 | |
*** sdake has joined #openstack-nova | 01:17 | |
*** yedongcan has joined #openstack-nova | 01:26 | |
*** sapd1 has joined #openstack-nova | 01:26 | |
*** stakeda has joined #openstack-nova | 01:27 | |
*** sdake has quit IRC | 01:29 | |
*** krasmussen has quit IRC | 01:30 | |
*** sdake has joined #openstack-nova | 01:32 | |
*** Dinesh_Bhor has joined #openstack-nova | 01:32 | |
*** markvoelker has quit IRC | 01:35 | |
*** hamzy has joined #openstack-nova | 01:39 | |
*** sdake has quit IRC | 01:39 | |
*** gyee has quit IRC | 01:47 | |
*** yankcrime has joined #openstack-nova | 02:01 | |
*** _fragatina has quit IRC | 02:02 | |
*** tetsuro has quit IRC | 02:05 | |
*** wolverineav has quit IRC | 02:18 | |
*** wolverineav has joined #openstack-nova | 02:19 | |
*** hongbin has joined #openstack-nova | 02:21 | |
*** wolverineav has quit IRC | 02:24 | |
*** macza has joined #openstack-nova | 02:35 | |
*** macza has quit IRC | 02:39 | |
*** Dinesh_Bhor has quit IRC | 02:44 | |
*** Dinesh_Bhor has joined #openstack-nova | 02:48 | |
*** takashin has joined #openstack-nova | 02:59 | |
*** sdake has joined #openstack-nova | 03:05 | |
*** wolverineav has joined #openstack-nova | 03:25 | |
*** zhanglong has quit IRC | 03:50 | |
*** spsurya has joined #openstack-nova | 03:54 | |
*** udesale has joined #openstack-nova | 03:55 | |
*** sdake has quit IRC | 04:15 | |
*** lbragstad has quit IRC | 04:32 | |
*** janki has joined #openstack-nova | 04:32 | |
*** macza has joined #openstack-nova | 04:43 | |
*** macza has quit IRC | 04:48 | |
*** hongbin has quit IRC | 04:55 | |
*** tetsuro has joined #openstack-nova | 04:59 | |
*** ratailor has joined #openstack-nova | 05:16 | |
*** penick has quit IRC | 05:24 | |
*** udesale has quit IRC | 05:36 | |
*** udesale has joined #openstack-nova | 05:42 | |
*** udesale has quit IRC | 05:48 | |
*** udesale has joined #openstack-nova | 05:48 | |
*** psachin has joined #openstack-nova | 05:49 | |
*** brinzhang has joined #openstack-nova | 05:50 | |
*** markvoelker has joined #openstack-nova | 06:00 | |
*** markvoelker has quit IRC | 06:04 | |
*** tetsuro has quit IRC | 06:09 | |
*** imacdonn has quit IRC | 06:13 | |
*** sridharg has joined #openstack-nova | 06:19 | |
*** sdake has joined #openstack-nova | 06:28 | |
*** Dinesh_Bhor has quit IRC | 06:30 | |
*** openstackgerrit has joined #openstack-nova | 06:34 | |
openstackgerrit | Merged openstack/nova master: Move binding ips to privsep. https://review.openstack.org/622151 | 06:34 |
*** Dinesh_Bhor has joined #openstack-nova | 06:39 | |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix resetting non-persistent fields when saving obj https://review.openstack.org/636271 | 06:41 |
*** Luzi has joined #openstack-nova | 06:43 | |
*** wolverineav has quit IRC | 06:45 | |
*** wolverineav has joined #openstack-nova | 06:46 | |
*** moshele has joined #openstack-nova | 06:47 | |
*** NewBruce has joined #openstack-nova | 06:49 | |
*** wolverineav has quit IRC | 06:51 | |
*** markvoelker has joined #openstack-nova | 07:00 | |
*** ileixe has joined #openstack-nova | 07:07 | |
*** sdake has quit IRC | 07:07 | |
*** NewBruce has quit IRC | 07:09 | |
*** udesale has quit IRC | 07:12 | |
*** wolverineav has joined #openstack-nova | 07:18 | |
*** belmoreira has joined #openstack-nova | 07:19 | |
*** udesale has joined #openstack-nova | 07:19 | |
*** wolverineav has quit IRC | 07:23 | |
*** udesale has quit IRC | 07:24 | |
*** udesale has joined #openstack-nova | 07:25 | |
*** takashin has left #openstack-nova | 07:29 | |
*** NewBruce has joined #openstack-nova | 07:30 | |
*** markvoelker has quit IRC | 07:34 | |
*** maciejjozefczyk has joined #openstack-nova | 07:38 | |
*** ccamacho has joined #openstack-nova | 07:48 | |
*** slaweq has joined #openstack-nova | 07:49 | |
*** rpittau has joined #openstack-nova | 07:57 | |
*** NewBruce has quit IRC | 08:04 | |
*** ralonsoh has joined #openstack-nova | 08:05 | |
*** yan0s has joined #openstack-nova | 08:07 | |
*** tetsuro has joined #openstack-nova | 08:09 | |
*** macza has joined #openstack-nova | 08:14 | |
*** NewBruce has joined #openstack-nova | 08:15 | |
*** tesseract has joined #openstack-nova | 08:16 | |
*** macza has quit IRC | 08:19 | |
*** markvoelker has joined #openstack-nova | 08:31 | |
*** helenafm has joined #openstack-nova | 08:31 | |
*** wolverineav has joined #openstack-nova | 08:31 | |
*** wolverineav has quit IRC | 08:35 | |
*** yedongcan has left #openstack-nova | 08:36 | |
*** tkajinam has quit IRC | 08:36 | |
*** NewBruce has quit IRC | 08:40 | |
*** NewBruce has joined #openstack-nova | 08:41 | |
*** NewBruce has quit IRC | 08:42 | |
*** NewBruce has joined #openstack-nova | 08:45 | |
*** NewBruce has quit IRC | 08:45 | |
*** NewBruce has joined #openstack-nova | 08:47 | |
*** dannins has joined #openstack-nova | 08:50 | |
*** NewBruce has quit IRC | 08:51 | |
*** xek has joined #openstack-nova | 08:54 | |
*** NewBruce has joined #openstack-nova | 08:55 | |
*** tetsuro has quit IRC | 08:57 | |
*** NewBruce3 has joined #openstack-nova | 08:58 | |
kashyap | Huzzah, all unit tests pass: https://review.openstack.org/#/c/632507/ | 09:00 |
*** tssurya has joined #openstack-nova | 09:00 | |
*** NewBruce3 has left #openstack-nova | 09:00 | |
openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP Restore connection_info after live migration rollback https://review.openstack.org/551349 | 09:00 |
kashyap | gibi: Morning, when you have a minute, would really appreciate a couple of eyes on this: https://review.openstack.org/#/c/632507/ (libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Stein") | 09:01 |
kashyap | stephenfin: ^ | 09:01 |
*** NewBruce has quit IRC | 09:01 | |
gibi | kashyap: looking | 09:01 |
*** NewBruce has joined #openstack-nova | 09:02 | |
kashyap | The commit message explains it (hopefully). And getting this in reduces expensive rebase churn. | 09:02 |
kashyap | I'll be here to answer any questions. | 09:03 |
*** markvoelker has quit IRC | 09:04 | |
kashyap | The code that is being removed in driver.py is what the original author (markus_z) asked us to do in his "NOTE" | 09:05 |
kashyap | And FWIW, I double-checked with the libvirt upstream developers to ensure what I'm removing makes sense. | 09:06 |
bauzas | gibi: FWIW, I had to bail out yesterday early | 09:06 |
bauzas | gibi: so I'll upload my series now with a big -W | 09:06 |
bauzas | so we could discuss with other folks later | 09:07 |
*** ociuhandu has quit IRC | 09:07 | |
gibi | bauzas: ack, I will look at it | 09:07 |
*** liumk2233 has joined #openstack-nova | 09:08 | |
gibi | kashyap: is there a connection between the min version bump and the removed virtlogd code? | 09:08 |
kashyap | gibi: Yes, it is unavoidable (as noted in the commit message) in this case | 09:08 |
kashyap | Due to the version constant check | 09:08 |
*** liumk_ has joined #openstack-nova | 09:09 | |
gibi | kashyap: so the min version now bigger than what the virtlogd needs | 09:10 |
openstackgerrit | Lee Yarwood proposed openstack/nova master: Avoid redundant initialize_connection on source post live migration https://review.openstack.org/551302 | 09:10 |
openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP Restore connection_info after live migration rollback https://review.openstack.org/551349 | 09:10 |
gibi | kashyap: so we can remove the check | 09:10 |
kashyap | Yes, indeed. | 09:10 |
kashyap | gibi: Oh, just noticed, I actually didn't remove two constant files themselves :-): | 09:12 |
kashyap | # Use the "logd" backend for handling stdout/stderr from QEMU processes. | 09:12 |
* kashyap goes to remove that | 09:12 | |
kashyap | MIN_LIBVIRT_VIRTLOGD = (1, 3, 3) | 09:12 |
kashyap | MIN_QEMU_VIRTLOGD = (2, 7, 0) | 09:12 |
gibi | kashyap: OK, then what I miss is the reason of the removal of self._create_file_device from _create_consoles_qemu_kvm() | 09:12 |
gibi | and the reason of the change in _create_consoles_s390x() | 09:12 |
*** sunnaichuan has joined #openstack-nova | 09:14 | |
*** liumk2233 has quit IRC | 09:14 | |
kashyap | gibi: First is that: we're adding two serial devices — one of them bypasses Nova | 09:14 |
kashyap | And Nova should not allow an extra serial console (so that one can use external tools like `virsh` to monitor the guest serial console) | 09:15 |
*** jangutter_ has joined #openstack-nova | 09:15 | |
*** jangutter_ has quit IRC | 09:15 | |
*** jangutter_ has joined #openstack-nova | 09:16 | |
kashyap | I'll the reasoning in the commit message (and for the _create_consoles_s390x() change) | 09:17 |
gibi | hm as far as I understand _create_file_device only makes sure that there is a console.log file | 09:17 |
gibi | ahh I see now | 09:17 |
gibi | that code is actually noop with virtlogd | 09:17 |
*** jangutter has quit IRC | 09:18 | |
gibi | then I got the reason of the change in _create_consoles_qemu_kvm() | 09:18 |
kashyap | Exactly | 09:18 |
kashyap | If 'virtlogd' is available, the _create_file_device() just returns. | 09:18 |
gibi | kashyap: correct | 09:18 |
kashyap | So, we just elided the whole function, and the places where it is called | 09:19 |
kashyap | Including the one in s390x | 09:19 |
kashyap | (_create_file_device() was called in two places, and both the two occurrences were removed) | 09:19 |
gibi | but in the s390x you replaced the removed _create_file_device call with a _create_pty_device call | 09:20 |
*** ociuhandu has joined #openstack-nova | 09:20 | |
gibi | is that _create_pty_device for sclplm needed for our always-use-virtlogd case? | 09:21 |
kashyap | That I can use some s390x eyes :-( | 09:21 |
kashyap | Let me check with a libvirt dev on that | 09:22 |
gibi | kashyap: thanks for taking care of it | 09:22 |
kashyap | gibi: Thanks for the careful review | 09:23 |
*** ociuhandu has quit IRC | 09:25 | |
*** avolkov has joined #openstack-nova | 09:28 | |
*** derekh has joined #openstack-nova | 09:29 | |
*** mma has joined #openstack-nova | 09:29 | |
openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Add method to allow reset fields for root bdm in BDM obj https://review.openstack.org/614672 | 09:30 |
kashyap | gibi: So, the "sclplm" is required, here is the background from the Nova spec that added support for s390x: | 09:30 |
kashyap | [quote] | 09:30 |
kashyap | For the interactive console and the log of OpenStack instances, console devices need to be generated (instead of serial devices), if the host architecture is System z (see [2]). These console devices need to have a target type "sclp" for the interactive console of OpenStack, and "sclplm" for the log of OpenStack. | 09:31 |
kashyap | [/quote] | 09:31 |
*** macza has joined #openstack-nova | 09:31 | |
*** ociuhandu has joined #openstack-nova | 09:34 | |
* kashyap is still digging: https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/libvirt-kvm-systemz.html | 09:35 | |
*** liumk_ has quit IRC | 09:35 | |
*** macza has quit IRC | 09:35 | |
*** ociuhandu has quit IRC | 09:38 | |
gibi | kashyap: do we have some kind of CI for s390x? | 09:46 |
*** mrch_ has joined #openstack-nova | 09:46 | |
kashyap | gibi: I see "IBM zKVM CI" | 09:46 |
kashyap | And when I click on its log, I get a "Not Found" | 09:47 |
gibi | it voted for PS2 and PS3 of your patch | 09:48 |
gibi | but nothing newer | 09:48 |
gibi | and yeah, I also cannot get to the logs | 09:48 |
kashyap | Hmm | 09:48 |
kashyap | gibi: I'm first asking an s390x developer about the behaviour ... because it is all black box for me | 09:49 |
gibi | me too | 09:51 |
*** cdent has joined #openstack-nova | 09:56 | |
Shilpa | coreycb: hi | 09:56 |
*** adrianreza has joined #openstack-nova | 10:00 | |
*** bhagyashris has quit IRC | 10:00 | |
*** markvoelker has joined #openstack-nova | 10:02 | |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Record requester in the InstancePCIRequest https://review.openstack.org/625310 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add pf_interface_name tag to passthrough_whitelist https://review.openstack.org/625311 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ensure that bandwidth and VF are from the same PF https://review.openstack.org/623543 | 10:05 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support server create with ports having resource request https://review.openstack.org/636360 | 10:05 |
*** ociuhandu has joined #openstack-nova | 10:12 | |
openstackgerrit | Silvan Kaiser proposed openstack/nova master: Added mount fstype based validation of Quobyte mounts https://review.openstack.org/522245 | 10:12 |
kashyap | gibi: One thing the s390x maintainer in QEMU confirmed is that: 'pty' for "sclplm" device will work. | 10:13 |
kashyap | But I want to get one more view from DanPB, who wrote the 'virtlogd' daemon. | 10:13 |
*** ralonsoh has quit IRC | 10:16 | |
gibi | kashyap: cool | 10:18 |
gibi | kashyap: you also have a nit in the commit message, about the new min Qemu version which should be 2.8.0 not 2.5.0 | 10:18 |
gibi | besides these I'm OK with your patch | 10:18 |
kashyap | Ah, thank you | 10:18 |
kashyap | gibi: Yeah, getting this will allow me to clean up all the now needless version constants | 10:19 |
kashyap | And getting that unit tests fixed took 8 days :/ | 10:19 |
gibi | yeah, I can imagine the pain | 10:19 |
*** wolverineav has joined #openstack-nova | 10:20 | |
openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Reject resize requests when the source host is down https://review.openstack.org/623489 | 10:20 |
stephenfin | kashyap: Sorry, forgot to start HexChat. I thought mriedem had suggested we do that min version bump early in Train instead? | 10:20 |
kashyap | stephenfin: He was ambivalent on it. | 10:20 |
*** jangutter_ has quit IRC | 10:20 | |
kashyap | stephenfin: IMHO, don't see any compelling blockers (/me hopes this statement doesn't come back to bite him) | 10:21 |
stephenfin | kashyap: Cool. I'll hold of +2ing personally until I double check, though I'd be happy to get it in of course | 10:22 |
kashyap | Re-working / rebasing the unit tests again for Train becomes a lot more more painful | 10:22 |
kashyap | stephenfin: I actually asked him (MattR) last night here, but his nick was _afk | 10:22 |
*** jangutter has joined #openstack-nova | 10:23 | |
kaisers | stephenfin: Thanks, i updated the CL. If you're rolling I'd also be happy to have somebody revisit https://review.openstack.org/#/c/554195/ | 10:24 |
kashyap | stephenfin: gibi: Release-wise, we're just about to reach M3 (Mar 04-08), and the GA is aroud 12-Apr — we still have decent time for this to soak in | 10:24 |
openstackgerrit | Silvan Kaiser proposed openstack/nova master: Exec systemd-run without --user flag in Quobyte driver https://review.openstack.org/554195 | 10:26 |
*** erlon has joined #openstack-nova | 10:31 | |
openstackgerrit | Silvan Kaiser proposed openstack/nova master: Exec systemd-run without --user flag in Quobyte driver https://review.openstack.org/554195 | 10:31 |
*** ralonsoh has joined #openstack-nova | 10:32 | |
*** markvoelker has quit IRC | 10:34 | |
*** jangutter has quit IRC | 10:38 | |
*** jangutter has joined #openstack-nova | 10:41 | |
stephenfin | kashyap: Comments left. Just the commit message fix needed, fwict | 10:45 |
*** ralonsoh has quit IRC | 10:45 | |
kashyap | stephenfin: Most excellent. | 10:45 |
*** ralonsoh has joined #openstack-nova | 10:45 | |
kashyap | gibi: stephenfin: Meanwhile, I double-checked with DanPB, and what I'm doing is "sane". Here's the conversation snippet: | 10:45 |
kashyap | < kashyap> danpb: Is changing the behaviour in this case, by using the console type 'pty' instead of 'file' for s390x's "sclplm" okay? | 10:46 |
kashyap | < kashyap> (FWIW, checking with Corneila of s390x, she said using 'pty' type for "sclplm" device will work.) | 10:46 |
kashyap | < kashyap> If you're wondering WTH is this "sclplm", s390x has two types of console devices: | 10:46 |
kashyap | < danpb> the guest doesnt' care what's done in the backend | 10:46 |
kashyap | < kashyap> (1) "sclp" -- for interactive consoles | 10:46 |
kashyap | < kashyap> (2) "sclplm" -- for log files | 10:46 |
kashyap | < kashyap> danpb: Ah, okay, then. | 10:46 |
kashyap | < danpb> personally i'd just get rid of that duplication | 10:46 |
kashyap | < kashyap> Okay, I'll make that in a separate logical change | 10:46 |
kashyap | stephenfin: Thanks for the quick review! | 10:46 |
kashyap | stephenfin: Related to your comment, look up "negative code" here :-) -- https://en.wikipedia.org/wiki/Douglas_McIlroy#Views_on_computing | 10:47 |
stephenfin | kashyap: Very true, but there's no reason removing code can't be done in multiple steps to help grok'ability. See https://review.openstack.org/#/c/636384/ and the preceding patch | 10:49 |
kashyap | stephenfin: Oh, absolutely: I try to be iron-clad on the "one logical change per commit" | 10:50 |
*** jangutter has quit IRC | 10:52 | |
*** wolverineav has quit IRC | 10:52 | |
*** jangutter has joined #openstack-nova | 10:55 | |
kashyap | stephenfin: Nice work there. | 10:57 |
openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208 | 10:59 |
openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add functional test for libvirt vgpu reshape https://review.openstack.org/631559 | 10:59 |
openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Use the correct mdev allocated from the pGPU https://review.openstack.org/636591 | 10:59 |
bauzas | gibi: ^ | 10:59 |
bauzas | gibi: I thought about the fact that we blindly use mdevs without checking the pGPU | 10:59 |
bauzas | gibi: and I think it's not a problem | 10:59 |
bauzas | at least until we support multiple types | 11:00 |
bauzas | since libvirt will be able to find a specific mdev | 11:00 |
bauzas | that said, I'll take my old internal hardware and do a lot of checks | 11:00 |
stephenfin | kaisers: More comments on https://review.openstack.org/#/c/522245/ after a second reading | 11:01 |
*** ociuhandu has quit IRC | 11:02 | |
*** ociuhandu has joined #openstack-nova | 11:03 | |
openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Stein" https://review.openstack.org/632507 | 11:04 |
kashyap | gibi: stephenfin: When you can, addressed both your remarks ^ | 11:04 |
*** ralonsoh has quit IRC | 11:05 | |
*** jangutter has quit IRC | 11:05 | |
*** jangutter has joined #openstack-nova | 11:06 | |
kaisers | stephenfin: thnx, checkin' in a min | 11:06 |
*** udesale has quit IRC | 11:13 | |
*** brinzhang has quit IRC | 11:14 | |
stephenfin | kaisers: And on https://review.openstack.org/#/c/554195/2 | 11:14 |
*** ileixe has quit IRC | 11:20 | |
*** ratailor has quit IRC | 11:20 | |
*** Dinesh_Bhor has quit IRC | 11:21 | |
kaisers | stephenfin: yup, thnx again! | 11:28 |
*** markvoelker has joined #openstack-nova | 11:30 | |
*** ccamacho has quit IRC | 11:39 | |
*** macza has joined #openstack-nova | 11:39 | |
*** janki has quit IRC | 11:40 | |
*** janki has joined #openstack-nova | 11:40 | |
*** macza has quit IRC | 11:44 | |
*** tbachman has quit IRC | 11:44 | |
*** sapd1 has quit IRC | 11:45 | |
*** wolverineav has joined #openstack-nova | 11:49 | |
*** rpittau has quit IRC | 11:52 | |
*** hamzy has quit IRC | 11:59 | |
*** panda is now known as panda|lunch | 12:00 | |
*** markvoelker has quit IRC | 12:04 | |
*** sdake has joined #openstack-nova | 12:04 | |
*** ccamacho has joined #openstack-nova | 12:09 | |
*** _fragatina has joined #openstack-nova | 12:11 | |
*** _fragatina_ has joined #openstack-nova | 12:12 | |
*** hamzy has joined #openstack-nova | 12:13 | |
*** _fragatina has quit IRC | 12:16 | |
*** sdake has quit IRC | 12:16 | |
*** wolverineav has quit IRC | 12:23 | |
*** rpittau has joined #openstack-nova | 12:29 | |
*** Cardoe has quit IRC | 12:45 | |
*** udesale has joined #openstack-nova | 12:50 | |
*** janki has quit IRC | 13:00 | |
openstackgerrit | Surya Seetharaman proposed openstack/nova master: Plumbing required in servers ViewBuilder to construct partial results https://review.openstack.org/635146 | 13:00 |
openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.69: Handles Down Cells https://review.openstack.org/591657 | 13:00 |
openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.69: Handles Down Cells Documentation https://review.openstack.org/635147 | 13:00 |
*** markvoelker has joined #openstack-nova | 13:01 | |
*** tbachman has joined #openstack-nova | 13:06 | |
*** mriedem has joined #openstack-nova | 13:08 | |
*** sapd1 has joined #openstack-nova | 13:08 | |
*** panda|lunch is now known as panda | 13:12 | |
openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.69: Handles Down Cells Documentation https://review.openstack.org/635147 | 13:13 |
openstackgerrit | Jim Rollenhagen proposed openstack/nova master: ironic: partition compute services by conductor group https://review.openstack.org/635006 | 13:15 |
openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Ensure config regexes match the entire string https://review.openstack.org/636627 | 13:18 |
*** wolverineav has joined #openstack-nova | 13:20 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Change nova-next tempest test regex https://review.openstack.org/636459 | 13:22 |
*** dannins has quit IRC | 13:27 | |
*** janki has joined #openstack-nova | 13:28 | |
Shilpa | stephenfin: hi, under Taskflow, added https://bugs.launchpad.net/taskflow/+bug/1815738, do let me know is this active channel? Unable to trace PTL for this, but https://pypi.org/project/taskflow/ reports latest release in Jan'19 | 13:28 |
openstack | Launchpad bug 1815738 in taskflow "Taskflow doesn't support to retrieve tasks progress details in the order of workflow task execution" [Undecided,New] | 13:28 |
mriedem | Shilpa: i think that would be in the #openstack-oslo channel now | 13:30 |
mriedem | or i guess #openstack-state-management https://wiki.openstack.org/wiki/TaskFlow#Contact_us.21 | 13:31 |
mriedem | tssurya: did you enjoy my tales from going down cell down street? | 13:31 |
Shilpa | mriedem: #openstack-state-management here no one responding, will check #openstack-oslo channel | 13:31 |
mriedem | tssurya: belmoreira: i'm interested - what do you configure [upgrade_levels]/compute to in nova.conf? do you pin to a release or have it set to auto? | 13:32 |
bauzas | mriedem: FWIW, I uploaded the new rev for reshape | 13:32 |
bauzas | mriedem: I found a potential blocker, but eventually it's not a problem | 13:32 |
mriedem | ok cool | 13:32 |
belmoreira | mriedem: in the past used to pin it to a release. Now we use auto | 13:33 |
*** Dinesh_Bhor has joined #openstack-nova | 13:34 | |
*** markvoelker has quit IRC | 13:34 | |
openstackgerrit | Lee Yarwood proposed openstack/nova master: Avoid redundant initialize_connection on source post live migration https://review.openstack.org/551302 | 13:36 |
*** eharney has quit IRC | 13:37 | |
mriedem | belmoreira: ok if you have this change from stable/rocky https://review.openstack.org/#/c/624982/ then you might want to be careful, | 13:37 |
mriedem | i came across something nasty last night in down cell testing | 13:37 |
mriedem | https://bugs.launchpad.net/nova/+bug/1815697 | 13:38 |
openstack | Launchpad bug 1815697 in OpenStack Compute (nova) "[upgrade_levels]compute=auto grinds the API response times when a cell is down" [Medium,Confirmed] | 13:38 |
mriedem | at this point i don't have any great solutions for that bug | 13:38 |
sean-k-mooney | mriedem: as a workaournd you set the retry interval to 1 and retries to 1 and set the exact version | 13:40 |
sean-k-mooney | that could lead to more api request failing in other cases but it should not be inherintly dangourous right | 13:40 |
mriedem | sean-k-mooney: no, the max_retries and retry_interval was just to quiet the logs | 13:40 |
sean-k-mooney | oh ok | 13:40 |
mriedem | the workaround for hanging the api on startpu was setting [upgrade_levels]/compute=rocky | 13:40 |
*** lbragstad has joined #openstack-nova | 13:41 | |
tssurya | mriedem: I did go through your tales :D I absolutely appreciate the testing :) I am adding more test cases for the parts you felt are lacking | 13:42 |
belmoreira | mriedem: we don't have that commit, yet. Thanks for the heads up. | 13:42 |
tssurya | and I am not so sure I get your worry here: https://review.openstack.org/#/c/591657/39/nova/api/openstack/compute/servers.py@135 | 13:42 |
Shilpa | mriedem: no one replies here too #openstack-oslo, will wait for more time | 13:43 |
*** Dinesh_Bhor has quit IRC | 13:44 | |
mriedem | Shilpa: ok well nova doesn't use taskflow. harlowja was the primary maintainer but i'm not sure if he's around anymore. oslo team would be your best bet i'd think, otherwise maybe #openstack-dev | 13:45 |
mriedem | efried loves taskflow though | 13:45 |
Shilpa | mriedem: ok, hope he will help me there | 13:46 |
mriedem | tssurya: my concern is if tooling is passing filter parameters by default, | 13:47 |
mriedem | normally in the API we filter out search options that the user isn't allowed to use without it being an error | 13:47 |
mriedem | so let's say i'm a non-admin and i pass some stuff which normally just gets filtered out in the remove_invalid_options method | 13:48 |
mriedem | essentially making search_opts = {} | 13:48 |
tssurya | like "deleted" and "tenant" ? | 13:48 |
mriedem | if i'm using 2.69, then empty search_opts would mean down_cell_support=True | 13:48 |
mriedem | tssurya: non-admins can't use the tenant filter | 13:48 |
tssurya | yea I know | 13:49 |
mriedem | servers are filtered by default from the project_id in the request context | 13:49 |
tssurya | which is why I was asking if that's what you mean by invalid options | 13:49 |
mriedem | so let's say i'm a non-admin and i'm using --deleted, here are some scenarios | 13:50 |
mriedem | 1. nova list --deleted with v2.1 - the deleted filter would be excluded w/o any errors in the api and i should list my servers | 13:50 |
mriedem | 2. nova list --deleted with v2.69 and list_records_by_skipping_down_cells=True (default), the down_cell_support flag would be set to False and i wouldn't get any results (if my servers are in a down cell) | 13:50 |
mriedem | 3. nova list --deleted with v2.69 and list_records_by_skipping_down_cells=False, if my servers are in a down cell i'll get a 500 | 13:51 |
mriedem | my point was if we calculated the down_cell_support flag *after* calling remove_invalid_options to filter the search_opts, then #3 becomes #2 | 13:51 |
mriedem | actually no that's not right, | 13:52 |
mriedem | i'd get partial results | 13:52 |
openstackgerrit | Lee Yarwood proposed openstack/nova master: Restore connection_info after live migration rollback https://review.openstack.org/551349 | 13:53 |
mriedem | to summarize, i think the down_cell_support variable should be set based on the search opts after we've filtered that set in remove_invalid_options to increase the chances of someone getting partial results | 13:53 |
lyarwood | mdbooth: ^ btw, feel free to respin the commit message if you have time. | 13:53 |
*** wolverineav has quit IRC | 13:53 | |
tssurya | mriedem: ok, I get what you mean | 13:53 |
*** jmlowe has quit IRC | 13:54 | |
mriedem | the tricky thing is if you do that, you have to explicitly look for paging and sorting parameters | 13:54 |
mriedem | because of https://review.openstack.org/#/c/591657/39/nova/api/openstack/compute/servers.py@1235 | 13:54 |
tssurya | yea | 13:54 |
mriedem | but i dont think that's so hard, we could just create a constant for ('sort_key', 'sort_dir', 'limit', 'marker') and use that in both places | 13:54 |
mriedem | and your _is_cell_down_supported method can check, "if any(PAGING_SORTING_PARAMS in reg.GET) | 13:55 |
tssurya | got it, let me dig more and incorporate that | 13:57 |
mriedem | err, if set(PAGING_SORTING_PARAMS) & set(list(req.GET.keys())) | 13:57 |
mriedem | something like that | 13:57 |
mriedem | does that make sense though? | 13:57 |
tssurya | it makes sense yea | 13:57 |
openstackgerrit | Hamdy Khader proposed openstack/os-vif master: Add create_ovs_port field in VIFPortProfileOpenVSwitch profile https://review.openstack.org/636061 | 13:57 |
tssurya | but I still don't get why the policy things are mixed with ignoring flags | 13:58 |
mriedem | cool. my goal is to maximize the chance that someone gets partial results if they have servers in a down cell | 13:58 |
mriedem | which policy things? | 13:58 |
tssurya | I mean if --deleted is an only admin option | 13:58 |
tssurya | it shuld barf | 13:58 |
tssurya | should* | 13:58 |
tssurya | saying you are not admin | 13:58 |
mriedem | legacy backward compat | 13:58 |
tssurya | just lke tenant or user | 13:58 |
tssurya | ahhh ok ok | 13:59 |
mriedem | same reason you can do GET /servers?foo=bar | 13:59 |
mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/servers.py#L606 | 13:59 |
openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Microversion 2.68: Remove 'forced' live migrations, evacuations https://review.openstack.org/635131 | 13:59 |
mriedem | the query parameter schema is whitelisted but also allows users to pass anything for backward compat, | 13:59 |
mriedem | except the stuff in https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/servers.py#L512 | 14:00 |
mriedem | those are 404ed here https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py#L125 | 14:00 |
tssurya | mriedem: wow its crazy, but I understand at least why it was done, thanks for patiently explaining that | 14:01 |
tssurya | I'll work on ironing out the wrinkles | 14:01 |
mriedem | cool. otherwise from my manual testing (once i worked around those other bugs with the service version checking) it was working pretty well | 14:02 |
stephenfin | mriedem: Not sure you already discussed this again with kashyap, but any serious objections to doing the libvirt min version bump now? https://review.openstack.org/#/c/632507/ gibi and I are happy but I recall some objections previously | 14:02 |
kashyap | stephenfin: Heh, was _just_ about to write a message here. | 14:03 |
kashyap | stephenfin: Not quite objections, mriedem was not sure, based on some discussion w/ an operator he had | 14:03 |
kashyap | Ideally, I should have done this at the _start_ of the cycle, but completely lost track of it | 14:03 |
mriedem | looking, but you should probably ask the PTL | 14:04 |
adrianc | sean-k-mooney: Hi, regarding sriov live-migration patches, id like to address some of the comments in https://review.openstack.org/#/c/620115/21, however i saw you are planning to upload a new PS for the series, will it be soon ? as id like to avoid merge conflicts etc... | 14:04 |
kashyap | mriedem: Yeah, will do. I thought melwitt was still catching up after being on PTO (?) | 14:04 |
mriedem | stephenfin: kashyap: the concern from an operator was they were upgrading from mitaka to i think queens and the minimum bump in between was just for maintenance, not related to any functional changes, and the computes they were upgrading didn't have that minimum libvirt (they were using older centos i think) | 14:05 |
mriedem | so this operator was asking me if there was a reason for the bump, or if they could just revert the change to keep the older centos compute node but still upgrade nova | 14:06 |
sean-k-mooney | adrianc: i got pulled into some other stuff so go ahead. | 14:06 |
kashyap | I see. Yeah, probably we should better advertize _why_ we do bumps and at what intervals? | 14:06 |
kashyap | mriedem: ^ Like me writing a reminder note to the mailing list about why we do, and give a gentle heads-up. | 14:06 |
mriedem | i just think in years past we weren't real aggressive about doing it every release | 14:06 |
sean-k-mooney | adrianc: ill work on adressing the base patches but ill mainly be adressing comment/commit messages so there whould be no conflicts | 14:07 |
kashyap | mriedem: Yes, I now kept a reminder on my phone to do the start of the cycle | 14:07 |
adrianc | sean-k-mooney: would you like me to address some of the nits in https://review.openstack.org/#/c/624842/ as well, or ill leave them to you ? | 14:07 |
*** sdake has joined #openstack-nova | 14:08 | |
mriedem | so looking at https://review.openstack.org/#/c/558171/ the last time we did a bump was pike, is that correct? | 14:08 |
sean-k-mooney | adrianc: am if you want to adress them i can focous on the os-vif brctl removal which is the other thing im context switch with | 14:09 |
mriedem | we advertized the next minimums in rocky | 14:09 |
*** sdake has quit IRC | 14:09 | |
mriedem | so really this would be the first required minimum bump since pike | 14:09 |
kashyap | mriedem: Yes, I did the bump last April-May (https://review.openstack.org/#/q/topic:bump_min_libvirt_and_qemu+(status:open+OR+status:abandoned+OR+status:merged) | 14:09 |
mriedem | giving at least a year to prepare | 14:09 |
kashyap | mriedem: Indeed | 14:09 |
mriedem | (knowing people skip several years of openstack releases now) | 14:09 |
sean-k-mooney | adrianc: i dont think there is anything controvertiol in any of the coments on the seirs so if you have time to make the requested change go for it | 14:10 |
mriedem | looks like we don't have zkvm 3rd party CI anymore... | 14:10 |
kashyap | Release-wise, we're just about to reach M3 (Mar 04-08), and the GA is aroud 12-Apr — I feel we're still okay | 14:10 |
*** gokhani has joined #openstack-nova | 14:10 | |
*** sdake has joined #openstack-nova | 14:11 | |
adrianc | sean-k-mooney: Ack, ill start from the base patch then :) | 14:11 |
kashyap | mriedem: Yeah, was wondering this morning about it | 14:11 |
mriedem | https://wiki.openstack.org/wiki/ThirdPartySystems/IBM_zKVM_CI | 14:11 |
kashyap | Today I spent 4 hours digging into s390x and their fun serial console stuff | 14:11 |
openstackgerrit | Alexandra Settle proposed openstack/nova master: Adding cross refs for config options in scheduler filter guide https://review.openstack.org/636635 | 14:11 |
mriedem | kashyap: so how about you send a reminder to the mailing list and also copy zkvm-ci@linux.vnet.ibm.com and bjzhjing@linux.vnet.ibm.com | 14:11 |
kashyap | MarkusZ left a note in the code with a pointer to a bug ... and it turned out it's a bug _I_ filed 4 years ago :P | 14:11 |
mriedem | markus_z is long gone from openstack | 14:12 |
kashyap | mriedem: I guess you mean now, right? | 14:12 |
mriedem | kashyap: yeah, i'd say do that now and if we don't hear any strong objections by end of week we go ahead | 14:12 |
kashyap | mriedem: I know, I just found it sadly funny that when I decided to learn more from the bug ... apparently I filed it. | 14:12 |
kashyap | mriedem: Alright, let me draft it right away. | 14:12 |
kashyap | mriedem: I spent a painful 8 days on-and-off arriving at the root cause. But hey, fun :-) | 14:13 |
*** sdake has quit IRC | 14:13 | |
kashyap | Thanks for your time. Appreciate it | 14:13 |
mriedem | you've grown as a person i'm sure | 14:13 |
kashyap | I did; I grew in "(positive) complexity" :D | 14:14 |
*** sdake has joined #openstack-nova | 14:15 | |
kashyap | What tags would I use? [ops][nova] suffices I guess? | 14:15 |
kashyap | s/suffices/suffice/ | 14:15 |
mriedem | yeah | 14:16 |
kashyap | mriedem++ | 14:17 |
kashyap | Hm, we should get a 'karma bot' in | 14:17 |
*** eharney has joined #openstack-nova | 14:22 | |
*** Dinesh_Bhor has joined #openstack-nova | 14:22 | |
*** sdake has quit IRC | 14:23 | |
*** dave-mccowan has joined #openstack-nova | 14:25 | |
*** dave-mccowan has quit IRC | 14:30 | |
*** markvoelker has joined #openstack-nova | 14:31 | |
efried | Shilpa: It looks to me like you're not specifying an engine for your task flow. I thought the default was a linear one, but it's possible it's not. Do you get the same results if you specify a linear engine explicitly? | 14:32 |
Shilpa | efried: tried with engine='serial' but no sucess | 14:32 |
Shilpa | i will check setting engine='linear' and get back to you | 14:33 |
efried | Shilpa: I don't remember exactly what the engine names are, would have to consult the docs. | 14:33 |
Shilpa | ok | 14:34 |
*** awaugama has joined #openstack-nova | 14:35 | |
efried | Shilpa: Beyond that, I'm probably not a whole lot of use to you. Josh Harlow (harlowja) is the guy who could help, but I don't see him in the usual haunts; possible he's moved on to greener pastures. | 14:35 |
efried | You may want to try posting your bug to the taskflow github repo | 14:35 |
Shilpa | efried: its serial but no sucess | 14:35 |
efried | Shilpa: Okay. | 14:36 |
Shilpa | efried: at taskflow github repo, in REDME posted Bugs: https://bugs.launchpad.net/taskflow/ | 14:36 |
*** lpetrut has joined #openstack-nova | 14:38 | |
bauzas | holy shit, I'm deploying devstack and placement gives me a 503 when being started | 14:38 |
bauzas | wtf | 14:38 |
sean-k-mooney | bauzas: its likely the same issue i had | 14:39 |
sean-k-mooney | you have old apache config files | 14:39 |
bauzas | efried: cdent: any thoughts on that ? | 14:39 |
bauzas | hah | 14:39 |
bauzas | i asked to reclone | 14:39 |
sean-k-mooney | it wont fix it | 14:39 |
bauzas | yeah | 14:39 |
bauzas | but then we're missing something in lib/placemenbt | 14:39 |
bauzas | because a clean.sh didn't fix it too | 14:40 |
openstackgerrit | Alexandra Settle proposed openstack/nova master: Adding cross refs for config options in scheduler filter guide https://review.openstack.org/636635 | 14:40 |
sean-k-mooney | nuke your /etc/http/site-available and sites-enabled | 14:40 |
sean-k-mooney | bauzas: ya i was ment to wite a change to devstack to fix it | 14:40 |
sean-k-mooney | its on my todo list | 14:40 |
efried | cool | 14:40 |
bauzas | ok | 14:40 |
bauzas | gtk | 14:40 |
sean-k-mooney | hopefully ill get to it by next week | 14:40 |
cdent | it's a generic problem with how apache stuff is config'd not just placement | 14:40 |
cdent | there are additional issues when you switch back and forth between using uwsgi and not | 14:41 |
sean-k-mooney | ya i hit the same issue with cinder i think | 14:41 |
openstackgerrit | Alexandra Settle proposed openstack/nova master: Adding cross refs for config options in scheduler filter guide https://review.openstack.org/636635 | 14:41 |
bauzas | sean-k-mooney: wait, I'm running devstack with uwsgi | 14:41 |
sean-k-mooney | yes | 14:41 |
bauzas | I don't have httpd site | 14:41 |
efried | cdent: is it the kind of thing it would be appropriate to put a hack into devstack to clean up / make sure it dtrt? | 14:41 |
sean-k-mooney | apache is still being used as a reverse proxy | 14:41 |
*** _fragatina_ has quit IRC | 14:42 | |
cdent | efried: sean-k-mooney plans to figure it out | 14:42 |
efried | woot | 14:42 |
bauzas | sean-k-mooney: wait, I don't know what to bitch | 14:42 |
sean-k-mooney | if your using ubnuntu its under /etc/apache2/... | 14:42 |
bauzas | my httpd directory is sane | 14:42 |
bauzas | it's RHEL, so /etc/httpd but whatever | 14:42 |
sean-k-mooney | bauzas: for me it was poining at an old socket name nova-placement.socket | 14:43 |
bauzas | ah, interesting | 14:43 |
sean-k-mooney | you could see the error in the apache errror log in /var/httpd/error.log | 14:43 |
bauzas | in conf.d, I have two files | 14:43 |
bauzas | nova-placement-api.conf and placement-api.conf | 14:44 |
sean-k-mooney | bauzas: the simple solution is to run clean unistall httpd and delete /etc/httpd then stack | 14:44 |
sean-k-mooney | it will fix everything | 14:44 |
sean-k-mooney | but really you just need to remove the placemetn config file | 14:44 |
bauzas | I guess | 14:45 |
bauzas | lemme try | 14:45 |
sean-k-mooney | i have done ^ 3 times in the last week for this issue on different systems | 14:46 |
sean-k-mooney | but ill look at actully just makeing devstack fix it properly next week | 14:46 |
bauzas | cdent: edleafe: efried: somehow, we changed from nova-placement-api.conf to placement-api.conf without thinking of folks who upgrade | 14:46 |
bauzas | :) | 14:46 |
bauzas | words of wisdom, people do upgrade :) | 14:46 |
sean-k-mooney | bauzas: yep but its also devstack soo.... | 14:47 |
cdent | of devstack? | 14:47 |
cdent | i thought the three letters on the front was a clue? | 14:47 |
sean-k-mooney | i assume grenade is just nuking the configs and replaceing them with new ones | 14:47 |
sean-k-mooney | cdent: it still shows up on the openstack survay in prod ocationally | 14:48 |
sean-k-mooney | cdent: honestly with the ansible rules we use in the gate for deploying it and the swap to systemd for all the things its actully a lot less dumb then it used to be | 14:48 |
sean-k-mooney | cdent: it now support host reboots again | 14:49 |
bauzas | cdent: you don't imagine how much devstack is insightful for deployers :) | 14:49 |
bauzas | but anyway, not a big deal | 14:49 |
cdent | I'm not disputing that devstack should clean up apache configs better. Just trying to point out that this is an old and not related to placement problem and not something that is super urgent. it's just a bug that needs to be fixed | 14:49 |
cdent | anyone can fix it | 14:49 |
sean-k-mooney | fixed and then backported as this only happens if you swap branches | 14:50 |
*** wolverineav has joined #openstack-nova | 14:50 | |
jroll | I'm surprised clean.sh doesn't remove apache configs, seems like an easy fix | 14:50 |
sean-k-mooney | which developer do a lot but ci systems and normal peole do not | 14:50 |
cdent | jroll: I really think it simply comes down to "nobody has got around to it yet or been burned enough to bother". sean-k-mooney is nearly there. | 14:51 |
sean-k-mooney | jroll: it also does not remove apache. i mean that is the super simple fix but not makeing the config generate condtional woudl also have worked | 14:51 |
cdent | sounds like bauzas is already there but doesn't have time | 14:51 |
*** sapd1 has quit IRC | 14:52 | |
sean-k-mooney | im takeing 2 days off this week and need to get other stuff done before i do which is the only reason i havent done it yet. | 14:53 |
*** sridharg has quit IRC | 14:54 | |
*** liuyulong has joined #openstack-nova | 14:55 | |
kashyap | As promised earlier: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002665.html -- [ops][nova] Heads-up: Upcoming version bump for libvirt and QEMU in 'Stein' | 14:58 |
*** penick has joined #openstack-nova | 14:59 | |
openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Stein" https://review.openstack.org/632507 | 14:59 |
openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Drop MIN_LIBVIRT_PARALLELS_SET_ADMIN_PASSWD https://review.openstack.org/632514 | 14:59 |
*** jmlowe has joined #openstack-nova | 14:59 | |
kashyap | gibi: stephenfin: Just a commit message update on the main patch ^ | 15:00 |
openstackgerrit | Alexandra Settle proposed openstack/nova master: Adding cross refs for config options in scheduler filter guide https://review.openstack.org/636635 | 15:02 |
*** markvoelker has quit IRC | 15:04 | |
bauzas | gibi: so, I uploaded my change | 15:06 |
bauzas | gibi: but now the func test you wrote is failing | 15:06 |
bauzas | weird, heh ? | 15:06 |
gibi | bauzas: sorry, I haven't looked it yet | 15:06 |
bauzas | no worries, just saying | 15:07 |
*** sapd1 has joined #openstack-nova | 15:08 | |
bauzas | gibi: I don't get why but the root RP has a VGPU inventory, hence the mismatch | 15:08 |
*** moshele has quit IRC | 15:09 | |
bauzas | gibi: http://logs.openstack.org/59/631559/5/check/nova-tox-functional/100cfb6/job-output.txt.gz#_2019-02-13_11_19_37_217415 | 15:09 |
bauzas | gibi: any reason why you don't provide the VGPU inventory first ? | 15:09 |
*** Dinesh_Bhor has quit IRC | 15:16 | |
*** jmlowe has quit IRC | 15:18 | |
*** mrch_ has quit IRC | 15:19 | |
openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Drop MIN_LIBVIRT_PARALLELS_SET_ADMIN_PASSWD https://review.openstack.org/632514 | 15:19 |
*** janki has quit IRC | 15:21 | |
*** mlavalle has joined #openstack-nova | 15:21 | |
*** _alastor_ has joined #openstack-nova | 15:22 | |
*** jmlowe has joined #openstack-nova | 15:23 | |
*** wolverineav has quit IRC | 15:23 | |
*** _alastor_ has quit IRC | 15:26 | |
*** ralonsoh has joined #openstack-nova | 15:27 | |
gibi | bauzas: sorry I was dragged away from the keyboard | 15:39 |
bauzas | gibi: heh, we all are | 15:39 |
bauzas | or, rather, always dragged from upstream | 15:39 |
gibi | bauzas: if https://review.openstack.org/#/c/631559/5/nova/tests/functional/libvirt/test_reshape.py@130 fails then it means that the reshape does not moved the VGPU resource away from the compute RP to the GPU RP | 15:42 |
gibi | bauzas: https://review.openstack.org/#/c/631559/5/nova/tests/functional/libvirt/test_reshape.py@123 should trigger the reshape | 15:43 |
bauzas | I'll dig into pdbing it | 15:43 |
bauzas | in the meantime, lemme mark my series as -2 | 15:43 |
*** maciejjozefczyk has quit IRC | 15:44 | |
openstackgerrit | Alexandra Settle proposed openstack/nova master: Adding cross refs for config options in scheduler filter guide https://review.openstack.org/636635 | 15:54 |
*** jmlowe has quit IRC | 15:54 | |
*** ociuhandu has quit IRC | 15:55 | |
*** Luzi has quit IRC | 15:55 | |
*** ociuhandu has joined #openstack-nova | 15:55 | |
*** jmlowe has joined #openstack-nova | 15:58 | |
gibi | bauzas: http://logs.openstack.org/59/631559/5/check/nova-tox-functional/100cfb6/job-output.txt.gz#_2019-02-13_11_19_37_215302 here is when the compute is restarted and should do the reshape | 15:59 |
*** tzumainn has joined #openstack-nova | 15:59 | |
*** lpetrut has quit IRC | 16:00 | |
*** markvoelker has joined #openstack-nova | 16:01 | |
openstackgerrit | Jack Ding proposed openstack/nova master: Flavor extra spec and image properties validation https://review.openstack.org/620706 | 16:02 |
*** sdake has joined #openstack-nova | 16:06 | |
*** macza has joined #openstack-nova | 16:11 | |
*** _alastor_ has joined #openstack-nova | 16:16 | |
*** samc-bbc has joined #openstack-nova | 16:17 | |
*** wolverineav has joined #openstack-nova | 16:20 | |
*** mrch_ has joined #openstack-nova | 16:22 | |
melwitt | o/ | 16:24 |
*** jangutter has quit IRC | 16:26 | |
*** markvoelker has quit IRC | 16:34 | |
*** Shilpa has quit IRC | 16:35 | |
*** jmlowe has quit IRC | 16:38 | |
*** jmlowe has joined #openstack-nova | 16:44 | |
*** sdake has quit IRC | 16:46 | |
*** sdake has joined #openstack-nova | 16:47 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Use math.gcd starting with python 3.5 https://review.openstack.org/636669 | 16:50 |
melwitt | mriedem: there's a patch proposed to add basic info about projects to the openstack-map, if you're interested in reviewing the nova info https://review.openstack.org/636604 | 16:51 |
*** wolverineav has quit IRC | 16:53 | |
*** gyee has joined #openstack-nova | 16:55 | |
*** helenafm has quit IRC | 16:57 | |
*** sdake has quit IRC | 16:58 | |
*** _fragatina has joined #openstack-nova | 16:59 | |
*** _fragatina has quit IRC | 17:00 | |
*** wolverineav has joined #openstack-nova | 17:01 | |
openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: Handle unicode characters in migration params https://review.openstack.org/636678 | 17:01 |
*** itlinux has joined #openstack-nova | 17:05 | |
*** tssurya has quit IRC | 17:06 | |
mriedem | melwitt: commented | 17:09 |
mriedem | dansmith: replied on https://review.openstack.org/#/c/469675/ | 17:09 |
*** itlinux has quit IRC | 17:09 | |
mriedem | dansmith: for sanity, i think we likely just want to stop using check_availability_zone in both the api and compute | 17:09 |
mriedem | and write an api-specific version of that | 17:09 |
*** itlinux_ has joined #openstack-nova | 17:09 | |
mriedem | from the api, gather all the precreated volumes and their azs, | 17:10 |
mriedem | if any are different, it's an error | 17:10 |
mriedem | if any are different from the requested server az, it's an error | 17:10 |
mriedem | if the server is created without a specific az, and the volumes all have the same az, then put that on the request spec | 17:10 |
*** ociuhandu_ has joined #openstack-nova | 17:11 | |
dansmith | mriedem: yeah, collect the data and make the call in the compute api | 17:12 |
dansmith | mriedem: like I said, if the "user requested an az and it doesn't match the volume" case still needs to be handled deep for cleanup reasons, then okay, | 17:12 |
dansmith | but (a) default_az means something totally different to me when reading the compute/api code and (b) I don't expect either of these mismatch checks to happen so deep down in that file | 17:13 |
dansmith | so if you don't want to refactor this at all, I'd at least ask that you change the name of that parameter with commentage up in compute/api aboutit | 17:14 |
*** itlinux_ has quit IRC | 17:14 | |
*** itlinux has joined #openstack-nova | 17:14 | |
mriedem | "the "user requested an az and it doesn't match the volume" case" is what would be in nova-compute after a volume is created | 17:15 |
mriedem | so yeah i think that remains | 17:15 |
dansmith | wait | 17:16 |
dansmith | oh right, yeah | 17:16 |
*** sapd1 has quit IRC | 17:16 | |
mriedem | which reminds me of https://bugs.launchpad.net/nova/+bug/1497253 | 17:17 |
openstack | Launchpad bug 1497253 in OpenStack Compute (nova) "different availability zone for nova and cinder when AZ is not explicitly given" [Low,Fix released] - Assigned to Dan Smith (danms) | 17:17 |
*** psachin has quit IRC | 17:18 | |
openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove get_config_vhostuser https://review.openstack.org/565471 | 17:19 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: Validate bandwidth configuration for other VIF types https://review.openstack.org/636383 | 17:19 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: Further de-dupe os-vif VIF tests https://review.openstack.org/636384 | 17:19 |
*** wolverineav has quit IRC | 17:20 | |
stephenfin | gibi: Trivial doc fix here, if you fancy taking a look https://review.openstack.org/#/c/636635/ | 17:21 |
*** udesale has quit IRC | 17:23 | |
*** jmlowe has quit IRC | 17:23 | |
*** rpittau has quit IRC | 17:24 | |
mriedem | dansmith: right so i'm thinking leaving the nova-compute behavior the same, and probably move that check_availability_zone method to nova.virt.block_device where it's actually used | 17:26 |
mriedem | and writing something new in the API | 17:26 |
dansmith | makes sense | 17:26 |
mriedem | that has the user-requested (or not) AZ context | 17:26 |
mriedem | having said that, it's pot-pie-o-clock and i've got a physical this afternoon (first in 10+ years) so my attendance will be spotty | 17:26 |
mriedem | oh and (don't tell laura) i have to get a valentine | 17:27 |
dansmith | mriedem: you might be approaching that age where the doctor starts wanting to take your relationship to the next level | 17:27 |
mriedem | i'm holding out for my 40s | 17:27 |
dansmith | heh | 17:28 |
*** sapd1 has joined #openstack-nova | 17:29 | |
*** itlinux has quit IRC | 17:30 | |
*** _fragatina has joined #openstack-nova | 17:31 | |
*** markvoelker has joined #openstack-nova | 17:32 | |
tzumainn | hi! quick question about the expected behavior with nova and ironic; I have four baremetal nodes, and I've created one server that runs on one of the nodes, but if I run "nova hypervisor-servers" against each node, that server shows up for every single node | 17:34 |
*** itlinux has joined #openstack-nova | 17:36 | |
*** dtantsur is now known as dtantsur|afk | 17:37 | |
*** sapd1 has quit IRC | 17:37 | |
*** ociuhand_ has joined #openstack-nova | 17:39 | |
*** itlinux_ has joined #openstack-nova | 17:39 | |
melwitt | tzumainn: are you running one nova-compute service for all four baremetal nodes? if so, I think that might be why it shows up for every node | 17:40 |
melwitt | I'd have to look into the code to find how the hostname => server lookup is done to confirm | 17:41 |
*** itlinux has quit IRC | 17:41 | |
melwitt | jroll ^ | 17:41 |
tzumainn | melwitt, ah, yeah, I am | 17:41 |
jroll | I'm not sure what "nova hypervisor-servers" is | 17:42 |
jroll | but yeah, probably only goes by compute service hostname, not node name | 17:42 |
*** ociuhandu has quit IRC | 17:42 | |
melwitt | it's a command to get a list of servers given a hostname | 17:43 |
bauzas | dansmith: stop me 1 sec, do we now need to set the host mappings when deploying with devstack ? | 17:46 |
dansmith | bauzas: devstack should do that for you | 17:46 |
jroll | melwitt: ah, sounds like a bug in that API response | 17:47 |
bauzas | dansmith: that was my assumption but I got a Host 'XXX' is not mapped to any cell | 17:47 |
* bauzas goes into looking n-api | 17:47 | |
dansmith | bauzas: on a vanilla devstack setup? shouldn't happen afaik | 17:47 |
*** ociuhand_ has quit IRC | 17:48 | |
bauzas | technically, a reinstalled devstack but with a ./clean.sh and a reclone=true | 17:48 |
melwitt | jroll: I get what you mean now, might be returning host instead of node | 17:48 |
bauzas | weirdo | 17:48 |
jroll | melwitt: yeah | 17:49 |
melwitt | thanks | 17:49 |
bauzas | dansmith: nevermind, looks like a rabbit issue | 17:50 |
mriedem | bauzas: i'm pretty sure clean.sh isn't maintained | 17:50 |
mriedem | i never use it | 17:50 |
bauzas | mriedem: okay, how do we clean the data then ? | 17:51 |
bauzas | rm -rf ? | 17:51 |
melwitt | tzumainn: ok, there might be a bug there. I can do some digging later to confirm and check if we have any open bugs or patches about it already | 17:51 |
mriedem | bauzas: if i need a clean slate i create a new devstack vm | 17:51 |
bauzas | lucky you :p | 17:51 |
mriedem | lemme guess, you're running devstack on baremetal with gpus | 17:52 |
bauzas | here I'm talking of some internal server that I got by accident | 17:52 |
bauzas | mriedem: yeah, I want to test the reshape series on real hardware | 17:52 |
bauzas | -where we have RHEL7.5... - | 17:53 |
bauzas | but anyway, will continue to look | 17:53 |
tzumainn | melwitt, thanks! I didn't see an existing bug, but looking at https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L167-L168 maybe what I'm seeing is intended behavior, and it's matching based on the hypervisor host_ip (which is the same for all four baremetal nodes)? | 17:53 |
mriedem | tzumainn: it's a fuzzy search on hypervisor hostname https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L680 | 17:55 |
mriedem | for ironic nodes that should actually be a uuid | 17:55 |
tzumainn | mriedem, yep, that's what I'm using | 17:55 |
*** derekh has quit IRC | 17:55 | |
mriedem | and the API code is taking that compute node record and getting all instances on that compute_nodes.host | 17:56 |
mriedem | which is the same for all of your nodes | 17:56 |
mriedem | compute_nodes.host == nova-compute service hostname == instances.host | 17:57 |
mriedem | so it's working as expected, i.e. that API is not ironic-aware | 17:57 |
mriedem | nor is most of the compute API | 17:57 |
mriedem | hence MOGAN | 17:57 |
melwitt | tzumainn: yeah, internally in nova we have hypervisor "hostname" which is associated with the nova-compute service and we have "nodename" which is the ironic nodename. if you're not using ironic hostname == nodename. else, it will be different and to get the right answer (better answer?) for ironic, we'd need to do a lookup based on nodename, not "hostname" | 17:57 |
openstackgerrit | Adrian Chiris proposed openstack/nova master: Libvirt: do not set MAC when unplugging macvtap VF https://review.openstack.org/624842 | 17:57 |
openstackgerrit | Adrian Chiris proposed openstack/nova master: Add free for claimed, allocated devices https://review.openstack.org/616120 | 17:57 |
openstackgerrit | Adrian Chiris proposed openstack/nova master: Allow per-port modification of vnic_type and profile https://review.openstack.org/607365 | 17:57 |
openstackgerrit | Adrian Chiris proposed openstack/nova master: Add get_instance_pci_request_from_vif https://review.openstack.org/619929 | 17:57 |
openstackgerrit | Adrian Chiris proposed openstack/nova master: SR-IOV Live migration indirect port support https://review.openstack.org/620115 | 17:57 |
openstackgerrit | Adrian Chiris proposed openstack/nova master: libvirt: auto detach/attach sriov ports on migration https://review.openstack.org/629589 | 17:57 |
*** awalende has joined #openstack-nova | 17:58 | |
tzumainn | mriedem, melwitt gotcha - okay, thanks! | 17:59 |
mriedem | tzumainn: without an api change the best we can do right now is document the wrinkle in the api reference | 18:00 |
tzumainn | mriedem, would an API change be possible in the longer-term, or would you consider this a tragic wrinkle in how nova/ironic interact? | 18:01 |
mriedem | tzumainn: i think it could be a possible change on a new microversion - if the resulting compute node we found is of type ironic, we have to lookup instances by nodename, not host | 18:01 |
melwitt | ++ | 18:02 |
mriedem | tzumainn: feel free to file a bug so it could at least be documented and then a blueprint, if someone wants to work on that, could be written based on the bug | 18:02 |
mriedem | jaypipes might like that | 18:02 |
tzumainn | mriedem, I'll do that - thanks very much! | 18:02 |
mriedem | yw | 18:02 |
melwitt | would also be good for a newer contributor bc it seems like it should be pretty small change | 18:03 |
openstackgerrit | Jim Rollenhagen proposed openstack/nova master: ironic: check fresh data when sync_power_state doesn't line up https://review.openstack.org/636699 | 18:03 |
mriedem | until they have to write functional api samples tests | 18:03 |
mriedem | and their head explodes | 18:03 |
melwitt | ah, I didn't think of that part | 18:04 |
*** markvoelker has quit IRC | 18:04 | |
*** erlon has quit IRC | 18:05 | |
* cdent lights the gabbi bat signal | 18:06 | |
cdent | but that's not why I'm here, I have an actual question | 18:07 |
cdent | it seems that most of the volume related methods in compute manager are synchronized on instance uuid, but detach volume is not. Is there a reason it is not? | 18:07 |
*** jmlowe has joined #openstack-nova | 18:08 | |
*** itlinux_ has quit IRC | 18:08 | |
sean-k-mooney | instnace uuid or volume uuid | 18:09 |
*** agopi has quit IRC | 18:09 | |
sean-k-mooney | if its instance could there be issues with multiattach volumes? | 18:10 |
sean-k-mooney | cdent: also i dont know | 18:10 |
openstackgerrit | Jim Rollenhagen proposed openstack/nova master: ironic: check fresh data when sync_power_state doesn't line up https://review.openstack.org/636699 | 18:10 |
cdent | sean-k-mooney: yeah, I was confused by the choice of lock id | 18:11 |
cdent | (too) | 18:11 |
sean-k-mooney | cdent: i have not looked at the code but is your requestion related to the use of the uuid for synconisation or is detach ungaurded and your asking should it be | 18:14 |
*** hongbin has joined #openstack-nova | 18:15 | |
cdent | sean-k-mooney: I'm asking if it should be. In k8s-on-openstack environment there are issues happening with volume attachements getting into a weird state | 18:15 |
openstackgerrit | garyk proposed openstack/nova master: Lock detach_volume https://review.openstack.org/636701 | 18:15 |
cdent | that's ^ the proposed fix | 18:15 |
sean-k-mooney | cdent: lyarwood and mdbooth are usually who i ask when it comes to volumn/storage things | 18:16 |
*** moshele has joined #openstack-nova | 18:17 | |
cdent | yeah, me too, but I'm hoping they've already stepped awy, keeping reasonable hours for this timezone, unlike you and me | 18:17 |
lyarwood | I'm around, just | 18:17 |
lyarwood | we already lock within the libvirt driver, I assume this is for VMWare? | 18:18 |
cdent | lyarwood: in the compute manager itself | 18:18 |
*** erlon has joined #openstack-nova | 18:18 | |
cdent | not the virtdrivers | 18:18 |
cdent | see the patch | 18:18 |
lyarwood | cdent: right, and I assume the issues you're seeing are with the underlying detach/attach on the virt drivers? | 18:18 |
lyarwood | cdent: or is it with the attachments themselves in c-api? | 18:19 |
*** awalende has quit IRC | 18:19 | |
*** sdake has joined #openstack-nova | 18:19 | |
cdent | lyarwood: I'm short on details. At the moment I'm just feeling around trying to gather info. I wasn't present at the discovery of the issue. | 18:19 |
lyarwood | kk np | 18:19 |
* lyarwood reads the bug | 18:19 | |
openstackgerrit | garyk proposed openstack/nova master: Lock detach_volume https://review.openstack.org/636701 | 18:20 |
*** awalende has joined #openstack-nova | 18:21 | |
cdent | lyarwood: as far as I can tell whatever the client is (some k8s thing) is going async on the volume delete and instance delete | 18:22 |
openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: mark os-cells as deprecated https://review.openstack.org/636708 | 18:22 |
mriedem | cdent: i seem to remember mdbooth having a change to lock detach_volume on instance.uuid | 18:23 |
lyarwood | cdent: yeah and as vmware isn't using os-brick you don't get any of the locking within it, adding a lock in the compute layer for detach seems reasonable tbh. | 18:23 |
*** wolverineav has joined #openstack-nova | 18:23 | |
cdent | mriedem: you mean in progress? | 18:23 |
* cdent looks | 18:23 | |
mriedem | might be abandoned by now | 18:23 |
mriedem | https://review.openstack.org/#/c/430213/ | 18:24 |
mriedem | bingo | 18:24 |
*** betherly has joined #openstack-nova | 18:25 | |
sean-k-mooney | lyarwood: well as cdent pointed out we already lock in the compute manager for attach | 18:25 |
*** erlon_ has joined #openstack-nova | 18:25 | |
mriedem | cdent: there you go - want me to restore and you can rebase? | 18:25 |
lyarwood | sean-k-mooney: right, I'm agreeing | 18:25 |
sean-k-mooney | lyarwood: yep i know. | 18:26 |
* mriedem notes he remembered a patch from Feb 7, 2017 | 18:26 | |
sean-k-mooney | mriedem: is there any reason to restor it vs https://review.openstack.org/#/c/636701/2 | 18:26 |
sean-k-mooney | the two are identical | 18:26 |
mriedem | how about because it came first? | 18:26 |
*** sapd1 has joined #openstack-nova | 18:27 | |
cdent | thanks mriedem I looked, but not that far back | 18:27 |
*** hongbin has quit IRC | 18:27 | |
*** wolverineav has quit IRC | 18:27 | |
sean-k-mooney | ok i gues garyk could take over rebaseing the old patch | 18:27 |
*** hongbin has joined #openstack-nova | 18:27 | |
* cdent worries about mriedem's brain | 18:27 | |
*** erlon has quit IRC | 18:28 | |
cdent | since it is in merge conflict, how about we just booth as a co-author on the clean one, and add the older bug too? | 18:29 |
*** ociuhandu has joined #openstack-nova | 18:29 | |
mriedem | that works too... | 18:29 |
*** betherly has quit IRC | 18:29 | |
mriedem | i've duplicated the bug already | 18:29 |
*** wolverineav has joined #openstack-nova | 18:29 | |
mriedem | sec | 18:29 |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Lock detach_volume https://review.openstack.org/636701 | 18:30 |
mriedem | comment inline | 18:32 |
cdent | damn dood, I think I'm just gonna let you do everything ;) | 18:32 |
mriedem | i'm not sure we need to hold the lock while doing the db query | 18:32 |
cdent | yeah | 18:32 |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Lock detach_volume https://review.openstack.org/636701 | 18:33 |
*** ociuhandu has quit IRC | 18:33 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Lock detach_volume https://review.openstack.org/636701 | 18:34 |
mriedem | there god | 18:34 |
mriedem | coding in gerrit ui | 18:34 |
*** moshele has quit IRC | 18:34 | |
cdent | Is there any chance that get_by_volume_and_instance itself can race? | 18:35 |
openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: Handle unicode characters in migration params https://review.openstack.org/636678 | 18:35 |
*** awalende has quit IRC | 18:35 | |
mriedem | worst case is you'd get a not found error from the db query | 18:37 |
*** sdake has quit IRC | 18:37 | |
*** sunnaichuan has quit IRC | 18:38 | |
*** awalende has joined #openstack-nova | 18:38 | |
*** avolkov has quit IRC | 18:42 | |
*** betherly has joined #openstack-nova | 18:47 | |
*** tssurya has joined #openstack-nova | 18:49 | |
*** agopi has joined #openstack-nova | 18:50 | |
*** mriedem is now known as mriedem_away | 18:52 | |
openstackgerrit | Merged openstack/nova master: Change nova-next job to run with python3 https://review.openstack.org/634739 | 18:53 |
openstackgerrit | Merged openstack/nova master: Adding cross refs for config options in scheduler filter guide https://review.openstack.org/636635 | 18:53 |
*** tbachman has quit IRC | 18:58 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove cells v1 (for the most part) from the docs https://review.openstack.org/636718 | 19:01 |
*** markvoelker has joined #openstack-nova | 19:01 | |
*** wolverineav has quit IRC | 19:04 | |
*** wolverineav has joined #openstack-nova | 19:08 | |
*** ociuhandu has joined #openstack-nova | 19:16 | |
*** ociuhandu has quit IRC | 19:28 | |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects to transmit NUMA config from dest to source https://review.openstack.org/634827 | 19:30 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Introduce live_migration_claim() https://review.openstack.org/635669 | 19:30 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Use live_migration_claim() to check dest resources https://review.openstack.org/634606 | 19:30 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Make dest send NUMAMigrateData to the source https://review.openstack.org/634828 | 19:30 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: update NUMA-related XML on the source https://review.openstack.org/635229 | 19:30 |
*** cdent has quit IRC | 19:31 | |
*** eharney has quit IRC | 19:31 | |
*** cdent has joined #openstack-nova | 19:32 | |
*** erlon_ has quit IRC | 19:34 | |
*** markvoelker has quit IRC | 19:34 | |
*** awalende has quit IRC | 19:39 | |
aspiers | what's the easiest way to set the logger to DEBUG level in functional tests? | 19:46 |
cdent | if you're based off the nova base class, you probably already have it, but if not self.flag('debug', 'True') might do it. I can't remember for sure | 19:49 |
dansmith | there's an environment variable | 19:49 |
aspiers | ^^^ that's the answer I was hoping for ;-) | 19:50 |
melwitt | aspiers: OS_DEBUG=1 | 19:50 |
cdent | there's a fixture that will swallow a lot of stuff too | 19:50 |
dansmith | yeah that^ | 19:50 |
aspiers | awesome, thanks! :-D | 19:50 |
melwitt | have fun | 19:50 |
aspiers | found it in nova/tests/fixtures.py now | 19:50 |
* cdent can't decide if he misunderstood the question, or what | 19:51 | |
melwitt | OS_DEBUG=1 tox -efunctional test_stuff | 19:51 |
melwitt | that will show you log level debug and less during a test run | 19:51 |
aspiers | cdent: I could maybe answer that if I understood your answer, but I fear it's too sophisticated for me ;-) | 19:52 |
aspiers | melwitt: yep the env var is working great, although I'm running stestr directly in order to run individual groups of tests quicker | 19:52 |
melwitt | smart | 19:52 |
aspiers | going through tox is slow :-/ | 19:52 |
cdent | aspiers: I think I was answering "I want to manipulate log handling in a functional test I'm making" not "I want to see the debug logs" | 19:52 |
aspiers | cdent: ah yeah, thought it might be something like that :) | 19:52 |
aspiers | sorry for not being clear enough in the original q | 19:53 |
cdent | no worries, worked out in the end | 19:53 |
aspiers | melwitt: shall I add OS_DEBUG to HACKING.rst? | 19:54 |
dansmith | aspiers: wouldn't you rather keep it secret and seem smart the next time a newbie comes along? | 19:55 |
aspiers | dansmith: tempting, but I think it's even more satisfying to be able to RTFH somebody by pointing them to docs I wrote myself ;-p | 19:56 |
dansmith | aspiers: okay as long as shame is involved either way | 19:56 |
aspiers | that way I can appear smart *and* conceited | 19:57 |
aspiers | exactly | 19:57 |
melwitt | haha, nice | 19:57 |
aspiers | after all I only went into coding in order to be able to shame others :-p | 19:57 |
melwitt | that's the right motivation | 19:58 |
aspiers | glad we're on the same page XD | 19:58 |
melwitt | and yeah, it would be a good addition to HACKING.rst. and also our contributor doc/ | 19:59 |
aspiers | melwitt: under https://docs.openstack.org/nova/latest/contributor/testing.html or a new page? | 20:00 |
aspiers | presumably https://docs.openstack.org/nova/latest/contributor/index.html#testing isn't a good place for it | 20:00 |
melwitt | lemme see, I think I saw a place | 20:01 |
melwitt | I was looking at https://docs.openstack.org/nova/latest/contributor/development-environment.html#running-unit-tests but yeah, similarly doesn't have the same level of detail that HACKING.rst does | 20:02 |
aspiers | I could just add a link from there to HACKING? | 20:03 |
melwitt | that also looks really old :\ | 20:03 |
*** wolverineav has quit IRC | 20:03 | |
aspiers | Yeah | 20:03 |
*** panda is now known as panda|off | 20:04 | |
*** _fragatina has quit IRC | 20:05 | |
melwitt | aspiers: I think a link to HACKING for nova-specific test running notes would be a good fit there on that last page. because the existing doc is a link to the general openstack-wide test running guide | 20:07 |
aspiers | melwitt: yeah OK ... but how do I link to HACKING? since it's not published with the docs :-/ | 20:08 |
aspiers | link to cgit? | 20:08 |
aspiers | http://git.openstack.org/cgit/openstack/nova/tree/HACKING.rst doesn't render | 20:09 |
aspiers | https://github.com/openstack/nova/blob/master/HACKING.rst might be nicer | 20:09 |
aspiers | but kind of breaks the unspoken "avoid GitHub" convention | 20:10 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Document how to make tests log at DEBUG level https://review.openstack.org/636744 | 20:10 |
aspiers | I guess the link to HACKING can be done separately anyway | 20:10 |
melwitt | yeah, we can punt that idea since there's not an obvious counterpart in our docs | 20:11 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Drop specific versions of openSUSE-based distributions https://review.openstack.org/636745 | 20:11 |
melwitt | I had been thinking about this I think, which is not nova docs (or even docs.o.o) https://wiki.openstack.org/wiki/Testr | 20:12 |
melwitt | from back in the day | 20:12 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Document how to make tests log at DEBUG level https://review.openstack.org/636744 | 20:12 |
aspiers | ah yeah | 20:12 |
*** ralonsoh has quit IRC | 20:18 | |
*** eharney has joined #openstack-nova | 20:25 | |
*** jmlowe has quit IRC | 20:28 | |
*** ociuhandu has joined #openstack-nova | 20:31 | |
*** markvoelker has joined #openstack-nova | 20:31 | |
*** medberry has joined #openstack-nova | 20:33 | |
*** wolverineav has joined #openstack-nova | 20:33 | |
*** wolverineav has quit IRC | 20:38 | |
*** hongbin has quit IRC | 20:40 | |
openstackgerrit | Adam Spiers proposed openstack/nova master: Document how to make tests log at DEBUG level https://review.openstack.org/636744 | 20:46 |
*** hongbin has joined #openstack-nova | 20:49 | |
*** hongbin has quit IRC | 20:54 | |
*** samc-bbc has quit IRC | 20:58 | |
aspiers | please can someone explain how the functional tests mock responses from the placement API? | 21:04 |
*** itlinux has joined #openstack-nova | 21:04 | |
*** ociuhandu has quit IRC | 21:04 | |
aspiers | I'm lost in a twisty maze of fixture code | 21:04 |
*** mriedem_away is now known as mriedem | 21:04 | |
*** wolverineav has joined #openstack-nova | 21:04 | |
mriedem | i believe the PlacementFixture is pulled in from the placement repo | 21:05 |
mriedem | via tox siblings | 21:05 |
aspiers | ahhhhh | 21:05 |
*** markvoelker has quit IRC | 21:05 | |
aspiers | that's why I couldn't find it | 21:05 |
aspiers | thanks! | 21:05 |
mriedem | https://github.com/openstack/placement/blob/master/placement/tests/functional/fixtures/placement.py#L27 | 21:05 |
*** hongbin has joined #openstack-nova | 21:06 | |
aspiers | yeah, looks like the actual fixtures are in gabbits.py | 21:07 |
cdent | aspiers: not mocks. 100% functionally real. | 21:07 |
*** xek has quit IRC | 21:07 | |
mriedem | aspiers: the gabbits are placement functional tests in placement | 21:07 |
aspiers | cdent: yeah, I see that now :) | 21:07 |
mriedem | nova uses the PlacementFixture for stubbing out the nova.scheduler.client.report.SchedulerClient calls, which is essentially nova's placement client | 21:07 |
mriedem | but yeah not really stubs | 21:07 |
mriedem | hits the actual placement api and db | 21:07 |
aspiers | mriedem: I'm trying to figure out why the HW_CPU_X86_VMX trait is missing in your traits functional test which I've rebased | 21:08 |
aspiers | the expectation that it's there is hard-coded, and I couldn't see why | 21:09 |
aspiers | actually I still can't | 21:13 |
aspiers | does it come from GranularFixture maybe? | 21:13 |
aspiers | https://review.openstack.org/#/c/538498/8/nova/tests/functional/test_servers.py@2007 | 21:14 |
aspiers | ^^^ this is what's confusing me | 21:14 |
openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Microversion 2.68: Remove 'forced' live migrations, evacuations https://review.openstack.org/635131 | 21:15 |
*** jmlowe has joined #openstack-nova | 21:15 | |
mriedem | i'm not sure what you're hitting, | 21:18 |
mriedem | but i see this in the test failure | 21:18 |
mriedem | AttributeError: <class 'nova.virt.fake.SmallFakeDriver'> does not have the attribute 'get_traits' | 21:18 |
mriedem | that method used to exist in the base class https://review.openstack.org/#/c/538498/7/nova/virt/driver.py@905 | 21:19 |
mriedem | but no longer does | 21:19 |
mriedem | HW_CPU_X86_VMX is a standard trait in the os-traits library, | 21:19 |
mriedem | the placement api puts that into the traits table on startup | 21:19 |
mriedem | dansmith: this had to be tweaked but do you want to re-approve https://review.openstack.org/#/c/551302/ | 21:21 |
dansmith | ack | 21:22 |
*** tbachman has joined #openstack-nova | 21:29 | |
*** whoami-rajat has quit IRC | 21:32 | |
*** awaugama has quit IRC | 21:35 | |
*** wolverineav has quit IRC | 21:37 | |
melwitt | mriedem: commented on https://review.openstack.org/636701, it seems to me we do need to lock around the bdm lookup, unless I'm missing something | 21:39 |
*** _fragatina has joined #openstack-nova | 21:45 | |
*** itlinux has quit IRC | 21:58 | |
*** rchurch has joined #openstack-nova | 22:00 | |
*** rchurch_ has quit IRC | 22:01 | |
*** markvoelker has joined #openstack-nova | 22:02 | |
*** wolverineav has joined #openstack-nova | 22:03 | |
*** tssurya has quit IRC | 22:07 | |
*** wolverineav has quit IRC | 22:08 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Lock detach_volume https://review.openstack.org/636701 | 22:14 |
mriedem | changed | 22:14 |
aspiers | mriedem: ignore the get_traits failure, I already fixed it but didn't push a new patchset yet | 22:17 |
aspiers | I guess I can push a new one and you'll see the exact failure | 22:18 |
openstackgerrit | Adam Spiers proposed openstack/nova master: WIP: Convert driver supported capabilities to compute node provider traits https://review.openstack.org/538498 | 22:20 |
*** openstackgerrit has quit IRC | 22:22 | |
*** mdbooth_ has joined #openstack-nova | 22:31 | |
*** eandersson has quit IRC | 22:33 | |
*** markvoelker has quit IRC | 22:34 | |
*** mdbooth has quit IRC | 22:34 | |
aspiers | mriedem: https://gist.github.com/aspiers/d64eff8a1d5594ad9763ebbd192d36b1 | 22:34 |
*** agopi_ has joined #openstack-nova | 22:35 | |
*** gouthamr has quit IRC | 22:35 | |
*** gouthamr has joined #openstack-nova | 22:36 | |
*** cburgess has quit IRC | 22:36 | |
*** agopi has quit IRC | 22:37 | |
*** cburgess has joined #openstack-nova | 22:38 | |
*** sapd1 has quit IRC | 22:40 | |
*** agopi__ has joined #openstack-nova | 22:40 | |
mriedem | yeah it's failing because you removed the get_traits mock which was returning ['CUSTOM_FOO', 'HW_CPU_X86_VMX'] | 22:42 |
aspiers | DOH | 22:42 |
mriedem | from the drier | 22:42 |
mriedem | *driver | 22:42 |
aspiers | lol | 22:42 |
*** agopi__ is now known as agopi | 22:42 | |
aspiers | missed that | 22:42 |
*** agopi_ has quit IRC | 22:42 | |
mriedem | you essentially need to stub the driver.update_provider_tree method with a fake so you can intercept the provider tree and shove your traits into it | 22:43 |
mriedem | i think | 22:43 |
aspiers | yeah sure, got it now | 22:43 |
aspiers | I was assuming it was hardcoded somewhere - was searching the fixtures high and low X-D | 22:44 |
mriedem | or just change FakeDriver.update_provider_tree to always return those traits but that could affect other tests | 22:44 |
mriedem | i'd do something like, | 22:44 |
mriedem | original_upt = self.compute.driver.update_provider_tree | 22:45 |
mriedem | def fake_upt(...): | 22:45 |
mriedem | original_upt(...) | 22:45 |
mriedem | # inject the traits i want in the tree object | 22:45 |
mriedem | return that thing | 22:45 |
mriedem | self.stub_out('nova.virt.fake.FakeDriver.update_provider_tree', fake_upt) | 22:46 |
mriedem | i think that should do it | 22:46 |
mriedem | if you need that to be temporary, you can use self.useFixture(fixtures.MonkeyPatch(old, new)) directly as a context manager | 22:47 |
aspiers | cool, thanks! | 22:48 |
*** tesseract has quit IRC | 22:49 | |
*** sapd1 has joined #openstack-nova | 22:55 | |
*** wolverineav has joined #openstack-nova | 22:59 | |
*** eandersson has joined #openstack-nova | 22:59 | |
*** david-lyle has quit IRC | 23:02 | |
*** dklyle has joined #openstack-nova | 23:02 | |
*** tkajinam has joined #openstack-nova | 23:05 | |
*** tbachman has quit IRC | 23:08 | |
*** tbachman has joined #openstack-nova | 23:09 | |
*** mrjk_ has quit IRC | 23:10 | |
*** slaweq has quit IRC | 23:11 | |
*** tbachman_ has joined #openstack-nova | 23:15 | |
*** sapd1 has quit IRC | 23:16 | |
*** tbachman has quit IRC | 23:17 | |
*** tbachman_ is now known as tbachman | 23:17 | |
*** mlavalle has quit IRC | 23:17 | |
*** wolverineav has quit IRC | 23:30 | |
*** wolverineav has joined #openstack-nova | 23:30 | |
*** markvoelker has joined #openstack-nova | 23:31 | |
*** openstackgerrit has joined #openstack-nova | 23:32 | |
openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add support for microversion 2.69 https://review.openstack.org/636779 | 23:32 |
openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add support for microversion 2.69 - expose device tags https://review.openstack.org/636779 | 23:36 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Remove _legacy_dict methods https://review.openstack.org/636210 | 23:40 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Add migration param to check_can_live_migrate_destination https://review.openstack.org/634605 | 23:40 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects to transmit NUMA config from dest to source https://review.openstack.org/634827 | 23:40 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Introduce live_migration_claim() https://review.openstack.org/635669 | 23:40 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Use live_migration_claim() to check dest resources https://review.openstack.org/634606 | 23:40 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Make dest send NUMAMigrateData to the source https://review.openstack.org/634828 | 23:40 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: update NUMA-related XML on the source https://review.openstack.org/635229 | 23:40 |
mriedem | efried: question in https://review.openstack.org/#/c/636378/ | 23:47 |
mriedem | oh gdi | 23:48 |
mriedem | now i see the follow up | 23:48 |
openstackgerrit | Merged openstack/nova master: Avoid redundant initialize_connection on source post live migration https://review.openstack.org/551302 | 23:48 |
efried | mriedem: Yeah, I considered actually referring to the fup in the commit message, but stephenfin's suggestion was intended to make it stand on its own, so... | 23:51 |
mriedem | which i guess is this https://review.openstack.org/#/c/582392/2/tox.ini@a26 | 23:52 |
*** lbragstad has quit IRC | 23:52 | |
mriedem | now to see how many people complain about unit tests not working tomorrow | 23:53 |
efried | heh | 23:54 |
efried | I did happen to notice that bionic doesn't have tox 3.x, I had to install it by hand. | 23:54 |
mriedem | i'm still on xenial | 23:55 |
mriedem | i get tox from pypi anyway | 23:55 |
*** agopi is now known as agopi|out | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!