Tuesday, 2019-01-29

*** imacdonn has quit IRC00:00
*** imacdonn has joined #openstack-nova00:00
*** _fragatina has joined #openstack-nova00:01
*** wolverineav has joined #openstack-nova00:02
*** itlinux has joined #openstack-nova00:10
*** BjoernT_ has quit IRC00:10
*** tetsuro has joined #openstack-nova00:15
openstackgerritEric Fried proposed openstack/nova master: FUPs: ReportClient traffic series  https://review.openstack.org/63360900:16
*** macza has quit IRC00:20
*** sdake has quit IRC00:23
*** sdake has joined #openstack-nova00:24
*** hongbin has joined #openstack-nova00:27
*** wolverineav has quit IRC00:34
*** wolverineav has joined #openstack-nova00:35
*** edmondsw has quit IRC00:38
*** edleafe- has joined #openstack-nova00:39
*** edleafe has quit IRC00:39
*** edleafe- is now known as edleafe00:39
*** wolverineav has quit IRC00:39
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to spec obj.  https://review.openstack.org/63124300:50
*** gyee has quit IRC00:51
*** wolverineav has joined #openstack-nova00:52
*** wolverineav has quit IRC00:56
*** markvoelker has joined #openstack-nova01:10
*** macza has joined #openstack-nova01:10
*** tiendc has joined #openstack-nova01:13
*** markvoelker has quit IRC01:15
*** macza has quit IRC01:15
*** hongbin has quit IRC01:18
*** hongbin has joined #openstack-nova01:20
*** _fragatina has quit IRC01:27
*** sdake has quit IRC01:28
*** _fragatina has joined #openstack-nova01:28
*** Dinesh_Bhor has joined #openstack-nova01:32
*** _fragatina has quit IRC01:32
*** tosky has quit IRC01:34
*** lbragstad has quit IRC01:35
*** wolverineav has joined #openstack-nova01:37
*** artom has joined #openstack-nova01:37
*** _alastor_ has quit IRC01:38
*** TxGirlGeek has quit IRC01:40
*** yedongcan has joined #openstack-nova02:29
*** sdake has joined #openstack-nova02:31
*** sdake has quit IRC02:37
*** whoami-rajat has joined #openstack-nova02:37
*** sdake has joined #openstack-nova02:38
*** ileixe has joined #openstack-nova02:39
*** _alastor_ has joined #openstack-nova02:39
*** cfriesen has quit IRC02:42
*** _alastor_ has quit IRC02:44
*** khomesh is now known as khomesh|brb02:50
*** psachin has joined #openstack-nova02:52
*** sdake has quit IRC02:58
*** sdake has joined #openstack-nova02:59
*** khomesh|brb is now known as khomesh|bfst03:04
*** _fragatina has joined #openstack-nova03:12
*** _fragatina has quit IRC03:13
*** _fragatina has joined #openstack-nova03:14
*** sdake has quit IRC03:26
*** sdake has joined #openstack-nova03:29
*** abhishek has joined #openstack-nova03:38
*** alex_xu has quit IRC03:40
*** sdake has quit IRC03:42
*** Dinesh_Bhor has quit IRC03:43
*** khomesh|bfst is now known as khomesh03:44
*** alex_xu has joined #openstack-nova03:46
*** lbragstad has joined #openstack-nova03:47
*** Dinesh_Bhor has joined #openstack-nova03:50
*** udesale has joined #openstack-nova04:02
*** macza has joined #openstack-nova04:05
*** igordc has quit IRC04:06
*** macza_ has joined #openstack-nova04:07
*** _fragatina has quit IRC04:08
*** macza has quit IRC04:09
*** macza_ has quit IRC04:11
*** macza has joined #openstack-nova04:21
*** nehaalhat_ has joined #openstack-nova04:23
openstackgerritSundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access  https://review.openstack.org/63124204:24
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to spec obj.  https://review.openstack.org/63124304:24
*** macza has quit IRC04:25
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to spec obj.  https://review.openstack.org/63124304:27
*** itlinux has quit IRC04:41
openstackgerritHesam Chobanlou proposed openstack/nova master: docs: ComputeDriver.update_provider_tree in nova  https://review.openstack.org/63337504:42
*** sdake has joined #openstack-nova04:50
*** spsurya has joined #openstack-nova05:05
*** _alastor_ has joined #openstack-nova05:12
*** _alastor_ has quit IRC05:17
*** ratailor has joined #openstack-nova05:23
*** udesale has quit IRC05:29
*** abhishek has quit IRC05:32
*** nehaalhat_ has quit IRC05:38
*** sdake has quit IRC05:41
*** sdake has joined #openstack-nova05:50
*** udesale has joined #openstack-nova05:51
*** udesale has quit IRC05:59
*** sridharg has joined #openstack-nova05:59
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove deprecated 'os-flavor-manage' policy  https://review.openstack.org/63365606:00
*** udesale has joined #openstack-nova06:00
*** khomesh is now known as khomesh|brb06:04
*** bhagyashris has joined #openstack-nova06:08
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix string interpolations in logging calls  https://review.openstack.org/63365706:10
*** edmondsw has joined #openstack-nova06:19
*** markvoelker has joined #openstack-nova06:20
*** liuyulong has joined #openstack-nova06:20
*** hongbin has quit IRC06:21
*** igordc has joined #openstack-nova06:23
*** markvoelker has quit IRC06:24
*** khomesh|brb is now known as khomesh06:44
*** ccamacho has quit IRC06:44
*** takashin has left #openstack-nova06:50
*** belmoreira has quit IRC06:53
*** Luzi has joined #openstack-nova06:56
*** wolverineav has quit IRC07:03
*** wolverineav has joined #openstack-nova07:04
*** moshele has joined #openstack-nova07:04
*** mrjk has joined #openstack-nova07:05
*** mrjk has joined #openstack-nova07:06
*** mrjk has quit IRC07:07
*** wolverineav has quit IRC07:09
*** udesale has quit IRC07:10
*** _alastor_ has joined #openstack-nova07:13
*** _alastor_ has quit IRC07:18
*** udesale has joined #openstack-nova07:18
*** ccamacho has joined #openstack-nova07:20
*** udesale has quit IRC07:22
*** udesale has joined #openstack-nova07:26
*** igordc has quit IRC07:28
*** alex_xu has quit IRC07:28
*** liuyulong has quit IRC07:29
*** moshele has quit IRC07:31
*** wolverineav has joined #openstack-nova07:34
*** wolverineav has quit IRC07:38
*** mrjk has joined #openstack-nova07:40
*** mrjk has quit IRC07:40
*** mrjk has joined #openstack-nova07:40
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove deprecated 'flavors' policy  https://review.openstack.org/63366407:41
openstackgerritSundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access  https://review.openstack.org/63124207:42
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to spec obj.  https://review.openstack.org/63124307:42
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs.  https://review.openstack.org/63124407:42
*** slaweq has joined #openstack-nova07:43
*** sapd1 has joined #openstack-nova07:48
*** rpittau has joined #openstack-nova08:04
openstackgerritYongli He proposed openstack/nova master: Adds the server group info into show server detail API.  https://review.openstack.org/62147408:06
*** khomesh is now known as khomesh|brb08:10
*** sdake has quit IRC08:11
*** ralonsoh has joined #openstack-nova08:11
*** awalende has joined #openstack-nova08:12
*** khomesh|brb is now known as khomesh08:14
*** wolverineav has joined #openstack-nova08:15
openstackgerritYongli He proposed openstack/nova master: Clean up orphan instances  https://review.openstack.org/62776508:15
*** wolverineav has quit IRC08:19
*** markvoelker has joined #openstack-nova08:20
*** ttsiouts has joined #openstack-nova08:27
*** helenafm has joined #openstack-nova08:28
*** bnemec has joined #openstack-nova08:29
*** janki has joined #openstack-nova08:36
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove deprecated 'os-server-groups' policy  https://review.openstack.org/63367208:36
*** ttsiouts has quit IRC08:39
*** ttsiouts has joined #openstack-nova08:40
*** tosky has joined #openstack-nova08:42
*** ttsiouts has quit IRC08:44
*** yan0s has joined #openstack-nova08:46
*** tkajinam has quit IRC08:48
*** khomesh is now known as khomesh|bomgar08:48
*** pcaruana has joined #openstack-nova08:51
*** markvoelker has quit IRC08:54
*** priteau has joined #openstack-nova08:54
*** khomesh|bomgar is now known as khomesh|mtg08:56
kashyapgibi: Morning, are others seeing these timeouts too?  https://review.openstack.org/#/c/630980/09:02
kashyapIt's been like that for more than 10 days09:02
*** ttsiouts has joined #openstack-nova09:02
*** helenafm has quit IRC09:04
*** sridharg has quit IRC09:16
*** panda is now known as panda|numb09:28
*** sridharg has joined #openstack-nova09:28
*** khomesh|mtg is now known as khomesh09:29
*** helenafm has joined #openstack-nova09:30
jangutterkashyap: you can get a quick "feel" by checking http://zuul.openstack.org/builds?job_name=tempest-full&pipeline=gate and there does indeed seem to be a few hits.09:30
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix a missing policy in test policy data  https://review.openstack.org/63368609:30
kashyapjangutter: Hi, a few hits of what?09:30
* kashyap clicks09:31
jangutterkashyap: TIMED_OUT on tempest-full builds on the gate. Obviously this is highly unscientific, but a quick way to check.09:31
*** mvkr has joined #openstack-nova09:33
*** dtantsur|afk is now known as dtantsur09:35
*** derekh has joined #openstack-nova09:36
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix a missing policy in test policy data  https://review.openstack.org/63368609:37
jangutterkashyap: oh wow, that review seems to love hitting random gate failures.09:38
kashyapYeah :-(09:38
kashyapAnyway to fix that?09:39
*** bhagyashris has quit IRC09:39
jangutterkashyap: that's down to bugfixing, unfortunately. Digging into the logs might help to try to find out if it's something you can spot.09:43
kashyapjangutter: Will do.  But the last time I did, nothing was related to my patch :-(09:43
jangutterkashyap: yep, I can almost 100% guarantee that there isn't.09:43
jangutterkashyap: (a fix related to your patch, I mean).09:44
kashyapI don't see any failures here either: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/testr_results.html.gz09:44
jangutterkashyap: I think devstack took too long to spin up.09:45
jangutterkashyap: If you look at http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/ara-report/ <--- it took 30 minutes for the run-devstack task.09:50
jangutterkashyap: some of these failures require a skill level that waay beyond my ability to comprehend.09:50
*** markvoelker has joined #openstack-nova09:51
kashyapHmm09:52
kashyapThanks for looking; I'll check w/ the -infra folks09:52
jangutterkashyap: yeah, digging into that ara-report, there's a 30 minute timeout on tasks, and the task went too long. Could be that the VM was running under contention or something so there's a very high chance a recheck will succeed.09:56
kashyapAlright, we've had about 6 re-checks09:56
kashyapAnother one is in progress; let's see.09:57
*** maciejjozefczyk has quit IRC10:00
*** khomesh is now known as khomesh|gone10:01
*** ttsiouts has quit IRC10:02
*** ttsiouts has joined #openstack-nova10:02
*** ttsiouts has quit IRC10:07
*** jbernard has quit IRC10:14
*** jbernard has joined #openstack-nova10:17
*** sapd1 has quit IRC10:23
*** markvoelker has quit IRC10:24
*** ttsiouts has joined #openstack-nova10:35
*** Dinesh_Bhor has quit IRC10:38
*** mvkr has quit IRC10:46
*** belmoreira has joined #openstack-nova10:55
*** moshele has joined #openstack-nova10:55
*** tssurya has joined #openstack-nova11:00
*** janki has quit IRC11:02
*** mvkr has joined #openstack-nova11:03
*** udesale has quit IRC11:09
*** erlon has joined #openstack-nova11:12
*** tiendc has quit IRC11:14
*** _alastor_ has joined #openstack-nova11:15
*** sapd1 has joined #openstack-nova11:17
*** _alastor_ has quit IRC11:19
*** markvoelker has joined #openstack-nova11:21
tinwoodMorning.  Is there anyone who could take a look at https://review.openstack.org/#/c/633279/ for me; I needed to add the LXD bit, but not sure if I should have version bumped.  Many thanks!11:34
*** sapd1 has quit IRC11:36
*** maciejjozefczyk has joined #openstack-nova11:39
*** ttsiouts has quit IRC11:40
*** ttsiouts has joined #openstack-nova11:40
*** tbachman has quit IRC11:44
*** ttsiouts has quit IRC11:44
*** maciejjozefczyk has quit IRC11:47
gibikashyap: I didn't see such tempest timeout recently but not long ago we had problems with tempest slowness and moved some of the tests to tempest-slow11:49
ionihello guys11:51
gibikashyap: if you can dig up what are the slowest test in those runs that timed out then we can move some more test to tempest-slow11:51
ioniwhat's the right way to disable out of date images ?11:51
ioni--deactivate or --private or other way?11:51
ionibecause now i'm using --deactive and if an instance that is using this image wants to resize, nova returnes and error that the project doesn't have permission to that specific image11:52
ionithat is now in error state11:52
ionii'm doing something wrong or is a bug?11:52
*** markvoelker has quit IRC11:53
ionithe error log: https://paste.xinu.at/b6ZsNA/11:55
*** kukacz has quit IRC12:04
*** kukacz has joined #openstack-nova12:04
*** moshele has quit IRC12:04
*** cdent has joined #openstack-nova12:08
*** ttsiouts has joined #openstack-nova12:11
*** liuyulong has joined #openstack-nova12:12
*** ttsiouts has quit IRC12:16
*** ttsiouts has joined #openstack-nova12:18
*** moshele has joined #openstack-nova12:28
*** tetsuro has quit IRC12:28
*** ratailor has quit IRC12:31
*** cdent has quit IRC12:36
*** pcaruana has quit IRC12:40
*** pcaruana has joined #openstack-nova12:50
*** markvoelker has joined #openstack-nova12:50
jaosoriorkashyap: Hello! Are you the right person to ask a libvirtd related question?12:56
kashyapgibi: Give me a few; check out this: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/ara-report/13:02
kashyapgibi: Pointed out by jangutter earlier; where the 'run-devstack' task was taking 30 minutes13:02
kashyapjaosorior: Hi, you can you ask, I can try :-)13:02
jaosoriorkashyap: I'm trying to reload libvirtd, given that it's certificate got rotated. in the man pages it says SIGHUP should do that... but it seems it doesn't... wanted to know if this is a bug, or if there's some other mechanism I could/should use instead.13:03
*** priteau has quit IRC13:04
*** yan0s has quit IRC13:04
*** tbachman has joined #openstack-nova13:04
*** priteau has joined #openstack-nova13:04
kashyapjaosorior: Hmm, interesting problem13:06
kashyapjaosorior: Can you afford to do an explicit stop+start of the daemon?13:07
kashyapIf so, does that pick up the new certs?13:07
*** ileixe has quit IRC13:08
jaosoriorkashyap: would that disrupt instances running on the host?13:08
kashyapjaosorior: Good news: no.13:08
kashyapjaosorior: Hang on, though.13:09
jaosoriorkashyap: what about connections from nova?13:09
kashyapFirst, is that a production instance?13:09
jaosorioryes13:09
jaosoriorwell... not where I'm testing all this stuff :D13:09
*** cdent has joined #openstack-nova13:09
kashyapjaosorior: But know that: restarting `libvirtd` will _not_ bring your instances down.13:09
*** tbachman has quit IRC13:09
jaosoriorbut that's the goal, yes13:09
*** xek has joined #openstack-nova13:11
kashyapjaosorior: BTW, the SIGHUP config is ill-defined in the `libvirtd` man page :-(13:11
kashyapIt says: "On receipt of SIGHUP libvirtd will reload its configuration."13:11
jaosoriorright13:11
kashyapBut doesn't expand what is "its configuration".13:11
kashyapI'll get that sorted w/ the libvirt upstream to expand and elaborate.13:12
jaosoriorTLS certs/keys are configuration IMO :D13:12
*** priteau has quit IRC13:12
kashyapjaosorior: Okay, so you're using TLS.13:12
kashyapAnd you want to pick up renewed certificates.  Then problem is not libvirtd, I'm afraid — QEMU is the real blocker13:13
jaosoriorwhat are my options?13:14
kashyapBecause it (QEMU) can't reload certificates on the fly.  (And also QEMU can't just be "restarted")13:14
*** tbachman has joined #openstack-nova13:14
kashyapLet me check with libvirt folks.13:14
*** artom has quit IRC13:15
jaosoriorkashyap: thanks!13:15
*** artom has joined #openstack-nova13:15
kashyapjaosorior: So, your only viable option is a bit heavy: live migration of the QEMU instance(s).13:16
kashyap(I.e. Nova instances)13:16
ioniwhere the certs are used?13:16
*** yan0s has joined #openstack-nova13:17
kashyapioni: For TLS for various Nova services, I presume13:19
kashyapjaosorior: Related, recently we merged this: http://git.openstack.org/cgit/openstack/nova/tree/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst13:19
kashyapThere is a nicer rendered version13:20
*** markvoelker has quit IRC13:20
*** sean-k-mooney has quit IRC13:20
kashyapjaosorior: http://logs.openstack.org/83/631283/3/check/openstack-tox-docs/615979c/html/admin/secure-live-migration-with-qemu-native-tls.html13:20
*** ade_lee has quit IRC13:21
*** sean-k-mooney has joined #openstack-nova13:21
jaosoriorkashyap: I'm a little confused though. the TLS implementation for connections to libvirtd is not the same one then as the one for qemu, right?13:22
kashyap(But the above URL doesn't solve the existing problem, though.  It is "future goodness", when you upgrade.)13:22
*** yan0s has quit IRC13:22
kashyapjaosorior: Are you using the 'live_migration_tunnelled' Nova config?13:23
kashyapjaosorior: They're somewhat connected: when you use TLS with libvirt, it is passing the TLS creds under the hood to QEMU.13:24
jaosoriorkashyap: we are, yeah13:24
kashyapjaosorior: I can tell more details, if you post your QEMU command-line of your Nova instance (from the TLS env)13:24
jaosoriorkashyap: sure13:24
kashyapI need to grab a quick bite (not going to eat in front of the computer, as a rule).  Be back in 15 mins13:24
jaosoriorkashyap: sure. Good time for me to make some coffee13:25
*** mriedem has joined #openstack-nova13:28
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Skip checking of target_dev for vhostuser  https://review.openstack.org/63373413:32
*** ttsiouts has quit IRC13:32
*** ttsiouts has joined #openstack-nova13:33
*** yan0s has joined #openstack-nova13:34
*** ttsiouts has quit IRC13:37
*** zul has quit IRC13:38
*** zul has joined #openstack-nova13:41
*** agopi has joined #openstack-nova13:43
kashyapjaosorior: Let me know when you have the output.13:43
kashyapjaosorior: But regardless, this requires support in QEMU and libvirt.  I'll file RFEs w/ relevant upstreams13:50
*** ttsiouts has joined #openstack-nova13:50
jaosoriorkashyap: that would be great! (sorry, I'm in a meeting)13:50
*** mmethot has joined #openstack-nova13:55
*** panda|numb is now known as panda13:56
*** psachin has quit IRC13:56
*** pelement has joined #openstack-nova13:58
*** mlavalle has joined #openstack-nova13:58
*** jdillaman has joined #openstack-nova14:02
*** efried has quit IRC14:15
*** efried has joined #openstack-nova14:15
*** mchlumsky has joined #openstack-nova14:16
*** ade_lee has joined #openstack-nova14:17
*** efried has quit IRC14:19
*** efried has joined #openstack-nova14:19
*** BjoernT has joined #openstack-nova14:22
*** udesale has joined #openstack-nova14:24
*** yan0s has quit IRC14:30
*** yedongcan has left #openstack-nova14:36
*** dave-mccowan has joined #openstack-nova14:38
*** samc-bbc has joined #openstack-nova14:41
*** dave-mccowan has quit IRC14:45
*** pcaruana has quit IRC14:45
*** sdake has joined #openstack-nova14:48
*** eharney has joined #openstack-nova14:49
*** yan0s has joined #openstack-nova14:50
*** sdake has quit IRC14:51
*** pcaruana has joined #openstack-nova14:53
*** sdake has joined #openstack-nova14:54
*** awaugama has joined #openstack-nova14:59
*** cfriesen has joined #openstack-nova15:02
*** Luzi has quit IRC15:05
gibijaypipes, mriedem: thanks for the reviews on the bandwidth series. I will propose the fup patch based on your comments15:06
jaypipesgibi: ack. will try to get through remaining patches later.15:07
*** mmethot_ has joined #openstack-nova15:08
gibijaypipes: I saw your -1 in the middle. I will need to get back to dansmith about the second microversion15:08
*** mmethot has quit IRC15:09
mriedemgibi: yw, glad to help move this along15:09
mriedemalso hopefully we can maybe merge some code yet this month https://review.openstack.org/#/c/633566/15:10
*** hongbin has joined #openstack-nova15:13
openstackgerritSurya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells  https://review.openstack.org/59165715:15
*** READ10 has joined #openstack-nova15:16
gibimriedem: yeah, the gate is not on my side :)15:17
*** udesale has quit IRC15:18
*** wolverineav has joined #openstack-nova15:21
*** eharney has quit IRC15:21
openstackgerritMatt Riedemann proposed openstack/nova master: Add configuration of maximum disk devices to attach  https://review.openstack.org/61677715:22
*** ttsiouts has quit IRC15:23
mriedemi'm +2 on ^ now15:23
mriedemand the one below15:23
*** ttsiouts has joined #openstack-nova15:23
jaypipesmriedem: amen on the merging code comment.15:24
*** eharney has joined #openstack-nova15:24
*** moshele has quit IRC15:25
*** wolverineav has quit IRC15:25
kashyapmriedem: I'm only 3 months late on your question here: https://review.openstack.org/#/c/506720/15:26
kashyap(In short, I'm planning to revive that Secure Boot spec for Train.)15:26
*** ttsiouts has quit IRC15:28
kashyapI posted a few interim updates there based on the completed work in OVMF, QEMU, libvirt & related tooling.  I didn't post one since last August due to the libvirt bug still being in ASSIGNED state: https://bugzilla.redhat.com/show_bug.cgi?id=160512715:29
openstackbugzilla.redhat.com bug 1605127 in libvirt "RFE: QEMU firmware metadata format - libvirt support" [Unspecified,Assigned] - Assigned to phrdina15:29
*** sdake has quit IRC15:32
*** awalende has quit IRC15:33
mriedemjangutter: i looked at the evacuate network-vif-plugged timeout errors in https://review.openstack.org/#/c/571325/ - tldr is it looks like we get the event before we're ready for it, which is weird15:33
*** awalende has joined #openstack-nova15:33
janguttermriedem: think it's something transient or still surviving in master branches?15:34
mriedemi assume it's transient otherwise that job would be 100% failure15:34
mriedemwould be useful if we logged when we register the event callbacks15:34
*** sdake has joined #openstack-nova15:36
mriedemoh we do15:37
mriedemJan 29 01:11:55.567189 ubuntu-xenial-rax-ord-0002201755 nova-compute[15903]: DEBUG nova.compute.manager [None req-252304a1-6eff-4ff3-aa4d-b4e0ab87601c demo admin] [instance: 2e82ddcd-75b8-4a41-8ecd-ce175adbdc67] Preparing to wait for external event network-vif-plugged-e241f79f-fb0d-4961-b0c8-aea9de2755bf {{(pid=15903) prepare_for_instance_event /opt/stack/new/nova/nova/compute/manager.py:325}}15:37
*** awalende has quit IRC15:37
mriedemwhich is too late15:38
mriedemJan 29 01:11:49.707004 ubuntu-xenial-rax-ord-0002201755  devstack@n-api.service[22319]: DEBUG nova.api.openstack.wsgi  [req-3ed0ada1-7328-4d5a-a3ea-da34dcdb252d  req-1f5aeede-83c6-44d5-afd4-b5435e71d61f service nova] Action: 'create',  calling method: <bound method ServerExternalEventsController.create  of  <nova.api.openstack.compute.server_external_events.ServerExternalEventsController  object at 0x7fb0133aa050>>, body: {"e15:39
mriedems": [{"status":  "completed", "tag": "e241f79f-fb0d-4961-b0c8-aea9de2755bf", "name":  "network-vif-plugged", "server_uuid":  "2e82ddcd-75b8-4a41-8ecd-ce175adbdc67"}]} {{(pid=22322) _process_stack  /opt/stack/new/nova/nova/api/openstack/wsgi.py:520}}15:39
janguttermriedem: nicely reversed causality there.15:41
*** ttsiouts has joined #openstack-nova15:42
mriedemhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Timeout%20waiting%20for%20%5B('network-vif-plugged'%5C%22%20AND%20message%3A%5C%22for%20instance%20with%20vm_state%20error%20and%20task_state%20rebuild_spawning.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d15:43
*** ccamacho has quit IRC15:45
*** ccamacho has joined #openstack-nova15:46
mnaseris there any stable cores around to +w this? https://review.openstack.org/#/c/619352/15:46
mriedemjangutter: https://bugs.launchpad.net/nova/+bug/181378915:47
openstackLaunchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,Confirmed]15:47
*** liuyulong has quit IRC15:47
*** openstackgerrit has quit IRC15:51
*** _alastor_ has joined #openstack-nova16:01
*** _mmethot_ has joined #openstack-nova16:06
mriedemjaypipes: since melwitt is out this week i replied on https://review.openstack.org/#/c/632904/ for that TooManyDiskDevices API error check16:08
mriedemtl;dr there was already a lot of discussion between myself, melwitt, cdent and edleafe on that 40316:08
*** mmethot_ has quit IRC16:08
mriedemi personally don't care if we change to 409, but i don't really think 400 is correct16:09
*** yan0s has quit IRC16:10
cdentyeah, 409 is most correct, but 403 is consistent16:11
efriedmriedem: Where's the code that converts ComputeDriver.capabilities into traits on the compute node RP??16:17
mriedemefried: https://review.openstack.org/#/c/538498/16:18
*** openstackgerrit has joined #openstack-nova16:18
openstackgerritBalazs Gibizer proposed openstack/nova master: Fup for the bandwidth resource provider series  https://review.openstack.org/63377616:18
efriedo16:18
efriedWhy hasn't this merged yet <== because some a-hole -1'd it.16:18
mriedembecause i haven't updated it16:19
gibijaypipes, mriedem: here is the followup fixing comments in patches that are approved https://review.openstack.org/63377616:19
mriedemgibi: ack16:19
efriedmriedem: Cool beans, just knew I had seen said code somewhere, but couldn't find it in the master branch. Thanks.16:19
kashyapjangutter: Where do you see the 'run-devstack' taking 30 minutes here: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/ara-report/16:21
*** wolverineav has joined #openstack-nova16:21
kashyapjangutter: The one I found actually successfully "FINISHED"16:22
kashyap(Although taking 30 mins & completing are not mutually exclusive.)16:22
jangutterkashyap: in the devstack-tempest.yaml line, click on the "16 Tasks" button to pull them down. The run-devstack : Run devstack task went over 30 minutes.16:22
kashyapjangutter: Right, there the status of 'run-devstack' task is "CHANGED".16:23
jangutterkashyap: if you click on the parameters for the plays, you'll also see "timeout 30"16:23
kashyapjangutter: Right, there I see it's 30 mins.  And when I click on its 'Status' ("CHANGED" - what does it even mean), the job finished.16:24
kashyapSo I don't see any action item there.16:24
jangutterkashyap: the playbook was interrupted, so the ARA results are inconsistent16:24
kashyapHmm16:24
kashyapJust want to put this change out of its misery; it's been in time-out/recheck hell for 9 days16:25
jangutterkashyap: http://logs.openstack.org/80/630980/7/gate/tempest-full/5b6ebba/job-output.txt.gz#_2019-01-29_00_14_28_138371 <--- this Ansible telling you the play timed out.16:26
kashyapAh, _there_ is the specific piece of hay in the haystack!16:27
jangutterkashyap: is the recheck still in the gate?16:27
kashyapYeah, 8 hours and running.16:27
kashyapOr something like that.  But it doesn't bother me much, I've got a more than a few irons in the fire.16:28
kashyapJust that it's annoying to wake up each morning to see that thing time out.16:28
jangutterkashyap: the sarlacc pit is slow, but thorough. (plagiarizing and butchering jaypipes).16:28
openstackgerritAdrian Chiris proposed openstack/nova master: Add free for claimed, allocated devices  https://review.openstack.org/61612016:28
openstackgerritAdrian Chiris proposed openstack/nova master: Allow per-port modification of vnic_type and profile  https://review.openstack.org/60736516:28
openstackgerritAdrian Chiris proposed openstack/nova master: Add get_instance_pci_request_from_vif  https://review.openstack.org/61992916:28
openstackgerritAdrian Chiris proposed openstack/nova master: SR-IOV Live migration indirect port support  https://review.openstack.org/62011516:28
mriedemgibi: regarding the microversion for move operations comment in https://review.openstack.org/#/c/630722/ i replied that i don't think we have to make that decision at that point in the series16:28
kashyapjangutter: Hehe16:28
jangutterkashyap: you'll see the tempest tests above that line have got times attached. Not sure if you could also grab them from the testr results, but chances are one of them is a candidate for the -slow set.16:29
*** wolverineav has quit IRC16:30
*** pelement has left #openstack-nova16:30
jangutterkashyap: I did see your review seems to have hit _different_ snags in the gate each time, so congratulations on being kind of a lightning rod.16:30
kashyapjangutter: Yeah, I see some of the network tests are taking terribly long (test_server_connectivity_reboot)16:31
jangutterkashyap: a heck of a lot of neutron and keystone reviews are getting booted because of what seems to be a sudden tightening up of pep8 enforcement.16:32
kashyapjangutter: Yep, mumble-mumble something about "good deeds".16:32
kashyapjangutter: I see, thanks for the info.16:33
*** Nel1x has joined #openstack-nova16:36
*** tstrul has joined #openstack-nova16:37
*** tstrul has quit IRC16:38
*** agopi is now known as agopi|lunch16:41
*** TxGirlGeek has joined #openstack-nova16:42
*** mchlumsky has quit IRC16:45
*** mdbooth has joined #openstack-nova16:45
*** sdake has quit IRC16:46
*** mdbooth_ has quit IRC16:47
*** mchlumsky has joined #openstack-nova16:48
*** wwriverrat has joined #openstack-nova16:51
*** hamzy has quit IRC16:56
*** helenafm has quit IRC16:57
*** hamzy has joined #openstack-nova17:00
*** pcaruana has quit IRC17:01
*** gyee has joined #openstack-nova17:03
*** ttsiouts has quit IRC17:06
*** ttsiouts has joined #openstack-nova17:06
*** macza has joined #openstack-nova17:07
*** tssurya has quit IRC17:09
*** ttsiouts has quit IRC17:10
*** hamzy has quit IRC17:11
*** hongbin has quit IRC17:14
*** sdake has joined #openstack-nova17:17
*** sdake has quit IRC17:18
*** ccamacho has quit IRC17:18
*** rpittau has quit IRC17:21
*** rpittau has joined #openstack-nova17:21
stephenfinNice docs patch here if anyone fancies taking a look https://review.openstack.org/#/c/62693117:23
*** bnemec has quit IRC17:27
kashyapstephenfin: Yes, that's an important point easy to miss.17:31
kashyap+1ed, FWIW.17:32
*** rpittau has quit IRC17:35
openstackgerritMerged openstack/nova master: libvirt: A few miscellaneous items related to "native TLS"  https://review.openstack.org/63098017:36
openstackgerritMerged openstack/nova master: docs: Update references to "QEMU-native TLS" document  https://review.openstack.org/63128317:36
jangutterhehehe, kashyap, another timeout http://logs.openstack.org/59/624959/5/gate/tempest-full/ce2253e/job-output.txt.gz#_2019-01-29_17_20_17_952753 but looks like your reviews got merged!17:41
kashyapVery bizarre17:42
kashyapBut I won't complain17:42
kashyapThat's a good excuse to go cook some dinner.17:42
kashyapjangutter: The difference is time-out was not in the _same_ job ("tempest-full"), which succeeded this time.17:43
*** sdake has joined #openstack-nova17:44
*** sridharg has quit IRC17:44
*** agopi|lunch is now known as agopi17:46
*** tbachman has quit IRC17:55
*** derekh has quit IRC18:01
*** igordc has joined #openstack-nova18:03
*** sdake has quit IRC18:10
openstackgerritJack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store  https://review.openstack.org/62164618:10
openstackgerritJack Ding proposed openstack/nova master: Improve libvirt image and snapshot handling  https://review.openstack.org/61669218:12
openstackgerritJack Ding proposed openstack/nova master: Correct instance port binding for rebuilds  https://review.openstack.org/60384418:14
*** ralonsoh has quit IRC18:22
*** sdake has joined #openstack-nova18:37
*** awalende has joined #openstack-nova18:45
*** tomtom001 has joined #openstack-nova18:49
*** awalende has quit IRC18:49
tomtom001hello, I am running openstack queens and I am having trouble getting nova instances to boot using emphemeral storage.  I have force_raw_images = false and use_cow_images = true for my settings, would these conflict with that?  I also have Host aggregates setting ephemeralcomputestorage = false  would this cause a problem for the boot process?  When my instances boot they just go to "No bootable18:51
tomtom001device"  nova, cinder, and ceph seem to be all functioning correctly.18:52
*** hamzy has joined #openstack-nova18:52
tomtom001While I've found documentation for the nova settings, I have not found any for the Host Aggregate settings and how it affects the storage selection.18:53
*** mdbooth_ has joined #openstack-nova18:54
*** mdbooth has quit IRC18:57
*** _fragatina has joined #openstack-nova19:01
*** mvkr has quit IRC19:06
*** whoami-rajat has quit IRC19:07
mriedemephemeralcomputestorage isn't a special key in nova,19:07
mriedempresumably you're trying to use that to restrict certain flavors or images to that host aggregate and linking it by the ephemeralcomputestorage key19:07
mriedeme.g. https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregateimagepropertiesisolation and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregateinstanceextraspecsfilter19:08
*** pelement has joined #openstack-nova19:09
tomtom001mriedem: so if I've done that then I wouldn't be able to use ephemeral on our host aggregates with that key set?19:09
mriedemare you trying to boot from volume?19:10
mriedemi don't know where the "no bootable device" error is coming from - libvirt?19:10
mriedemalso not sure what you mean by ephemeral storage - do you mean local storage on the compute node?19:11
mriedemi.e. not volume backed19:11
tomtom001from the console of the instance, showing no bootable device, ephemeral is not local nova disk, just not volume backed.19:12
mriedemsounds like a problem with the image19:13
tomtom001The same images work if booted with a volume backing.19:14
*** dtantsur is now known as dtantsur|afk19:17
openstackgerritHang Yang proposed openstack/nova stable/rocky: Fix port dns_name reset  https://review.openstack.org/63380619:23
openstackgerritHang Yang proposed openstack/nova stable/queens: Fix port dns_name reset  https://review.openstack.org/63380719:24
*** tbachman has joined #openstack-nova19:31
ionihello guys19:34
ioniwhat's the right way to disable out of date images ?19:34
ioni--deactivate or --private or other way?19:34
ionibecause now i'm using --deactive and if an instance that is using this image wants to resize, nova returnes and error that the project doesn't have permission to that specific image19:35
ionithe error log: https://paste.xinu.at/b6ZsNA/19:35
*** tbachman has quit IRC19:35
*** tbachman has joined #openstack-nova19:41
jaypipesioni: I would do the --private (or --protected + --private) route.19:48
ionijaypipes, that will prevent having issue with resizing ?19:49
ionithats what i want to resolve19:49
jaypipesioni: is your entire goal just to remove the image from the list of images that users see in Horizon or `openstack image list`?19:50
ionijaypipes, indeed19:50
*** awaugama has quit IRC19:51
openstackgerritJack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store  https://review.openstack.org/62164619:52
jaypipesioni: you're on Queens, right?19:52
ionijaypipes, yes19:52
jaypipesioni: k.19:52
jaypipesmriedem: if ioni deletes an image will that image still be available for resize operations? I know we continue to store the image metadata for instances but not sure if our resize operation will barf at deleted images or not.19:53
jaypipesioni: I'm assuming here you're talking about same-original-image resizes, not resize with a different image...19:53
ionijaypipes, that's mostly the case I didn't want to delete images, i don't know what happens19:53
jaypipesioni: ack. I'm not entirely sure either, which is why I called out the big guns (mriedem)19:54
ionijaypipes, openstack image resize, so yes, the same original image19:54
ioni*openstack server resize19:54
mriedemthere is no resize with a different image19:54
jaypipesmriedem: ah, sorry, yea, was thinking rebuild.19:54
mriedemi suspect that if you delete the backing image and try to resize it will fail b/c the dest host won't be able to pull down the image19:55
ionii don't know why is trying to check the original image19:55
ionii have like 7 images with centos 7,  built in various time acros years and some instances are running on old images19:55
ionii do update images from time to time to include up to date packages19:56
ionii only want the most up to date image to be available19:56
ionibut also resize to work19:56
mriedemioni: have you asked the glance folk in #openstack-glance or the operators in #openstack-operators?19:58
mriedemi'm not sure what the difference is between a deactivated image and a deleted image19:58
*** erlon has quit IRC19:58
mriedemhttps://docs.openstack.org/glance/latest/admin/index.html isn't helping me19:58
ionimriedem, i didn't ask in there, mostly because nova fails with an error19:58
ioniso i was thinking that's a nova issue19:58
mriedemyou could maybe make the image visibility 'shared' and then only grant access to the projects that need access to that image to resize https://developer.openstack.org/api-ref/image/v2/index.html#sharing19:59
mriedembut then it's not public for all projects to use19:59
mriedemand it wouldn't show up in the default image list for those sharing member projects, only the owner (admin i presume)19:59
mriedemanyway seems like a pretty common problem, i'd try #openstack-glance or #openstack-operators20:00
mriedemit's not really a nova issue20:00
ioniok20:00
ionithanks20:00
mriedemif you find out some juicy details please report back :)20:00
mriedemhttps://docs.openstack.org/operations-guide/ops-user-facing-operations.html#deleting-images also isn't very helpful20:02
mriedem"deleting images is fine unless you need to migrate servers"20:02
mriedem:(20:02
ioniso is mostly about driver20:03
ionior about the storage backend20:04
ioniqcow, raw or other20:04
mriedemwell, libvirt is by far the most widely used compute driver,20:05
mriedemif you were using shared instance storage across all of your computes, then the image might be cached and you don't need to fetch it from glance,20:05
mriedembut that's not a deployment strategy that everyone uses20:05
*** awaugama has joined #openstack-nova20:06
mriedemioni: btw i'm asking in #openstack-operators20:07
ionimriedem, yes, i saw, i'm going to continue there20:07
*** agopi is now known as agopi|brb20:10
*** agopi|brb has quit IRC20:10
mriedemjaypipes: the answer was change the image status to "community" which hides it from the image list for all users except the owner, but you can reach it by uuid if you know it20:11
mriedemso like shared but less private20:11
jaypipesmriedem: ugh20:18
jaypipesmriedem: that is so very non-obvious.20:18
mriedemat least there is a solution20:20
*** ttsiouts has joined #openstack-nova20:26
cdent"at least there is a solution" is the first album from my band "so very non-obvious"20:26
*** awaugama has quit IRC20:30
artomI thought My Chemical Romance already did that?20:35
*** READ10 has quit IRC20:40
cdentmriedem, jaypipes : can either of you remember the details of what's left on the nova side to make shared disk a possibility?20:43
cdentThe reason I ask is because I've been modelling various disk set ups in placement with the usual gabbits things I do, and it works very nicely from that side of things.20:44
*** sdake has quit IRC20:45
mriedemthe bug about it has details20:52
jaypipescdent: mostly I think it's just that nova and the virt drivers simply have no idea that instances use shared storage or not. There needs to be something added to the resource tracker (I had proposed that provider YAML config file for this type of purpose) that communicates to the nova-compute service that the compute node has access to shared storage.20:52
mriedemhttps://bugs.launchpad.net/nova/+bug/178402020:52
openstackLaunchpad bug 1784020 in OpenStack Compute (nova) "Shared storage providers are not supported and will break things if used" [Medium,Fix released]20:52
mriedemthere are lots of TODOs and FIXMEs in the code with things like "this will need to change to support sharing providers"20:53
mriedemlike in the cold migration flow20:53
cdentthanks, I had lost track of that bug20:53
* cdent nods20:53
mriedemalso L220 https://etherpad.openstack.org/p/nova-ptg-stein20:53
jaypipescdent: along with a giant-ass reshape-the-world-for-shared-storage function that would "correct" the allocation of resources.20:53
*** xek has quit IRC20:54
cdentIt's looking like shared disk could make multiple datastores in one vsphere cluster way more sensible. nested could also work, but shared will work in both the "this cluster owns this store" and "this store is used by multiple clusters" set ups20:57
mriedemfyi, new pip release of something is destroying pep8 in the gate21:05
mriedemthis https://pypi.org/project/pycodestyle/2.5.0/21:06
cdentjoy!21:11
*** munimeha1 has joined #openstack-nova21:11
cdentthis is the disk playing about I was talking about ^^ https://github.com/cdent/placement-disk-models21:11
*** cdent has quit IRC21:17
*** ade_lee has quit IRC21:38
*** hamzy has quit IRC21:42
*** eharney has quit IRC21:48
*** pelement has left #openstack-nova21:56
*** mchlumsky has quit IRC22:00
mriedemgood lord i really dislike the PciDevice* object interfaces22:04
mriedemso different from everything else we have for versioned objects22:04
mriedemburying too much business logic in the data model22:06
mriedemit's that some kind of anti-pattern...22:06
mriedem*isn't that22:06
*** BjoernT has quit IRC22:15
*** sdake has joined #openstack-nova22:17
*** igordc has quit IRC22:21
*** ade_lee has joined #openstack-nova22:24
*** jdillaman has quit IRC22:32
*** TxGirlGeek has quit IRC22:32
*** igordc has joined #openstack-nova22:34
*** gyee has quit IRC22:35
*** slaweq has quit IRC22:36
mriedemdansmith: interesting fun fact of the day: you can't load 'fault' using expected_attrs with Instance.get_by_uuid22:38
mriedemhttp://paste.openstack.org/show/744212/22:40
mriedemsomething everyone cares about22:40
*** ade_lee has quit IRC22:41
*** ade_lee has joined #openstack-nova22:42
*** munimeha1 has quit IRC22:43
*** tkajinam has joined #openstack-nova22:55
-openstackstatus- NOTICE: http://zuul.openstack.org is not working. https://zuul.openstack.org does work. Please use that while we investigate.23:14
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove deprecated 'os-server-groups' policy  https://review.openstack.org/63367223:24
*** TxGirlGeek has joined #openstack-nova23:27
*** TxGirlGeek has quit IRC23:29
*** TxGirlGe_ has joined #openstack-nova23:29
*** sdake has quit IRC23:35
*** sdake has joined #openstack-nova23:37
*** rcernin has quit IRC23:53
*** sdake has quit IRC23:55
*** sdake has joined #openstack-nova23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!