Thursday, 2017-02-02

openstackgerritMatt Riedemann proposed openstack/nova: Fix the terminated_at field in the server query params schema  https://review.openstack.org/42796400:00
*** erlon has quit IRC00:00
*** krtaylor has quit IRC00:02
openstackgerritMatt Riedemann proposed openstack/nova: Handle NotImplementedError in _process_instance_vif_deleted_event  https://review.openstack.org/42797400:05
*** karimb has quit IRC00:06
*** mriedem has quit IRC00:06
*** xyang1 has quit IRC00:12
*** hongbin has quit IRC00:16
*** tbachman has quit IRC00:18
*** karimb has joined #openstack-nova00:18
*** sdague has quit IRC00:21
*** jaosorior has quit IRC00:27
*** unicell has quit IRC00:34
*** dtp-afk has quit IRC00:34
*** yamahata has quit IRC00:35
*** owalsh-afk has quit IRC00:37
*** chyka has quit IRC00:38
*** dimtruck is now known as zz_dimtruck00:38
*** ducttape_ has quit IRC00:39
*** ducttape_ has joined #openstack-nova00:39
*** ducttape_ has quit IRC00:41
*** _ducttape_ has joined #openstack-nova00:41
*** thorst_ has joined #openstack-nova00:41
*** _ducttape_ has quit IRC00:42
*** ducttape_ has joined #openstack-nova00:42
*** nic has quit IRC00:43
*** unicell has joined #openstack-nova00:48
*** tbachman has joined #openstack-nova00:48
*** ijw has quit IRC00:50
*** jamesdenton has joined #openstack-nova00:53
*** rfolco has quit IRC00:58
*** openstackgerrit has quit IRC01:02
*** thorst_ has quit IRC01:05
*** dave-mccowan has quit IRC01:05
*** thorst_ has joined #openstack-nova01:06
*** john51 has joined #openstack-nova01:06
*** rfolco has joined #openstack-nova01:08
*** rfolco has quit IRC01:08
*** thorst_ has quit IRC01:10
*** john51_ has quit IRC01:11
*** raunak has quit IRC01:13
*** unicell has quit IRC01:14
*** raunak has joined #openstack-nova01:15
*** raunak has quit IRC01:15
*** thorst_ has joined #openstack-nova01:16
oomichimriedem: cool, nice catch01:22
*** krtaylor has joined #openstack-nova01:22
*** thorst_ has quit IRC01:25
*** rfolco has joined #openstack-nova01:27
*** rfolco has quit IRC01:27
*** rfolco has joined #openstack-nova01:27
*** rfolco has quit IRC01:28
*** karimb has quit IRC01:30
*** openstackgerrit has joined #openstack-nova01:31
openstackgerritZhenyu Zheng proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42797801:31
*** rfolco has joined #openstack-nova01:31
*** rfolco has quit IRC01:31
*** rfolco has joined #openstack-nova01:32
*** unicell has joined #openstack-nova01:32
*** rfolco has quit IRC01:32
*** mriedem has joined #openstack-nova01:32
*** tlian has quit IRC01:32
*** thorst_ has joined #openstack-nova01:33
*** thorst_ has quit IRC01:33
*** zz_dimtruck is now known as dimtruck01:35
mriedemhttps://review.openstack.org/#/c/427782/ is merged01:37
mriedemwoot01:37
mriedemhopefully we see that bug drop off01:37
mriedemthe postgres fix is merged too https://review.openstack.org/#/c/427667/01:38
openstackgerritZhenyu Zheng proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42797801:40
*** tlian has joined #openstack-nova01:41
*** Apoorva has quit IRC01:41
*** sree has joined #openstack-nova01:44
*** jose-phillips has quit IRC01:44
*** hfu has joined #openstack-nova01:46
melwittmriedem: I had forgotten I said I'd write up step-by-step examples for setting up cells v2, so I did it just now. do you think it's too late for that?01:46
*** thorst_ has joined #openstack-nova01:53
*** mtanino has quit IRC01:53
mriedemno01:53
mriedemlink?01:53
mriedemwe really needed some upgrade steps01:53
mriedembut i'll take anything01:53
melwittyay good. one sec, getting some sphinx warnings01:53
melwitt"Title level inconsistent" I'm trying to figure out what level is one step smaller than ~~~~~ for a section01:54
*** thorst_ has quit IRC01:57
mriedem----- ?01:59
mriedemmelwitt: ^01:59
mriedemCells V2 ========  Manifesto ~~~~~~~~~  Proposal --------02:00
melwittyeah, I think you're right. thanks02:00
*** Sukhdev has quit IRC02:02
*** Apoorva has joined #openstack-nova02:02
openstackgerritmelanie witt proposed openstack/nova master: Add step-by-step examples for Cells V2 setup  https://review.openstack.org/42798102:03
*** jdillaman has quit IRC02:04
openstackgerritZhenyu Zheng proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42797802:04
melwittmriedem: ^02:04
gmannKevin_Zheng: looks like you are back, updating doc patch as per mriedem  comments ?02:05
*** Apoorva has quit IRC02:07
openstackgerritZhenyu Zheng proposed openstack/nova master: Trivial Fix typo in schema  https://review.openstack.org/42798302:08
*** zhurong has joined #openstack-nova02:08
Kevin_Zhenggmann: yeah I'm trying to02:08
*** tanee is now known as tanee_away02:08
*** efried has quit IRC02:09
gmannKevin_Zheng: thanks, please let me know any help. we should get those last bits in this week02:09
*** Jeffrey4l_ has quit IRC02:10
Kevin_Zhenggmann: sure02:10
Kevin_ZhengThanks02:10
*** tanee_away is now known as tanee02:10
*** efried has joined #openstack-nova02:11
*** dimtruck is now known as zz_dimtruck02:11
*** unicell has quit IRC02:12
gmannKevin_Zheng: terminated_at is fixed in https://review.openstack.org/#/c/427964/02:15
Kevin_ZhengYeah just saw it02:15
gmannok02:16
*** jdillaman has joined #openstack-nova02:17
*** cdelatte has quit IRC02:19
openstackgerritTakashi NATSUME proposed openstack/nova-specs master: Enable cold migration with target host  https://review.openstack.org/42663602:20
*** zhurong has quit IRC02:20
openstackgerritTakashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration  https://review.openstack.org/33473202:21
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Fix deprecated proxy API parameters  https://review.openstack.org/41688202:21
openstackgerritTakashi NATSUME proposed openstack/nova master: Use ListOfIntegersField in oslo.versionedobjects  https://review.openstack.org/42391202:22
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Fix parameters and examples in aggregate API  https://review.openstack.org/42551202:22
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Fix parameters in os-hosts.inc  https://review.openstack.org/41644402:22
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix doc generation warnings  https://review.openstack.org/42554902:22
openstackgerritTakashi NATSUME proposed openstack/nova master: Refactor a test method including 3 test cases  https://review.openstack.org/39473802:22
mriedemmelwitt: comments inline02:24
mriedemmelwitt: i worry a little about duplicating some of the stuff that dan wrote above that in the first time setup section02:24
*** zz_dimtruck is now known as dimtruck02:25
melwittthanks02:26
melwittmriedem: I thought about that a little too. I was thinking we should probably have short "do these steps" things for people to look at to help understand the different commands. well, based on the convos we had with operators in the channel about it02:28
mriedemyeah,02:29
*** thorst_ has joined #openstack-nova02:29
*** thorst_ has quit IRC02:29
mriedemi don't have a great solution to not be redundant but at the same time, have everything anyone would ever want or need to know in a single section and not make that so overly complicate with if/else conditions that no one can understand it02:29
mriedemcan't win really02:29
*** sree has quit IRC02:29
mriedemi've always failed at process/guide docs like that02:30
*** dimtruck is now known as zz_dimtruck02:30
mriedemthe balance between too much and too little info, because no matter how you write it, someone won't be able to follow it02:30
mriedemlived with that for years in my last project before openstack02:30
melwittyeah. what's there already is a thorough explanation of all the things, but for the sort of roll up or quick reference, we don't have anything02:31
melwittmaybe that page isn't the best place for it but I wasn't sure where else we would put it02:31
openstackgerritHidekazu Nakamura proposed openstack/nova master: Do not attempt volume swap when guest is stopped  https://review.openstack.org/38979802:33
*** raunak has joined #openstack-nova02:38
*** raunak has quit IRC02:40
openstackgerritMatt Riedemann proposed openstack/nova master: Handle NotImplementedError in _process_instance_vif_deleted_event  https://review.openstack.org/42797402:41
*** raunak has joined #openstack-nova02:42
*** raunak has quit IRC02:42
*** hieulq has joined #openstack-nova02:48
mriedemmelwitt: so i thought about this on the way home for some reason https://review.openstack.org/#/c/355091/02:51
mriedemi don't really want to open a can of worms,02:51
mriedembut i'm not sure if what we have for placement right now resolves that with reporting 0 DISK_GB for boot from volume or not02:51
mriedemi think it does02:52
*** thorst_ has joined #openstack-nova02:52
*** thorst_ has quit IRC02:52
*** kaisers_ has joined #openstack-nova02:53
melwittmriedem: yeah, I skimmed some ML threads on that and it wasn't clear to me if the boot from volume disk reporting problem is solved yet with placement02:53
mriedem"We're already reporting proper allocation information for instances with  no disk (volume-backed and others) and as soon as we look to the  placement API in the scheduler we'll stop suffering from this problem."02:53
mriedem^ as long as the scheduler is using placement, we're filtering on DISK_GB now02:53
melwittmriedem: is it now? I've been way out of the loop on what's merged02:54
*** hongbin has joined #openstack-nova02:55
mriedemmelwitt: yeah https://review.openstack.org/#/c/417961/02:56
mriedemif the minimum nova-compute service version in the deployment is new enough, the filter scheduler will use placement now,02:56
mriedemincluding filtering on DISK_GB02:56
*** kaisers has quit IRC02:56
mriedemhttps://review.openstack.org/#/c/417961/41/nova/scheduler/filter_scheduler.py@13902:56
melwittokay, cool02:57
melwittwell, that's a win, glad to see that02:58
mriedemhttps://review.openstack.org/#/c/427367/ adds some upgrade notes to the placement devref02:58
mriedemmight help you catch up a bit02:58
melwittthanks02:59
mriedemi missed pretty much all of the moving instance create to conductor, but after the last 24 hours of bugs i feel like i've caught up :)02:59
*** zerda3 has joined #openstack-nova02:59
melwittI wonder how it makes sure volume backed instance doesn't request the flavor's amount of disk when rebuilding etc03:00
melwitthaha, yeah really03:00
*** raunak has joined #openstack-nova03:01
*** bkopilov has quit IRC03:02
*** ssurana has quit IRC03:02
*** tlian has quit IRC03:06
*** jschlueter has quit IRC03:08
*** jschlueter has joined #openstack-nova03:09
*** hieulq has quit IRC03:09
*** jamesdenton has quit IRC03:10
*** jamesden_ has joined #openstack-nova03:10
mriedemgot some other cells v2 related docs changes here, small ones https://review.openstack.org/#/q/topic:man+project:openstack/nova+status:open03:11
*** thorst_ has joined #openstack-nova03:13
*** tlian has joined #openstack-nova03:18
*** mtanino has joined #openstack-nova03:20
*** thorst_ has quit IRC03:20
*** gouthamr has quit IRC03:24
*** mtanino has quit IRC03:24
*** abalutoiu has quit IRC03:34
*** armax has quit IRC03:40
*** Jeffrey4l has joined #openstack-nova03:41
*** jamesden_ has quit IRC03:41
*** jamesdenton has joined #openstack-nova03:41
*** jamesdenton has quit IRC03:50
*** tlian has quit IRC03:52
*** jamesdenton has joined #openstack-nova03:53
openstackgerritMatt Riedemann proposed openstack/nova master: Fix typo in help for discover_hosts_in_cells_interval  https://review.openstack.org/42799703:54
*** thorst_ has joined #openstack-nova03:55
*** thorst_ has quit IRC03:55
*** armax has joined #openstack-nova03:56
*** baoli has joined #openstack-nova03:57
*** tbachman has quit IRC03:57
*** nicolasbock has quit IRC04:01
*** raunak has quit IRC04:01
*** baoli has quit IRC04:01
*** tbachman has joined #openstack-nova04:02
*** armax has quit IRC04:10
*** diga has joined #openstack-nova04:14
*** jamesdenton has quit IRC04:17
*** tbachman has quit IRC04:17
*** psachin has joined #openstack-nova04:18
*** bkopilov has joined #openstack-nova04:19
mriedemgibi: https://review.openstack.org/#/c/427647/ apparently the cinder people have figured out how to short circuit sending notifications if they aren't enabled04:21
mriedemhttps://review.openstack.org/#/c/42654704:22
*** armax has joined #openstack-nova04:22
*** dikonoor has joined #openstack-nova04:27
*** mdnadeem has joined #openstack-nova04:29
*** jamesdenton has joined #openstack-nova04:30
*** guchihiro has joined #openstack-nova04:34
*** armax has quit IRC04:34
*** jamesdenton has quit IRC04:35
*** jamesden_ has joined #openstack-nova04:35
*** abhishek_k has joined #openstack-nova04:35
*** abhishekk has quit IRC04:35
*** ducttape_ has quit IRC04:36
*** udesale has joined #openstack-nova04:36
*** Kevin_Zheng has quit IRC04:37
*** jamesdenton has joined #openstack-nova04:37
*** jamesden_ has quit IRC04:39
*** ssurana has joined #openstack-nova04:40
*** jamesdenton has quit IRC04:42
*** jamesden_ has joined #openstack-nova04:43
*** ayogi has joined #openstack-nova04:48
*** adisky_ has joined #openstack-nova04:50
*** sree has joined #openstack-nova04:51
*** hongbin_ has joined #openstack-nova04:53
*** archies has joined #openstack-nova04:55
*** hongbin has quit IRC04:56
*** sree has quit IRC04:56
*** sree has joined #openstack-nova04:58
*** efried has quit IRC04:59
*** hfu has quit IRC04:59
*** hfu has joined #openstack-nova05:01
*** hfu has quit IRC05:01
*** pbandark has joined #openstack-nova05:01
*** hfu has joined #openstack-nova05:01
*** hfu has quit IRC05:02
*** sree has quit IRC05:02
*** hfu has joined #openstack-nova05:02
*** hfu has quit IRC05:03
archies@team, I am facing an interesting issue while attaching a volume. In libvirt the error is "operation failed: open disk image file failed"05:03
archiesAny idea?05:03
*** sree has joined #openstack-nova05:04
*** ratailor has joined #openstack-nova05:08
*** guchihiro has quit IRC05:09
*** efried has joined #openstack-nova05:12
*** prateek has joined #openstack-nova05:12
*** jamesden_ has quit IRC05:24
*** bkopilov_ has joined #openstack-nova05:25
*** amotoki has joined #openstack-nova05:28
*** thorst_ has joined #openstack-nova05:33
*** satyar has joined #openstack-nova05:37
*** ducttape_ has joined #openstack-nova05:37
*** thorst_ has quit IRC05:38
*** ducttape_ has quit IRC05:42
openstackgerritZhenyu Zheng proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42797805:42
*** ayogi has quit IRC05:46
*** Sukhdev has joined #openstack-nova05:46
*** trinaths has joined #openstack-nova05:47
*** ayogi has joined #openstack-nova05:49
openstackgerritZhenyu Zheng proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42797805:53
*** hongbin_ has quit IRC05:53
*** unicell has joined #openstack-nova05:55
*** unicell has quit IRC05:57
*** unicell has joined #openstack-nova06:01
trinaths  Hi-     While enabling VM with huge pages, I get the following error.  Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology.   I’m using devstack (stable/newton) and nova (6.0.0).      I have configured hugepages and flavors following the guide at http://docs.openstack.org/admin-guide/compute-huge-pages.html  .   Here is the configuration of the host.     http://paste.openstack.org/sh06:01
*** hfu has joined #openstack-nova06:04
*** mdnadeem has quit IRC06:06
*** hfu has quit IRC06:06
*** efried has quit IRC06:07
*** sree has quit IRC06:08
*** kaisers_ has quit IRC06:09
*** artom_ has joined #openstack-nova06:09
*** markvoelker has joined #openstack-nova06:10
*** satyar_ has joined #openstack-nova06:10
openstackgerritZhenyu Zheng proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42797806:11
*** efried has joined #openstack-nova06:11
*** rnoriega has quit IRC06:11
*** pkoniszewski has quit IRC06:11
*** mdnadeem has joined #openstack-nova06:12
*** artom has quit IRC06:12
*** timello has quit IRC06:12
*** markvoelker_ has quit IRC06:12
*** egarbade- has quit IRC06:12
*** John341_ has quit IRC06:12
*** mnaser has quit IRC06:12
*** satyar has quit IRC06:13
*** abhishek_k has quit IRC06:13
*** abhishek_k has joined #openstack-nova06:13
*** John341_ has joined #openstack-nova06:14
*** pkoniszewski has joined #openstack-nova06:15
*** rnoriega has joined #openstack-nova06:18
*** lpetrut has joined #openstack-nova06:18
*** egarbade- has joined #openstack-nova06:19
*** jdurgin1 has quit IRC06:19
*** unicell1 has joined #openstack-nova06:20
*** unicell has quit IRC06:21
*** sree has joined #openstack-nova06:21
*** mnaser has joined #openstack-nova06:23
*** timello has joined #openstack-nova06:24
*** Jeffrey4l has quit IRC06:25
*** avolkov has joined #openstack-nova06:26
*** Jeffrey4l has joined #openstack-nova06:26
*** kaisers has joined #openstack-nova06:30
*** mjura has joined #openstack-nova06:31
*** vsaienko has joined #openstack-nova06:32
*** Hosam has joined #openstack-nova06:35
*** vsaienko has quit IRC06:37
*** sudipto_ has joined #openstack-nova06:38
*** sudipto has joined #openstack-nova06:38
*** psachin has quit IRC06:42
*** clayton has quit IRC06:46
*** markus_z has joined #openstack-nova06:49
*** raunak has joined #openstack-nova06:56
*** psachin has joined #openstack-nova06:58
*** thorst_ has joined #openstack-nova07:00
*** rcernin has joined #openstack-nova07:06
*** sridharg has joined #openstack-nova07:06
*** thorst_ has quit IRC07:06
*** lpetrut has quit IRC07:07
*** ducttape_ has joined #openstack-nova07:08
*** zhurong has joined #openstack-nova07:13
*** ducttape_ has quit IRC07:13
*** andreas_s has joined #openstack-nova07:15
*** tesseract has joined #openstack-nova07:16
*** kaisers has quit IRC07:16
openstackgerritZhenyu Zheng proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42797807:16
*** kaisers has joined #openstack-nova07:17
*** lpetrut has joined #openstack-nova07:29
*** Apoorva has joined #openstack-nova07:29
*** psachin has quit IRC07:30
markus_zmikal: Would you maybe take a look at this newton backport? https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/newton+topic:%22bug+1455252%2207:30
*** mptacekx has joined #openstack-nova07:30
markus_zmikal: ^ lyarwood already checked it.07:31
*** unicell1 has quit IRC07:36
*** Apoorva has quit IRC07:36
*** unicell has joined #openstack-nova07:36
*** pcaruana has joined #openstack-nova07:39
*** raunak has quit IRC07:40
*** pcaruana has quit IRC07:42
*** psachin has joined #openstack-nova07:42
*** pcaruana has joined #openstack-nova07:43
openstackgerritTakashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2)  https://review.openstack.org/40895507:43
*** nkrinner_afk is now known as nkrinner07:45
*** yamahata has joined #openstack-nova07:47
*** Hosam has quit IRC07:47
*** Hosam has joined #openstack-nova07:47
*** unicell has quit IRC07:49
*** efried has quit IRC07:51
*** Hosam has quit IRC07:52
*** efried has joined #openstack-nova07:55
*** karimb has joined #openstack-nova07:56
*** hfu has joined #openstack-nova08:06
*** lpetrut has quit IRC08:07
*** Sukhdev has quit IRC08:15
*** karthiks has quit IRC08:16
*** lpetrut has joined #openstack-nova08:18
*** ygl has joined #openstack-nova08:26
*** Hosam has joined #openstack-nova08:27
*** zhurong has quit IRC08:28
*** hfu has quit IRC08:28
*** rcernin has quit IRC08:28
*** ygl has quit IRC08:31
*** slaweq has joined #openstack-nova08:33
*** karthiks has joined #openstack-nova08:35
gibiQ~.08:35
gibisorry ^^08:35
gibimriedem: thanks for the link, I will check the cinder solution08:35
*** ducttape_ has joined #openstack-nova08:38
*** Hosam has quit IRC08:40
*** Hosam has joined #openstack-nova08:41
*** ducttape_ has quit IRC08:43
*** edmondsw has joined #openstack-nova08:43
*** hfu has joined #openstack-nova08:45
*** edmondsw has quit IRC08:48
openstackgerritTakashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2)  https://review.openstack.org/40896408:49
*** rcernin has joined #openstack-nova08:52
*** karimb has quit IRC08:53
*** lpetrut has quit IRC08:57
*** pbandark has quit IRC08:58
*** gema has joined #openstack-nova08:59
gemacan someone point me in the right direction regarding how to update this page: http://docs.openstack.org/developer/nova/feature_classification.html08:59
gemaare there any requirements in terms of testing that need to be met or is this manually maintained?08:59
*** zzzeek has quit IRC09:00
*** lpetrut has joined #openstack-nova09:01
*** zzzeek has joined #openstack-nova09:01
*** pbandark has joined #openstack-nova09:02
*** thorst_ has joined #openstack-nova09:02
BobBalljohnthetubaguy: OK - so the XenServer CI problem is that https://review.openstack.org/#/c/396269/ merged breaking support for the Ubuntu we're using.09:05
*** thorst_ has quit IRC09:08
*** karimb has joined #openstack-nova09:13
*** karthiks has quit IRC09:13
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Remove redundant bdm serial mangling and saving during swap_volume  https://review.openstack.org/42736409:15
openstackgerritTakashi NATSUME proposed openstack/nova master: Add functional tests for cold migration to same host  https://review.openstack.org/41492609:16
openstackgerritTakashi NATSUME proposed openstack/nova master: Add functional tests for cold migration to same host  https://review.openstack.org/41492609:16
*** lpetrut has quit IRC09:19
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action  https://review.openstack.org/41004209:19
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action  https://review.openstack.org/41004209:19
*** gszasz has joined #openstack-nova09:20
johnthetubaguyBobBall: really :S09:21
BobBallIndeed.09:22
johnthetubaguyBobBall: how did that break the uploads from inside the XenAPI plugin?09:23
BobBallIt broke Cinder which meant that the glance upload couldn't create the volume it was going to upload to09:23
johnthetubaguybut I didn't think the xenapi plugin is used for any cinder volumes?09:23
johnthetubaguyoh, it breaks all glance uploads in the destination sense?09:24
BobBalluhhhh ok now you've got me confused...09:24
*** lucas-afk is now known as lucasagomes09:25
*** gszasz has quit IRC09:27
*** ralonsoh has joined #openstack-nova09:27
*** karthiks has joined #openstack-nova09:27
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action  https://review.openstack.org/41004209:29
*** Jeffrey4l has quit IRC09:30
*** Jeffrey4l has joined #openstack-nova09:31
*** ssurana has quit IRC09:37
*** panda|off is now known as panda09:47
*** yamahata has quit IRC09:48
*** sambetts_ is now known as sambetts09:57
mdboothlyarwood: Thinking about https://review.openstack.org/#/c/427364/10:04
mdboothSpecifically about the chunk in libvirt driver which you delete that we discussed yesterday10:05
*** owalsh-afk has joined #openstack-nova10:05
*** owalsh-afk is now known as owalsh10:05
mdboothI think the fundamental issue there is that libvirt volume drivers mutate connection_info10:05
mdboothThe example which is specifically called out is LibvirtISCSIVolumeDriver. This adds the local device path that created to connect to the remote iscsi volume to connection_info.10:06
lyarwoodmdbooth: that's still added after my change10:07
mdboothBut this isn't strictly related to the volume connection.10:07
lyarwoodmdbooth: we pass the modified connection_info back to the compute layer and update the BDM with it10:07
mdboothlyarwood: Yeah, I'm being expansive here :)10:07
lyarwoodah10:07
lyarwood:D10:07
mdboothSo, anyway, I think the bug here is in LibvirtISCSIVolumeDriver and all other places which mutate connection_info10:08
openstackgerritTomi Juvonen proposed openstack/nova-specs master: Add Host Details  https://review.openstack.org/42807010:08
mdboothThey need to stash that metadata somewhere else.10:08
mdboothI also disapprove of even looking inside connection_info, tbh10:08
lyarwoodmdbooth: yeah that's all going to change with cinder v3 anyway10:09
*** ducttape_ has joined #openstack-nova10:09
mdboothIf we could have cinder pass it to us as an opaque packed C struct, that would be awesome10:09
mdboothThe only thing you can do with this data is give it back to cinder10:09
lyarwoodmdbooth: but I can't see how that relates to this issue tbh10:09
*** sree has quit IRC10:09
mdboothIt relates because it means we *can't* refetch connection info from cinder after cinder also mutates it10:09
*** sree has joined #openstack-nova10:10
mdboothBecause Nova futzes with it, we're now stuck trying to second guess one or the other.10:10
lyarwoodmdbooth: ah right you're talking about the cinder retype situation10:11
mdboothcinder volume migration10:11
lyarwoodmdbooth: yeah same thing10:11
* mdbooth has never worked out what retype means10:11
mdboothvolume migration is meaningful to me :)10:11
lyarwoodmdbooth: I need to drop now, can we pick this up when I get back?10:12
mdboothlyarwood: Sure thing.10:12
lyarwoodmdbooth: cool10:12
*** satyar_ has quit IRC10:12
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix access_ip_v4/6 filters params for servers filter  https://review.openstack.org/42807110:13
gmannjohnthetubaguy:  mriedem alex_xu  seems like we missed another bit in filter/sort thing - https://bugs.launchpad.net/nova/+bug/166119510:13
openstackLaunchpad bug 1661195 in OpenStack Compute (nova) "Servers filter by access_ip_v4 does not filter servers" [Undecided,In progress] - Assigned to Ghanshyam Mann (ghanshyammann)10:13
gmannjohnthetubaguy:  alex_xu  mriedem  this is fix - https://review.openstack.org/42807110:14
*** ducttape_ has quit IRC10:14
*** sree has quit IRC10:14
johnthetubaguygmann: oh, you mean we should allow them, because we used to allow them10:14
gmannbut different filed name in filter and sort than what API represent is odd to me10:15
gmannjohnthetubaguy:  i think so, or no ?10:15
johnthetubaguygmann: don't they now match what the API has?10:15
gmanni may be missed the discussion on that if any10:15
gmannjohnthetubaguy:  oh so you mean new list will match the what API has either it used to work before or not10:16
johnthetubaguygmann: the whole plan is the filter name matches what the API names are now, rather than matching the DB10:17
gmannjohnthetubaguy:  yea10:17
gmannjohnthetubaguy:  ok, i think m going in opposite way :)10:18
gmannjohnthetubaguy: i agree on that point10:18
gmannjohnthetubaguy:  but bug seems now accessIPv4 work on filter or not as DB fileds are different for those?10:18
gmannjohnthetubaguy: and sort has 'access_ip_v4' not accessIPv4 so it is issue on sort list then10:19
johnthetubaguygmann: so we must have accessipv4 and accessipv6 as thats what in the API, in both filter and sort10:19
gmannjohnthetubaguy:  yea and those should work as filter. because i doubt currently those works as we did not add translation layer for those10:20
johnthetubaguygmann: but... for backwards compatibility, we should probably also allow access_ip_v4 and access_ip_v6, and mark them for removal in a future microversion10:20
mdboothCan anybody confirm that schema migrations are intended to be idempotent? Specifically that I should be able to run db_sync multiple times?10:20
johnthetubaguygmann: yeah, we should add a test for both10:20
johnthetubaguymdbooth: thats correct10:20
gmannjohnthetubaguy:  yea, that was actually i was thinking initially10:21
gmannjohnthetubaguy: cool,10:21
gmannjohnthetubaguy: ll update10:21
johnthetubaguygmann: cool10:21
johnthetubaguygmann: we should give that the tag, and get it on mriedem's etherpad I think10:22
johnthetubaguyRC potential thing10:22
mdboothjohnthetubaguy: Thought so, thanks. I'm seeing an error in 216_havana creating the instances table,  specifically line 1046 where it calls instances.create() raises an error containing: (1050, u"Table 'instances' already exists")10:22
gmannjohnthetubaguy:  but last doubt, is it ok to introduced new param without version up. i mean accessIPv4 are kind of new addition as filter10:22
gmannjohnthetubaguy: sure, ll do on etherpad10:22
mdboothIt does already exist. Any idea how that's supposed to work?10:22
* mdbooth suspects a higher level failure, but isn't sure what to look for.10:23
*** udesale has quit IRC10:24
mdboothOr maybe it's a lower level failure... Perhaps sqlalchemy is supposed to transmute it into gold.10:24
*** hfu has quit IRC10:25
johnthetubaguymriedem: hmm thats odd, thats the our base one that everyone has run again on upgrade for years, its curious that has broken10:25
gmannjohnthetubaguy: i remember we did microversion when adding ipv6 as filter10:25
mdboothjohnthetubaguy: Yeah, suspect it's something weird in this environment.10:26
*** Hosam has quit IRC10:26
mdboothFile hasn't changed in ages.10:26
mdboothIt could be an interesting candidate for robustification, though.10:26
mdboothIf I can work out why it's broken.10:27
*** Hosam has joined #openstack-nova10:27
johnthetubaguygmann: yes, thats a good point, we also have these: https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/servers.py#L354 :S10:27
johnthetubaguymdbooth: generally we try never to change those files, but yeah10:27
johnthetubaguyflavors was the exception in there, I think10:27
gmannjohnthetubaguy:  i thought of adding accessIPv4 with version up fir interop things10:28
* mdbooth is suspecting sqla weirdness, tbh.10:28
johnthetubaguygmann: we haven't yet released the last fix, so I am OK with that for right now10:28
johnthetubaguymdbooth: maybe you died half way through the migration, and somehow that got committed into the DB? That might confuse things10:30
gmannjohnthetubaguy:  but accessIPv4 are always new even before filter/sort fix. i means they are totally new to users after Ocata10:31
*** Hosam has quit IRC10:31
johnthetubaguygmann: yeah, I am not sure why we added new ones without a microversion, now you meantion it, I got distracted with killing bad ones10:32
*** hfu has joined #openstack-nova10:33
gmannjohnthetubaguy:  yea, list is too huge we did not notice those10:33
*** owalsh is now known as owalsh-brb10:34
gmannjohnthetubaguy: i have added that in  etherpad also (so mriedem can mark that rc potential), ll check from home.10:35
johnthetubaguygmann: thanks10:36
johnthetubaguygmann: it was such a massive list, there were always going to be niggles, would love to squash then before release, if possible10:36
gmann+110:36
*** owalsh-brb is now known as owalsh10:41
*** erlon has joined #openstack-nova10:45
sfinucanbauzas: Can I +W this? https://review.openstack.org/#/c/426864/610:48
sfinucanI assume it won't do anything silly like tag the release or whatnot10:48
*** rfolco has joined #openstack-nova10:51
johnthetubaguysfinucan: that sounds fine to me, we have added lots of preludes by accident in the past10:51
sfinucanjohnthetubaguy: Righto - +W it is10:51
* sfinucan just noticed there's a unapproved dependency anyway so it won't merge yet10:52
johnthetubaguyoh year, I should review that one again10:52
bauzassfinucan: cool with me10:52
bauzasjohnthetubaguy: I'm already on it10:52
*** derekh has joined #openstack-nova10:52
sfinucanbauzas: Could you also take a look at https://review.openstack.org/#/c/427499/, seeing as you're far more familiar with that feature10:53
bauzassfinucan: mmm, I had a concern on the bottom patch10:55
bauzasat least a typo10:55
sfinucanbauzas: Hmm, if you want to fix it now we can just re-approve, right?10:56
sfinucanAlso, do you know what "barber pole sorting" is? :D10:56
*** tbachman has joined #openstack-nova10:56
*** efried has quit IRC10:57
sfinucanI'm guessing it's Python's 'zip', but that's a guess10:57
bauzassfinucan: https://en.wikipedia.org/wiki/Barber's_pole#Computer_science10:58
sfinucanbauzas: I don't think that's what he's talking about though10:58
sfinucanThat describes a loading bar - he's on about sorting something10:59
bauzassfinucan: that would mean it takes time IIUC10:59
bauzassfinucan: oh, you removed +W10:59
bauzassfinucan: but that doesn't mean the gate jobs will stop11:00
bauzassfinucan: we have to trigger a new PS11:00
sfinucanbauzas: it doesn't?11:00
sfinucanOh11:00
bauzasnope11:00
sfinucanWell, should we fix that typo?11:00
bauzaswe could do that in a follow-up11:00
bauzasnow it's in the gate11:00
sfinucanThat sounds fair11:00
bauzasand given we have 3 patches that could be trampled if we provide a new rev11:00
* sfinucan is disgusted that he missed a typo :(11:01
bauzasso, adding the fix at a top patch11:01
*** lpetrut has joined #openstack-nova11:01
mdboothjohnthetubaguy: In case you're interested, looks like the db state was probably corrupt, perhaps due to a previous incomplete cleanup. This was in a CI system. So robustification likely not relevant to anybody.11:01
bauzassfinucan: restored the +W11:02
sfinucan(y)11:02
*** pbandark has quit IRC11:03
*** shaner has quit IRC11:03
*** thorst_ has joined #openstack-nova11:04
*** shaner has joined #openstack-nova11:04
*** aarefiev_afk is now known as aarefiev11:08
*** thorst_ has quit IRC11:08
sfinucanAre we still merging bugfixes? If so, could I get some eyes on this (to be backported) change? https://review.openstack.org/#/c/396184/11:09
*** efried has joined #openstack-nova11:09
openstackgerritStephen Finucane proposed openstack/nova master: Fix backwards compatibility for InstanceNUMACell  https://review.openstack.org/39618411:09
*** ducttape_ has joined #openstack-nova11:10
*** pbandark has joined #openstack-nova11:12
johnthetubaguysfinucan: I think so, although we should avoid high risk ones till tomorrow11:13
johnthetubaguysfinucan: we usually try to keep master not too different to stable/ocata so any backports we need are likely to be easier11:13
*** ducttape_ has quit IRC11:14
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Make 'get_domain' private  https://review.openstack.org/41737811:15
*** ociuhandu has quit IRC11:17
johnthetubaguymdbooth: ah, good context, in complete cleanup would do it11:17
*** nicolasbock has joined #openstack-nova11:18
sfinucanGerrit's "Add Reviewer" dialog is truly awful11:18
*** sudipto_ has quit IRC11:19
*** sudipto has quit IRC11:19
openstackgerritJohn Garbutt proposed openstack/nova-specs master: WIP: write up policy ideas for pike  https://review.openstack.org/42787211:22
*** ducttape_ has joined #openstack-nova11:30
*** aloga has quit IRC11:35
*** aloga has joined #openstack-nova11:36
openstackgerritJohn Garbutt proposed openstack/nova-specs master: WIP: write up policy ideas for pike  https://review.openstack.org/42787211:37
*** tbachman has quit IRC11:39
pandaany hints on what may cause this ? http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/632c7f2/logs/overcloud-novacompute-0/var/log/nova/nova-compute.txt.gz#_2017-02-02_08_13_29_75911:40
*** psachin has quit IRC11:43
*** sree has joined #openstack-nova11:43
kashyapmdbooth: or johnthetubaguy: When you have a moment, need an opinon11:45
kashyapWhere do you think is appropriate for these kinds of tests to live? https://review.openstack.org/#/c/315786/11:46
kashyapThey are trying to add the NMI Watchdog test for a Nova instance in Tempest.  But Tempest upstream, rightfully rejects it, as there are no API operations, and it is too Nova-specific11:46
johnthetubaguykashyap: sounds like we need a nova tempest plugin11:47
johnthetubaguyI would say nova functional tests, but they don't normally need images setup up, AFAIK, I could be wrong11:47
johnthetubaguykashyap: you really want to be talking to sdague about all that11:47
kashyapjohnthetubaguy: Does such a thing exists (or is it exist)?11:47
*** catintheroof has joined #openstack-nova11:47
kashyapjohnthetubaguy: Yeah, maybe sdague is still getting ready to wake up11:48
johnthetubaguykashyap: there are some details here: http://docs.openstack.org/developer/nova/test_strategy.html#types-of-tests11:48
*** amoralej has joined #openstack-nova11:48
johnthetubaguybut looks like it might need updating11:48
* kashyap clicks11:48
kashyapjohnthetubaguy: So, one place you are suggesting is - nova/nova/tests/functional11:48
*** ftersin has joined #openstack-nova11:52
mdboothjohnthetubaguy: More context on the db_sync thing, looks like we had a puppet job which retried on failure, which included exceeding its timeout. Seems db sync was taking too long and we ended up running 2 simultaneously.11:53
mdboothSo perhaps robustification might be in order.11:53
mdboothPerhaps the migration should hold an exclusive lock while it runs.11:53
johnthetubaguymdbooth: it should be all done in a single transaction, for each version, I thought?11:53
johnthetubaguymaybe we never added that stuff in db sync11:54
johnthetubaguymaybe thats not possible...11:54
mdboothDoes msql honour transactions for ddl?11:54
johnthetubaguyI was just thinking the same question, I don't know11:55
johnthetubaguyI had always assume that, but you might be right11:55
johnthetubaguynothing like an olso.concurrency lock to stop that being an issue, like you say11:55
*** psachin has joined #openstack-nova11:55
mdboothInterwebs say no transactional ddl11:55
mdboothThink it would have to be a db lock, because we can't be sure where db sync is running.11:56
mdboothAlso, think change would have to be in sqla migrations11:57
*** ducttape_ has quit IRC11:57
*** bkopilov_ has quit IRC11:57
*** bkopilov has quit IRC11:58
johnthetubaguymdbooth: true, DB lock is safer11:58
*** archies has quit IRC11:59
*** ducttape_ has joined #openstack-nova12:00
*** prateek has quit IRC12:00
*** Hosam has joined #openstack-nova12:01
*** ratailor has quit IRC12:02
*** openstackgerrit has quit IRC12:03
*** edmondsw has joined #openstack-nova12:04
*** hfu has quit IRC12:09
*** hfu has joined #openstack-nova12:09
*** hfu has quit IRC12:09
*** sdague has joined #openstack-nova12:09
*** hfu has joined #openstack-nova12:10
owalshmdbooth: DB lock might be overkill...12:10
*** hfu has quit IRC12:10
*** jpena is now known as jpena|lunch12:10
*** karimb has quit IRC12:10
owalshthe first db sync succeeded IIRC12:11
mdboothowalsh: I don't think so. It would only have to used for exclusion of the migration itself.12:11
mdboothI think it could be pretty simple to implement, tbh.12:11
*** hfu has joined #openstack-nova12:11
*** openstackgerrit has joined #openstack-nova12:12
openstackgerritDerek Higgins proposed openstack/nova master: Simplify uses of assert_has_calls  https://review.openstack.org/42810812:12
*** hfu has quit IRC12:12
derekhjohnthetubaguy: sfinucan ^^ sorry forgot to push it up yesterday12:12
mdboothWhen you initially read the current version from the migrate_version table you'd take an exclusive lock on it.12:12
*** hfu has joined #openstack-nova12:12
*** hfu has quit IRC12:12
mdboothThen you'd exclude any other migration from starting until this once completed, but other db operations can continue.12:12
*** clayton has joined #openstack-nova12:13
sfinucanderekh: Sweet. Cheers :) Think it can/should wait until Ocata is cut though12:13
derekhsfinucan: sounds fine by me, just letting ye know its there12:13
*** dave-mccowan has joined #openstack-nova12:14
*** Hosam has quit IRC12:16
*** bvanhav has joined #openstack-nova12:18
*** panda is now known as panda|lch12:18
owalshmdbooth: hmm the oslo.db tests used to lock - https://review.openstack.org/5883012:20
mdboothowalsh: A good reason to use a db lock12:20
*** ociuhandu has joined #openstack-nova12:21
owalshbut why protect the tests with a lock instead of the migration?12:21
* owalsh wonders if there was a good reason for this12:21
mdboothowalsh: I think every test runs the migrations.12:21
*** Hosam has joined #openstack-nova12:21
mdboothUnless... optimisation.12:22
mdboothBut mostly.12:22
*** ducttape_ has quit IRC12:22
owalshforget that, (should have looked at the diff), it's just locking the engine reset12:26
mdboothowalsh: I've emailed zzzeek for an opinion. cc'd you.12:27
*** READ10 has quit IRC12:31
*** thorst_ has joined #openstack-nova12:33
*** vsaienko has joined #openstack-nova12:34
*** thorst_ has quit IRC12:34
*** ayogi has quit IRC12:40
*** ducttape_ has joined #openstack-nova12:42
lyarwoodhopefully a stupid question, does the gate rebase changes? I keep hitting Bug 1660878 in https://review.openstack.org/#/c/427267/ and wonder if I need to manually rebase it to pick up https://review.openstack.org/#/c/427782/12:42
openstackbug 1660878 in OpenStack Compute (nova) "test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"" [High,In progress] https://launchpad.net/bugs/1660878 - Assigned to Matt Riedemann (mriedem)12:42
*** sree has quit IRC12:46
lyarwoodhttp://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e4e67b3/logs/devstacklog.txt.gz#_2017-02-02_10_50_57_699 - okay, so we merge the change, so I should have https://review.openstack.org/#/c/427782/12:46
*** takashin has left #openstack-nova12:47
*** jpena|lunch is now known as jpena12:50
*** satyar has joined #openstack-nova12:51
*** ducttape_ has quit IRC12:53
openstackgerritJohn Garbutt proposed openstack/nova master: Report reserved_host_disk_mb in GB not KB  https://review.openstack.org/42812012:54
*** vsaienko has quit IRC12:55
*** mvk has quit IRC12:55
johnthetubaguybauzas: sfinucan: a nasty placement issue the openstack ansible guys were hitting ^12:55
johnthetubaguyjust want to make sure I am not reading that incorrectly12:55
* johnthetubaguy goes for some lunch12:56
*** panda|lch is now known as panda12:57
*** tbachman has joined #openstack-nova12:58
*** gcb has joined #openstack-nova13:00
*** thorst_ has joined #openstack-nova13:01
*** ducttape_ has joined #openstack-nova13:02
mdboothlyarwood: Yep13:04
lyarwoodmdbooth: thanks, I'll wait for dansmith and mriedem to come online before I recheck https://review.openstack.org/#/c/427267/ again13:05
mdboothjohnthetubaguy: Do you think you'll get around to https://review.openstack.org/#/c/383859/ before release? Fixes NFS, and makes doing the same for other filesystem-backed drivers trivial. See glusterfs follow-on for eg: https://review.openstack.org/#/c/423402/13:08
gcbsfinucan, mriedem:  As I know we have been preparing for final octata release, Can we still merge python 3 unit tests patches now ?13:08
mdboothjohnthetubaguy: So the NFS change is somewhat complex because it includes infrastructure applicable to all filesystem-backed volume backends.13:09
sfinucangcb: They're pretty low risk (unit test only) but I think we're only accepting critical bugfixes and doc patches at the moment13:09
mdboothIt's pretty well tested, though.13:09
sfinucanThe Pike window will be open tomorrow though13:09
mdboothIt's a bug. Are we passed the bugfix window?13:10
mdboothpast13:10
mdboothThat said, if I can get it into Pike I'll take it and propose for backport.13:10
mdboothI just want to land the damned thing.13:10
*** karimb has joined #openstack-nova13:10
gcbsfinucan: thanks, got it. The only two patches to make all unit tests work on Python 3.5  https://review.openstack.org/#/q/topic:bug/148263313:11
gcbit's okay to make them merge into Pike :)13:11
sfinucangcb: Perfect. I'll take a look tomorrow or Monday :) Will be good to get that all closed off13:11
*** gouthamr has joined #openstack-nova13:11
gcbsfinucan: cool, thanks13:12
*** Hosam has quit IRC13:15
sfinucanjohnthetubaguy: Took a gawk. Makes sense to me13:18
*** esberglu has joined #openstack-nova13:21
*** jheroux has joined #openstack-nova13:22
*** sree has joined #openstack-nova13:22
jaypipesmorning folks13:24
bauzasmriedem: FWIW, we have another related problem13:24
bauzasjohnthetubaguy: https://bugs.launchpad.net/nova/+bug/165926613:24
openstackLaunchpad bug 1659266 in OpenStack Compute (nova) "Disk allocation for instance is not good with swap" [High,Confirmed] - Assigned to Tanvir Talukder (tanvirt16)13:24
bauzasbut fixing that could be difficult given we already have allocations13:25
sfinucantrinaths: Not sure if you got a response to your question or not. Have you tried building an instance with a NUMA topology but without hugepages?13:25
* sfinucan is reading through scrollback to find patches to review13:25
mriedemgcb: sfinucan: mdbooth: we shouldn't be putting things into the gate or check queue that don't need to be, py3 unit test cleanup included13:28
sfinucanmriedem: Morning. That's what we thought13:28
gcbmriedem: ack13:28
*** eharney has quit IRC13:29
*** mvk has joined #openstack-nova13:29
*** trinaths has quit IRC13:30
mdboothlyarwood: I'm just going to grab coffee. Want to talk about swap_volume in a few minutes?13:31
lyarwoodmdbooth: sure13:31
mriedembauzas: is that related to the thing that johnthetubaguy has a patch up for?13:31
*** cdent has joined #openstack-nova13:32
mriedemhttps://review.openstack.org/#/c/428120/13:32
sfinucanmriedem, bauzas: That's the one, yes13:32
bauzasmriedem: not really but the same problme13:33
bauzasmriedem: honestly, MHO is that we should not use GBs for disk resources but rather bytes13:33
bauzasor KB13:33
bauzasif that's too big13:33
bauzasbut, we have lots of ceilings like the one I wrote13:33
mriedemdansmith: melwitt: hmm, lyarwood's patch failed in the gate on the cells inst delete/create race thing http://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e4e67b3/console.html#_2017-02-02_11_48_07_75524313:34
bauzasmriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L14813:34
mriedemhttps://review.openstack.org/#/c/427782/13:34
mriedemwhich merged at 7:34pm13:35
mriedemand lyarwood's change failed at 2017-02-02 11:48:07.75524313:35
lyarwoodmriedem: twice, I think the previous run also hit this a few hours before that13:35
* lyarwood gets the link13:35
mriedemlyarwood: well i could see it hitting before the other fix was merged,13:36
mriedembut i'm worried if it hit after the fix was merged13:36
mriedemi think ES is backed up too so we probably can't tell if the hits have dropped off after the fix merged13:36
mriedemyeah we can't tell http://status.openstack.org//elastic-recheck/index.html13:36
mriedemeverything is backed up13:36
lyarwoodmriedem: http://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cc0ed47/console.html#_2017-02-02_05_45_00_341186 was the other hit13:37
*** Hosam has joined #openstack-nova13:38
*** diga has quit IRC13:38
*** nmathew- has joined #openstack-nova13:38
mriedemyeah that's before the fix was merged13:39
mriedemso not a surprise13:39
mriedembut failing 3 hours after the merge worries me13:39
lyarwoodmriedem: ah did it land at 7:34am?13:39
mriedempm13:39
lyarwoodmriedem: you said pm above13:39
lyarwoodmriedem: that's 05:45am13:39
mriedemoh yeah13:40
lyarwoodanyway13:40
nmathew-hi all13:41
nmathew-can anybody point to a material on flavors13:41
mriedemlyarwood: apparently the times in the zuul log don't match up with what's in the gerrit ui13:41
mriedemnmathew-: http://docs.openstack.org/admin-guide/compute-flavors.html13:41
mriedemdocs.o.o has a search box13:42
nmathew-mriedem, say I not enabling hugepages in flavors, I can stil enable that by kernel parameters to that vm kernel13:44
nmathew-mriedem, what is the difference between specifying in flavors and not specifying in flavors?13:44
lyarwoodnmathew-: see the topic, if you have questions like that #openstack is a better place to ask :)13:44
nmathew-ok13:45
nmathew-oh sorry13:45
lyarwoodnmathew-: np at all13:45
mriedemso looking at my fix for the delete/create race bug,13:46
mriedemhttps://review.openstack.org/#/c/427775/1/nova/conductor/manager.py13:46
mriedemi see that debug log message hit 3 times in the n-cond logs on a run of that patch http://logs.openstack.org/75/427775/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/c906c15/logs/screen-n-cond.txt.gz#_2017-02-01_16_14_03_27313:46
mriedemso it's obviously catching the issue13:46
mriedemdansmith: bauzas: ^ so i'm thinking we should probably just get that in given the time13:46
*** annegentle has joined #openstack-nova13:47
*** zhurong has joined #openstack-nova13:48
*** owalsh is now known as owalsh-brb13:51
*** satyar has quit IRC13:51
*** pbandark has quit IRC13:52
cdentmriedem: unless you're already on it, I'll fix the nits in https://review.openstack.org/#/c/42736713:53
mriedemi'll get it quick13:53
*** burt has joined #openstack-nova13:54
openstackgerritMatt Riedemann proposed openstack/nova master: doc: add upgrade notes to the placement devref  https://review.openstack.org/42736713:55
mriedemsfinucan: bauzas: ^ fixed the typo13:55
*** psachin has quit IRC13:55
*** sree has quit IRC13:55
*** sree has joined #openstack-nova13:56
bauzasmriedem: okay, let's merge https://review.openstack.org/#/c/427775/1 then13:56
*** sree has quit IRC13:56
*** crushil has quit IRC13:56
*** ducttape_ has quit IRC13:57
sfinucanmriedem: Job. Done.13:57
mriedemi wonder if it would be better to destroy the build request right before creating the instance?13:57
mriedembut, i don't really want to ponder that today13:57
bauzasmriedem: possibly13:58
bauzasmriedem: the main issue I see here is timing13:58
bauzasso let's merge your change is quite impactless and think about the best approach for Pike13:58
johnthetubaguymriedem: FWIW, I think its best to create the instance first, so we don't "loose" the instance, although marking the build request in someway that stops deletes might make sense, maybe thats the cell mapping entry?14:00
johnthetubaguyhmm, I can't 100% remember the ordering now14:01
johnthetubaguymriedem: this one the openstack ansible folks I spotted feels nasty, but we totally could back port it later, I believe14:02
*** eharney has joined #openstack-nova14:02
mriedemjohnthetubaguy: https://review.openstack.org/#/c/428120/ right?14:02
mriedemi left some comments in there14:02
mriedemthere is a bug report i think that fits the same thing already14:03
mriedemi've marked it rc114:03
*** owalsh-brb is now known as owalsh14:03
*** annegentle has quit IRC14:04
*** amoralej is now known as amoralej|lunch14:05
*** bvanhav_ has joined #openstack-nova14:05
*** gcb has quit IRC14:05
*** bvanhav has quit IRC14:05
alex_xujohnthetubaguy: I think gmann is right on https://review.openstack.org/#/c/428071, we agreed on we have filter which match the name in the API representation, but that will be done in the future with microversion.14:06
*** zhurong has quit IRC14:06
*** eharney has quit IRC14:07
*** mdrabe has joined #openstack-nova14:08
*** Hosam has quit IRC14:10
*** Hosam has joined #openstack-nova14:10
johnthetubaguymriedem: the bug is different I think, the bug talks about the allocation, this my bug is about the inventory (due to the reservation)14:11
jrollmriedem: fyi this one looks fun https://bugs.launchpad.net/nova/+bug/166125814:12
openstackLaunchpad bug 1661258 in OpenStack Compute (nova) "Deleted ironic node has an inventory in nova_api database" [Undecided,New]14:12
mriedemjroll: damn14:13
jrollit might actually be easy14:13
mriedemjroll: will need cdent, edleafe and/or jaypipes to look at that one14:13
jrollmriedem: thank you for pinging them :P14:14
mriedemi was thinking the periodic update would remove the inventory for the removed node but vdrok said in the bug that it didn't14:14
cdentjoy14:14
*** Hosam has quit IRC14:14
*** sree has joined #openstack-nova14:15
*** READ10 has joined #openstack-nova14:15
edleafemriedem: I can take a look at that14:15
jrollright, I assume we're missing something like this for inventory https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L657714:15
*** ducttape_ has joined #openstack-nova14:16
*** zhurong has joined #openstack-nova14:17
mriedemjroll: yeah that looks reasonable14:18
mriedemjaypipes refactored a ton of that code so he'll know best14:19
*** crushil has joined #openstack-nova14:19
jaypipesjroll, vdrok, mriedem, edleafe, cdent: I have a suspicion on cause of that bug.14:19
*** eharney has joined #openstack-nova14:19
*** bkopilov_ has joined #openstack-nova14:19
edleafejaypipes: do tell14:19
*** bkopilov has joined #openstack-nova14:19
*** tbachman has quit IRC14:20
*** tbachman_ has joined #openstack-nova14:20
*** nmathew has joined #openstack-nova14:20
jaypipesjroll, vdrok, mriedem, edleafe, cdent: Ironic virt driver is probably returning False for node_is_available() and therefore when update_available_resource() runs, we never even get to the call for InstanceList.get_by_host_and_node() and therefore never clean up the now-deleted instance.14:20
mriedemsdague: how do you feel about this? https://review.openstack.org/#/c/426926/14:20
jaypipesmriedem: remember when you asked me to change RT.disabled(nodename) to account for the Ironic-only node_is_available() call? :)14:20
mriedemsdague: because i'd like to get the dependent nova change in for ocata14:20
mriedemjaypipes: vaguely14:21
*** nmathew- has quit IRC14:21
mriedemjaypipes: i'm pretty sure i was drunk reviewing most of that refactor series :)14:21
jrolljaypipes: this is about deleted nodes, not deleted instances, to be clear14:21
jrolljaypipes: so it wouldn't even be returned in get_available_nodes()14:22
jaypipesvdrok: I asked for you to do a flash-patch in the bug above.14:22
cdentis running two nova-computes on the same machine normal and/or an important consideration in this bug?14:22
jaypipesjroll: right, but the inventory won't get cleaned up in the placement DB if the node_is_available() returns False here:14:23
vdrokjaypipes: ok, thanks, will do after lunch14:23
jaypipeshttps://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L41414:23
*** esberglu has quit IRC14:23
jaypipescdent: no.14:23
sdaguemriedem: yeh, that seems fine.14:23
sdaguemriedem: under what circumstances does that error out?14:23
jrolljaypipes: yeah, it'll definitely return False there if the node is deleted14:24
mriedemsdague: would have to dig into nova-status code for specifics, but things like not being able to talk to the placement api14:24
*** Jeffrey4l has quit IRC14:24
sdaguemriedem: yeh, seems fine14:24
mriedemno cell mappings, no cell0, no host mappings when there are computes, etc14:25
mriedemthanks14:25
edleafejroll: So if the node is deleted, the problem is just orphaned invntory, right? IOW, it isn't in the DB anymore, so it won't get returned from a placement GET14:26
jrolledleafe: AIUI it *is* being returned from the placement API14:26
*** psachin has joined #openstack-nova14:26
mriedemthe error in the bug report was missing some details, like what the actual build failure was14:27
*** amoralej|lunch is now known as amoralej14:27
jrollmriedem: fault | {"message": "Node 6cc8803d-4e77-4948-b653-663d8d5e52b7 could not be found. (HTTP 404)", "code": 500, "details": " File \"/opt/stack/nova/.......14:27
*** ducttape_ has quit IRC14:27
*** mlavalle has joined #openstack-nova14:27
jrollright at the top14:27
mriedemyeah, but14:27
mriedemthat doesn't really have anything to do with placement...14:28
mriedemit's in _build_and_run_instance14:28
*** satyar has joined #openstack-nova14:28
jrollwell14:28
jrollit selected a node that doesn't exist in ironic14:28
jrollit also doesn't exist in the compute_nodes table, but does exist in inventory14:29
mriedemunless it's blowing up here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L190514:29
*** udesale has joined #openstack-nova14:29
*** liverpooler has joined #openstack-nova14:30
*** nmathew- has joined #openstack-nova14:30
edleafejroll: so it's gone from the compute_nodes table in the cell, but not in the resource_providers table in the API/placement db14:30
edleafejroll: So placement still thinks it's there14:30
jrolledleafe: AIUI, yes14:30
*** liverpooler has quit IRC14:30
jrollkeep in mind I did not file this bug14:30
edleafejroll: ack14:30
jrollI have not independently reproduced it14:30
*** liverpooler has joined #openstack-nova14:30
jrollI'm just trying to answer question while vdrok is at lunch :)14:31
vdrokjaypipes: so, I tried that, return (nodename not in self.compute_nodes) only in the disabled, restarted both computes, still the same14:31
jrollmriedem: nah, I think all the placement/allocations stuff is working fine, except that the node being selected doesn't exist, so spawn() explodes14:31
*** prateek has joined #openstack-nova14:32
mriedemwhat is selecting the node? the scheduler?14:32
jrollI assume so14:33
jrollthat's what selects things, right14:33
edleafejroll: jaypipes: so maybe we need to also add a call to the placement API after https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L324 to delete the compute node as a resource provider?14:33
vdrokmriedem: I added a bit more log from the n-cpu14:33
mriedemi guess if everything is fully master then the filter scheduler will use placement14:33
*** sdague has quit IRC14:34
*** nmathew has quit IRC14:34
*** baoli has joined #openstack-nova14:34
jrollyeah, looks recent enough for placement api, that's why I bring it up :)14:34
*** sandanar has joined #openstack-nova14:35
jrolledleafe: that seems sensible to my feeble brain14:35
mriedemedleafe: then the compute node object has to use the scheduler client...14:35
vdroknow going to lunch for real :)14:35
mriedemi'd rather we just call the scheduler client to delete the RP after deleting the CN here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L658514:35
mriedemsince the compute manager already has it's grubby hands into that code14:35
*** Jeffrey4l has joined #openstack-nova14:36
edleafemriedem: yeah, makes sense14:36
*** psachin has quit IRC14:37
*** zzzeek has quit IRC14:40
*** zzzeek has joined #openstack-nova14:40
mriedemas far as i can tell, we only ever delete inventories or allocations, but never the actual resource provider14:40
mriedemand since the CN is the RP, and we delete the CN, i guess we need to also delete the RP14:40
*** jamesdenton has joined #openstack-nova14:41
mriedemi'm going to head into the office, bbiab14:41
*** mriedem has quit IRC14:41
cdentyes14:41
cdent(the delete the rp)14:42
*** Jeffrey4l has quit IRC14:42
*** tblakes has joined #openstack-nova14:43
*** crushil has quit IRC14:43
*** crushil has joined #openstack-nova14:44
*** Jeffrey4l has joined #openstack-nova14:44
*** m1dev has joined #openstack-nova14:44
*** esberglu has joined #openstack-nova14:45
*** nmathew- has quit IRC14:46
*** nmathew has joined #openstack-nova14:46
*** krtaylor has quit IRC14:48
*** nmathew has quit IRC14:52
*** hongbin has joined #openstack-nova14:52
mdboothlyarwood: I've reviewed the swap_volume change. Haven't looked at the tests yet, btw. Doing that now.14:56
*** m1dev has quit IRC14:58
lyarwoodmdbooth: thanks again, working on it now.14:59
*** udesale has quit IRC15:00
*** prateek has quit IRC15:02
*** sudipto_ has joined #openstack-nova15:03
*** sudipto has joined #openstack-nova15:03
*** sree_ has joined #openstack-nova15:04
*** nmathew has joined #openstack-nova15:04
*** sree_ is now known as Guest1558615:04
*** baoli has quit IRC15:05
*** sree has quit IRC15:06
*** karts has joined #openstack-nova15:06
*** jamesden_ has joined #openstack-nova15:06
*** mriedem has joined #openstack-nova15:07
*** sridharg has quit IRC15:07
*** karthiks has quit IRC15:07
mriedemplease all be fixed, please all be fixed...15:07
mriedemno whammies15:07
*** jamesdenton has quit IRC15:07
lyarwoodmriedem: my fix is failing in check, looking now15:07
*** jamesden_ has quit IRC15:08
*** zz_dimtruck is now known as dimtruck15:09
*** zhurong has quit IRC15:09
*** jamesdenton has joined #openstack-nova15:10
*** dikonoor has quit IRC15:10
*** karts has quit IRC15:10
lyarwoodmriedem: http://logs.openstack.org/67/427267/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0f048f1/console.html#_2017-02-02_15_03_33_671655 - sorry if I've missed a previous discussion about this still being broken btw15:10
*** sridhargaddam has joined #openstack-nova15:11
mriedemlyarwood: another fix for that is in the gate15:12
lyarwoodmriedem: ah great, sorry I missed that fly past15:13
*** mtanino has joined #openstack-nova15:13
mriedemlyarwood: this one https://review.openstack.org/#/c/427775/15:13
mriedem2 in the gate15:13
*** ducttape_ has joined #openstack-nova15:14
* lyarwood fires up openstack-gerrit-dashboard/dash.py to watch15:14
*** johndperkins has quit IRC15:17
*** baoli has joined #openstack-nova15:17
mriedemdo we have a definition of barber-pole sorting anywhere?15:17
*** dave-mccowan has quit IRC15:18
*** mptacekx has quit IRC15:18
mriedemjust means it wraps around from cell to cell right?15:18
*** Guest15586 has quit IRC15:19
mriedem"striped across the cell boundaries"15:19
openstackgerritMatt Riedemann proposed openstack/nova master: Fix typo in cells v2 ocata reno  https://review.openstack.org/42819815:19
*** sdague has joined #openstack-nova15:21
*** sridhargaddam has quit IRC15:21
*** sridharg has joined #openstack-nova15:21
jaypipesedleafe: when (if ever) do compute_nodes table records get deleted?15:22
*** karts has joined #openstack-nova15:22
*** kfarr has joined #openstack-nova15:22
jaypipesmriedem: more generally, I wonder when we ever delete compute nodes...15:24
*** tbachman_ is now known as tbachman15:24
mriedemin the case of ironic, i guess when they stop reporting available nodes15:24
mriedemjaypipes: also when the nova-compute service is deleted via the api i think15:25
mriedemwill have to look15:25
openstackgerritBalazs Gibizer proposed openstack/nova master: Improve assertJsonEqual error reporting  https://review.openstack.org/42819915:25
*** baoli has quit IRC15:25
mriedemi remember fixinga 404 in the api around that recently15:25
jaypipesmriedem: that would be a big mistake in the case of Ironic then... (since service != compute_node for Ironic)15:25
mriedemhttps://review.openstack.org/#/c/406627/ was the 404 i was thinking of15:26
mriedem"Compute node resources must currently be deleted manually in the database"15:26
*** baoli has joined #openstack-nova15:27
edleafejaypipes: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L658515:27
mriedembut we do have this https://github.com/openstack/nova/blob/13.1.2/nova/compute/manager.py#L651315:27
mriedemyeah15:27
edleafejaypipes: which in turn calls https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L32415:27
jaypipesmriedem: yes, and I had some strong words on that patch :)15:27
diana_clarkesfinucan: I heard you like docs ;) https://review.openstack.org/#/c/411441/15:27
diana_clarkesfinucan: (but only if you're not otherwise busy)15:28
jaypipesedleafe: doink.15:28
mriedemhttp://www.thereluctanttecchie.com/openstack-removing-a-compute-node-in-icehouse/15:28
jaypipesedleafe: right on, brother, nice detective work.15:28
jaypipesedleafe: we need to have the resource tracker do that, not the compute manager.15:28
jaypipesedleafe: I can work on a patch.15:28
edleafejaypipes: already working on it15:29
jaypipesedleafe: nice find.15:29
mriedem" The fact that the crap hypervisor-list/hypervisor-show REST APIs are  still manually attempting to conflate a ComputeNode with a Service is a  problem that should be fixed entirely in the REST API, since it is the  REST API that is borked."15:29
jaypipesedleafe: oh? even better :)15:29
mriedemthose strong words? :)15:29
jaypipesmriedem: heh, yes sir.15:29
*** sridharg has quit IRC15:29
edleafejaypipes: but it's the manager that has to make the call. I don't see the benefit in adding the RT as an indirection layer15:29
*** sree has joined #openstack-nova15:29
jaypipesedleafe: because the RT has the placement client, not the manager.15:29
jaypipesedleafe: and the RT has all the compute node objects now, not the manager.15:30
mriedemthe compute manager is the one looking for orphaned computes15:30
mriedemnot the RT15:30
edleafejaypipes: the manager has the scheduler client15:30
*** prateek has joined #openstack-nova15:30
*** prateek has quit IRC15:30
*** sdague has quit IRC15:30
mriedemunless you just move that into RT.update_available_resource15:30
jaypipesmriedem: all of that update_available_resource() code in the manager should be in the RT, IMHO.15:30
sfinucandiana_clarke: ack15:31
edleafejaypipes: we can do scheduler_client.report_client like we added to RT15:31
*** gabor_antal_ has joined #openstack-nova15:32
*** gabor_antal has quit IRC15:32
jaypipesmriedem: jinx.15:32
mriedemjaypipes: well,15:32
mriedemit's a chicken/egg,15:32
mriedembecause we only call rt.update_available_resource(context, nodename)15:32
mriedemfor an available node15:32
mriedemand in this case, it's a not available node15:32
mriedemso orphaned and we delete the CN record15:32
mriedemwe could build a new method into the RT to remove it if that's cleaner15:32
edleafemriedem: that sounds clunky to me15:32
edleafethere's nothing magic about the client15:32
*** artom_ has quit IRC15:32
mriedemeither way, i'm totally fine with a quick rt.report_client.delete_resource_provider from the compute manager in ocata if it gets the job done and saves time15:32
*** edmondsw_ has joined #openstack-nova15:32
jaypipesedleafe, mriedem: what I'm saying is that all the code in the compute manager's update_available_resource() method should be moved to the RT now. There's no reason to have that in the compute manager.15:32
mriedemand a TODO to move that into RT later15:32
*** cleong has joined #openstack-nova15:32
*** artom_ has joined #openstack-nova15:32
mriedemjaypipes: *now* is not the day to do it15:32
jaypipesmriedem: sure, I'm also fine with a quick fix.15:32
jaypipesmriedem: understood.15:32
edleafemriedem: gimme a few to get something done15:32
jaypipesedleafe: cheers15:33
edleafeall this IRC chat is slowing me down :)15:33
*** sdague has joined #openstack-nova15:33
*** prateek has joined #openstack-nova15:33
jaypipesmriedem: in that future cleanup, I will also change the InstanceList.get_all_by_host_and_nodename() to be an InstanceList.get_all_by_host() and fix the performance issues with the Ironic RT once and for all.15:33
*** breitz has quit IRC15:33
mriedemyeah i guess compute manager has self.scheduler_client = scheduler_client.SchedulerClient()15:33
jaypipesmriedem: that was the end goal of moving all the compute_node -> compute_nodes handling in the RT.15:34
*** dave-mccowan has joined #openstack-nova15:34
mriedemjaypipes: ok15:34
*** breitz has joined #openstack-nova15:34
*** edmondsw has quit IRC15:34
*** konan_abhi has joined #openstack-nova15:34
jaypipesmriedem: in the meantime, lemme look at the latest patch from edleafe on the Ironic inventory management.15:34
*** abhishek_k has quit IRC15:35
*** baoli has quit IRC15:35
mriedemthe harder part of this change is probably going to be the method in the scheduler to remove the RP b/c we'll also have to make sure it doesn't have allocations and inventory in order to delete the RP, else it will 40915:35
jaypipesmriedem: yup15:36
*** sandanar has quit IRC15:36
mriedemdiana_clarke: wow nice catch L(15:36
mriedem:)15:36
openstackgerritMatt Riedemann proposed openstack/nova master: Fix typo in cells v2 ocata reno  https://review.openstack.org/42819815:36
edleafemriedem: add 'force=True'?15:36
mriedemedleafe: is that a thing in the REST API?15:37
jaypipesedleafe: I assigned that bug to you, FYI15:37
edleafemriedem: nope - I was joking15:37
*** jaosorior has joined #openstack-nova15:37
jaypipesmriedem: no.15:37
edleafeforgot the smiley15:37
*** baoli has joined #openstack-nova15:37
jaypipesedleafe: nah, it's easy enough to write code that looks at existence of allocations and deletes them, then inventory, then resource provider.15:37
jaypipesheh15:37
mriedemwe already have methods to delete the allocations and inventory, so probably can just re-use those15:38
* mriedem crosses fingers15:38
edleafejaypipes: it's the same code; doing it in the client just makes more HTTP calls15:38
edleafemriedem: yep, that's the plan15:38
jaypipesmriedem: indeed. in fact, https://review.openstack.org/#/c/404472 has examples of using those exact calls :)15:38
mriedemi'm going to need to see johnthetubaguy be happy with that before i can look at it probably15:39
jaypipesedleafe: understood. I'd prefer to keep a single HTTP call to a single thing unless there's a strong argument not to.15:39
mriedemjaypipes: we also have 2 bugs for disk inventory/allocation in the report client15:39
jaypipess/to a single thing/to do a single thing/15:39
jaypipesmriedem: links pls?15:39
mriedemhttps://review.openstack.org/#/c/428120/15:40
mriedemhttps://bugs.launchpad.net/nova/+bug/165926615:40
openstackLaunchpad bug 1659266 in OpenStack Compute (nova) "Disk allocation for instance is not good with swap" [High,Confirmed] - Assigned to Tanvir Talukder (tanvirt16)15:40
edleafejaypipes: there is no HTTP method that will remove a RP and all of its associated invetory and allocations15:40
*** mdnadeem has quit IRC15:40
*** dimtruck is now known as zz_dimtruck15:40
edleafejaypipes: so adding one will mean a new microversion :(15:40
jaypipesedleafe: I know, and I think that's appropriate.15:40
edleafeI would certainly prefer to add it that way, but with RC1...15:41
jaypipesedleafe: meaning DELETE /resource-providers/<UUID> should return a 409 if there are existing allocations and inventories. The user should be required to DELETE /allocations|inventories in order to perform the DELETE /r-p15:41
edleafejaypipes: of course15:41
*** zz_dimtruck is now known as dimtruck15:41
jaypipesedleafe: I don't care for implementing DELETE CASCADE in a single HTTP call.15:41
mriedemlet us rathole on that in ATL15:42
jaypipesyup15:42
edleafejaypipes: well, that's why I said several HTTP calls15:42
edleafejaypipes: one to delete allocation, one to delete inventory, and the final to delete the RP15:42
jaypipesedleafe: right, which is how things currently work.15:42
mriedemthink if we actually had hard constraints on the instances table and nova didn't auto delete all related things in code...15:43
*** annegentle has joined #openstack-nova15:43
mriedem*auto soft-delete...15:43
dims_mriedem : dansmith : melwitt : should ServersNegativeTestJSON have been fixed? (with https://review.openstack.org/#/c/427782)?15:43
dansmithdims_: yes, but we have another patch in the queue for it15:43
mriedemdims_: yes, but https://review.openstack.org/#/c/427775/15:43
*** krtaylor has joined #openstack-nova15:44
mriedemsfinucan: bauzas: want to send it on https://review.openstack.org/#/c/428198/ ?15:44
dims_mriedem : dansmith : melwitt : gotcha, saw it pop back up. will track 427775. thanks (http://logs.openstack.org/43/408143/9/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/eb7a75d/logs/testr_results.html.gz)15:44
*** nmathew has quit IRC15:45
*** baoli has quit IRC15:45
jrolljaypipes: I'm putting this on a plaque "in the future I will fix the performance issues with the Ironic RT once and for all."15:45
dims_oh boy!15:46
jaypipesjroll: heh.15:46
*** sree has quit IRC15:46
*** tbachman has quit IRC15:47
*** abalutoiu has joined #openstack-nova15:47
*** prateek has quit IRC15:48
*** baoli has joined #openstack-nova15:48
openstackgerritPushkar Umaranikar proposed openstack/nova master: [WIP] Placement api: Add json_error_formatter to defaults  https://review.openstack.org/39519415:49
*** annegentle has quit IRC15:51
*** sree has joined #openstack-nova15:52
*** jamesdenton has quit IRC15:52
*** baoli has quit IRC15:54
johnthetubaguymriedem: jaypipes: I can take a look at that in a few mins15:54
jaypipesjohnthetubaguy: actually, just reviewing you patch to fix the MB -> GB thing. running some tests on it locally, but overall the code looks fine (but also bauzas' identified issue with the allocation side also is a good find)15:55
openstackgerritLance Bragstad proposed openstack/nova-specs master: Cleanup pike policy spec  https://review.openstack.org/42822215:56
bauzasjaypipes: mriedem: FWIW I'm about to add a PTG bullet about discussing on modifying the disk resources so it's bytes15:57
lbragstadjohnthetubaguy ^ feel free to squash that into your original version (it's just cleanup to get your draft passing)15:57
johnthetubaguylbragstad: oh, cool, thanks15:57
*** marst has joined #openstack-nova15:58
jaypipesbauzas: I support the general idea. but would be a pain in the ass to migrate all the MEMORY_MB and DISK_GB inventory and allocation records to bytes...15:58
johnthetubaguylbragstad: feel free to just push to my change if you want, thats cool15:58
johnthetubaguyjaypipes: great, thank you15:58
lbragstadjohnthetubaguy yep! also - you have a todo in there for the keystone model15:58
jaypipesbauzas: in addition to the fact that MEMORY_MB and DISK_GB are standard resource classes.15:58
lbragstadjohnthetubaguy I can take that action item - but where do you want that documented? I felt weird putting keystone things in a nova specific spec15:58
*** armax has joined #openstack-nova15:58
bauzasjaypipes: exactly15:59
*** burgerk has joined #openstack-nova15:59
bauzasjaypipes: that's why I think we should be discussing about it during the PTG15:59
bauzasjaypipes: or during a hangout15:59
bauzasbecause yes, changing that could be a PITA15:59
jaypipescool with me.15:59
jaypipes(the discussions) :)16:00
*** _ducttape_ has joined #openstack-nova16:00
*** jamesdenton has joined #openstack-nova16:01
*** _ducttape_ has quit IRC16:01
edleafejaypipes: cdent just found a flaw in this approach16:02
edleafejaypipes: we can't delete allocations by RP or ID16:02
edleafejaypipes: only by consumer_id16:02
edleafejaypipes: and a consumer might have allocations across multiple RPs16:02
jaypipesedleafe: yes, and?16:03
*** ducttape_ has quit IRC16:03
dansmithso after an evacuate,16:04
dansmithwe wouldn't want to delete allocations for not-our-instance-anymore16:04
johnthetubaguylbragstad: ah, sorry, I missundertood, got you now16:04
jaypipesedleafe: should still be able to delete the resource provider if the deletion of the consumers on that resource provider (identified with an InstanceList.get_all_by_host()) is called to DELETE /allocations/<instance_uuid>16:05
lbragstadjohnthetubaguy yep - line 102 here https://review.openstack.org/#/c/427872/3/specs/pike/approved/policy-cleanup-pike.rst16:05
jaypipesdansmith: yeah, we would. the evacuate would result in a *new* allocation on a new resource provider. we would definitely want to delete the allocation on the old one.16:06
*** stvnoyes has quit IRC16:06
edleafejaypipes: yeah, that could work.16:06
*** ducttape_ has joined #openstack-nova16:06
dansmithjaypipes: edleafe just said you can only delete by consumer id, which is instance uuid right?16:06
*** stvnoyes has joined #openstack-nova16:06
openstackgerritBalazs Gibizer proposed openstack/nova master: Improve assertJsonEqual error reporting  https://review.openstack.org/42819916:07
edleafedansmith: yeah, so to delete by compute node, you'd have to get all the instances for that node, and delete their allocations first16:08
jaypipesdansmith: yes, what edleafe said.16:08
dansmithedleafe: right, but after an evacuation we have allocations on two compute nodes for that instance, right?16:08
edleafedansmith: wouldn't the evacuation delete allocations?16:09
jaypipesdansmith: I think what we're saying is that the evacuation process would delete the allocs on the source node before adding them in the dest node.16:09
*** mdrabe has quit IRC16:09
dansmithedleafe: how? allocations are only managed from the compute node's RT AFAIK16:09
dansmithif we add that, then maybe that's okay16:10
dansmithbut I don't think it happens today16:10
sfinucandiana_clarke: https://review.openstack.org/#/c/411441/2/nova/tests/unit/api/openstack/compute/test_cells.py16:10
* edleafe wonders why he started this without a new pot of coffee...16:10
jaypipesdansmith: no, does not happen today.16:10
dansmithjaypipes: so today we probably already corrupt the allocations after an evacuate anyway I guess16:11
edleafeso let me summarize to make sure I'm clear on the evacuation scenario16:11
jaypipesdansmith: sounds like it.16:11
dansmithjaypipes: we just allocate again on the new node, and then maybe when the old node comes up, we delete the new and old ones16:11
jaypipesdansmith: sounds right.16:12
edleafewhen we evacuate an instance to a new host, the instance ID stays the same, so the allocations for that instance remain16:12
*** Oku_OS is now known as Oku_OS-away16:12
dansmithjaypipes: sounds wrong.. but accurate, unfortunately.16:12
jaypipesdansmith: I mean, not right, but sounds like what currently would happen.16:12
dansmithheh16:12
jaypipesjinx :)16:12
edleafethe new host for that instance would periodically update, see the new instance, and then re-allocate against the new RP16:12
jaypipesdansmith: want to add a bug? or should I?16:12
dansmithjaypipes: I have done nothing useful so far today, so let me16:12
jaypipesedleafe: yes, eventually it would resolve itself... but still not ideal16:13
edleafeIf we then delete all allocations for that instance / consumer, there will be a window where there are no allocations for that instance16:13
dansmithjaypipes: oh yeah I guess it will heal, although not after cdent's proposed lessening of the healing action I think16:13
dansmithor at least, heal slower16:13
dansmithjaypipes: but opens us up for a poor decision because we think that host is less loaded than it is16:13
cdentdansmith: i've punted on that until we figure out how that stuff even works16:14
dansmithcdent: yeah, I know, I'm just saying16:14
dansmithcdent: either (a) this is a good reason to take the hit on the healing or (b) need to consider this kind of thing if/when we proceed with that16:14
cdentI'd vote for b16:14
cdentso we can get rid of some of the implictness and confusion when we talk about this stuff :)16:14
*** tbachman has joined #openstack-nova16:15
dansmithwell, things like this being papered over because we heal aggressively is problematic I think,16:15
dansmithbecause before we heal, we're making bad decisions16:15
*** ftersin has quit IRC16:15
dansmithwhich I think is an argument for (b), but..16:15
cdent( )|( )16:16
dansmithum16:17
* dansmith squints to see the real picture16:17
dansmitha butt?16:17
cdenta big ol butt16:17
dansmithclassy, cdent, classy16:18
*** annegentle has joined #openstack-nova16:18
openstackgerritMaciej Szankin proposed openstack/nova master: conf: mark glance.api_servers opt as required  https://review.openstack.org/42824116:19
*** mdrabe has joined #openstack-nova16:19
*** xinli has joined #openstack-nova16:20
dansmithedleafe: https://bugs.launchpad.net/nova/+bug/166131216:21
openstackLaunchpad bug 1661312 in OpenStack Compute (nova) "Evacuation will corrupt instance allocations" [Undecided,New]16:21
edleafedansmith: Thanks.16:22
lbragstadjohnthetubaguy i'm going to propose a keystone spec for policy in code and I'll get a follow on to that to address your keystone TODOs in https://review.openstack.org/#/c/427872 (then we can reassess)16:24
*** kberger has joined #openstack-nova16:25
*** jamesden_ has joined #openstack-nova16:26
*** jamesdenton has quit IRC16:28
*** tlian has joined #openstack-nova16:28
*** rcernin has quit IRC16:29
openstackgerritMatt Riedemann proposed openstack/nova master: Fix access_ip_v4/6 filters params for servers filter  https://review.openstack.org/42807116:30
mriedemgmann: any idea if Kevin had an update for this https://review.openstack.org/#/c/421760/?16:35
mriedemor if the release note was going to be split out?16:35
mriedemb/c we probably don't have time to wait for china to wake up16:36
*** ducttape_ has quit IRC16:38
*** ducttape_ has joined #openstack-nova16:39
*** sree has quit IRC16:39
*** baoli has joined #openstack-nova16:40
*** chyka has joined #openstack-nova16:41
*** tesseract has quit IRC16:41
*** markus_z has quit IRC16:43
*** browne has joined #openstack-nova16:47
*** artom_ is now known as artom16:47
*** baoli has quit IRC16:48
*** annegentle has quit IRC16:48
*** andreas_s has quit IRC16:49
*** annegentle has joined #openstack-nova16:50
openstackgerritBalazs Gibizer proposed openstack/nova master: Short circuit notifiations when not enabled  https://review.openstack.org/42826016:52
*** dikonoor has joined #openstack-nova16:52
openstackgerritBalazs Gibizer proposed openstack/nova master: handle uninited fields in notification payload  https://review.openstack.org/41585716:54
*** tblakes has quit IRC16:55
jaypipessdague, dims: new box, trying to run some py 3.5-specific things. getting this: http://paste.openstack.org/show/597397/. Any ideas?16:55
*** bvanhav__ has joined #openstack-nova16:56
*** bvanhav_ has quit IRC16:56
*** tlian has quit IRC16:58
openstackgerritArtom Lifshitz proposed openstack/nova master: Pass APIVersionRequest to extensions  https://review.openstack.org/42587616:58
mriedemjaypipes: read the note in tox.ini16:59
mriedemyou need to delete .testrepository16:59
jaypipesah. thx mriedem.16:59
*** dimtruck is now known as zz_dimtruck16:59
alex_xumriedem: let me try to take care of https://review.openstack.org/#/c/42176017:01
openstackgerritJohn Garbutt proposed openstack/nova master: [WIP] Placement api: Add json_error_formatter to defaults  https://review.openstack.org/39519417:02
*** liverpooler has quit IRC17:02
ildikovjohnthetubaguy: mriedem: can you come to the Cinder-NOva meeting?17:02
*** liverpooler has joined #openstack-nova17:03
*** hoonetorg has quit IRC17:03
*** Apoorva has joined #openstack-nova17:03
*** Apoorva has quit IRC17:03
johnthetubaguyildikov: ah, thats moved to today then I guess17:04
*** hoonetorg has joined #openstack-nova17:04
johnthetubaguyildikov: I have updated my calendar, but honestly I have to chase a few of these RC related patches17:04
mriedemildikov: busy with rc1 things17:05
*** tblakes has joined #openstack-nova17:05
ildikovjohnthetubaguy: mriedem: yep, it got moved, but will ping you directly if something pops up then17:05
mriedemalex_xu: thanks, i think at this point i'd rather just make the reno a separate patch17:06
mriedemthe api-ref change is too large and needs a much more detailed review17:07
alex_xumriedem: yea, agree with that, just working on separating the patch17:07
*** raunak has joined #openstack-nova17:08
*** kfarr has quit IRC17:10
*** mriedem1 has joined #openstack-nova17:17
*** mriedem has quit IRC17:17
*** mtanino_ has joined #openstack-nova17:18
*** nic has joined #openstack-nova17:18
*** kfarr has joined #openstack-nova17:19
*** mtanino__ has joined #openstack-nova17:19
*** mtanino has quit IRC17:19
*** sree has joined #openstack-nova17:20
*** mtanino_ has quit IRC17:22
*** panda is now known as panda|bbl17:26
*** raunak has quit IRC17:27
*** raunak has joined #openstack-nova17:28
openstackgerritDiana Clarke proposed openstack/nova master: Add offset & limit docs & tests  https://review.openstack.org/41144117:29
openstackgerritJohn Garbutt proposed openstack/nova master: Report reserved_host_disk_mb in GB not KB  https://review.openstack.org/42812017:33
*** alexpilo_ has joined #openstack-nova17:35
*** dtp has joined #openstack-nova17:35
*** alexpilotti has quit IRC17:37
*** hoonetorg has quit IRC17:38
*** ducttape_ has quit IRC17:39
jaypipesjohnthetubaguy: lol between 2.7 and 3.5 float handling, eh?17:39
*** hoonetorg has joined #openstack-nova17:40
*** nkrinner is now known as nkrinner_afk17:41
*** karimb has quit IRC17:41
*** annegentle has quit IRC17:43
melwittjaypipes: mriedem1 and I were discussing yesterday, is everything already merged that handles the local disk for volume backed instances to treat it as 0? does it also request 0 disk during a rebuild, etc?17:43
*** mriedem1 is now known as mriedem17:43
*** aarefiev is now known as aarefiev_afk17:44
*** liangy has joined #openstack-nova17:45
*** zz_dimtruck is now known as dimtruck17:45
mriedemmdbooth: i'm going through your nfs mount patch now17:46
mriedemmdbooth: so don't drop off yet17:46
openstackgerritAlex Xu proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42828517:46
alex_xumriedem: ^ done17:46
mriedemthanks17:46
openstackgerritMaciej Szankin proposed openstack/nova master: conf: mark glance.api_servers opt as required  https://review.openstack.org/42824117:46
alex_xunp17:46
dansmithmelwitt: yeah, I think we should be good there17:46
*** hoonetorg has quit IRC17:47
melwittsweet17:47
*** satyar_ has joined #openstack-nova17:48
*** baoli has joined #openstack-nova17:48
*** satyar has quit IRC17:49
*** annegentle has joined #openstack-nova17:50
*** lucasagomes is now known as lucas-afk17:51
*** cdent has quit IRC17:52
*** sudipto_ has quit IRC17:53
*** sudipto has quit IRC17:53
*** vsaienko has joined #openstack-nova17:53
johnthetubaguyjaypipes: the joys!17:53
johnthetubaguyjaypipes: I have a concern on this ironic patch still: https://review.openstack.org/#/c/40447217:54
johnthetubaguyjaypipes: but something tells me I am miss understanding the resource tracker sequence17:54
*** Apoorva has joined #openstack-nova17:54
*** mjura has quit IRC17:54
diana_clarkesfinucan: thanks a bunch for reviewing that!17:55
*** jose-phillips has joined #openstack-nova17:55
*** lpetrut has quit IRC17:56
*** liangy has quit IRC17:57
*** owalsh is now known as owalsh-afk17:57
*** baoli has quit IRC17:58
*** hoonetorg has joined #openstack-nova17:59
*** baoli has joined #openstack-nova17:59
*** vsaienko has quit IRC17:59
*** vsaienko has joined #openstack-nova18:00
*** raunak has quit IRC18:01
*** ralonsoh has quit IRC18:02
sfinucandiana_clarke: np :)18:02
mriedemmdbooth: lyarwood: before i go further, some comments/questions inline https://review.openstack.org/#/c/383859/13/nova/virt/libvirt/volume/mount.py18:02
*** kfarr has quit IRC18:02
*** owalsh-afk has quit IRC18:03
*** derekh has quit IRC18:03
*** raunak has joined #openstack-nova18:03
*** vsaienko has quit IRC18:04
*** vsaienko has joined #openstack-nova18:05
*** sree has quit IRC18:05
BobBalljohnthetubaguy / mriedem FYI I believe https://review.openstack.org/#/c/428180/ will fix the Citrix XenServer CI18:09
*** Swami has joined #openstack-nova18:09
mriedemcool18:10
BobBalljohnthetubaguy / mriedem: A dodgy Cinder commit meant that "older" versions of pvs would all fail (as they don't have the newly required option) - the check for the version was broken.18:10
BobBallWe will monitor the CI after this cinder commit is merged and let you know the results and when we'd like to re-start voting.18:11
mriedemsure18:11
openstackgerritMaciej Szankin proposed openstack/nova master: conf: resolved TODOs in compute configuration  https://review.openstack.org/42679318:11
mriedemi guess if LVM.supports_pvs_ignoreskippedcluster was just set to False18:11
mriedembecause of the old pvs18:11
mriedem*lvm18:12
BobBallActually it was always returning true... LVM.supports_pvs_ignoreskippedcluster was *meant* to be reading a property with a method behind it (using self.supportsXXX) but LVM.supportsXXX is the method itself so that was not None.18:13
openstackgerritmelanie witt proposed openstack/nova master: Add step-by-step examples for Cells V2 setup  https://review.openstack.org/42798118:13
BobBalloh sorr I mis-read.18:13
BobBallCouldn't "set" it because it's just a method on the class18:13
*** alexpilotti has joined #openstack-nova18:15
*** cdent has joined #openstack-nova18:17
*** dikonoor has quit IRC18:18
*** alexpilo_ has quit IRC18:18
*** alexpilotti has quit IRC18:20
mriedemmdbooth: lyarwood: done with that review, only real questions are in that mount module18:20
*** alexpilotti has joined #openstack-nova18:20
jaypipesjohnthetubaguy: +W from me on you disk_gb patch.18:21
* mriedem gets lunch18:21
*** unicell has joined #openstack-nova18:26
*** unicell has quit IRC18:26
*** unicell has joined #openstack-nova18:26
*** yamahata has joined #openstack-nova18:27
mriedemis anyone working a fix for this? https://bugs.launchpad.net/nova/+bug/165926618:28
openstackLaunchpad bug 1659266 in OpenStack Compute (nova) "Disk allocation for instance is not good with swap" [High,Confirmed] - Assigned to Tanvir Talukder (tanvirt16)18:28
*** hieulq has joined #openstack-nova18:29
mriedembauzas: ^ you reported it18:29
EmilienM2 questions :18:29
EmilienM1. Where [cells] section needs to be configured (on which services, eg: nova-api? etc)18:29
mriedem(1) who is your daddy and (2) what does he do?18:29
mriedemEmilienM: cells config is only for cells v118:30
mriedemyou don't care about it18:30
EmilienM2. Do we *need* cells/enable= true in Ocata,18:30
mriedemno18:30
EmilienMamoralej: ^18:30
mriedem[cells] is all about cells v118:30
EmilienMmriedem: ok18:30
amoralejthanks mriedem18:30
EmilienMyeah thx18:30
* EmilienM goes back in debugging18:31
mriedemi'll update the config option help to make that more clear18:31
EmilienMmriedem: we're unable to spawn VMs anymore18:31
EmilienMmaybe you'll know why18:31
EmilienMhttp://logs.openstack.org/15/424915/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b29f35b/logs/nova/nova-compute.txt.gz#_2017-02-02_12_57_00_75418:31
amoralejmriedem, we getg "Instance 6a26cec0-3769-44b6-851c-b77359897e07 could not be found." kind of errors in certain tempest tests18:32
EmilienMmriedem: we're deploying trunk18:32
EmilienMwith UC18:32
johnthetubaguyjaypipes: thanks!18:33
*** dharinic is now known as dharinic|lunch18:34
*** mvk has quit IRC18:35
openstackgerritMatt Riedemann proposed openstack/nova master: Clarify the [cells] config option help  https://review.openstack.org/42834418:35
mriedemmelwitt: dansmith: ^18:35
*** annegentle has quit IRC18:36
mriedemEmilienM: that trace says the bdm is invalid18:37
EmilienMbdm?18:37
mriedemEmilienM: because the volume isn't found http://logs.openstack.org/15/424915/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b29f35b/logs/nova/nova-compute.txt.gz#_2017-02-02_12_56_57_27418:37
mriedembdm == block_device_mapping18:37
EmilienMBlock Device Mapping18:37
EmilienMok18:37
EmilienMwell, it worked yesterday18:38
mriedemEmilienM: did it work with devstack? :P18:38
johnthetubaguymriedem: I can put something up for that bug 165926618:38
openstackbug 1659266 in OpenStack Compute (nova) "Disk allocation for instance is not good with swap" [High,Confirmed] https://launchpad.net/bugs/1659266 - Assigned to Tanvir Talukder (tanvirt16)18:38
* mriedem ducks18:38
mriedemjohnthetubaguy: thanks18:38
EmilienMmriedem: I was about to say it but well...18:38
johnthetubaguyI have to run, soon, but lets see what can be done18:38
mriedemjohnthetubaguy: you're not allowed to leave18:38
EmilienMmriedem: FYI we're delaying TripleO release because upgrades are not working currently from Newton to Ocata18:38
mriedemsally will understand18:38
EmilienMmriedem: we hope to solve blockers by next week :/18:39
johnthetubaguymriedem: you haven't met the conductor, she is scary18:39
*** ducttape_ has joined #openstack-nova18:40
mriedemEmilienM: my wild guess would be something related to https://review.openstack.org/#/c/427782/18:41
mriedemwhich merged last night18:41
*** salv-orlando has joined #openstack-nova18:41
mriedemEmilienM: but would also mean you don't have things setup for cells v218:41
EmilienMmriedem: do you run experimental job sometimes?18:42
EmilienMit seems like you're not18:42
EmilienMmriedem: I think we are pretty well covered for cells v2 but I'll need to confirm18:42
diana_clarkeEmilienM: I've run it, but I've yet to see it pass ;(18:43
*** vsaienko has quit IRC18:43
EmilienMdiana_clarke: where?18:43
diana_clarkeEmilienM: for example: https://review.openstack.org/#/c/420186/18:43
*** vsaienko has joined #openstack-nova18:44
diana_clarke(it's not recent, but people are running it)18:44
*** ducttape_ has quit IRC18:44
EmilienMdiana_clarke: cool18:44
*** READ10 has quit IRC18:45
dansmithI run it too on occasion, but it takes a long time, fails often, and the logs are harder to sift through18:46
*** vsaienko has quit IRC18:46
EmilienMdansmith: ping me when it fails18:47
EmilienMI'm happy to help18:47
openstackgerritJohn Garbutt proposed openstack/nova master: Stop swap allocations being wrong due to MB vs GB  https://review.openstack.org/42835218:47
EmilienMdansmith: any idea about my bug mentioned above ^18:48
dansmithEmilienM: I looked a bit earlier, but I saw no debug logs to help track anything down18:48
EmilienMdansmith: no debug on http://logs.openstack.org/15/424915/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b29f35b/logs/nova/nova-compute.txt.gz#_2017-02-02_12_56_57_274 ?18:49
johnthetubaguyjaypipes: mriedem: its a quick cut and refactor from the previous fix, hopefully, at least thats what the unit tests suggested: https://review.openstack.org/#/c/42835218:49
EmilienMdansmith: could it be related to cells again?18:49
dansmithEmilienM: no, I was looking at another run that had no debug, periodic maybe?18:50
EmilienMdansmith: yeah periodic but this one has debug ^18:50
*** vsaienko has joined #openstack-nova18:50
dansmithEmilienM: ah, I hadn't seen that volume error, and no that's not cells related AFAIK18:51
openstackgerritJohn Garbutt proposed openstack/nova master: Stop swap allocations being wrong due to MB vs GB  https://review.openstack.org/42835218:51
EmilienMdansmith: could it be related to https://review.openstack.org/#/c/427782/ ?18:52
*** karimb has joined #openstack-nova18:52
*** annegentle has joined #openstack-nova18:52
openstackgerritJohn Garbutt proposed openstack/nova master: DNM: hack ironic with resource providers  https://review.openstack.org/42629618:52
* johnthetubaguy runs to rehersal18:53
dansmithEmilienM: not the volume error for sure no18:53
EmilienMwell, if you have any idea so far, please let me know. We're stuck.18:54
*** vsaienko has quit IRC18:55
openstackgerritMaciej Szankin proposed openstack/nova master: conf: add min parameter to scheduler opts  https://review.openstack.org/42835518:56
mriedemEmilienM: so the instance not found you were talking about is like this http://logs.openstack.org/15/424915/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b29f35b/console.html#_2017-02-02_13_04_48_20418518:56
mriedemin a bunch of the tempest tests18:57
*** vsaienko has joined #openstack-nova18:57
mriedemit looks like it's waiting for the server to be active, it's gone18:57
amoralejEmilienM, dansmith, reported in https://bugs.launchpad.net/nova/+bug/166136018:58
openstackLaunchpad bug 1661360 in OpenStack Compute (nova) "tempest test fails with "Instance not found" error" [Undecided,New]18:58
jrolljohnthetubaguy: there's an experimental job for resource class stuff, so I was going to abandon https://review.openstack.org/#/c/426296/18:58
jrolljohnthetubaguy: but can keep it around if you rather18:58
mriedemEmilienM: i can put up a debug patch if you guys can depend on it18:58
amoralejmriedem, i've reproduced locally, nova server returns "instance not found" but it's found in "nova list" command18:58
mriedemamoralej: that's probably a race between when we have a build request and the actual instance,18:59
mriedemthe api returns either18:59
mriedemsince they both represent a server at different stages of the lifecycle18:59
EmilienMmriedem: yes I can18:59
EmilienMmriedem: you can run check experimental18:59
*** carthaca_ has quit IRC19:00
*** sapcc-bot has quit IRC19:00
dansmithmriedem: what race?19:00
*** eharney has quit IRC19:00
*** carthaca_ has joined #openstack-nova19:00
*** dgonzalez_ has joined #openstack-nova19:00
*** sapcc-bot has joined #openstack-nova19:00
*** tpatzig_ has joined #openstack-nova19:00
melwittmriedem: isn't that issue fixed by https://review.openstack.org/#/c/427775 this morning?19:01
dansmithmelwitt: there's no delete going on here, which is what ^ is for right?19:01
melwittoh. yeah the above is for delete19:01
*** ociuhandu has quit IRC19:01
*** dgonzalez_ has quit IRC19:02
*** tpatzig_ has quit IRC19:02
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: debugging for instance not found in tripleo  https://review.openstack.org/42836019:04
mriedemEmilienM: amoralej: ^19:04
cdentedleafe: do you recall if this allocation checking is happening anywhere else because this seems important but is stuck behind something else: https://review.openstack.org/#/c/416752/19:04
mriedemmelwitt: dansmith: even with my fix,19:05
mriedemcouldn't we have a window of time where the build request exists, the instance exists, but the instance mapping doesn't?19:05
dansmithmriedem: sure, but then we return the buildrequest, that's the point19:05
dansmithmriedem: in their logs, I do not see "Found 1 cells"19:05
edleafecdent: it's not really "stuck"; just hasn't gotten core attention19:05
*** trevormc has joined #openstack-nova19:06
*** raunak has quit IRC19:06
dansmithmriedem: I wonder if it's the change from melwitt that skips looking in all cells -- maybe incorrectly following that path?19:06
cdentedleafe: what I meant was that fix is +W and something we ought to release, but the other thing less so19:06
*** Sukhdev has joined #openstack-nova19:06
mriedemdansmith: so maybe it hits a window when the api code sees there is no instance mapping, but then by the time it pulls the build request, that's gone19:06
edleafebut yeah, it's not technically dependent on the one below it19:06
mriedemthat would be tight19:06
dansmithmriedem: the api isn't even looking at the cells, though19:07
melwittthat's only if CONF.cells.enable is True though19:07
edleafecdent: the other thing is as important19:07
dansmithmelwitt: I know, but19:07
edleafethe create() method allows all sorts of bad stuff in19:07
EmilienMmriedem: "check experimental" and we'll see what happens19:07
amoralejI'm testing locally19:07
dansmithmelwitt: where is this in their log? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L251819:07
*** _gryf has quit IRC19:08
*** _gryf has joined #openstack-nova19:08
*** lpetrut has joined #openstack-nova19:08
*** dharinic|lunch is now known as dharinic19:08
cdentedleafe: but it's not wired into the api is it? that's what I'm trying to get at. the patch above prevents the api being used badly, the patch below removes something that was never exposed19:08
dansmithcells.enable=False in their startup dump19:08
melwitthm19:09
dansmithmelwitt: see what I mean though?19:09
melwittyeah19:09
dansmithcompare to a devstack run: http://logs.openstack.org/03/427903/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fb34df9/logs/screen-n-api.txt.gz#_2017-02-01_21_00_47_55219:10
*** trevormc_ has joined #openstack-nova19:10
edleafecdent: true19:10
*** trevormc_ has left #openstack-nova19:10
edleafecdent: but rebasing on master will lose the +W19:10
dansmithnow it's possible that they have never had it right, and removing those pre-cellsv2 short circuits are why they're failing now, but that's exactly why those have to be gone19:10
melwittdansmith: I'm also wondering about your change that changed the fallbacks into raises of InstanceNotFound19:10
cdentedleafe: so the question is: how much of a bug is it that it is possible to write negative allocations?19:10
*** trevormc has quit IRC19:10
* cdent tries it19:10
*** raunak has joined #openstack-nova19:11
dansmithmelwitt: right, that's what I just mentioned, but we *have* to remove those else we'll paper over not having things19:11
melwittyeah19:11
edleafecdent: negative not so much. Zero allocations was the issue19:11
*** trevormc has joined #openstack-nova19:11
cdentedleafe: by "tries it", I mean "will they get all the way through"19:11
cdentif they get through, that seems a fair risk19:12
mriedemdansmith: i'll throw this into the get instance flow in my debug patch https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2515-L252219:13
melwittdansmith: the 'Found cells' message is only for the get_all so that's why we're not seeing it I think. they're seeing a fail on a get of a single instance and said they see it in the list of all instances if they do a 'nova list'19:14
mriedemright19:14
dansmithmelwitt: which is a get all19:14
mriedemso i'll throw that into the debug patch19:14
mriedemthe test is failing on a single GET19:14
mriedemnot list19:14
melwittdansmith: yeah, it shows up in get_all but not in get19:14
mriedemthat's why we don't see it19:14
melwittis the problem they described19:14
melwittyeah19:14
dansmithmelwitt: I know, but they've done at least one get_all by that point right?19:14
mriedembut, we should see it in the logs somewhere,19:14
mriedembecause tempest will list instances at some point19:14
dansmithright19:14
melwitthm19:15
*** mlavalle has quit IRC19:15
*** bvanhav__ has quit IRC19:15
*** andrewbogott has quit IRC19:16
*** bvanhav__ has joined #openstack-nova19:16
melwittit's possible it didn't yet, but let me see if I can locate a list all in the log19:16
*** karimb has quit IRC19:16
dansmitham I missing where our requests are logged in n-api log?19:17
*** karlamrhein has quit IRC19:17
melwittI'm not seeing it them yet. this doesn't look usual to me19:17
*** karimb has joined #openstack-nova19:17
dansmithmelwitt: http://logs.openstack.org/15/424915/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b29f35b/logs/tempest.txt.gz#_2017-02-02_12_53_52_73519:18
dansmithmelwitt: yeah, something seems wrong about it19:18
dansmithmelwitt: but the above should have triggered a list all right?19:18
jaypipesjohnthetubaguy: +219:18
melwittdansmith: no, I think that's a create. it will return the one instance, or the first instance if it's a multi create19:18
melwittthat's a POST19:19
dansmithoh you're right19:19
openstackgerritEd Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted  https://review.openstack.org/42837519:19
edleafejaypipes: cdent: ^^ please review19:20
cdentedleafe: yessir19:20
mriedemthere is no "200 GET https://[::1]:8774/v2.1/servers" in that log19:20
dansmithhow is that possible19:20
melwittthis is very odd19:21
*** karlamrhein has joined #openstack-nova19:22
*** andrewbogott has joined #openstack-nova19:22
dansmithmriedem: so your race...19:22
dansmithmriedem: we look up the map, if we fail, we look up the buildreq, if that fails we re-check for the map19:22
dansmithby the time we've deleted the buildreq, we've finished up the instance map and saved it19:23
dansmithso I don't see that there is any window there19:23
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: debugging for instance not found in tripleo  https://review.openstack.org/42836019:23
dansmithare you thinking of a different place?19:23
mriedemdansmith: i was thinking it's like this19:23
mriedem1. https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2271 we don't have an instance mappingyet19:24
mriedemwell, shit that's it19:24
mriedemif there is no instance mapping, that doesn't lookup the build req19:24
mriedemat all, right?19:24
mriedemthe conditionals in there are only caring if there is an instance mapping AND it has a cell mapping19:25
dansmithbut we create the instance map early19:25
melwittuh oh, looks tha tway19:25
dansmithlemme look19:25
mriedemwe create the instance mapping after the instance is created, but before the build request is deleted19:25
dansmithno19:25
dansmithwe update it19:25
dansmithwe create the instance mapping at almost the same time as the buildreq19:25
dansmithalthough the buildreq is first by a few python cycles19:26
dansmiththis would be nanoseconds of a window19:26
mriedemoh right19:26
mriedemhttps://github.com/openstack/nova/blob/master/nova/compute/api.py#L99719:26
mriedemhttps://github.com/openstack/nova/blob/master/nova/compute/api.py#L101019:26
dansmithpicoseconds even19:26
dansmithyeah19:26
mriedemi thought you just started making up small time units now19:26
mriedemdanoseconds19:26
dansmithfemtoseconds19:27
melwittsmithoseconds19:27
mriedemonly Vek can say femto-anything19:27
dansmithmriedem: so I can swap the order of those in a test patch to run experimental on, but I'm highly dubious19:27
dansmithmriedem: Vek gave me the femtotorch19:27
mriedemha19:27
dansmithit was a small and quick handoff19:27
mriedemhow appropriate19:28
melwitt /facepalm19:28
mriedemrimshot?19:28
*** ociuhandu has joined #openstack-nova19:28
openstackgerritDan Smith proposed openstack/nova master: DNM: Reorder two electrons to see if it fixes a race  https://review.openstack.org/42838519:30
melwitthaha, like that commit message19:30
dansmithso this probably comes from the logic in get_instance needing to be defensive against pre-cellsv2 data, so we could also just rearrange that and probably simplify19:30
*** vsaienko has quit IRC19:31
*** ducttape_ has joined #openstack-nova19:32
*** eharney has joined #openstack-nova19:37
*** mtanino__ has quit IRC19:39
*** kfarr has joined #openstack-nova19:52
amoralejmriedem, dansmith, i've been debugging by printing value of service_version right before https://github.com/openstack/nova/blob/master/nova/compute/api.py#L226819:54
amoralejand i'm getting allways 019:54
amoralejin fact if y remove the "service_version < 15" part of the if, tempest test passes19:55
amoralejhow can i see the versions of each service?19:55
amoralejEmilienM ^19:55
dansmithamoralej: orly19:55
dansmithamoralej: maybe we're grabbing a cached version of zero before we start computes19:56
dansmithnot sure what would have changed around that specific thing recently though19:56
amoraleji can try to remove the compute node and register it again19:56
dansmithwe really should never cache version=0, which I think we're probably doing right now19:58
*** raunak has quit IRC20:00
*** annegentle has quit IRC20:00
*** rfolco has quit IRC20:01
*** panda|bbl is now known as panda20:02
*** annegentle has joined #openstack-nova20:02
*** vsaienko has joined #openstack-nova20:06
*** salv-orlando has quit IRC20:09
amoralejdansmith, in nova database all services have version 1620:10
amoralejbut nova_cell0 has no services20:10
mriedemthere are no hosts for nova_cell020:10
mriedemcell0 is just a place to dump dead instances that didn't get built20:11
cdentedleafe: my testing suggests there's no urgency, but it does suggest that checking the allocations is not necessary if we change (and validate) the default min_unit when creating inventory20:11
cdentif that's done, then there's no need to check the allocations manually20:11
dansmithamoralej: the problem is it may have 16 at the end, but might've had zero early such that it cached zero and never saw the 1620:11
amoralejdansmith, in that case it should work if i restart nova api?20:12
amoraleji can try20:12
dansmithamoralej: yeah20:12
openstackgerritEd Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted  https://review.openstack.org/42837520:13
mriedemvdrok: want to test this? https://review.openstack.org/#/c/428375/20:14
openstackgerritDan Smith proposed openstack/nova master: Don't ever cache service versions at zero  https://review.openstack.org/42840420:14
*** lpetrut has quit IRC20:15
amoralejdansmith, reboot didn't work20:16
amoralejbut20:16
*** kfarr has quit IRC20:16
dansmithamoralej: did you get all api workers on all nodes?20:16
amoralejreplacing nova-osapi_compute by nova-compute worked20:16
dansmithoh jeez20:16
amoralejso20:16
dansmithwhere?20:16
amoralejhttps://github.com/openstack/nova/blame/master/nova/compute/api.py#L226820:16
dansmithcripes I see20:16
dansmithjeezus20:16
amoraleji mean https://github.com/openstack/nova/blame/master/nova/compute/api.py#L226420:17
dansmithffs20:17
dansmithhow.20:17
dansmithmany.20:17
dansmithtimes.20:17
amoralejis service-name configurable?20:17
dansmithno20:17
dansmiththis is a bug20:17
dansmithI dunno why we're not hitting it elsewhere20:17
amoralejEmilienM ^20:17
EmilienMamoralej: nice catch!20:18
amoralejdansmith, and https://github.com/openstack/nova/blame/master/nova/compute/api.py#L170720:18
dansmithwait wait wai20:18
*** kfarr has joined #openstack-nova20:18
dansmithI think that's actually looking for the api service not the compute service20:19
dansmithso it should be right20:19
melwittyeah, that should be right. it used to be wrong as 'nova-api' in the past, somewhere in compute/api I recall20:19
cdentedleafe: I left more about the min_unit stuff on https://review.openstack.org/#/c/416752/720:19
dansmithright, and I was thinking it was looking for nova-compute, which is what the change was20:19
dansmithso I jumped20:19
amoralejand where api services are registered?20:19
dansmithamoralej: you changing it just means you're looking up a different service20:19
amoraleji don't see it in services table20:19
amoralejwe are running it as wsgi20:20
dansmithoh, hmm, I bet that is a problem20:20
edleafecdent: I can get to those changes today20:20
amoralejdansmith, this is my services table http://paste.openstack.org/show/597426/20:20
dansmithamoralej: you see no service records in the services table for nova-osapi_compute?20:20
dansmithyeah20:21
dansmithbecause you don't use our service infrastructure20:21
dansmithmriedem: we've never said we're okay to run under wsgi did we?20:21
mriedemdansmith: we don't have anything official20:21
amoralejEmilienM ^20:21
mriedembut i know people run it20:21
mriedemi think Fuel did too20:21
cdentedleafe: feel free, but I don't think there's any rush: because the RT sets min_unit correctly, any "proper" use of the placement api will work okay and a bad admin can't send 0 allocations to the inventory that's been written (because min_unit is 1)20:21
cdent(if you do do it, add me to the review so it is on my radar so I know not to do it)20:22
dansmiththe thing is, I'm not sure what in the last day would have broken20:22
edleafecdent: Understood, but I hate building those checks into things external to placement20:22
mriedemdansmith: even with your change, we would have hit this i think https://review.openstack.org/#/c/427782/3/nova/compute/api.py@226920:23
cdentedleafe: oh yeah, totally agree20:23
EmilienMnova gate is not running api in wsgi with apache?20:23
edleafecdent: because the next time some service tries to use placement, we have to copy/paste all that external stuff there, too20:23
mriedemEmilienM: no20:23
* cdent nods20:23
dansmithmriedem: on wsgi, yeah20:23
EmilienMwhy?20:23
mriedemEmilienM: no one has made that happen20:23
dansmithand we have things like this that hook into our service framework20:24
mriedemapache support was hacked in several releases ago as a kind of 'i want to see if i can do this' but it wasn't like a supported new feature or anything20:24
amoraleji'm trying to test it with nova-api20:24
mriedemit was a little toy patch20:24
EmilienMtripleo has been running it in productince since Newton20:25
EmilienMproduction*20:25
*** crushil has quit IRC20:26
dansmithso few real-world users at this point I guess20:26
*** crushil has joined #openstack-nova20:26
dansmithdims_: since you were originally involved, any idea how many real people running fuel might be running nova api under apache?20:26
EmilienMdansmith: few real-world users? It's in OSP1020:27
dansmithEmilienM: yeah, I dunno what the footprint of that looks like20:27
*** mlavalle has joined #openstack-nova20:28
mriedemjaypipes: "any idea how many real people running fuel might be running nova api under apache?"20:28
*** Apoorva has quit IRC20:28
mwhahahamriedem: a bunch, it's in 9 believe20:29
* mwhahaha goes looking20:29
mriedemhttps://github.com/openstack/nova/blob/ed55dcad83d5db2fa7e43fc3d5465df1550b554c/nova/wsgi/nova-api.py#L1520:30
mriedemEXPERIMENTAL support script for running Nova API under Apache2 etc.20:30
mriedemanywho20:30
jrollexperimental means battle-tested production grade software, right20:30
jrolllike nova-bm20:31
mriedemcarrier grade20:31
jroll++20:31
cdentI'm unclear why exporting deploy() as application is so fraught with risk?20:31
EmilienMmriedem: yeah a lot of operators are doing that20:32
EmilienMmriedem: I don't have numbers but it's something common20:32
*** kaisers_ has joined #openstack-nova20:32
mriedemcdent: like i said earlier, it was just a 'lets see if this can happen' thing long ago that we never built CI around20:32
jrollcdent: I think the current theory is that it doesn't get any of this code: https://github.com/openstack/nova/blob/ed55dcad83d5db2fa7e43fc3d5465df1550b554c/nova/service.py#L13920:32
melwittyeah, we didn't do it but thought we should have since things like the api rate limiting can't work as-is20:33
jrollspecifically line 15420:33
mriedemso i think we provide a workaround hack in ocata20:33
cdentthank jroll that's what I was after20:33
mriedemand work on running nova-api under apache in one of the gate jobs in pike20:33
jrollcdent: it's overridden for the wsgi thing, I guess, but same idea: https://github.com/openstack/nova/blob/ed55dcad83d5db2fa7e43fc3d5465df1550b554c/nova/service.py#L37620:33
jrollno problem20:33
*** baoli has quit IRC20:34
mwhahahaso to answer, fuel doesn't use apache for the nova api. it's haproxy fronting the standalone service20:35
EmilienMI think Nova team should consider moving the experimental tripleo job to the gate. We have seen over the last weeks (more probably) that it brings very useful feedback on how it works outside devstack20:35
*** salv-orlando has joined #openstack-nova20:35
dansmithmwhahaha: ah, good to know20:35
*** bvanhav__ has quit IRC20:35
*** kaisers has quit IRC20:35
*** baoli has joined #openstack-nova20:37
EmilienMmriedem: I think testing api service with apache is part of the goal for Pike20:37
mriedemwhich goal?20:37
mriedemhow many goals are we up to now?20:37
mriedemi said we can start running it in one gating job for nova in pike as a start20:37
EmilienMin https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html#completion-criteria20:38
*** mvk has joined #openstack-nova20:38
mriedembecause we have a unicorn snowflake job for stuff like that already20:38
EmilienMmriedem: 2 goals20:38
*** amoralej is now known as amoralej|off20:38
mriedemok, so i guess it's a goal for us then anyway20:39
mriedemi'll throw it in our ptg etherpad20:39
*** annegentle has quit IRC20:40
*** Sukhdev has quit IRC20:40
dims_dansmith : i don't think the fuel team ever picked it up from where we left off here in nova. so probably not very many (http://git.openstack.org/cgit/openstack/fuel-library/tree/deployment/puppet/openstack_tasks/manifests/openstack_controller/openstack_controller.pp)20:40
dims_jaypipes : amirite?20:41
dansmithdims_: gotcha20:41
EmilienMmriedem: I will propose to run tripleo job in the gate for nova20:42
mriedemEmilienM: i will -120:42
EmilienMof course you will20:42
EmilienMbut you'll have to explain why20:42
mriedembecause it's not even non-voting in the nova check queue for starters20:42
mriedemwe aren't going to gate on something right off the bat20:42
EmilienMok, let's make non voting first20:43
EmilienMand see how it goes20:43
mriedemare you talking 1 job or 6?20:43
EmilienMbut we need to stop ignoring the world outside devstack20:43
EmilienM1 job20:43
EmilienMwhich takes 50 min max20:43
mriedemwfm20:43
EmilienMcool20:43
*** annegentle has joined #openstack-nova20:43
openstackgerritDan Smith proposed openstack/nova master: Don't ever cache service versions at zero  https://review.openstack.org/42840420:43
openstackgerritDan Smith proposed openstack/nova master: Work around deployments with unsupported API services  https://review.openstack.org/42841520:43
dansmithmriedem: ^ untested, but might workaround the issue for experimental and unsupported setups20:44
mriedemyup20:44
*** hieulq has quit IRC20:48
openstackgerritDan Smith proposed openstack/nova master: Don't ever cache service versions at zero  https://review.openstack.org/42840420:48
openstackgerritDan Smith proposed openstack/nova master: Clarify the language in the apache wsgi sample  https://review.openstack.org/42841720:48
dansmithmriedem: also recommend this, since there's clearly confusion around the word "experimental"20:48
dansmithand with that, I'm going to walk away for a bit.20:49
mriedemjaypipes: https://review.openstack.org/#/c/428375/20:49
EmilienMmriedem: the tripleo job has finished to run, http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/6ca334c/logs/subnode-2/var/log/nova/nova-compute.txt.gz20:49
EmilienMmriedem: let me know if logs are not enough I can increase it20:50
EmilienMmriedem: the error is here: http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/6ca334c/logs/subnode-2/var/log/nova/nova-api.txt.gz#_2017-02-02_20_42_17_09720:50
mriedemwish i could filter these logs based on level20:51
mriedemi expected to see different errors there20:51
mriedemlike real ERROR level issues20:51
edleafecdent: what about just changing objects.Allocation.used to a NonNegativeIntegerield?20:51
cdentedleafe: I thought we established those didn't exist?20:52
cdentedleafe: and also, in the past, even if we did have that we still wanted the trap at the schema level20:52
edleafecdent: doh!20:53
edleafeNonNegative allows zero20:53
* edleafe feels especially dumb20:53
*** takashin has joined #openstack-nova20:53
mriedembauzas: if you're still around https://review.openstack.org/#/c/428352/21:00
bauzasmriedem: I'm just writing a comment :)21:00
*** harlowja has quit IRC21:00
mriedemgreat21:00
bauzasmriedem: there is a py3 issue21:00
bauzasbut I'm verifying it21:01
mriedemwouldn't the py3 job fail on that?21:01
bauzasnot sure21:01
mriedemit's voting21:01
bauzasmriedem: do we have meeting ?21:01
edleafemeeting today?21:01
mriedemcrap21:01
mriedemyes21:01
edleafebauzas: jinx21:01
mriedemshort one21:01
bauzasmriedem: nevermind my comment on py321:04
*** salv-orlando has quit IRC21:05
*** liverpooler has quit IRC21:06
*** _ducttape_ has joined #openstack-nova21:06
*** raunak has joined #openstack-nova21:09
*** karimb has quit IRC21:09
tonybmriedem: I get the quote I'm just not sure how to interpret it in the context of this discussion21:10
*** ducttape_ has quit IRC21:10
*** karimb has joined #openstack-nova21:10
*** raunak has quit IRC21:10
*** dave-mccowan has quit IRC21:10
mriedemtonyb: that the reqs team can wait :)21:11
mriedemand enjoy some cake while they are waiting21:11
tonybmriedem: :)21:11
tonybthe cake is a lie!21:11
mriedemunless it's mikal's birthday21:12
*** slaweq has quit IRC21:12
*** raunak has joined #openstack-nova21:12
*** catintheroof has quit IRC21:14
*** catintheroof has joined #openstack-nova21:15
*** pshedimb has joined #openstack-nova21:16
pshedimbIf anyone has idea about multi node devstack (latest version) setup21:17
pshedimbMy comupte node is generating this error when creating an instance21:17
pshedimbERROR nova.compute.manager [req-392e02de-349a-4791-b390-4449ff466427 None None] No compute node record for host fmwpaxeon3921:17
pshedimbIntially I had placement config error21:17
pshedimbI copied placement config from controller to compute node21:18
pshedimbAnd now I'm having this error21:18
*** catintheroof has quit IRC21:19
*** Apoorva has joined #openstack-nova21:19
diana_clarkepshedimb: is placement-api in your ENABLED_SERVICES?21:22
*** raunak has quit IRC21:23
dansmiththat error isn't related to placement21:23
*** _ducttape_ has quit IRC21:23
dansmithand it's not an error if it's followed by "created a service record..."21:23
*** ducttape_ has joined #openstack-nova21:24
pshedimbdansmith, On controller placement-api is enabled21:25
pshedimbOn compute node, I copied placement-api related conf to nova.conf manually21:25
*** andrewbogott has quit IRC21:26
*** andrewbogott has joined #openstack-nova21:26
*** raunak has joined #openstack-nova21:26
mriedembauzas: thanks for https://review.openstack.org/#/c/428352/21:26
diana_clarkedansmith: oh, I just recall that being one of the answers to the same question here: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-02-01.log.html#t2017-02-01T13:08:3321:27
pshedimbdansmith, It is not followed by " "created a service record..."21:27
mriedembauzas: if you still have some stomach left, https://review.openstack.org/#/c/427499/21:27
*** srwilkers has joined #openstack-nova21:27
diana_clarkenevermind21:27
dansmithdiana_clarke: the compute node record is service and old-school RT related21:27
bauzasmriedem: I had an open question on https://review.openstack.org/#/c/428352/ about older allocation records21:28
bauzasmriedem: I'm not sure we auto-heal previous records21:28
mriedembauzas: probably a question for jaypipes21:28
mriedemor cdent21:28
bauzasyup21:28
bauzasfolks ?21:28
* cdent looks21:28
bauzasmriedem: dansmith: oh man, totally forgot to amend the nova-status check for filters21:29
mriedemi've got to run for awhile21:29
mriedembauzas: that's not stopship imo21:29
mriedemand we could backport if needed21:29
* mriedem is out21:29
*** mriedem has quit IRC21:29
bauzasmriedem: okay, go to your meeting21:29
cdentbauzas: we check allocations every periodic job in the resource tracker. if what's on the server is different from what we have now, we rewrite it21:29
bauzascdent: that's what I thought but I wanted to be sure21:30
cdentbauzas: you want me to leave that response on the review too, or okay?21:30
bauzascdent: so the patch above will autofix allocations after the 60-sec default period, cool21:30
cdentyes21:30
bauzascdent: nope, not needed, I +Wd it21:31
cdentgreat21:31
*** sapcc-bot has quit IRC21:33
*** carthaca_ has quit IRC21:33
*** Swami has quit IRC21:34
*** cleong has quit IRC21:34
*** Jeffrey4l has quit IRC21:35
*** sapcc-bot has joined #openstack-nova21:35
*** tpatzig_ has joined #openstack-nova21:35
*** Swami has joined #openstack-nova21:36
*** tpatzig_ has quit IRC21:37
*** ijw has joined #openstack-nova21:38
*** Jeffrey4l has joined #openstack-nova21:38
*** cdent has quit IRC21:38
*** dimtruck is now known as zz_dimtruck21:46
*** karimb has quit IRC21:46
*** dave-mccowan has joined #openstack-nova21:47
*** sambetts is now known as sambetts|afk21:49
*** avolkov has quit IRC21:53
*** srwilkers has left #openstack-nova21:53
*** salv-orlando has joined #openstack-nova21:57
*** karimb has joined #openstack-nova22:03
*** flwang has left #openstack-nova22:03
openstackgerritJiri Suchomel proposed openstack/nova master: Retry few times when creating the initial connection to libvirtd.  https://review.openstack.org/41692822:04
*** vsaienko has quit IRC22:04
*** vsaienko has joined #openstack-nova22:05
*** satyar_ has quit IRC22:06
*** thorst_ has quit IRC22:06
*** breitz has quit IRC22:14
*** salv-orl_ has joined #openstack-nova22:15
*** slaweq has joined #openstack-nova22:16
*** salv-orlando has quit IRC22:18
*** crushil has quit IRC22:19
*** esberglu has quit IRC22:24
*** esberglu has joined #openstack-nova22:24
*** mriedem has joined #openstack-nova22:24
mriedemi'm back22:24
*** jheroux has quit IRC22:25
*** xinli has quit IRC22:25
*** harlowja has joined #openstack-nova22:25
*** breitz has joined #openstack-nova22:26
*** ducttape_ has quit IRC22:26
EmilienMmriedem: https://review.openstack.org/#/c/428424/ is available for review when you have time (as PTL you need to sign-off)22:28
*** esberglu has quit IRC22:28
*** AAR has joined #openstack-nova22:29
*** AAR has left #openstack-nova22:29
*** tlian has joined #openstack-nova22:30
*** ducttape_ has joined #openstack-nova22:31
*** zz_dimtruck is now known as dimtruck22:31
*** thorst_ has joined #openstack-nova22:31
*** harlowja has quit IRC22:32
*** harlowja has joined #openstack-nova22:33
*** takashin has left #openstack-nova22:33
*** takashin has joined #openstack-nova22:34
*** Sukhdev has joined #openstack-nova22:34
mriedemEmilienM: i'm not going to dig into that one today probably22:35
*** thorst_ has quit IRC22:35
openstackgerritTakashi NATSUME proposed openstack/nova master: Add functional tests for cold migration to same host  https://review.openstack.org/41492622:36
melwittdansmith: do you know which part of the scheduler/placement code treats RequestSpec root_gb as 0 if BFV during scheduling? I haven't found it yet22:38
EmilienMmriedem: ok, just don't forget it please22:38
*** slaweq has quit IRC22:38
*** annegentle has quit IRC22:38
*** marst has quit IRC22:39
*** erlon has quit IRC22:40
dansmithmelwitt: here's where we request it: https://review.openstack.org/#/c/417961/41/nova/scheduler/filter_scheduler.py22:42
mriedemedleafe: are you working on the unit tests in https://review.openstack.org/#/c/428375/ ?22:42
dansmithmelwitt: but looking at that, we probably have to adjust that logic?22:42
*** ducttape_ has quit IRC22:43
dansmithmelwitt: like if is_bfv(), don't include root_gb in there22:43
melwittdansmith: yeah, I think so. right22:43
dansmithfor some reason I had in my head that the math there equaling zero means we don't ask for DISK_GB,22:44
dansmithbut if we want root_gb!=0 then we need one more if in there22:44
dansmithmelwitt: you want me to work that up?22:44
mriedemi swear i mentioned something about that to bauzas during that review22:45
mriedembecause the report client was checking is_bfv22:45
dansmithI did too22:45
dansmithbut in irc22:45
melwittyeah, the reporting is working I think. that means compute hosts won't advertise less disk available if they have bfv instances on them22:45
*** esberglu has joined #openstack-nova22:45
openstackgerritEd Leafe proposed openstack/nova master: Add check for invalid allocation amounts  https://review.openstack.org/41675222:45
dansmithyeah22:45
dansmiththat's been there for a while22:45
melwittbut that doesn't address the issue of asking for disk when during a request if bfv22:46
dansmithyep, it's a two-liner I think22:46
melwitt-when22:46
edleafemriedem: yeah - just updated https://review.openstack.org/416752first22:46
melwittdansmith: yeah, if you want to add that, sure. what's there currently is just making sure disk isn't requested if the flavor has 0 disk. but the bfv thing is flavor can have disk but it ends up on a volume and not on the compute host22:47
dansmithyep yep22:47
*** kaisers_ has quit IRC22:47
mriedemhttps://review.openstack.org/#/c/417961/22/nova/scheduler/filter_scheduler.py@15122:48
mriedemwas my comment / question i think22:48
dansmithgdi gerrit22:49
mriedemit seems like the ultimate agreement was we needed to check for bfv in there, and then didn't22:50
melwittokay, so bauzas was saying wait until cinder is telling the capacity and then check against that. in the future22:50
mriedemwhich would still be a different resource provider22:50
mriedemis what dan clarified22:51
*** mtanino has joined #openstack-nova22:51
melwittokay, so that means we have to pass by this RP then? sorry I don't know how this works22:51
mriedemyou'd have an aggregate for the shared storage pool22:51
mriedemwith compute RPs in that aggregate22:51
dansmiththat's just to get the math right,22:51
dansmithwe still don't want to charge it against the compute node if it's a volume22:51
-openstackstatus- NOTICE: Restarting gerrit due to performance problems22:51
melwittyeah22:51
dansmithwe have BDMs by this point, but I'm not sure they make it to the scheduler to make this easy ...22:52
*** salv-orl_ has quit IRC22:53
*** edmondsw_ has quit IRC22:53
mriedemedleafe: so https://review.openstack.org/#/c/428375/ depends on https://review.openstack.org/416752 ?22:54
mriedemedleafe: if so, can you rebase them to stack up that way?22:54
*** edmondsw has joined #openstack-nova22:54
edleafemriedem: I separated them at cdent's request22:54
*** marst has joined #openstack-nova22:56
mriedemsdague: jaypipes: if you're around, would be good to get this in https://review.openstack.org/#/c/427499/22:57
mriedemb/c anyone using nova-status when upgrading to ocata will be possibly busted w/o it22:57
mriedemwell, the status check will fail22:57
mriedemwhich we probably don't want to happen on it's maiden voyage22:57
*** burt has quit IRC22:58
*** edmondsw has quit IRC22:58
*** burgerk_ has joined #openstack-nova23:00
*** panda is now known as panda|zZ23:00
edleafeSo what's the preferred way of dealing with keystone auth exceptions when placement isn't configured? Seems that 726 tests hit the compute node destroy path, which will call placement in https://review.openstack.org/#/c/428375/23:02
edleafeI would prefer to configure it, if possible :)23:02
*** tbachman has quit IRC23:02
*** burgerk has quit IRC23:03
*** burgerk_ has quit IRC23:04
*** mdrabe has quit IRC23:06
*** jamesden_ has quit IRC23:06
openstackgerritDan Smith proposed openstack/nova master: Request zero root disk for boot-from-volume instances  https://review.openstack.org/42848123:08
dansmithmelwitt: working on tests now, but want to see a run of that ^ to make sure I've correctly guessed which of the ten BDM formats we have right there23:08
mriedemedleafe: the @safe_connect decorator23:10
dansmithif we had more than a couple hours I'd just rev the rpc api, but..23:10
edleafemriedem: heh, I was just trying that out...23:10
*** dimtruck is now known as zz_dimtruck23:11
mriedemedleafe: although you might need to use the scheduler report client / placement fixture that bauzas used in his scheduler tests23:12
edleafemriedem: we'll see23:12
mriedemdansmith: root_bdm() can return None23:15
mriedemso that will likely explode for a bunch of tests23:15
dansmithmriedem: I know23:15
dansmithduh23:15
dansmithsheesh23:15
dansmithjust wanted to make sure YOU knew23:15
dansmithand 6 is the number of tests23:15
*** zz_dimtruck is now known as dimtruck23:15
mriedemplus23:17
mriedemPLUS23:17
mriedemaccording to https://github.com/openstack/nova/blob/master/nova/objects/block_device.py#L364 it has to be per-instance23:17
dansmithmriedem: I don't think that applies here,23:18
mriedemi meant tempest tests btw23:18
dansmithbecause this is a BDM that we replicate for each,23:18
dansmithnot a set of bdms from multuple instances23:18
dansmithmultiple23:18
mriedemoh right23:18
mriedemcopies the list, and then sets the instance uuid on each bdm in thelist23:19
dansmithyar23:19
mriedemmultuple is py4 btw23:19
mriedemwhich is an openstack goal for T23:19
dansmithbadumpum23:19
*** thorst_ has joined #openstack-nova23:20
dansmithugh, requestspec is the new bdm23:20
*** trevormc has quit IRC23:21
*** owalsh-afk has joined #openstack-nova23:24
dansmithalso I'm pretty sure we're serializing to legacy, then passing it to a method that unserializes it23:24
dansmithbut I don't even want to think about it right now23:24
melwittyeah, there's a small piece in between that can't yet handle objects, so it does that23:24
dansmithno,23:24
dansmithI think it's direct23:24
dansmithlemme get links23:24
melwittoh. I know there's a thing where it serializes to pass to a specific method, then after it comes back it turns it back into an object. or something23:25
dansmithhttps://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L86523:25
dansmithhttps://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L59523:25
dansmithyeah23:25
*** takashin has left #openstack-nova23:25
openstackgerritDan Smith proposed openstack/nova master: Request zero root disk for boot-from-volume instances  https://review.openstack.org/42848123:25
dansmithmriedem: tests never failed if you push before jenkins finished23:26
melwittI wonder if I could bolt my sweet functional test onto that patch to further test it. or I could stack it over yours https://review.openstack.org/#/c/355091/8/nova/tests/functional/test_boot_from_volume.py23:27
dansmiththat is the worst test I've ever seen23:27
* dansmith just wants to balance jaypipes' gushing23:28
*** dave-mccowan has quit IRC23:28
melwittI was about to be like *gasp* how could you say that!23:28
dansmithheh23:28
*** baoli has quit IRC23:28
dansmithyeah, I say stack it on for now at least23:29
melwittcool. this is exciting23:29
* dansmith thinks melwitt needs her meter recalibrated23:29
melwittI feel like I'm the champion of boot from volume and root_gb = 0 at this point23:30
gmannmriedem: alex_xu this one from kevin on release note (we should have this on etherpad) - https://review.openstack.org/#/c/427978/23:31
mriedemgmann: alex_xu split the release note out https://review.openstack.org/#/c/428285/23:32
mriedemi'm reviewing and cleaning it up now23:32
gmannmriedem: cool, that also fine23:32
gmannmriedem:  i did not see doc patch update as per comments  - https://review.openstack.org/#/c/421760/23:32
mriedemdansmith: is that like a 5 second rule?23:33
*** gouthamr has quit IRC23:33
dansmithmriedem: similar, but with less salmonella23:33
gmannmriedem:  if we are taking that too, i can work on that23:33
*** esberglu has quit IRC23:33
*** Sukhdev has quit IRC23:33
mriedemgmann: api-ref can be in pike23:33
mriedemimo23:33
*** tblakes has quit IRC23:33
mriedemapi-ref is built from master anyway23:33
gmannyea, that makes no hurry23:34
melwittmriedem: that reminds me, the cells v2 step-by-step stuff is also not related to deadline right? since it's devref23:34
*** owalsh-afk is now known as owalsh23:34
*** tlian has quit IRC23:35
mriedemmelwitt: let's say yes23:37
melwittcools23:38
*** owalsh is now known as owalsh-afk23:38
melwittmy test didn't work -- NoValidHost. I wonder if I need to do something to enable placement in functional test23:39
openstackgerritMatt Riedemann proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42828523:39
mriedemgmann: see how this looks ^ i touched up the wording and fancied it up a bit23:39
mriedemmelwitt: placement fixture?23:39
* melwitt looks23:39
mriedemhttps://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L117523:39
melwittthanks23:40
melwittI <3 fixtures23:40
jaypipesmriedem: done.23:40
mriedemthanks23:41
mriedemoh also, i might just +W this myself if no one else does https://review.openstack.org/#/c/428344/23:41
dansmithmriedem: I'll get it23:42
mriedemthanks23:42
dansmithmriedem: we should also put the wsgi clarification in, I need to de-parent it23:42
mriedemok23:42
jaypipesmriedem: lemme review real quick.23:42
mriedemthis is going to be the most gd docstastic release ever23:42
melwittmriedem: I looked at that earlier and wondered if you should note that [cells] options aren't required for cells v2 in the first blurb too23:42
mriedemshrug23:43
mriedemit's approved now so i have no incentive23:43
mriedem:)23:43
openstackgerritDan Smith proposed openstack/nova master: Clarify the language in the apache wsgi sample  https://review.openstack.org/42841723:43
dansmithmriedem: ^23:43
mriedemwipped as bauzas would say23:43
dansmithheh23:44
openstackgerritEd Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted  https://review.openstack.org/42837523:44
edleafemriedem: jaypipes: ^^ Had to rebase this on https://review.openstack.org/#/c/404472 to get safe_connect23:47
*** salv-orlando has joined #openstack-nova23:48
*** amotoki has quit IRC23:48
melwittlooks like it's still trying to claim the flavor root_gb23:48
*** vsaienko has quit IRC23:49
dansmithmelwitt: in your test or somewhere else?23:51
dansmithmelwitt: or do you mean on the compute node?23:51
melwittdansmith: running my test I see it making the claim with the instance flavor root_gb on the compute host23:52
dansmithmelwitt: that's expected right?23:52
dansmiththe claim stuff knows nothing about this,23:52
melwittdansmith: yeah but it will fail because the compute host free disk figure will kick it out23:52
dansmithbut we should be reporting the right thing to placement, and not considering it on scheduling23:52
*** salv-orlando has quit IRC23:52
dansmithuntil we remove that stuff, right?23:53
melwittso it can't schedule to a compute host that isn't reporting enough local disk to fit the flavor23:53
dansmithso maybe we need the claims bit of your patch?23:53
*** claudiub has quit IRC23:53
*** karimb has quit IRC23:53
dansmithI wish we didn't though, since that's a bunch of extra pieces :/23:54
melwittit appears so. it's pretty ugly though https://review.openstack.org/#/c/355091/8/nova/compute/resource_tracker.py@15523:54
dansmithyeah23:54
melwittI dunno what resource providers plan is for fixing this23:55
dansmithI don't know that we keep the claims stuff past ocata23:55
dansmithI'm not sure though23:55
dansmithmelwitt: your stuff in claims all says "remove this after resource providers"23:55
melwittyeah, that wasn't based on knowing how it would be solved. my understanding was that RP fixes all the things23:56
mriedemi'm going to go on a limb and say,23:56
mriedemi really like joe walsh23:56
dansmithwell, when we get allocations in the scheduler it definitely goes away, AFAIK23:56
melwittokay. then I should change my TODOs to say that. I didn't know what specifically would make those changes obsolete in the future23:57
dansmithman, that's a lot of copying the instance object every time we report resources23:58
mriedemgmann: found a problem in the reno23:58
mriedemfixing quick23:58
*** dtp has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!