Tuesday, 2016-09-06

*** f10d4 has quit IRC00:22
*** mriedem has joined #openstack-nova00:26
*** mriedem has quit IRC00:27
*** felix23ma has quit IRC00:29
*** vishwanathj has joined #openstack-nova00:35
*** felix23ma has joined #openstack-nova00:40
*** thorst has joined #openstack-nova00:40
*** hoangcx has joined #openstack-nova00:45
*** thorst has quit IRC00:47
*** thorst has joined #openstack-nova00:47
*** thorst has quit IRC00:47
*** ssurana has joined #openstack-nova00:48
*** kaisers1 has joined #openstack-nova00:51
*** kaisers has quit IRC00:51
*** kaisers has joined #openstack-nova00:52
*** kaisers2 has quit IRC00:52
*** markvoelker has joined #openstack-nova00:53
*** thorst has joined #openstack-nova00:54
*** lixiaoy1 has joined #openstack-nova01:04
*** thorst has quit IRC01:05
openstackgerritMaciej Szankin proposed openstack/nova: Deprecate all topic conf options  https://review.openstack.org/35160801:06
*** matias has quit IRC01:11
*** yamahata has joined #openstack-nova01:21
*** READ10 has joined #openstack-nova01:24
*** kevinz has joined #openstack-nova01:28
*** hoangcx has quit IRC01:32
*** wangqun has joined #openstack-nova01:43
*** liuyulong_ is now known as liuyulong01:48
*** baoli has joined #openstack-nova01:51
*** baoli_ has joined #openstack-nova01:53
*** ijw has joined #openstack-nova01:55
*** akshai has joined #openstack-nova01:56
*** baoli has quit IRC01:56
openstackgerritChen Fan proposed openstack/nova: pci: nova fail to boot when wrong setting in pci_whitelist  https://review.openstack.org/35760401:58
*** code-R has joined #openstack-nova02:00
*** ijw has quit IRC02:01
*** EinstCrazy has joined #openstack-nova02:01
*** baoli_ has quit IRC02:03
*** code-R_ has joined #openstack-nova02:03
*** wangqun has quit IRC02:03
*** hoangcx has joined #openstack-nova02:03
*** wangqun_ has joined #openstack-nova02:03
*** ijw has joined #openstack-nova02:03
*** ijw has quit IRC02:04
*** ijw has joined #openstack-nova02:04
*** thorst has joined #openstack-nova02:05
*** phuongnh has joined #openstack-nova02:05
*** code-R has quit IRC02:06
*** ijw has quit IRC02:10
openstackgerritNguyen Phuong An proposed openstack/nova: [api-ref]: Correcting server_groups_list parameter's type.  https://review.openstack.org/36555402:13
*** thorst has quit IRC02:13
*** armax has quit IRC02:13
*** junjie has quit IRC02:14
*** code-R_ has quit IRC02:17
*** rfolco has quit IRC02:18
*** agireud has quit IRC02:18
*** kevinz has quit IRC02:18
*** kevinz has joined #openstack-nova02:19
*** agireud has joined #openstack-nova02:25
*** akshai_ has joined #openstack-nova02:29
*** salv-orl_ has joined #openstack-nova02:29
*** agireud has quit IRC02:30
*** akshai has quit IRC02:32
*** salv-orlando has quit IRC02:32
*** RuiChen has joined #openstack-nova02:33
*** annp has joined #openstack-nova02:34
*** ijw has joined #openstack-nova02:36
*** thorst has joined #openstack-nova02:40
*** agireud has joined #openstack-nova02:40
*** ijw has quit IRC02:40
*** vishwanathj is now known as vishwanathj_zzz02:47
*** amotoki has joined #openstack-nova02:48
*** bkopilov has quit IRC02:50
*** amotoki has quit IRC02:50
openstackgerritLi Yingjun proposed openstack/nova: Adds view builders for hypervisors controller  https://review.openstack.org/33528202:51
*** akshai_ has quit IRC02:52
*** markvoelker has quit IRC02:53
*** ijw has joined #openstack-nova03:00
*** sudipto has joined #openstack-nova03:00
*** sudipto_ has joined #openstack-nova03:00
*** hoangcx has quit IRC03:01
*** houming has joined #openstack-nova03:04
*** ijw has quit IRC03:05
*** amotoki has joined #openstack-nova03:07
*** hoangcx has joined #openstack-nova03:12
*** ssurana has quit IRC03:13
*** amotoki has quit IRC03:18
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox from compute/test_volumes.py  https://review.openstack.org/32325703:40
*** READ10 has quit IRC03:42
*** ijw has joined #openstack-nova03:44
*** adisky has joined #openstack-nova03:47
*** ijw has quit IRC03:48
openstackgerritJianghua Wang proposed openstack/nova: XenAPI: resolve the fetch_bandwidth failure  https://review.openstack.org/36583803:50
*** diga has joined #openstack-nova03:51
*** coolsvap_ has joined #openstack-nova03:52
*** amotoki has joined #openstack-nova03:54
openstackgerritEli Qiao proposed openstack/nova: Libvirt: Correct PERF_EVENTS_CPU_FLAG_MAPPING  https://review.openstack.org/36584203:54
*** thorst has quit IRC03:58
*** thorst has joined #openstack-nova03:58
*** ijw has joined #openstack-nova04:00
*** thorst has quit IRC04:00
*** prateek has joined #openstack-nova04:01
*** Sukhdev has joined #openstack-nova04:01
*** prateek has quit IRC04:03
*** thorst has joined #openstack-nova04:03
*** sdake has joined #openstack-nova04:05
*** sudipto_ has quit IRC04:06
*** sudipto has quit IRC04:06
*** thorst has quit IRC04:08
*** ijw has quit IRC04:09
*** thorst has joined #openstack-nova04:10
*** lixiaoy1 has quit IRC04:12
*** thorst has quit IRC04:15
*** bkopilov has joined #openstack-nova04:24
*** sdake has quit IRC04:25
*** ijw has joined #openstack-nova04:30
murali__Hello04:31
murali__Someone have a look https://ask.openstack.org/en/question/96444/nova-docker-issue-in-launching-the-container/04:31
*** moshele has joined #openstack-nova04:34
*** links has joined #openstack-nova04:39
*** ijw has quit IRC04:40
*** ijw has joined #openstack-nova04:41
*** ijw has quit IRC04:42
*** harlowja_at_home has joined #openstack-nova04:42
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in compute/test_neutron_security_groups.py  https://review.openstack.org/32309304:44
*** harlowja_at_home has quit IRC04:47
*** lixiaoy1 has joined #openstack-nova04:47
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in unit/fake_processutils.py  https://review.openstack.org/32423904:48
*** sdake has joined #openstack-nova04:50
*** ekuris has joined #openstack-nova04:51
*** sudipto has joined #openstack-nova04:53
*** sudipto_ has joined #openstack-nova04:53
*** dikonoor has joined #openstack-nova04:55
*** thorst has joined #openstack-nova04:55
*** su_zhang has joined #openstack-nova04:57
*** thorst has quit IRC04:59
*** sdake has quit IRC05:01
*** irenab has joined #openstack-nova05:04
*** oanson has joined #openstack-nova05:05
*** irenab_ has joined #openstack-nova05:06
*** dikonoor has quit IRC05:06
*** thorst has joined #openstack-nova05:08
*** irenab has quit IRC05:08
*** irenab_ is now known as irenab05:08
*** sdake has joined #openstack-nova05:10
*** ratailor has joined #openstack-nova05:11
*** mamandle has joined #openstack-nova05:11
*** wangqun_ has quit IRC05:11
*** wangqun_ has joined #openstack-nova05:11
*** sdake has quit IRC05:17
*** diogogmt has quit IRC05:19
*** su_zhang has quit IRC05:21
*** jaosorior has joined #openstack-nova05:21
*** su_zhang has joined #openstack-nova05:22
*** thorst has quit IRC05:25
*** prateek has joined #openstack-nova05:26
*** su_zhang has quit IRC05:26
*** ijw has joined #openstack-nova05:29
*** thorst has joined #openstack-nova05:29
*** zzzeek has quit IRC05:30
*** sridharg has joined #openstack-nova05:31
*** ijw_ has joined #openstack-nova05:31
*** nkrinner_afk is now known as nkrinner05:32
*** ijw has quit IRC05:32
*** thorst has quit IRC05:33
*** ijw_ has quit IRC05:35
*** ijw has joined #openstack-nova05:38
*** EinstCrazy has quit IRC05:40
*** ijw has quit IRC05:40
*** diogogmt has joined #openstack-nova05:41
*** liverpooler has quit IRC05:42
*** ijw has joined #openstack-nova05:43
*** lixiaoy1 has quit IRC05:45
*** lixiaoy11 has joined #openstack-nova05:45
*** ijw has quit IRC05:46
*** sudipto has quit IRC05:47
*** sudipto_ has quit IRC05:47
*** ijw has joined #openstack-nova05:47
*** hoonetorg has quit IRC05:48
*** mjura has joined #openstack-nova05:51
*** EinstCrazy has joined #openstack-nova05:52
*** sudipto has joined #openstack-nova05:53
openstackgerritNguyen Phuong An proposed openstack/nova: Update info of machine-processable description  https://review.openstack.org/36542605:55
*** unicell1 has joined #openstack-nova05:55
*** unicell has quit IRC05:56
*** ijw has quit IRC05:57
*** ijw has joined #openstack-nova05:57
murali__Someone have a look here : https://ask.openstack.org/en/question/96444/nova-docker-issue-in-launching-the-container/05:58
murali__I am checking on this issue now05:58
*** mjura has quit IRC05:58
*** mjura has joined #openstack-nova05:59
openstackgerrithuan proposed openstack/nova: xenapi: support the hotplug of a neutron port  https://review.openstack.org/36550506:00
*** ijw has quit IRC06:01
*** rcarrillocruz has quit IRC06:02
*** ijw has joined #openstack-nova06:02
*** ijw has quit IRC06:03
openstackgerritEli Qiao proposed openstack/nova: Libvirt: Correct PERF_EVENTS_CPU_FLAG_MAPPING  https://review.openstack.org/36584206:03
*** paul-carlton has joined #openstack-nova06:04
*** flip214 has left #openstack-nova06:09
*** ijw has joined #openstack-nova06:10
*** hoonetorg has joined #openstack-nova06:12
*** raunak has joined #openstack-nova06:14
*** ttrumm has joined #openstack-nova06:21
*** EinstCrazy has quit IRC06:21
*** sean-k-m1 has joined #openstack-nova06:22
*** EinstCrazy has joined #openstack-nova06:23
*** sean-k-mooney has quit IRC06:24
slaweq_mdbooth: hello, I pushed yesterday clean-up patch which we talked about: https://review.openstack.org/#/c/365756/06:25
slaweq_please take a look if You will have time06:25
*** ijw has quit IRC06:25
*** dimtruck is now known as zz_dimtruck06:27
*** ihrachys has joined #openstack-nova06:29
*** claudiub|2 has joined #openstack-nova06:29
*** claudiub has joined #openstack-nova06:31
*** raunak_ has joined #openstack-nova06:34
*** raunak has quit IRC06:34
*** raunak_ is now known as raunak06:34
*** claudiub|2 has quit IRC06:34
*** felix23ma has quit IRC06:42
*** felix23ma has joined #openstack-nova06:43
*** andreas_s has joined #openstack-nova06:45
*** zz_dimtruck is now known as dimtruck06:48
*** pcaruana has joined #openstack-nova06:48
*** felix23ma has quit IRC06:54
*** markvoelker has joined #openstack-nova06:54
*** dimtruck is now known as zz_dimtruck06:58
*** markvoelker has quit IRC06:59
*** tesseract- has joined #openstack-nova07:00
*** EinstCra_ has joined #openstack-nova07:00
*** EinstCrazy has quit IRC07:00
*** ijw has joined #openstack-nova07:00
*** zz_dimtruck is now known as dimtruck07:00
*** ijw has quit IRC07:00
*** ijw has joined #openstack-nova07:00
*** hoangcx has quit IRC07:01
*** felix23ma has joined #openstack-nova07:02
*** hoangcx has joined #openstack-nova07:04
*** sudipto has quit IRC07:05
*** wangqun has joined #openstack-nova07:05
*** wangqun_ has quit IRC07:06
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in api/openstack/compute/test_floating_ips.py  https://review.openstack.org/32266807:06
*** Sukhdev has quit IRC07:06
*** Sukhdev has joined #openstack-nova07:06
*** EinstCra_ has quit IRC07:09
*** matrohon has joined #openstack-nova07:09
*** dimtruck is now known as zz_dimtruck07:10
*** liverpooler has joined #openstack-nova07:10
*** sahid has joined #openstack-nova07:10
*** Sukhdev has quit IRC07:11
*** ihrachys has quit IRC07:12
*** ihrachys has joined #openstack-nova07:12
*** slaweq__ has joined #openstack-nova07:14
*** ttrumm has quit IRC07:14
*** lixiaoy11 has quit IRC07:14
*** ttrumm has joined #openstack-nova07:15
*** lixiaoy1 has joined #openstack-nova07:16
*** EinstCrazy has joined #openstack-nova07:17
*** slaweq_ has quit IRC07:17
*** raunak has quit IRC07:18
ratailoropenstack-nova, openstack pl review https://review.openstack.org/#/c/312887/07:21
*** ihrachys has quit IRC07:23
*** ljxiash has joined #openstack-nova07:24
*** chrichip has quit IRC07:25
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in  api/openstack/compute/test_remote_consoles.py  https://review.openstack.org/32310207:28
*** felix23ma has quit IRC07:32
*** ijw has quit IRC07:35
*** gszasz has joined #openstack-nova07:35
*** ijw has joined #openstack-nova07:36
*** mamandle has quit IRC07:39
*** felix23ma has joined #openstack-nova07:41
prateekchal07:43
*** zenoway has joined #openstack-nova07:47
*** daemontool has joined #openstack-nova07:54
*** paul-carlton has quit IRC07:59
*** lpetrut has joined #openstack-nova08:02
*** yassine has joined #openstack-nova08:04
*** yassine is now known as Guest6574508:04
*** Guest65745 is now known as yassou08:05
*** lpetrut has quit IRC08:07
*** jistr is now known as jistr|dentist08:07
*** EinstCrazy has quit IRC08:11
openstackgerritChris Dent proposed openstack/nova: [placement] Add some tests ensuring unicode resource provider name  https://review.openstack.org/36568808:12
*** mamandle has joined #openstack-nova08:14
openstackgerritHa Van Tu proposed openstack/nova: Remove unused parameters  https://review.openstack.org/36592908:15
*** derekh has joined #openstack-nova08:15
*** lucas-dinner is now known as lucasagomes08:16
*** ijw has quit IRC08:17
*** shangxdy has joined #openstack-nova08:17
*** markvoelker has joined #openstack-nova08:19
*** code-R has joined #openstack-nova08:20
*** mvk has quit IRC08:22
*** code-R_ has joined #openstack-nova08:22
*** markus_z has joined #openstack-nova08:22
*** paul-carlton has joined #openstack-nova08:22
*** ihrachys has joined #openstack-nova08:23
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in compute/test_security_groups.py  https://review.openstack.org/32312208:23
*** markvoelker has quit IRC08:24
*** hshiina has joined #openstack-nova08:24
*** code-R has quit IRC08:25
*** gongysh has quit IRC08:25
*** gongysh has joined #openstack-nova08:26
*** EinstCrazy has joined #openstack-nova08:27
*** Jeffrey4l has quit IRC08:28
*** EinstCrazy has quit IRC08:28
*** EinstCrazy has joined #openstack-nova08:28
*** mnestratov has joined #openstack-nova08:29
*** salv-orlando has joined #openstack-nova08:29
openstackgerritZhenyu Zheng proposed openstack/nova: Trival fix a typo  https://review.openstack.org/36593808:31
*** salv-orl_ has quit IRC08:32
*** roeyc has joined #openstack-nova08:32
*** sudipto has joined #openstack-nova08:34
*** danpawlik1 has joined #openstack-nova08:34
danpawlik1Hello, if you have some free time pls check https://review.openstack.org/#/c/314928/ :)08:35
openstackgerritMaxim Nestratov proposed openstack/nova: libvirt: add supported vif types for virtuozzo virt_type  https://review.openstack.org/36594308:35
*** ababich has joined #openstack-nova08:35
*** ijw has joined #openstack-nova08:39
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in virt/libvirt/test_imagecache.py  https://review.openstack.org/32371708:44
*** ecelik has quit IRC08:47
*** ecelik has joined #openstack-nova08:48
*** hshiina has quit IRC08:50
*** mvk has joined #openstack-nova08:51
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in compute/test_server_actions.py  https://review.openstack.org/32318608:52
*** lixiaoy1 has quit IRC08:57
*** EinstCrazy has quit IRC08:59
*** hoangcx has quit IRC08:59
*** ecelik has quit IRC09:00
*** ecelik has joined #openstack-nova09:00
*** zz_dimtruck is now known as dimtruck09:01
*** EinstCrazy has joined #openstack-nova09:01
*** ecelik has quit IRC09:04
*** ecelik has joined #openstack-nova09:04
*** ijw has quit IRC09:07
*** rwmjones has quit IRC09:08
*** dimtruck is now known as zz_dimtruck09:10
*** rwmjones has joined #openstack-nova09:12
*** danpb has joined #openstack-nova09:16
*** ijw has joined #openstack-nova09:18
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox int unit/virt/xenapi/image/test_utils.py  https://review.openstack.org/34262609:18
*** gongysh has quit IRC09:19
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in virt/xenapi/image/test_vdi_through_dev.py  https://review.openstack.org/34268209:21
*** gongysh has joined #openstack-nova09:21
*** jistr|dentist is now known as jistr09:23
*** mfedosin has joined #openstack-nova09:25
*** gongysh has quit IRC09:26
*** daemontool has quit IRC09:29
*** lpetrut has joined #openstack-nova09:30
*** lpetrut has quit IRC09:30
*** lpetrut has joined #openstack-nova09:31
*** shangxdy has quit IRC09:31
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in unit/virt/xenapi/test_vm_utils.py (1)  https://review.openstack.org/34405409:35
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in unit/virt/test_block_device.py  https://review.openstack.org/34259509:37
*** roeyc has quit IRC09:38
*** mvk has quit IRC09:38
*** cdent has joined #openstack-nova09:39
*** PaulMurray has joined #openstack-nova09:39
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in unit/cells/test_cells_rpc_driver.py  https://review.openstack.org/33310109:39
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox int unit/virt/xenapi/stubs.py  https://review.openstack.org/33484809:43
*** sdake has joined #openstack-nova09:45
openstackgerritDao Cong Tien proposed openstack/nova: Stop using mox in unit/virt/xenapi/test_vmops.py (2/2)  https://review.openstack.org/34401009:46
*** zenoway has quit IRC09:46
*** zenoway has joined #openstack-nova09:47
*** zenoway has quit IRC09:51
*** sean-k-m1 is now known as sean-k-mooney09:53
*** annp has quit IRC09:57
*** mvk has joined #openstack-nova09:58
*** mnestratov has quit IRC10:00
*** zenoway has joined #openstack-nova10:00
*** zenoway has quit IRC10:01
*** zenoway has joined #openstack-nova10:01
openstackgerritGhanshyam Mann proposed openstack/nova: Body Verification of os-aggregates.inc  https://review.openstack.org/36602310:03
*** zz_dimtruck is now known as dimtruck10:06
*** Sukhdev has joined #openstack-nova10:07
mdboothftersin: Sorry, you got a load of my frustration this morning :) Thanks for all the reviews.10:11
* mdbooth tries to keep it bottled, but doesn't always quite succeed.10:12
mdboothThat's healthy, right?10:12
*** Sukhdev has quit IRC10:13
*** phuongnh has quit IRC10:14
*** liuyulong is now known as liuyulong|away10:15
*** riccardo has joined #openstack-nova10:16
ftersinbdbooth: just found a bug of ephemeral resizing, went to find a solution, remembered that you discussed it here, and start to investigate the current state of your work10:16
*** roeyc has joined #openstack-nova10:18
mdboothftersin: Yeah, hasn't worked ever, I think10:18
mdboothThat patch half fixes it. I think I note that it's not fixed in one of the unit tests, don't I?10:19
*** ijw has quit IRC10:20
mdboothIncidentally, remember that 'disk.local' is never created for new instances, and hasn't been for releases covering several years at least.10:20
riccardohi guys, when I google for something like, how to connect to DB, it always displays results for openstack administrators. How can I search for contributer/programmer stuff? Any idea?10:20
mdboothWhich basically means that 'disk.local' is rare in the wild, and quite possibly extinct10:20
mdboothA single ephemeral disks created today is 'disk.eph0'10:21
ftersin'disk.local' is a clear example of the unit tests quality :/10:22
mdboothYup10:22
ftersinephN is used since bdm v210:22
mdboothIndeed10:22
openstackgerritRajesh Tailor proposed openstack/nova: Fix instance delete stuck in deleting task_state  https://review.openstack.org/29449110:24
*** sahid has quit IRC10:28
ftersinmdbooth: "That patch half fixes it" - which patch do you mean?10:28
mdboothYou commented on it in a review one of my patches10:28
mdboothIt's one of the _create_image patches10:29
*** kaisers2 has joined #openstack-nova10:29
*** sahid has joined #openstack-nova10:29
*** kaisers1 has quit IRC10:29
*** kaisers1 has joined #openstack-nova10:30
mdboothftersin: This one: https://review.openstack.org/#/c/331118/15/nova/virt/libvirt/driver.py10:30
*** Oku_OS is now known as Oku_OS-away10:30
*** kaisers has quit IRC10:30
*** Oku_OS-away is now known as Oku_OS10:31
mdboothHalf fixed == not fixed, of course :)10:32
* mdbooth is just trying to find the code which mangles bdms in that call path. I've found it before, but it's currently eluding me.10:32
*** ecelik has quit IRC10:32
mdboothIt's either in compute api, conductor, or compute manager.10:32
*** PaulMurray has quit IRC10:33
ftersinmdboot: well, i think that if nova with this fix can resize or add a single ephemeral, we may amend the case with numerous ephemerals10:33
mdboothAnyway, the instance we're passed has had its metadata resized, but its bdms have not been resized.10:33
*** sdake has quit IRC10:33
mdboothBecause the resize code uses bdms, it means we resize, but to the same size as before10:33
mdboothAnyway, with that change it will at least iterate over ephemerals, but it'll uselessly resize them to their existing size.10:34
mdboothWe need to also fix the caller.10:34
mdboothI'm determined to find that code again :)10:34
ftersinif your series is blocked by newton code freeze, does it make sence to push https://review.openstack.org/#/c/346263 at the moment?10:35
*** ttrumm_ has joined #openstack-nova10:35
*** zenoway has quit IRC10:36
mdboothftersin: I have been +1 on several patches already which will conflict with my series for exactly that reason.10:36
mdboothYes, expedient bug fixes are always good.10:36
*** zenoway has joined #openstack-nova10:36
*** ecelik has joined #openstack-nova10:37
mdboothThey're also easier to backport.10:37
mdboothftersin: I've already reviewed that patch and was in favour of it, but it needs a test.10:38
*** roeyc has quit IRC10:38
slaweq__mdbooth: hello10:38
*** wangqun has quit IRC10:38
slaweq__mdbooth: do You know who from core team I could ask to check https://review.openstack.org/#/c/363926/ ?10:38
*** tbachman has quit IRC10:39
*** ttrumm has quit IRC10:39
ftersinmdboot: but you just sayed that it requires bdm update as well, right?10:39
mdboothAnybody you can get :) Bug fixes are the priority right now, so hopefully you'll get some attention.10:39
*** code-R_ has quit IRC10:39
ftersinso that not test only, but some other updates...10:40
mdboothftersin: Ah, yes, I didn't read the whole context. Yes, I don't think that will adress the whole problem.10:40
*** zenoway has quit IRC10:40
*** ecelik has quit IRC10:41
ftersini've asked cfriesen, if he wants to push it to newton and legacy releases. waiting for an answer10:41
ftersinok, thanks10:41
*** cdent has quit IRC10:42
openstackgerritAlberto Planas proposed openstack/nova: Add migrate_uri for invoking the migration  https://review.openstack.org/35655810:43
*** zenoway has joined #openstack-nova10:44
*** PaulMurray has joined #openstack-nova10:44
*** roeyc has joined #openstack-nova10:45
*** HenryG_ is now known as HenryG10:46
*** lyarwood is now known as lyarwood_10:46
*** baoli has joined #openstack-nova10:46
*** lyarwood_ is now known as lyarwood10:46
*** esracelik has joined #openstack-nova10:47
*** zengine has quit IRC10:48
mdboothftersin: It's in ComputeManager._finish_resize10:49
mdboothThis is where compute manager resizes the instance's metadata10:49
mdboothHowever, it doesn't update bdms, which are the only thing we can reasonably look at for ephemerals10:50
mdboothWe need to also poke bdms in there before calling driver.finish_migration()10:50
*** _gryf has quit IRC10:50
*** zzzeek has joined #openstack-nova10:51
mdboothWith that, and the fix the both cfriesen and I have done independently, resize of ephemerals should work.10:51
*** _gryf has joined #openstack-nova10:51
ftersinbut whether we must distinguish between bdms added implicitly by flavor and explicitly by user/snapshot?10:52
sfinucanmriedem: When you're online, could I get the -2 removed from this centralize-config-options patch? https://review.openstack.org/#/c/357061/10:53
ftersini.e. if a user run instance on a flavor with 20 GB ephemeral, specified 10 GB only in bdm, and then resize the instance to 40 GB ephemeral flavor, what to do with the ephemeral?10:54
*** EinstCrazy has quit IRC10:54
mdboothdanpawlik1: I'd recommend against block adding core reviewers, btw. You risk annoying them :/10:55
danpbindeed, reviews with people block-added go to the bottom of my priority list :-)10:55
mdboothftersin: Whatever we do, that's nothing to do with the driver.10:56
*** zzzeek has quit IRC10:56
mdboothBut yes, that's a good question. There's also:10:56
danpawlik1mdbooth: sorry for that10:56
mdboothFlavor specifies 20GB ephemeral, user adds 2 ephemerals 10GB each.10:56
mdboothUser resizes to flavor with 40GB ephemeral. What does the resize do?10:57
ftersinmdbooth: this is another case - how to split inreased space onto exising ephemerals10:57
*** zzzeek has joined #openstack-nova10:57
danpbmdbooth: i'd just split it proportionally across the disks10:57
*** baoli has quit IRC10:57
danpbthere's nothing much else you can do that's sane, given the API limitations10:57
riccardohi guys, when I google for something like, "how to connect to DB", it always displays results for openstack administrators. How can I search for contributer/programmer stuff? Any idea?10:57
mdboothftersin: We discussed this incredibly briefly ages ago, and the answer at the time was: it's always been broken anyway, so don't sweat it. Just fix the case where there's a single, full size ephemeral disk and leave anything else for a future design change.10:58
*** kaisers2 has quit IRC10:58
mdboothHowever, ephemerals, especially multiple ephemerals, have been broken for so long in so many ways, I'm not personally convinced anybody's using them.10:58
*** kaisers1 has quit IRC10:59
mdboothdanpb: I think it's safer to leave it alone in that case, tbh.10:59
*** kaisers has joined #openstack-nova10:59
*** kaisers1 has joined #openstack-nova10:59
*** baoli has joined #openstack-nova10:59
mdboothSeems like a lot of hassle for an unused feature.10:59
*** baoli has quit IRC11:01
openstackgerritClaudiu Belu proposed openstack/nova-specs: Adds spec for instance live resize  https://review.openstack.org/14121911:02
ftersinmdbooth: but if a feature is broken, who uses it? how customers do live now with no ephemeral resize? do workaround via snapshots?11:03
mdboothThat's the thing, maybe they don't use it. What's the use case for multiple ephemerals anyway?11:04
mdboothWhy would a user ever request an ephemeral smaller than what they've paid for?11:04
*** Jeffrey4l has joined #openstack-nova11:05
mdboothI wonder if we've designed features which users don't need or use.11:05
danpbmdbooth: i can see people wanting separate disks for separate usage scenarios11:05
*** cdent has joined #openstack-nova11:05
ftersini would want to split if i wanted to separate fast space between different services or data purposes11:06
danpbmdbooth: eg, one might contain /var/www/ while the other contained /home11:06
mdboothdanpb: Maybe if their workload uses raw disk?11:06
danpbyes you could use partitions, but using separate disks means you avoid contention in the i.o stack so theoretically could get better performance11:06
danpbi agree though that use separate disks is a niche use case11:06
mdboothIt's still the same disk on the backend :)11:07
danpbmdbooth: that's a mere impl detail11:07
mdboothSo qemu might use a slightly different path, but the spinning rust/magical cloud bits are the same.11:07
danpbfrom an API pov it makes sense11:07
ftersini would want to limit my ephemeral, if i did not need it so much, but needed vcpus for example, and given flavor set did not give me a big choice, but i needed to honor disk quota11:07
danpbits just that our impl isn't taking advantage of it11:07
danpbmdbooth: you will at least have separate i/o queues in the guest os side which can improve performance in a multi vcpu guest11:08
mdboothdanpb: That's somewhere I wanted to go after persistent storage metadata, btw. That will allow us to specify different backends for different disks.11:08
mdboothRight now, though, there's nothing.11:08
danpbyeah, people have requested separate backends for local disks before11:08
mdboothftersin: Anyway, I'm not disagreeing with you, just expounding the virtues of expediently fixing the thing with an obvious fix, and leaving the rest for another time.11:10
mdboothftersin: How important is ephemeral resize to you, btw?11:10
mdboothHave you got users complaining about this?11:10
ftersinthat's thing i agree too11:11
*** PaulMurray has quit IRC11:11
ftersini'm blocked with this on scalio driver11:11
mdboothEphemeral resize?11:11
ftersinnot directly, but yes11:12
mdboothWell, it doesn't work with any other driver, so why should it block you?11:12
ftersinscaleio volumes need to be mapped to a compute host11:12
danpbyeah, i'm not seeing why instance resize is a blocker in general11:12
*** ljxiash has quit IRC11:13
ftersinwhen the resizing moves the instance to another host, volumes must be remapped to that host11:13
*** PaulMurray has joined #openstack-nova11:13
*** roeyc has quit IRC11:14
ftersinimagebackend's create_image is the only place where i can do it11:14
ftersinso i need create_image to be called inside finish_migration11:15
ftersinfor each disk, including ephemerals11:15
mdboothI see no problem with that.11:15
ftersineven if size of ephemeral is not changed11:15
mdboothDoes cfriesen's patch fix it for you?11:15
ftersinhope for that, but have not checked yet11:16
mdboothHowever, I think that's the wrong place to do that.11:16
mdboothI'm going to open a can of worms here, stay with me, but this is also broken for Rbd.11:17
mdboothWhich, iiuc, is the same model.11:17
mdboothi.e. remote storage, which we're going to connect to from the new host and not actually copy. Right?11:17
ftersinrbd does not require mapping operation, though11:17
mdboothRight, but it does exclude it from the copy.11:17
mdboothWhich I'm assuming you also have to be careful to do.11:18
ftersinwhich copy operation you mean?11:18
mdboothmigrate_disk_and_power_off11:18
mdboothYou'll presumably need to ensure that your scaleio disks aren't copied there if they're just going to be moved11:18
mdboothInvolving some kind of shared storage negotiation11:18
*** cdelatte has joined #openstack-nova11:19
mdboothSo, for scaleio do you need to attach the disks to the *host*?11:19
ftersinrbd and scaleio are shared block storage, no one need to copy data of them11:19
mdboothSo, for Rbd there's no host operation required because qemu talks rbd directly11:20
mdboothBut iiuc, the scaleio thing is more akin to attaching a volume11:20
mdboothIs it persistent across host reboots?11:20
ftersinmoreover, a lot of driver code does copy and other things with files, which comes from libvirt domain via get_instance_disk_info11:20
ftersinbut rbd devices are not returned from there11:21
mdboothYeah, that's gross.11:21
ftersini did the same for scaleio ones11:21
mdboothThat's how they're exluded from the copy.11:21
mdboothI'm just specifically thinking about the attach thing.11:21
mdboothIf it's not persistent across host reboots, then you can't do it in create_image anyway.11:22
*** ljxiash has joined #openstack-nova11:22
ftersinyes. otherwise it do not work at all11:22
mdboothYou need another hook.11:22
*** lucasagomes is now known as lucas-hungry11:22
ftersinscaleio mapping is persistent11:22
*** rtheis has joined #openstack-nova11:22
mdboothOk, cool.11:23
mdboothSo, now I think I understand your comment about ensure_backing.11:23
*** roeyc has joined #openstack-nova11:23
mdboothBecause that's basically exactly the same hook you want for scalio.11:23
mdboothNamely, don't copy anything, just ensure the disk exists at the remote end.11:24
*** bkopilov has quit IRC11:27
*** ijw has joined #openstack-nova11:27
riccardohow can i read and write to MariaDB? Is there some code in oslo or do I need to write everything from the scratch?11:29
ftersinbut.. since i already added hooks to map and unmap all volumes, probably it is better just to add another hook to map all volumes in finish_migration11:29
*** amotoki_ has joined #openstack-nova11:29
*** ijw has quit IRC11:29
*** ijw has joined #openstack-nova11:30
*** amotoki has quit IRC11:31
ftersinmdbooth: exactly. i added a hook to map all instance disks to pre_live_migration, so i just add the same to finish_migration, and do not worry about resizing11:31
ftersinexcellent solution. thanks for the useful conversation11:32
*** ljxiash has quit IRC11:33
ftersindanpb: about people question. did anyone ask to have cinder as the ephemeral backend?11:36
*** mfedosin has quit IRC11:37
danpbi don't know11:37
*** sdague has joined #openstack-nova11:39
riccardohow can I read in the database how many hosts are online?11:39
*** PaulMurray has quit IRC11:41
*** kaisers has quit IRC11:41
*** fragatina has joined #openstack-nova11:42
*** kaisers has joined #openstack-nova11:42
*** kaisers1 has quit IRC11:43
*** kaisers1 has joined #openstack-nova11:43
*** fragatina has quit IRC11:43
*** mamandle has quit IRC11:45
*** kaisers2 has joined #openstack-nova11:46
*** kaisers3 has joined #openstack-nova11:47
*** kaisers1 has quit IRC11:48
*** kaisers has quit IRC11:49
*** mfedosin has joined #openstack-nova11:51
*** salv-orl_ has joined #openstack-nova11:52
*** salv-orlando has quit IRC11:52
*** code-R has joined #openstack-nova11:54
*** sudipto has quit IRC11:55
*** kaisers3 has quit IRC11:55
*** kaisers has joined #openstack-nova11:55
*** code-R_ has joined #openstack-nova11:58
*** code-R has quit IRC12:01
*** daemontool has joined #openstack-nova12:04
*** gcb has quit IRC12:04
sdaguecdent: are you planning on cleaning up the log string nits in - https://review.openstack.org/#/c/365015/ ?12:04
*** ijw has quit IRC12:05
cdentsdague: yes, but I can do that later if you think we should get that in sooner than later12:06
cdentthe number of 409s that are happening should be improved by some of the stuff above that patch12:06
cdentthe client side doesn't have any magical generation counting so it sometimes has to 409 to know that it needs to get a new generation to rectify itself12:07
cdentsdague: I was waiting for the US to wake up before making any more adjustments on that stack so that they can happen in a batch.12:08
*** tbachman has joined #openstack-nova12:08
sdaguecdent: ok, fair.12:09
*** rfolco has joined #openstack-nova12:09
openstackgerritKashyap Chamarthy proposed openstack/nova: libvirt: fix detach volume race condition  https://review.openstack.org/29051012:09
*** daemontool_ has joined #openstack-nova12:10
kashyapmdbooth: The above is just a rebase to Git first.  Now reworking the feedback from reviewers12:10
kashyaps/Git/Git master/12:10
*** daemontool has quit IRC12:12
*** fragatina has joined #openstack-nova12:13
*** tbachman_ has joined #openstack-nova12:13
*** edmondsw has joined #openstack-nova12:13
sdaguecdent: even at top of stack the 409ing seems pretty high on allocations - http://logs.openstack.org/19/365719/1/check/gate-tempest-dsvm-neutron-placement-full-ubuntu-xenial-nv/a1c6cc4/logs/apache/placement-api.txt.gz12:14
sdaguewhich is definitely leading to a lot of 404s on deletes, so explains why the allocations might not line up with reality12:14
*** tbachman has quit IRC12:15
*** tbachman_ is now known as tbachman12:15
cdentsdague: I don't think the allocations handling on the client side is very smart yet, so we'll need to do some analysis across the placement log and the n-cpu log to find out what's going on with a bit more detail. I'll suck down the n-cpu log and see if any patterns leap out, but we'll probably want Dan too.12:19
*** rlrossit has joined #openstack-nova12:19
*** ttrumm_ has quit IRC12:19
cdent(I've got a meeting in a few minutes that's going to delay any progress for me)12:20
*** liverpooler has quit IRC12:21
sdagueyep, no prob12:21
cdent"Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider '4a25cc16-0b8a-4759-a29f-c17c6650dd39'. The requested amount would exceed the capacity" makes no sense so should be a clue12:21
*** oanson has quit IRC12:22
*** fragatina has quit IRC12:22
*** ttrumm has joined #openstack-nova12:24
cdentone thing sdague looks like we still need to change things so that placement api starts before n-cpu. I think we talked about that last week, but I forgot about it.12:24
sdagueok, though shouldn't we be handling that in code?12:25
sdagueI thought we were handling the connect issue and it would just mean a missed request12:25
*** woodster_ has joined #openstack-nova12:26
cdentit would, and it's fine. I don't think it is causing problems, it's just a "wasted" error.12:26
*** PaulMurray has joined #openstack-nova12:26
*** clenimar has quit IRC12:26
*** lucas-hungry is now known as lucasagomes12:27
cdentsdague: there's a 147 difference instances being allocated. is that around what we expect? do we even know?12:30
*** thorst has joined #openstack-nova12:30
*** jaypipes has joined #openstack-nova12:30
*** clenimar has joined #openstack-nova12:31
*** xinliang has quit IRC12:32
sdaguethat sounds ball park correct12:32
sdagueI was pretty sure it was > 10012:32
*** mamandle has joined #openstack-nova12:34
sdaguecdent: which run are you looking at?12:35
cdentthe one you pasted above12:35
*** mdrabe has joined #openstack-nova12:37
cdentthat's using awks, greps, sorts uniq on the n-cpu log12:37
*** tongli has joined #openstack-nova12:40
*** EinstCrazy has joined #openstack-nova12:43
*** xinliang has joined #openstack-nova12:44
*** baoli has joined #openstack-nova12:44
*** ttrumm has quit IRC12:44
*** baoli_ has joined #openstack-nova12:45
*** figleaf is now known as edleafe12:45
*** ttrumm has joined #openstack-nova12:46
*** baoli has quit IRC12:48
*** jaosorior has quit IRC12:48
*** jaosorior has joined #openstack-nova12:49
*** mmedvede has quit IRC12:49
*** jaosorior has quit IRC12:49
*** pcaruana has quit IRC12:50
*** tbachman_ has joined #openstack-nova12:51
*** ratailor has quit IRC12:51
*** jaosorior has joined #openstack-nova12:51
*** tbachman has quit IRC12:51
*** tbachman_ is now known as tbachman12:51
*** sudipto has joined #openstack-nova12:51
*** sridharg has quit IRC12:56
*** vishwanathj_zzz is now known as vishwanathj12:57
cdentsdague: when I get off this call I'm going to make sure that the capacity violation tests on the api side are sufficient so we can be sure to bisect the problem properly12:58
*** diga has quit IRC12:59
*** edand has joined #openstack-nova13:00
*** tbachman has quit IRC13:00
*** zengine has joined #openstack-nova13:00
*** vladikr has joined #openstack-nova13:00
*** houming has quit IRC13:01
*** bkopilov has joined #openstack-nova13:03
*** mingyu has joined #openstack-nova13:06
sdaguecdent: are we handling cpu overcommit fine in the placement API?13:07
*** tbachman has joined #openstack-nova13:08
cdentsdague: that's what allocation ratio means in the inventory, and what I'm about to check is that the tests for that are sufficient to be sure it is correct13:08
cdentthen we can work back from there to where the two concepts comingle13:08
*** sridharg has joined #openstack-nova13:09
*** mingyu_ has joined #openstack-nova13:09
openstackgerritJianghua Wang proposed openstack/nova: XenAPI: resolve the fetch_bandwidth failure  https://review.openstack.org/36583813:10
*** mmedvede has joined #openstack-nova13:11
*** mingyu has quit IRC13:12
*** mnestratov has joined #openstack-nova13:15
*** Guest36215 has quit IRC13:15
openstackgerritJianghua Wang proposed openstack/nova: XenAPI: resolve the fetch_bandwidth failure  https://review.openstack.org/36583813:15
*** akshai has joined #openstack-nova13:18
openstackgerritAlex Szarka proposed openstack/nova-specs: Typo: very instead of vary  https://review.openstack.org/36612513:18
*** ttrumm_ has joined #openstack-nova13:22
*** ljxiash has joined #openstack-nova13:23
*** tblakes has joined #openstack-nova13:24
*** eharney has joined #openstack-nova13:25
*** ttrumm has quit IRC13:25
*** tblakes_ has joined #openstack-nova13:25
*** tbachman has quit IRC13:25
*** hemanthm|away is now known as hemanthm13:27
markus_zbauzas: Can I interest you in a scheduler bug fix? https://review.openstack.org/#/c/352344/13:29
*** tblakes has quit IRC13:29
*** tblakes_ is now known as tblakes13:30
*** irenab has quit IRC13:31
*** pwnall1 has joined #openstack-nova13:32
*** tbachman has joined #openstack-nova13:32
*** irenab has joined #openstack-nova13:32
bauzasmarkus_z: cool :)13:33
*** mriedem has joined #openstack-nova13:34
*** mingyu_ has quit IRC13:34
*** zengine has quit IRC13:35
*** markvoelker has joined #openstack-nova13:36
openstackgerritSlawek Kaplonski proposed openstack/nova: libvirt: Clean up block device job status reporting code  https://review.openstack.org/36575613:36
*** andymaier has joined #openstack-nova13:37
*** mingyu has joined #openstack-nova13:37
*** pwnall1 has quit IRC13:38
*** pwnall1 has joined #openstack-nova13:38
*** sridharg has quit IRC13:39
riccardohi guys, can anyone help? how can i get data from MariaDB?13:39
*** shangxdy has joined #openstack-nova13:40
riccardois there some python example code?13:40
openstackgerritMJWurtz proposed openstack/nova: Refresh info_cache after deleting floating IP  https://review.openstack.org/35749413:42
*** ttrumm_ has quit IRC13:43
*** yamahata has quit IRC13:44
*** ttrumm has joined #openstack-nova13:44
openstackgerritPrateek Arora proposed openstack/nova: Handle ConnectFailure from keystone when neutron is down  https://review.openstack.org/32770613:45
*** cleong has joined #openstack-nova13:45
*** TravT has joined #openstack-nova13:45
*** roeyc has quit IRC13:46
*** dimtruck is now known as zz_dimtruck13:47
*** roeyc has joined #openstack-nova13:48
*** ttrumm has quit IRC13:49
openstackgerritPrateek Arora proposed openstack/nova: Handle ConnectFailure from keystone when neutron is down  https://review.openstack.org/32770613:49
prateekmriedem, can you please have a look at https://review.openstack.org/#/c/294513/ whenever you have some time ?13:50
openstackgerritChris Dent proposed openstack/nova: [placement] Add additional allocation tests  https://review.openstack.org/36614413:50
cdentsdague: that ^ is more tests, but they didn't reveal anything on the server side13:51
*** mlavalle has joined #openstack-nova13:51
cdentsdague I'm going to add some more logging to the resource tracker so we can see what it is writing more clearly. Or I can put it on the server side if that seems better?13:51
*** gongysh has joined #openstack-nova13:52
sdagueif it's on the server side we could also add a debug log which dumps the current allocation state of the world on every request, to see what the system thinks is there13:52
mdboothftersin: Hey, if I asked you to finish the cleanup in https://review.openstack.org/#/c/334609/, would it get done?13:53
mdboothftersin: There are 2 identical misuses of that function in driver, but you only clean up one.13:53
mdboothFixing 1 is obviously better than fixing none :)13:53
mdboothBut fixing both is best.13:54
* cdent spins up devstack13:54
ftersini remove the second in the next patch13:54
mdboothftersin: Ah, ok.13:54
mdboothHadn't seen that.13:55
*** rlrossit has quit IRC13:56
PaulMurrayLive migration meeting in a few minutes on #openstack-meeting-3 PaulMurray paul-carlton2 eliqiao shaohe_feng pkoniszewski jlanoux yuntongjin mdbooth kashyap kmartin claudiub tobasco tdurakov andreas_s abhishekk mrhillsman raj_singh13:56
*** browne has joined #openstack-nova13:57
*** burt has joined #openstack-nova13:57
ftersinmdbooth: i abandoned my cleaning patch and promoted https://review.openstack.org/#/c/337056 instead13:57
*** annegentle has joined #openstack-nova13:58
riccardohi, i have 5 compute nodes and one controller in place. Is there something I can help you guys?13:58
mdboothriccardo: This is a channel for nova development. See /topic for user discussion.13:59
riccardosure, I would like to help with python coding14:00
*** luis5tb has joined #openstack-nova14:00
openstackgerritGleb Stepanov proposed openstack/nova: Skip malformed cookies  https://review.openstack.org/34609014:00
*** anush has joined #openstack-nova14:00
sfinucanriccardo: aye, but the focus should be on specific aspects of nova14:00
sfinucanfor things like how to access a database there are many better forums14:01
riccardook, just let me know how i can help :-) The only constraint: I am a beginner.14:01
sfinucanriccardo: https://wiki.openstack.org/wiki/How_To_Contribute14:01
sfinucanriccardo: bug fixing would be a very good start14:01
*** krtaylor has joined #openstack-nova14:02
riccardoit hasn't to do something with my DB problem. i just wanna contribute anyhow14:02
*** ashyoung has joined #openstack-nova14:02
riccardobug fix sounds good14:02
riccardoi check the site...14:03
bauzasalaski: around ?14:03
bauzasalaski: back from holiday ?14:03
alaskibauzas: yep14:03
bauzasalaski: so, we have a bug report about updating the instance name https://bugs.launchpad.net/nova/+bug/162024814:04
openstackLaunchpad bug 1620248 in OpenStack Compute (nova) "Can't rename instance right after creation (regression)" [High,In progress] - Assigned to Sylvain Bauza (sylvain-bauza)14:04
bauzasalaski: if you remember our discussion we had 2 weeks ago...14:05
*** thorst has quit IRC14:05
alaskiyeah14:05
openstackgerritSylvain Bauza proposed openstack/nova: WIP: Update BuildRequest if instance currently being scheduled  https://review.openstack.org/36574014:05
*** thorst has joined #openstack-nova14:05
bauzasalaski: so, I wonder why we should provide for that...14:06
bauzasalaski: I wrote a quickly patch for modifying then the BuildRequest14:06
*** ametts has joined #openstack-nova14:06
bauzasalaski: https://review.openstack.org/36574014:06
alaskiI'm amazed that anyone is doing that rather than just passing the name in the original request14:06
bauzasalaski: agreed, but I think it could be understanding14:07
openstackgerritdane-fichter proposed openstack/nova: Use cursive for signature verification  https://review.openstack.org/35123214:08
*** marst has quit IRC14:08
bauzasalaski: so, either we don't accept people updating the instance until it's scheduled, or we need to update the BuildReq14:08
alaskiright. I favor the former, but if we can't change that without a microversion there's no point14:09
*** amotoki_ has quit IRC14:09
bauzasalaski: I'm pretty sure some people would like the latter14:09
*** hshiina has joined #openstack-nova14:10
alaskiUpdating the buildreq is mostly fairly simple. But we have to account for the race where instance creation could be happening as the buildreq is updated.14:10
bauzasalaski: see my change then14:10
*** liang_ has joined #openstack-nova14:10
alaskilooking, but I need a sec to digest this14:11
mriedemcdent: we don't need to use the generic-resource-pools bp link in the new commits for bug fixes and unit test adds14:11
mriedemcdent: since we're past FF it's weird to see new patches for a closed bp14:11
alaskibauzas: I think you can simplify this a lot by checking if instance.id is set. If it is the instance exists in a db and instance.save() will work14:11
*** ducttape_ has joined #openstack-nova14:12
cdentmriedem: ah, okay, I was using that to make sure they get linked somewhere in case people start doing a "where's the stuff I'm suppose to look at". but will stop now.14:12
*** ekuris has quit IRC14:12
mriedemcdent: i'd probably just associate them to a bug14:13
mriedemsince there should be a bug for the overcommit stuff you and dansmith are trying to hunt down14:13
bauzasalaski: sure, but that will won't work until we delay to persist the instance down in the conductor, right?14:13
bauzasalaski: I mean, here, we already have a created instance14:14
*** thorst has quit IRC14:14
alaskibauzas: currently there are two instances, one from the instance table and buildreq.instance. if .id is set you know you have the one from the instance table14:14
*** ababich has quit IRC14:14
*** ducttape_ has quit IRC14:15
bauzasalaski: right, but I was thinking the build_req.instance was having the id set14:15
alaskiyou might get buildreq.instance even though there is an instance in the instances table, until instance creation moves to the conductor14:15
*** moshele has quit IRC14:16
alaskibauzas: nope. id is only set if the instance has been written to the db but we persist the buildreq one before doing that14:16
*** amotoki has joined #openstack-nova14:17
bauzasalaski: oh right14:17
bauzasalaski: I just saw that14:17
*** michchap has quit IRC14:17
bauzasI was first thinking about looking at instance._context but lookup at instance.id is better14:17
alaskiyeah. .id is the definitive guide to whether or not it's from the instances table14:18
*** ducttape_ has joined #openstack-nova14:18
*** ababich has joined #openstack-nova14:18
openstackgerritPrateek Arora proposed openstack/nova: Handle ConnectFailure from keystone when neutron is down  https://review.openstack.org/32770614:18
openstackgerritChristopher Martin proposed openstack/nova: Fix for Nova API Dev Guide  https://review.openstack.org/36616214:20
*** mamandle has quit IRC14:20
*** takedakn has joined #openstack-nova14:20
*** gongysh has quit IRC14:20
*** diogogmt has quit IRC14:20
*** links has quit IRC14:21
*** marst has joined #openstack-nova14:21
*** syjulian_ has joined #openstack-nova14:21
*** shangxdy has quit IRC14:23
prateekmriedem, thanks for the reviewing  https://review.openstack.org/#/c/327706/,even though i had requested a review for  https://review.openstack.org/#/c/294513/ , in which i had worked on some of your comments and it's been pending for a long time :)14:23
*** ametts has quit IRC14:24
mriedemprateek: i'm in 2 meetings atm, so i'll review what i can review when i can get the time :)14:24
*** BjoernT has joined #openstack-nova14:25
*** zigo has quit IRC14:25
prateekmriedem, sure, thanks !14:25
*** michchap has joined #openstack-nova14:26
*** akshai has quit IRC14:27
*** zz_dimtruck is now known as dimtruck14:29
*** ametts has joined #openstack-nova14:29
*** salv-orlando has joined #openstack-nova14:29
*** prateek has quit IRC14:30
*** felix23ma has quit IRC14:30
*** brian_price has joined #openstack-nova14:32
*** takedakn has quit IRC14:32
*** zigo has joined #openstack-nova14:32
*** salv-orl_ has quit IRC14:32
*** zigo is now known as Guest3914714:33
*** syjulian_ has quit IRC14:33
mriedemmdbooth: if you can point me at where functional changes start for the imagebackend series that would save me some time14:33
mdboothmriedem: Sec14:33
*** syjulian_ has joined #openstack-nova14:33
openstackgerritSamuel Matzek proposed openstack/nova: Reschedule boot when attach volume fails  https://review.openstack.org/24650514:34
mdboothDepends how you define functional :) This is the first non-test cleanup: https://review.openstack.org/#/c/333272/14:34
mdboothHowever, its a NFC14:34
cdentsdague, dansmith I'm struggling to replicate the allocation problem seen in temest, locally. And now I have to go to an appt. Will check in afterwards to see what you guys know and what I should do. Can one of you make the bug (if it doesn't already exist) suggested by mriedem a few minutes back. biab.14:35
*** jamesdenton has joined #openstack-nova14:35
mdboothmriedem: The next one is functional: https://review.openstack.org/#/c/331115/14:35
sdaguecdent: sure, how parallel is your local environment?14:35
tojuvonesdague, dansmith, johnthetubaguy: If some of you could take a glance at this as familiar. I also have POC quite ready against the spec: https://review.openstack.org/31051014:35
mriedemmdbooth: national football conference?14:35
mdboothmriedem: Are you suggesting that I continue pushing on the test changes? I'd pretty much abandoned them for now, but I'll happily keep going for a bit.14:35
mdboothNon Functional Change :)14:36
mriedemmdbooth: at this point with 2 weeks left i'd rather avoid big test refactors14:36
mriedemmdbooth: especially if you're looking at fixing that post-copy network issue14:36
mdboothmriedem: That's what I thought. I'll leave it.14:36
cdentsdague: not very, which is probably part of the problem, but I'm struggling to come up with a mental model that would explain things, even when parallel. I think my picture of "what's wrong" is incomplete, so mostly I'm just trying to narrow reality a bit. If the bug is able to explain the main issue, that'll probably help.14:36
sdaguemdbooth: yeh, typically post freeze we don't merge any non test / non docs code without a bug14:36
*** cdent has quit IRC14:37
*** akshai has joined #openstack-nova14:37
*** amotoki has quit IRC14:38
*** jdurgin1 has joined #openstack-nova14:38
*** Guest39147 has quit IRC14:39
sdaguemriedem / dansmith - do we have some reasonable query / interface on allocations to get a debug log dump of what the state of the world looks like?14:39
*** zigo_ has joined #openstack-nova14:42
mdboothsfinucan: https://bugs.launchpad.net/nova/+bug/160799614:43
openstackLaunchpad bug 1607996 in OpenStack Compute (nova) "Live migration does not update numa hugepages info in xml" [High,In progress]14:43
*** krtaylor has quit IRC14:43
mdboothAre you definitely not looking at that any more?14:43
mriedemsdague: idk14:44
sfinucanmdbooth: Not at present, no. I didn't quite get the problem14:44
*** claudiub has quit IRC14:44
sfinucanwas he referring to the host XML or guest?14:44
sfinucan*she14:44
*** sdake has joined #openstack-nova14:44
mriedemsdague: we could pull from the DB in a DNM debug patch if needed, directly from the RT i mean14:44
mriedemif necessary14:44
mdboothsfinucan: Don't know. The bug was/is under discussion in #openstack-meeting-314:44
*** raj_singh_ has joined #openstack-nova14:45
mriedembut i don't know what the GET looks like from the placement API14:45
mriedemi thought there was talk on friday that there wasn't a GET14:45
sfinucanmdbooth: under what context?14:45
mdboothLive migration meeting14:45
mdboothOngoing14:45
sfinucanah14:45
sfinucanafraid there isn't much I can contribute. I can look at it again, but I'd need to ask for more info first14:45
openstackgerritGleb Stepanov proposed openstack/python-novaclient: Add evacuate host method  https://review.openstack.org/34917914:46
mdboothsfinucan: NP14:47
*** daemontool_ has quit IRC14:47
*** zigo_ has quit IRC14:47
*** rrecio_ has joined #openstack-nova14:47
*** daemontool has joined #openstack-nova14:48
*** zigo_ has joined #openstack-nova14:48
mriedemrpodolyaka: sdague: jaypipes: i'm going to -2 the separate placement API DB change for newton https://review.openstack.org/#/c/362766/ - we're past FF and since we aren't testing this i don't think it would be good to land it14:52
*** mamandle has joined #openstack-nova14:52
*** annegentle has quit IRC14:53
sdaguemriedem: +114:53
mriedemmdbooth: probably a good one for you https://bugs.launchpad.net/nova/+bug/162034114:53
openstackLaunchpad bug 1620341 in OpenStack Compute (nova) "Removing unused base images removes backing files of active instances" [Undecided,New]14:53
*** prateek has joined #openstack-nova14:53
mdboothTitle sounds fun14:54
* mdbooth reads14:54
*** raj_singh_ has quit IRC14:54
mdboothmriedem: Paraphrase: image cache manager is broken14:54
mdboothThat's about the level of detail there14:55
mriedem+NFS14:55
mriedembut yeah14:55
* mdbooth has a quick look at ICM's shared storage handling14:55
mdboothI'm pretty sure it stores a file containing every host which accesses the shared store14:56
mdboothAnd filters based on all of them14:56
mdboothBut IIRC there's nothing to stop cleanup running concurrently on different hosts.14:56
*** tbachman has quit IRC14:57
kashyapmdbooth: Randomly chiming in, there are multiple (2 IIRC) config variables related to removing unused base image files.  You plan to remove / rework some of them as part of your larger work on Image Cache?14:59
*** _ducttape_ has joined #openstack-nova14:59
mdboothkashyap: I have no larger work on Image Cache!14:59
* kashyap stops making mdbooth responsible for imaginary stuff14:59
mdboothI occasionally fix bits of it when I'm compelled to read enough of it to understand some of it.14:59
*** pradk has quit IRC15:00
kashyapNoted :-)15:00
*** mjura has quit IRC15:01
*** ashyoung has quit IRC15:01
openstackgerritSean Dague proposed openstack/nova: WIP: additional logging to debug allocations  https://review.openstack.org/36618615:02
sdaguemriedem: it will be interesting to see what the run logs for that look like15:02
*** rodrigods has quit IRC15:02
*** rodrigods has joined #openstack-nova15:02
*** ducttape_ has quit IRC15:02
markus_zPaulMurray: apologies for not responding in the live migrate meeting, I was in another meeting too and didn't see it.15:03
PaulMurraymarkus_z, np15:04
PaulMurraymarkus_z, does anyone look after the lm bugs ?15:04
* PaulMurray expects I should be doing it15:04
*** edand has quit IRC15:05
markus_zPaulMurray: I'm not aware of anyone, tbh.15:05
markus_zPaulMurray: I keep them alive and wait for my turn for reviews15:05
PaulMurrayI will go through them in the next couple of days15:05
*** rmart04_ has joined #openstack-nova15:06
markus_zPaulMurray: One was a regression. The other one an older issue.15:06
*** syjulian_ has quit IRC15:06
markus_zPaulMurray: thanks15:06
openstackgerritMatt Riedemann proposed openstack/nova: libvirt: improve logging for shared storage check  https://review.openstack.org/36514015:06
openstackgerritMatt Riedemann proposed openstack/nova: nfs fix for xenial images  https://review.openstack.org/32946615:06
rpodolyakamriedem: oh, I missed the fact the original change was reverted while I was on vacation :P15:06
rpodolyakait's a bit disappointing of course, but yeah, I agree it may be too late now15:06
*** tbachman has joined #openstack-nova15:08
*** annegentle has joined #openstack-nova15:08
openstackgerritSylvain Bauza proposed openstack/nova: WIP: Update BuildRequest if instance currently being scheduled  https://review.openstack.org/36574015:08
bauzasalaski: just updated based on your comments15:08
bauzasalaski: I think we still need to have lots of conditionals for making sure we don't forget a possible race condition15:09
*** andreas_s has quit IRC15:09
*** gouthamr has joined #openstack-nova15:10
*** amotoki has joined #openstack-nova15:11
markus_zdansmith: FYI: fixes nova-manage privilege escalation: https://review.openstack.org/#/c/352866/15:11
*** thorst has joined #openstack-nova15:11
markus_zmaybe you could have a look at it15:11
*** diogogmt has joined #openstack-nova15:11
alaskibauzas: I don't think you can avoid the race condition without a lock. And I think a first patch should fix the simple issue of updating buildreq.instance and then deal with races in a second patch.15:12
*** gouthamr_ has joined #openstack-nova15:12
*** sayalilunkad has quit IRC15:13
*** sayalilunkad has joined #openstack-nova15:13
*** hemnafk has quit IRC15:13
bauzasalaski: well...15:14
*** yamahata has joined #openstack-nova15:14
*** PaulMurray has quit IRC15:15
*** gouthamr has quit IRC15:15
bauzasalaski: you mean, just amending the BuildReq without really checking whether we should rather use the cellsv2 contextmanager?15:16
bauzasalaski: I thought it was better to add it too, but okaty15:16
alaskibauzas: I mean first solve the issue of not being able to update the buildreq at all. check for instance.id: if it exists instance.update; instance.save, it it doesn't exist pull the buildreq and update it.15:17
alaskithen worry about the small race15:17
*** nkrinner is now known as nkrinner_afk15:18
*** hemna has joined #openstack-nova15:18
*** thorst has quit IRC15:19
*** pradk has joined #openstack-nova15:20
*** zigo_ is now known as zigo15:20
*** fragatina has joined #openstack-nova15:20
*** mamandle has quit IRC15:24
*** cdent has joined #openstack-nova15:26
*** EinstCrazy has quit IRC15:26
*** fragatina has quit IRC15:27
*** gokrokve has joined #openstack-nova15:27
*** mriedem1 has joined #openstack-nova15:27
*** armax has joined #openstack-nova15:29
*** mriedem has quit IRC15:30
*** thorst has joined #openstack-nova15:30
*** whenry has joined #openstack-nova15:31
*** tbachman has quit IRC15:31
*** sudipto has quit IRC15:31
*** rmart04_ has quit IRC15:33
*** mingyu has quit IRC15:34
sdaguedansmith: I think I figured out the vcpu issue15:34
dansmithyeah?15:34
sdaguemath is hard15:35
*** Jeffrey4l has quit IRC15:35
dansmithin my local db we are reporting the proper allocation ratio, so that kinda shoots my theory15:35
dansmithunless we're not factoring it into the actual capacity calculation15:36
openstackgerritSean Dague proposed openstack/nova: WIP: calculate capacity correctly  https://review.openstack.org/36621415:36
sdaguedansmith: I think that's the fix15:36
*** rmart04 has joined #openstack-nova15:36
dansmithhmm, why?15:36
dansmiththat would mean we're effectively scaling the reserve right?15:37
*** PaulMurray has joined #openstack-nova15:37
dansmithi.e. we reserve only that amount from the scaled amount15:37
*** paul-carlton has quit IRC15:37
*** tonytan4ever has joined #openstack-nova15:38
sdagueok, well what do we expect total, reserved, allocation_ratio to be there?15:38
openstackgerritPrateek Arora proposed openstack/nova: Handle ConnectFailure from keystone when neutron is down  https://review.openstack.org/32770615:38
*** tonytan_brb has joined #openstack-nova15:39
dansmithreserved is often zero or small15:39
danpbso consider total capacity is 8 GB and we want 1 GB reserved and alloc ratio is 2 - with old logic we'd report 14 GB, but with fixed logic we're report 15 GB15:39
danpbthe new logic sdague suggests does make more sense to me15:39
sdagueok, well it's definitely capacity where we go funky15:39
*** andymaier has quit IRC15:39
sdaguelet me get some more debug in there to figure that out15:39
dansmithbut we should be 4*16 in the jenkins workers15:39
dansmithand I doubt we're starting that many instances in parallel right?15:40
bauzasare you guys talking about how we reserve space in the RT ?15:40
*** tblakes has quit IRC15:40
bauzasthat's just a silly math15:40
danpbof course our reserved memory calcs all have a level of crapiness about them15:40
sdaguedansmith: we should be 4 * 1615:40
sdaguehowever, any time we go over 8 we seem to fail15:40
sdagueon the capacity check15:40
dansmithsdague: reserved for vcpus is zero right?15:40
dansmithit is on my machine15:41
dansmithso I don't see why your math makes any difference15:41
*** pcaruana has joined #openstack-nova15:42
*** rmart04 has quit IRC15:42
*** tonytan4ever has quit IRC15:42
openstackgerritSivasathurappan Radhakrishnan proposed openstack/nova: Remove context object in oslo.log method  https://review.openstack.org/31710915:43
sdaguedansmith: ok, let me figure out what's really going on here15:43
sdagueI'm adding some more debug there15:43
*** sdake has quit IRC15:44
dansmithokay15:44
bauzasdansmith: sdague: just in case you haven't seen that yet, we just reserve a bit of space by this https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L88615:45
bauzasdansmith: sdague: then we look at the free ram https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L88815:45
prateekcdent, sahid , can you please take a look when you have some free time ? https://review.openstack.org/#/c/327706/15:46
bauzasdansmith: sdague: and then we sum up the ram size for each instance https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L69915:46
sdaguebauzas: right, so this actually won't make any difference on the cpu side15:47
sdagueas dansmith correctly states15:47
sdaguebecause the reserved is 015:47
prateekjaypipes, can you please take a look when you have some time ? https://review.openstack.org/#/c/294513/15:48
mdboothWhich release was 13.0.0?15:48
mdboothWas that liberty?15:48
mdboothAh, no. That was mitaka, I think?15:49
_gryf13 is mitaka, liberty was 12.x15:49
mdboothThanks15:49
_gryfmdbooth, https://releases.openstack.org/mitaka/15:50
*** tonytan_brb has quit IRC15:51
bauzassdague: correct15:51
*** paul-carlton2 has quit IRC15:52
sahidprateek: done15:52
*** tbachman has joined #openstack-nova15:52
*** mriedem1 is now known as mriedem15:53
*** tonytan4ever has joined #openstack-nova15:53
cdentsdague, dansmith I think we've yet to identify or articulate the actual problem in a complete fashion, and until we can generate a minimal test case that aggrevates the problem we're going to be throwing spitballs that don't stick15:54
*** matrohon has quit IRC15:55
*** unicell1 has quit IRC15:55
sdaguecdent: well, some more logging would help, it's exploding in _check_capacity_exceeded15:57
sdagueand it really shouldn't be, given what is available at the time15:57
cdentso the questions become: can we construct a minimal test case? Can we do it in a way that it can be run indepdendently of tempest?15:57
*** _ducttape_ has quit IRC15:58
*** ducttape_ has joined #openstack-nova15:59
mriedemyou could potentially create a functional test15:59
mriedembut it's a fake virt driver15:59
mriedemso fake RT inventory15:59
*** claudiub has joined #openstack-nova16:00
sdagueright, but test for what? We have a real world behavior happening here that we don't yet understand even the bad math that is happening16:00
cdentsdague: basically so we can inspect more quickly. the iteration cycle here is painful. If the answer is "no we can't" that's okay, but I've still yet to read a concise description of the problem so I'm not able to believe "no we can't", yet.16:01
*** nic has joined #openstack-nova16:01
*** mgoddard_ has joined #openstack-nova16:02
sdague2016-09-06 12:01:16.928 WARNING nova.objects.resource_provider [req-9c93fbd0-ab72-47ad-9479-1132cb85f470 placement service] Attempting to allocate 1 for VCPU. Currently using 1024, amount available 64.016:02
sdaguethat is the crux of the problem16:02
*** unicell has joined #openstack-nova16:02
dansmithsdague: capacity says 1024 are in use and 64 are available for cpus?16:03
cdentIf we're driving on guesses at the moment I'd guess that the join or the group by in the sql in the exceed function is wonky16:03
sdaguedansmith: yes16:03
*** luis5tb has quit IRC16:03
dansmith1024/16 = 64 which makes no sense for that worker16:03
sdaguecorrect16:03
sdague1024 is a crazy value16:03
dansmiththat's weird.16:03
sdaguethis is my nuc16:03
dansmithoh okay, but still16:04
sdagueyeh, it's 4 core16:04
sdagueand, I know that's not actual usage16:04
* cdent inspects the sql16:04
sdaguebecause I've got other debug logs which are dumping the usage16:04
sdague2016-09-06 12:01:16.904 DEBUG nova.api.openstack.placement.handlers.allocation [req-9c93fbd0-ab72-47ad-9479-1132cb85f470 placement service] Current Usages for 62b97cd4-5dc8-45d9-89ad-988273895635: Usage(resource_class='VCPU',usage=16), Usage(resource_class='MEMORY_MB',usage=1024), Usage(resource_class='DISK_GB',usage=0),16:04
*** TravT_ has joined #openstack-nova16:04
*** gokrokve has quit IRC16:05
sdaguewhich means for this resource check, it is using the memory value for cpu16:05
dansmithso you booted a guest with 16 vcpus yes?16:05
openstackgerritChristopher Martin proposed openstack/nova: Fix Boot From Volume size error  https://review.openstack.org/36433716:05
*** mgoddard has quit IRC16:05
*** david-lyle_ has joined #openstack-nova16:05
*** david-lyle_ has quit IRC16:05
*** Swami has joined #openstack-nova16:06
sdaguedansmith: nope16:06
*** hshiina has quit IRC16:06
sdaguethat's complete usages16:06
dansmithsdague: an allocation is one instance16:06
sdaguedansmith: it's not an allocation16:06
dansmithoh you're logging usage from the allocation handler I see16:06
sdaguehttps://review.openstack.org/#/c/366186/1/nova/api/openstack/placement/handlers/allocation.py16:07
sdagueyeh16:07
dansmithgotcha16:07
dansmithyeah, kinda seems like it's confusing the columns in the sql query or something16:07
*** TravT has quit IRC16:08
*** TravT has joined #openstack-nova16:10
openstackgerritMatthew Booth proposed openstack/nova: imagecache: Hold shared storage registry lock while running  https://review.openstack.org/36623916:10
mdboothmriedem: ^^^ But I think the real fix is https://review.openstack.org/#/c/185549/ in mitaka16:11
*** paul-carlton2 has joined #openstack-nova16:11
sdagueI'm going to stick another bit of debug on the records to see if there anything useful there16:12
*** TravT_ has quit IRC16:12
mriedemmdbooth: what do you mean by the real fix is in mitaka?16:13
mdboothmriedem: The reporter mentions they hit the problem in several release, the last of which was liberty16:13
mdboothThe above fix was first in mitaka, and the backport reject for liberty16:13
mdboothBy you :)16:13
sdagueok... so this is weird16:13
mdboothAnd I suspect it would resolve their problem16:14
mriedemmdbooth: yeah b/c it required a grenade change for rootwrap filters16:14
mdboothmriedem: Right, but I thought it was amusing anyway :)16:14
*** derekh is now known as derekh_afk16:14
* cdent is increasingly feeling like this sql is not right16:14
sdaguehttp://paste.openstack.org/show/567204/16:15
*** lpetrut has quit IRC16:15
sdaguewe seem to have a joined in a way that we get usage for every resource class type crossed against every resource type16:15
*** riccardo has quit IRC16:15
*** lpetrut has joined #openstack-nova16:15
*** rlrossit has joined #openstack-nova16:16
mdboothmriedem: Anyway, that code still needs a lock. It came up here: https://review.openstack.org/#/c/334956/16:16
cdentfeh16:16
sdaguefield 4 is the resource class16:16
mdboothThe code gives the impression it might run concurrently, which raised flags with jaypipes16:16
sdaguelet me push so I can gitlink things16:16
mdboothWhen in fact, that would certainly break.16:16
openstackgerritSean Dague proposed openstack/nova: WIP: more and more and more debug  https://review.openstack.org/36624116:16
cdentsdague: i've repeated what you're seeing. nice work. I think that points the finger of doom pretty strongly at the sql16:19
sdaguehttps://github.com/openstack/nova/blob/f9103f6a93b2dcc15e613ee54235fdf54d60027f/nova/objects/resource_provider.py#L705 - so I'm assuming that we should only see 3 records there16:20
*** lpetrut has quit IRC16:20
*** gouthamr_ has quit IRC16:20
*** thorst has quit IRC16:21
*** daemontool has quit IRC16:21
*** thorst has joined #openstack-nova16:21
cdentthat's what I think, yeah16:21
sdagueok, so minimally we should probably update the code to explode if we find duplicate keys for our map I would think16:22
*** yamahata has quit IRC16:23
*** raunak has joined #openstack-nova16:24
*** rlrossit has quit IRC16:25
*** daemontool has joined #openstack-nova16:25
*** raunak has quit IRC16:25
*** akshai has quit IRC16:26
*** akshai has joined #openstack-nova16:29
*** tonytan_brb has joined #openstack-nova16:29
*** daemontool has quit IRC16:29
*** Apoorva has joined #openstack-nova16:29
*** Sukhdev has joined #openstack-nova16:30
sdagueyeh, it's the left outer join that's the issue16:30
*** thorst has quit IRC16:30
sdaguethat's why we get usage across all the records, and all the records duplicated16:30
sdaguewe really want to do a real join with inventory.resource_class_id == allocations.resource_class_id16:31
*** stevemar has quit IRC16:31
*** tonytan4ever has quit IRC16:32
*** gouthamr has joined #openstack-nova16:33
sdagueok, I might have that16:33
sdagueI at least get sane allocation records now16:33
avolkovsdague: hi, from https://review.openstack.org/#/c/365015/, do we have some stat about http responses?16:33
openstackgerritSean Dague proposed openstack/nova: WIP: correctly join the usage to inventory  https://review.openstack.org/36624516:34
sdaguecdent / dansmith ^^^ how does that look to you?16:34
cdentsdague: one sec16:34
dansmithI wouldn't know16:34
dansmithI'll believe it if the logs look righter :)16:34
sdaguewell, I get sane records now16:35
sdagueonly 3 of them16:35
*** mingyu has joined #openstack-nova16:35
sdagueinstead of 916:35
sdagueand the usage lines up with the resource class I expect16:35
*** derekh_afk is now known as derekh16:35
*** raunak has joined #openstack-nova16:35
cdentsdague: that's the exact same thing that I came up with (while I was one seccing) so I guess that's two votes16:35
cdentjust confirming with some runs16:36
*** zenoway has quit IRC16:36
*** READ10 has joined #openstack-nova16:36
*** zenoway has joined #openstack-nova16:36
dansmithsdague: should be able to prove that with a functional test that breaks on the current code right?16:36
sdaguedansmith: maybe16:36
sdagueso the thing is, I expect this comes and goes because the join columns have no guarunteed order16:37
*** tonytan_brb is now known as tonytan4ever16:37
sdaguesorry join rows16:37
openstackgerritSivasathurappan Radhakrishnan proposed openstack/nova: Using get() method to prevent KeyError  https://review.openstack.org/35942416:38
cdentsdague: I'm wondering what's missing from the existing allocation violation tests. probably not enough resoruces clases16:38
sdaguecdent: enough range of values?16:38
cdentwell: not enough classes of resources in the tests such that the cross product happens16:38
cdentalthough I thought I had one with at least two resource classes in the mix16:39
sdagueright, but the issue would only arise if you exceeded one of them16:39
sdagueand you managed to get things returned in a way that you were comparing it to the other16:39
*** mingyu has quit IRC16:40
cdentI guess it wasn't quite good enough: https://review.openstack.org/#/c/366144/16:40
cdentanyway, my local tests confirm your join changes16:40
cdent3 records with the right data16:40
*** zenoway has quit IRC16:41
mdboothsdague: That looks correct to me also..16:41
*** sahid has quit IRC16:41
mdboothThat primary join is essentially joining allocation to inventory16:41
*** READ10 has quit IRC16:41
mdboothUsing only resource_provider in the where clause16:41
mdboothBut there are multiple inventories per resource provider16:41
sdagueso, we probably just need a safety check that record length is not greater than len(res_classes) * len(provider_uuids)16:42
*** xyang1 has joined #openstack-nova16:42
sdaguethat would at least have blown up early on the sql16:42
*** fragatina has joined #openstack-nova16:42
mdboothresource_provider/resource_class uniquely describes an inventory16:42
cdentif we have functional tests that confirm the sql is doing what it says it should, then having an assertion in the code path seems redundant to me: we need to be able to rely on the sql. If we can't, we need to fix it.16:43
sdaguemdbooth: ok, sure, so we do an 'in' check before adding the usage map16:43
cdentthe problem in this case is we didnt' have good tests for the sql16:43
cdents/good/good enough/16:43
mdboothsdague: Agreeing with you completely, just showing my working :)16:43
sdaguea safety measure is still good16:43
*** fragatina has quit IRC16:43
*** lucasagomes is now known as lucas-dinner16:44
*** fragatina has joined #openstack-nova16:44
sdagueok, I need to get lunch, then I'll go clean this up and rebase to master16:44
* cdent tries to think up a reasonable functional test16:45
*** markus_z has quit IRC16:46
*** roeyc has quit IRC16:47
*** pradk has quit IRC16:47
mdboothcdent: A resource provider with 2 inventories, with allocations against both.16:48
*** jamesdenton has quit IRC16:48
cdentmdbooth: yeah, I thought I had that, but it appears not16:48
cdentor rather I have it, but I don't have a failure against it16:49
* mdbooth is just looking at what the duplicate entries in 'records' result in16:49
*** rlrossit has joined #openstack-nova16:49
*** gszasz has quit IRC16:49
* cdent nods16:50
*** lpetrut has joined #openstack-nova16:52
*** fragatina has quit IRC16:52
mdboothThis is pretty dense :)16:52
*** Sukhdev has quit IRC16:53
*** rama_y has joined #openstack-nova16:54
mdboothcdent: So, primary_join is a cartesian join of usage (correct) against all inventories in a single rp16:55
*** ihrachys has quit IRC16:55
mdboothrecords, therefore contains a mix of joins, where all but one of the inventories are incorrect16:55
openstackgerritdane-fichter proposed openstack/nova: Use cursive for signature verification  https://review.openstack.org/35123216:56
mdboothSo the 'last' record may or may not be the correct one16:56
sdaguemdbooth: yeh, it's super dense. Which is why I'm in favor of inline safety measures because we've proven that no one can eyeball review it correctly :)16:56
*** sdake has joined #openstack-nova16:57
mdboothsdague: Sounds like a good idea :)16:57
* cdent has a failing test16:58
mdboothcdent: provs_with_inv contains all relevant rps16:58
cdent(in the good way)16:58
mdboothusage_map contains only the last record16:58
sdagueheh, you know, jay's comment of "this is this sql" is the right sql he wanted16:58
mdboothcdent: Now to ensure it's deterministicly failing :)16:58
sdaguejust missed a join16:58
*** hogepodge has quit IRC16:58
*** lpetrut has quit IRC16:59
cdentsdague: heh, yeah16:59
mdboothcdent: I assume your test involves executing the sql, right?16:59
mdboothI wonder what assurances sqlite makes about returned row order in this case17:00
*** syjulian_ has joined #openstack-nova17:00
mdboothProbably not many :)17:00
*** anush has quit IRC17:00
mdboothIt's a pretty funky query17:00
cdentmdbooth: yeah, this is a functional test that is calling the outer layer object methods, not a unit test on the sql methods17:00
mdboothcdent: If you're wondering what I'm going on about, btw, it's this:17:02
mdbooth    for record in records:17:02
mdbooth        usage_map[(record['uuid'], record['resource_class_id'])] = record17:02
mdboothThat will result in usage_map being correct or incorrect depending on whether the last returned row was the correct one or now17:03
mdboothnot17:03
*** unicell has quit IRC17:03
*** mgoddard_ has quit IRC17:03
*** hogepodge has joined #openstack-nova17:03
*** mgoddard has joined #openstack-nova17:03
cdentmdbooth: can you expand on what you're saying there a bit, because, assuming we fix the join as described, why is that still true?17:05
mdboothcdent: If we fix the join it's not true17:05
sdaguedo we have a tracking bug yet?17:05
cdentah, okay, now I get where you're coming from17:05
*** bradjones has quit IRC17:06
mdboothHowever, it would make your failing test determined by the returned row order17:06
mdboothi.e. Your failing test might get false positives with an alternative row order17:06
cdentmdbooth: yeah, but it turns out I made my failing test incorrectly otherwise anyway, so I'm still at the drawing board17:06
*** yamahata has joined #openstack-nova17:06
openstackgerritTracy Jones proposed openstack/nova: Fix spelling mistake  https://review.openstack.org/36625417:07
cdentsdague: not that I'm aware of, when I left for my appointment I thought you said you were gonna17:07
sdagueoh, I was just digging on the problem, I didn't build the bug yet17:07
sdagueI'll do that now17:07
*** code-R_ has quit IRC17:07
*** su_zhang has joined #openstack-nova17:09
mdboothcdent: Sorry to intrude, I was just being nosy. Enjoy :)17:10
*** lpetrut has joined #openstack-nova17:10
* mdbooth has to head home17:10
cdentmdbooth: not an intrusion at all, thanks for the insight17:10
*** armax has quit IRC17:11
*** bradjones has joined #openstack-nova17:11
*** bradjones has quit IRC17:11
*** bradjones has joined #openstack-nova17:11
*** sdake_ has joined #openstack-nova17:12
*** paul-carlton2 has quit IRC17:13
*** syjulian_ has quit IRC17:14
*** sdake has quit IRC17:15
*** pradk has joined #openstack-nova17:16
openstackgerritSean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting  https://review.openstack.org/36624517:20
sdaguecdent: so, I think if you *actually* want to unit test that, you are going to need to split the _check_capacity_exceeded function in 217:21
sdaguewith the first half being just the SQL17:21
sdagueand basically ending at where records are returned17:21
cdentsdague: I think a functional test is more useful in this case, or at least, for me, I'm able to exercise the problem more effectively and realistically (and create a better understanding of what's going) with a functional test17:22
sdaguecdent: sure, that's fine too17:22
sdagueI do think the key in the bug is complicated sql is complicated, and we got back an unexpected row structure17:22
sdagueand long term, isolating that so we know we always get back expected row structure would be nice17:23
sdagueanyway, https://review.openstack.org/366245 I think is in the state we want to merge, and it's rebased to master so doesn't depend on anything else17:23
*** stevemar has joined #openstack-nova17:26
*** paul-carlton2 has joined #openstack-nova17:26
*** armax has joined #openstack-nova17:26
*** mriedem1 has joined #openstack-nova17:26
*** thorst has joined #openstack-nova17:26
openstackgerritTracy Jones proposed openstack/nova: Fix spelling mistake in libvirt  https://review.openstack.org/36626217:28
*** PaulMurray has quit IRC17:28
*** tjones has joined #openstack-nova17:28
*** mriedem has quit IRC17:28
*** sdake_ has quit IRC17:28
*** rama_y has quit IRC17:29
*** sdake has joined #openstack-nova17:29
*** ababich has quit IRC17:29
*** su_zhang has quit IRC17:30
*** jdurgin1 has quit IRC17:30
*** su_zhang has joined #openstack-nova17:30
*** akshai has quit IRC17:30
*** TravT has quit IRC17:31
*** fragatina has joined #openstack-nova17:32
*** fragatina has quit IRC17:32
*** tonytan4ever has quit IRC17:33
*** TravT has joined #openstack-nova17:33
*** fragatina has joined #openstack-nova17:33
*** edand has joined #openstack-nova17:33
*** jheroux has joined #openstack-nova17:34
*** thorst has quit IRC17:35
sdaguegah, pep817:36
*** jaosorior has quit IRC17:36
openstackgerritSean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting  https://review.openstack.org/36624517:38
*** harlowja has joined #openstack-nova17:38
cdentfound another (minor) bug so just reporting it for now but will fix it tomorrow. i'm fried at this point: https://bugs.launchpad.net/nova/+bug/162074817:38
openstackLaunchpad bug 1620748 in OpenStack Compute (nova) "In placement when an attempt is made to write to missing inventory the error message is ugly" [Undecided,New] - Assigned to Chris Dent (cdent)17:38
*** amotoki has quit IRC17:39
sdagueyeh, the int ids definitely make dealing with the raw data a bit odd17:40
*** tbachman has quit IRC17:40
sdagueit would be nice if the repr for the objects resolved that17:40
cdentyou still got the set problem17:40
cdentI'm going to have to give up for the evening; i've reached stupid. will check in later or catch up tomorrow17:42
*** cdent has quit IRC17:42
sdagueok, no worries17:43
*** tonytan4ever has joined #openstack-nova17:44
*** tbachman has joined #openstack-nova17:44
*** lamt has joined #openstack-nova17:45
sdagueyay, my run tempest a lot and leak resources meant I even got to see the warning I put in place17:48
sdague2016-09-06 13:47:13.766 WARNING nova.objects.resource_provider [req-a3758596-5182-4e9e-b7a2-04462bf27f48 placement service] Over capacity for VCPU on resource provider 62b97cd4-5dc8-45d9-89ad-988273895635. Needed: 1, Used: 64, Capacity: 64.017:48
sdague17:48
*** oomichi- is now known as oomichi17:49
auggymriedem1: Jenkins protested at the merged policy files change, https://review.openstack.org/#/c/363768/ so I'm looking into it now17:49
*** sdake has quit IRC17:49
dansmithsdague: I didn't parse that first statement, but is this in CI or locally?17:49
sdaguedansmith: it's locally17:50
dansmithsdague: so using some flavor that could really result in 64 vcpus being allocated?17:50
*** paul-carlton2 has quit IRC17:50
sdaguemy env is pretty dirty now with lots of tempest fails for other reasons, so I've got a leak17:50
dansmithoh17:50
sdaguethis is trying to allocate a 1 vcpu guest17:50
dansmithyou mean you think you've got leaked allocations that don't corespond to instances?17:50
sdaguebut nova still thinks I've got 64 allocated17:51
sdagueyeh17:51
dansmithso you can look at the allocations and see the instance uuids that they came from17:51
auggyjohnthetubaguy: the admin port changes, https://review.openstack.org/#/c/357540/ and https://review.openstack.org/#/c/357726/ are all green now and I removed the WIP, let me know if there's anything else you need help with regarding those17:51
sdaguedansmith: I think so, what view do you want?17:51
dansmithsdague: it would be good to have a manage command that will do a consistency check on instances and allocations17:51
dansmithsdague: I don't want a view, I'm just saying if you want to go spot check some things to see if there are allocations for deleted instances, you should be able to do that fairly easily17:52
sdaguedansmith: yeh, I just assume that's the case. I was just happy to see that warning message was formatted correctly17:52
dansmithsdague: consumer_id in allocations should be the instance uuid17:52
sdaguenow that it isn't blowing up all the time randomly17:52
*** mriedem1 is now known as mriedem17:53
sdagueyeh, based on timestamps it looks like I was leaking 2 - 8 per tempest run17:54
*** tblakes has joined #openstack-nova17:54
sdaguebut I wasn't even really caring what was passing / failing17:54
*** fragatina has quit IRC17:54
sdagueas I was just trying to generate load17:54
mriedemauggy: unrelated, just recheck it17:54
dansmithI wonder if allocation delete is ever blocked for some wrong math reason?17:54
mriedemthat cellsv1 job seems to be failing more often on that same test17:54
auggymriedem: kk will do, just wanted to make sure it wasn't a real issue17:55
*** fragatina has joined #openstack-nova17:55
mriedemauggy: we shouldn't even be running dsvm jobs on a .gitignore change17:55
mriedemwe should update project-config to not run dsvm jobs on a change like that, same as how we don't run dsvm jobs for docs or unit test only changes17:55
auggymriedem: fair point, it seemed odd to me too but i'm paranoid17:55
*** jamesdenton has joined #openstack-nova17:55
auggymriedem: should i file a bug or something for that as a reminder to make that change? i can work on that17:55
*** sdake has joined #openstack-nova17:56
mriedemauggy: see https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L133917:56
mriedemdon't need a bug17:56
auggykk17:56
auggyi'll make a todo for myself and get that in today17:56
sdaguedansmith: I am sure there are other things we should check here about how things skew from reality over time17:57
*** akshai has joined #openstack-nova17:57
mriedemauggy: and probably this one https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L199917:57
dansmithsdague: you mean more things we should sanity check inside that capacity transaction?17:57
mriedemso .git.* files should be skipped, like .gitignore and .gitreview17:58
sdaguedansmith: we probably need an audit tool that goes across the API17:58
*** lyarwood has quit IRC17:58
sdaguedansmith: I just mean "in general"17:58
*** mingyu has joined #openstack-nova17:58
dansmithsdague: such a thing would have to be able to read deleted instances17:58
auggymriedem: awesome, thanks!17:58
openstackgerritAndrey Volkov proposed openstack/nova: WIP Move instance creation to conductor  https://review.openstack.org/31937917:58
dansmithso I'd kinda rather do it from manage so we're not building more stuff that relies on deleted instances through the api17:58
openstackgerritDan Smith proposed openstack/nova: [placement] Allow inventory to violate allocations  https://review.openstack.org/36506817:59
dansmithsdague: ^ jaypipes is okay with this and it's pretty critical for correctness17:59
dansmithI just removed some more cruft that alex_xu found17:59
sdaguedansmith: ok, I'll hit that if you hit my join fix18:00
sdaguedansmith: doesn't a nova manage command get weird once placement is separated? especially once it's another db next cycle some time18:01
dansmithsdague: nova manage can hit the placement api if you want, but the instance and these records are already spread across the api and cell databases18:02
*** edand has quit IRC18:02
dansmithsdague: I'm not sure I'm really able to validate that sql change.. and I thought we were going to wait for a functional test?18:03
dansmithwell, maybe I could validate it if I wrote said tests, but..18:03
*** mingyu has quit IRC18:04
sdaguewell, the validation at this point is just that KeyError isn't raised18:05
sdaguethat's where we went wrong, we took 9 rows back from the sql query, squashed them down to 3 rows, kind of randomly in a dictionary, and assumed they were the rows we wanted (they often weren't)18:06
*** crinkle_ is now known as crinkle18:07
*** rlrossit_ has joined #openstack-nova18:07
*** tbachman_ has joined #openstack-nova18:07
*** tbachman has quit IRC18:08
*** tbachman_ is now known as tbachman18:08
dansmithsdague: if I remove that functional change and leave the key assert thing, I don't get any extra fails18:09
sdaguedansmith: in a real environment?18:09
dansmithno, in functional tests18:09
*** oanson has joined #openstack-nova18:10
*** rlrossit has quit IRC18:10
*** ssurana has joined #openstack-nova18:12
dansmithso I modified one of our tests to hit that keyerror, but your functional change doesn't make it go away18:14
sdaguedansmith: what did you modify?18:14
sdagueso, I've tested this live, and it's definitely the real fix. I am sure the functional tests are lacking here18:14
dansmither, wait18:14
dansmithactually, I'm hitting a different keyerror in the code you added, but not the raise you want18:15
auggyoh wow i just read markus isn't doing bugs or much nova anymore :(18:16
auggyi swear, go on a holiday for a day...18:17
*** pradk has quit IRC18:18
*** fragatin_ has joined #openstack-nova18:22
tjonesyeah too bad - he did a great job.  it's a hard one18:23
mriedemSwami: reviewed https://review.openstack.org/#/c/275073/18:24
Swamimriedem: thanks18:24
*** fragatina has quit IRC18:25
dansmithsdague: so you're adding an extra filter on resource_class, but it seems like the check you're doing is to filter out multiple rows of the same class for two resource providers?18:25
sdaguedansmith: yeh, it looks like the functional tests are too clever by half and try to do a whole bunch of stuff in a sequence so it's a bit less real. I'll leave someone else to make those sane. That's the real18:25
sdaguedansmith: no, the key is a tuple18:25
*** tri2sing has joined #openstack-nova18:25
sdagueof rp uuid and resource class18:25
dansmithsdague: I know18:26
tjonesmriedem: are we at the point where we are pausing on removing mox for newton ?18:26
mriedemtjones: yes18:26
tjonesok thanks18:26
mriedemthere is a thread in the ML on moving those to an ocata blueprint18:26
*** oomichi has quit IRC18:26
*** oomichi has joined #openstack-nova18:27
tjonesyeah i  missed it but i figured we were there.18:27
*** danpb has quit IRC18:27
dansmithsdague: I modified a test so that one RP has two resource classes, which is realistic and I thought what you were saying was the problem that we get the cross-product of the thing we're looking for, such that we have multiple (uuid, class) pairs in one result set18:28
openstackgerritRyan Rossiter proposed openstack/nova: Log trace when updating host states  https://review.openstack.org/36628318:29
*** oanson has quit IRC18:29
dansmithsdague: but I don't hit that trigger if I remove your extra resource_class_id filter from the query18:29
*** thorst has joined #openstack-nova18:32
*** jamesdenton has quit IRC18:32
sdaguedansmith: yep, I don't know why the tests don't match the reality. I'll let jaypipes / cdent run with that one18:33
dansmithokay18:34
jaypipesdansmith, sdague: what's up? sorry, been in meetings all day :(18:35
dansmithjaypipes: summarizing on a review, just a sec18:35
jaypipesk18:35
*** pradk has joined #openstack-nova18:35
dansmithjaypipes: https://review.openstack.org/#/c/366245/318:37
dansmithjaypipes: sdague figured out that the VCPU thing is due to us not always picking the right inventory result row during the capacity check, but I can't write a functional test to reproduce/prove it which I think is important18:37
openstackgerritRyan Rossiter proposed openstack/nova: Warn when using scheduler with debug mode  https://review.openstack.org/36628918:37
*** thorst has quit IRC18:39
*** lamt has quit IRC18:40
*** mfedosin has quit IRC18:40
jaypipesdansmith: k. do you need me to pull the above patch and write a func test for the multiple-providers-multiple-resources case?18:41
dansmithjaypipes: yeah18:41
dansmithjaypipes: write a test that fails without the extra query addition18:41
jaypipesdansmith: you got it.18:42
dansmithjaypipes: that's what I've been trying to do, but I'm not smart enough apparently18:42
*** oomichi has quit IRC18:42
jaypipesdansmith: I've got more meetings but will try my best...18:42
dansmithare you saying you think we need multipiple RP each with multiple classes to hit that?18:42
*** baoli has joined #openstack-nova18:42
jaypipesdansmith: yeah.18:42
dansmithwe should only have one RP in the current test tempest runs I think, so I'm not sure how we'd have hit that...18:42
*** baoli_ has quit IRC18:42
jaypipes(I think) :)18:43
sdagueyeh, this seemed hittable with only 1 RP18:43
* dansmith puzzles18:43
sdaguehttp://logs.openstack.org/41/366241/1/check/gate-tempest-dsvm-neutron-placement-full-ubuntu-xenial-nv/48a0a01/logs/apache/placement-api.txt.gz#_2016-09-06_16_36_22_24218:44
sdaguethat's in a run with a bunch of unmerged logging18:44
*** lamt has joined #openstack-nova18:44
*** TravT has quit IRC18:45
sdaguewe definitely only have 1 resource provider there18:45
dansmithsdague: what allocation has 192 of anything?18:45
sdaguemem18:45
dansmithwe're using a 192MB memory flavor in tempest in the gate?18:46
sdaguewe're using 64MB flavors18:46
*** prateek_ has joined #openstack-nova18:46
*** TravT has joined #openstack-nova18:46
sdaguethat's the mem usage18:46
dansmiththat's what I thought, so no allocation should be more than 6418:46
jaypipesdansmith: the resource tracker WARN log message right above there shows 64 vCPU available, using 128 vCPU, requesting 1 vCPU.18:46
sdaguedansmith: that's *usage*18:46
dansmithsdague: oh the log says "allocation record: ..."18:47
sdagueyes18:47
sdagueit's the rows from the giant sql monster18:47
sdagueit was the best name I had for it18:47
*** rama_y has joined #openstack-nova18:47
dansmithsdague: then that log message should be different :)18:47
jaypipessdague: that's the correct name/term :)18:47
sdagueok, tell me what the debug message should be :)18:47
dansmithjaypipes: he says "allocation record: $usage" .. that's not right to me :)18:47
*** raunak has quit IRC18:48
*** priteau has joined #openstack-nova18:48
dansmithjaypipes: as in Usage() which is a sum from the resource provider18:48
*** oomichi has joined #openstack-nova18:48
sdaguehttps://review.openstack.org/#/c/366241/1/nova/objects/resource_provider.py@70618:48
jaypipesdansmith: yes, true. I was just saying "allocation record" is the right term here.18:48
dansmithjaypipes: how is it right?18:48
jaypipesdansmith: doh, no, you are 100% correct, sorry.18:49
dansmithsdague: yeah, I see why you said that, but since we have actual "Allocation" reocrds, I think it's wrong to say that :)18:49
*** prateek has quit IRC18:49
sdaguedansmith: ok, tell me a different word, that's the best I had when I was hacking to the bottom of this18:49
sdagueanyway, terminology asside18:49
dansmithsdague: capacity or usage or something18:49
sdagueyou can see the N^2 matrix there18:50
*** prateek has joined #openstack-nova18:50
dansmithsdague: the terminology had me concerned we were trying to allocate a bad value is my point18:50
sdagueok18:50
dansmithsdague: anyway, those are just three records, which we expect.. this is after your fix, right?18:51
*** prateek_ has quit IRC18:51
sdaguedansmith: no, that's before the fix18:51
dansmithsdague: okay, but that's not NxM18:51
sdagueand there a 9 records18:51
dansmiththat's what I would expect to see18:51
dansmithoh, lol.. scroll up18:51
sdague:)18:52
dansmithyeah, so I don't know why I can18:52
*** oomichi has quit IRC18:52
dansmithwhy I can't repro that18:52
*** oomichi has joined #openstack-nova18:52
sdagueyeh, me either18:52
dansmiththat plus your fix kindaish makes sense to my SQL-dumb brain, but.. I want to see it in a test and I can't make it do that18:52
openstackgerritSylvain Bauza proposed openstack/nova: WIP: Update BuildRequest if instance currently being scheduled  https://review.openstack.org/36574018:52
bauzasalaski: ^ updated for a quickiest patch18:53
bauzasalaski: if you agree with it, I'll add the UTS18:53
bauzasUTs18:53
alaskibauzas: was just commenting on the previous version. I'll finish that and update and look again18:54
sdagueok, well, I'll leave that to jaypipes. Going to walk away from computer for a bit.18:54
jaypipesdansmith: ah, got it...18:54
bauzasalaski: heh, sorry about that :)18:55
alaskino worries, I do it too :)18:55
jaypipesdansmith: TestAllocationListCreateDelete.test_alllocation_list_create() uses multiple resource providers, but only one resource class inventory record for each resource provider. If we change that to be multiple resource classes per provider, it should fail.18:55
jaypipesdansmith: I will update and test pre-patch and post-patch.18:56
*** raunak has joined #openstack-nova18:59
dansmithjaypipes: that is *precisely* what I have done and I dont' see the fail18:59
dansmithjaypipes: http://pastebin.com/cmcvaLqJ19:00
dansmithjaypipes: that case of multiple classes for one RP is definitely a giant gap in our current tests for sure19:00
dansmithespecially since that's a realistic scenario19:01
*** prateek_ has joined #openstack-nova19:02
*** raunak has quit IRC19:02
jaypipesdansmith: ok. well, lemme play around with this and see if I can trigger it.19:02
dansmithjaypipes: yep, cool19:02
dansmithin about 30 I have to run off for a bit, but will check in with you this afternoon19:02
*** prateek has quit IRC19:05
*** prateek_ has quit IRC19:06
alaskibauzas: added some comments. It still has some checks I think you can remove, but looks close19:07
*** prateek_ has joined #openstack-nova19:07
bauzasalaski: okay, fair enough, I'll work on that tomorrow with the UTs so we could try to merge it before RC119:08
openstackgerritRyan Rossiter proposed openstack/nova: Warn when using scheduler with debug mode  https://review.openstack.org/36628919:08
bauzasalaski: that's a regression hence me running for it19:08
bauzasauggy: FWIW, saw your email19:08
*** parora has joined #openstack-nova19:09
bauzasauggy: I think we need to pay close attention to new and in progress bugs to see whether they're related to some regression19:09
*** gjayavelu has joined #openstack-nova19:09
alaskibauzas: yep, we need to get this fixed19:09
*** akshai has quit IRC19:09
bauzasauggy: I tried to look around all the open bugs, but I really need help here :)19:09
*** Sukhdev has joined #openstack-nova19:09
openstackgerritRyan Rossiter proposed openstack/nova: Log trace when updating host states  https://review.openstack.org/36628319:11
*** pcaruana has quit IRC19:11
*** prateek_ has quit IRC19:12
*** prateek_ has joined #openstack-nova19:13
*** cdent has joined #openstack-nova19:14
*** Guest78236 is now known as melwitt19:14
bauzasrlrossit_: so, yeah, I agree that running DEBUG for the scheduler is a very bad idea...19:14
bauzasrlrossit_: like drinking while driving19:14
bauzasrlrossit_: but there, should we just add some ads saying that it's bad, or rather find some situations where DEBUG logs are unnecessary?19:15
cdentdansmith: you've made me feel better. I was unable to write a failing functional test too and left for dinner in frustration. I wonder if it is sqlite artifact (or mysql). I can demonstrate the bad behavior in devstack no problem.19:15
bauzasrlrossit_: I tend to like the latter :)19:15
auggybauzas: lol i'm not a pro either but i will do what i can ;)19:15
dansmithcdent: yeah, I dunno, but this seems like an odd thing to be an artifact19:15
bauzasauggy: bug triaging is one way to come by Nova :)19:15
*** cleong has quit IRC19:15
auggybauzas: yeah, i've been pretty active with helping markus this past year19:16
rlrossit_bauzas: I have decided to do both19:16
bauzasrlrossit_: \o/19:16
* auggy has learned a lot from nova bugs19:16
*** mriedem has quit IRC19:16
rlrossit_I'm saying debug=True with scheduler is a bad thing, and then I also move the main debug culprit to trace()19:16
*** parora has quit IRC19:16
rlrossit_there's 2 patches19:16
bauzasrlrossit_: I just saw the second one19:16
bauzasrlrossit_: tbh, I'm very not sold about saying it's a bad thing19:17
bauzasrlrossit_: that's just life19:17
cdentdansmith: that's all I had, going back to not being here19:17
*** cdent has quit IRC19:17
rlrossit_https://review.openstack.org/366289 is "Hey you shouldn't do this", and https://review.openstack.org/366283 is "well, you did it, let's not make it hurt so much"19:17
bauzasrlrossit_: yeah I very gently -1d the former :D19:17
*** fragatin_ has quit IRC19:17
*** fragatina has joined #openstack-nova19:18
bauzasrlrossit_: so the problem is more that you need DEBUG for getting important info when some bad scheduling happens19:19
bauzasrlrossit_: I mean, some operators want to use DEBUG in production because they think they miss some crucial information about some bad scheduling19:19
bauzasrlrossit_: but just changing the log level doesn't help them IMHO19:20
bauzasrlrossit_: we should rather think about why they need to use DEBUG19:20
*** pradk- has joined #openstack-nova19:20
bauzasrlrossit_: see ?19:20
*** pradk has quit IRC19:21
rlrossit_yeah, I don't disagree with that19:21
*** parora has joined #openstack-nova19:21
rlrossit_and granted, when I was doing this performance stuff, I wouldn't go looking in the logs to see that I shouldn't be running with debug on.19:21
rlrossit_it was slow, and I didn't know why or where until I commented logging out19:22
*** pradk- is now known as pradk19:22
*** lamt has quit IRC19:24
*** rmart04 has joined #openstack-nova19:24
bauzasrlrossit_: that's why we had some BP related to that19:24
bauzasrlrossit_: lemme find it19:25
*** tesseract- has quit IRC19:25
rlrossit_yeah mriedem linked it to me on Friday19:25
*** prateek_ has quit IRC19:25
rlrossit_cfriesen is working on it I believe19:25
*** lpetrut has quit IRC19:25
bauzasrlrossit_: the idea was to stack all the filter logs so that where we were having a NoValidHost, then it was popped19:25
*** akshai has joined #openstack-nova19:25
rlrossit_yup19:25
rlrossit_"in-memory logging" of sorts19:26
bauzasrlrossit_: well, I haven't seen much of a progress here, so I think you should maybe see how cfriesen and you could resurrect that19:26
melwittI had thought turning off debug log level for performance testing was a given, be it openstack or not19:26
bauzasthe Ocata cycle will open soon and I'd be very happy to see that19:26
rlrossit_melwitt: well, I didn't think debug logging would be *that* bad19:27
bauzasmelwitt: we should rename s/DEBUG/EDONTDOTHISFORPRODUCTION I think :p19:27
dansmithI thought that we actually hear that most big ops actually run with debug on all the time because they have to have it to diagnose any real issue19:27
rlrossit_bauzas: still doesn't stop some people ;)19:27
dansmithi.e. they pay the penalty for the benefit of being able to fix things19:27
bauzasdansmith: yup, why I pointed out cfriesen's BP19:28
dansmiththat may have come via sdague a while ago19:28
melwittwe didn't. debug is definitely not on unless something bad happens and debugging is needed19:28
rlrossit_dansmith: yeah... that's what we're starting to live with19:28
dansmithmelwitt: okay19:28
*** diogogmt has quit IRC19:29
*** hemanthm is now known as hemanthm|away19:29
openstackgerritSwaminathan Vasudevan proposed openstack/nova: Implement setup_networks_on_host for Neutron networks  https://review.openstack.org/27507319:30
bauzasrlrossit_: grah, can't find cfriesen's BP19:30
bauzasrlrossit_: I'd have checked the whiteboard to see the status19:30
*** diogogmt has joined #openstack-nova19:31
*** tbachman_ has joined #openstack-nova19:31
bauzasgrah, can someone tell me how I can find the gerrit username for cfriesen so I could search for his open changes ?19:31
*** coolsvap_ has quit IRC19:32
*** tbachman has quit IRC19:32
*** tbachman_ is now known as tbachman19:32
bauzasnevermind, found it19:32
bauzasrlrossit_: https://review.openstack.org/#/c/331306/ has some implementation concerns, possibly good to disccuss that with the owner19:33
*** claudiub has quit IRC19:34
bauzasrlrossit_: in particular given your testbed of 400 nodes, could be interesting to see the memory usage for that19:34
*** Apoorva has quit IRC19:34
alaskiI know that rax runs(ran) with debug logs on, because tracking down issues would have been near impossible without it. But cells may have been (i.e. was) a factor there.19:35
*** rmart04 has quit IRC19:35
*** parora has quit IRC19:36
alaskiefforts like cfriesens to only dump logs on failure would be a huge help there19:36
bauzasalaski: yup19:36
rlrossit_bauzas: I don't see that using much more memory than holding 400 host state maps in memory19:36
*** thorst has joined #openstack-nova19:36
bauzasrlrossit_: well, that's different19:36
bauzasrlrossit_: holding 400 host state dicts is one thing19:37
bauzasrlrossit_: allowing to write in memory long strings is another, I guess19:37
bauzasas you stated, logs can be very verbose :)19:37
rlrossit_I guess what I'm trying to say is: overall, it's not that much *more* memory19:38
rlrossit_ram is cheap, file I/O is expensive19:38
rlrossit_well, that debug that I moved down to trace doesn't really help much in the case of scheduling failures anyways :)19:39
bauzasrlrossit_: but you can define a log factory for each level you know :)19:40
*** rcernin has joined #openstack-nova19:41
bauzasanyway, I just want to tell that merging your patch won't really solve the problem19:41
bauzasit could just reduce the verbosity, sure19:41
bauzasbut that would still need some further improvment19:41
*** thorst has quit IRC19:45
*** sdake has quit IRC19:46
*** moshele has joined #openstack-nova19:48
*** rcernin has quit IRC19:48
*** pradk has quit IRC19:49
sdaguejaypipes: you have that test fix, because if not, I have it, now that you gave the hint19:50
jaypipessdague: meetings :(19:52
sdagueok19:52
*** TravT has quit IRC19:55
openstackgerritSean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting  https://review.openstack.org/36624519:56
openstackgerritSean Dague proposed openstack/nova: DNM: test SQL fix  https://review.openstack.org/36631319:56
*** eharney has quit IRC19:56
*** devananda is now known as devananda|travel19:57
sdagueug, that's not really it, I need to futz with this a bit more19:59
*** davidlenwell has quit IRC19:59
*** mvk has quit IRC20:01
*** kfarr has joined #openstack-nova20:02
*** raunak has joined #openstack-nova20:03
*** sdake has joined #openstack-nova20:03
*** joshblake has joined #openstack-nova20:04
cfriesensfinucan: you on?20:06
*** eharney has joined #openstack-nova20:08
openstackgerritSean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting  https://review.openstack.org/36624520:09
*** raunak has quit IRC20:09
openstackgerritSean Dague proposed openstack/nova: DNM: test SQL fix  https://review.openstack.org/36631320:09
*** tbachman has quit IRC20:10
*** tbachman has joined #openstack-nova20:11
sdagueyeh, still doesn't quite trigger it, I'm not sure now20:11
*** ljxiash has quit IRC20:11
*** davidlenwell has joined #openstack-nova20:12
*** TravT has joined #openstack-nova20:13
*** edmondsw has quit IRC20:13
bauzascfriesen: sfinucan lives in Ireland, so probably off for tonight20:15
*** raunak has joined #openstack-nova20:18
*** unicell has joined #openstack-nova20:18
*** kevinz has quit IRC20:23
openstackgerritAndrew Laski proposed openstack/nova: Annotate db models that have moved to the nova_api db  https://review.openstack.org/36632720:26
*** praneshp has joined #openstack-nova20:29
*** salv-orl_ has joined #openstack-nova20:30
*** moshele has quit IRC20:33
*** salv-orlando has quit IRC20:33
*** thorst has joined #openstack-nova20:33
cfriesenbauzas: ah, thanks.  will try tomorrow20:33
bauzasnp20:34
bauzascfriesen: I had a convo with rlrossit_ about the scheduler logging BP you wrote20:35
bauzascfriesen: hopefully you'll have time to work on it for Ocata ?20:35
*** thorst has quit IRC20:39
openstackgerritSwaminathan Vasudevan proposed openstack/nova: Implement setup_networks_on_host for Neutron networks  https://review.openstack.org/27507320:40
*** ayoung has quit IRC20:41
*** moshele has joined #openstack-nova20:41
*** pradk has joined #openstack-nova20:44
*** cdelatte has quit IRC20:44
sdaguejaypipes: ok, I think I figured it out20:46
sdagueI needed 2 consumers20:46
sdagueotherwise during allocations it deleted all the usage for the current consumer20:47
openstackgerritSean Dague proposed openstack/nova: DNM: test SQL fix  https://review.openstack.org/36631320:47
openstackgerritSean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting  https://review.openstack.org/36624520:47
jaypipessdague: aha.20:47
sdagueso 1 resource_provider, > 1 active consumer, > 1 resource class20:47
jaypipesright, that makes sense.20:47
sdaguethere is a dedicated test for this in the patch now20:48
jaypipessdague: nice digging. i literally just got off my 11th meeting today (not kidding)20:48
sdagueyep, no worries. I learned a bunch about the whole system in the process, so that's always good20:48
cfriesenbauzas: hopefully, yes.  we were kind of late out the gate for N...other stuff going on.20:49
jaypipessdague: thank you very much for your help on all this over last couple weeks. I really appreciate it.20:49
sdaguejaypipes: happy to help20:49
*** agireud has quit IRC20:52
sdagueI think you owe me a cookie or drink or something in barcelona for debugging your SQL though :)20:52
*** rlrossit_ has quit IRC20:54
*** TravT has quit IRC20:55
*** agireud has joined #openstack-nova20:59
*** mriedem has joined #openstack-nova21:01
*** sdake has quit IRC21:02
sdaguejaypipes: it would be cool if you could review - https://review.openstack.org/#/c/366245/ for correctness21:03
*** kylek3h has quit IRC21:04
*** derekh has quit IRC21:05
*** raunak has quit IRC21:06
openstackgerritAnusha Unnam proposed openstack/nova: List soft_deleted instances  https://review.openstack.org/36634721:08
dansmithsdague: I'm pulling it down to play now21:09
sdaguedansmith: you should get the fail on the DNM patch21:10
*** mmedvede has quit IRC21:11
*** mmedvede has joined #openstack-nova21:11
*** raunak has joined #openstack-nova21:11
*** ljxiash has joined #openstack-nova21:12
*** raunak has quit IRC21:12
dansmithsdague: found a bug, fixing locally and will push it up21:12
*** mvk has joined #openstack-nova21:13
sdaguecool21:13
openstackgerritDan Smith proposed openstack/nova: correctly join the usage to inventory for capacity accounting  https://review.openstack.org/36624521:13
sdaguedansmith: you deleted the fix21:14
dansmithhah21:14
sdaguehttps://review.openstack.org/#/c/366245/6..7/nova/objects/resource_provider.py21:14
dansmithbecause I was testing21:14
dansmithsorry21:14
sdaguethe map_key bit works21:14
sdaguewithout the explicit str()21:14
jaypipessdague: +2 from me.21:15
*** joshblake has quit IRC21:15
dansmithsdague: it doesn't here, it fails21:15
dansmithsdague: with string format21:16
*** ljxiash has quit IRC21:16
openstackgerritDan Smith proposed openstack/nova: correctly join the usage to inventory for capacity accounting  https://review.openstack.org/36624521:16
sdagueodd21:16
dansmithand it should fail of course21:16
*** tonytan_brb has joined #openstack-nova21:17
sdagueoh... right21:17
dansmith        raise KeyError("%s already in usage_map, bad query" % map_key)21:17
dansmith    TypeError: not all arguments converted during string formatting21:17
*** raunak has joined #openstack-nova21:18
*** tri2sing has quit IRC21:19
*** tonytan4ever has quit IRC21:19
*** tri2sing has joined #openstack-nova21:19
*** Apoorva has joined #openstack-nova21:21
*** dimtruck is now known as zz_dimtruck21:23
sdagueyep21:25
sdagueI guess I had just gone blind in my *stacktest* buffer21:26
*** rmart04 has joined #openstack-nova21:26
dansmithyou get a similar enough error that it seems like it's working21:27
sdagueyeh21:27
sdaguejaypipes: ok, you need to swing around with your +2 again, as dansmith had to fix the fix21:27
jaypipessdague: done21:29
sdague\o/ ... ok, time to go make dinner. See you all tomorrow21:29
*** ametts has quit IRC21:30
mriedemalaski: rax doesn't support floating IPs right?21:32
*** rfolco has quit IRC21:32
*** rtheis has quit IRC21:33
jrollmriedem: nope, not today21:34
mriedemok, was just reading this 'Launch instance with Floating IP' dev list thread21:34
*** rmart04 has left #openstack-nova21:34
mriedemabout industry standards21:34
clarkbalso no security groups though thats orthogonal to the thread youare talking about21:35
*** thorst has joined #openstack-nova21:37
*** annegentle has quit IRC21:39
mriedemindustry standard21:41
mriedem'nuff said21:41
*** moshele has quit IRC21:42
*** pradk has quit IRC21:43
* dansmith is "industry standard"21:45
*** thorst has quit IRC21:45
*** diogogmt has quit IRC21:45
*** liang_ has quit IRC21:45
*** esmiurium has quit IRC21:46
jaypipesif I'm ever referred to as "industry standard", it is dansmith's sworn oath to shoot me in the head with a grenade launcher.21:46
dansmithwow, that may be a little overkill21:47
dansmithI think a small-calibre handgun would be plenty21:47
jaypipesdansmith: it's an industry standard response, really.21:47
*** esmiurium has joined #openstack-nova21:47
* dansmith checks on the industry standard for executions21:47
*** tri2sing has quit IRC21:49
*** tri2sing has joined #openstack-nova21:50
*** Sukhdev has quit IRC21:51
*** stevemar has quit IRC21:52
*** Sukhdev has joined #openstack-nova21:55
*** Sukhdev has quit IRC21:59
*** burt has quit IRC22:00
*** zz_dimtruck is now known as dimtruck22:02
*** annegentle has joined #openstack-nova22:02
*** gyee has joined #openstack-nova22:05
*** BjoernT has quit IRC22:05
*** kylek3h has joined #openstack-nova22:06
*** Sukhdev has joined #openstack-nova22:09
*** tri2sing has quit IRC22:10
*** tri2sing has joined #openstack-nova22:11
*** tri2sing has quit IRC22:11
*** baoli has quit IRC22:12
*** baoli has joined #openstack-nova22:13
*** kfarr has quit IRC22:13
*** su_zhang has quit IRC22:14
*** liangy has joined #openstack-nova22:15
jrollgrenade will just bounce off your head and blow off your legs, though :(22:15
jroll:P22:16
*** priteau has quit IRC22:16
*** xyang1 has quit IRC22:17
jaypipesjroll: honestly, after my day today, even that would be a welcome event.22:17
jroll:(22:18
*** eharney has quit IRC22:18
*** mdrabe has quit IRC22:22
*** priteau has joined #openstack-nova22:22
*** stevemar has joined #openstack-nova22:24
*** stevemar has quit IRC22:26
*** marst has quit IRC22:26
*** _ducttape_ has joined #openstack-nova22:26
*** priteau has quit IRC22:26
*** stevemar has joined #openstack-nova22:26
openstackgerritEric Brown proposed openstack/nova: VMware: deprecate wsdl_location conf option  https://review.openstack.org/34247022:26
*** liangy has quit IRC22:27
*** tblakes has quit IRC22:28
*** ducttape_ has quit IRC22:29
*** stevemar has quit IRC22:31
*** liangy has joined #openstack-nova22:32
*** krtaylor has joined #openstack-nova22:32
*** dimtruck is now known as zz_dimtruck22:34
*** yamahata has quit IRC22:40
*** thorst has joined #openstack-nova22:43
*** annegentle has quit IRC22:46
*** zz_dimtruck is now known as dimtruck22:47
*** thorst has quit IRC22:49
*** yamahata has joined #openstack-nova22:49
*** thorst has joined #openstack-nova22:49
*** akshai has quit IRC22:53
*** baoli has quit IRC22:58
*** krtaylor has quit IRC23:06
*** stevemar has joined #openstack-nova23:07
*** mingyu has joined #openstack-nova23:08
*** mingyu has quit IRC23:10
*** ayoung has joined #openstack-nova23:10
*** sdake has joined #openstack-nova23:10
*** marst has joined #openstack-nova23:12
*** ljxiash has joined #openstack-nova23:13
harlowjaqq guys, i'm trying to find a good set of links to share with folks on the topic of (what is cells v1); what do u guys think the best source is? (even though i know its cells v1...)23:18
*** ljxiash has quit IRC23:18
*** markvoelker has quit IRC23:19
harlowjasaid folks are going to try to debug some of the sync issues, so any links people have might be useful (vs me just searching around, ha)23:20
*** jamesdenton has joined #openstack-nova23:21
*** akshai has joined #openstack-nova23:22
*** jaypipes has quit IRC23:25
*** jamesdenton has quit IRC23:26
*** mriedem has quit IRC23:30
*** mriedem has joined #openstack-nova23:32
*** dimtruck is now known as zz_dimtruck23:32
*** akshai has quit IRC23:33
*** bapalm has quit IRC23:35
openstackgerritmelanie witt proposed openstack/nova: Make req spec and RT aware of volume-backed instances  https://review.openstack.org/35509123:38
*** Sukhdev has quit IRC23:40
*** rama_y has quit IRC23:40
*** bapalm has joined #openstack-nova23:41
*** raunak has quit IRC23:41
*** zz_dimtruck is now known as dimtruck23:46
*** mingyu has joined #openstack-nova23:48
*** Jeffrey4l has joined #openstack-nova23:49
*** Jeffrey4l has quit IRC23:55
*** markvoelker has joined #openstack-nova23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!