*** f10d4 has quit IRC | 00:22 | |
*** mriedem has joined #openstack-nova | 00:26 | |
*** mriedem has quit IRC | 00:27 | |
*** felix23ma has quit IRC | 00:29 | |
*** vishwanathj has joined #openstack-nova | 00:35 | |
*** felix23ma has joined #openstack-nova | 00:40 | |
*** thorst has joined #openstack-nova | 00:40 | |
*** hoangcx has joined #openstack-nova | 00:45 | |
*** thorst has quit IRC | 00:47 | |
*** thorst has joined #openstack-nova | 00:47 | |
*** thorst has quit IRC | 00:47 | |
*** ssurana has joined #openstack-nova | 00:48 | |
*** kaisers1 has joined #openstack-nova | 00:51 | |
*** kaisers has quit IRC | 00:51 | |
*** kaisers has joined #openstack-nova | 00:52 | |
*** kaisers2 has quit IRC | 00:52 | |
*** markvoelker has joined #openstack-nova | 00:53 | |
*** thorst has joined #openstack-nova | 00:54 | |
*** lixiaoy1 has joined #openstack-nova | 01:04 | |
*** thorst has quit IRC | 01:05 | |
openstackgerrit | Maciej Szankin proposed openstack/nova: Deprecate all topic conf options https://review.openstack.org/351608 | 01:06 |
---|---|---|
*** matias has quit IRC | 01:11 | |
*** yamahata has joined #openstack-nova | 01:21 | |
*** READ10 has joined #openstack-nova | 01:24 | |
*** kevinz has joined #openstack-nova | 01:28 | |
*** hoangcx has quit IRC | 01:32 | |
*** wangqun has joined #openstack-nova | 01:43 | |
*** liuyulong_ is now known as liuyulong | 01:48 | |
*** baoli has joined #openstack-nova | 01:51 | |
*** baoli_ has joined #openstack-nova | 01:53 | |
*** ijw has joined #openstack-nova | 01:55 | |
*** akshai has joined #openstack-nova | 01:56 | |
*** baoli has quit IRC | 01:56 | |
openstackgerrit | Chen Fan proposed openstack/nova: pci: nova fail to boot when wrong setting in pci_whitelist https://review.openstack.org/357604 | 01:58 |
*** code-R has joined #openstack-nova | 02:00 | |
*** ijw has quit IRC | 02:01 | |
*** EinstCrazy has joined #openstack-nova | 02:01 | |
*** baoli_ has quit IRC | 02:03 | |
*** code-R_ has joined #openstack-nova | 02:03 | |
*** wangqun has quit IRC | 02:03 | |
*** hoangcx has joined #openstack-nova | 02:03 | |
*** wangqun_ has joined #openstack-nova | 02:03 | |
*** ijw has joined #openstack-nova | 02:03 | |
*** ijw has quit IRC | 02:04 | |
*** ijw has joined #openstack-nova | 02:04 | |
*** thorst has joined #openstack-nova | 02:05 | |
*** phuongnh has joined #openstack-nova | 02:05 | |
*** code-R has quit IRC | 02:06 | |
*** ijw has quit IRC | 02:10 | |
openstackgerrit | Nguyen Phuong An proposed openstack/nova: [api-ref]: Correcting server_groups_list parameter's type. https://review.openstack.org/365554 | 02:13 |
*** thorst has quit IRC | 02:13 | |
*** armax has quit IRC | 02:13 | |
*** junjie has quit IRC | 02:14 | |
*** code-R_ has quit IRC | 02:17 | |
*** rfolco has quit IRC | 02:18 | |
*** agireud has quit IRC | 02:18 | |
*** kevinz has quit IRC | 02:18 | |
*** kevinz has joined #openstack-nova | 02:19 | |
*** agireud has joined #openstack-nova | 02:25 | |
*** akshai_ has joined #openstack-nova | 02:29 | |
*** salv-orl_ has joined #openstack-nova | 02:29 | |
*** agireud has quit IRC | 02:30 | |
*** akshai has quit IRC | 02:32 | |
*** salv-orlando has quit IRC | 02:32 | |
*** RuiChen has joined #openstack-nova | 02:33 | |
*** annp has joined #openstack-nova | 02:34 | |
*** ijw has joined #openstack-nova | 02:36 | |
*** thorst has joined #openstack-nova | 02:40 | |
*** agireud has joined #openstack-nova | 02:40 | |
*** ijw has quit IRC | 02:40 | |
*** vishwanathj is now known as vishwanathj_zzz | 02:47 | |
*** amotoki has joined #openstack-nova | 02:48 | |
*** bkopilov has quit IRC | 02:50 | |
*** amotoki has quit IRC | 02:50 | |
openstackgerrit | Li Yingjun proposed openstack/nova: Adds view builders for hypervisors controller https://review.openstack.org/335282 | 02:51 |
*** akshai_ has quit IRC | 02:52 | |
*** markvoelker has quit IRC | 02:53 | |
*** ijw has joined #openstack-nova | 03:00 | |
*** sudipto has joined #openstack-nova | 03:00 | |
*** sudipto_ has joined #openstack-nova | 03:00 | |
*** hoangcx has quit IRC | 03:01 | |
*** houming has joined #openstack-nova | 03:04 | |
*** ijw has quit IRC | 03:05 | |
*** amotoki has joined #openstack-nova | 03:07 | |
*** hoangcx has joined #openstack-nova | 03:12 | |
*** ssurana has quit IRC | 03:13 | |
*** amotoki has quit IRC | 03:18 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox from compute/test_volumes.py https://review.openstack.org/323257 | 03:40 |
*** READ10 has quit IRC | 03:42 | |
*** ijw has joined #openstack-nova | 03:44 | |
*** adisky has joined #openstack-nova | 03:47 | |
*** ijw has quit IRC | 03:48 | |
openstackgerrit | Jianghua Wang proposed openstack/nova: XenAPI: resolve the fetch_bandwidth failure https://review.openstack.org/365838 | 03:50 |
*** diga has joined #openstack-nova | 03:51 | |
*** coolsvap_ has joined #openstack-nova | 03:52 | |
*** amotoki has joined #openstack-nova | 03:54 | |
openstackgerrit | Eli Qiao proposed openstack/nova: Libvirt: Correct PERF_EVENTS_CPU_FLAG_MAPPING https://review.openstack.org/365842 | 03:54 |
*** thorst has quit IRC | 03:58 | |
*** thorst has joined #openstack-nova | 03:58 | |
*** ijw has joined #openstack-nova | 04:00 | |
*** thorst has quit IRC | 04:00 | |
*** prateek has joined #openstack-nova | 04:01 | |
*** Sukhdev has joined #openstack-nova | 04:01 | |
*** prateek has quit IRC | 04:03 | |
*** thorst has joined #openstack-nova | 04:03 | |
*** sdake has joined #openstack-nova | 04:05 | |
*** sudipto_ has quit IRC | 04:06 | |
*** sudipto has quit IRC | 04:06 | |
*** thorst has quit IRC | 04:08 | |
*** ijw has quit IRC | 04:09 | |
*** thorst has joined #openstack-nova | 04:10 | |
*** lixiaoy1 has quit IRC | 04:12 | |
*** thorst has quit IRC | 04:15 | |
*** bkopilov has joined #openstack-nova | 04:24 | |
*** sdake has quit IRC | 04:25 | |
*** ijw has joined #openstack-nova | 04:30 | |
murali__ | Hello | 04:31 |
murali__ | Someone have a look https://ask.openstack.org/en/question/96444/nova-docker-issue-in-launching-the-container/ | 04:31 |
*** moshele has joined #openstack-nova | 04:34 | |
*** links has joined #openstack-nova | 04:39 | |
*** ijw has quit IRC | 04:40 | |
*** ijw has joined #openstack-nova | 04:41 | |
*** ijw has quit IRC | 04:42 | |
*** harlowja_at_home has joined #openstack-nova | 04:42 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in compute/test_neutron_security_groups.py https://review.openstack.org/323093 | 04:44 |
*** harlowja_at_home has quit IRC | 04:47 | |
*** lixiaoy1 has joined #openstack-nova | 04:47 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in unit/fake_processutils.py https://review.openstack.org/324239 | 04:48 |
*** sdake has joined #openstack-nova | 04:50 | |
*** ekuris has joined #openstack-nova | 04:51 | |
*** sudipto has joined #openstack-nova | 04:53 | |
*** sudipto_ has joined #openstack-nova | 04:53 | |
*** dikonoor has joined #openstack-nova | 04:55 | |
*** thorst has joined #openstack-nova | 04:55 | |
*** su_zhang has joined #openstack-nova | 04:57 | |
*** thorst has quit IRC | 04:59 | |
*** sdake has quit IRC | 05:01 | |
*** irenab has joined #openstack-nova | 05:04 | |
*** oanson has joined #openstack-nova | 05:05 | |
*** irenab_ has joined #openstack-nova | 05:06 | |
*** dikonoor has quit IRC | 05:06 | |
*** thorst has joined #openstack-nova | 05:08 | |
*** irenab has quit IRC | 05:08 | |
*** irenab_ is now known as irenab | 05:08 | |
*** sdake has joined #openstack-nova | 05:10 | |
*** ratailor has joined #openstack-nova | 05:11 | |
*** mamandle has joined #openstack-nova | 05:11 | |
*** wangqun_ has quit IRC | 05:11 | |
*** wangqun_ has joined #openstack-nova | 05:11 | |
*** sdake has quit IRC | 05:17 | |
*** diogogmt has quit IRC | 05:19 | |
*** su_zhang has quit IRC | 05:21 | |
*** jaosorior has joined #openstack-nova | 05:21 | |
*** su_zhang has joined #openstack-nova | 05:22 | |
*** thorst has quit IRC | 05:25 | |
*** prateek has joined #openstack-nova | 05:26 | |
*** su_zhang has quit IRC | 05:26 | |
*** ijw has joined #openstack-nova | 05:29 | |
*** thorst has joined #openstack-nova | 05:29 | |
*** zzzeek has quit IRC | 05:30 | |
*** sridharg has joined #openstack-nova | 05:31 | |
*** ijw_ has joined #openstack-nova | 05:31 | |
*** nkrinner_afk is now known as nkrinner | 05:32 | |
*** ijw has quit IRC | 05:32 | |
*** thorst has quit IRC | 05:33 | |
*** ijw_ has quit IRC | 05:35 | |
*** ijw has joined #openstack-nova | 05:38 | |
*** EinstCrazy has quit IRC | 05:40 | |
*** ijw has quit IRC | 05:40 | |
*** diogogmt has joined #openstack-nova | 05:41 | |
*** liverpooler has quit IRC | 05:42 | |
*** ijw has joined #openstack-nova | 05:43 | |
*** lixiaoy1 has quit IRC | 05:45 | |
*** lixiaoy11 has joined #openstack-nova | 05:45 | |
*** ijw has quit IRC | 05:46 | |
*** sudipto has quit IRC | 05:47 | |
*** sudipto_ has quit IRC | 05:47 | |
*** ijw has joined #openstack-nova | 05:47 | |
*** hoonetorg has quit IRC | 05:48 | |
*** mjura has joined #openstack-nova | 05:51 | |
*** EinstCrazy has joined #openstack-nova | 05:52 | |
*** sudipto has joined #openstack-nova | 05:53 | |
openstackgerrit | Nguyen Phuong An proposed openstack/nova: Update info of machine-processable description https://review.openstack.org/365426 | 05:55 |
*** unicell1 has joined #openstack-nova | 05:55 | |
*** unicell has quit IRC | 05:56 | |
*** ijw has quit IRC | 05:57 | |
*** ijw has joined #openstack-nova | 05:57 | |
murali__ | Someone have a look here : https://ask.openstack.org/en/question/96444/nova-docker-issue-in-launching-the-container/ | 05:58 |
murali__ | I am checking on this issue now | 05:58 |
*** mjura has quit IRC | 05:58 | |
*** mjura has joined #openstack-nova | 05:59 | |
openstackgerrit | huan proposed openstack/nova: xenapi: support the hotplug of a neutron port https://review.openstack.org/365505 | 06:00 |
*** ijw has quit IRC | 06:01 | |
*** rcarrillocruz has quit IRC | 06:02 | |
*** ijw has joined #openstack-nova | 06:02 | |
*** ijw has quit IRC | 06:03 | |
openstackgerrit | Eli Qiao proposed openstack/nova: Libvirt: Correct PERF_EVENTS_CPU_FLAG_MAPPING https://review.openstack.org/365842 | 06:03 |
*** paul-carlton has joined #openstack-nova | 06:04 | |
*** flip214 has left #openstack-nova | 06:09 | |
*** ijw has joined #openstack-nova | 06:10 | |
*** hoonetorg has joined #openstack-nova | 06:12 | |
*** raunak has joined #openstack-nova | 06:14 | |
*** ttrumm has joined #openstack-nova | 06:21 | |
*** EinstCrazy has quit IRC | 06:21 | |
*** sean-k-m1 has joined #openstack-nova | 06:22 | |
*** EinstCrazy has joined #openstack-nova | 06:23 | |
*** sean-k-mooney has quit IRC | 06:24 | |
slaweq_ | mdbooth: hello, I pushed yesterday clean-up patch which we talked about: https://review.openstack.org/#/c/365756/ | 06:25 |
slaweq_ | please take a look if You will have time | 06:25 |
*** ijw has quit IRC | 06:25 | |
*** dimtruck is now known as zz_dimtruck | 06:27 | |
*** ihrachys has joined #openstack-nova | 06:29 | |
*** claudiub|2 has joined #openstack-nova | 06:29 | |
*** claudiub has joined #openstack-nova | 06:31 | |
*** raunak_ has joined #openstack-nova | 06:34 | |
*** raunak has quit IRC | 06:34 | |
*** raunak_ is now known as raunak | 06:34 | |
*** claudiub|2 has quit IRC | 06:34 | |
*** felix23ma has quit IRC | 06:42 | |
*** felix23ma has joined #openstack-nova | 06:43 | |
*** andreas_s has joined #openstack-nova | 06:45 | |
*** zz_dimtruck is now known as dimtruck | 06:48 | |
*** pcaruana has joined #openstack-nova | 06:48 | |
*** felix23ma has quit IRC | 06:54 | |
*** markvoelker has joined #openstack-nova | 06:54 | |
*** dimtruck is now known as zz_dimtruck | 06:58 | |
*** markvoelker has quit IRC | 06:59 | |
*** tesseract- has joined #openstack-nova | 07:00 | |
*** EinstCra_ has joined #openstack-nova | 07:00 | |
*** EinstCrazy has quit IRC | 07:00 | |
*** ijw has joined #openstack-nova | 07:00 | |
*** zz_dimtruck is now known as dimtruck | 07:00 | |
*** ijw has quit IRC | 07:00 | |
*** ijw has joined #openstack-nova | 07:00 | |
*** hoangcx has quit IRC | 07:01 | |
*** felix23ma has joined #openstack-nova | 07:02 | |
*** hoangcx has joined #openstack-nova | 07:04 | |
*** sudipto has quit IRC | 07:05 | |
*** wangqun has joined #openstack-nova | 07:05 | |
*** wangqun_ has quit IRC | 07:06 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in api/openstack/compute/test_floating_ips.py https://review.openstack.org/322668 | 07:06 |
*** Sukhdev has quit IRC | 07:06 | |
*** Sukhdev has joined #openstack-nova | 07:06 | |
*** EinstCra_ has quit IRC | 07:09 | |
*** matrohon has joined #openstack-nova | 07:09 | |
*** dimtruck is now known as zz_dimtruck | 07:10 | |
*** liverpooler has joined #openstack-nova | 07:10 | |
*** sahid has joined #openstack-nova | 07:10 | |
*** Sukhdev has quit IRC | 07:11 | |
*** ihrachys has quit IRC | 07:12 | |
*** ihrachys has joined #openstack-nova | 07:12 | |
*** slaweq__ has joined #openstack-nova | 07:14 | |
*** ttrumm has quit IRC | 07:14 | |
*** lixiaoy11 has quit IRC | 07:14 | |
*** ttrumm has joined #openstack-nova | 07:15 | |
*** lixiaoy1 has joined #openstack-nova | 07:16 | |
*** EinstCrazy has joined #openstack-nova | 07:17 | |
*** slaweq_ has quit IRC | 07:17 | |
*** raunak has quit IRC | 07:18 | |
ratailor | openstack-nova, openstack pl review https://review.openstack.org/#/c/312887/ | 07:21 |
*** ihrachys has quit IRC | 07:23 | |
*** ljxiash has joined #openstack-nova | 07:24 | |
*** chrichip has quit IRC | 07:25 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in api/openstack/compute/test_remote_consoles.py https://review.openstack.org/323102 | 07:28 |
*** felix23ma has quit IRC | 07:32 | |
*** ijw has quit IRC | 07:35 | |
*** gszasz has joined #openstack-nova | 07:35 | |
*** ijw has joined #openstack-nova | 07:36 | |
*** mamandle has quit IRC | 07:39 | |
*** felix23ma has joined #openstack-nova | 07:41 | |
prateek | chal | 07:43 |
*** zenoway has joined #openstack-nova | 07:47 | |
*** daemontool has joined #openstack-nova | 07:54 | |
*** paul-carlton has quit IRC | 07:59 | |
*** lpetrut has joined #openstack-nova | 08:02 | |
*** yassine has joined #openstack-nova | 08:04 | |
*** yassine is now known as Guest65745 | 08:04 | |
*** Guest65745 is now known as yassou | 08:05 | |
*** lpetrut has quit IRC | 08:07 | |
*** jistr is now known as jistr|dentist | 08:07 | |
*** EinstCrazy has quit IRC | 08:11 | |
openstackgerrit | Chris Dent proposed openstack/nova: [placement] Add some tests ensuring unicode resource provider name https://review.openstack.org/365688 | 08:12 |
*** mamandle has joined #openstack-nova | 08:14 | |
openstackgerrit | Ha Van Tu proposed openstack/nova: Remove unused parameters https://review.openstack.org/365929 | 08:15 |
*** derekh has joined #openstack-nova | 08:15 | |
*** lucas-dinner is now known as lucasagomes | 08:16 | |
*** ijw has quit IRC | 08:17 | |
*** shangxdy has joined #openstack-nova | 08:17 | |
*** markvoelker has joined #openstack-nova | 08:19 | |
*** code-R has joined #openstack-nova | 08:20 | |
*** mvk has quit IRC | 08:22 | |
*** code-R_ has joined #openstack-nova | 08:22 | |
*** markus_z has joined #openstack-nova | 08:22 | |
*** paul-carlton has joined #openstack-nova | 08:22 | |
*** ihrachys has joined #openstack-nova | 08:23 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in compute/test_security_groups.py https://review.openstack.org/323122 | 08:23 |
*** markvoelker has quit IRC | 08:24 | |
*** hshiina has joined #openstack-nova | 08:24 | |
*** code-R has quit IRC | 08:25 | |
*** gongysh has quit IRC | 08:25 | |
*** gongysh has joined #openstack-nova | 08:26 | |
*** EinstCrazy has joined #openstack-nova | 08:27 | |
*** Jeffrey4l has quit IRC | 08:28 | |
*** EinstCrazy has quit IRC | 08:28 | |
*** EinstCrazy has joined #openstack-nova | 08:28 | |
*** mnestratov has joined #openstack-nova | 08:29 | |
*** salv-orlando has joined #openstack-nova | 08:29 | |
openstackgerrit | Zhenyu Zheng proposed openstack/nova: Trival fix a typo https://review.openstack.org/365938 | 08:31 |
*** salv-orl_ has quit IRC | 08:32 | |
*** roeyc has joined #openstack-nova | 08:32 | |
*** sudipto has joined #openstack-nova | 08:34 | |
*** danpawlik1 has joined #openstack-nova | 08:34 | |
danpawlik1 | Hello, if you have some free time pls check https://review.openstack.org/#/c/314928/ :) | 08:35 |
openstackgerrit | Maxim Nestratov proposed openstack/nova: libvirt: add supported vif types for virtuozzo virt_type https://review.openstack.org/365943 | 08:35 |
*** ababich has joined #openstack-nova | 08:35 | |
*** ijw has joined #openstack-nova | 08:39 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in virt/libvirt/test_imagecache.py https://review.openstack.org/323717 | 08:44 |
*** ecelik has quit IRC | 08:47 | |
*** ecelik has joined #openstack-nova | 08:48 | |
*** hshiina has quit IRC | 08:50 | |
*** mvk has joined #openstack-nova | 08:51 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in compute/test_server_actions.py https://review.openstack.org/323186 | 08:52 |
*** lixiaoy1 has quit IRC | 08:57 | |
*** EinstCrazy has quit IRC | 08:59 | |
*** hoangcx has quit IRC | 08:59 | |
*** ecelik has quit IRC | 09:00 | |
*** ecelik has joined #openstack-nova | 09:00 | |
*** zz_dimtruck is now known as dimtruck | 09:01 | |
*** EinstCrazy has joined #openstack-nova | 09:01 | |
*** ecelik has quit IRC | 09:04 | |
*** ecelik has joined #openstack-nova | 09:04 | |
*** ijw has quit IRC | 09:07 | |
*** rwmjones has quit IRC | 09:08 | |
*** dimtruck is now known as zz_dimtruck | 09:10 | |
*** rwmjones has joined #openstack-nova | 09:12 | |
*** danpb has joined #openstack-nova | 09:16 | |
*** ijw has joined #openstack-nova | 09:18 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox int unit/virt/xenapi/image/test_utils.py https://review.openstack.org/342626 | 09:18 |
*** gongysh has quit IRC | 09:19 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in virt/xenapi/image/test_vdi_through_dev.py https://review.openstack.org/342682 | 09:21 |
*** gongysh has joined #openstack-nova | 09:21 | |
*** jistr|dentist is now known as jistr | 09:23 | |
*** mfedosin has joined #openstack-nova | 09:25 | |
*** gongysh has quit IRC | 09:26 | |
*** daemontool has quit IRC | 09:29 | |
*** lpetrut has joined #openstack-nova | 09:30 | |
*** lpetrut has quit IRC | 09:30 | |
*** lpetrut has joined #openstack-nova | 09:31 | |
*** shangxdy has quit IRC | 09:31 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in unit/virt/xenapi/test_vm_utils.py (1) https://review.openstack.org/344054 | 09:35 |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in unit/virt/test_block_device.py https://review.openstack.org/342595 | 09:37 |
*** roeyc has quit IRC | 09:38 | |
*** mvk has quit IRC | 09:38 | |
*** cdent has joined #openstack-nova | 09:39 | |
*** PaulMurray has joined #openstack-nova | 09:39 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in unit/cells/test_cells_rpc_driver.py https://review.openstack.org/333101 | 09:39 |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox int unit/virt/xenapi/stubs.py https://review.openstack.org/334848 | 09:43 |
*** sdake has joined #openstack-nova | 09:45 | |
openstackgerrit | Dao Cong Tien proposed openstack/nova: Stop using mox in unit/virt/xenapi/test_vmops.py (2/2) https://review.openstack.org/344010 | 09:46 |
*** zenoway has quit IRC | 09:46 | |
*** zenoway has joined #openstack-nova | 09:47 | |
*** zenoway has quit IRC | 09:51 | |
*** sean-k-m1 is now known as sean-k-mooney | 09:53 | |
*** annp has quit IRC | 09:57 | |
*** mvk has joined #openstack-nova | 09:58 | |
*** mnestratov has quit IRC | 10:00 | |
*** zenoway has joined #openstack-nova | 10:00 | |
*** zenoway has quit IRC | 10:01 | |
*** zenoway has joined #openstack-nova | 10:01 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova: Body Verification of os-aggregates.inc https://review.openstack.org/366023 | 10:03 |
*** zz_dimtruck is now known as dimtruck | 10:06 | |
*** Sukhdev has joined #openstack-nova | 10:07 | |
mdbooth | ftersin: Sorry, you got a load of my frustration this morning :) Thanks for all the reviews. | 10:11 |
* mdbooth tries to keep it bottled, but doesn't always quite succeed. | 10:12 | |
mdbooth | That's healthy, right? | 10:12 |
*** Sukhdev has quit IRC | 10:13 | |
*** phuongnh has quit IRC | 10:14 | |
*** liuyulong is now known as liuyulong|away | 10:15 | |
*** riccardo has joined #openstack-nova | 10:16 | |
ftersin | bdbooth: just found a bug of ephemeral resizing, went to find a solution, remembered that you discussed it here, and start to investigate the current state of your work | 10:16 |
*** roeyc has joined #openstack-nova | 10:18 | |
mdbooth | ftersin: Yeah, hasn't worked ever, I think | 10:18 |
mdbooth | That patch half fixes it. I think I note that it's not fixed in one of the unit tests, don't I? | 10:19 |
*** ijw has quit IRC | 10:20 | |
mdbooth | Incidentally, remember that 'disk.local' is never created for new instances, and hasn't been for releases covering several years at least. | 10:20 |
riccardo | hi guys, when I google for something like, how to connect to DB, it always displays results for openstack administrators. How can I search for contributer/programmer stuff? Any idea? | 10:20 |
mdbooth | Which basically means that 'disk.local' is rare in the wild, and quite possibly extinct | 10:20 |
mdbooth | A single ephemeral disks created today is 'disk.eph0' | 10:21 |
ftersin | 'disk.local' is a clear example of the unit tests quality :/ | 10:22 |
mdbooth | Yup | 10:22 |
ftersin | ephN is used since bdm v2 | 10:22 |
mdbooth | Indeed | 10:22 |
openstackgerrit | Rajesh Tailor proposed openstack/nova: Fix instance delete stuck in deleting task_state https://review.openstack.org/294491 | 10:24 |
*** sahid has quit IRC | 10:28 | |
ftersin | mdbooth: "That patch half fixes it" - which patch do you mean? | 10:28 |
mdbooth | You commented on it in a review one of my patches | 10:28 |
mdbooth | It's one of the _create_image patches | 10:29 |
*** kaisers2 has joined #openstack-nova | 10:29 | |
*** sahid has joined #openstack-nova | 10:29 | |
*** kaisers1 has quit IRC | 10:29 | |
*** kaisers1 has joined #openstack-nova | 10:30 | |
mdbooth | ftersin: This one: https://review.openstack.org/#/c/331118/15/nova/virt/libvirt/driver.py | 10:30 |
*** Oku_OS is now known as Oku_OS-away | 10:30 | |
*** kaisers has quit IRC | 10:30 | |
*** Oku_OS-away is now known as Oku_OS | 10:31 | |
mdbooth | Half fixed == not fixed, of course :) | 10:32 |
* mdbooth is just trying to find the code which mangles bdms in that call path. I've found it before, but it's currently eluding me. | 10:32 | |
*** ecelik has quit IRC | 10:32 | |
mdbooth | It's either in compute api, conductor, or compute manager. | 10:32 |
*** PaulMurray has quit IRC | 10:33 | |
ftersin | mdboot: well, i think that if nova with this fix can resize or add a single ephemeral, we may amend the case with numerous ephemerals | 10:33 |
mdbooth | Anyway, the instance we're passed has had its metadata resized, but its bdms have not been resized. | 10:33 |
*** sdake has quit IRC | 10:33 | |
mdbooth | Because the resize code uses bdms, it means we resize, but to the same size as before | 10:33 |
mdbooth | Anyway, with that change it will at least iterate over ephemerals, but it'll uselessly resize them to their existing size. | 10:34 |
mdbooth | We need to also fix the caller. | 10:34 |
mdbooth | I'm determined to find that code again :) | 10:34 |
ftersin | if your series is blocked by newton code freeze, does it make sence to push https://review.openstack.org/#/c/346263 at the moment? | 10:35 |
*** ttrumm_ has joined #openstack-nova | 10:35 | |
*** zenoway has quit IRC | 10:36 | |
mdbooth | ftersin: I have been +1 on several patches already which will conflict with my series for exactly that reason. | 10:36 |
mdbooth | Yes, expedient bug fixes are always good. | 10:36 |
*** zenoway has joined #openstack-nova | 10:36 | |
*** ecelik has joined #openstack-nova | 10:37 | |
mdbooth | They're also easier to backport. | 10:37 |
mdbooth | ftersin: I've already reviewed that patch and was in favour of it, but it needs a test. | 10:38 |
*** roeyc has quit IRC | 10:38 | |
slaweq__ | mdbooth: hello | 10:38 |
*** wangqun has quit IRC | 10:38 | |
slaweq__ | mdbooth: do You know who from core team I could ask to check https://review.openstack.org/#/c/363926/ ? | 10:38 |
*** tbachman has quit IRC | 10:39 | |
*** ttrumm has quit IRC | 10:39 | |
ftersin | mdboot: but you just sayed that it requires bdm update as well, right? | 10:39 |
mdbooth | Anybody you can get :) Bug fixes are the priority right now, so hopefully you'll get some attention. | 10:39 |
*** code-R_ has quit IRC | 10:39 | |
ftersin | so that not test only, but some other updates... | 10:40 |
mdbooth | ftersin: Ah, yes, I didn't read the whole context. Yes, I don't think that will adress the whole problem. | 10:40 |
*** zenoway has quit IRC | 10:40 | |
*** ecelik has quit IRC | 10:41 | |
ftersin | i've asked cfriesen, if he wants to push it to newton and legacy releases. waiting for an answer | 10:41 |
ftersin | ok, thanks | 10:41 |
*** cdent has quit IRC | 10:42 | |
openstackgerrit | Alberto Planas proposed openstack/nova: Add migrate_uri for invoking the migration https://review.openstack.org/356558 | 10:43 |
*** zenoway has joined #openstack-nova | 10:44 | |
*** PaulMurray has joined #openstack-nova | 10:44 | |
*** roeyc has joined #openstack-nova | 10:45 | |
*** HenryG_ is now known as HenryG | 10:46 | |
*** lyarwood is now known as lyarwood_ | 10:46 | |
*** baoli has joined #openstack-nova | 10:46 | |
*** lyarwood_ is now known as lyarwood | 10:46 | |
*** esracelik has joined #openstack-nova | 10:47 | |
*** zengine has quit IRC | 10:48 | |
mdbooth | ftersin: It's in ComputeManager._finish_resize | 10:49 |
mdbooth | This is where compute manager resizes the instance's metadata | 10:49 |
mdbooth | However, it doesn't update bdms, which are the only thing we can reasonably look at for ephemerals | 10:50 |
mdbooth | We need to also poke bdms in there before calling driver.finish_migration() | 10:50 |
*** _gryf has quit IRC | 10:50 | |
*** zzzeek has joined #openstack-nova | 10:51 | |
mdbooth | With that, and the fix the both cfriesen and I have done independently, resize of ephemerals should work. | 10:51 |
*** _gryf has joined #openstack-nova | 10:51 | |
ftersin | but whether we must distinguish between bdms added implicitly by flavor and explicitly by user/snapshot? | 10:52 |
sfinucan | mriedem: When you're online, could I get the -2 removed from this centralize-config-options patch? https://review.openstack.org/#/c/357061/ | 10:53 |
ftersin | i.e. if a user run instance on a flavor with 20 GB ephemeral, specified 10 GB only in bdm, and then resize the instance to 40 GB ephemeral flavor, what to do with the ephemeral? | 10:54 |
*** EinstCrazy has quit IRC | 10:54 | |
mdbooth | danpawlik1: I'd recommend against block adding core reviewers, btw. You risk annoying them :/ | 10:55 |
danpb | indeed, reviews with people block-added go to the bottom of my priority list :-) | 10:55 |
mdbooth | ftersin: Whatever we do, that's nothing to do with the driver. | 10:56 |
*** zzzeek has quit IRC | 10:56 | |
mdbooth | But yes, that's a good question. There's also: | 10:56 |
danpawlik1 | mdbooth: sorry for that | 10:56 |
mdbooth | Flavor specifies 20GB ephemeral, user adds 2 ephemerals 10GB each. | 10:56 |
mdbooth | User resizes to flavor with 40GB ephemeral. What does the resize do? | 10:57 |
ftersin | mdbooth: this is another case - how to split inreased space onto exising ephemerals | 10:57 |
*** zzzeek has joined #openstack-nova | 10:57 | |
danpb | mdbooth: i'd just split it proportionally across the disks | 10:57 |
*** baoli has quit IRC | 10:57 | |
danpb | there's nothing much else you can do that's sane, given the API limitations | 10:57 |
riccardo | hi guys, when I google for something like, "how to connect to DB", it always displays results for openstack administrators. How can I search for contributer/programmer stuff? Any idea? | 10:57 |
mdbooth | ftersin: We discussed this incredibly briefly ages ago, and the answer at the time was: it's always been broken anyway, so don't sweat it. Just fix the case where there's a single, full size ephemeral disk and leave anything else for a future design change. | 10:58 |
*** kaisers2 has quit IRC | 10:58 | |
mdbooth | However, ephemerals, especially multiple ephemerals, have been broken for so long in so many ways, I'm not personally convinced anybody's using them. | 10:58 |
*** kaisers1 has quit IRC | 10:59 | |
mdbooth | danpb: I think it's safer to leave it alone in that case, tbh. | 10:59 |
*** kaisers has joined #openstack-nova | 10:59 | |
*** kaisers1 has joined #openstack-nova | 10:59 | |
*** baoli has joined #openstack-nova | 10:59 | |
mdbooth | Seems like a lot of hassle for an unused feature. | 10:59 |
*** baoli has quit IRC | 11:01 | |
openstackgerrit | Claudiu Belu proposed openstack/nova-specs: Adds spec for instance live resize https://review.openstack.org/141219 | 11:02 |
ftersin | mdbooth: but if a feature is broken, who uses it? how customers do live now with no ephemeral resize? do workaround via snapshots? | 11:03 |
mdbooth | That's the thing, maybe they don't use it. What's the use case for multiple ephemerals anyway? | 11:04 |
mdbooth | Why would a user ever request an ephemeral smaller than what they've paid for? | 11:04 |
*** Jeffrey4l has joined #openstack-nova | 11:05 | |
mdbooth | I wonder if we've designed features which users don't need or use. | 11:05 |
danpb | mdbooth: i can see people wanting separate disks for separate usage scenarios | 11:05 |
*** cdent has joined #openstack-nova | 11:05 | |
ftersin | i would want to split if i wanted to separate fast space between different services or data purposes | 11:06 |
danpb | mdbooth: eg, one might contain /var/www/ while the other contained /home | 11:06 |
mdbooth | danpb: Maybe if their workload uses raw disk? | 11:06 |
danpb | yes you could use partitions, but using separate disks means you avoid contention in the i.o stack so theoretically could get better performance | 11:06 |
danpb | i agree though that use separate disks is a niche use case | 11:06 |
mdbooth | It's still the same disk on the backend :) | 11:07 |
danpb | mdbooth: that's a mere impl detail | 11:07 |
mdbooth | So qemu might use a slightly different path, but the spinning rust/magical cloud bits are the same. | 11:07 |
danpb | from an API pov it makes sense | 11:07 |
ftersin | i would want to limit my ephemeral, if i did not need it so much, but needed vcpus for example, and given flavor set did not give me a big choice, but i needed to honor disk quota | 11:07 |
danpb | its just that our impl isn't taking advantage of it | 11:07 |
danpb | mdbooth: you will at least have separate i/o queues in the guest os side which can improve performance in a multi vcpu guest | 11:08 |
mdbooth | danpb: That's somewhere I wanted to go after persistent storage metadata, btw. That will allow us to specify different backends for different disks. | 11:08 |
mdbooth | Right now, though, there's nothing. | 11:08 |
danpb | yeah, people have requested separate backends for local disks before | 11:08 |
mdbooth | ftersin: Anyway, I'm not disagreeing with you, just expounding the virtues of expediently fixing the thing with an obvious fix, and leaving the rest for another time. | 11:10 |
mdbooth | ftersin: How important is ephemeral resize to you, btw? | 11:10 |
mdbooth | Have you got users complaining about this? | 11:10 |
ftersin | that's thing i agree too | 11:11 |
*** PaulMurray has quit IRC | 11:11 | |
ftersin | i'm blocked with this on scalio driver | 11:11 |
mdbooth | Ephemeral resize? | 11:11 |
ftersin | not directly, but yes | 11:12 |
mdbooth | Well, it doesn't work with any other driver, so why should it block you? | 11:12 |
ftersin | scaleio volumes need to be mapped to a compute host | 11:12 |
danpb | yeah, i'm not seeing why instance resize is a blocker in general | 11:12 |
*** ljxiash has quit IRC | 11:13 | |
ftersin | when the resizing moves the instance to another host, volumes must be remapped to that host | 11:13 |
*** PaulMurray has joined #openstack-nova | 11:13 | |
*** roeyc has quit IRC | 11:14 | |
ftersin | imagebackend's create_image is the only place where i can do it | 11:14 |
ftersin | so i need create_image to be called inside finish_migration | 11:15 |
ftersin | for each disk, including ephemerals | 11:15 |
mdbooth | I see no problem with that. | 11:15 |
ftersin | even if size of ephemeral is not changed | 11:15 |
mdbooth | Does cfriesen's patch fix it for you? | 11:15 |
ftersin | hope for that, but have not checked yet | 11:16 |
mdbooth | However, I think that's the wrong place to do that. | 11:16 |
mdbooth | I'm going to open a can of worms here, stay with me, but this is also broken for Rbd. | 11:17 |
mdbooth | Which, iiuc, is the same model. | 11:17 |
mdbooth | i.e. remote storage, which we're going to connect to from the new host and not actually copy. Right? | 11:17 |
ftersin | rbd does not require mapping operation, though | 11:17 |
mdbooth | Right, but it does exclude it from the copy. | 11:17 |
mdbooth | Which I'm assuming you also have to be careful to do. | 11:18 |
ftersin | which copy operation you mean? | 11:18 |
mdbooth | migrate_disk_and_power_off | 11:18 |
mdbooth | You'll presumably need to ensure that your scaleio disks aren't copied there if they're just going to be moved | 11:18 |
mdbooth | Involving some kind of shared storage negotiation | 11:18 |
*** cdelatte has joined #openstack-nova | 11:19 | |
mdbooth | So, for scaleio do you need to attach the disks to the *host*? | 11:19 |
ftersin | rbd and scaleio are shared block storage, no one need to copy data of them | 11:19 |
mdbooth | So, for Rbd there's no host operation required because qemu talks rbd directly | 11:20 |
mdbooth | But iiuc, the scaleio thing is more akin to attaching a volume | 11:20 |
mdbooth | Is it persistent across host reboots? | 11:20 |
ftersin | moreover, a lot of driver code does copy and other things with files, which comes from libvirt domain via get_instance_disk_info | 11:20 |
ftersin | but rbd devices are not returned from there | 11:21 |
mdbooth | Yeah, that's gross. | 11:21 |
ftersin | i did the same for scaleio ones | 11:21 |
mdbooth | That's how they're exluded from the copy. | 11:21 |
mdbooth | I'm just specifically thinking about the attach thing. | 11:21 |
mdbooth | If it's not persistent across host reboots, then you can't do it in create_image anyway. | 11:22 |
*** ljxiash has joined #openstack-nova | 11:22 | |
ftersin | yes. otherwise it do not work at all | 11:22 |
mdbooth | You need another hook. | 11:22 |
*** lucasagomes is now known as lucas-hungry | 11:22 | |
ftersin | scaleio mapping is persistent | 11:22 |
*** rtheis has joined #openstack-nova | 11:22 | |
mdbooth | Ok, cool. | 11:23 |
mdbooth | So, now I think I understand your comment about ensure_backing. | 11:23 |
*** roeyc has joined #openstack-nova | 11:23 | |
mdbooth | Because that's basically exactly the same hook you want for scalio. | 11:23 |
mdbooth | Namely, don't copy anything, just ensure the disk exists at the remote end. | 11:24 |
*** bkopilov has quit IRC | 11:27 | |
*** ijw has joined #openstack-nova | 11:27 | |
riccardo | how can i read and write to MariaDB? Is there some code in oslo or do I need to write everything from the scratch? | 11:29 |
ftersin | but.. since i already added hooks to map and unmap all volumes, probably it is better just to add another hook to map all volumes in finish_migration | 11:29 |
*** amotoki_ has joined #openstack-nova | 11:29 | |
*** ijw has quit IRC | 11:29 | |
*** ijw has joined #openstack-nova | 11:30 | |
*** amotoki has quit IRC | 11:31 | |
ftersin | mdbooth: exactly. i added a hook to map all instance disks to pre_live_migration, so i just add the same to finish_migration, and do not worry about resizing | 11:31 |
ftersin | excellent solution. thanks for the useful conversation | 11:32 |
*** ljxiash has quit IRC | 11:33 | |
ftersin | danpb: about people question. did anyone ask to have cinder as the ephemeral backend? | 11:36 |
*** mfedosin has quit IRC | 11:37 | |
danpb | i don't know | 11:37 |
*** sdague has joined #openstack-nova | 11:39 | |
riccardo | how can I read in the database how many hosts are online? | 11:39 |
*** PaulMurray has quit IRC | 11:41 | |
*** kaisers has quit IRC | 11:41 | |
*** fragatina has joined #openstack-nova | 11:42 | |
*** kaisers has joined #openstack-nova | 11:42 | |
*** kaisers1 has quit IRC | 11:43 | |
*** kaisers1 has joined #openstack-nova | 11:43 | |
*** fragatina has quit IRC | 11:43 | |
*** mamandle has quit IRC | 11:45 | |
*** kaisers2 has joined #openstack-nova | 11:46 | |
*** kaisers3 has joined #openstack-nova | 11:47 | |
*** kaisers1 has quit IRC | 11:48 | |
*** kaisers has quit IRC | 11:49 | |
*** mfedosin has joined #openstack-nova | 11:51 | |
*** salv-orl_ has joined #openstack-nova | 11:52 | |
*** salv-orlando has quit IRC | 11:52 | |
*** code-R has joined #openstack-nova | 11:54 | |
*** sudipto has quit IRC | 11:55 | |
*** kaisers3 has quit IRC | 11:55 | |
*** kaisers has joined #openstack-nova | 11:55 | |
*** code-R_ has joined #openstack-nova | 11:58 | |
*** code-R has quit IRC | 12:01 | |
*** daemontool has joined #openstack-nova | 12:04 | |
*** gcb has quit IRC | 12:04 | |
sdague | cdent: are you planning on cleaning up the log string nits in - https://review.openstack.org/#/c/365015/ ? | 12:04 |
*** ijw has quit IRC | 12:05 | |
cdent | sdague: yes, but I can do that later if you think we should get that in sooner than later | 12:06 |
cdent | the number of 409s that are happening should be improved by some of the stuff above that patch | 12:06 |
cdent | the client side doesn't have any magical generation counting so it sometimes has to 409 to know that it needs to get a new generation to rectify itself | 12:07 |
cdent | sdague: I was waiting for the US to wake up before making any more adjustments on that stack so that they can happen in a batch. | 12:08 |
*** tbachman has joined #openstack-nova | 12:08 | |
sdague | cdent: ok, fair. | 12:09 |
*** rfolco has joined #openstack-nova | 12:09 | |
openstackgerrit | Kashyap Chamarthy proposed openstack/nova: libvirt: fix detach volume race condition https://review.openstack.org/290510 | 12:09 |
*** daemontool_ has joined #openstack-nova | 12:10 | |
kashyap | mdbooth: The above is just a rebase to Git first. Now reworking the feedback from reviewers | 12:10 |
kashyap | s/Git/Git master/ | 12:10 |
*** daemontool has quit IRC | 12:12 | |
*** fragatina has joined #openstack-nova | 12:13 | |
*** tbachman_ has joined #openstack-nova | 12:13 | |
*** edmondsw has joined #openstack-nova | 12:13 | |
sdague | cdent: even at top of stack the 409ing seems pretty high on allocations - http://logs.openstack.org/19/365719/1/check/gate-tempest-dsvm-neutron-placement-full-ubuntu-xenial-nv/a1c6cc4/logs/apache/placement-api.txt.gz | 12:14 |
sdague | which is definitely leading to a lot of 404s on deletes, so explains why the allocations might not line up with reality | 12:14 |
*** tbachman has quit IRC | 12:15 | |
*** tbachman_ is now known as tbachman | 12:15 | |
cdent | sdague: I don't think the allocations handling on the client side is very smart yet, so we'll need to do some analysis across the placement log and the n-cpu log to find out what's going on with a bit more detail. I'll suck down the n-cpu log and see if any patterns leap out, but we'll probably want Dan too. | 12:19 |
*** rlrossit has joined #openstack-nova | 12:19 | |
*** ttrumm_ has quit IRC | 12:19 | |
cdent | (I've got a meeting in a few minutes that's going to delay any progress for me) | 12:20 |
*** liverpooler has quit IRC | 12:21 | |
sdague | yep, no prob | 12:21 |
cdent | "Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider '4a25cc16-0b8a-4759-a29f-c17c6650dd39'. The requested amount would exceed the capacity" makes no sense so should be a clue | 12:21 |
*** oanson has quit IRC | 12:22 | |
*** fragatina has quit IRC | 12:22 | |
*** ttrumm has joined #openstack-nova | 12:24 | |
cdent | one thing sdague looks like we still need to change things so that placement api starts before n-cpu. I think we talked about that last week, but I forgot about it. | 12:24 |
sdague | ok, though shouldn't we be handling that in code? | 12:25 |
sdague | I thought we were handling the connect issue and it would just mean a missed request | 12:25 |
*** woodster_ has joined #openstack-nova | 12:26 | |
cdent | it would, and it's fine. I don't think it is causing problems, it's just a "wasted" error. | 12:26 |
*** PaulMurray has joined #openstack-nova | 12:26 | |
*** clenimar has quit IRC | 12:26 | |
*** lucas-hungry is now known as lucasagomes | 12:27 | |
cdent | sdague: there's a 147 difference instances being allocated. is that around what we expect? do we even know? | 12:30 |
*** thorst has joined #openstack-nova | 12:30 | |
*** jaypipes has joined #openstack-nova | 12:30 | |
*** clenimar has joined #openstack-nova | 12:31 | |
*** xinliang has quit IRC | 12:32 | |
sdague | that sounds ball park correct | 12:32 |
sdague | I was pretty sure it was > 100 | 12:32 |
*** mamandle has joined #openstack-nova | 12:34 | |
sdague | cdent: which run are you looking at? | 12:35 |
cdent | the one you pasted above | 12:35 |
*** mdrabe has joined #openstack-nova | 12:37 | |
cdent | that's using awks, greps, sorts uniq on the n-cpu log | 12:37 |
*** tongli has joined #openstack-nova | 12:40 | |
*** EinstCrazy has joined #openstack-nova | 12:43 | |
*** xinliang has joined #openstack-nova | 12:44 | |
*** baoli has joined #openstack-nova | 12:44 | |
*** ttrumm has quit IRC | 12:44 | |
*** baoli_ has joined #openstack-nova | 12:45 | |
*** figleaf is now known as edleafe | 12:45 | |
*** ttrumm has joined #openstack-nova | 12:46 | |
*** baoli has quit IRC | 12:48 | |
*** jaosorior has quit IRC | 12:48 | |
*** jaosorior has joined #openstack-nova | 12:49 | |
*** mmedvede has quit IRC | 12:49 | |
*** jaosorior has quit IRC | 12:49 | |
*** pcaruana has quit IRC | 12:50 | |
*** tbachman_ has joined #openstack-nova | 12:51 | |
*** ratailor has quit IRC | 12:51 | |
*** jaosorior has joined #openstack-nova | 12:51 | |
*** tbachman has quit IRC | 12:51 | |
*** tbachman_ is now known as tbachman | 12:51 | |
*** sudipto has joined #openstack-nova | 12:51 | |
*** sridharg has quit IRC | 12:56 | |
*** vishwanathj_zzz is now known as vishwanathj | 12:57 | |
cdent | sdague: when I get off this call I'm going to make sure that the capacity violation tests on the api side are sufficient so we can be sure to bisect the problem properly | 12:58 |
*** diga has quit IRC | 12:59 | |
*** edand has joined #openstack-nova | 13:00 | |
*** tbachman has quit IRC | 13:00 | |
*** zengine has joined #openstack-nova | 13:00 | |
*** vladikr has joined #openstack-nova | 13:00 | |
*** houming has quit IRC | 13:01 | |
*** bkopilov has joined #openstack-nova | 13:03 | |
*** mingyu has joined #openstack-nova | 13:06 | |
sdague | cdent: are we handling cpu overcommit fine in the placement API? | 13:07 |
*** tbachman has joined #openstack-nova | 13:08 | |
cdent | sdague: that's what allocation ratio means in the inventory, and what I'm about to check is that the tests for that are sufficient to be sure it is correct | 13:08 |
cdent | then we can work back from there to where the two concepts comingle | 13:08 |
*** sridharg has joined #openstack-nova | 13:09 | |
*** mingyu_ has joined #openstack-nova | 13:09 | |
openstackgerrit | Jianghua Wang proposed openstack/nova: XenAPI: resolve the fetch_bandwidth failure https://review.openstack.org/365838 | 13:10 |
*** mmedvede has joined #openstack-nova | 13:11 | |
*** mingyu has quit IRC | 13:12 | |
*** mnestratov has joined #openstack-nova | 13:15 | |
*** Guest36215 has quit IRC | 13:15 | |
openstackgerrit | Jianghua Wang proposed openstack/nova: XenAPI: resolve the fetch_bandwidth failure https://review.openstack.org/365838 | 13:15 |
*** akshai has joined #openstack-nova | 13:18 | |
openstackgerrit | Alex Szarka proposed openstack/nova-specs: Typo: very instead of vary https://review.openstack.org/366125 | 13:18 |
*** ttrumm_ has joined #openstack-nova | 13:22 | |
*** ljxiash has joined #openstack-nova | 13:23 | |
*** tblakes has joined #openstack-nova | 13:24 | |
*** eharney has joined #openstack-nova | 13:25 | |
*** ttrumm has quit IRC | 13:25 | |
*** tblakes_ has joined #openstack-nova | 13:25 | |
*** tbachman has quit IRC | 13:25 | |
*** hemanthm|away is now known as hemanthm | 13:27 | |
markus_z | bauzas: Can I interest you in a scheduler bug fix? https://review.openstack.org/#/c/352344/ | 13:29 |
*** tblakes has quit IRC | 13:29 | |
*** tblakes_ is now known as tblakes | 13:30 | |
*** irenab has quit IRC | 13:31 | |
*** pwnall1 has joined #openstack-nova | 13:32 | |
*** tbachman has joined #openstack-nova | 13:32 | |
*** irenab has joined #openstack-nova | 13:32 | |
bauzas | markus_z: cool :) | 13:33 |
*** mriedem has joined #openstack-nova | 13:34 | |
*** mingyu_ has quit IRC | 13:34 | |
*** zengine has quit IRC | 13:35 | |
*** markvoelker has joined #openstack-nova | 13:36 | |
openstackgerrit | Slawek Kaplonski proposed openstack/nova: libvirt: Clean up block device job status reporting code https://review.openstack.org/365756 | 13:36 |
*** andymaier has joined #openstack-nova | 13:37 | |
*** mingyu has joined #openstack-nova | 13:37 | |
*** pwnall1 has quit IRC | 13:38 | |
*** pwnall1 has joined #openstack-nova | 13:38 | |
*** sridharg has quit IRC | 13:39 | |
riccardo | hi guys, can anyone help? how can i get data from MariaDB? | 13:39 |
*** shangxdy has joined #openstack-nova | 13:40 | |
riccardo | is there some python example code? | 13:40 |
openstackgerrit | MJWurtz proposed openstack/nova: Refresh info_cache after deleting floating IP https://review.openstack.org/357494 | 13:42 |
*** ttrumm_ has quit IRC | 13:43 | |
*** yamahata has quit IRC | 13:44 | |
*** ttrumm has joined #openstack-nova | 13:44 | |
openstackgerrit | Prateek Arora proposed openstack/nova: Handle ConnectFailure from keystone when neutron is down https://review.openstack.org/327706 | 13:45 |
*** cleong has joined #openstack-nova | 13:45 | |
*** TravT has joined #openstack-nova | 13:45 | |
*** roeyc has quit IRC | 13:46 | |
*** dimtruck is now known as zz_dimtruck | 13:47 | |
*** roeyc has joined #openstack-nova | 13:48 | |
*** ttrumm has quit IRC | 13:49 | |
openstackgerrit | Prateek Arora proposed openstack/nova: Handle ConnectFailure from keystone when neutron is down https://review.openstack.org/327706 | 13:49 |
prateek | mriedem, can you please have a look at https://review.openstack.org/#/c/294513/ whenever you have some time ? | 13:50 |
openstackgerrit | Chris Dent proposed openstack/nova: [placement] Add additional allocation tests https://review.openstack.org/366144 | 13:50 |
cdent | sdague: that ^ is more tests, but they didn't reveal anything on the server side | 13:51 |
*** mlavalle has joined #openstack-nova | 13:51 | |
cdent | sdague I'm going to add some more logging to the resource tracker so we can see what it is writing more clearly. Or I can put it on the server side if that seems better? | 13:51 |
*** gongysh has joined #openstack-nova | 13:52 | |
sdague | if it's on the server side we could also add a debug log which dumps the current allocation state of the world on every request, to see what the system thinks is there | 13:52 |
mdbooth | ftersin: Hey, if I asked you to finish the cleanup in https://review.openstack.org/#/c/334609/, would it get done? | 13:53 |
mdbooth | ftersin: There are 2 identical misuses of that function in driver, but you only clean up one. | 13:53 |
mdbooth | Fixing 1 is obviously better than fixing none :) | 13:53 |
mdbooth | But fixing both is best. | 13:54 |
* cdent spins up devstack | 13:54 | |
ftersin | i remove the second in the next patch | 13:54 |
mdbooth | ftersin: Ah, ok. | 13:54 |
mdbooth | Hadn't seen that. | 13:55 |
*** rlrossit has quit IRC | 13:56 | |
PaulMurray | Live migration meeting in a few minutes on #openstack-meeting-3 PaulMurray paul-carlton2 eliqiao shaohe_feng pkoniszewski jlanoux yuntongjin mdbooth kashyap kmartin claudiub tobasco tdurakov andreas_s abhishekk mrhillsman raj_singh | 13:56 |
*** browne has joined #openstack-nova | 13:57 | |
*** burt has joined #openstack-nova | 13:57 | |
ftersin | mdbooth: i abandoned my cleaning patch and promoted https://review.openstack.org/#/c/337056 instead | 13:57 |
*** annegentle has joined #openstack-nova | 13:58 | |
riccardo | hi, i have 5 compute nodes and one controller in place. Is there something I can help you guys? | 13:58 |
mdbooth | riccardo: This is a channel for nova development. See /topic for user discussion. | 13:59 |
riccardo | sure, I would like to help with python coding | 14:00 |
*** luis5tb has joined #openstack-nova | 14:00 | |
openstackgerrit | Gleb Stepanov proposed openstack/nova: Skip malformed cookies https://review.openstack.org/346090 | 14:00 |
*** anush has joined #openstack-nova | 14:00 | |
sfinucan | riccardo: aye, but the focus should be on specific aspects of nova | 14:00 |
sfinucan | for things like how to access a database there are many better forums | 14:01 |
riccardo | ok, just let me know how i can help :-) The only constraint: I am a beginner. | 14:01 |
sfinucan | riccardo: https://wiki.openstack.org/wiki/How_To_Contribute | 14:01 |
sfinucan | riccardo: bug fixing would be a very good start | 14:01 |
*** krtaylor has joined #openstack-nova | 14:02 | |
riccardo | it hasn't to do something with my DB problem. i just wanna contribute anyhow | 14:02 |
*** ashyoung has joined #openstack-nova | 14:02 | |
riccardo | bug fix sounds good | 14:02 |
riccardo | i check the site... | 14:03 |
bauzas | alaski: around ? | 14:03 |
bauzas | alaski: back from holiday ? | 14:03 |
alaski | bauzas: yep | 14:03 |
bauzas | alaski: so, we have a bug report about updating the instance name https://bugs.launchpad.net/nova/+bug/1620248 | 14:04 |
openstack | Launchpad bug 1620248 in OpenStack Compute (nova) "Can't rename instance right after creation (regression)" [High,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | 14:04 |
bauzas | alaski: if you remember our discussion we had 2 weeks ago... | 14:05 |
*** thorst has quit IRC | 14:05 | |
alaski | yeah | 14:05 |
openstackgerrit | Sylvain Bauza proposed openstack/nova: WIP: Update BuildRequest if instance currently being scheduled https://review.openstack.org/365740 | 14:05 |
*** thorst has joined #openstack-nova | 14:05 | |
bauzas | alaski: so, I wonder why we should provide for that... | 14:06 |
bauzas | alaski: I wrote a quickly patch for modifying then the BuildRequest | 14:06 |
*** ametts has joined #openstack-nova | 14:06 | |
bauzas | alaski: https://review.openstack.org/365740 | 14:06 |
alaski | I'm amazed that anyone is doing that rather than just passing the name in the original request | 14:06 |
bauzas | alaski: agreed, but I think it could be understanding | 14:07 |
openstackgerrit | dane-fichter proposed openstack/nova: Use cursive for signature verification https://review.openstack.org/351232 | 14:08 |
*** marst has quit IRC | 14:08 | |
bauzas | alaski: so, either we don't accept people updating the instance until it's scheduled, or we need to update the BuildReq | 14:08 |
alaski | right. I favor the former, but if we can't change that without a microversion there's no point | 14:09 |
*** amotoki_ has quit IRC | 14:09 | |
bauzas | alaski: I'm pretty sure some people would like the latter | 14:09 |
*** hshiina has joined #openstack-nova | 14:10 | |
alaski | Updating the buildreq is mostly fairly simple. But we have to account for the race where instance creation could be happening as the buildreq is updated. | 14:10 |
bauzas | alaski: see my change then | 14:10 |
*** liang_ has joined #openstack-nova | 14:10 | |
alaski | looking, but I need a sec to digest this | 14:11 |
mriedem | cdent: we don't need to use the generic-resource-pools bp link in the new commits for bug fixes and unit test adds | 14:11 |
mriedem | cdent: since we're past FF it's weird to see new patches for a closed bp | 14:11 |
alaski | bauzas: I think you can simplify this a lot by checking if instance.id is set. If it is the instance exists in a db and instance.save() will work | 14:11 |
*** ducttape_ has joined #openstack-nova | 14:12 | |
cdent | mriedem: ah, okay, I was using that to make sure they get linked somewhere in case people start doing a "where's the stuff I'm suppose to look at". but will stop now. | 14:12 |
*** ekuris has quit IRC | 14:12 | |
mriedem | cdent: i'd probably just associate them to a bug | 14:13 |
mriedem | since there should be a bug for the overcommit stuff you and dansmith are trying to hunt down | 14:13 |
bauzas | alaski: sure, but that will won't work until we delay to persist the instance down in the conductor, right? | 14:13 |
bauzas | alaski: I mean, here, we already have a created instance | 14:14 |
*** thorst has quit IRC | 14:14 | |
alaski | bauzas: currently there are two instances, one from the instance table and buildreq.instance. if .id is set you know you have the one from the instance table | 14:14 |
*** ababich has quit IRC | 14:14 | |
*** ducttape_ has quit IRC | 14:15 | |
bauzas | alaski: right, but I was thinking the build_req.instance was having the id set | 14:15 |
alaski | you might get buildreq.instance even though there is an instance in the instances table, until instance creation moves to the conductor | 14:15 |
*** moshele has quit IRC | 14:16 | |
alaski | bauzas: nope. id is only set if the instance has been written to the db but we persist the buildreq one before doing that | 14:16 |
*** amotoki has joined #openstack-nova | 14:17 | |
bauzas | alaski: oh right | 14:17 |
bauzas | alaski: I just saw that | 14:17 |
*** michchap has quit IRC | 14:17 | |
bauzas | I was first thinking about looking at instance._context but lookup at instance.id is better | 14:17 |
alaski | yeah. .id is the definitive guide to whether or not it's from the instances table | 14:18 |
*** ducttape_ has joined #openstack-nova | 14:18 | |
*** ababich has joined #openstack-nova | 14:18 | |
openstackgerrit | Prateek Arora proposed openstack/nova: Handle ConnectFailure from keystone when neutron is down https://review.openstack.org/327706 | 14:18 |
openstackgerrit | Christopher Martin proposed openstack/nova: Fix for Nova API Dev Guide https://review.openstack.org/366162 | 14:20 |
*** mamandle has quit IRC | 14:20 | |
*** takedakn has joined #openstack-nova | 14:20 | |
*** gongysh has quit IRC | 14:20 | |
*** diogogmt has quit IRC | 14:20 | |
*** links has quit IRC | 14:21 | |
*** marst has joined #openstack-nova | 14:21 | |
*** syjulian_ has joined #openstack-nova | 14:21 | |
*** shangxdy has quit IRC | 14:23 | |
prateek | mriedem, thanks for the reviewing https://review.openstack.org/#/c/327706/,even though i had requested a review for https://review.openstack.org/#/c/294513/ , in which i had worked on some of your comments and it's been pending for a long time :) | 14:23 |
*** ametts has quit IRC | 14:24 | |
mriedem | prateek: i'm in 2 meetings atm, so i'll review what i can review when i can get the time :) | 14:24 |
*** BjoernT has joined #openstack-nova | 14:25 | |
*** zigo has quit IRC | 14:25 | |
prateek | mriedem, sure, thanks ! | 14:25 |
*** michchap has joined #openstack-nova | 14:26 | |
*** akshai has quit IRC | 14:27 | |
*** zz_dimtruck is now known as dimtruck | 14:29 | |
*** ametts has joined #openstack-nova | 14:29 | |
*** salv-orlando has joined #openstack-nova | 14:29 | |
*** prateek has quit IRC | 14:30 | |
*** felix23ma has quit IRC | 14:30 | |
*** brian_price has joined #openstack-nova | 14:32 | |
*** takedakn has quit IRC | 14:32 | |
*** zigo has joined #openstack-nova | 14:32 | |
*** salv-orl_ has quit IRC | 14:32 | |
*** zigo is now known as Guest39147 | 14:33 | |
*** syjulian_ has quit IRC | 14:33 | |
mriedem | mdbooth: if you can point me at where functional changes start for the imagebackend series that would save me some time | 14:33 |
mdbooth | mriedem: Sec | 14:33 |
*** syjulian_ has joined #openstack-nova | 14:33 | |
openstackgerrit | Samuel Matzek proposed openstack/nova: Reschedule boot when attach volume fails https://review.openstack.org/246505 | 14:34 |
mdbooth | Depends how you define functional :) This is the first non-test cleanup: https://review.openstack.org/#/c/333272/ | 14:34 |
mdbooth | However, its a NFC | 14:34 |
cdent | sdague, dansmith I'm struggling to replicate the allocation problem seen in temest, locally. And now I have to go to an appt. Will check in afterwards to see what you guys know and what I should do. Can one of you make the bug (if it doesn't already exist) suggested by mriedem a few minutes back. biab. | 14:35 |
*** jamesdenton has joined #openstack-nova | 14:35 | |
mdbooth | mriedem: The next one is functional: https://review.openstack.org/#/c/331115/ | 14:35 |
sdague | cdent: sure, how parallel is your local environment? | 14:35 |
tojuvone | sdague, dansmith, johnthetubaguy: If some of you could take a glance at this as familiar. I also have POC quite ready against the spec: https://review.openstack.org/310510 | 14:35 |
mriedem | mdbooth: national football conference? | 14:35 |
mdbooth | mriedem: Are you suggesting that I continue pushing on the test changes? I'd pretty much abandoned them for now, but I'll happily keep going for a bit. | 14:35 |
mdbooth | Non Functional Change :) | 14:36 |
mriedem | mdbooth: at this point with 2 weeks left i'd rather avoid big test refactors | 14:36 |
mriedem | mdbooth: especially if you're looking at fixing that post-copy network issue | 14:36 |
mdbooth | mriedem: That's what I thought. I'll leave it. | 14:36 |
cdent | sdague: not very, which is probably part of the problem, but I'm struggling to come up with a mental model that would explain things, even when parallel. I think my picture of "what's wrong" is incomplete, so mostly I'm just trying to narrow reality a bit. If the bug is able to explain the main issue, that'll probably help. | 14:36 |
sdague | mdbooth: yeh, typically post freeze we don't merge any non test / non docs code without a bug | 14:36 |
*** cdent has quit IRC | 14:37 | |
*** akshai has joined #openstack-nova | 14:37 | |
*** amotoki has quit IRC | 14:38 | |
*** jdurgin1 has joined #openstack-nova | 14:38 | |
*** Guest39147 has quit IRC | 14:39 | |
sdague | mriedem / dansmith - do we have some reasonable query / interface on allocations to get a debug log dump of what the state of the world looks like? | 14:39 |
*** zigo_ has joined #openstack-nova | 14:42 | |
mdbooth | sfinucan: https://bugs.launchpad.net/nova/+bug/1607996 | 14:43 |
openstack | Launchpad bug 1607996 in OpenStack Compute (nova) "Live migration does not update numa hugepages info in xml" [High,In progress] | 14:43 |
*** krtaylor has quit IRC | 14:43 | |
mdbooth | Are you definitely not looking at that any more? | 14:43 |
mriedem | sdague: idk | 14:44 |
sfinucan | mdbooth: Not at present, no. I didn't quite get the problem | 14:44 |
*** claudiub has quit IRC | 14:44 | |
sfinucan | was he referring to the host XML or guest? | 14:44 |
sfinucan | *she | 14:44 |
*** sdake has joined #openstack-nova | 14:44 | |
mriedem | sdague: we could pull from the DB in a DNM debug patch if needed, directly from the RT i mean | 14:44 |
mriedem | if necessary | 14:44 |
mdbooth | sfinucan: Don't know. The bug was/is under discussion in #openstack-meeting-3 | 14:44 |
*** raj_singh_ has joined #openstack-nova | 14:45 | |
mriedem | but i don't know what the GET looks like from the placement API | 14:45 |
mriedem | i thought there was talk on friday that there wasn't a GET | 14:45 |
sfinucan | mdbooth: under what context? | 14:45 |
mdbooth | Live migration meeting | 14:45 |
mdbooth | Ongoing | 14:45 |
sfinucan | ah | 14:45 |
sfinucan | afraid there isn't much I can contribute. I can look at it again, but I'd need to ask for more info first | 14:45 |
openstackgerrit | Gleb Stepanov proposed openstack/python-novaclient: Add evacuate host method https://review.openstack.org/349179 | 14:46 |
mdbooth | sfinucan: NP | 14:47 |
*** daemontool_ has quit IRC | 14:47 | |
*** zigo_ has quit IRC | 14:47 | |
*** rrecio_ has joined #openstack-nova | 14:47 | |
*** daemontool has joined #openstack-nova | 14:48 | |
*** zigo_ has joined #openstack-nova | 14:48 | |
mriedem | rpodolyaka: sdague: jaypipes: i'm going to -2 the separate placement API DB change for newton https://review.openstack.org/#/c/362766/ - we're past FF and since we aren't testing this i don't think it would be good to land it | 14:52 |
*** mamandle has joined #openstack-nova | 14:52 | |
*** annegentle has quit IRC | 14:53 | |
sdague | mriedem: +1 | 14:53 |
mriedem | mdbooth: probably a good one for you https://bugs.launchpad.net/nova/+bug/1620341 | 14:53 |
openstack | Launchpad bug 1620341 in OpenStack Compute (nova) "Removing unused base images removes backing files of active instances" [Undecided,New] | 14:53 |
*** prateek has joined #openstack-nova | 14:53 | |
mdbooth | Title sounds fun | 14:54 |
* mdbooth reads | 14:54 | |
*** raj_singh_ has quit IRC | 14:54 | |
mdbooth | mriedem: Paraphrase: image cache manager is broken | 14:54 |
mdbooth | That's about the level of detail there | 14:55 |
mriedem | +NFS | 14:55 |
mriedem | but yeah | 14:55 |
* mdbooth has a quick look at ICM's shared storage handling | 14:55 | |
mdbooth | I'm pretty sure it stores a file containing every host which accesses the shared store | 14:56 |
mdbooth | And filters based on all of them | 14:56 |
mdbooth | But IIRC there's nothing to stop cleanup running concurrently on different hosts. | 14:56 |
*** tbachman has quit IRC | 14:57 | |
kashyap | mdbooth: Randomly chiming in, there are multiple (2 IIRC) config variables related to removing unused base image files. You plan to remove / rework some of them as part of your larger work on Image Cache? | 14:59 |
*** _ducttape_ has joined #openstack-nova | 14:59 | |
mdbooth | kashyap: I have no larger work on Image Cache! | 14:59 |
* kashyap stops making mdbooth responsible for imaginary stuff | 14:59 | |
mdbooth | I occasionally fix bits of it when I'm compelled to read enough of it to understand some of it. | 14:59 |
*** pradk has quit IRC | 15:00 | |
kashyap | Noted :-) | 15:00 |
*** mjura has quit IRC | 15:01 | |
*** ashyoung has quit IRC | 15:01 | |
openstackgerrit | Sean Dague proposed openstack/nova: WIP: additional logging to debug allocations https://review.openstack.org/366186 | 15:02 |
sdague | mriedem: it will be interesting to see what the run logs for that look like | 15:02 |
*** rodrigods has quit IRC | 15:02 | |
*** rodrigods has joined #openstack-nova | 15:02 | |
*** ducttape_ has quit IRC | 15:02 | |
markus_z | PaulMurray: apologies for not responding in the live migrate meeting, I was in another meeting too and didn't see it. | 15:03 |
PaulMurray | markus_z, np | 15:04 |
PaulMurray | markus_z, does anyone look after the lm bugs ? | 15:04 |
* PaulMurray expects I should be doing it | 15:04 | |
*** edand has quit IRC | 15:05 | |
markus_z | PaulMurray: I'm not aware of anyone, tbh. | 15:05 |
markus_z | PaulMurray: I keep them alive and wait for my turn for reviews | 15:05 |
PaulMurray | I will go through them in the next couple of days | 15:05 |
*** rmart04_ has joined #openstack-nova | 15:06 | |
markus_z | PaulMurray: One was a regression. The other one an older issue. | 15:06 |
*** syjulian_ has quit IRC | 15:06 | |
markus_z | PaulMurray: thanks | 15:06 |
openstackgerrit | Matt Riedemann proposed openstack/nova: libvirt: improve logging for shared storage check https://review.openstack.org/365140 | 15:06 |
openstackgerrit | Matt Riedemann proposed openstack/nova: nfs fix for xenial images https://review.openstack.org/329466 | 15:06 |
rpodolyaka | mriedem: oh, I missed the fact the original change was reverted while I was on vacation :P | 15:06 |
rpodolyaka | it's a bit disappointing of course, but yeah, I agree it may be too late now | 15:06 |
*** tbachman has joined #openstack-nova | 15:08 | |
*** annegentle has joined #openstack-nova | 15:08 | |
openstackgerrit | Sylvain Bauza proposed openstack/nova: WIP: Update BuildRequest if instance currently being scheduled https://review.openstack.org/365740 | 15:08 |
bauzas | alaski: just updated based on your comments | 15:08 |
bauzas | alaski: I think we still need to have lots of conditionals for making sure we don't forget a possible race condition | 15:09 |
*** andreas_s has quit IRC | 15:09 | |
*** gouthamr has joined #openstack-nova | 15:10 | |
*** amotoki has joined #openstack-nova | 15:11 | |
markus_z | dansmith: FYI: fixes nova-manage privilege escalation: https://review.openstack.org/#/c/352866/ | 15:11 |
*** thorst has joined #openstack-nova | 15:11 | |
markus_z | maybe you could have a look at it | 15:11 |
*** diogogmt has joined #openstack-nova | 15:11 | |
alaski | bauzas: I don't think you can avoid the race condition without a lock. And I think a first patch should fix the simple issue of updating buildreq.instance and then deal with races in a second patch. | 15:12 |
*** gouthamr_ has joined #openstack-nova | 15:12 | |
*** sayalilunkad has quit IRC | 15:13 | |
*** sayalilunkad has joined #openstack-nova | 15:13 | |
*** hemnafk has quit IRC | 15:13 | |
bauzas | alaski: well... | 15:14 |
*** yamahata has joined #openstack-nova | 15:14 | |
*** PaulMurray has quit IRC | 15:15 | |
*** gouthamr has quit IRC | 15:15 | |
bauzas | alaski: you mean, just amending the BuildReq without really checking whether we should rather use the cellsv2 contextmanager? | 15:16 |
bauzas | alaski: I thought it was better to add it too, but okaty | 15:16 |
alaski | bauzas: I mean first solve the issue of not being able to update the buildreq at all. check for instance.id: if it exists instance.update; instance.save, it it doesn't exist pull the buildreq and update it. | 15:17 |
alaski | then worry about the small race | 15:17 |
*** nkrinner is now known as nkrinner_afk | 15:18 | |
*** hemna has joined #openstack-nova | 15:18 | |
*** thorst has quit IRC | 15:19 | |
*** pradk has joined #openstack-nova | 15:20 | |
*** zigo_ is now known as zigo | 15:20 | |
*** fragatina has joined #openstack-nova | 15:20 | |
*** mamandle has quit IRC | 15:24 | |
*** cdent has joined #openstack-nova | 15:26 | |
*** EinstCrazy has quit IRC | 15:26 | |
*** fragatina has quit IRC | 15:27 | |
*** gokrokve has joined #openstack-nova | 15:27 | |
*** mriedem1 has joined #openstack-nova | 15:27 | |
*** armax has joined #openstack-nova | 15:29 | |
*** mriedem has quit IRC | 15:30 | |
*** thorst has joined #openstack-nova | 15:30 | |
*** whenry has joined #openstack-nova | 15:31 | |
*** tbachman has quit IRC | 15:31 | |
*** sudipto has quit IRC | 15:31 | |
*** rmart04_ has quit IRC | 15:33 | |
*** mingyu has quit IRC | 15:34 | |
sdague | dansmith: I think I figured out the vcpu issue | 15:34 |
dansmith | yeah? | 15:34 |
sdague | math is hard | 15:35 |
*** Jeffrey4l has quit IRC | 15:35 | |
dansmith | in my local db we are reporting the proper allocation ratio, so that kinda shoots my theory | 15:35 |
dansmith | unless we're not factoring it into the actual capacity calculation | 15:36 |
openstackgerrit | Sean Dague proposed openstack/nova: WIP: calculate capacity correctly https://review.openstack.org/366214 | 15:36 |
sdague | dansmith: I think that's the fix | 15:36 |
*** rmart04 has joined #openstack-nova | 15:36 | |
dansmith | hmm, why? | 15:36 |
dansmith | that would mean we're effectively scaling the reserve right? | 15:37 |
*** PaulMurray has joined #openstack-nova | 15:37 | |
dansmith | i.e. we reserve only that amount from the scaled amount | 15:37 |
*** paul-carlton has quit IRC | 15:37 | |
*** tonytan4ever has joined #openstack-nova | 15:38 | |
sdague | ok, well what do we expect total, reserved, allocation_ratio to be there? | 15:38 |
openstackgerrit | Prateek Arora proposed openstack/nova: Handle ConnectFailure from keystone when neutron is down https://review.openstack.org/327706 | 15:38 |
*** tonytan_brb has joined #openstack-nova | 15:39 | |
dansmith | reserved is often zero or small | 15:39 |
danpb | so consider total capacity is 8 GB and we want 1 GB reserved and alloc ratio is 2 - with old logic we'd report 14 GB, but with fixed logic we're report 15 GB | 15:39 |
danpb | the new logic sdague suggests does make more sense to me | 15:39 |
sdague | ok, well it's definitely capacity where we go funky | 15:39 |
*** andymaier has quit IRC | 15:39 | |
sdague | let me get some more debug in there to figure that out | 15:39 |
dansmith | but we should be 4*16 in the jenkins workers | 15:39 |
dansmith | and I doubt we're starting that many instances in parallel right? | 15:40 |
bauzas | are you guys talking about how we reserve space in the RT ? | 15:40 |
*** tblakes has quit IRC | 15:40 | |
bauzas | that's just a silly math | 15:40 |
danpb | of course our reserved memory calcs all have a level of crapiness about them | 15:40 |
sdague | dansmith: we should be 4 * 16 | 15:40 |
sdague | however, any time we go over 8 we seem to fail | 15:40 |
sdague | on the capacity check | 15:40 |
dansmith | sdague: reserved for vcpus is zero right? | 15:40 |
dansmith | it is on my machine | 15:41 |
dansmith | so I don't see why your math makes any difference | 15:41 |
*** pcaruana has joined #openstack-nova | 15:42 | |
*** rmart04 has quit IRC | 15:42 | |
*** tonytan4ever has quit IRC | 15:42 | |
openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova: Remove context object in oslo.log method https://review.openstack.org/317109 | 15:43 |
sdague | dansmith: ok, let me figure out what's really going on here | 15:43 |
sdague | I'm adding some more debug there | 15:43 |
*** sdake has quit IRC | 15:44 | |
dansmith | okay | 15:44 |
bauzas | dansmith: sdague: just in case you haven't seen that yet, we just reserve a bit of space by this https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L886 | 15:45 |
bauzas | dansmith: sdague: then we look at the free ram https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L888 | 15:45 |
prateek | cdent, sahid , can you please take a look when you have some free time ? https://review.openstack.org/#/c/327706/ | 15:46 |
bauzas | dansmith: sdague: and then we sum up the ram size for each instance https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L699 | 15:46 |
sdague | bauzas: right, so this actually won't make any difference on the cpu side | 15:47 |
sdague | as dansmith correctly states | 15:47 |
sdague | because the reserved is 0 | 15:47 |
prateek | jaypipes, can you please take a look when you have some time ? https://review.openstack.org/#/c/294513/ | 15:48 |
mdbooth | Which release was 13.0.0? | 15:48 |
mdbooth | Was that liberty? | 15:48 |
mdbooth | Ah, no. That was mitaka, I think? | 15:49 |
_gryf | 13 is mitaka, liberty was 12.x | 15:49 |
mdbooth | Thanks | 15:49 |
_gryf | mdbooth, https://releases.openstack.org/mitaka/ | 15:50 |
*** tonytan_brb has quit IRC | 15:51 | |
bauzas | sdague: correct | 15:51 |
*** paul-carlton2 has quit IRC | 15:52 | |
sahid | prateek: done | 15:52 |
*** tbachman has joined #openstack-nova | 15:52 | |
*** mriedem1 is now known as mriedem | 15:53 | |
*** tonytan4ever has joined #openstack-nova | 15:53 | |
cdent | sdague, dansmith I think we've yet to identify or articulate the actual problem in a complete fashion, and until we can generate a minimal test case that aggrevates the problem we're going to be throwing spitballs that don't stick | 15:54 |
*** matrohon has quit IRC | 15:55 | |
*** unicell1 has quit IRC | 15:55 | |
sdague | cdent: well, some more logging would help, it's exploding in _check_capacity_exceeded | 15:57 |
sdague | and it really shouldn't be, given what is available at the time | 15:57 |
cdent | so the questions become: can we construct a minimal test case? Can we do it in a way that it can be run indepdendently of tempest? | 15:57 |
*** _ducttape_ has quit IRC | 15:58 | |
*** ducttape_ has joined #openstack-nova | 15:59 | |
mriedem | you could potentially create a functional test | 15:59 |
mriedem | but it's a fake virt driver | 15:59 |
mriedem | so fake RT inventory | 15:59 |
*** claudiub has joined #openstack-nova | 16:00 | |
sdague | right, but test for what? We have a real world behavior happening here that we don't yet understand even the bad math that is happening | 16:00 |
cdent | sdague: basically so we can inspect more quickly. the iteration cycle here is painful. If the answer is "no we can't" that's okay, but I've still yet to read a concise description of the problem so I'm not able to believe "no we can't", yet. | 16:01 |
*** nic has joined #openstack-nova | 16:01 | |
*** mgoddard_ has joined #openstack-nova | 16:02 | |
sdague | 2016-09-06 12:01:16.928 WARNING nova.objects.resource_provider [req-9c93fbd0-ab72-47ad-9479-1132cb85f470 placement service] Attempting to allocate 1 for VCPU. Currently using 1024, amount available 64.0 | 16:02 |
sdague | that is the crux of the problem | 16:02 |
*** unicell has joined #openstack-nova | 16:02 | |
dansmith | sdague: capacity says 1024 are in use and 64 are available for cpus? | 16:03 |
cdent | If we're driving on guesses at the moment I'd guess that the join or the group by in the sql in the exceed function is wonky | 16:03 |
sdague | dansmith: yes | 16:03 |
*** luis5tb has quit IRC | 16:03 | |
dansmith | 1024/16 = 64 which makes no sense for that worker | 16:03 |
sdague | correct | 16:03 |
sdague | 1024 is a crazy value | 16:03 |
dansmith | that's weird. | 16:03 |
sdague | this is my nuc | 16:03 |
dansmith | oh okay, but still | 16:04 |
sdague | yeh, it's 4 core | 16:04 |
sdague | and, I know that's not actual usage | 16:04 |
* cdent inspects the sql | 16:04 | |
sdague | because I've got other debug logs which are dumping the usage | 16:04 |
sdague | 2016-09-06 12:01:16.904 DEBUG nova.api.openstack.placement.handlers.allocation [req-9c93fbd0-ab72-47ad-9479-1132cb85f470 placement service] Current Usages for 62b97cd4-5dc8-45d9-89ad-988273895635: Usage(resource_class='VCPU',usage=16), Usage(resource_class='MEMORY_MB',usage=1024), Usage(resource_class='DISK_GB',usage=0), | 16:04 |
*** TravT_ has joined #openstack-nova | 16:04 | |
*** gokrokve has quit IRC | 16:05 | |
sdague | which means for this resource check, it is using the memory value for cpu | 16:05 |
dansmith | so you booted a guest with 16 vcpus yes? | 16:05 |
openstackgerrit | Christopher Martin proposed openstack/nova: Fix Boot From Volume size error https://review.openstack.org/364337 | 16:05 |
*** mgoddard has quit IRC | 16:05 | |
*** david-lyle_ has joined #openstack-nova | 16:05 | |
*** david-lyle_ has quit IRC | 16:05 | |
*** Swami has joined #openstack-nova | 16:06 | |
sdague | dansmith: nope | 16:06 |
*** hshiina has quit IRC | 16:06 | |
sdague | that's complete usages | 16:06 |
dansmith | sdague: an allocation is one instance | 16:06 |
sdague | dansmith: it's not an allocation | 16:06 |
dansmith | oh you're logging usage from the allocation handler I see | 16:06 |
sdague | https://review.openstack.org/#/c/366186/1/nova/api/openstack/placement/handlers/allocation.py | 16:07 |
sdague | yeh | 16:07 |
dansmith | gotcha | 16:07 |
dansmith | yeah, kinda seems like it's confusing the columns in the sql query or something | 16:07 |
*** TravT has quit IRC | 16:08 | |
*** TravT has joined #openstack-nova | 16:10 | |
openstackgerrit | Matthew Booth proposed openstack/nova: imagecache: Hold shared storage registry lock while running https://review.openstack.org/366239 | 16:10 |
mdbooth | mriedem: ^^^ But I think the real fix is https://review.openstack.org/#/c/185549/ in mitaka | 16:11 |
*** paul-carlton2 has joined #openstack-nova | 16:11 | |
sdague | I'm going to stick another bit of debug on the records to see if there anything useful there | 16:12 |
*** TravT_ has quit IRC | 16:12 | |
mriedem | mdbooth: what do you mean by the real fix is in mitaka? | 16:13 |
mdbooth | mriedem: The reporter mentions they hit the problem in several release, the last of which was liberty | 16:13 |
mdbooth | The above fix was first in mitaka, and the backport reject for liberty | 16:13 |
mdbooth | By you :) | 16:13 |
sdague | ok... so this is weird | 16:13 |
mdbooth | And I suspect it would resolve their problem | 16:14 |
mriedem | mdbooth: yeah b/c it required a grenade change for rootwrap filters | 16:14 |
mdbooth | mriedem: Right, but I thought it was amusing anyway :) | 16:14 |
*** derekh is now known as derekh_afk | 16:14 | |
* cdent is increasingly feeling like this sql is not right | 16:14 | |
sdague | http://paste.openstack.org/show/567204/ | 16:15 |
*** lpetrut has quit IRC | 16:15 | |
sdague | we seem to have a joined in a way that we get usage for every resource class type crossed against every resource type | 16:15 |
*** riccardo has quit IRC | 16:15 | |
*** lpetrut has joined #openstack-nova | 16:15 | |
*** rlrossit has joined #openstack-nova | 16:16 | |
mdbooth | mriedem: Anyway, that code still needs a lock. It came up here: https://review.openstack.org/#/c/334956/ | 16:16 |
cdent | feh | 16:16 |
sdague | field 4 is the resource class | 16:16 |
mdbooth | The code gives the impression it might run concurrently, which raised flags with jaypipes | 16:16 |
sdague | let me push so I can gitlink things | 16:16 |
mdbooth | When in fact, that would certainly break. | 16:16 |
openstackgerrit | Sean Dague proposed openstack/nova: WIP: more and more and more debug https://review.openstack.org/366241 | 16:16 |
cdent | sdague: i've repeated what you're seeing. nice work. I think that points the finger of doom pretty strongly at the sql | 16:19 |
sdague | https://github.com/openstack/nova/blob/f9103f6a93b2dcc15e613ee54235fdf54d60027f/nova/objects/resource_provider.py#L705 - so I'm assuming that we should only see 3 records there | 16:20 |
*** lpetrut has quit IRC | 16:20 | |
*** gouthamr_ has quit IRC | 16:20 | |
*** thorst has quit IRC | 16:21 | |
*** daemontool has quit IRC | 16:21 | |
*** thorst has joined #openstack-nova | 16:21 | |
cdent | that's what I think, yeah | 16:21 |
sdague | ok, so minimally we should probably update the code to explode if we find duplicate keys for our map I would think | 16:22 |
*** yamahata has quit IRC | 16:23 | |
*** raunak has joined #openstack-nova | 16:24 | |
*** rlrossit has quit IRC | 16:25 | |
*** daemontool has joined #openstack-nova | 16:25 | |
*** raunak has quit IRC | 16:25 | |
*** akshai has quit IRC | 16:26 | |
*** akshai has joined #openstack-nova | 16:29 | |
*** tonytan_brb has joined #openstack-nova | 16:29 | |
*** daemontool has quit IRC | 16:29 | |
*** Apoorva has joined #openstack-nova | 16:29 | |
*** Sukhdev has joined #openstack-nova | 16:30 | |
sdague | yeh, it's the left outer join that's the issue | 16:30 |
*** thorst has quit IRC | 16:30 | |
sdague | that's why we get usage across all the records, and all the records duplicated | 16:30 |
sdague | we really want to do a real join with inventory.resource_class_id == allocations.resource_class_id | 16:31 |
*** stevemar has quit IRC | 16:31 | |
*** tonytan4ever has quit IRC | 16:32 | |
*** gouthamr has joined #openstack-nova | 16:33 | |
sdague | ok, I might have that | 16:33 |
sdague | I at least get sane allocation records now | 16:33 |
avolkov | sdague: hi, from https://review.openstack.org/#/c/365015/, do we have some stat about http responses? | 16:33 |
openstackgerrit | Sean Dague proposed openstack/nova: WIP: correctly join the usage to inventory https://review.openstack.org/366245 | 16:34 |
sdague | cdent / dansmith ^^^ how does that look to you? | 16:34 |
cdent | sdague: one sec | 16:34 |
dansmith | I wouldn't know | 16:34 |
dansmith | I'll believe it if the logs look righter :) | 16:34 |
sdague | well, I get sane records now | 16:35 |
sdague | only 3 of them | 16:35 |
*** mingyu has joined #openstack-nova | 16:35 | |
sdague | instead of 9 | 16:35 |
sdague | and the usage lines up with the resource class I expect | 16:35 |
*** derekh_afk is now known as derekh | 16:35 | |
*** raunak has joined #openstack-nova | 16:35 | |
cdent | sdague: that's the exact same thing that I came up with (while I was one seccing) so I guess that's two votes | 16:35 |
cdent | just confirming with some runs | 16:36 |
*** zenoway has quit IRC | 16:36 | |
*** READ10 has joined #openstack-nova | 16:36 | |
*** zenoway has joined #openstack-nova | 16:36 | |
dansmith | sdague: should be able to prove that with a functional test that breaks on the current code right? | 16:36 |
sdague | dansmith: maybe | 16:36 |
sdague | so the thing is, I expect this comes and goes because the join columns have no guarunteed order | 16:37 |
*** tonytan_brb is now known as tonytan4ever | 16:37 | |
sdague | sorry join rows | 16:37 |
openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova: Using get() method to prevent KeyError https://review.openstack.org/359424 | 16:38 |
cdent | sdague: I'm wondering what's missing from the existing allocation violation tests. probably not enough resoruces clases | 16:38 |
sdague | cdent: enough range of values? | 16:38 |
cdent | well: not enough classes of resources in the tests such that the cross product happens | 16:38 |
cdent | although I thought I had one with at least two resource classes in the mix | 16:39 |
sdague | right, but the issue would only arise if you exceeded one of them | 16:39 |
sdague | and you managed to get things returned in a way that you were comparing it to the other | 16:39 |
*** mingyu has quit IRC | 16:40 | |
cdent | I guess it wasn't quite good enough: https://review.openstack.org/#/c/366144/ | 16:40 |
cdent | anyway, my local tests confirm your join changes | 16:40 |
cdent | 3 records with the right data | 16:40 |
*** zenoway has quit IRC | 16:41 | |
mdbooth | sdague: That looks correct to me also.. | 16:41 |
*** sahid has quit IRC | 16:41 | |
mdbooth | That primary join is essentially joining allocation to inventory | 16:41 |
*** READ10 has quit IRC | 16:41 | |
mdbooth | Using only resource_provider in the where clause | 16:41 |
mdbooth | But there are multiple inventories per resource provider | 16:41 |
sdague | so, we probably just need a safety check that record length is not greater than len(res_classes) * len(provider_uuids) | 16:42 |
*** xyang1 has joined #openstack-nova | 16:42 | |
sdague | that would at least have blown up early on the sql | 16:42 |
*** fragatina has joined #openstack-nova | 16:42 | |
mdbooth | resource_provider/resource_class uniquely describes an inventory | 16:42 |
cdent | if we have functional tests that confirm the sql is doing what it says it should, then having an assertion in the code path seems redundant to me: we need to be able to rely on the sql. If we can't, we need to fix it. | 16:43 |
sdague | mdbooth: ok, sure, so we do an 'in' check before adding the usage map | 16:43 |
cdent | the problem in this case is we didnt' have good tests for the sql | 16:43 |
cdent | s/good/good enough/ | 16:43 |
mdbooth | sdague: Agreeing with you completely, just showing my working :) | 16:43 |
sdague | a safety measure is still good | 16:43 |
*** fragatina has quit IRC | 16:43 | |
*** lucasagomes is now known as lucas-dinner | 16:44 | |
*** fragatina has joined #openstack-nova | 16:44 | |
sdague | ok, I need to get lunch, then I'll go clean this up and rebase to master | 16:44 |
* cdent tries to think up a reasonable functional test | 16:45 | |
*** markus_z has quit IRC | 16:46 | |
*** roeyc has quit IRC | 16:47 | |
*** pradk has quit IRC | 16:47 | |
mdbooth | cdent: A resource provider with 2 inventories, with allocations against both. | 16:48 |
*** jamesdenton has quit IRC | 16:48 | |
cdent | mdbooth: yeah, I thought I had that, but it appears not | 16:48 |
cdent | or rather I have it, but I don't have a failure against it | 16:49 |
* mdbooth is just looking at what the duplicate entries in 'records' result in | 16:49 | |
*** rlrossit has joined #openstack-nova | 16:49 | |
*** gszasz has quit IRC | 16:49 | |
* cdent nods | 16:50 | |
*** lpetrut has joined #openstack-nova | 16:52 | |
*** fragatina has quit IRC | 16:52 | |
mdbooth | This is pretty dense :) | 16:52 |
*** Sukhdev has quit IRC | 16:53 | |
*** rama_y has joined #openstack-nova | 16:54 | |
mdbooth | cdent: So, primary_join is a cartesian join of usage (correct) against all inventories in a single rp | 16:55 |
*** ihrachys has quit IRC | 16:55 | |
mdbooth | records, therefore contains a mix of joins, where all but one of the inventories are incorrect | 16:55 |
openstackgerrit | dane-fichter proposed openstack/nova: Use cursive for signature verification https://review.openstack.org/351232 | 16:56 |
mdbooth | So the 'last' record may or may not be the correct one | 16:56 |
sdague | mdbooth: yeh, it's super dense. Which is why I'm in favor of inline safety measures because we've proven that no one can eyeball review it correctly :) | 16:56 |
*** sdake has joined #openstack-nova | 16:57 | |
mdbooth | sdague: Sounds like a good idea :) | 16:57 |
* cdent has a failing test | 16:58 | |
mdbooth | cdent: provs_with_inv contains all relevant rps | 16:58 |
cdent | (in the good way) | 16:58 |
mdbooth | usage_map contains only the last record | 16:58 |
sdague | heh, you know, jay's comment of "this is this sql" is the right sql he wanted | 16:58 |
mdbooth | cdent: Now to ensure it's deterministicly failing :) | 16:58 |
sdague | just missed a join | 16:58 |
*** hogepodge has quit IRC | 16:58 | |
*** lpetrut has quit IRC | 16:59 | |
cdent | sdague: heh, yeah | 16:59 |
mdbooth | cdent: I assume your test involves executing the sql, right? | 16:59 |
mdbooth | I wonder what assurances sqlite makes about returned row order in this case | 17:00 |
*** syjulian_ has joined #openstack-nova | 17:00 | |
mdbooth | Probably not many :) | 17:00 |
*** anush has quit IRC | 17:00 | |
mdbooth | It's a pretty funky query | 17:00 |
cdent | mdbooth: yeah, this is a functional test that is calling the outer layer object methods, not a unit test on the sql methods | 17:00 |
mdbooth | cdent: If you're wondering what I'm going on about, btw, it's this: | 17:02 |
mdbooth | for record in records: | 17:02 |
mdbooth | usage_map[(record['uuid'], record['resource_class_id'])] = record | 17:02 |
mdbooth | That will result in usage_map being correct or incorrect depending on whether the last returned row was the correct one or now | 17:03 |
mdbooth | not | 17:03 |
*** unicell has quit IRC | 17:03 | |
*** mgoddard_ has quit IRC | 17:03 | |
*** hogepodge has joined #openstack-nova | 17:03 | |
*** mgoddard has joined #openstack-nova | 17:03 | |
cdent | mdbooth: can you expand on what you're saying there a bit, because, assuming we fix the join as described, why is that still true? | 17:05 |
mdbooth | cdent: If we fix the join it's not true | 17:05 |
sdague | do we have a tracking bug yet? | 17:05 |
cdent | ah, okay, now I get where you're coming from | 17:05 |
*** bradjones has quit IRC | 17:06 | |
mdbooth | However, it would make your failing test determined by the returned row order | 17:06 |
mdbooth | i.e. Your failing test might get false positives with an alternative row order | 17:06 |
cdent | mdbooth: yeah, but it turns out I made my failing test incorrectly otherwise anyway, so I'm still at the drawing board | 17:06 |
*** yamahata has joined #openstack-nova | 17:06 | |
openstackgerrit | Tracy Jones proposed openstack/nova: Fix spelling mistake https://review.openstack.org/366254 | 17:07 |
cdent | sdague: not that I'm aware of, when I left for my appointment I thought you said you were gonna | 17:07 |
sdague | oh, I was just digging on the problem, I didn't build the bug yet | 17:07 |
sdague | I'll do that now | 17:07 |
*** code-R_ has quit IRC | 17:07 | |
*** su_zhang has joined #openstack-nova | 17:09 | |
mdbooth | cdent: Sorry to intrude, I was just being nosy. Enjoy :) | 17:10 |
*** lpetrut has joined #openstack-nova | 17:10 | |
* mdbooth has to head home | 17:10 | |
cdent | mdbooth: not an intrusion at all, thanks for the insight | 17:10 |
*** armax has quit IRC | 17:11 | |
*** bradjones has joined #openstack-nova | 17:11 | |
*** bradjones has quit IRC | 17:11 | |
*** bradjones has joined #openstack-nova | 17:11 | |
*** sdake_ has joined #openstack-nova | 17:12 | |
*** paul-carlton2 has quit IRC | 17:13 | |
*** syjulian_ has quit IRC | 17:14 | |
*** sdake has quit IRC | 17:15 | |
*** pradk has joined #openstack-nova | 17:16 | |
openstackgerrit | Sean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting https://review.openstack.org/366245 | 17:20 |
sdague | cdent: so, I think if you *actually* want to unit test that, you are going to need to split the _check_capacity_exceeded function in 2 | 17:21 |
sdague | with the first half being just the SQL | 17:21 |
sdague | and basically ending at where records are returned | 17:21 |
cdent | sdague: I think a functional test is more useful in this case, or at least, for me, I'm able to exercise the problem more effectively and realistically (and create a better understanding of what's going) with a functional test | 17:22 |
sdague | cdent: sure, that's fine too | 17:22 |
sdague | I do think the key in the bug is complicated sql is complicated, and we got back an unexpected row structure | 17:22 |
sdague | and long term, isolating that so we know we always get back expected row structure would be nice | 17:23 |
sdague | anyway, https://review.openstack.org/366245 I think is in the state we want to merge, and it's rebased to master so doesn't depend on anything else | 17:23 |
*** stevemar has joined #openstack-nova | 17:26 | |
*** paul-carlton2 has joined #openstack-nova | 17:26 | |
*** armax has joined #openstack-nova | 17:26 | |
*** mriedem1 has joined #openstack-nova | 17:26 | |
*** thorst has joined #openstack-nova | 17:26 | |
openstackgerrit | Tracy Jones proposed openstack/nova: Fix spelling mistake in libvirt https://review.openstack.org/366262 | 17:28 |
*** PaulMurray has quit IRC | 17:28 | |
*** tjones has joined #openstack-nova | 17:28 | |
*** mriedem has quit IRC | 17:28 | |
*** sdake_ has quit IRC | 17:28 | |
*** rama_y has quit IRC | 17:29 | |
*** sdake has joined #openstack-nova | 17:29 | |
*** ababich has quit IRC | 17:29 | |
*** su_zhang has quit IRC | 17:30 | |
*** jdurgin1 has quit IRC | 17:30 | |
*** su_zhang has joined #openstack-nova | 17:30 | |
*** akshai has quit IRC | 17:30 | |
*** TravT has quit IRC | 17:31 | |
*** fragatina has joined #openstack-nova | 17:32 | |
*** fragatina has quit IRC | 17:32 | |
*** tonytan4ever has quit IRC | 17:33 | |
*** TravT has joined #openstack-nova | 17:33 | |
*** fragatina has joined #openstack-nova | 17:33 | |
*** edand has joined #openstack-nova | 17:33 | |
*** jheroux has joined #openstack-nova | 17:34 | |
*** thorst has quit IRC | 17:35 | |
sdague | gah, pep8 | 17:36 |
*** jaosorior has quit IRC | 17:36 | |
openstackgerrit | Sean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting https://review.openstack.org/366245 | 17:38 |
*** harlowja has joined #openstack-nova | 17:38 | |
cdent | found another (minor) bug so just reporting it for now but will fix it tomorrow. i'm fried at this point: https://bugs.launchpad.net/nova/+bug/1620748 | 17:38 |
openstack | Launchpad bug 1620748 in OpenStack Compute (nova) "In placement when an attempt is made to write to missing inventory the error message is ugly" [Undecided,New] - Assigned to Chris Dent (cdent) | 17:38 |
*** amotoki has quit IRC | 17:39 | |
sdague | yeh, the int ids definitely make dealing with the raw data a bit odd | 17:40 |
*** tbachman has quit IRC | 17:40 | |
sdague | it would be nice if the repr for the objects resolved that | 17:40 |
cdent | you still got the set problem | 17:40 |
cdent | I'm going to have to give up for the evening; i've reached stupid. will check in later or catch up tomorrow | 17:42 |
*** cdent has quit IRC | 17:42 | |
sdague | ok, no worries | 17:43 |
*** tonytan4ever has joined #openstack-nova | 17:44 | |
*** tbachman has joined #openstack-nova | 17:44 | |
*** lamt has joined #openstack-nova | 17:45 | |
sdague | yay, my run tempest a lot and leak resources meant I even got to see the warning I put in place | 17:48 |
sdague | 2016-09-06 13:47:13.766 WARNING nova.objects.resource_provider [req-a3758596-5182-4e9e-b7a2-04462bf27f48 placement service] Over capacity for VCPU on resource provider 62b97cd4-5dc8-45d9-89ad-988273895635. Needed: 1, Used: 64, Capacity: 64.0 | 17:48 |
sdague | 17:48 | |
*** oomichi- is now known as oomichi | 17:49 | |
auggy | mriedem1: Jenkins protested at the merged policy files change, https://review.openstack.org/#/c/363768/ so I'm looking into it now | 17:49 |
*** sdake has quit IRC | 17:49 | |
dansmith | sdague: I didn't parse that first statement, but is this in CI or locally? | 17:49 |
sdague | dansmith: it's locally | 17:50 |
dansmith | sdague: so using some flavor that could really result in 64 vcpus being allocated? | 17:50 |
*** paul-carlton2 has quit IRC | 17:50 | |
sdague | my env is pretty dirty now with lots of tempest fails for other reasons, so I've got a leak | 17:50 |
dansmith | oh | 17:50 |
sdague | this is trying to allocate a 1 vcpu guest | 17:50 |
dansmith | you mean you think you've got leaked allocations that don't corespond to instances? | 17:50 |
sdague | but nova still thinks I've got 64 allocated | 17:51 |
sdague | yeh | 17:51 |
dansmith | so you can look at the allocations and see the instance uuids that they came from | 17:51 |
auggy | johnthetubaguy: the admin port changes, https://review.openstack.org/#/c/357540/ and https://review.openstack.org/#/c/357726/ are all green now and I removed the WIP, let me know if there's anything else you need help with regarding those | 17:51 |
sdague | dansmith: I think so, what view do you want? | 17:51 |
dansmith | sdague: it would be good to have a manage command that will do a consistency check on instances and allocations | 17:51 |
dansmith | sdague: I don't want a view, I'm just saying if you want to go spot check some things to see if there are allocations for deleted instances, you should be able to do that fairly easily | 17:52 |
sdague | dansmith: yeh, I just assume that's the case. I was just happy to see that warning message was formatted correctly | 17:52 |
dansmith | sdague: consumer_id in allocations should be the instance uuid | 17:52 |
sdague | now that it isn't blowing up all the time randomly | 17:52 |
*** mriedem1 is now known as mriedem | 17:53 | |
sdague | yeh, based on timestamps it looks like I was leaking 2 - 8 per tempest run | 17:54 |
*** tblakes has joined #openstack-nova | 17:54 | |
sdague | but I wasn't even really caring what was passing / failing | 17:54 |
*** fragatina has quit IRC | 17:54 | |
sdague | as I was just trying to generate load | 17:54 |
mriedem | auggy: unrelated, just recheck it | 17:54 |
dansmith | I wonder if allocation delete is ever blocked for some wrong math reason? | 17:54 |
mriedem | that cellsv1 job seems to be failing more often on that same test | 17:54 |
auggy | mriedem: kk will do, just wanted to make sure it wasn't a real issue | 17:55 |
*** fragatina has joined #openstack-nova | 17:55 | |
mriedem | auggy: we shouldn't even be running dsvm jobs on a .gitignore change | 17:55 |
mriedem | we should update project-config to not run dsvm jobs on a change like that, same as how we don't run dsvm jobs for docs or unit test only changes | 17:55 |
auggy | mriedem: fair point, it seemed odd to me too but i'm paranoid | 17:55 |
*** jamesdenton has joined #openstack-nova | 17:55 | |
auggy | mriedem: should i file a bug or something for that as a reminder to make that change? i can work on that | 17:55 |
*** sdake has joined #openstack-nova | 17:56 | |
mriedem | auggy: see https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1339 | 17:56 |
mriedem | don't need a bug | 17:56 |
auggy | kk | 17:56 |
auggy | i'll make a todo for myself and get that in today | 17:56 |
sdague | dansmith: I am sure there are other things we should check here about how things skew from reality over time | 17:57 |
*** akshai has joined #openstack-nova | 17:57 | |
mriedem | auggy: and probably this one https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1999 | 17:57 |
dansmith | sdague: you mean more things we should sanity check inside that capacity transaction? | 17:57 |
mriedem | so .git.* files should be skipped, like .gitignore and .gitreview | 17:58 |
sdague | dansmith: we probably need an audit tool that goes across the API | 17:58 |
*** lyarwood has quit IRC | 17:58 | |
sdague | dansmith: I just mean "in general" | 17:58 |
*** mingyu has joined #openstack-nova | 17:58 | |
dansmith | sdague: such a thing would have to be able to read deleted instances | 17:58 |
auggy | mriedem: awesome, thanks! | 17:58 |
openstackgerrit | Andrey Volkov proposed openstack/nova: WIP Move instance creation to conductor https://review.openstack.org/319379 | 17:58 |
dansmith | so I'd kinda rather do it from manage so we're not building more stuff that relies on deleted instances through the api | 17:58 |
openstackgerrit | Dan Smith proposed openstack/nova: [placement] Allow inventory to violate allocations https://review.openstack.org/365068 | 17:59 |
dansmith | sdague: ^ jaypipes is okay with this and it's pretty critical for correctness | 17:59 |
dansmith | I just removed some more cruft that alex_xu found | 17:59 |
sdague | dansmith: ok, I'll hit that if you hit my join fix | 18:00 |
sdague | dansmith: doesn't a nova manage command get weird once placement is separated? especially once it's another db next cycle some time | 18:01 |
dansmith | sdague: nova manage can hit the placement api if you want, but the instance and these records are already spread across the api and cell databases | 18:02 |
*** edand has quit IRC | 18:02 | |
dansmith | sdague: I'm not sure I'm really able to validate that sql change.. and I thought we were going to wait for a functional test? | 18:03 |
dansmith | well, maybe I could validate it if I wrote said tests, but.. | 18:03 |
*** mingyu has quit IRC | 18:04 | |
sdague | well, the validation at this point is just that KeyError isn't raised | 18:05 |
sdague | that's where we went wrong, we took 9 rows back from the sql query, squashed them down to 3 rows, kind of randomly in a dictionary, and assumed they were the rows we wanted (they often weren't) | 18:06 |
*** crinkle_ is now known as crinkle | 18:07 | |
*** rlrossit_ has joined #openstack-nova | 18:07 | |
*** tbachman_ has joined #openstack-nova | 18:07 | |
*** tbachman has quit IRC | 18:08 | |
*** tbachman_ is now known as tbachman | 18:08 | |
dansmith | sdague: if I remove that functional change and leave the key assert thing, I don't get any extra fails | 18:09 |
sdague | dansmith: in a real environment? | 18:09 |
dansmith | no, in functional tests | 18:09 |
*** oanson has joined #openstack-nova | 18:10 | |
*** rlrossit has quit IRC | 18:10 | |
*** ssurana has joined #openstack-nova | 18:12 | |
dansmith | so I modified one of our tests to hit that keyerror, but your functional change doesn't make it go away | 18:14 |
sdague | dansmith: what did you modify? | 18:14 |
sdague | so, I've tested this live, and it's definitely the real fix. I am sure the functional tests are lacking here | 18:14 |
dansmith | er, wait | 18:14 |
dansmith | actually, I'm hitting a different keyerror in the code you added, but not the raise you want | 18:15 |
auggy | oh wow i just read markus isn't doing bugs or much nova anymore :( | 18:16 |
auggy | i swear, go on a holiday for a day... | 18:17 |
*** pradk has quit IRC | 18:18 | |
*** fragatin_ has joined #openstack-nova | 18:22 | |
tjones | yeah too bad - he did a great job. it's a hard one | 18:23 |
mriedem | Swami: reviewed https://review.openstack.org/#/c/275073/ | 18:24 |
Swami | mriedem: thanks | 18:24 |
*** fragatina has quit IRC | 18:25 | |
dansmith | sdague: so you're adding an extra filter on resource_class, but it seems like the check you're doing is to filter out multiple rows of the same class for two resource providers? | 18:25 |
sdague | dansmith: yeh, it looks like the functional tests are too clever by half and try to do a whole bunch of stuff in a sequence so it's a bit less real. I'll leave someone else to make those sane. That's the real | 18:25 |
sdague | dansmith: no, the key is a tuple | 18:25 |
*** tri2sing has joined #openstack-nova | 18:25 | |
sdague | of rp uuid and resource class | 18:25 |
dansmith | sdague: I know | 18:26 |
tjones | mriedem: are we at the point where we are pausing on removing mox for newton ? | 18:26 |
mriedem | tjones: yes | 18:26 |
tjones | ok thanks | 18:26 |
mriedem | there is a thread in the ML on moving those to an ocata blueprint | 18:26 |
*** oomichi has quit IRC | 18:26 | |
*** oomichi has joined #openstack-nova | 18:27 | |
tjones | yeah i missed it but i figured we were there. | 18:27 |
*** danpb has quit IRC | 18:27 | |
dansmith | sdague: I modified a test so that one RP has two resource classes, which is realistic and I thought what you were saying was the problem that we get the cross-product of the thing we're looking for, such that we have multiple (uuid, class) pairs in one result set | 18:28 |
openstackgerrit | Ryan Rossiter proposed openstack/nova: Log trace when updating host states https://review.openstack.org/366283 | 18:29 |
*** oanson has quit IRC | 18:29 | |
dansmith | sdague: but I don't hit that trigger if I remove your extra resource_class_id filter from the query | 18:29 |
*** thorst has joined #openstack-nova | 18:32 | |
*** jamesdenton has quit IRC | 18:32 | |
sdague | dansmith: yep, I don't know why the tests don't match the reality. I'll let jaypipes / cdent run with that one | 18:33 |
dansmith | okay | 18:34 |
jaypipes | dansmith, sdague: what's up? sorry, been in meetings all day :( | 18:35 |
dansmith | jaypipes: summarizing on a review, just a sec | 18:35 |
jaypipes | k | 18:35 |
*** pradk has joined #openstack-nova | 18:35 | |
dansmith | jaypipes: https://review.openstack.org/#/c/366245/3 | 18:37 |
dansmith | jaypipes: sdague figured out that the VCPU thing is due to us not always picking the right inventory result row during the capacity check, but I can't write a functional test to reproduce/prove it which I think is important | 18:37 |
openstackgerrit | Ryan Rossiter proposed openstack/nova: Warn when using scheduler with debug mode https://review.openstack.org/366289 | 18:37 |
*** thorst has quit IRC | 18:39 | |
*** lamt has quit IRC | 18:40 | |
*** mfedosin has quit IRC | 18:40 | |
jaypipes | dansmith: k. do you need me to pull the above patch and write a func test for the multiple-providers-multiple-resources case? | 18:41 |
dansmith | jaypipes: yeah | 18:41 |
dansmith | jaypipes: write a test that fails without the extra query addition | 18:41 |
jaypipes | dansmith: you got it. | 18:42 |
dansmith | jaypipes: that's what I've been trying to do, but I'm not smart enough apparently | 18:42 |
*** oomichi has quit IRC | 18:42 | |
jaypipes | dansmith: I've got more meetings but will try my best... | 18:42 |
dansmith | are you saying you think we need multipiple RP each with multiple classes to hit that? | 18:42 |
*** baoli has joined #openstack-nova | 18:42 | |
jaypipes | dansmith: yeah. | 18:42 |
dansmith | we should only have one RP in the current test tempest runs I think, so I'm not sure how we'd have hit that... | 18:42 |
*** baoli_ has quit IRC | 18:42 | |
jaypipes | (I think) :) | 18:43 |
sdague | yeh, this seemed hittable with only 1 RP | 18:43 |
* dansmith puzzles | 18:43 | |
sdague | http://logs.openstack.org/41/366241/1/check/gate-tempest-dsvm-neutron-placement-full-ubuntu-xenial-nv/48a0a01/logs/apache/placement-api.txt.gz#_2016-09-06_16_36_22_242 | 18:44 |
sdague | that's in a run with a bunch of unmerged logging | 18:44 |
*** lamt has joined #openstack-nova | 18:44 | |
*** TravT has quit IRC | 18:45 | |
sdague | we definitely only have 1 resource provider there | 18:45 |
dansmith | sdague: what allocation has 192 of anything? | 18:45 |
sdague | mem | 18:45 |
dansmith | we're using a 192MB memory flavor in tempest in the gate? | 18:46 |
sdague | we're using 64MB flavors | 18:46 |
*** prateek_ has joined #openstack-nova | 18:46 | |
*** TravT has joined #openstack-nova | 18:46 | |
sdague | that's the mem usage | 18:46 |
dansmith | that's what I thought, so no allocation should be more than 64 | 18:46 |
jaypipes | dansmith: the resource tracker WARN log message right above there shows 64 vCPU available, using 128 vCPU, requesting 1 vCPU. | 18:46 |
sdague | dansmith: that's *usage* | 18:46 |
dansmith | sdague: oh the log says "allocation record: ..." | 18:47 |
sdague | yes | 18:47 |
sdague | it's the rows from the giant sql monster | 18:47 |
sdague | it was the best name I had for it | 18:47 |
*** rama_y has joined #openstack-nova | 18:47 | |
dansmith | sdague: then that log message should be different :) | 18:47 |
jaypipes | sdague: that's the correct name/term :) | 18:47 |
sdague | ok, tell me what the debug message should be :) | 18:47 |
dansmith | jaypipes: he says "allocation record: $usage" .. that's not right to me :) | 18:47 |
*** raunak has quit IRC | 18:48 | |
*** priteau has joined #openstack-nova | 18:48 | |
dansmith | jaypipes: as in Usage() which is a sum from the resource provider | 18:48 |
*** oomichi has joined #openstack-nova | 18:48 | |
sdague | https://review.openstack.org/#/c/366241/1/nova/objects/resource_provider.py@706 | 18:48 |
jaypipes | dansmith: yes, true. I was just saying "allocation record" is the right term here. | 18:48 |
dansmith | jaypipes: how is it right? | 18:48 |
jaypipes | dansmith: doh, no, you are 100% correct, sorry. | 18:49 |
dansmith | sdague: yeah, I see why you said that, but since we have actual "Allocation" reocrds, I think it's wrong to say that :) | 18:49 |
*** prateek has quit IRC | 18:49 | |
sdague | dansmith: ok, tell me a different word, that's the best I had when I was hacking to the bottom of this | 18:49 |
sdague | anyway, terminology asside | 18:49 |
dansmith | sdague: capacity or usage or something | 18:49 |
sdague | you can see the N^2 matrix there | 18:50 |
*** prateek has joined #openstack-nova | 18:50 | |
dansmith | sdague: the terminology had me concerned we were trying to allocate a bad value is my point | 18:50 |
sdague | ok | 18:50 |
dansmith | sdague: anyway, those are just three records, which we expect.. this is after your fix, right? | 18:51 |
*** prateek_ has quit IRC | 18:51 | |
sdague | dansmith: no, that's before the fix | 18:51 |
dansmith | sdague: okay, but that's not NxM | 18:51 |
sdague | and there a 9 records | 18:51 |
dansmith | that's what I would expect to see | 18:51 |
dansmith | oh, lol.. scroll up | 18:51 |
sdague | :) | 18:52 |
dansmith | yeah, so I don't know why I can | 18:52 |
*** oomichi has quit IRC | 18:52 | |
dansmith | why I can't repro that | 18:52 |
*** oomichi has joined #openstack-nova | 18:52 | |
sdague | yeh, me either | 18:52 |
dansmith | that plus your fix kindaish makes sense to my SQL-dumb brain, but.. I want to see it in a test and I can't make it do that | 18:52 |
openstackgerrit | Sylvain Bauza proposed openstack/nova: WIP: Update BuildRequest if instance currently being scheduled https://review.openstack.org/365740 | 18:52 |
bauzas | alaski: ^ updated for a quickiest patch | 18:53 |
bauzas | alaski: if you agree with it, I'll add the UTS | 18:53 |
bauzas | UTs | 18:53 |
alaski | bauzas: was just commenting on the previous version. I'll finish that and update and look again | 18:54 |
sdague | ok, well, I'll leave that to jaypipes. Going to walk away from computer for a bit. | 18:54 |
jaypipes | dansmith: ah, got it... | 18:54 |
bauzas | alaski: heh, sorry about that :) | 18:55 |
alaski | no worries, I do it too :) | 18:55 |
jaypipes | dansmith: TestAllocationListCreateDelete.test_alllocation_list_create() uses multiple resource providers, but only one resource class inventory record for each resource provider. If we change that to be multiple resource classes per provider, it should fail. | 18:55 |
jaypipes | dansmith: I will update and test pre-patch and post-patch. | 18:56 |
*** raunak has joined #openstack-nova | 18:59 | |
dansmith | jaypipes: that is *precisely* what I have done and I dont' see the fail | 18:59 |
dansmith | jaypipes: http://pastebin.com/cmcvaLqJ | 19:00 |
dansmith | jaypipes: that case of multiple classes for one RP is definitely a giant gap in our current tests for sure | 19:00 |
dansmith | especially since that's a realistic scenario | 19:01 |
*** prateek_ has joined #openstack-nova | 19:02 | |
*** raunak has quit IRC | 19:02 | |
jaypipes | dansmith: ok. well, lemme play around with this and see if I can trigger it. | 19:02 |
dansmith | jaypipes: yep, cool | 19:02 |
dansmith | in about 30 I have to run off for a bit, but will check in with you this afternoon | 19:02 |
*** prateek has quit IRC | 19:05 | |
*** prateek_ has quit IRC | 19:06 | |
alaski | bauzas: added some comments. It still has some checks I think you can remove, but looks close | 19:07 |
*** prateek_ has joined #openstack-nova | 19:07 | |
bauzas | alaski: okay, fair enough, I'll work on that tomorrow with the UTs so we could try to merge it before RC1 | 19:08 |
openstackgerrit | Ryan Rossiter proposed openstack/nova: Warn when using scheduler with debug mode https://review.openstack.org/366289 | 19:08 |
bauzas | alaski: that's a regression hence me running for it | 19:08 |
bauzas | auggy: FWIW, saw your email | 19:08 |
*** parora has joined #openstack-nova | 19:09 | |
bauzas | auggy: I think we need to pay close attention to new and in progress bugs to see whether they're related to some regression | 19:09 |
*** gjayavelu has joined #openstack-nova | 19:09 | |
alaski | bauzas: yep, we need to get this fixed | 19:09 |
*** akshai has quit IRC | 19:09 | |
bauzas | auggy: I tried to look around all the open bugs, but I really need help here :) | 19:09 |
*** Sukhdev has joined #openstack-nova | 19:09 | |
openstackgerrit | Ryan Rossiter proposed openstack/nova: Log trace when updating host states https://review.openstack.org/366283 | 19:11 |
*** pcaruana has quit IRC | 19:11 | |
*** prateek_ has quit IRC | 19:12 | |
*** prateek_ has joined #openstack-nova | 19:13 | |
*** cdent has joined #openstack-nova | 19:14 | |
*** Guest78236 is now known as melwitt | 19:14 | |
bauzas | rlrossit_: so, yeah, I agree that running DEBUG for the scheduler is a very bad idea... | 19:14 |
bauzas | rlrossit_: like drinking while driving | 19:14 |
bauzas | rlrossit_: but there, should we just add some ads saying that it's bad, or rather find some situations where DEBUG logs are unnecessary? | 19:15 |
cdent | dansmith: you've made me feel better. I was unable to write a failing functional test too and left for dinner in frustration. I wonder if it is sqlite artifact (or mysql). I can demonstrate the bad behavior in devstack no problem. | 19:15 |
bauzas | rlrossit_: I tend to like the latter :) | 19:15 |
auggy | bauzas: lol i'm not a pro either but i will do what i can ;) | 19:15 |
dansmith | cdent: yeah, I dunno, but this seems like an odd thing to be an artifact | 19:15 |
bauzas | auggy: bug triaging is one way to come by Nova :) | 19:15 |
*** cleong has quit IRC | 19:15 | |
auggy | bauzas: yeah, i've been pretty active with helping markus this past year | 19:16 |
rlrossit_ | bauzas: I have decided to do both | 19:16 |
bauzas | rlrossit_: \o/ | 19:16 |
* auggy has learned a lot from nova bugs | 19:16 | |
*** mriedem has quit IRC | 19:16 | |
rlrossit_ | I'm saying debug=True with scheduler is a bad thing, and then I also move the main debug culprit to trace() | 19:16 |
*** parora has quit IRC | 19:16 | |
rlrossit_ | there's 2 patches | 19:16 |
bauzas | rlrossit_: I just saw the second one | 19:16 |
bauzas | rlrossit_: tbh, I'm very not sold about saying it's a bad thing | 19:17 |
bauzas | rlrossit_: that's just life | 19:17 |
cdent | dansmith: that's all I had, going back to not being here | 19:17 |
*** cdent has quit IRC | 19:17 | |
rlrossit_ | https://review.openstack.org/366289 is "Hey you shouldn't do this", and https://review.openstack.org/366283 is "well, you did it, let's not make it hurt so much" | 19:17 |
bauzas | rlrossit_: yeah I very gently -1d the former :D | 19:17 |
*** fragatin_ has quit IRC | 19:17 | |
*** fragatina has joined #openstack-nova | 19:18 | |
bauzas | rlrossit_: so the problem is more that you need DEBUG for getting important info when some bad scheduling happens | 19:19 |
bauzas | rlrossit_: I mean, some operators want to use DEBUG in production because they think they miss some crucial information about some bad scheduling | 19:19 |
bauzas | rlrossit_: but just changing the log level doesn't help them IMHO | 19:20 |
bauzas | rlrossit_: we should rather think about why they need to use DEBUG | 19:20 |
*** pradk- has joined #openstack-nova | 19:20 | |
bauzas | rlrossit_: see ? | 19:20 |
*** pradk has quit IRC | 19:21 | |
rlrossit_ | yeah, I don't disagree with that | 19:21 |
*** parora has joined #openstack-nova | 19:21 | |
rlrossit_ | and granted, when I was doing this performance stuff, I wouldn't go looking in the logs to see that I shouldn't be running with debug on. | 19:21 |
rlrossit_ | it was slow, and I didn't know why or where until I commented logging out | 19:22 |
*** pradk- is now known as pradk | 19:22 | |
*** lamt has quit IRC | 19:24 | |
*** rmart04 has joined #openstack-nova | 19:24 | |
bauzas | rlrossit_: that's why we had some BP related to that | 19:24 |
bauzas | rlrossit_: lemme find it | 19:25 |
*** tesseract- has quit IRC | 19:25 | |
rlrossit_ | yeah mriedem linked it to me on Friday | 19:25 |
*** prateek_ has quit IRC | 19:25 | |
rlrossit_ | cfriesen is working on it I believe | 19:25 |
*** lpetrut has quit IRC | 19:25 | |
bauzas | rlrossit_: the idea was to stack all the filter logs so that where we were having a NoValidHost, then it was popped | 19:25 |
*** akshai has joined #openstack-nova | 19:25 | |
rlrossit_ | yup | 19:25 |
rlrossit_ | "in-memory logging" of sorts | 19:26 |
bauzas | rlrossit_: well, I haven't seen much of a progress here, so I think you should maybe see how cfriesen and you could resurrect that | 19:26 |
melwitt | I had thought turning off debug log level for performance testing was a given, be it openstack or not | 19:26 |
bauzas | the Ocata cycle will open soon and I'd be very happy to see that | 19:26 |
rlrossit_ | melwitt: well, I didn't think debug logging would be *that* bad | 19:27 |
bauzas | melwitt: we should rename s/DEBUG/EDONTDOTHISFORPRODUCTION I think :p | 19:27 |
dansmith | I thought that we actually hear that most big ops actually run with debug on all the time because they have to have it to diagnose any real issue | 19:27 |
rlrossit_ | bauzas: still doesn't stop some people ;) | 19:27 |
dansmith | i.e. they pay the penalty for the benefit of being able to fix things | 19:27 |
bauzas | dansmith: yup, why I pointed out cfriesen's BP | 19:28 |
dansmith | that may have come via sdague a while ago | 19:28 |
melwitt | we didn't. debug is definitely not on unless something bad happens and debugging is needed | 19:28 |
rlrossit_ | dansmith: yeah... that's what we're starting to live with | 19:28 |
dansmith | melwitt: okay | 19:28 |
*** diogogmt has quit IRC | 19:29 | |
*** hemanthm is now known as hemanthm|away | 19:29 | |
openstackgerrit | Swaminathan Vasudevan proposed openstack/nova: Implement setup_networks_on_host for Neutron networks https://review.openstack.org/275073 | 19:30 |
bauzas | rlrossit_: grah, can't find cfriesen's BP | 19:30 |
bauzas | rlrossit_: I'd have checked the whiteboard to see the status | 19:30 |
*** diogogmt has joined #openstack-nova | 19:31 | |
*** tbachman_ has joined #openstack-nova | 19:31 | |
bauzas | grah, can someone tell me how I can find the gerrit username for cfriesen so I could search for his open changes ? | 19:31 |
*** coolsvap_ has quit IRC | 19:32 | |
*** tbachman has quit IRC | 19:32 | |
*** tbachman_ is now known as tbachman | 19:32 | |
bauzas | nevermind, found it | 19:32 |
bauzas | rlrossit_: https://review.openstack.org/#/c/331306/ has some implementation concerns, possibly good to disccuss that with the owner | 19:33 |
*** claudiub has quit IRC | 19:34 | |
bauzas | rlrossit_: in particular given your testbed of 400 nodes, could be interesting to see the memory usage for that | 19:34 |
*** Apoorva has quit IRC | 19:34 | |
alaski | I know that rax runs(ran) with debug logs on, because tracking down issues would have been near impossible without it. But cells may have been (i.e. was) a factor there. | 19:35 |
*** rmart04 has quit IRC | 19:35 | |
*** parora has quit IRC | 19:36 | |
alaski | efforts like cfriesens to only dump logs on failure would be a huge help there | 19:36 |
bauzas | alaski: yup | 19:36 |
rlrossit_ | bauzas: I don't see that using much more memory than holding 400 host state maps in memory | 19:36 |
*** thorst has joined #openstack-nova | 19:36 | |
bauzas | rlrossit_: well, that's different | 19:36 |
bauzas | rlrossit_: holding 400 host state dicts is one thing | 19:37 |
bauzas | rlrossit_: allowing to write in memory long strings is another, I guess | 19:37 |
bauzas | as you stated, logs can be very verbose :) | 19:37 |
rlrossit_ | I guess what I'm trying to say is: overall, it's not that much *more* memory | 19:38 |
rlrossit_ | ram is cheap, file I/O is expensive | 19:38 |
rlrossit_ | well, that debug that I moved down to trace doesn't really help much in the case of scheduling failures anyways :) | 19:39 |
bauzas | rlrossit_: but you can define a log factory for each level you know :) | 19:40 |
*** rcernin has joined #openstack-nova | 19:41 | |
bauzas | anyway, I just want to tell that merging your patch won't really solve the problem | 19:41 |
bauzas | it could just reduce the verbosity, sure | 19:41 |
bauzas | but that would still need some further improvment | 19:41 |
*** thorst has quit IRC | 19:45 | |
*** sdake has quit IRC | 19:46 | |
*** moshele has joined #openstack-nova | 19:48 | |
*** rcernin has quit IRC | 19:48 | |
*** pradk has quit IRC | 19:49 | |
sdague | jaypipes: you have that test fix, because if not, I have it, now that you gave the hint | 19:50 |
jaypipes | sdague: meetings :( | 19:52 |
sdague | ok | 19:52 |
*** TravT has quit IRC | 19:55 | |
openstackgerrit | Sean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting https://review.openstack.org/366245 | 19:56 |
openstackgerrit | Sean Dague proposed openstack/nova: DNM: test SQL fix https://review.openstack.org/366313 | 19:56 |
*** eharney has quit IRC | 19:56 | |
*** devananda is now known as devananda|travel | 19:57 | |
sdague | ug, that's not really it, I need to futz with this a bit more | 19:59 |
*** davidlenwell has quit IRC | 19:59 | |
*** mvk has quit IRC | 20:01 | |
*** kfarr has joined #openstack-nova | 20:02 | |
*** raunak has joined #openstack-nova | 20:03 | |
*** sdake has joined #openstack-nova | 20:03 | |
*** joshblake has joined #openstack-nova | 20:04 | |
cfriesen | sfinucan: you on? | 20:06 |
*** eharney has joined #openstack-nova | 20:08 | |
openstackgerrit | Sean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting https://review.openstack.org/366245 | 20:09 |
*** raunak has quit IRC | 20:09 | |
openstackgerrit | Sean Dague proposed openstack/nova: DNM: test SQL fix https://review.openstack.org/366313 | 20:09 |
*** tbachman has quit IRC | 20:10 | |
*** tbachman has joined #openstack-nova | 20:11 | |
sdague | yeh, still doesn't quite trigger it, I'm not sure now | 20:11 |
*** ljxiash has quit IRC | 20:11 | |
*** davidlenwell has joined #openstack-nova | 20:12 | |
*** TravT has joined #openstack-nova | 20:13 | |
*** edmondsw has quit IRC | 20:13 | |
bauzas | cfriesen: sfinucan lives in Ireland, so probably off for tonight | 20:15 |
*** raunak has joined #openstack-nova | 20:18 | |
*** unicell has joined #openstack-nova | 20:18 | |
*** kevinz has quit IRC | 20:23 | |
openstackgerrit | Andrew Laski proposed openstack/nova: Annotate db models that have moved to the nova_api db https://review.openstack.org/366327 | 20:26 |
*** praneshp has joined #openstack-nova | 20:29 | |
*** salv-orl_ has joined #openstack-nova | 20:30 | |
*** moshele has quit IRC | 20:33 | |
*** salv-orlando has quit IRC | 20:33 | |
*** thorst has joined #openstack-nova | 20:33 | |
cfriesen | bauzas: ah, thanks. will try tomorrow | 20:33 |
bauzas | np | 20:34 |
bauzas | cfriesen: I had a convo with rlrossit_ about the scheduler logging BP you wrote | 20:35 |
bauzas | cfriesen: hopefully you'll have time to work on it for Ocata ? | 20:35 |
*** thorst has quit IRC | 20:39 | |
openstackgerrit | Swaminathan Vasudevan proposed openstack/nova: Implement setup_networks_on_host for Neutron networks https://review.openstack.org/275073 | 20:40 |
*** ayoung has quit IRC | 20:41 | |
*** moshele has joined #openstack-nova | 20:41 | |
*** pradk has joined #openstack-nova | 20:44 | |
*** cdelatte has quit IRC | 20:44 | |
sdague | jaypipes: ok, I think I figured it out | 20:46 |
sdague | I needed 2 consumers | 20:46 |
sdague | otherwise during allocations it deleted all the usage for the current consumer | 20:47 |
openstackgerrit | Sean Dague proposed openstack/nova: DNM: test SQL fix https://review.openstack.org/366313 | 20:47 |
openstackgerrit | Sean Dague proposed openstack/nova: correctly join the usage to inventory for capacity accounting https://review.openstack.org/366245 | 20:47 |
jaypipes | sdague: aha. | 20:47 |
sdague | so 1 resource_provider, > 1 active consumer, > 1 resource class | 20:47 |
jaypipes | right, that makes sense. | 20:47 |
sdague | there is a dedicated test for this in the patch now | 20:48 |
jaypipes | sdague: nice digging. i literally just got off my 11th meeting today (not kidding) | 20:48 |
sdague | yep, no worries. I learned a bunch about the whole system in the process, so that's always good | 20:48 |
cfriesen | bauzas: hopefully, yes. we were kind of late out the gate for N...other stuff going on. | 20:49 |
jaypipes | sdague: thank you very much for your help on all this over last couple weeks. I really appreciate it. | 20:49 |
sdague | jaypipes: happy to help | 20:49 |
*** agireud has quit IRC | 20:52 | |
sdague | I think you owe me a cookie or drink or something in barcelona for debugging your SQL though :) | 20:52 |
*** rlrossit_ has quit IRC | 20:54 | |
*** TravT has quit IRC | 20:55 | |
*** agireud has joined #openstack-nova | 20:59 | |
*** mriedem has joined #openstack-nova | 21:01 | |
*** sdake has quit IRC | 21:02 | |
sdague | jaypipes: it would be cool if you could review - https://review.openstack.org/#/c/366245/ for correctness | 21:03 |
*** kylek3h has quit IRC | 21:04 | |
*** derekh has quit IRC | 21:05 | |
*** raunak has quit IRC | 21:06 | |
openstackgerrit | Anusha Unnam proposed openstack/nova: List soft_deleted instances https://review.openstack.org/366347 | 21:08 |
dansmith | sdague: I'm pulling it down to play now | 21:09 |
sdague | dansmith: you should get the fail on the DNM patch | 21:10 |
*** mmedvede has quit IRC | 21:11 | |
*** mmedvede has joined #openstack-nova | 21:11 | |
*** raunak has joined #openstack-nova | 21:11 | |
*** ljxiash has joined #openstack-nova | 21:12 | |
*** raunak has quit IRC | 21:12 | |
dansmith | sdague: found a bug, fixing locally and will push it up | 21:12 |
*** mvk has joined #openstack-nova | 21:13 | |
sdague | cool | 21:13 |
openstackgerrit | Dan Smith proposed openstack/nova: correctly join the usage to inventory for capacity accounting https://review.openstack.org/366245 | 21:13 |
sdague | dansmith: you deleted the fix | 21:14 |
dansmith | hah | 21:14 |
sdague | https://review.openstack.org/#/c/366245/6..7/nova/objects/resource_provider.py | 21:14 |
dansmith | because I was testing | 21:14 |
dansmith | sorry | 21:14 |
sdague | the map_key bit works | 21:14 |
sdague | without the explicit str() | 21:14 |
jaypipes | sdague: +2 from me. | 21:15 |
*** joshblake has quit IRC | 21:15 | |
dansmith | sdague: it doesn't here, it fails | 21:15 |
dansmith | sdague: with string format | 21:16 |
*** ljxiash has quit IRC | 21:16 | |
openstackgerrit | Dan Smith proposed openstack/nova: correctly join the usage to inventory for capacity accounting https://review.openstack.org/366245 | 21:16 |
sdague | odd | 21:16 |
dansmith | and it should fail of course | 21:16 |
*** tonytan_brb has joined #openstack-nova | 21:17 | |
sdague | oh... right | 21:17 |
dansmith | raise KeyError("%s already in usage_map, bad query" % map_key) | 21:17 |
dansmith | TypeError: not all arguments converted during string formatting | 21:17 |
*** raunak has joined #openstack-nova | 21:18 | |
*** tri2sing has quit IRC | 21:19 | |
*** tonytan4ever has quit IRC | 21:19 | |
*** tri2sing has joined #openstack-nova | 21:19 | |
*** Apoorva has joined #openstack-nova | 21:21 | |
*** dimtruck is now known as zz_dimtruck | 21:23 | |
sdague | yep | 21:25 |
sdague | I guess I had just gone blind in my *stacktest* buffer | 21:26 |
*** rmart04 has joined #openstack-nova | 21:26 | |
dansmith | you get a similar enough error that it seems like it's working | 21:27 |
sdague | yeh | 21:27 |
sdague | jaypipes: ok, you need to swing around with your +2 again, as dansmith had to fix the fix | 21:27 |
jaypipes | sdague: done | 21:29 |
sdague | \o/ ... ok, time to go make dinner. See you all tomorrow | 21:29 |
*** ametts has quit IRC | 21:30 | |
mriedem | alaski: rax doesn't support floating IPs right? | 21:32 |
*** rfolco has quit IRC | 21:32 | |
*** rtheis has quit IRC | 21:33 | |
jroll | mriedem: nope, not today | 21:34 |
mriedem | ok, was just reading this 'Launch instance with Floating IP' dev list thread | 21:34 |
*** rmart04 has left #openstack-nova | 21:34 | |
mriedem | about industry standards | 21:34 |
clarkb | also no security groups though thats orthogonal to the thread youare talking about | 21:35 |
*** thorst has joined #openstack-nova | 21:37 | |
*** annegentle has quit IRC | 21:39 | |
mriedem | industry standard | 21:41 |
mriedem | 'nuff said | 21:41 |
*** moshele has quit IRC | 21:42 | |
*** pradk has quit IRC | 21:43 | |
* dansmith is "industry standard" | 21:45 | |
*** thorst has quit IRC | 21:45 | |
*** diogogmt has quit IRC | 21:45 | |
*** liang_ has quit IRC | 21:45 | |
*** esmiurium has quit IRC | 21:46 | |
jaypipes | if I'm ever referred to as "industry standard", it is dansmith's sworn oath to shoot me in the head with a grenade launcher. | 21:46 |
dansmith | wow, that may be a little overkill | 21:47 |
dansmith | I think a small-calibre handgun would be plenty | 21:47 |
jaypipes | dansmith: it's an industry standard response, really. | 21:47 |
*** esmiurium has joined #openstack-nova | 21:47 | |
* dansmith checks on the industry standard for executions | 21:47 | |
*** tri2sing has quit IRC | 21:49 | |
*** tri2sing has joined #openstack-nova | 21:50 | |
*** Sukhdev has quit IRC | 21:51 | |
*** stevemar has quit IRC | 21:52 | |
*** Sukhdev has joined #openstack-nova | 21:55 | |
*** Sukhdev has quit IRC | 21:59 | |
*** burt has quit IRC | 22:00 | |
*** zz_dimtruck is now known as dimtruck | 22:02 | |
*** annegentle has joined #openstack-nova | 22:02 | |
*** gyee has joined #openstack-nova | 22:05 | |
*** BjoernT has quit IRC | 22:05 | |
*** kylek3h has joined #openstack-nova | 22:06 | |
*** Sukhdev has joined #openstack-nova | 22:09 | |
*** tri2sing has quit IRC | 22:10 | |
*** tri2sing has joined #openstack-nova | 22:11 | |
*** tri2sing has quit IRC | 22:11 | |
*** baoli has quit IRC | 22:12 | |
*** baoli has joined #openstack-nova | 22:13 | |
*** kfarr has quit IRC | 22:13 | |
*** su_zhang has quit IRC | 22:14 | |
*** liangy has joined #openstack-nova | 22:15 | |
jroll | grenade will just bounce off your head and blow off your legs, though :( | 22:15 |
jroll | :P | 22:16 |
*** priteau has quit IRC | 22:16 | |
*** xyang1 has quit IRC | 22:17 | |
jaypipes | jroll: honestly, after my day today, even that would be a welcome event. | 22:17 |
jroll | :( | 22:18 |
*** eharney has quit IRC | 22:18 | |
*** mdrabe has quit IRC | 22:22 | |
*** priteau has joined #openstack-nova | 22:22 | |
*** stevemar has joined #openstack-nova | 22:24 | |
*** stevemar has quit IRC | 22:26 | |
*** marst has quit IRC | 22:26 | |
*** _ducttape_ has joined #openstack-nova | 22:26 | |
*** priteau has quit IRC | 22:26 | |
*** stevemar has joined #openstack-nova | 22:26 | |
openstackgerrit | Eric Brown proposed openstack/nova: VMware: deprecate wsdl_location conf option https://review.openstack.org/342470 | 22:26 |
*** liangy has quit IRC | 22:27 | |
*** tblakes has quit IRC | 22:28 | |
*** ducttape_ has quit IRC | 22:29 | |
*** stevemar has quit IRC | 22:31 | |
*** liangy has joined #openstack-nova | 22:32 | |
*** krtaylor has joined #openstack-nova | 22:32 | |
*** dimtruck is now known as zz_dimtruck | 22:34 | |
*** yamahata has quit IRC | 22:40 | |
*** thorst has joined #openstack-nova | 22:43 | |
*** annegentle has quit IRC | 22:46 | |
*** zz_dimtruck is now known as dimtruck | 22:47 | |
*** thorst has quit IRC | 22:49 | |
*** yamahata has joined #openstack-nova | 22:49 | |
*** thorst has joined #openstack-nova | 22:49 | |
*** akshai has quit IRC | 22:53 | |
*** baoli has quit IRC | 22:58 | |
*** krtaylor has quit IRC | 23:06 | |
*** stevemar has joined #openstack-nova | 23:07 | |
*** mingyu has joined #openstack-nova | 23:08 | |
*** mingyu has quit IRC | 23:10 | |
*** ayoung has joined #openstack-nova | 23:10 | |
*** sdake has joined #openstack-nova | 23:10 | |
*** marst has joined #openstack-nova | 23:12 | |
*** ljxiash has joined #openstack-nova | 23:13 | |
harlowja | qq guys, i'm trying to find a good set of links to share with folks on the topic of (what is cells v1); what do u guys think the best source is? (even though i know its cells v1...) | 23:18 |
*** ljxiash has quit IRC | 23:18 | |
*** markvoelker has quit IRC | 23:19 | |
harlowja | said folks are going to try to debug some of the sync issues, so any links people have might be useful (vs me just searching around, ha) | 23:20 |
*** jamesdenton has joined #openstack-nova | 23:21 | |
*** akshai has joined #openstack-nova | 23:22 | |
*** jaypipes has quit IRC | 23:25 | |
*** jamesdenton has quit IRC | 23:26 | |
*** mriedem has quit IRC | 23:30 | |
*** mriedem has joined #openstack-nova | 23:32 | |
*** dimtruck is now known as zz_dimtruck | 23:32 | |
*** akshai has quit IRC | 23:33 | |
*** bapalm has quit IRC | 23:35 | |
openstackgerrit | melanie witt proposed openstack/nova: Make req spec and RT aware of volume-backed instances https://review.openstack.org/355091 | 23:38 |
*** Sukhdev has quit IRC | 23:40 | |
*** rama_y has quit IRC | 23:40 | |
*** bapalm has joined #openstack-nova | 23:41 | |
*** raunak has quit IRC | 23:41 | |
*** zz_dimtruck is now known as dimtruck | 23:46 | |
*** mingyu has joined #openstack-nova | 23:48 | |
*** Jeffrey4l has joined #openstack-nova | 23:49 | |
*** Jeffrey4l has quit IRC | 23:55 | |
*** markvoelker has joined #openstack-nova | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!