Thursday, 2017-08-10

*** thorst has joined #openstack-nova00:04
*** mingyu has joined #openstack-nova00:05
*** mingyu has quit IRC00:10
*** rajathagasthya has quit IRC00:13
*** yangyapeng has quit IRC00:16
*** yassine has joined #openstack-nova00:19
*** thingee_ has quit IRC00:19
*** suresh12 has joined #openstack-nova00:20
*** itlinux has joined #openstack-nova00:20
*** kornicameister has quit IRC00:25
*** kornicameister has joined #openstack-nova00:26
*** edmondsw has joined #openstack-nova00:30
masbergood morning nova team, I am not sure whether this is the right place to ask this question (please forgive me if not) I am testing openstack vs raw performance in terms of disk latency using ioping and realized that disk latency from the host is around 159us whole on the vm is around 388us. Could someone please help me to understand why is that? thank you very much00:30
masberby the way I am using openstack all in one flavor using packstack so instance storage is local lvm00:31
*** thorst has quit IRC00:31
*** mamandle has joined #openstack-nova00:34
*** edmondsw has quit IRC00:34
*** Nel1x has joined #openstack-nova00:35
*** yamamoto_ has quit IRC00:37
*** yamamoto has joined #openstack-nova00:39
*** kiennt has joined #openstack-nova00:39
*** kornicameister has quit IRC00:40
*** Sukhdev has quit IRC00:41
*** tyrefors has joined #openstack-nova00:42
*** Apoorva_ has quit IRC00:43
*** sbezverk has quit IRC00:44
*** kornicameister has joined #openstack-nova00:45
*** xinliang has joined #openstack-nova00:45
*** xinliang has quit IRC00:45
*** xinliang has joined #openstack-nova00:45
*** markvoelker has joined #openstack-nova00:45
*** mdnadeem has joined #openstack-nova00:46
openstackgerritMerged openstack/nova master: Add release note for shared storage known issue  https://review.openstack.org/49158200:48
*** kristian__ has joined #openstack-nova00:49
openstackgerritMerged openstack/nova master: add cli to support matrix  https://review.openstack.org/48149800:51
*** kristian__ has quit IRC00:54
*** eharney has joined #openstack-nova00:59
*** tetsuro has joined #openstack-nova00:59
*** mingyu has joined #openstack-nova01:01
*** suresh12 has quit IRC01:02
*** jichen has joined #openstack-nova01:04
*** phuongnh has joined #openstack-nova01:07
*** dave-mccowan has joined #openstack-nova01:07
*** Shunli has joined #openstack-nova01:08
*** thorst has joined #openstack-nova01:08
*** thorst has quit IRC01:08
*** thorst has joined #openstack-nova01:09
alex_xugit show ls01:12
alex_xuoops01:12
edleafealex_xu: good thing I've never typed in the wrong window!01:13
alex_xuedleafe: :)01:13
*** thorst has quit IRC01:13
*** gbarros has joined #openstack-nova01:14
alex_xuthat is fault of multiple screens01:14
*** yangyapeng has joined #openstack-nova01:17
*** tuanluong has joined #openstack-nova01:19
*** crushil has joined #openstack-nova01:23
*** mamandle has quit IRC01:24
*** thorst has joined #openstack-nova01:25
*** thorst has quit IRC01:27
*** litao__ has joined #openstack-nova01:29
*** kornicameister has quit IRC01:30
*** kornicameister has joined #openstack-nova01:35
*** bkopilov has quit IRC01:43
*** gongysh has joined #openstack-nova01:46
openstackgerrithuangtianhua proposed openstack/python-novaclient master: Allow boot server with multiple nics  https://review.openstack.org/49200301:58
*** crushil has quit IRC01:59
*** diga has quit IRC02:18
*** chyka has joined #openstack-nova02:19
*** yuchenghui has quit IRC02:20
*** chyka has quit IRC02:23
*** Sukhdev_ has quit IRC02:23
*** yamahata has quit IRC02:25
*** eharney has quit IRC02:26
*** dave-mccowan has quit IRC02:27
*** rajathagasthya has joined #openstack-nova02:28
*** thorst has joined #openstack-nova02:28
*** hongbin has joined #openstack-nova02:28
*** thorst has quit IRC02:29
*** hemna_ has quit IRC02:29
*** kornicameister has quit IRC02:29
*** kornicameister has joined #openstack-nova02:35
*** esberglu has quit IRC02:40
*** rajathagasthya has quit IRC02:44
*** kbaegis has quit IRC02:45
*** rajathagasthya has joined #openstack-nova02:47
*** hongbin has quit IRC02:48
*** hongbin has joined #openstack-nova02:49
*** rajathagasthya has quit IRC02:49
*** dklyle has quit IRC02:49
*** zhurong has joined #openstack-nova02:52
*** hongbin has quit IRC02:52
*** hongbin has joined #openstack-nova02:53
*** Nel1x has quit IRC02:53
*** dave-mccowan has joined #openstack-nova02:54
*** yamahata has joined #openstack-nova02:54
*** kbaegis has joined #openstack-nova02:55
*** rajathagasthya has joined #openstack-nova02:58
*** markvoelker has quit IRC03:02
*** markvoelker has joined #openstack-nova03:03
*** bkopilov has joined #openstack-nova03:04
*** mamandle has joined #openstack-nova03:08
*** mamandle has quit IRC03:19
*** itlinux has quit IRC03:20
*** david-lyle has joined #openstack-nova03:22
*** vks1 has joined #openstack-nova03:25
*** Shunli has quit IRC03:25
*** kbaegis1 has joined #openstack-nova03:26
*** nicolasbock has joined #openstack-nova03:29
*** kbaegis has quit IRC03:29
*** suresh12 has joined #openstack-nova03:32
*** yamamoto has quit IRC03:36
*** suresh12 has quit IRC03:37
*** gbarros has quit IRC03:37
*** udesale has joined #openstack-nova03:40
*** hongbin has quit IRC03:43
*** rajathagasthya has quit IRC03:47
*** yamamoto has joined #openstack-nova03:47
*** dave-mccowan has quit IRC03:48
*** yamamoto has quit IRC03:48
*** yamamoto has joined #openstack-nova03:49
openstackgerritMerged openstack/nova master: Test resize with too big flavor  https://review.openstack.org/49081403:49
openstackgerritNaichuan Sun proposed openstack/nova master: xenapi: Live migration failed in xapi pool  https://review.openstack.org/48945103:49
*** kornicameister has quit IRC03:53
*** kornicameister has joined #openstack-nova03:54
*** diga has joined #openstack-nova03:56
*** markvoelker has quit IRC04:01
*** markvoelker has joined #openstack-nova04:01
*** mamandle has joined #openstack-nova04:01
*** thorst has joined #openstack-nova04:04
*** rmcall has quit IRC04:07
*** suresh12 has joined #openstack-nova04:09
*** thorst has quit IRC04:09
*** sridharg has joined #openstack-nova04:11
*** suresh12 has quit IRC04:14
*** moshele has joined #openstack-nova04:18
*** Sukhdev has joined #openstack-nova04:20
*** thorst has joined #openstack-nova04:26
*** moshele has quit IRC04:28
*** gongysh has quit IRC04:30
*** thorst has quit IRC04:30
*** gouthamr has joined #openstack-nova04:31
*** shan has joined #openstack-nova04:33
*** trinaths has joined #openstack-nova04:35
*** sbezverk has joined #openstack-nova04:35
*** adisky__ has joined #openstack-nova04:37
*** hareesh has joined #openstack-nova04:38
*** sbezverk has quit IRC04:40
*** rmcall has joined #openstack-nova04:40
*** claudiub has joined #openstack-nova04:40
*** ekuris has joined #openstack-nova04:40
*** suresh12 has joined #openstack-nova04:42
*** ratailor has joined #openstack-nova04:43
*** david-lyle has quit IRC04:44
*** rmcall has quit IRC04:45
*** dklyle has joined #openstack-nova04:45
*** gyee has joined #openstack-nova04:48
*** mingyu has quit IRC04:49
*** mingyu has joined #openstack-nova04:50
*** rajathagasthya has joined #openstack-nova04:50
*** cfriesen has quit IRC04:50
*** mingyu has quit IRC04:50
*** rajathagasthya has quit IRC04:51
*** rajathagasthya has joined #openstack-nova04:51
*** suresh12 has quit IRC04:53
openstackgerritSpencer Yu proposed openstack/python-novaclient master: Nova client should retry with Retry-After value  https://review.openstack.org/44776604:56
*** gongysh has joined #openstack-nova05:09
openstackgerritNaichuan Sun proposed openstack/nova master: xenapi: Live migration failed in xapi pool  https://review.openstack.org/48945105:10
jianghuawmriedem, thanks for the comments for https://bugs.launchpad.net/nova/+bug/1709594.05:11
openstackLaunchpad bug 1709594 in OpenStack Compute (nova) "live-migration without '--block-migrate" failed with "No sql_connection parameter is established"" [Undecided,New]05:11
jianghuawYes, this is a devstack configuration and it's running in superconductor mode.05:11
*** sree has joined #openstack-nova05:11
jianghuawthe database configuration in nova.conf is:05:12
jianghuaw[database]05:12
jianghuawconnection = mysql+pymysql://root:citrix@127.0.0.1/nova_cell0?charset=utf805:12
jianghuaw[api_database]05:12
jianghuawconnection = mysql+pymysql://root:citrix@127.0.0.1/nova_api?charset=utf805:12
jianghuawmriedem, the nova_api database connection settings seems correct.05:14
*** gouthamr has quit IRC05:16
*** moshele has joined #openstack-nova05:16
*** gmann has joined #openstack-nova05:17
*** litao__ has quit IRC05:19
*** gjayavelu has joined #openstack-nova05:20
*** suresh12 has joined #openstack-nova05:21
*** mamandle has quit IRC05:22
openstackgerritAlex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it  https://review.openstack.org/48037905:24
*** Sukhdev has quit IRC05:30
*** mamandle has joined #openstack-nova05:30
*** kiennt has quit IRC05:34
*** jamesdenton has quit IRC05:36
*** abalutoiu has quit IRC05:39
*** mingyu has joined #openstack-nova05:46
*** suresh12 has quit IRC05:47
*** armax has joined #openstack-nova05:48
*** abalutoiu has joined #openstack-nova05:48
*** armax has quit IRC05:48
*** mvk has joined #openstack-nova05:49
*** mingyu has quit IRC05:49
*** psachin has joined #openstack-nova05:52
*** kristian__ has joined #openstack-nova05:52
*** suresh12 has joined #openstack-nova05:53
*** kristian__ has quit IRC05:56
*** jamesdenton has joined #openstack-nova05:57
*** kiennt has joined #openstack-nova06:00
*** mamandle_ has joined #openstack-nova06:01
*** yamamoto has quit IRC06:01
*** mamandle has quit IRC06:01
*** mamandle_ is now known as mamandle06:01
*** slaweq has quit IRC06:05
*** rcernin has joined #openstack-nova06:05
*** lpetrut has joined #openstack-nova06:05
*** yamamoto has joined #openstack-nova06:05
*** mamandle has quit IRC06:09
*** gmann has quit IRC06:11
*** Oku_OS-away is now known as Oku_OS06:12
*** junbo has quit IRC06:12
*** gmann has joined #openstack-nova06:12
*** junbo has joined #openstack-nova06:15
*** jamesdenton has quit IRC06:16
*** suresh12 has quit IRC06:20
openstackgerritAlex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it  https://review.openstack.org/48037906:22
*** thorst has joined #openstack-nova06:26
*** slaweq has joined #openstack-nova06:29
*** thorst has quit IRC06:31
*** moshele has quit IRC06:33
*** jamesdenton has joined #openstack-nova06:34
*** kristian__ has joined #openstack-nova06:34
*** rajathagasthya has quit IRC06:36
*** yuchenghui has joined #openstack-nova06:37
*** itlinux has joined #openstack-nova06:38
*** jamesdenton has quit IRC06:39
*** sridharg has quit IRC06:39
*** kukacz has joined #openstack-nova06:43
openstackgerritAlex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it  https://review.openstack.org/48037906:43
openstackgerritAlex Xu proposed openstack/nova master: placement: avoid to return duplicated alloc_reqs when no sharing rp  https://review.openstack.org/49239506:43
*** yuchenghui has quit IRC06:43
*** kukacz_ has joined #openstack-nova06:44
*** gjayavelu has quit IRC06:45
*** sridharg has joined #openstack-nova06:46
*** kukacz has quit IRC06:47
*** jamesdenton has joined #openstack-nova06:48
*** itlinux has quit IRC06:54
*** sshwarts has joined #openstack-nova06:56
*** lpetrut has quit IRC06:57
openstackgerritMerged openstack/nova master: [placement] Add api-ref for usages  https://review.openstack.org/48056306:58
*** abalutoiu has quit IRC06:59
*** sridharg has quit IRC06:59
*** sridharg has joined #openstack-nova07:00
openstackgerritMerged openstack/nova master: Raise NoValidHost if no allocation candidates  https://review.openstack.org/49149107:03
openstackgerritAlex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it  https://review.openstack.org/48037907:04
*** sridharg has quit IRC07:07
*** gjayavelu has joined #openstack-nova07:09
*** sridharg has joined #openstack-nova07:09
*** kukacz_ has quit IRC07:10
*** kukacz has joined #openstack-nova07:10
*** psachin has quit IRC07:12
*** gcb has joined #openstack-nova07:13
*** tesseract has joined #openstack-nova07:13
openstackgerritAlex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it  https://review.openstack.org/48037907:14
*** sahid has joined #openstack-nova07:14
alex_xuemm...finally07:14
*** sridharg has quit IRC07:17
*** sridharg has joined #openstack-nova07:19
openstackgerritjichenjc proposed openstack/nova master: Enhance support matrix document  https://review.openstack.org/48201307:20
*** markmc has quit IRC07:21
*** markus_z has joined #openstack-nova07:23
*** aarefiev_afk is now known as aarefiev07:29
*** mdnadeem has quit IRC07:42
*** xinliang has quit IRC07:48
*** ralonsoh has joined #openstack-nova07:48
*** alexchadin has joined #openstack-nova07:48
*** gjayavelu has quit IRC07:57
*** xinliang has joined #openstack-nova08:00
*** mdnadeem has joined #openstack-nova08:01
*** markus_z has quit IRC08:03
*** mdnadeem has quit IRC08:03
*** mdnadeem has joined #openstack-nova08:03
*** markus_z has joined #openstack-nova08:04
*** gjayavelu has joined #openstack-nova08:09
*** jpena|off has quit IRC08:12
openstackgerritBalazs Gibizer proposed openstack/nova master: replace chance with filter scheduler in func tests  https://review.openstack.org/49152908:13
*** mriedem has quit IRC08:16
*** moshele has joined #openstack-nova08:17
*** Drankis has joined #openstack-nova08:17
*** abalutoiu has joined #openstack-nova08:18
gibigood morning08:19
maciejjozefczykhello08:20
*** suresh12 has joined #openstack-nova08:21
*** xinliang has quit IRC08:23
*** gjayavelu has quit IRC08:24
*** lucas-afk is now known as lucasagomes08:25
*** derekh has joined #openstack-nova08:25
*** suresh12 has quit IRC08:25
*** thorst has joined #openstack-nova08:27
*** yamamoto has quit IRC08:27
*** yamamoto has joined #openstack-nova08:28
*** psachin has joined #openstack-nova08:29
*** yamamoto has quit IRC08:31
*** yamamoto has joined #openstack-nova08:32
*** hoonetorg has quit IRC08:32
*** thorst has quit IRC08:32
*** gjayavelu has joined #openstack-nova08:33
*** egonzalez has joined #openstack-nova08:36
*** jaosorior has quit IRC08:37
*** ekuris has quit IRC08:37
*** moshele has quit IRC08:37
*** avolkov has joined #openstack-nova08:38
*** tetsuro has quit IRC08:38
*** xinliang has joined #openstack-nova08:39
*** xinliang has quit IRC08:39
*** xinliang has joined #openstack-nova08:39
bauzasmorning Novaists08:40
*** jaosorior has joined #openstack-nova08:41
*** ociuhandu has quit IRC08:41
*** tosika has quit IRC08:46
*** hoonetorg has joined #openstack-nova08:49
*** tosika has joined #openstack-nova08:52
*** xinliang has quit IRC08:52
*** jaosorior has quit IRC08:54
*** ekuris has joined #openstack-nova08:54
ratailoralex_xu, you around ?09:05
alex_xuratailor: yea09:05
ratailoralex_xu, regarding this bug https://bugs.launchpad.net/nova/+bug/170926009:06
openstackLaunchpad bug 1709260 in OpenStack Compute (nova) "Addition of host to host-aggregate should be case -sensitive" [Low,Confirmed] - Assigned to Rajesh Tailor (ratailor)09:06
*** ralonsoh has quit IRC09:06
*** xinliang has joined #openstack-nova09:06
ratailoralex_xu, I have done something like this.. https://github.com/openstack/nova/commit/402b3abf990d08d2af8331079d36a92d84d84b80#diff-342854d9b83486a3db6189bc21be6266L102609:06
*** ralonsoh has joined #openstack-nova09:07
ratailoralex_xu, for host_mappings table in api_db.09:07
ratailoralex_xu, I was able to pass through this code block. https://github.com/openstack/nova/blob/master/nova/objects/host_mapping.py#L91-L9309:08
alex_xuratailor: not sure your first link is correct, that is pointed to a empty migration check09:09
alex_xuratailor: oh, sorry, I saw the entire patch now09:09
alex_xuratailor: I'm not sure we need to convert the table to case-sensitive09:10
ratailoralex_xu, but again I got stuck in this code block. https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L571-L57509:10
alex_xuat least, the hostname should be case-insensitive09:10
*** tetsuro has joined #openstack-nova09:11
ratailoralex_xu, in that case, do I need to look for alternative approach if possible ?09:11
alex_xuratailor: I guess it won't work after you change the collation of the table? since the in-memory copy of host<->aggregation mapping in the scheduler is still case-sensitive09:14
*** yuchenghui has joined #openstack-nova09:15
alex_xuratailor: ah, I see, you want to stop the api user input a hostname which differnet with the Db one09:16
ratailoralex_xu, I also applied the collation logic to service table in nova_cell1 db, but here, case-insensitive check is getting passed.09:16
alex_xubut the hostname should be case-insensitive? whatever the user input COMPUTE0.example.com or compute0.example.com, it should works09:16
*** gjayavelu has quit IRC09:16
ratailoralex_xu, yes, I wan't to prevent from adding the case-insensitive host to host-aggregate.09:17
ratailorbut aggregates store the hostname as COMPUTE0.example.com, whereas in db its compute0.example.com09:17
ratailoralex_xu, so that is what might be confusing scheduler.09:18
*** gongysh has quit IRC09:19
alex_xuratailor: there is a related bug https://bugs.launchpad.net/nova/+bug/153801109:20
openstackLaunchpad bug 1538011 in OpenStack Compute (nova) "Metadata keys are not case sensitive" [Medium,In progress] - Assigned to Augustina Ragwitz (auggy)09:20
*** yuchenghui has quit IRC09:20
ratailoralex_xu, Thanks! let me check it.09:20
alex_xuratailor: it would be great to check those two bugs, see the best way to fix09:21
ratailoralex_xu, sure.09:21
*** abhishekk has joined #openstack-nova09:22
alex_xuratailor: and there is a spec for that bug https://review.openstack.org/#/c/350843/, but didn't get merged09:22
*** gmann has quit IRC09:28
*** xinliang has quit IRC09:31
*** sree has quit IRC09:34
*** sree has joined #openstack-nova09:34
*** cdent has joined #openstack-nova09:35
*** jichen has quit IRC09:35
*** sambetts|afk is now known as sambetts09:36
openstackgerritVladyslav Drok proposed openstack/nova master: [placement] Add api-ref for RP usages  https://review.openstack.org/45010509:37
*** gongysh has joined #openstack-nova09:37
*** kukacz has quit IRC09:39
*** sree has quit IRC09:39
*** sree has joined #openstack-nova09:40
*** markvoelker has quit IRC09:42
*** xinliang has joined #openstack-nova09:44
*** kukacz has joined #openstack-nova09:47
*** sdague has joined #openstack-nova09:50
openstackgerritRodolfo Alonso Hernandez proposed openstack/nova master: Add datapath type information to OVS vif objects  https://review.openstack.org/47489209:51
*** jaosorior has joined #openstack-nova09:54
openstackgerritVladyslav Drok proposed openstack/nova master: [placement] Add api-ref for allocation_candidates  https://review.openstack.org/48111209:57
openstackgerritVladyslav Drok proposed openstack/nova master: [placement] Make placement_api_docs.py failing  https://review.openstack.org/48092409:57
alex_xucdent: bauzas this is ready for review https://review.openstack.org/#/q/topic:bug/170242010:01
cdentalex_xu: thanks alex, I’ve got that in my queue to look at soon. Was it hard to resolve?10:01
bauzasalex_xu: thanks, will look to it asap10:02
*** mvk_ has joined #openstack-nova10:02
bauzasjust have some internal paperwork to do10:02
alex_xucdent: after I pull some code into a sub-method, then thing gets a little better10:02
cdentah, yeah, good idea10:02
*** gcb has quit IRC10:04
maciejjozefczykcdent: please add also this to your queue https://review.openstack.org/#/c/491808/10:04
cdentmaciejjozefczyk: already there10:04
maciejjozefczykcdent: :) thx10:04
*** mvk has quit IRC10:05
*** kiennt has quit IRC10:07
*** phuongnh has quit IRC10:10
*** kukacz_ has joined #openstack-nova10:11
*** kukacz has quit IRC10:12
*** kukacz_ has quit IRC10:13
*** mdnadeem has quit IRC10:16
*** bkopilov has quit IRC10:19
*** yamamoto has quit IRC10:21
*** kukacz has joined #openstack-nova10:22
*** lpetrut has joined #openstack-nova10:22
*** kukacz has quit IRC10:22
*** yamamoto has joined #openstack-nova10:25
*** kukacz has joined #openstack-nova10:26
*** kukacz has quit IRC10:26
*** moshele has joined #openstack-nova10:26
*** yamamoto has quit IRC10:28
*** yamamoto has joined #openstack-nova10:28
*** thorst has joined #openstack-nova10:28
*** ekuris has quit IRC10:29
*** mdnadeem has joined #openstack-nova10:30
*** thorst has quit IRC10:33
*** yangyapeng has quit IRC10:33
*** dtantsur|afk is now known as dtantsur10:35
*** sree has quit IRC10:35
*** sree has joined #openstack-nova10:35
*** udesale has quit IRC10:36
*** kristian__ has quit IRC10:37
*** kristian__ has joined #openstack-nova10:38
*** zhurong has quit IRC10:39
*** tetsuro has quit IRC10:39
*** sree has quit IRC10:40
*** yamamoto has quit IRC10:41
openstackgerritSean Dague proposed openstack/nova master: Clarify that vlan feature means nova-network support  https://review.openstack.org/47855110:42
*** ekuris has joined #openstack-nova10:44
*** yamahata has quit IRC10:48
openstackgerritMerged openstack/nova master: Remove ram/disk sched filters from default list  https://review.openstack.org/49185410:54
*** shan has quit IRC10:54
openstackgerritMerged openstack/nova master: Mark Chance and Caching schedulers as deprecated  https://review.openstack.org/49221010:57
*** gongysh has quit IRC11:01
*** mkrcmari__ has joined #openstack-nova11:01
*** psachin has quit IRC11:03
*** mvk_ has quit IRC11:04
*** mdnadeem has quit IRC11:06
*** yamamoto has joined #openstack-nova11:09
openstackgerritBalazs Gibizer proposed openstack/nova master: replace chance with filter scheduler in func tests  https://review.openstack.org/49152911:10
*** mkucia has joined #openstack-nova11:11
*** mdnadeem has joined #openstack-nova11:12
openstackgerritMerged openstack/python-novaclient master: Remove substitutions for command error msg  https://review.openstack.org/49070511:12
*** mdnadeem has quit IRC11:19
*** yamamoto_ has joined #openstack-nova11:20
*** diga has quit IRC11:22
*** yamamoto has quit IRC11:24
*** yangyapeng has joined #openstack-nova11:25
openstackgerritMerged openstack/nova master: [placement] Avoid error log on 405 response  https://review.openstack.org/49002111:26
openstackgerritVladyslav Drok proposed openstack/nova master: [placement] Add api-ref for allocation_candidates  https://review.openstack.org/48111211:26
openstackgerritVladyslav Drok proposed openstack/nova master: [placement] Make placement_api_docs.py failing  https://review.openstack.org/48092411:27
openstackgerritMerged openstack/nova master: api-ref: fix security_groups response parameter in os-security-groups  https://review.openstack.org/48927411:27
*** mkrcmari__ has quit IRC11:28
openstackgerritMerged openstack/nova master: api-ref: requested security groups are not applied to pre-existing ports  https://review.openstack.org/48927511:28
*** mkrcmari__ has joined #openstack-nova11:29
*** kbaegis1 has quit IRC11:34
*** kbaegis has joined #openstack-nova11:34
openstackgerritOpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata  https://review.openstack.org/47709111:34
openstackgerritMerged openstack/nova master: Remove translation of log messages  https://review.openstack.org/46663711:39
maciejjozefczyksdague: Could you check this one? https://review.openstack.org/#/c/491808/ please?11:39
openstackgerritMerged openstack/nova master: remove mox from unit/virt/vmwareapi/test_driver_api.py  https://review.openstack.org/45212811:39
sdaguemaciejjozefczyk: how does this handle the synchronization problem that now computes might be trying to delete the same instances at the same time?11:41
sdaguepreviously, by being host scoped, this was a sharded problem11:41
*** lucasagomes is now known as lucas-hungry11:42
*** dave-mccowan has joined #openstack-nova11:43
*** alexchadin has quit IRC11:43
*** alexchadin has joined #openstack-nova11:44
*** alexchadin has quit IRC11:44
*** mkucia has quit IRC11:45
*** alexchadin has joined #openstack-nova11:45
*** alexchadin has quit IRC11:45
*** alexchadin has joined #openstack-nova11:45
*** alexchadin has quit IRC11:46
openstackgerritMerged openstack/nova master: imagebackend: cleanup constructor args to Rbd  https://review.openstack.org/49049911:46
*** alexchadin has joined #openstack-nova11:46
openstackgerritMerged openstack/nova master: Add policy granularity to the Flavors API  https://review.openstack.org/44928811:47
*** mtanino has joined #openstack-nova11:47
sdagueah, I see now11:49
*** kbaegis has quit IRC11:49
*** smatzek has joined #openstack-nova11:49
*** kbaegis has joined #openstack-nova11:49
maciejjozefczyksdague: The problem is about instance (which is deleted from nova side) is still running on compute A, but nova says that its deleted and it was on host B11:49
sdaguemaciejjozefczyk: yeh, I get the problem11:50
sdagueI was just trying to make sure that this didn't make it so that multiple computes were trying to delete the same instance11:50
sdaguebut _get_instances_on_driver should keep us sharded11:50
maciejjozefczykyes11:51
*** alexchadin has quit IRC11:51
maciejjozefczykI'm going to work on patch to rollback migration if deletion of instance will be triggered, in near future11:52
sdaguecool11:52
maciejjozefczykbut this fix for already 'lost' and 'working' zombiee instances i think should be in nova11:52
maciejjozefczykin my installation I have hundreds of them11:53
*** kbaegis has quit IRC11:53
*** kbaegis has joined #openstack-nova11:54
*** trinaths has quit IRC11:56
*** esberglu has joined #openstack-nova11:59
sdaguemaciejjozefczyk: yep, +2 on this fix11:59
*** tbachman has quit IRC11:59
*** _pewp_ has quit IRC12:00
maciejjozefczyksdague: thx12:00
*** thorst has joined #openstack-nova12:02
*** esberglu has quit IRC12:04
*** abhishekk has quit IRC12:04
*** alexchadin has joined #openstack-nova12:04
*** tuanluong has quit IRC12:06
*** kbaegis has quit IRC12:06
*** kbaegis has joined #openstack-nova12:06
openstackgerritAlex Xu proposed openstack/nova master: placement: avoid returning duplicated alloc_reqs when no sharing rp  https://review.openstack.org/49239512:06
openstackgerritAlex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it  https://review.openstack.org/48037912:06
alex_xucdent: thanks12:06
openstackgerritIlya Popov proposed openstack/nova master: Tests: Add cleanup of 'instances' directory  https://review.openstack.org/49158912:06
sdaguealex_xu: can I tempt you with doc patches? :)12:08
*** _pewp_ has joined #openstack-nova12:08
sdaguemostly I'd like to get the manuals stuff merged before I go on vacation next week12:08
*** alexchadin has quit IRC12:09
*** sree has joined #openstack-nova12:09
sdaguehttps://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/doc-migration12:09
*** vks1 has quit IRC12:09
alex_xusdague: yea, let me try12:10
*** jpena has joined #openstack-nova12:10
*** kbaegis1 has joined #openstack-nova12:12
*** kbaegis has quit IRC12:12
sdaguealex_xu: thank you12:13
*** sree has quit IRC12:13
*** kbaegis1 has quit IRC12:17
*** kbaegis has joined #openstack-nova12:17
*** xinliang has quit IRC12:20
openstackgerritChris Dent proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it  https://review.openstack.org/48037912:20
*** MVenesio has joined #openstack-nova12:20
*** sree has joined #openstack-nova12:20
*** mtanino has quit IRC12:25
*** edmondsw has joined #openstack-nova12:27
*** slaweq has quit IRC12:31
*** slaweq has joined #openstack-nova12:32
*** lucas-hungry is now known as lucasagomes12:32
*** xinliang has joined #openstack-nova12:33
*** moshele has quit IRC12:34
*** jamesdenton has quit IRC12:35
*** slaweq has quit IRC12:36
*** jamesdenton has joined #openstack-nova12:36
*** karthiks has quit IRC12:37
*** moshele has joined #openstack-nova12:40
*** ratailor has quit IRC12:41
*** gbarros has joined #openstack-nova12:42
*** alexchadin has joined #openstack-nova12:44
*** catintheroof has joined #openstack-nova12:45
*** lyan has joined #openstack-nova12:45
*** mriedem has joined #openstack-nova12:45
*** mkrcmari__ has quit IRC12:46
*** sbezverk has joined #openstack-nova12:48
*** moshele has quit IRC12:49
*** slaweq has joined #openstack-nova12:51
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Reconnect volumes and encryptors during a hard reboot  https://review.openstack.org/40038412:54
*** mvk has joined #openstack-nova12:54
openstackgerritLee Yarwood proposed openstack/nova master: compute: Detach volumes on _rebuild_default_impl failure  https://review.openstack.org/44210512:54
*** markmc has joined #openstack-nova12:55
*** mdnadeem has joined #openstack-nova12:57
*** jpena is now known as jpena|mtg12:59
cdentmriedem: i was partly thinking in terms of “don’t add more churn to zuul, now”12:59
*** esberglu has joined #openstack-nova13:00
*** moshele has joined #openstack-nova13:00
*** pchavva has joined #openstack-nova13:01
mriedemmmm zuul churn13:01
cdentfresh and tasty13:02
mriedemartom: you love evacuate right?13:02
mriedemgibi: you love finding bugs right?13:02
gibimriedem: I would put it I like finding them now than getting it from production :)13:05
artommriedem, in the same way I love, err...13:05
artomCrap, it's too early for witty wordplay13:06
artommriedem, anyways, what's up?13:06
mriedemmy main worry evacuate from an ocata compute messing this up https://review.openstack.org/#/c/491012/13:06
mriedemartom: i don't know how much you've followed this13:06
mriedembut basically the filter scheduler creates allocations in placement now,13:06
mriedemon both the source and dest computes during a move13:06
mriedemlike evacuate13:06
*** mingyu has joined #openstack-nova13:07
mriedemthe problem is that the resource tracker has no concept of other providers than itself, so during it's periodic accounting updates, it overwrites allocations in placement for any other provider13:07
mriedemthat patch ^ attempts to resolve that13:07
mriedemby using a minimum compute service version check - so once all of the computes are pike, it will stop doing it's local accounting13:08
mriedemand overwriting the stuff the scheduler created13:08
mriedemone of my worries is that we have an ocata compute that is forced-down, which takes it out of the service version check, but could still be running and trampling on things13:08
*** lucasxu has joined #openstack-nova13:09
mriedemi think it's probably a small window because if you are forcing a compute down and evacuating from it, (1) you're likely to stop that host at some point and (2) once the instances move, the dest compute should be accounting for them - and the scheduler will also do that13:09
artomI'm fuzzy on the resource tracker having no concept of other providers than itself13:10
artomI thought compute nodes were resource providers?13:11
mriedemthey are13:11
openstackgerritMerged openstack/nova master: Keep the code consistent  https://review.openstack.org/49030413:11
mriedemcdent: am i correct in saying that https://review.openstack.org/#/c/491012/ only applies to the periodic update_available_resource task?13:12
mriedemlooks like that's the only place that _update_usage_from_instances is called from13:12
openstackgerritMerged openstack/nova master: add description about key_name  https://review.openstack.org/48952513:12
* cdent refreshes himself13:12
*** bkopilov has joined #openstack-nova13:12
cdentmriedem: yes13:13
mriedemjust thinking that if that code thinks everything is pike and doesn't auto-heal,13:13
mriedemand it misses the instance moving from the ocata compute,13:13
mriedemthen we have to be sure that the rebuild_claim handles it13:13
*** moshele has quit IRC13:14
cdentmurgh13:14
gibimriedem: if a compute host is forced_down it should mean that that compute host is fenced13:14
gibimriedem: therefore it cannot tramp on allocations13:15
*** kylek3h has joined #openstack-nova13:15
*** baoli has joined #openstack-nova13:15
mriedemgibi: that doesn't mean the nova-compute service is not running on that host13:15
mriedemand if the service is running, it's update_available_resource periodic is running and could be overwriting allocations for the instance that's being evacuated13:16
mriedemthe forced_down flag doesn't do anything besides let the evacuate API proceed before the servicegroup api checkin says the compute is down13:16
openstackgerritMarkus Zoeller (markus_z) proposed openstack/nova master: docs: Explain the flow of the "serial console" feature  https://review.openstack.org/47618813:16
*** tbachman has joined #openstack-nova13:17
mriedemcdent: so i don't think rebuild_claim will update allocations at all13:17
sdaguemriedem: the contract with the user is forced_down means they killed that compute13:17
gibimriedem: if the compute is stull running but the admin set force-down then it is a user error13:17
gibiadmin should fence first then set forced-down flag13:17
mriedemrebuild_call calls _move_call which calls _update_usage_from_migration which calls _update_usage which doesn't call the report client13:17
sdagueit is only meant to be used if they've taken that system out of communication13:17
sdagueagree with gibi, that's admin error, and we've never attempted to correct for that13:18
mriedemthat's not documented anywhere https://developer.openstack.org/api-ref/compute/#update-forced-down13:18
sdaguemriedem: ok, well we should document it, that was the whole point of that feature13:18
*** ociuhandu has joined #openstack-nova13:18
sdaguefor HA systems to override nova when it knew better13:19
cdentmriedem: yeah, it looks like the allocation creation is all happening outside the various _claim* methods13:19
mriedemi had reported a bug related to docs on this at one point https://bugs.launchpad.net/nova/+bug/169187113:19
openstackLaunchpad bug 1691871 in OpenStack Compute (nova) "forced-down vs service disable is not documented well in the compute API reference" [Medium,Confirmed]13:19
gibimriedem: it is at least in the original spec https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/mark-host-down.html13:19
cdentwhich is somewhat weird13:19
cdentbut probably good given what we want eventually13:20
sdaguemriedem: sure but as you know, there aren't many idle folks looking for bugs to fix. Docs bugs mostly languish in our tracker.13:21
mriedemgibi: ok so the doctor project is supposed to get an alarm that something is wrong with the host, fence it and then force it down and start evacuating?13:21
mriedemsdague: i planned on fixing that docs gap myself13:21
mriedembut $time13:21
*** tbachman has quit IRC13:21
gibimriedem: I think so, yes13:21
sdaguemriedem: sure, that's fine13:22
mriedemwhat does 'fencing' mean in this case?13:22
bauzasmriedem: do we have a devstack change for https://review.openstack.org/#/c/491854/ ?13:22
*** tbachman has joined #openstack-nova13:22
cdentre: $time [t 1zqO]13:22
purplerbot<cdent> When do we start asking if the concept of PTL, as currently constructed, is sustainable? [2017-08-10 13:21:20.802315] [n 1zqO]13:22
mriedembauzas: no13:22
gibimriedem: power off, or cut the network13:22
gibimriedem: mostly power off via IPMI13:22
bauzasmriedem: okay, I'll write it13:22
sdaguewhat gibi said13:22
mriedemok powering off would be ideal13:22
sdaguemriedem: but, it could be lots of things. They could also decide to network fence the node13:23
mriedembut network is also good so the compute couldn't send changes to the report client (to conductor i mean)13:23
mriedemas long as it can't get to the placement api then that's sufficient13:23
cdentthat comment from bauzas reminds me of something I read while reading gerrit messages last night: did some test have to be changed so that it was running one of the filters we have no declared no longer default?13:23
gibimriedem: I'm not 100% sure doctor also starts the evacuation automatically after force_down13:23
mriedemgibi: ok but some project, maybe watcher13:23
gibimriedem: sure13:23
sdaguemriedem: I can take a spin on the api-ref, if you review my other doc patches :)13:23
gibimriedem: we have our on internally :)13:24
bauzaswhat's the problem with evacuation ?13:24
mriedemsdague: i have one change i need to get in today and that's https://review.openstack.org/#/c/491012/13:24
mriedembauzas: see https://review.openstack.org/#/c/491012/13:24
bauzasmriedem: ah, this...13:25
*** kbaegis1 has joined #openstack-nova13:27
mriedemcdent: ok left more comments in there13:29
mriedemi think i'm ok with this now, or as much as i can be,13:29
mriedemcdent: i'll fix the functional test quick and push that up, just that change13:29
*** kbaegis has quit IRC13:29
mriedemand then i'll probably spend some time this afternoon working on a functional test for evacuate13:29
*** kfarr has joined #openstack-nova13:29
mriedemwith the service version stuff13:30
mriedembauzas: devstack sets this13:30
mriedeminiset $NOVA_CONF filter_scheduler enabled_filters "RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,RamFilter,DiskFilter"13:30
*** mdnadeem has quit IRC13:30
*** jaypipes has joined #openstack-nova13:30
bauzasmriedem: I know13:30
mriedemnvm, that's only for the fake hypervisor13:30
bauzasmriedem: I'm just writing atm the change to remove this13:30
*** kbaegis has joined #openstack-nova13:31
mriedemwhich was only used in the large tests, which we don't run anymore13:31
bauzaswe run Core and Ram by default AFAIK13:31
*** kbaegis1 has quit IRC13:31
bauzasnot Disk tho13:31
mriedemFILTERS="RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter"13:31
mriedem^ is what we run by default13:32
bauzasso, we need to remove Ram *and* Diks13:32
bauzasI was just remembering we still run 2 legacy filters13:32
mriedemhttp://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n10413:32
mriedemok13:33
bauzasanyway, patch is on its way13:33
*** kbaegis has quit IRC13:33
mriedemsdague: oh right, that docs bug was because cfriesen was asking about this, see the last paragraph or two in that bug report,13:34
*** kbaegis has joined #openstack-nova13:34
*** kbaegis has quit IRC13:34
mriedemsomeone had forced the service down, upgraded everything else, and then when they tried to set forced_down=False, it failed with ServiceTooOld13:34
*** kbaegis has joined #openstack-nova13:34
mriedemwhich i think just caused some confusion in how that all works13:34
openstackgerritSean Dague proposed openstack/nova master: Update api-guide and api-ref to be clear about forced-down  https://review.openstack.org/49253313:35
mriedemthe ServiceTooOld is i'm pretty sure a 500 out of the PUT /os-services API too13:35
*** moshele has joined #openstack-nova13:35
sdaguemriedem: yeh, I don't know what the recovery path is on that all, but at least we can be clear on what the state transition into it should look like13:35
cdentdoes anyone have anything specific for me to do, or shall I continue chasing reviews and randomly testing things by hand?13:35
*** kbaegis1 has joined #openstack-nova13:36
*** kbaegis2 has joined #openstack-nova13:36
*** kbaegis has quit IRC13:36
edleafecdent: sudo make me a sandwich13:37
* cdent flies to new york13:37
*** kbaegis has joined #openstack-nova13:38
*** egonzalez has quit IRC13:38
bauzassdague: mriedem: https://review.openstack.org/#/c/492537/13:38
bauzas^ devstack change removing legacy filters FTW13:39
sdaguebauzas: ok, cool13:39
sdaguebauzas: is there a reason we override those in the first place?13:39
bauzassdague: correct me if I'm wrong but we don't need to modify grenade since it uses default config optoions ?13:39
sdaguewould we just use defaults?13:39
*** moshele has quit IRC13:39
mriedemsdague: because devstack had the default + SameHost + DifferentHost13:39
bauzaswhat mriedem said13:39
mriedemtempest has tests for SameHost/DifferentHost filters13:39
sdagueah, cool, good reason13:39
mriedemwhich aren't defaults13:40
bauzaswe could tho make += "SameHost"13:40
bauzassince it's a listopt13:40
*** kbaegis has quit IRC13:40
mriedemin bash?13:40
*** kbaegis3 has joined #openstack-nova13:40
*** kbaegis2 has quit IRC13:40
*** kbaegis3 has quit IRC13:40
*** kbaegis1 has quit IRC13:40
sdaguebauzas: that doesn't work in setting nova.conf13:40
bauzasah, right13:40
bauzasI usually do this directly :)13:40
bauzassdague: anyway, my question still remains wrt grenade13:41
bauzassdague: do we need to s// something when we upgrade the node ?13:41
sdaguebauzas: grenade should be fine as long as those things didn't get deleted13:41
bauzasin terms of nova.conf -ism13:41
sdagueit will use the ocata config in pike13:41
bauzasyeah, my thoughts13:41
bauzasnot super then13:41
bauzaswe don't remove those filters because CachingScheduler still requires them13:42
bauzasbut in theory, you don't need to run those filters since Ocata13:42
sdagueright as long as there is normal deprecation cycles you don't need to do anything in grenade13:42
bauzasthat's just we wanted to be gentle in Ocata and not ask operators to change their conf when upgrading from Newton13:42
sdagueas long as you remove the deprecated setting of things in devstack in the release where you deprecate them13:42
bauzasokay, I think it's reasonable then13:43
sdagueit just naturally rolls through it all13:43
bauzask13:43
bauzasmriedem: looping over https://etherpad.openstack.org/p/nova-pike-release-candidate-todo13:44
sdaguemriedem: I think putting the entire recovery path inside the API ref for forced-down is not idea - https://review.openstack.org/#/c/492533/1/api-ref/source/os-services.inc13:44
bauzasmriedem: AFAICT, the top prio for reviewing is then https://review.openstack.org/#/c/491012/ ?13:45
sdaguethere probably needs to be a specific HA recovery document for the whole thing13:45
sdaguethis is just "don't use this unless you really are sure"13:45
bauzasit's a force thing13:45
bauzas:p13:45
bauzas'force' is meaningful in Linux terminology :)13:45
mriedembauzas: yes13:46
bauzasdon't expect good things to happen magically if you use a force flag13:46
mriedemsdague: i didn't say to put the entire recovery path in there...13:46
mriedemjust 'never come back' is not really correct,13:46
mriedemi think maybe you mean 'never come back in it's current state'13:46
mriedemor something13:46
openstackgerritMatt Riedemann proposed openstack/nova master: Resource tracker compatibility with Ocata and Pike  https://review.openstack.org/49101213:47
mriedemcdent: jaypipes: ^ fixed the functional test failure13:47
bauzasyou can "force enable" back13:47
cdentmriedem: roger13:47
bauzasbut honestly, that forced-down thing is just a hack because of our SG API13:47
bauzasI'm not sure we should be explicit here13:48
*** egonzalez has joined #openstack-nova13:51
openstackgerritEd Leafe proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration  https://review.openstack.org/48795413:51
openstackgerritSean Dague proposed openstack/nova master: Update api-guide and api-ref to be clear about forced-down  https://review.openstack.org/49253313:53
*** marst_ has joined #openstack-nova13:54
*** alexchadin has quit IRC13:55
*** takedakn has joined #openstack-nova13:56
*** takashin has joined #openstack-nova13:58
mriedemnova meeting in 1 minute13:59
bauzassdague: tried to help with defining why we have a force-down flag https://review.openstack.org/#/c/492533/213:59
bauzasseeing that being used by operators scares me13:59
bauzasit was never the intent13:59
*** takedakn has quit IRC13:59
bauzasand like the bug report mentions, we have the disabling thing14:00
*** ioggstream has joined #openstack-nova14:00
ioggstreamhi@all14:00
*** gouthamr has joined #openstack-nova14:00
sdaguebauzas: right, so I think we have to be careful about putting too much into api-ref because it's supposed to be a reference14:00
sdaguewe probably need an HA guide here that explains the whole thing14:00
ioggstreamabout OS:: Nova::ServerGroup policy, how will anti-affinity (default) policy work if I have more vms than compute nodes?14:00
bauzassdague: well the problem is that our API can be consumed by both end-users and automation tools14:00
mriedembauzas: the api-ref could link to a more detailed guide14:01
sdaguebauzas: I'll be honest, I'm not going to spend all day redrafting this docs patch. I think what I have up there is an improvement, if folks want to take it over and write it instead, I'm good with that14:01
bauzassdague: to be frank, I do wonder if we should just cut that flag from the nova CLI14:01
mriedemsdague: there is, or was, a maintenance guide14:01
bauzassdague: okay, I'll push a rev then14:01
mriedemplanned and unplanned14:01
sdaguemriedem: sure14:01
mriedemi'd put it in there, but it might have been in the now defunct ops guide14:01
mriedemwould have to dig14:01
bauzasmriedem: sdague: what are you thinking of just removing that method from our nova CLI ?14:01
bauzassince only scripts should call it14:01
mriedemmeeting time14:02
sdaguebauzas: people might have manually downed their nodes as well14:02
sdaguebauzas: I think it's fine to be in the nova cli14:02
sdaguethat's an admin tool14:02
*** gongysh has joined #openstack-nova14:02
sdaguethey just need to realize what they are doing14:02
bauzasagreed14:02
bauzassdague: I'll try a new rev14:02
*** gongysh has quit IRC14:03
sdaguebauzas: I think your assumption that no one should ever call this manually is wrong14:03
jianghuawjianghua14:03
sdaguethe important point is they met preconditions14:04
sdaguethe service is fenced14:04
sdaguewe don't care how they met those14:04
sdaguebut they are expressing to nova that they did14:04
bauzassdague: we have a service group API for that14:04
sdagueit might have been a tool14:04
bauzassdague: the only usecase I heard of was that the SG API was lacking of functionality and either lagging or totally missing the host being down14:05
bauzassdague: so, eventually, the SG API would meet those preconds14:05
bauzasthat's just because Nova isn't intented to be a Nagios system, we allow other tools to fence the host for us14:05
sdaguebauzas: that's not good enough if I need it now14:05
*** crushil has joined #openstack-nova14:05
bauzasfrom our behalf, I mean14:06
*** awaugama has joined #openstack-nova14:06
dtantsuredleafe: hi! my last attempt to use resource classes in the CI ended up with RamFilter removing the nodes14:06
*** moshele has joined #openstack-nova14:06
dtantsurI wonder if I'm missing something.. I thought I disabled requesting RAM/disk/CPU14:06
bauzasdtantsur: Nova by default was still running those filters until yesterday14:07
dtantsuroh14:08
edleafedtantsur: probably the request was for 1 of the resource class, along with the disk/ram/cpu in the flavor14:08
dtantsuredleafe: I assume I'm removing the request for disk/ram/cpu from flavor14:08
dtantsurlemme get a link14:08
dtantsuredleafe: https://review.openstack.org/#/c/476968/13/devstack/lib/ironic@188914:09
dtantsurbauzas: this one, right? https://github.com/openstack/nova/commit/2fe96819c24eff5a9493a6559f3e8d5b4624a8c914:09
edleafedtantsur: ok, then that should work14:09
bauzasdtantsur: correct14:09
edleafedtantsur: do you have the call to placement anywhere in logs?14:10
dtantsurthanks, I'll see how it looks nowadays14:10
bauzasdtantsur: oh wait14:10
bauzasdtantsur: Ironic is special-case IIRC14:10
bauzasdtantsur: you folks have your own czay filters list :)14:10
bauzascrazy14:10
bauzasin nova14:10
dtantsuryeah, we did change something around it.. lemme check14:10
* bauzas is facepalming14:11
dtantsurhttps://review.openstack.org/#/c/490459/14:11
dtantsurthis is why I'm seeing the RamFilter, not the ExactRamFilter14:12
dtantsuroh, and by the way. should we kill Exact filters with fire14:12
dtantsur(well, I meant deprecate)14:12
dtantsur?14:12
edleafedtantsur: those were there only to support the pretense that an ironic node was a vm14:13
edleafedtantsur: so yeah, kill 'em!14:13
dtantsuredleafe: wanna get a deprecation patch or should I?14:13
edleafedtantsur: we should also deprecate the separate ironic filter options, no?14:13
dtantsuredleafe: yep14:14
*** jpena|mtg is now known as jpena|off14:14
edleafedtantsur: I may have time, but not much14:14
edleafedtantsur: if you want to start and post a WIP, I can pick it up14:14
dtantsurENOTMUCHTIME is a common error code nowadays14:14
dtantsursure, will do14:14
* dtantsur git clone nova14:15
*** sree has quit IRC14:18
*** slaweq has quit IRC14:19
bauzasdtantsur: those Exact* filters could just be treated like the other legacy filters14:20
dtantsurand how do you treat legacy filters? :)14:20
bauzasdtantsur: being removed from the list of filters to run by default, but still in tree for upgrade concerns14:20
bauzaswhen you upgrade from Ocata, you certainly don't want to update nova.conf for that14:21
dtantsurthey're not on by default, there is an option to enable them..14:21
openstackgerritBalazs Gibizer proposed openstack/nova master: test server evacuation with placement  https://review.openstack.org/49254814:21
*** sshwarts has quit IRC14:21
dtantsurfolks, what's your next version? (to use with deprecated_since)14:22
bauzaswell, in theory you could use CachingScheduler with IroncHostManager I guess14:22
bauzasin that case, you'd still require Exact* filters14:22
gibimriedem: just out of curiosity create a Pike -> Pike evac test and it seems the allocation on the source host has never cleaned up https://review.openstack.org/#/c/492548/14:22
bauzasyet another call for deprecating the other scheduler driver we have in tree14:22
dtantsurbauzas: I've never heard of people using it, but yeah. For every crazy feature there are people to try it in production..14:23
ioggstreamdoes anybody knows if soft-anti-affinity may be enabled in newton ?14:23
bauzasioggstream: IIRC, yes14:23
bauzasioggstream: https://blueprints.launchpad.net/nova/+spec/soft-affinity-for-server-group is Mitaka complete14:24
ioggstreambauzas: by default it doesn't work but I saw that mitaka has an ERRATA14:24
mriedemgibi: because the periodic task doesn't cleanup allocations anymore14:24
mriedemgibi: it assumes the scheduler has everything correct14:24
mriedemand the source node is 'down'14:24
gibimriedem: but not even the source compute cleans up?14:24
gibimriedem: after started up again?14:24
mriedemif it's down we probably don't care about it14:24
mriedemoh, we'll talk after the meeting14:24
gibimriedem: sure14:24
*** sridharg has quit IRC14:26
mriedembut yeah the update_available_resource code in pike now does'nt overwrite the allocatoins14:26
mriedemper that change14:26
mriedemso that's why the source compute won't cleanup once it comes back up14:26
mriedemit should remove though....14:26
mriedemgibi: this one https://review.openstack.org/#/c/491850/14:27
*** lbragstad has quit IRC14:28
gibimriedem: my test https://review.openstack.org/#/c/492548/ is top of https://review.openstack.org/#/c/491850/ and I still see the allocation on the source host14:28
*** tbachman has quit IRC14:34
*** takashin has left #openstack-nova14:35
*** takedakn has joined #openstack-nova14:35
mriedemgibi: i think that's probably due to https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@104714:35
gibimriedem: checking the debug log...14:36
mriedemgibi: but we should get into https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@114514:36
mriedem_remove_deleted_instances_allocations14:36
gibimriedem: I see the debug log you pointed at14:37
mriedemoh it could be https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@118714:38
mriedemif (instance.host == cn.host and14:38
mriedem                    instance.node == cn.hypervisor_hostname):14:38
mriedemwe continue there14:38
mriedemor if instance.host != cn.host:14:38
mriedemwe also continue there14:38
*** moshele has quit IRC14:38
mriedemseems we should check to see if the instance is in self.tracked_migrations14:38
gibiI can insert some extra log to confirm14:38
*** priteau has joined #openstack-nova14:39
gibiahh there is logs already14:40
gibiit is the instance.host == cn.host where we continue14:41
jaypipesmriedem: ty14:43
jaypipesmriedem: fyi, kinda vacationing today...14:43
jaypipesmriedem: will work on my patches thouhg14:43
mriedemjaypipes: don't think you have anything to work on14:45
mriedemexcept follow ups for additional testing and whatnot14:45
*** yamahata has joined #openstack-nova14:46
*** cleong has joined #openstack-nova14:46
mriedemgibi: ok so self.tracked_migrations probably won't help after we restart the compute service since that dict will probably be empty14:46
mriedemgibi: i left some comments in your test change,14:46
mriedemwe could maybe do some allocation cleanup in https://github.com/openstack/nova/blob/9a66d039a14afd591f4a3b6e655580aeeed17d29/nova/compute/manager.py#L64914:46
jaypipesmriedem: yeah14:46
mriedemgibi: this is similar to https://bugs.launchpad.net/nova/+bug/1679750 where we don't delete the allocations for the instance on the compute host during a 'local delete' in the API14:47
openstackLaunchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,In progress]14:47
mriedemwhen we can't get to the compute host b/c it's down14:47
sdaguemriedem: ok, stepping away from the computer for a bit...14:47
*** vks1 has joined #openstack-nova14:48
bauzasmriedem: https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1134 isn't a cellv2-compatible call since it would be an upcall, no ?14:48
gibimriedem: should I file a bug or we will squeez the fix into https://review.openstack.org/#/c/491012 ?14:48
mriedemgibi: i think it's a new bug14:49
mriedemgibi: similar to bug 167975014:49
openstackbug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,In progress] https://launchpad.net/bugs/167975014:49
mriedembut slightly different14:49
gibimriedem: OK, I will report it14:49
mriedembug 1679750 is about how we don't remove the allocations when we delete the instance and the compute is down14:49
mriedemin your case, the instance isn't deleted, it's evacuated, and we just don't cleanup the allocations from the original host when it comes back up14:49
mriedembauzas: services are per-cell14:50
mriedemso it's not an upcall14:50
*** PsionTheory has joined #openstack-nova14:50
gibimriedem: thank for the pointers14:50
mriedembauzas: this is why we needed to expose the service uuids out of the API in 2.5314:50
mriedemto identify nova-compute 1 from nova-compute 1 in cell A and cell B14:51
bauzasmriedem: so it only verifies if all the local-cell computes are having that version, right?14:51
bauzasmriedem: since we're in a local compute14:52
mriedemyes14:52
mriedemmoves are also restricted to the cell that the instance already lives in14:52
*** armax has joined #openstack-nova14:53
ioggstreambauzas: https://bugzilla.redhat.com/show_bug.cgi?id=1447798 thx++14:53
openstackbugzilla.redhat.com bug 1447798 in python-openstackclient "soft-anti-affinity is not accepted by python-openstackclient but is with python-novaclient" [High,New] - Assigned to jruzicka14:53
ioggstreamfyi14:53
mriedemhttps://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@116614:53
mriedemioggstream: i don't think osc is handling nova microversions14:53
bauzasmriedem: correct, so it's probably not a problem14:53
mriedemand the soft affinity policy rules were in a microversion14:53
mriedembauzas: yes of all the problems we could have with this series of changes, that's one i'm not worried about :)14:54
bauzasfortunately because we don't support multi-cell moves14:54
bauzasokay14:54
bauzasgtk :)14:54
mriedemand likely won't for a long time14:54
ioggstreammriedem: you mean osc challenges the server for the supported options?14:54
mriedemioggstream: no, i mean i believe osc only ever requests compute API things with 2.114:55
*** sbezverk has quit IRC14:55
mriedemand the soft affinity group policy was added in 2.15 https://docs.openstack.org/nova/latest/reference/api-microversion-history#id1314:55
mriedemso if you're using osc and try to create a server group with policy 'soft-anti-affinity' using microversion 2.1, it's going to fail with a 40014:56
mriedemsaying it doesn't know what that policy is14:56
gibimriedem: I filed https://bugs.launchpad.net/nova/+bug/170990214:56
openstackLaunchpad bug 1709902 in OpenStack Compute (nova) "source host allocation not cleaned up in placement after evacuation" [Undecided,New]14:56
ioggstreammriedem:  ok, I didn't know the microversion stuff, let me read about it14:56
mriedemioggstream: you might be able to pass an --openstack-compute-version or something option to osc cli, idk14:56
*** Swami has joined #openstack-nova14:58
openstackgerritEric Fried proposed openstack/nova master: Use ksa adapter for placement conf & requests  https://review.openstack.org/49224714:58
*** cfriesen has joined #openstack-nova14:58
ioggstreammriedem: I'll check and let you know, thx++14:59
openstackgerritDmitry Tantsur proposed openstack/nova master: Deprecate bare metal filters  https://review.openstack.org/49256314:59
*** lbragstad has joined #openstack-nova14:59
dtantsuredleafe: ^^^14:59
* dtantsur did not expect nova unit tests to run 15 minutes, wow :D14:59
*** sree has joined #openstack-nova15:00
edleafedtantsur: get a faster machine! :)15:00
*** felipemonteiro has joined #openstack-nova15:00
bauzasmriedem: do you agree with my comment on https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1073 and if so, possibly me passing a new revision ?15:03
mriedemdtantsur: heh yeah you need more VCPUs15:05
dtantsurLOOL15:05
*** jaypipes has quit IRC15:05
bauzast470s FTW15:05
*** jaypipes has joined #openstack-nova15:05
openstackgerritEric Fried proposed openstack/nova master: Use ksa adapter for placement conf & requests  https://review.openstack.org/49224715:06
openstackgerritEric Fried proposed openstack/nova master: nova.utils.get_ksa_adapter()  https://review.openstack.org/48813715:06
* bauzas just need to migrate from his old t440s15:06
mriedembauzas: given the amount of shit we log continually all the time, like lazy-loading pci request and pci devices for every claim, then i'd say leave that for a follow up patch15:06
openstackgerritEric Fried proposed openstack/nova master: Get auth from context for glance endpoint  https://review.openstack.org/49005715:06
bauzasmriedem: ok, firing it up15:06
bauzasand then I'll +W that one15:06
*** derekh has quit IRC15:06
*** suresh12 has joined #openstack-nova15:07
*** sapcc-bot1 has joined #openstack-nova15:07
bauzas(story-telling, if you remember my current laptop, you'll understand how much I'll enjoy to have a new Enter key w/ my new laptop :D )15:07
*** yamamoto_ has quit IRC15:08
dtantsurbauzas: I have t460s. too old for nova, I guess :D15:08
mriedembauzas: if you're going to +W anyway, just do it now15:08
mriedemwith the gate time as it is15:09
bauzasmriedem: sure15:09
mriedemi don't really want to be doing rc1 at midnight15:09
mriedemmidnight is matty's time15:09
*** sapcc-bot has quit IRC15:09
dtantsuroh, rc1.. does this ironic-related stuff have a chance to get  in?15:09
bauzasmriedem: yeah I should consider relocating to Hawaii15:10
bauzasgiven the TZ15:10
*** yamamoto has joined #openstack-nova15:10
bauzasor French Polynesia, that could help too15:10
*** slaweq has joined #openstack-nova15:10
openstackgerritBalazs Gibizer proposed openstack/nova master: test server evacuation with placement  https://review.openstack.org/49254815:10
mriedemgibi: per your bug, i'm curious why TestLocalDeleteAllocations isn't failing then15:11
*** yamamoto has quit IRC15:11
mriedemoh i know why15:11
mriedemit's deleted15:11
mriedemand when the RT gets InstanceNotFound, it deletes the allocations15:11
*** hareesh has quit IRC15:12
jaypipesis it just me or is the gate getting better...15:12
mriedemdtantsur: i've been leaning on dansmith for that change15:12
mriedemdtantsur: but he's out the rest of the week15:12
mriedemdtantsur: so it's likely rc2 at this point15:12
dtantsuryeah, it's fine, as long as it gets into Pike at all15:12
*** burt has joined #openstack-nova15:12
mriedemjaypipes: except for the slow nodes and timeouts, it's not bad15:12
*** kfarr has quit IRC15:12
mriedemgibi: i wonder why https://review.openstack.org/#/c/492548/ keeps saying it can't merge?15:14
mriedembut don't rebase the entire series...15:14
gibimriedem: I think it is because I adding the test at the end of the test_server.py15:15
jaypipesmriedem, bauzas, sdague: should be simple review: https://review.openstack.org/#/c/491822/15:15
gibimriedem: but I can try locally15:15
*** Oku_OS is now known as Oku_OS-away15:15
gibimriedem: yes, the evac patch conflicts on the file ending15:16
gibimriedem: master has test_resize_not_enough_resource at the end15:16
gibimriedem: but Jay's branch doesn't contain that15:16
gibimriedem: I will have no time today to do the evac fix in the init_host so if this is important somebody can take over. But tomorrow I can continue15:17
openstackgerritSylvain Bauza proposed openstack/nova master: Make compute log less verbose with allocs autocorrection  https://review.openstack.org/49257115:17
bauzasmriedem: HTH ^15:18
openstackgerritDmitry Tantsur proposed openstack/nova master: Deprecate bare metal filters  https://review.openstack.org/49256315:18
*** itlinux has joined #openstack-nova15:19
dtantsurbauzas, mriedem, easy win for your review stats ^^ (like if you needed more, heh)15:19
*** suresh12 has quit IRC15:20
*** annegentle has joined #openstack-nova15:21
gibimriedem, dansmith: I think this is ready to go in, Jay was +2 already but I needed to rebase: https://review.openstack.org/#/c/491529/15:21
jaypipesgibi: hey, there's an urgent request for your assistance here: http://lists.openstack.org/pipermail/openstack/2017-August/045339.html15:22
* jaypipes runs away15:22
*** namnh has joined #openstack-nova15:22
gibijaypipes: seems like s spam :)15:26
*** annegentle has quit IRC15:26
mriedembauzas: sdague: melwitt: i triaged https://bugs.launchpad.net/nova/+bug/170959415:27
openstackLaunchpad bug 1709594 in OpenStack Compute (nova) "live-migration without '--block-migrate" failed with "No sql_connection parameter is established"" [High,Triaged]15:27
mriedemit's a superconductor issue,15:27
mriedemthe xenapi driver tries to get host aggregates during a live migration,15:27
mriedemand the aggregates are in the API DB now,15:27
jaypipesgibi: was just joking you.15:27
mriedemso this is an upcall attempt from the compute service, in a cell with no access to the api db15:27
*** sdake_ is now known as sdake15:27
gibijaypipes: ohh. OK. You got me :)15:28
*** annegentle has joined #openstack-nova15:28
*** thingee_ has joined #openstack-nova15:31
mriedemgibi: i think we could deal with the source allocation cleanup bug as a backport15:34
*** takedakn has quit IRC15:37
gibimriedem: ack15:39
*** moshele has joined #openstack-nova15:40
mriedemit's only a problem unless the compute comes back online15:41
mriedems/unless/until/15:41
*** moshele has quit IRC15:43
ioggstreammriedem: +1 [stack@c1f-ops-dirp00 openshift-on-openstack]$ o server group create deleteme-sg-3 --policy soft-anti-affinity --os-compute-api-version 2.1515:44
sdaguemriedem: good call15:44
*** gbarros has quit IRC15:45
mriedemioggstream: cool15:46
*** chyka has joined #openstack-nova15:46
*** tbachman has joined #openstack-nova15:49
openstackgerritSpencer Yu proposed openstack/python-novaclient master: Nova client should retry with Retry-After value  https://review.openstack.org/44776615:49
mriedemioggstream: going to update that bugzilla?15:49
mriedembtw, it's nice to see my ibm red hat bugzilla account is still active15:50
mriedemand has permissions to ibm bugs15:50
*** jpena|off is now known as jpena15:50
ioggstreammriedem: yes, I'll do it. I've an rh account: I'll try with it.15:51
ioggstreammriedem: my issue is now that heat doesn't seem to support soft-anti-affinity15:51
bauzasdtantsur: edleafe: I'm confused by the ironic upgrade path (see https://review.openstack.org/#/c/492563/2)15:52
bauzasdtantsur: edleafe: we could potentially have old ironic nodes to schedule with a pike scheduler15:52
dtantsurbauzas: well, we're not removing anything, are we?15:53
bauzasdtantsur: sure, but you send a signal with those deprecations15:53
bauzasI just don't want operators to freak out15:53
dtantsurwell, in Queens the old style of scheduling is not going to be possible15:53
dtantsurwe have to send some signal about that coming15:53
bauzasso it's just a wording issue15:54
dtantsurre upgrade https://docs.openstack.org/ironic/latest/admin/upgrade-guide.html#upgrading-from-ocata-to-pike15:54
dtantsurwe should change s/recommended/required/, I'm going to have another update anyway15:54
bauzasbut the point is, if you deprecate in Pike, that means you accept operators to not use those opts by the Pike timeframe, which could be a bit of concern if they roll out upgrades15:54
bauzasdtantsur: I briefly looked at your relnotes too15:55
bauzasdtantsur: and haven't found any clear ask for providing resource classes for nodes15:55
bauzas(talking of https://docs.openstack.org/releasenotes/ironic/unreleased.html15:55
dtantsurbauzas: this is coming as part of https://review.openstack.org/#/c/491773/15:55
bauzasah cool15:55
bauzasdtantsur: so I guess some linkage between relnotes could be appreciated15:56
bauzaslike, ironic has to be upgraded before compute nodes obviously15:56
bauzasbut ironic can be upgraded after placement and scheduler, right15:56
*** Apoorva has joined #openstack-nova15:56
dtantsurI guess so, yeah15:56
bauzasand before you run a fresh pike nova-compute, you have to update your ironic cloud to set the resource classes15:57
bauzasthat is the upgrade ordering I care, since operators would query for15:57
dtantsurright, we can do it while still running Ocata15:57
dtantsurshould this patch be finished today? I have meetings, then I'd prefer to bail out (I'm in EU time)15:58
*** aarefiev is now known as aarefiev_afk16:00
bauzasdtantsur: you mean the notes ?16:00
ildikovmriedem: quick meeting if you're available16:00
dtantsurbauzas: this nova patch16:00
bauzasdtantsur: well, mriedem will cut the rc1 tag tonight for us (as well, I'm CEST) so that would mean those notes would require a backport if we want them in the pike tree16:01
*** annegentle has quit IRC16:01
dtantsurthis sounds like "today"..16:01
*** suresh12 has joined #openstack-nova16:01
dtantsuredleafe: is it possible you take on fixing release note wording for ^^^?16:01
bauzasdtantsur: ideally, I'd have appreciated to see https://review.openstack.org/#/c/491773/ landed first, but I guess we need to send them concurrently to the gate16:02
*** dklyle is now known as david-lyle16:02
dtantsurbauzas: ironic is not branched today, so it may wait a bit, I think16:03
edleafebauzas: dtantsur: I'll review after API-WG meeting16:03
dtantsurcool16:03
bauzasdtantsur: oh right, not the same cadence than us16:03
bauzaswell, I can push a new rev16:03
bauzasedleafe: ^16:04
dtantsurif you don't mind!16:04
edleafeI never mind16:04
edleafe:)16:04
mriedempushing what now?16:06
mriedemwe already have stuff in the nova release notes that say ironic gets upgraded before nova16:06
*** lyan has quit IRC16:06
mriedemfor some unrelated features, like boot from volume16:06
mriedemwe don't need to say that 5 times16:06
bauzasmriedem: context is https://review.openstack.org/#/c/492563/16:07
edleafemriedem: it's the sixth time that will sink in16:10
*** suresh12 has quit IRC16:10
*** sahid has quit IRC16:11
*** yamamoto has joined #openstack-nova16:12
*** crushil has quit IRC16:12
dtantsuredleafe, bauzas: updated ironic part (docs and reno)16:13
dtantsurwording suggestions are welcome, my wording can be awful sometimes16:13
mriedemthis doesn't have to be done in pike does it?16:14
mriedemseems like too many moving parts16:14
dtantsurmriedem: these filters won't work in Queens, no?16:14
mriedemidk16:14
mriedemi just,16:14
* dtantsur +100 to "too many moving parts"16:14
mriedemi can only have my head wrapped around 10 different RC1 stop ship omfg blocking issues at one time16:14
mriedemand this is #1116:15
edleafeunless we drag it out further, the plan is that in queens, ironic will only be scheduled using custom resource classes16:15
* dtantsur feels sorry for mriedem, and shares his pain with his own PTL struggles16:15
edleafemriedem: this doesn't change functionality, right? Just marks things as deprecated16:15
*** Drankis has quit IRC16:15
efriedIf the placement API gives me a 500, and I'm using mod_wsgi, where do I find the log that's gonna tell me WHY?  (devstack)16:16
mriedemefried: the placement-api logs16:16
sdagueefried: uwsgi?16:16
efriedmod-wsgi16:16
efriedI know, I know, it's deprecated.  Is that my actual problem?16:16
sdagueummm... I thought we deleted that bit16:17
*** yamamoto has quit IRC16:17
sdagueefried: mod-wsgi is going to be an apache log16:17
efriedHeh.  If it's deleted, not just deprecated, I guess that'd do it.16:17
efriedOkay, where's that guy?16:17
sdagueefried: we may not have deleted it16:17
sdagueefried: whereever your apache logs16:17
efriedI mean, our CI is running with it and (mostly) succeeding.16:17
sdague /var/log/apache/placement*16:17
sdagueon an ubuntu like thing16:17
sdagueit will be a different place on a redhat like thing16:18
openstackgerritMerged openstack/nova master: remove log message with potential stale info  https://review.openstack.org/49224216:18
*** kristian__ has quit IRC16:18
efriednyaha! Thanks sdague16:18
*** Swami has quit IRC16:19
*** mriedem is now known as mriedem_away16:20
openstackgerritSylvain Bauza proposed openstack/nova master: Deprecate bare metal filters  https://review.openstack.org/49256316:21
bauzasdtantsur: edleafe: tried some rewording ^16:21
dtantsurthanks, looking16:21
efriedThat placement log has no entries since two weeks ago :(16:21
efried(and the 500 happened two minutes ago)16:21
openstackgerritMerged openstack/nova master: placement: refactor healing of allocations in RT  https://review.openstack.org/49185016:23
*** pcaruana has quit IRC16:25
sdagueefried: it is hard for me to debug remotely, if there is a box I can connect to I can look16:26
sdagueI have IBM vpn access so hopefully a path16:26
*** psachin has joined #openstack-nova16:27
*** itlinux has quit IRC16:27
*** lucasxu has quit IRC16:27
*** gjayavelu has joined #openstack-nova16:28
cdentefried: there may still be a non wsgi log where the other nova logs are. another place to check are any other apache log you can find (grep for ‘resource_providers’). where things end up gets weird16:29
*** s1061123 has quit IRC16:29
cdentand where a 500 ends up with mod_wsgi in the first place (even outside openstack) can be odd16:29
cdentsometimes it will be the central apache error.log16:29
*** namnh has quit IRC16:30
efriedcdent Yeah, I'm finding it in the horizon_access.log (which is odd - I'm not using horizon at all).  But just the request/response headers, not the error trace.16:30
cdentis a horizon_error.log? or error.log?16:30
efriedThere's an error.log16:31
*** saphi has joined #openstack-nova16:31
efriedit has nothing for the last 6h16:31
*** lucasagomes is now known as lucas-afk16:32
cdentefried: I missed the earlier discussion, what time does this code come from?16:32
efriedI'm using latest master nova.  nova-powervm driver plugged in, but wouldn't think that's in the code path.16:32
cdentwhat installed placement?16:33
*** hongbin has joined #openstack-nova16:33
sdaguecdent: I'm going to try to get on the box and poke to see if we can iterate through it16:33
*** rcernin has quit IRC16:33
cdentIf it’s nova master there should still be a non apache log for errors, based on the work you did last summer sdague16:34
cdent_unless_ the 500 is in the calling of the wsgi app (rather than within the wsgi app)16:35
cdentI’d be curious to hear what it turns out to be16:35
efriedesberglu We're still using mod-wsgi in our CI, right?16:40
bauzasfolks, bailing out for a few hours, \o16:41
efriedsdague cdent Ya know, my nova is current, but my other stacky stuff is a couple weeks old.  Could that do it?16:41
efriedI mean, regardless, I would like to be able to figure out where to look for the real cause of a generic 500...16:42
*** mingyu has quit IRC16:43
cdentefried: that’s why I asked about what’s doing your deployment (devstack, whatever)16:43
efriedoh, sorry, I misunderstood the question then.  Yeah, I did a devstack a couple weeks ago, been dinking around with the nova code since then.  But hadn't yet had occasion to get this far (been just working conf stuff, so stopping early).16:44
*** lyan has joined #openstack-nova16:44
cdenta devstack from two weeks ago that is still mod_wsgi surprises me16:45
cdentso it may be that your error is in journalctl16:45
cdent journalctl --unit devstack@placement-api16:46
efriedcdent That journalctl unit doesn't exist.16:48
cdentI’ll leave you in sdague’s good hands then. You seem to have some weird. :)16:49
efriedcdent The placement log is in /var/log/apache216:49
efriedcdent But it doesn't have anything in it.  Leading me to believe we aren't getting that far.16:49
cdentoh yeah, I forgot that you have that file16:49
cdentlook in all the other files in /var/log/apache2 for whatever is the central error file16:50
sdagueefried: yeh, it's super weird, it looks like placement isn't running16:50
*** Swami has joined #openstack-nova16:50
efriedsdague I'm not real savvy here, but I thought placement was (as of yet) part of n-cpu.16:50
sdagueefried: it's a wsgi script16:51
sdagueso, actually it looks like it's there under apache, but not running for some reason16:51
sdagueor running but not connected16:51
cdentor died16:51
efriedShould I try restarting httpd?16:52
sdagueefried: wait a sec before doing that16:52
efriedsuresure.16:52
sdagueI want to figure out if there is any other postmortem here16:52
sdaguehttp://paste.openstack.org/show/618073/ - apache status16:52
*** slaweq has quit IRC16:53
efriedFWIW, the stack itself would have been done a couple weeks ago, and the only service I've been mucking with is n-cpu.16:53
*** marst_ has quit IRC16:54
*** marst_ has joined #openstack-nova16:54
sdaguehmmm... well all the access for all services is in the horizon access log16:55
sdaguewhich, is surely a bug16:55
sdaguehowever, it's a bread trail16:55
efriedsdague If it's unique to mod-wsgi, it might not be a bug that'll get attention.16:55
sdagueyeh16:55
sdaguewe also don't enable horizon in normal testing16:56
cdentthat’s not a mod-wsgi bug, that’s an apache misconfiguration, and is part of the driver to switch to uwsgi in devstack16:56
cdentwhich ever section ends up listening on port 80 (or 443) with a log configuration will end up getting lots of things that listen on that port16:56
efriedI can try to restack with uwsgi, but last time I tried, it didn't work.  (Which is why we're still using mod-wsgi in our CI.)16:57
*** yamahata has quit IRC16:57
efried(esberglu We either need to figure out the uwsgi thing or add /var/log/apache2/* to our CI log dump.)16:57
sdagueah, you know, I think the crux of it is after the uwsgi cut over we started dropping ports16:58
cdentapache makes the assumption that if you are onthe same port, you’re using the same log16:58
sdagueyeh16:58
sdagueyeh, this is part of why mod-wsgi is such a pain on the dev side16:59
cdentand when horizon is enabled it is consuming port 80 in a weird way16:59
sdagueefried: so my starting point was "nova-manage upgrade check"16:59
sdaguewhich went all 500 stack tracy16:59
sdagueyes, placement is returning 500s, however I can't find them landing in a log anywhere17:00
*** dtantsur is now known as dtantsur|afk17:00
sdaguehttp://paste.openstack.org/show/618074/ horizon_access.log17:01
*** baoli has quit IRC17:01
efriedyeah, just what I was seeing.17:01
*** ralonsoh has quit IRC17:01
efriedMine were coming from the compute service trying to suss out the resource providers17:01
efriedsdague Well, if you're game to help me debug why uwsgi stack is failing, I can try restacking thusly.17:02
sdagueefried: yeh, I would do that17:02
efriedThat would be a big help in general.17:02
sdaguemake sure you do a ./clean.sh17:02
efriedOkay, rippinit.17:02
*** igordcard is now known as igordc17:02
sdagueyeh, I don't think anything else useful can come from this install17:02
efriedI did a lot of useful stuff wrt service catalog lookups.17:03
efriedBut I was breaking right at driver init to muck around in pdb17:03
efriedBut now I'm working on converting the placement API over (https://review.openstack.org/#/c/492247/) so I kinda wanted to see it working :)17:04
*** baoli has joined #openstack-nova17:04
efried...and you can see from ^^ that our CI isn't having any trouble with it.17:04
sdague:)17:05
efried(just checked the compute logs to be sure - as if we could pass without placement being happy in the first place - and there's no 500s)17:05
*** ociuhandu has quit IRC17:06
*** kfarr has joined #openstack-nova17:06
*** abalutoiu has quit IRC17:06
sdagueyeh, it's in a weird intermediate state I think, there is a reason why mod_wsgi is something i wanted to remove from the dev/test stack. It just hits a bunch of odd apachisms that don't fit well with our other assumptions17:06
efriedsdague To be clear, all I need to do to switch is remove WSGI_MODE=mod_wsgi from my local.conf?17:07
sdagueyep17:07
efriedk.  Refreshing other project clones...17:07
*** sree has quit IRC17:08
efriedaaand stacking...17:10
melwittmriedem_away: I was wondering if we need to document that for multi-cell with nova-network, upcalls from compute are required for quota checks. I was thinking we might not have to because IIRC we're not supporting multi-cell + nova-network17:10
melwittI think single cell would be okay because there's no isolation from the API DB there17:11
efriedMeanwhile, any idea what glorious magic makes a .txt.gz journalctl log show up in the browser with color codes translated?17:13
*** marst_ has quit IRC17:13
*** Apoorva_ has joined #openstack-nova17:14
*** mvk has quit IRC17:14
*** annegentle has joined #openstack-nova17:14
*** slaweq has joined #openstack-nova17:16
*** itlinux has joined #openstack-nova17:17
*** Apoorva has quit IRC17:18
*** lpetrut has quit IRC17:19
*** rajathagasthya has joined #openstack-nova17:20
sdagueefried: magic yet to be written17:21
*** mingyu has joined #openstack-nova17:21
efriedsdague But it works17:22
efriedAt least in my browser17:22
sdagueefried: interesting17:22
efriedExcept that for powervm logs, it doesn't *quite* work.17:22
efriedsdague What, it doesn't do that for you?17:22
sdagueefried: oh, it's not color codes translated17:23
sdagueyou mean the coloring of - http://logs.openstack.org/81/488381/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/2b8b331/logs/screen-c-api.txt.gz ?17:23
efriedsdague Yup.17:23
sdaguehttps://github.com/openstack-infra/os-loganalyze17:23
*** suresh12 has joined #openstack-nova17:24
sdagueis a wsgi filter for the logs17:24
efried...that runs under the auspices of apached?17:24
sdagueyeh, log access is proxied through that17:24
efried...and would need to exist on the log server.17:24
sdagueyes17:24
sdagueit processes in real time17:24
efriedSo ours must be outdated or something.  Cool.  esberglu ^^ !17:24
sdaguethere is a regex on the files it will attempt to do it with17:25
efriedsdague Cause for us, it shows up HTMLified, with the log filters, but no color translation.17:25
efriedhttp://184.172.12.213/manual/rest_logs/logs/n-cpu.txt.gz17:25
efried(This isn't in production yet, cause we've been naming 'em .log.gz, but we're converting over)17:25
sdaguehttps://github.com/openstack-infra/os-loganalyze/blob/master/os_loganalyze/filter.py#L24-L3117:25
sdagueyeh, you'd need to setup this code on apache17:25
*** baoli has quit IRC17:26
sdaguehttps://github.com/openstack-infra/puppet-openstackci/blob/master/templates/logs.vhost.erb#L106 and friends to see how you'd set it up on apache17:27
sdagueit requires a few rewrite rules17:27
*** baoli has joined #openstack-nova17:27
*** baoli has quit IRC17:27
efriedsdague Is that not the same script that does the filtering capability?17:27
*** jpena is now known as jpena|mtg17:30
efriedsdague Okay that stack failed, on image create.17:30
*** markus_z has quit IRC17:30
efriedapache error log shows an ENOSPC in /tmp.  I've got .8GB in there; is it really trying to copy the image through /tmp??17:31
*** yamahata has joined #openstack-nova17:31
*** hemna_ has joined #openstack-nova17:32
*** saphi has quit IRC17:34
*** baoli has joined #openstack-nova17:37
*** krtaylor has quit IRC17:38
*** vks1 has quit IRC17:40
sdagueefried: it might be17:40
efriedsdague Restacking, gonna watch /tmp...17:41
sdaguemtreinish: ^^^17:41
sdagueit might be specific on the uwsgi glance17:41
sdagueand, if so, that could be an interesting issue17:41
* efried googles how to grow /tmp...17:41
*** ioggstream has quit IRC17:42
sdagueyou all are booting a real image, not a cirros, right?17:42
sdagueI could totally imagine that issue to have slipped through because cirros17:42
cdentexcellent point, sdague, at least one interation of the uwsgiification of glance needed to do some writes to disk, but I don’t know if that version landed, and if it did where on disk17:43
*** annegentle has quit IRC17:44
efriedsdague Yeah, it's a real image - though it's pretty small  (2GB)17:44
sdagueefried: yeh... but there was all manner of tricksyness to get uwsgi glance to work17:45
sdagueand I can totally imagine full write of the upload to tmp was part of that17:45
efriedHow... precious.17:45
*** slaweq_ has joined #openstack-nova17:46
*** mriedem_away is now known as mriedem17:48
mriedemmelwitt: hmm, nova-network only starts in a cells v1 config17:49
mriedemso yeah, i don't think we probably have to worry about that for cells v2 multi-cell17:49
*** slaweq has quit IRC17:50
*** ociuhandu has joined #openstack-nova17:50
*** gouthamr_ has joined #openstack-nova17:50
*** gouthamr has quit IRC17:51
*** lpetrut has joined #openstack-nova17:51
efriedsdague Yup, thar she blows.  During that image create command, /tmp fills quickly, then stack bails.17:56
mriedemdude17:57
mriedemthar she blars17:57
mriedembauzas: jaypipes: we want this in for pike too https://review.openstack.org/#/c/492234/17:58
mriedemmelwitt: ^17:58
*** lpetrut has quit IRC17:58
jaypipesmriedem: done17:58
*** SamYaple has joined #openstack-nova17:59
mriedemthanks17:59
SamYapleany chance I will see this backport merge https://review.openstack.org/#/c/486272/ ? im not sure what novas backport policies are18:00
mriedemmelwitt: can you also help go through the set of docs changes leading up to https://review.openstack.org/#/c/491853/ ? that's the target for the docs migration for rc118:00
mriedemSamYaple: maybe18:01
mriedemthere should be a push for stable branch reviews once we are clear of pike rc118:01
SamYaplemriedem: ok thanks. i can't run ocata in my setup without this patch and im not a huge fan of carrying patches which im currently doing with it18:03
sdagueSamYaple: +2 on that.18:03
SamYaplenice sdague! thanks18:04
mtreinishsdague, cdent: we ended up doing mod_proxy with http traffic on a local port18:04
mtreinishglance runs uwsgi in http mode and apache just forwards the traffic to the local uwsgi daemon18:04
sdaguemtreinish: yeh, file upload seems to create a full temp file though18:04
sdaguewhich, is actually kind of an issue18:05
*** imadjerra has joined #openstack-nova18:05
cdentyeah, that will be because of a tweak to mod proxy’s behavior18:05
sdaguefrom nova bugs, 120GB snapshots are a thing18:05
sdagueno one is planning N * 120GB free in /tmp18:05
*** imadjerra has quit IRC18:07
cdentI guess that means the question becomes: is anyone planning to duplicate devstack’s way of running glance?18:08
mtreinishsdague: I'm not sure how the uwsgi change would effect things there. We ran basically the same config before just instead of uwsgi running glance it was glance standalone. The apache config was more or less the same for tls18:08
mtreinishunless uwsgi is using /tmp as a cache18:08
cdentit’s mod proxy18:08
*** Apoorva_ has quit IRC18:08
* cdent finds reference18:08
sdaguecdent: that's an impact of the tls termination?18:08
*** Apoorva has joined #openstack-nova18:09
efriedmtreinish Is there a uwsgi config somewhere that I can tell to use something other than /tmp ?18:09
efriedI'm trying again with TMPDIR=/var/tmp (where I have some more space) - but that'll only affect stuff running directly under the auspices of stack.sh18:10
efriedmm, unless I set it in /etc/environment.  Let's see if that flies...18:11
*** annegentle has joined #openstack-nova18:11
cdentsdague, mtreinish, efried : https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#request-bodies18:11
cdentit’s the sendcl thing18:11
cdentthat was added to the proxy config when glance switched to uwsg18:12
mtreinishcdent: the mod_proxy config for glance should not be using sendcl18:12
cdentdid you take it back out?18:12
mtreinishthat was one of the paths I was investigating and it didn't work for various reasons (including this issue growing tmp space)18:13
melwittmriedem: sure thing18:13
mtreinishcdent: https://github.com/openstack-dev/devstack/blob/master/lib/apache#L333 is the path for glance18:14
cdentit’s currently on line 276 of lib/apache18:14
mtreinishcdent: right that's the path that's not glance18:14
cdentis setenv iglobal to apache ?18:14
cdentIt looks like it may be18:15
*** Apoorva_ has joined #openstack-nova18:15
sdagueso... if you hit uwsgi directly without the tls proxy, this wouldn't be an issue?18:16
cdentsdague: unclear, because it may be the mod_proxy_uwsgi also attends to it?18:17
cdentI’m not certain this is the problem, but I do think sendcl is a factor, or something like it18:18
cdenthard to be clear what uwsgi does in the face of file uploads18:18
openstackgerritMerged openstack/nova master: Imported Translations from Zanata  https://review.openstack.org/47709118:18
sdagueyeh, it's supposed to be chunk encoded upload18:18
sdaguebut bouncing through proxies... things get tricky18:18
*** Apoorva has quit IRC18:19
mtreinishis there a running system with this issue? because it's easy to confirm if apache is passing chunked data to uwsgi with a tcpump on the local port18:19
cdentif you are sending chunked but demanding content-lenght (from whatever directive) a write will be required18:19
mriedemsdague: i've got a couple of comments in here https://review.openstack.org/#/c/49181518:20
mtreinishcdent: if setenv is globally scoped across different site files we probaly just need to create a virtualhost for the glance path to isolate it from the other sites18:20
mriedemmelwitt: take a look at my comments if you update18:20
mtreinishcdent: fwiw, the glance path doesn't use mod_proxy_uwsgi18:21
cdentI know, but it does use mod_proxy doesn’t it? At least that’s what I thought I heard you say?18:21
mtreinishyes it does18:21
melwittmriedem: phew, in the nick of time18:21
melwittlooks like sdague is on it already18:22
sdaguemriedem: ok, so an inline change to that is going to be 4 - 6 hours to get check results back18:22
sdaguehow late you staying up for the branch?18:22
sdaguevs. fix later18:22
mriedemit's just docs, shouldn't run all jobs18:23
sdagueit doesn't18:23
sdaguethe check queue is starved out18:23
*** liverpooler has joined #openstack-nova18:23
melwitthm, yeah I didn't think about going through the check queue again. though I thought it would be fast for docs18:23
sdagueit's hours before you get a slot18:24
mriedemare you already doing a dump of review comment follow up stuff at the end of the series?18:24
mriedemlike https://review.openstack.org/#/c/491517/18:24
sdagueI rebased that and stuck it on the end18:24
sdagueas it was in merge conflict, and I didn't want to generate more conflicts18:25
mriedemalright i'm fine with following up at the end of the series18:25
sdagueok, can do18:25
mriedemmelwitt: you want the +W ?18:25
*** rajathagasthya has quit IRC18:25
openstackgerritMerged openstack/nova master: Add format_dom for PCI device addresses  https://review.openstack.org/49182218:25
melwittmriedem: hell yes I do18:25
sdagueI also tried to set expectations here - https://review.openstack.org/#/c/492124 how we really do need to think about docs reviews a bit differently than code reviews, especially because they are merge magnets18:26
sdagueyeh, it looks like we're at 3.5 hours hold to get check nodes18:27
cdentmtreinish: you added the sendcl, do you remember why? https://review.openstack.org/#/c/459451/18:27
melwitt+Warthogged18:27
cdentit looks like maybe you accidentally left it in18:27
cdentafter choosing a different approach18:27
sdaguemelwitt: thanks18:27
sdagueok, I'll work on follow up fix patch at top of stack18:28
* cdent likes warthogs18:28
mtreinishcdent: my intent there was specifically for the non-glance case. If nova or others gets a chunked request we don't want it to blow up an invalid content length error18:28
*** liverpooler has quit IRC18:29
mtreinishso I wanted sendcl for the mod_proxy_uwsgi case so it would handle those requests properly18:29
mtreinishnone of the projects using that path should be taking large chunked uploads over the api (it's only nova, cinder, and placement)18:29
mriedemoh no18:30
mriedemsdague: you've just started two major new ML thread wars https://review.openstack.org/#/c/491834/3/doc/source/index.rst18:31
mriedemsdague: crisis the first: you omitted CONTAINERS!18:31
mriedemcrisis the second: OpenSack is a new competing community of cloud hacky sack based software that will kill us all!18:31
sdaguemriedem: the only in tree container support is untested libvirt lxc, right?18:32
mriedemvirtuozzo18:32
*** smatzek has quit IRC18:32
sdagueoh, right18:32
mriedemand lxc18:32
mriedembut yes lxc is likely a floating turd18:32
*** krtaylor has joined #openstack-nova18:32
mriedemtried getting CI running for that several releases ago18:32
sdagueyeh, I debated it, someone else can tweak that18:32
sdaguemy goals, make this page navigable18:32
mriedem"limited container support" is probably ok18:33
mriedemheh, nova-net! https://review.openstack.org/#/c/492105/2/doc/source/figures/nova-conf-xen-flat.png,unified18:34
*** tosika has quit IRC18:35
*** rcernin has joined #openstack-nova18:35
*** smatzek has joined #openstack-nova18:36
sdagueyeh, there is all kinds of fun that came over with that import18:36
openstackgerritIlya Popov proposed openstack/nova master: Tests: Add cleanup of 'instances' directory  https://review.openstack.org/49158918:38
*** rajathagasthya has joined #openstack-nova18:38
mriedemyeah, i finally know about the EMC system enabler https://review.openstack.org/#/c/492105/2/doc/source/figures/emc-enabler.png,unified18:40
*** gbarros has joined #openstack-nova18:41
* cdent waves18:41
*** cdent has quit IRC18:41
mtreinishcdent: looks like you're right the file is coming from mod_proxy18:42
mtreinishI'll write up a patch to create an isolated virtual host for the glance apache endpoint and hopefully that should isolate the sendcl param18:43
openstackgerritSean Dague proposed openstack/nova master: clean ups from I6815958b2533d462a2e5d27e7be57440d9f4f40a  https://review.openstack.org/49264518:43
*** Apoorva_ has quit IRC18:43
sdaguemriedem / melwitt ^^^ ok, go to town ripping on that18:43
sdagueI'll iterate until we get it good enough18:43
*** Apoorva has joined #openstack-nova18:44
*** Sukhdev has joined #openstack-nova18:44
melwittcool, thanks18:45
*** kfarr has quit IRC18:49
*** psachin has quit IRC18:52
*** lpetrut has joined #openstack-nova18:55
sdaguemelwitt: on https://review.openstack.org/#/c/491853 realize that's just the import plus reference corrections to make it build18:55
melwittsdague: ah, I see that now. I thought the Cells v1 vs v2 warning was new but I see now that it's not18:56
sdagueyeh18:56
sdaguelet us say... the existing docs are a mixed bag. It's probably better to have them in, but I think we're going to need a doc sprint early in the cycle to clean up a bunch of this18:56
melwittsdague: I think the only other thing not imported is the database config section that's not mentioned as skipped in the commit message. is that one supposed to be skipped?18:57
sdagueyeh, it turned into a really goofy thing18:57
*** awaugama has quit IRC18:57
melwittk18:57
sdaguelet me pull up a couple of links18:57
sdaguethe database doc in question was this - https://github.com/openstack/openstack-manuals/blob/6f9fc171800e8a435011f38cd4558e900884ce86/doc/config-reference/source/compute/database-connections.rst#L218:58
sdaguewhich besides being kind of inaccurate with the "any sqla" bit, it was also pretty vapid in what it said18:59
*** rcernin has quit IRC18:59
sdagueand was mostly a link to https://github.com/openstack/openstack-manuals/blob/6f9fc171800e8a435011f38cd4558e900884ce86/doc/config-reference/source/tables/common-database.rst#L1118:59
melwittcool. just making sure it wasn't missed. I'm kind of just skimming this18:59
sdagueyeh18:59
sdaguethe common tables things was kind of a bear19:00
sdaguesteven spent a day or two fixing references, and I spent another day after him doing that. There was a lot more common normalization in the openstack-manuals repo that had to be denormalized for all of this.19:01
*** adisky__ has quit IRC19:02
*** efried is now known as efried_afk19:02
sdagueok, afk for a bit19:03
*** jamesdenton has quit IRC19:03
mtreinishefried_afk: https://review.openstack.org/#/c/492651/19:03
*** openstackgerrit has quit IRC19:03
mtreinishsdague: ^^^ that should fix the glance thing19:03
*** jamesdenton has joined #openstack-nova19:03
mriedemsdague: melwitt: ok several comments in the config reference one https://review.openstack.org/#/c/491853/619:07
mriedemlooks like some are duplicates of mel's19:07
melwittmriedem: yeah, turns out this is a straight forklift and I didn't realize that at first19:07
melwittso there's lots of latent issues in it19:07
mriedemheh,19:08
mriedem"The nova-conductor service is the only service that writes to the database."19:08
mriedem"If nova-conductor is not used, entries to the database are mostly written by the nova-scheduler service, although all services must be able to update entries in the database."19:08
mriedemffs19:08
melwittnoice19:08
mriedemonly conductor can write to the database, except everything else can and must be able to also, including your dog19:08
melwitto local conductor, I barely remember you19:08
mriedemmr ruffles19:08
melwittheh19:09
mriedemsome of the other stuff i pointed out is just plain wrong, like "kvm is the default compute driver"19:09
*** sree has joined #openstack-nova19:09
mriedem"to change compute drivers, configure the [libvirt] section"19:09
mriedemhello libvirt, please to be giving me vmware19:09
mriedemthanks19:09
melwittas you do19:10
mriedemsdague: i think we should rename that cells.rst to cellsv1.rst at some point19:11
mriedemsince it's totally cellsv1 content19:11
mriedemmikal: this is for you https://review.openstack.org/#/c/491853/6/doc/source/admin/configuration/samples/rootwrap.conf.rst@519:11
*** baoli has quit IRC19:13
*** sree has quit IRC19:13
*** openstackgerrit has joined #openstack-nova19:19
openstackgerritEd Leafe proposed openstack/nova master: Deprecate bare metal filters  https://review.openstack.org/49256319:19
mnaserjaypipes whenever you have a minute re https://review.openstack.org/#/c/491561/ -- i was thinking about this and realized that all_weighers will now include this which might mean that both ramweigher and rampercentageweigher would both be used (and maybe that might cause some weird scheduling and a change in default behaviour) .. any thoughts (more than welcome anyone else input too)19:20
jaypipesmnaser: hmm...19:21
*** nicolasbock has quit IRC19:21
mriedemok the other rc1 things are approved, that just leaves https://review.openstack.org/#/c/487954/19:21
jaypipesmnaser: yeah, that's a good point. keep it in the separate file then...19:21
jaypipesmnaser: mb add a note to the top saying "this is a separate file because..."19:22
mnaserjaypipes i think that even if its a separate file it'll automatically get included with all_weighers :X19:22
mnaserlet me find the code for that again19:22
mnaserif i remember right, its not hard coded, used some loadable stuff to autoload all of the stuff in the folder19:22
*** gbarros has quit IRC19:23
mnaserhttps://github.com/openstack/nova/blob/master/nova/scheduler/weights/__init__.py#L46-L48 => https://github.com/openstack/nova/blob/master/nova/loadables.py#L78-L9619:23
jaypipesmnaser: ugh...19:24
mnaserthe reason this test didnt catch it because it doesnt check for an exact match but "if it contains" https://github.com/openstack/nova/blob/master/nova/tests/unit/scheduler/weights/test_weights_hosts.py19:24
mnaseryeah.. its not pretty19:24
jaypipesmnaser: well, even if it's included in the list of filters, that doesn't necessarily mean it's enabled, right?19:25
mnaserjaypipes this is weigher and afaik there is no enabled or disabled19:25
mnaserjaypipes ouch.. "but wait, there's more" - https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L389-L39419:26
mnaserwe couldnt even drop it from all_weighers lol19:26
*** efried_afk is now known as efried19:27
mriedemweighers are all enabled by default19:27
efriedmtreinish Sweet, trying...19:27
mriedemmnaser: you can't set weight_classes ?19:28
mriedemdefault=["nova.scheduler.weights.all_weighers"],19:28
mriedemmnaser: you should be able to specify the specific list of weighers if you want to exclude some19:28
*** kristian__ has joined #openstack-nova19:28
jaypipesmnaser: yeah, you'd essentially have to list all the other weight classes, minus RamWeigher, and add in RamPercentageWeigher. ugh.19:28
mnasermriedem - i could, but the idea is when introducing this change -- https://review.openstack.org/#/c/491561/ -- that weigher will become included with the list of default all_weighers19:28
mnaseri have them manually listed and it works fine for me right now19:29
mriedemso you want an excluded_weighers option19:29
mriedemhow about, included_by_conditional_on_virt_backend_weighers? :)19:29
mriedems/by/but./19:29
mnaseri dont need it, i can manually list it, but if that change merges, the behaviour of all_weighers will change .. which is probably not okay?19:29
jaypipesmriedem: no, he needs to change the behaviour of the existing RamWeigher if, say, a [scheduler]ram_weight_use_percentage configuration option is set...19:29
mriedemok, so you need a config option to change the behavior of the existing ram weigher,19:30
jaypipesmnaser: right, it's not ok, since both Ram weighers will produce different weights.19:30
mriedemand default it to behave the old way19:30
mnaserthats a good idea19:30
jaypipesyeah19:30
mnaseri can do that19:30
mriedem$5 please19:30
jaypipesmnaser: thanks man. :)19:30
mnaserjaypipes np19:30
mnasermriedem i'll start a tab19:31
mriedemspeaking of,19:31
mriedemcraiglist person is coming over in an hour to buy a $10 kids wagon, emailed me asking if i have change for a $2019:31
mriedemthis will be fun19:31
mriedem"i do, if you're ok with singles and $1 coins"19:31
mriedem"here is a bag of nickels"19:32
melwittmriedem: anything else need review for rc1 that I can look at?19:32
mriedemmelwitt: hell yes you can https://review.openstack.org/#/c/487954/19:32
melwittsweet19:32
*** slaweq has joined #openstack-nova19:33
*** kristian__ has quit IRC19:33
*** xyang1 has joined #openstack-nova19:35
*** slaweq_ has quit IRC19:36
*** kristian__ has joined #openstack-nova19:38
*** sbezverk has joined #openstack-nova19:39
*** baoli has joined #openstack-nova19:40
sdaguemriedem: I agree. I honestly think we should probably rename all the cells v1 stuff to legacy cells actually19:41
*** baoli has quit IRC19:41
sdagueespecially when dealing with bugs, people don't really understand the v1 v2 thing19:41
sdaguelegacy cells vs. cells would probably make things clearer19:41
* bauzas is back around19:42
sdaguemriedem: can we get ironic folks voting on https://review.openstack.org/#/c/48795419:43
bauzasdo I need to chime in somewhere?19:44
sdaguebauzas: I think 487954 is the last patch required for the branch19:45
*** baoli has joined #openstack-nova19:45
*** sambetts is now known as sambetts|afk19:45
mriedemsdague: they were ok with one of the earlier patch sets19:47
mriedemdtantsur|afk: is ok with it i think19:47
bauzassdague: a bit latey for the gate, nope?19:49
bauzasunless mriedem is a night owl19:49
mriedembauzas: i'll be up19:50
*** kristian__ has quit IRC19:50
sdaguewe apparently got more rax nodes too a couple hours ago as well, so we're actually starting to burn down the wait queue19:51
*** mingyu has quit IRC19:51
sdaguewhich is good19:51
sdagueat least its not still growing :)19:51
bauzascool :)19:51
*** lpetrut has quit IRC19:54
*** pchavva has quit IRC19:57
*** kristian__ has joined #openstack-nova20:00
*** smatzek has quit IRC20:07
*** baoli has quit IRC20:07
*** awaugama has joined #openstack-nova20:15
*** jaypipes has quit IRC20:20
*** Sukhdev has quit IRC20:21
*** gjayavelu has quit IRC20:22
mriedemmelwitt: are you still going through https://review.openstack.org/#/c/487954/ ?20:22
melwittmriedem: yes, going through the tests, trying to understand what "dupe" means and how the first dupe test is different from the second20:22
*** akhodos has joined #openstack-nova20:23
*** hamzy has quit IRC20:24
*** amotoki has quit IRC20:24
sdagueoof, citycloud lon1 fail again20:26
edleafemelwitt: the first verifies that only one call to save is made, since one instance already has been migrated20:26
mnaser2017-08-10 20:24:19.406877 | MAGNUM_GUEST_IMAGE_URL='\''http://mirror.mtl01.internap.openstack.org/fedora/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2'\''20:26
mnaseroops, wrong chan20:27
edleafethe second verifies that if refresh_cache is called a second time, the instances are not saved again20:27
melwittthanks edleafe20:27
edleafeit has some old code in that second call - I'm correcting that from _refresh_hash_ring() to _refresh_cache()20:27
mriedemsdague: per gmann's doc change, i don't see where this stable api doc is even linked from http://docs-draft.openstack.org/26/489926/9/check/gate-nova-docs-ubuntu-xenial/24e8cc4//doc/build/html/reference/stable-api.html20:31
mriedemlike https://developer.openstack.org/api-guide/compute/20:32
mriedemor http://docs-draft.openstack.org/26/489926/9/check/gate-nova-docs-ubuntu-xenial/24e8cc4//doc/build/html/contributor/index.html#the-nova-api20:32
sdaguereference/index20:33
sdaguehttp://docs-draft.openstack.org/26/489926/9/check/gate-nova-docs-ubuntu-xenial/24e8cc4//doc/build/html/reference/index.html20:33
mriedemoh20:34
mriedemsheesh20:34
sdaguefrom the reorged page it would be index -> Technical Reference Deep Dives20:34
sdague-> Nova Stable REST API20:34
mriedemyeah...20:34
mriedemthat was a hunt20:34
sdagueyeh, we're going to need a slice of time to figure out what the right information architecture is for all this stuff is at PTG20:35
sdaguethis first stage grouping was just that, a first stage grouping, without creating *more* 404s20:35
*** sdake is now known as k2so20:37
sdagueso, the contributor and reference consolidations were first, and I made sub pages for them. But the admin subpage was already claimed as the main url for the admin guide, so I inlined the equivalent user / admin chunks into the top index page20:37
sdagueit might be good to unwind all of that for the contributor / reference pages20:37
*** kristian__ has quit IRC20:38
sdagueso you get there without the intermediate hop20:38
*** yusef has joined #openstack-nova20:38
sdagueI also think we might be able to use refs in the toctree and get something more sane out of all of that20:39
*** jamesdenton has quit IRC20:39
openstackgerritEd Leafe proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration  https://review.openstack.org/48795420:41
efriedmtreinish With your fix I did indeed hit https://bugs.launchpad.net/glance/+bug/170385620:41
openstackLaunchpad bug 1703856 in Glance "502 Bad gateway error on image-create" [High,Confirmed]20:41
*** jamesdenton has joined #openstack-nova20:43
*** cleong has quit IRC20:43
efriedmtreinish ...and so did the gate.20:45
*** annegentle has quit IRC20:45
edleafemriedem: melwitt: ^^ addressed your comments (and your terrible taste in indentation)20:46
melwitthah20:47
*** esberglu has quit IRC20:47
melwitt+2, looks cool to me20:48
melwittthanks for changing those test names, makes it a lot clearer to someone not in-the-know20:49
*** _pewp_ has quit IRC20:50
*** gouthamr_ has quit IRC20:50
*** _pewp_ has joined #openstack-nova20:50
edleafemelwitt: yeah, well, they went through a bunch of back-and-forth as people had different opinions on how the migration should work.20:51
openstackgerritMerged openstack/nova master: update policy UT fixtures  https://review.openstack.org/39861020:57
*** elod has quit IRC20:58
openstackgerritMerged openstack/nova master: Require Placement 1.10 in nova-status upgrade check  https://review.openstack.org/49223420:58
openstackgerritMerged openstack/nova master: Add For Operators section to front page  https://review.openstack.org/49181520:59
mikalmriedem: replying now20:59
openstackgerritMerged openstack/nova master: rework index intro to describe nova  https://review.openstack.org/49183420:59
openstackgerritMerged openstack/nova master: Bulk import all config reference figures  https://review.openstack.org/49210521:00
*** esberglu has joined #openstack-nova21:01
mriedemif i work from home, and your dog next to my house barks non-stop for 30+ minutes,21:01
mriedemi should be able to do something bad21:01
openstackgerritMerged openstack/nova master: nova-manage: Deprecate '--version' parameters  https://review.openstack.org/45380821:02
*** esberglu_ has joined #openstack-nova21:02
*** jpena|mtg is now known as jpena|off21:02
*** esberglu has quit IRC21:02
openstackgerritMerged openstack/nova master: doc: Import configuration reference  https://review.openstack.org/49185321:04
openstackgerritMerged openstack/nova master: Structure cli page  https://review.openstack.org/49211121:04
*** ekuris has quit IRC21:04
*** tosika has joined #openstack-nova21:08
*** krtaylor has quit IRC21:08
*** baoli has joined #openstack-nova21:10
*** egonzalez has quit IRC21:11
openstackgerritMatt Riedemann proposed openstack/nova master: doc: address review comments in stable-api guide updates  https://review.openstack.org/49269021:12
mtreinishefried: hmm, the patch would have fixed the 502 errors. It likely means something is misconfigured elsewhere in the path21:12
mtreinishlet me take a look at the gate logs21:12
bauzasmriedem: le woof says hello to your neighbor :p21:13
mriedemgod21:13
mriedemi hope le woof isn't as dumb as the neighbor dog21:13
openstackgerritRobert Ellis proposed openstack/nova master: Clarifying node_uuid usage in ironic driver.  https://review.openstack.org/48580321:13
*** mriedem has left #openstack-nova21:14
*** mriedem has quit IRC21:14
*** mriedem has joined #openstack-nova21:14
bauzaseurasier FTW21:14
bauzasFWIW https://review.openstack.org/#/c/487954 looks good to me, but is the Ironic job working fine ?21:15
*** elod has joined #openstack-nova21:16
mriedemthis is just an uncut newfoundland21:16
*** suresh12 has quit IRC21:16
mriedemthe last ironic job run failed http://logs.openstack.org/54/487954/13/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/fa780de/console.html21:17
mriedemlooks like due to timeout21:17
mriedemPS12 was ok http://logs.openstack.org/54/487954/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/041c03a/21:17
mriedemPS14 was citycloud-lon1 which is a known slow node issue right now21:18
bauzashow many times is running the refresh cache?21:18
bauzasI mean the period21:18
*** suresh12 has joined #openstack-nova21:19
mriedembauzas: well, at least every 60 seconds by default because it's called from get_available_nodes which is called from the update_available_resources periodic task21:21
bauzasI'm trying to understand if concurrent runs would be some problems21:23
bauzaslooks not21:23
bauzasthat said, a question21:23
bauzasif we spawn, do we have the same cache ?21:24
bauzasmriedem: ^21:24
*** gouthamr has joined #openstack-nova21:24
*** thorst has quit IRC21:24
mriedemi don't understand the question21:24
bauzaswe would run multiple greenlets, right?21:25
bauzasso my question is about the node cache21:25
bauzasdo we share the same node cache object between greenlets ?21:26
mriedemoh eventlet spawn21:26
mriedemnot driver.spawn21:27
*** thorst has joined #openstack-nova21:27
*** Sukhdev has joined #openstack-nova21:27
bauzasyup eventlet.spawn_n even21:28
bauzasmaybe it's a stupid question, but I'm not remembering if coroutines accept to just share the same objects21:28
*** annegentle has joined #openstack-nova21:29
mriedem_pike_flavor_migration is passed to the spawn and updates self._migrated_instance_uuids and yes i'd assume that's all pointing back to self21:29
mriedemas the same object21:30
mriedemotherwise that would be crazy21:30
*** thorst has quit IRC21:31
mriedemyou see things getting hit in the logs21:32
mriedemhttp://logs.openstack.org/54/487954/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/041c03a/logs/screen-n-cpu.txt.gz#_Aug_09_19_31_21_25212721:32
mriedembecause the ironic jobs don't yet set resource_class on the nodes21:32
bauzasmriedem: I should explain more my concern21:32
bauzasmriedem: say we have pike flavor migration run that takes more than 60 secs21:32
bauzasthen, we would have 2 concurrent migrations21:33
bauzasthat looks okay to me because we check the cache21:33
bauzasbut the question I wonder is whether all the greenlets share the same cache, so when the first migration updates the cache, the latter gets the updates21:33
bauzasmaybe it's pointless, and I'm just silly21:34
bauzasbut I'm just thinking out loud21:34
mriedemboth greenlets should be working on the same self._migrated_instance_uuids21:34
mriedemmelwitt: edleafe: ^?21:34
bauzasmriedem: looking at StackOverflow, looks like yup21:35
melwittyeah, I think the self._migrated_instance_uuids would be shared between them (if one was still running longer than 60 sec). so maybe we need to synchronize access to that set?21:35
mriedemoh if only we could be using synchronized collections from java!21:36
mriedemwhat's the worst that would happen here?21:37
mriedemwouldn't we just double migrate the same instance.flavor.extra_spec?21:37
melwittthat's what I was trying to think about.21:37
mriedemif resource_key in specs:21:38
mriedem                # The compute must have been restarted, and the instance.flavor21:38
mriedem                # has already been migrated21:38
mriedem                continue21:38
mriedem^ should save it21:38
bauzasmriedem: melwitt: well, the more I think about the problem, the more I think it wouldn't be a prolem21:38
bauzasat least because it's for flavors21:38
melwittyeah21:38
bauzasnot sure operators have a lot of flavors needing more than 60 secs for a migration21:38
*** baoli has quit IRC21:38
*** baoli has joined #openstack-nova21:39
mriedemwell, unless you're hitting rpc timeouts on sending updates to conductor or something21:39
bauzasand if so, well, not a problem given the current code which is not synchronised but failproof21:39
melwittit's a good point to think about though. I think mriedem is right that it would skip an already migrated one even if it got a stale view of the shared set21:39
melwittand set doesn't raise if you add the same element twice, just a no-op21:43
mriedemlooks like dtantsur|afk's change in ironic to test this is blowing up during scheduling http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_53362121:44
mriedemedleafe: i left a couple of comments/questions in https://review.openstack.org/#/c/487954/21:44
mriedemhmm, the compute node should be reporting inventory to placement which would include the CUSTOM_BAREMETAL resource class21:46
mriedemhttp://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_53362121:46
*** gbarros has joined #openstack-nova21:48
*** thorst has joined #openstack-nova21:49
*** krtaylor has joined #openstack-nova21:49
mriedemlooks like we don't get inventory data before that21:50
*** xyang1 has quit IRC21:54
*** thorst has quit IRC21:54
mriedemi'm a bit torn on if we should land the nova change w/o actually seeing this working in the devstack test patch for ironic21:55
mriedemand i have to run right now anyway to pick up maya21:55
melwittshould I remove my vote for now?21:56
mriedemmelwitt: no it's ok21:56
mriedemi'm on the fence about just sending it in and then fixing later if there is something wrong21:56
melwittk21:56
mriedemwould be nice if someone from ironic could say, i tested this manually and it's fine21:56
mriedemedleafe: ^ did anyone test manually?21:57
edleafemriedem: not that I know of21:57
*** Sukhdev has quit IRC21:57
mriedemok i'll ask in -ironic but i think most of that team is gone for the day21:58
mriedembbibab21:58
mriedem*bbiab even21:58
openstackgerritMerged openstack/nova master: Improve stable-api doc with current API state  https://review.openstack.org/48992622:01
*** baoli has quit IRC22:01
*** tesseract has quit IRC22:02
*** gjayavelu has joined #openstack-nova22:02
*** Apoorva_ has joined #openstack-nova22:05
*** gjayavelu has quit IRC22:05
*** baoli has joined #openstack-nova22:05
*** ioggstream has joined #openstack-nova22:06
*** priteau has quit IRC22:07
*** kylek3h has quit IRC22:07
*** Apoorva has quit IRC22:09
*** Sukhdev has joined #openstack-nova22:09
*** suresh12 has quit IRC22:13
*** suresh12 has joined #openstack-nova22:14
*** priteau has joined #openstack-nova22:15
bauzasfolks, see you tomorrow22:18
bauzaswill look at the branch if any22:18
*** openstackgerrit has quit IRC22:18
*** priteau has quit IRC22:20
*** slaweq has quit IRC22:23
edleafeSo it looks like when Ironic sets the resource_class for a node, it doesn't do anything to create that in Placement22:27
edleafewhich would explain http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_53362122:28
edleafemriedem: should I add code to ensure that the custom resource class for a node exists in that migration? I don't see anything in ironic where it is created22:33
*** akhodos has quit IRC22:33
kevinbentonmriedem: kernel panic in VM http://logs.openstack.org/10/488510/33/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/8b65cd3/logs/testr_results.html.gz22:33
kevinbentonmriedem: how often does that happen?22:34
*** felipemonteiro has quit IRC22:36
*** rajathagasthya has quit IRC22:36
mriedemedleafe: no, not in your change. it's used in the scheduler. the custom resource class is created in placement via the periodic updates in the RT22:36
mriedemvia the get_inventory() method to the ironic driver22:36
edleafemriedem: so that's not being run before the test failure above22:37
mriedemedleafe: it's run on start of the compute service22:38
mriedemand in the update_available_resource periodic,22:39
mriedemthe problem is we're not reporting any inventory for the node22:39
mriedemso yeah, maybe the problem is a chicken and egg issue, idk22:39
mriedemdoes the inventory not show up until we have a node that we're tracking with an instance?22:39
mriedemand we don't have the instance w/o the custom resource class that the node is using22:39
edleafethe error isn't that there is no inventory; it's that there is no such resource class22:40
mriedemthat interaction is a black box to me right now22:40
mriedemedleafe: because we didn't PUT any inventory22:40
mriedemwhich would create the custom resource class from the node we're tracking22:40
mriedemthe admin could pre-create the custom resource classes, sure22:41
mriedembut nova is also trying to create them if they don't already exist in placement and we have inventory for them22:41
mriedemi'm no baremetal expert though, so i do'nt know the order in which things need to happen here to auto-create the custom resource class22:41
edleafewell, just trying to figure out how to fix this. Gotta run out in a few minutes22:41
mriedemi don't know if there is something to fix on the nova side,22:42
mriedemand i haven't dug into the devstack changes in the ironic WIP patch22:42
edleafethey're selecting a flavor with the custom RC, but placement is barfing on that since it never got created.22:42
mriedemso i might just throw this in rc122:42
mriedemand deal with any issues in rc222:42
edleafeyeah22:42
edleafei'm not sure that's the best response from placement22:42
mriedemrather than just not return allocation candidates you mean?22:43
edleafeI mean I understand preventing typos and stuff22:43
mriedemit seems ok, you're asking for allocation_candidates filtering on something which doesn't exist22:43
mriedemand you should probably make sure you can ask for something that's in GET /resource_classes22:44
mriedemit's a client side error somewhere22:44
edleafeok, so I guess the 400 is correct22:44
edleafethe client can determine how to deal with that22:44
*** catintheroof has quit IRC22:45
*** catintheroof has joined #openstack-nova22:45
edleafeI gotta run - will read the scrollback in the morning22:45
*** esberglu_ has quit IRC22:47
*** yusef has quit IRC22:48
*** annegentle has quit IRC22:48
*** annegentle has joined #openstack-nova22:49
*** catintheroof has quit IRC22:49
*** yamamoto has joined #openstack-nova22:51
*** mingyu has joined #openstack-nova22:51
*** annegentle has quit IRC22:53
*** edmondsw has quit IRC22:53
*** MVenesio has quit IRC22:54
*** mingyu has quit IRC22:55
*** rajathagasthya has joined #openstack-nova22:57
*** annegentle has joined #openstack-nova23:05
*** Swami has quit IRC23:05
*** gouthamr has quit IRC23:12
*** awaugama has quit IRC23:12
*** annegentle has quit IRC23:13
*** chyka has quit IRC23:15
*** hongbin has quit IRC23:21
*** lyan has quit IRC23:22
*** openstackgerrit has joined #openstack-nova23:22
openstackgerritMatt Riedemann proposed openstack/nova master: Remove "dhcp_options_for_instance"  https://review.openstack.org/49270723:22
mtreinishefried: ok try: https://review.openstack.org/#/c/492651/ again23:23
mtreinishI had to start playing with config settings locally and I think devstack should match the config which worked for me locally23:23
*** slaweq has joined #openstack-nova23:23
*** suresh12 has quit IRC23:26
*** suresh12 has joined #openstack-nova23:26
*** slaweq has quit IRC23:28
efriedmtreinish ack.23:30
efriedmtreinish Just got restacked with mod_wsgi, though, so I may push it out a bit.23:31
*** ioggstream has quit IRC23:31
*** yamamoto has quit IRC23:33
*** claudiub has quit IRC23:35
*** Swami has joined #openstack-nova23:35
*** rajathagasthya has quit IRC23:36
*** gyee has quit IRC23:36
*** gyee has joined #openstack-nova23:37
*** thorst has joined #openstack-nova23:38
*** yamamoto has joined #openstack-nova23:40
*** sdague has quit IRC23:47
mtreinishefried: no worries, I don't think it works anyway. It's still failing the gate23:48
mtreinishI'll have to play with it some more tomorrow23:48
*** suresh12 has quit IRC23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!