Monday, 2017-05-15

*** openstack has joined #openstack-nova06:59
*** aloga has quit IRC07:01
*** aloga has joined #openstack-nova07:01
*** _ioni is now known as ioni07:04
*** ioni has joined #openstack-nova07:05
*** belmoreira has joined #openstack-nova07:12
openstackgerritZhenyu Zheng proposed openstack/nova master: Catch exception.OverQuota when create image for volume backed instance  https://review.openstack.org/46454807:14
*** mlakat has joined #openstack-nova07:20
*** suresh12 has joined #openstack-nova07:20
*** Daisy has joined #openstack-nova07:22
*** suresh12 has quit IRC07:24
*** sridharg has quit IRC07:25
*** sridharg has joined #openstack-nova07:28
*** ralonsoh has joined #openstack-nova07:32
openstackgerritZhenyu Zheng proposed openstack/nova master: Catch exception.OverQuota when create image for volume backed instance  https://review.openstack.org/46454807:33
*** thorst_afk has joined #openstack-nova07:35
*** thorst_afk has quit IRC07:40
openstackgerritjichenjc proposed openstack/nova master: Use plain routes list for os-certificates endpoint instead of stevedore  https://review.openstack.org/46081307:45
*** masber has quit IRC07:45
*** salv-orlando has joined #openstack-nova07:46
openstackgerritHuan Xie proposed openstack/nova master: WIP: XenAPI use os-xenapi v2 in nova  https://review.openstack.org/45349307:46
*** rcernin has joined #openstack-nova07:47
*** pcaruana has joined #openstack-nova07:47
*** gszasz has joined #openstack-nova07:50
*** rwmjones has joined #openstack-nova07:55
*** sridhargaddam has joined #openstack-nova07:56
*** kylek3h has joined #openstack-nova07:57
*** salv-orlando has quit IRC07:57
*** sridharg has quit IRC07:58
*** zzzeek has quit IRC08:00
*** takashin has left #openstack-nova08:00
*** zzzeek has joined #openstack-nova08:00
*** bkopilov has quit IRC08:01
*** bkopilov has joined #openstack-nova08:01
*** kylek3h has quit IRC08:02
*** trinaths has quit IRC08:08
*** jaosorior has joined #openstack-nova08:09
*** faizy_ has quit IRC08:16
*** faizy_ has joined #openstack-nova08:16
*** lucas-afk is now known as lucasagomes08:19
*** trinaths has joined #openstack-nova08:22
*** pjm6_ has quit IRC08:27
*** abhi89 has joined #openstack-nova08:34
*** thorst_afk has joined #openstack-nova08:36
*** Daisy has quit IRC08:36
*** andymccr_ is now known as andymccr08:36
*** Daisy has joined #openstack-nova08:37
*** Daisy_ has joined #openstack-nova08:39
*** Daisy_ has quit IRC08:41
*** Daisy_ has joined #openstack-nova08:42
*** Daisy has quit IRC08:42
*** trinaths has quit IRC08:43
*** Qiming_ is now known as Qiming08:45
*** gomarivera has joined #openstack-nova08:46
*** rha has quit IRC08:48
*** rha has joined #openstack-nova08:50
*** gomarivera has quit IRC08:50
*** ZZelle_ has quit IRC08:51
*** ZZelle has joined #openstack-nova08:51
*** ZZelle is now known as Guest6838308:52
*** xinliang has quit IRC08:52
*** Guest68383 is now known as ZZelle_08:54
*** slaweq has joined #openstack-nova08:54
*** xinliang has joined #openstack-nova08:54
*** thorst_afk has quit IRC08:55
*** kylek3h has joined #openstack-nova08:58
*** slaweq has quit IRC08:59
*** liusheng has joined #openstack-nova09:01
*** kylek3h has quit IRC09:02
*** Yu-Chiang has joined #openstack-nova09:17
*** Yu-Chiang has quit IRC09:21
*** trinaths has joined #openstack-nova09:22
*** sulo_ is now known as sulo09:27
openstackgerritSongming Yan proposed openstack/nova-specs master: Sriov-bond  https://review.openstack.org/46352609:28
*** jistr_ is now known as jistr09:30
*** abhi89 has quit IRC09:33
*** tanee has quit IRC09:37
*** zsli_ has quit IRC09:38
*** aarefiev_afk is now known as aarefiev09:42
*** tovin07_ has quit IRC09:48
*** sdague has joined #openstack-nova09:48
*** Daisy_ has quit IRC09:53
*** Daisy has joined #openstack-nova09:53
*** Daisy has quit IRC09:58
*** tbachman has joined #openstack-nova09:58
*** kylek3h has joined #openstack-nova09:58
*** kylek3h has quit IRC10:03
*** tanee has joined #openstack-nova10:11
*** thorst_afk has joined #openstack-nova10:13
*** szaher_ has quit IRC10:14
*** szaher has joined #openstack-nova10:15
*** tbachman has quit IRC10:15
*** kevinz has quit IRC10:15
*** mariusv has quit IRC10:17
*** thorst_afk has quit IRC10:18
*** mariusv_ has quit IRC10:20
*** suresh12 has joined #openstack-nova10:20
*** murugesh has joined #openstack-nova10:22
murugeshHi There10:22
murugeshI have openstack mitaka version running on Centos10:22
murugeshsuddenly running instances are went to paused state10:23
murugeshunable to recover it and no error logs find in nova10:23
*** suresh12 has quit IRC10:24
*** sambetts|afk is now known as sambetts10:24
*** mariusv has joined #openstack-nova10:25
murugeshcan anyone help to to sort out this issue and fix10:25
*** mariusv has quit IRC10:25
*** mariusv has joined #openstack-nova10:25
*** lunarlamp has joined #openstack-nova10:26
*** belmoreira has quit IRC10:28
*** karthiks has joined #openstack-nova10:29
*** cNilesh has quit IRC10:31
*** nicolasbock has joined #openstack-nova10:33
murugeshcan somebody pls help me out on this issue10:40
slaweq_hello, I'm trying to run devstack on Ubuntu 17.04 and I have error https://pastebin.com/F1fsv9nH10:41
slaweq_maybe someone of You know how to fix it?10:41
slaweq_on Ubuntu 16.04 it's working fine10:41
*** gomarivera has joined #openstack-nova10:47
*** gongysh has quit IRC10:49
*** bkopilov_ has quit IRC10:49
*** bkopilov has quit IRC10:50
*** gomarivera has quit IRC10:52
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment  https://review.openstack.org/41562210:53
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface  https://review.openstack.org/43538710:54
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment  https://review.openstack.org/41810010:54
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume  https://review.openstack.org/43508210:54
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment  https://review.openstack.org/39194110:54
openstackgerritArtom Lifshitz proposed openstack/nova master: API support for tagged device attachment  https://review.openstack.org/37602110:54
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment  https://review.openstack.org/41552010:54
*** yolkfull has quit IRC10:55
*** yolkfull has joined #openstack-nova10:55
lyarwoodsdague: morning, when you have time could you take another look at https://review.openstack.org/#/c/462348/ and confirm if a releasenote is really required here?10:56
*** gongysh has joined #openstack-nova10:56
*** ZZelle_ is now known as ZZelle10:58
*** dikonoor has joined #openstack-nova10:58
*** kylek3h has joined #openstack-nova10:59
*** gcb has quit IRC11:02
*** artom__ has quit IRC11:04
*** kylek3h has quit IRC11:04
*** artom__ has joined #openstack-nova11:04
sdaguelyarwood: yeh, rootwrap changes11:05
*** artom__ is now known as artom11:05
sdagueonly redhat manages that file in share, everyone else is managing it as config (ubuntu, and all the from source alternatives)11:05
sdagueso required changes there need release note signaling as the code won't work without a config change11:06
*** artom has quit IRC11:06
*** artom has joined #openstack-nova11:06
lyarwoodsdague: right, but the removal change landed ~3 weeks ago so I'm not sure why we need to signal anything given this is basically a revert11:07
*** artom has quit IRC11:07
*** artom has joined #openstack-nova11:07
sdagueI thought the removal was last cycle?11:08
sdaguemaybe I was reading the git logs wrong11:08
lyarwoodsdague: nope, I started the change ages ago as a DNM11:08
sdagueah11:08
sdagueok, sure11:09
lyarwoodsdague: yeah the Author line has the original date11:09
sdaguecan you reference the other land in the commit message? Easier to see that then11:09
sdagueI'll +2 after that11:09
lyarwoodsdague: ack thanks11:09
sdaguesorry, without that I went digging, and apparently got the wrong answer11:10
lyarwoodnp at all11:10
*** mvk has quit IRC11:11
*** yolkfull has quit IRC11:12
*** baoli has joined #openstack-nova11:12
*** sree has quit IRC11:12
openstackgerritLee Yarwood proposed openstack/nova master: Add missing rootwrap filter for cryptsetup  https://review.openstack.org/46234811:13
*** sree has joined #openstack-nova11:13
lyarwoodsdague: ^ thanks again11:13
*** thorst_afk has joined #openstack-nova11:14
*** sree_ has joined #openstack-nova11:15
*** thorst_afk has quit IRC11:15
*** sree_ is now known as Guest578511:16
*** thorst_afk has joined #openstack-nova11:16
sdaguelyarwood: +211:17
*** sree has quit IRC11:17
*** Guest5785 has quit IRC11:20
*** gongysh has quit IRC11:21
openstackgerritSergey Nikitin proposed openstack/nova master: Replace diagnostics objects with Nova diagnostics objects  https://review.openstack.org/39448011:30
openstackgerritSergey Nikitin proposed openstack/nova master: Added nova objects for intance diagnostics  https://review.openstack.org/43474511:30
openstackgerritSergey Nikitin proposed openstack/nova master: Standardization of VM diagnostics info API.  https://review.openstack.org/35554011:30
openstackgerritSergey Nikitin proposed openstack/nova master: Completed implementation of instance diagnostics for Xen  https://review.openstack.org/39961311:30
*** lucasagomes is now known as lucas-hungry11:31
*** slaweq has joined #openstack-nova11:31
*** mariusv has quit IRC11:32
*** lunarlamp is now known as mariusv11:32
*** peter-hamilton has joined #openstack-nova11:33
peter-hamiltonjohnthetubaguy: can i get your eyes on some cert validation patches today?11:34
*** baoli has quit IRC11:34
peter-hamiltonjohnthetubaguy: https://review.openstack.org/#/c/357202/ and https://review.openstack.org/#/c/457678/ if you have time11:36
*** slaweq has quit IRC11:36
*** hieulq has quit IRC11:38
*** mvk has joined #openstack-nova11:40
murugeshHi Guys11:44
murugeshIs soembody there11:45
*** gianpietro has joined #openstack-nova11:50
*** gianpietro has quit IRC11:54
*** shaner has quit IRC11:55
*** ArchiFleKs has joined #openstack-nova12:04
*** shaner has joined #openstack-nova12:11
*** murugesh has quit IRC12:13
*** bkopilov has joined #openstack-nova12:13
*** bkopilov_ has joined #openstack-nova12:13
ArchiFleKshi, when using configdrive with nova, are the userdata still written inside the database ? it seems like it, I have a configdrive but still can retrieve userdata with tha metadata api. I thought i could use configdrive to bypass the 64k limit for user-data12:13
*** kylek3h has joined #openstack-nova12:16
*** zz_dimtruck is now known as dimtruck12:19
*** rfolco has joined #openstack-nova12:20
*** kylek3h has quit IRC12:20
*** edmondsw has joined #openstack-nova12:20
*** mlakat has quit IRC12:20
*** lucas-hungry is now known as lucasagomes12:22
*** edmondsw has quit IRC12:25
*** abhi89 has joined #openstack-nova12:25
*** edmondsw has joined #openstack-nova12:26
*** gianpietro has joined #openstack-nova12:27
*** edmondsw_ has joined #openstack-nova12:29
sdagueArchiFleKs: the 64k limit is API side enforced, back end implementation doesn't matter12:30
*** edmondsw has quit IRC12:30
*** vks1 has quit IRC12:30
*** mlakat has joined #openstack-nova12:31
*** gianpietro has quit IRC12:33
*** ayogi has quit IRC12:33
*** baoli has joined #openstack-nova12:35
ArchiFleKssdague: thanks, so there is not way to extend userdata size ? (https://bugs.launchpad.net/nova/+bug/1669979) , even if the database field is bigger. What is the right approach from your opinion ? heat-sofwaredeployment ?12:35
openstackLaunchpad bug 1669979 in OpenStack Compute (nova) "failed to boot instance when user data is larger than 65535 bytes" [Undecided,Incomplete] - Assigned to wind (better)12:35
*** erhudy has joined #openstack-nova12:36
*** vladikr has joined #openstack-nova12:37
johnthetubaguypeter-hamilton: sorry between jobs at the moment, so not going to be super active review wise right now, migrating all my stuff to personal machines at the moment12:38
ArchiFleKssdague: because config drive limit is 64mb12:40
*** ratailor has quit IRC12:40
sdagueArchiFleKs: expanding userdata size would definitely be a spec, there are lots of fall out implications of that, including memory bloat in the API server12:40
*** mdrabe has joined #openstack-nova12:41
*** deepak has quit IRC12:42
sdagueArchiFleKs: it also looks like zaneb recommended for folks to go in a different direction here - https://ask.openstack.org/en/question/105427/how-to-increase-user-data-limit-beyond-64k/12:42
*** belmoreira has joined #openstack-nova12:43
*** flaper87 has quit IRC12:43
*** tbachman has joined #openstack-nova12:44
*** flaper87 has joined #openstack-nova12:44
*** dimtruck is now known as zz_dimtruck12:45
*** flaper87 has quit IRC12:45
*** zz_dimtruck is now known as dimtruck12:48
*** khappone has quit IRC12:48
*** thorst_afk has quit IRC12:49
*** khappone has joined #openstack-nova12:51
*** dave-mccowan has joined #openstack-nova12:51
*** lyan has joined #openstack-nova12:54
abhi89johnthetubaguy: Hi John.. can you please review https://review.openstack.org/#/c/462561/12:54
peter-hamiltonjohnthetubaguy: not a problem, just trying to generate interest12:55
*** deepak has joined #openstack-nova12:55
*** kylek3h has joined #openstack-nova12:58
openstackgerritBéla Vancsics proposed openstack/nova master: Transform instance.trigger_crash_dump notification  https://review.openstack.org/39622513:03
openstackgerritDave Johnston proposed openstack/nova master: DocImpact Adds man page for nova-manage db online_data_migrations explaining the return codes  https://review.openstack.org/46463413:03
*** sree has joined #openstack-nova13:03
*** mriedem has joined #openstack-nova13:04
*** esberglu has joined #openstack-nova13:04
*** pchavva has joined #openstack-nova13:06
bauzasArchiFleKs: so, while you could use configdrivers for a quick resolution, I think you should possibly use other ways (like using a config management tool) for passing more to the instance in case 2^24 chars is not enough for you13:07
lyarwoodany fellow f25 devstack users seeing this trace when sync'ing the api_db? http://paste.openstack.org/show/609562/13:07
*** eharney has joined #openstack-nova13:08
*** liusheng has quit IRC13:13
*** liverpooler has joined #openstack-nova13:14
*** pabardina has joined #openstack-nova13:14
*** pabardina has left #openstack-nova13:14
*** jaosorior has quit IRC13:15
ArchiFleKsbauzas: yes but there no resolution at all because nova is still using the DB to store the data anyway and the 64k limit is still there. This is for magnum which uses heat, i think we should test heat-software-deployment so we do not depend on other config tools13:15
*** liusheng has joined #openstack-nova13:15
ArchiFleKssdague: there is a way for atomic to use swift as a storage for configdrive data, I know AWS also has a limit that can be bypass by storing stack template inside an S3 bucket. This might be a solution to not add issues to nova database13:17
ArchiFleKss/atomic/ironc13:17
ArchiFleKs*ironic13:17
bauzasArchiFleKs: the MediumText column accepts 2^24 chars, ie. 16M IIUC13:17
bauzasArchiFleKs: like sdague said, the 64K limit is an API one13:19
bauzasso, if you're using configdrives, you'll get a max size of 16M13:19
bauzasAFAIUI13:19
stvnoyes1mriedem: ildikov: please review cinder v3/live migration WIP when you have some time - https://review.openstack.org/#/c/463987/    thanks...13:20
*** faizy has joined #openstack-nova13:22
ArchiFleKsbauzas: yes but my point is that there is no way to bypass it unless this is change (https://github.com/openstack/nova/blob/master/nova/compute/api.py#L100) even when using confidrive with nova, right ?13:22
*** johnthetubaguy has quit IRC13:24
*** andreas_s has quit IRC13:25
*** faizy_ has quit IRC13:25
bauzasArchiFleKs: if you're using a configdrive, then it bypasses the metadata service https://docs.openstack.org/user-guide/cli-config-drive.html13:26
*** johnthetubaguy has joined #openstack-nova13:26
bauzasand http://cloudinit.readthedocs.io/en/latest/topics/datasources/configdrive.html13:26
*** lbragstad has joined #openstack-nova13:27
*** johnthetubaguy has quit IRC13:27
bauzasanyway, need to go find my daughter \o13:27
mriedemstvnoyes1: ack13:29
*** andreas_s has joined #openstack-nova13:29
*** johnthetubaguy has joined #openstack-nova13:29
bauzasArchiFleKs: ah, nevermind, you mean when passing the userdata for POST /servers then13:29
*** lbragstad has quit IRC13:29
*** lbragstad has joined #openstack-nova13:30
mriedemedleafe: i won't be able to make the scheduler meeting in 30 minutes, not sure how much was going to be talked about besides recaps from summit discussions13:31
*** jdandrea has joined #openstack-nova13:31
*** awaugama has joined #openstack-nova13:32
mriedemedleafe: for the most part claims in the scheduler/conductor had agreement from ops, although they still wanted us to pass the chosen hosts down to the cell conductor for retries13:33
*** johnthetubaguy has quit IRC13:33
ArchiFleKsbauzas: yes I mean even if i enable configdrive inside a Nova::Server heat resource, it will still use the API and apply the limits13:33
mriedemedleafe: and https://review.openstack.org/#/c/463597/ came up as a side item during that discussion, as just a good to have13:33
*** johnthetubaguy has joined #openstack-nova13:34
*** gcb has joined #openstack-nova13:36
*** johnthetubaguy has quit IRC13:37
mriedemjohnthetubaguy: doing summit recaps is going to be weird this time because there weren't a set of vertical 'nova' tracks; were you going to provide a recap of the 3 vm/bm sessions to the ML/13:37
mriedem?13:37
*** gcb has quit IRC13:37
mriedemb/c i didn't make it to all of them13:37
*** gouthamr has joined #openstack-nova13:38
edleafemriedem: thanks for the info. I'm sure we will discuss all of that at the meeting.13:39
edleafemriedem: I know I'll be asking about it :)13:39
*** gcb has joined #openstack-nova13:40
mriedemedleafe: cool; notes are in https://etherpad.openstack.org/p/BOS-forum-cellsv2-developer-community-coordination and https://etherpad.openstack.org/p/BOS-forum-move-claims-from-compute-to-scheduler13:40
mriedemi'll try to do some high level recaps in the ML this week13:40
mriedemKevin_Zheng: were you going to update https://review.openstack.org/#/c/459593/ soon?13:41
*** abhishek has joined #openstack-nova13:42
mriedemavolkov: btw thanks for the sqla make_url tip13:42
*** sree has quit IRC13:43
Kevin_Zhengmriedem: yeah, I will update tomorrow, btw, I left one remaining comment about deleting tags, could you have a look?13:43
ildikovstvnoyes1: on my list, thanks for the heads up13:43
mriedemKevin_Zheng: this? https://review.openstack.org/#/c/459593/10/nova/conductor/manager.py@108613:43
*** gongysh has joined #openstack-nova13:44
Kevin_ZhengAh yes13:44
mriedem:)13:44
*** efried has joined #openstack-nova13:44
*** gongysh has quit IRC13:44
*** sree has joined #openstack-nova13:45
Kevin_Zhengmriedem: thanks, we are flying to Suzhou tomorrow and I will update it tomorrow evening or the day after that13:45
edleafeScheduler subteam meeting in 15 minutes in #openstack-meeting-alt13:45
*** abhi89 has quit IRC13:46
*** dimtruck is now known as zz_dimtruck13:46
*** johnthetubaguy has joined #openstack-nova13:47
bauzasedleafe: just in case you don't know, I'm working on updating my series13:47
bauzasedleafe: please stop updating your ones, given I'd prefer you reviewing mine13:47
*** johnthetubaguy has quit IRC13:48
avolkovmriedem: you're welcome13:50
mriedemKevin_Zheng: thanks13:51
*** burt has joined #openstack-nova13:52
*** derekh has quit IRC13:54
*** sree has quit IRC13:55
*** gcb has quit IRC13:55
*** sree has joined #openstack-nova13:55
*** gcb has joined #openstack-nova13:56
edleafebauzas: I saw your notes13:57
bauzasack13:58
*** gongysh has joined #openstack-nova13:58
*** sree has quit IRC13:59
*** coreywright has quit IRC14:01
edleafeScheduler subteam meeting going on now in #openstack-meeting-alt14:02
*** johnthetubaguy has joined #openstack-nova14:04
*** mlavalle has joined #openstack-nova14:05
*** crushil has joined #openstack-nova14:05
*** johnthetubaguy has quit IRC14:07
*** johnthetubaguy has joined #openstack-nova14:07
*** johnthetubaguy has quit IRC14:08
*** gongysh has quit IRC14:09
*** johnthetubaguy has joined #openstack-nova14:10
*** johnthetubaguy has quit IRC14:10
*** johnthetubaguy has joined #openstack-nova14:10
*** zz_dimtruck is now known as dimtruck14:13
*** coreywright has joined #openstack-nova14:13
lyarwoodslaweq_: https://bugs.launchpad.net/nova/+bug/1690819 FYI, I'm also seeing that in F2514:13
openstackLaunchpad bug 1690819 in OpenStack Compute (nova) "nova-manage [..] api_db sync on F25 results in RuntimeError: no suitable implementation for this system" [Undecided,New]14:13
*** hongbin has joined #openstack-nova14:14
*** prateek has quit IRC14:16
*** johnthetubaguy has quit IRC14:17
*** johnthetubaguy has joined #openstack-nova14:17
*** suresh12 has joined #openstack-nova14:21
*** baoli has quit IRC14:21
sdaguelyarwood: that's a blow up deep in monotonic14:23
sdaguelyarwood: I wonder why it's blowing up there14:23
lyarwoodsdague: yeah this is odd, stepping through this in a venv now and getting the following trace http://paste.openstack.org/show/609576/14:25
*** suresh12 has quit IRC14:25
openstackgerritMathieu Gagné proposed openstack/nova master: Add ability to signal and perform online volume size change  https://review.openstack.org/45432214:27
artomSo, I tried find_library('foo') and it just returns False, without blowing up...14:27
* artom updates his F25 to be sure14:27
*** ratailor has joined #openstack-nova14:28
*** shaohe_feng has joined #openstack-nova14:30
* artom definitely can't reproduce on his F25...14:30
bauzasoh man14:30
bauzasI was about to write a reservoir sampling method for getting a random subset of a long iterator14:31
bauzasuntil I just discovered that we put all the hosts in memory after weighing those !14:31
bauzasso, weighed_hosts is a full list14:32
*** moshele has quit IRC14:32
bauzasI thought we were smarter14:32
bauzaslike getting weights for each of the items of an iterator and just sorting them by indexes instead of putting all in memory14:33
bauzasfairly inefficient14:33
*** jaypipes has joined #openstack-nova14:35
*** derekh has joined #openstack-nova14:35
stvnoyes1nova core: review 'add Cinder v3 detach to shutdown_instance' - https://review.openstack.org/#/c/463987/ - it has a +2,  but it needs a workflow +1.  thanks.14:36
bauzasoh man, even the filtering creates a list...14:36
bauzasso, we're gentle with operators by using iterators when getting a full list from DB, but then we put that all in memory once we loop over filters14:36
bauzasand then we do that again with weighers14:36
bauzasso we basically duplicate the size in memory14:37
* bauzas facepalms14:37
*** sree has joined #openstack-nova14:37
bauzas^ that could be a problem with large deployments14:38
*** derekh has quit IRC14:38
*** derekh has joined #openstack-nova14:39
lyarwoodsdague / artom ; I think it's https://github.com/eventlet/eventlet/issues/401 , I can't reproduce with eventlet 0.20.114:39
*** gomarivera has joined #openstack-nova14:40
*** thorst_afk has joined #openstack-nova14:41
sean-k-mooneysfinucan: hi o/, back from boston?14:41
sdaguelyarwood: yeh, that seems likely, I wonder why you are hitting it consistently but it's not showing up other places14:41
*** sree has quit IRC14:42
*** thorst_afk has quit IRC14:46
lyarwoodsdague: no idea, we have eventlet==0.21.0 in the gate at the moment14:46
*** marst has joined #openstack-nova14:46
*** suresh12 has joined #openstack-nova14:46
sdaguelyarwood: I guess it's a race, I wonder if your env is on the slower or faster side of things14:47
*** cfriesen has joined #openstack-nova14:47
sdaguejust to figure out when things expose14:47
*** gomarivera has quit IRC14:48
*** cfriesen_ has joined #openstack-nova14:48
lyarwoodsdague: hopefully on the faster side vs the gate given it's the only heavy workload on my current pretty beefy host14:49
*** yamahata has joined #openstack-nova14:49
*** cfriesen has quit IRC14:49
*** cfriesen_ has quit IRC14:50
*** cfriesen__ has joined #openstack-nova14:50
*** trinaths has left #openstack-nova14:50
openstackgerritMatt Riedemann proposed openstack/nova master: Make discover_hosts only query for unmapped ComputeNode records  https://review.openstack.org/42790314:50
*** dikonoor has quit IRC14:52
*** jaypipes has quit IRC14:53
*** gomarivera has joined #openstack-nova14:53
*** jaypipes has joined #openstack-nova14:53
*** itlinux has joined #openstack-nova14:53
*** dikonoor has joined #openstack-nova14:54
*** deepak has quit IRC14:54
*** eharney has quit IRC14:55
*** suresh12 has quit IRC14:56
*** cfriesen has joined #openstack-nova14:57
mriedemjaypipes: https://review.openstack.org/#/c/427903/ is the last piece in the 'make auto-discover_hosts faster' series14:58
*** sridhargaddam has quit IRC14:58
jaypipesmriedem: k, will review. currently on a plane out to China...14:59
jaypipesmriedem: spotty internet..14:59
mriedemoh fun14:59
*** cfriesen has quit IRC15:00
alex_xujaypipes: $20 for more than 10 hours flight, it is totally worth :)15:01
*** gomarivera has quit IRC15:01
*** cfriesen__ has quit IRC15:01
*** chyka has joined #openstack-nova15:01
*** cfriesen has joined #openstack-nova15:02
*** chyka has quit IRC15:02
*** chyka has joined #openstack-nova15:03
openstackgerritMonty Taylor proposed openstack/nova master: WIP Add description of server addresses objects  https://review.openstack.org/46433115:03
*** thorst_afk has joined #openstack-nova15:03
jaypipesmriedem: done.15:05
mriedemthanks15:05
jaypipesalex_xu: still on way to LAX first, then the biggie. :)15:05
* mriedem goes to cut the itchy tag off this summit t-shirt15:06
*** deepak has joined #openstack-nova15:06
jaypipesmriedem, sdague, anyone: seen this? from monotonic import monotonic as now  # noqa15:06
jaypipes  File "/home/jaypipes/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/monotonic.py", line 169, in <module>15:07
jaypipes    raise RuntimeError('no suitable implementation for this system')15:07
jaypipesRuntimeError: no suitable implementation for this system15:07
jaypipes15:07
*** rcernin has quit IRC15:07
jaypipesjust installed a fresh venv for tox/functional...15:07
mriedemi have no15:07
mriedem*not15:07
artomjaypipes, there's been some chatter about it, yeah15:07
artomjaypipes, see lyarwood's link above15:07
artomhttps://github.com/eventlet/eventlet/issues/40115:08
*** Oku_OS is now known as Oku_OS-away15:08
lyarwood& https://bugs.launchpad.net/nova/+bug/169081915:09
openstackLaunchpad bug 1690819 in OpenStack Compute (nova) "nova-manage [..] api_db sync on F25 results in RuntimeError: no suitable implementation for this system" [Undecided,New]15:09
lyarwoodjaypipes: going back to eventlet 0.20.1 works around the issue for now btw15:09
lyarwoodfor me at least15:09
sdaguethat looks like it landed today15:11
jaypipeslyarwood, artom: thx much for the help!15:11
jaypipesyet another buggy eventlet release...15:12
artomWell, if you monkey-patch all evarything...15:12
jaypipesindeed.15:12
sdagueI proposed a revert on it - https://review.openstack.org/#/c/464664/15:13
*** ociuhandu has joined #openstack-nova15:14
jaypipessdague: cool, cheers15:17
*** lpetrut has quit IRC15:17
*** jaypipes has quit IRC15:19
*** pcaruana has quit IRC15:25
*** annegentle has joined #openstack-nova15:25
artommriedem, mind checking out https://review.openstack.org/#/c/415520/33 again? There's a link down there to the passing Tempest test. Thanks :)15:27
mriedemyeah15:27
cfriesenhas anyone looked at the "fairness" behaviour of the oslo_concurrency.Lock() and Semaphore() and RWLock()?   We're seeing some indications that if one eventlet drops the lock and immediately reaquires it then it can "starve" other eventlets waiting on the same lock.15:27
mriedemdansmith: see alex_xu's question here https://review.openstack.org/#/c/461532/4/nova/compute/api.py - was there a reason we didn't do a 409 on HostAPI.compute_node_get when there are duplicates? apparently i pointed that out in the original review but never published my comments until just now.15:29
*** ociuhandu has quit IRC15:29
*** annegentle has quit IRC15:29
*** smatzek has joined #openstack-nova15:30
dansmithmriedem: will looks when I'm off this cal15:31
*** moshele has joined #openstack-nova15:37
*** andreas_s has quit IRC15:39
*** gcb has quit IRC15:39
*** lbragstad has quit IRC15:40
*** mlakat has quit IRC15:40
*** lbragstad has joined #openstack-nova15:40
openstackgerritMathieu Gagné proposed openstack/nova master: Add ability to signal and perform online volume size change  https://review.openstack.org/45432215:43
*** itlinux has quit IRC15:45
ralonsohHi, I have bug related to SR-IOV. When a PF is attached again to the OS (detached from the VM), the VF config (sriov_numvfs) is unset15:46
ralonsoh#link https://bugs.launchpad.net/nova/+bug/167970315:46
openstackLaunchpad bug 1679703 in OpenStack Compute (nova) "Unable to boot instance with VF( direct-port ) because the PF is not online" [Undecided,New] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)15:46
ralonsohI propose two ideas in the bug. Do you have any feedback?15:46
ralonsohIMO, the second one is cheaper and easier15:47
*** itlinux has joined #openstack-nova15:47
*** gjayavelu has joined #openstack-nova15:47
*** jaosorior has joined #openstack-nova15:51
*** itlinux has quit IRC15:52
*** esberglu has quit IRC15:55
*** jose-phi_ has quit IRC15:55
*** jose-phillips has joined #openstack-nova15:56
openstackgerritJay Pipes proposed openstack/nova master: placement: tests for non-shared with shared  https://review.openstack.org/46189915:56
openstackgerritJay Pipes proposed openstack/nova master: placement: test for agg association not sharing  https://review.openstack.org/46190115:56
openstackgerritJay Pipes proposed openstack/nova master: placement: test non-shared out of inventory  https://review.openstack.org/46190015:56
openstackgerritJay Pipes proposed openstack/nova master: placement: shared resources when finding providers  https://review.openstack.org/46157415:56
*** gyee has joined #openstack-nova15:58
*** ijw has joined #openstack-nova16:01
mriedemralonsoh: i'd ask moshele and/or cfriesen16:01
ralonsohmriedem: thanks!16:01
ralonsohmoshele, cfriesen: ping16:01
*** ratailor has quit IRC16:02
*** markus_z has quit IRC16:02
mosheleralonsoh: hi16:02
ralonsohmoshele: can you take a look at my comment in #link https://bugs.launchpad.net/nova/+bug/1679703?16:02
openstackLaunchpad bug 1679703 in OpenStack Compute (nova) "Unable to boot instance with VF( direct-port ) because the PF is not online" [Undecided,New] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)16:02
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment  https://review.openstack.org/41562216:02
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface  https://review.openstack.org/43538716:02
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment  https://review.openstack.org/41810016:02
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume  https://review.openstack.org/43508216:03
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment  https://review.openstack.org/39194116:03
openstackgerritArtom Lifshitz proposed openstack/nova master: API support for tagged device attachment  https://review.openstack.org/37602116:03
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment  https://review.openstack.org/41552016:03
*** Apoorva has joined #openstack-nova16:03
*** abhishek has quit IRC16:03
*** Apoorva has quit IRC16:03
ralonsohmoshele: it's about attaching a PF to a VM and then binding it again to the OS16:03
ralonsohmoshele: the VF information is lost16:03
*** Apoorva has joined #openstack-nova16:04
mosheleralonsoh: I would expect libvirt to restore the num of vfs16:04
mosheleralonsoh:  did you send mail in libvrit community?16:04
ralonsohmoshele: so you think the first option is the valid one16:04
ralonsohmoshele: ahhh, so libvirt itself16:05
ralonsohmoshele: ok, I'll do it16:05
mosheleralonsoh: let me know what they say16:07
*** artom has quit IRC16:07
*** artom has joined #openstack-nova16:08
*** artom has quit IRC16:10
*** artom has joined #openstack-nova16:10
*** nic1 has joined #openstack-nova16:10
*** esberglu has joined #openstack-nova16:14
*** karthiks has quit IRC16:14
*** belmoreira has quit IRC16:16
*** Apoorva has quit IRC16:17
*** esberglu has quit IRC16:18
lyarwoodmdbooth: https://review.openstack.org/#/c/463992/3/nova/tests/unit/virt/libvirt/test_config.py - I'm not missing a volume driver where we are using netfs am I?16:18
mdboothlyarwood: Bizarrely, I'm currently looking at exactly the same thing16:19
lyarwoodhaha great16:20
mdboothlyarwood: We don't appear to have a volume driver which sets netfs16:20
mdboothAlthough I'm just looking at all uses of source_protocol16:21
mdboothPossible it could be imported from somewhere16:21
mriedemmdbooth: yeah, that was added when gluster was added,16:21
mriedemi didn't remove it because i didn't know what might rely on it and break16:22
mriedemin https://review.openstack.org/#/c/463992/16:22
mdboothmriedem: I saw that, just looking for myself.16:22
lyarwoodyeah, I assumed we were setting it via connection_info in LibvirtNetVolumeDriver but I can't find anything on the cinder side16:22
mriedemok16:22
*** nicolasbock has quit IRC16:22
openstackgerritDan Smith proposed openstack/nova master: Make compute auto-disable itself if builds are failing  https://review.openstack.org/46359716:22
mriedemlyarwood: the source_protocol is set in nova16:22
mriedemin the volume driver's get_config method16:23
lyarwoodmriedem: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L4616:23
mriedemwe could potentially remove the netfs stuff after this since it appears to be dead code now16:23
mdboothnova/virt/libvirt/volume/net.py:        source_protocol = connection_info['driver_volume_type']16:23
*** nic1 is now known as nic16:23
lyarwoodargh wrong line number16:24
mriedemmdbooth: ok, i think that's just probably wrong16:24
mriedemi.e. bad var name16:24
*** crushil has quit IRC16:24
mdboothmriedem: It seems to be used in get_config there16:24
lyarwoodhttps://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L9616:24
mdboothYeah, that one16:24
*** moshele has quit IRC16:25
mdboothSo that means it could come from cinder and end up in the libvirt xml16:25
mdboothcinder doesn't appear to set driver_volume_type to netfs anywhere, though16:26
mriedemblech16:26
mriedembtw, the rbd and iscsi net stuff crammed into the same driver code is not fun16:26
mdboothYep16:26
mriedemthere could have been a cinder driver years ago that had netfs16:27
*** sree has joined #openstack-nova16:27
*** sree has quit IRC16:27
*** slaweq has joined #openstack-nova16:27
*** sree has joined #openstack-nova16:27
openstackgerritMoshe Levi proposed openstack/nova master: HW offload support for openvswitch  https://review.openstack.org/39826516:28
*** gianpietro has joined #openstack-nova16:28
*** slaweq has quit IRC16:28
*** Swami has joined #openstack-nova16:28
mriedemi checked grizzly, icehouse and kilo though and found nothing16:28
*** slaweq has joined #openstack-nova16:28
*** sree_ has joined #openstack-nova16:30
*** sree has quit IRC16:30
dansmithsdague: check it out: http://logs.openstack.org/37/458537/19/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e290ec2/logs/screen-n-api.txt.gz?level=TRACE#_May_11_20_08_20_21125616:30
*** sree_ is now known as Guest9357916:30
dansmithI assume that's the mysql server complaining16:31
*** jose-phillips has quit IRC16:31
sdagueyeh, probably16:33
sdagueI'm pretty sure when you see that it's coming back from the server16:33
sdaguethis the superconductor fleet?16:33
*** slaweq has quit IRC16:33
dansmithsdague: can I raise that in devstack easily or do I need to make something new?16:33
dansmithsdague: yeah but single node16:33
*** jose-phillips has joined #openstack-nova16:34
sdagueyou can change it in devstack, let me find the line16:34
dansmithsdague: do you know if I can control the size of our connection pool?16:34
openstackgerritMatt Riedemann proposed openstack/nova master: Handle uuid in HostAPI._find_service  https://review.openstack.org/46151916:34
openstackgerritMatt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get  https://review.openstack.org/46153216:34
openstackgerritMatt Riedemann proposed openstack/nova master: Use uuid for id in Services API  https://review.openstack.org/46428016:34
mriedemdansmith: you can16:34
mriedemwe had to do that for postgres at some point16:34
sdaguehttps://github.com/openstack-dev/devstack/blob/27df725179045a2e91a34eb91e29e52d0fe5f663/lib/databases/mysql#L99 ....16:34
mriedemhttps://github.com/openstack-dev/devstack/tree/master/lib/databases16:34
sdaguewe're exceeding 1024?16:34
sdaguethat seems suspicious to me16:34
*** artom_ has joined #openstack-nova16:35
*** gjayavelu has quit IRC16:35
dansmithwell, what's our connection pool size?16:35
*** artom_ has quit IRC16:36
mriedemwe used to save off the mysql config i thought16:36
mriedembut i can't find it16:36
*** artom_ has joined #openstack-nova16:36
dansmithconnection pool size would be something in our own config16:36
dansmithjust noticed by the way,16:37
*** gomarivera has joined #openstack-nova16:37
dansmiththat we don't get the config dump at the beginning of n-api logging anymore16:37
sdaguedansmith: I don't know that we're setting the pool16:37
dansmithmaybe because of the wsgi patch?16:37
dansmithlooks like 5 per worker maybe16:38
*** Apoorva has joined #openstack-nova16:38
*** artom has quit IRC16:38
dansmithwhich is pretty low16:38
dansmiththat would be ~204 workers to exhaust 102416:39
sdaguedansmith: I still see the dump16:39
sdaguehttp://logs.openstack.org/26/464326/1/check/gate-tempest-dsvm-neutron-src-oslo.context-ubuntu-xenial/4fc411c/logs/screen-n-api.txt.gz#_May_14_11_27_57_37007316:39
dansmithwhich seems unlikely on a single node16:39
sdagueit's just a little later16:39
dansmithoh okay, that seems odd given we're loading extensions before that, but okay16:39
*** baoli has joined #openstack-nova16:40
sdagueI think the new route system compiles early16:40
dansmithokay16:40
sdaguethe stuff that completely does away with the extension loader16:40
sdaguewe honestly could probably turn off debug on those16:40
dansmithnow that th3y16:41
dansmiththey are all implicit yeah16:41
dansmithanyway, I dunno what to do about this connection thing16:41
dansmithpresumably if we go higher we also need to set maxfds16:42
dansmithunless we already have for something else16:42
*** annegentle has joined #openstack-nova16:42
sdagueyeh, I don't know, but I was pretty sure that was the db rejecting things. Any chance we're leaking db connections or something?16:43
*** dikonoor has quit IRC16:43
*** harlowja has joined #openstack-nova16:44
dansmithnothing that fundamental has changed recently, so I dunno16:44
*** lucasagomes is now known as lucas-afk16:45
*** david-lyle has quit IRC16:46
*** dklyle has joined #openstack-nova16:46
dansmithlooks like we may be limited below max_connections due to ulimit on descriptors anyway16:46
dansmithso maybe we're not getting 1024 even16:46
*** itlinux has joined #openstack-nova16:48
cfriesenralonsoh: here now, you still have a question or has it been dealt with (I'm checking history now)16:49
*** crushil has joined #openstack-nova16:50
ralonsohcfriesen: well, I have an answer from libvirt guys16:51
ralonsohcfriesen: http://paste.openstack.org/show/609591/16:52
ralonsohcfriesen: at the first time, the suggest me to do what is NOT recommended in openstack manuals16:52
*** gomarivera has quit IRC16:53
ralonsohcfriesen: they recommended to modify the kernel module16:53
ralonsohcfriesen: https://github.com/openstack/openstack-manuals/blob/master/doc/networking-guide/source/config-sriov.rst16:53
*** gszasz has quit IRC16:54
*** gianpietro has quit IRC16:55
*** gianpietro has joined #openstack-nova16:55
*** suresh12 has joined #openstack-nova16:57
cfriesenralonsoh: technically you're not modifying the kernel module itself, just the parameter on loading it.   Arguably either the module parameter or setting it via /sys/class/net would work fine, it's just that they may at some point remove the module parameter in the future.16:57
melwittdansmith, sdague: on the db connection thing, coincidentally I'm building a new devstack right now to look into it. I have a guess at something that might be related but need to verify16:58
dansmithmelwitt: and you won't share until confirmed?16:58
sdaguemelwitt: ok, cool.16:58
dansmithcan I have a teaser for next week's episode?16:58
cfriesenralonsoh: the nice thing about setting it via /sys/class/net is that it gives you flexibility if you have different hardware using the same module.16:58
ralonsohcfriesen: I know and I agree. The problem is how to store it16:59
dansmithsdague: melwitt: I just pushed up a ulimit hack to see if that affects anything, fwiw16:59
*** derekh has quit IRC16:59
ralonsohcfriesen: do you think the idea of adding a pre-up command is enough?16:59
sdaguemelwitt: yeh, I would feel much more comfortable twiddling knobs if we understood why we were hitting that16:59
melwittdansmith: heh. well, I could be totally wrong, but I think it might be because we aren't caching the oslo.db transaction context managers. I had thought oslo.db connection pooling underneath took care of that, but now I want to try and see if there's a difference when doing it17:01
dansmithmelwitt: hmm17:01
*** suresh12 has quit IRC17:01
dansmithmelwitt: so we also disable some amount of tpooling, which isn't connection pooling that I know of, but..17:01
dansmithso maybe we're already breaking things and what I'm doing is just pushing us over a limit?17:01
*** ijw has quit IRC17:02
melwittthat's what I'm thinking, but it's a total guess17:02
*** snikitin_ has quit IRC17:02
*** nicolasbock has joined #openstack-nova17:03
dansmithhmm, that would be like bad and stuff17:03
sdaguelyarwood - https://review.openstack.org/#/c/464664/ approved and in gate, so hopefully you'll be back to working in ~ 1 hr17:03
dansmithnot really sure how we'd not be gc'ing them, but..17:03
*** swebster has joined #openstack-nova17:03
melwittI mean, I think cfriesen or somebody had mentioned recently they saw total db connections go way up since newton or ocata compared with before cells v2. so that had me thinking already that I wanted to look into this17:03
dansmithmaybe we're just hitting a rate that makes it a problem17:04
dansmithmelwitt: well, we would have an increase because of the api db17:04
cfriesenralonsoh: in our environment if a PF has VFs we only ever pass the VFs through to the guest so no personal experience with what you're trying to do.  In the case where you're using VFs are you "up"ing the PF?17:04
dansmithI guess if we're leaking somehow, we'd get worse the more targeting we do,17:05
dansmithbut..17:05
melwittyeah. so what I'm thinking might not be related, so I didn't want to get anyone worried about it untiI checked17:05
cfriesenmelwitt: we hit increased DB connections going from Mitaka to Newton17:05
*** ijw has joined #openstack-nova17:05
dansmithcfriesen: did you look at where those were coming from? i.e. only from api/conductor workers?17:05
*** jmlowe has joined #openstack-nova17:06
ralonsohcfriesen: it's a user case (and a possible bug). You can use both, PF and VFs, but not at the same time. But if you start a VM with a PF and then you need to use the VFs, those are lost17:06
dansmithwell, I guess if we're leaking those would be the services anyway, so.. that might not tell us anything17:06
*** esberglu has joined #openstack-nova17:06
*** mvk has quit IRC17:07
cfriesendansmith: I'm still catching up on email, but I don't think we've bottomed out exactly where they're coming from.  I think we just bumped the number of connections for now.17:09
*** slaweq has joined #openstack-nova17:09
dansmithalright17:09
cfriesendansmith: got any clever ways to figure out what code has open connections?17:10
dansmithnetstat?17:10
jmloweI've been having a problem with nova, I'm pretty sure bugs have been reported years ago (can't find them atm) but have languished, when an instance goes into error state the underlying disk is deleted, this is exactly the wrong thing to do17:10
cfriesendansmith: that's the easy part.  what about within the python code?17:11
dansmithcfriesen: well, knowing what binary has the excess is a start17:11
dansmithwe'd need extra instrumentation to know which code I think17:11
jmlowefor instance, Wednesday during the summit a user resumed a suspended instance that didn't have enough memory so qemu wouldn't start, this is eminently fixable unless nova comes along and deletes the underlying block device for the instance17:12
mriedemjmlowe: never heard of that17:12
*** eharney has joined #openstack-nova17:12
jmloweIf I dig back in my logs I even have errors from nova repeatedly deleting the same rbd device after the first wrongful deletion is succeeds17:13
jmloweno probably not just a gross conceptual error in the state diagram but also a bad implementation17:14
jmlowealso kind of hard to replicate reliably on a 300 compute node production system17:15
mriedemjmlowe: what you might be thinking of i,s17:15
mriedemresume in the libvirt driver calls this https://github.com/openstack/nova/blob/8d9eb670c75a8209901bac3b4dd94925b6b1f000/nova/virt/libvirt/driver.py#L256217:15
mriedemif that fails in some unexpected way, we go here https://github.com/openstack/nova/blob/8d9eb670c75a8209901bac3b4dd94925b6b1f000/nova/virt/libvirt/driver.py#L512017:16
jmlowewill also trigger on error state due to migration error17:16
mriedemdestroy_disks_on_failure defaults to False here17:17
mriedemand isn't set to True for the resume case17:17
mriedemso when calling cleanup, destroy_disks would be False https://github.com/openstack/nova/blob/8d9eb670c75a8209901bac3b4dd94925b6b1f000/nova/virt/libvirt/driver.py#L504217:17
*** Guest93579 has quit IRC17:18
mriedemjmlowe: i'm confused, are you doing a migration or a resume?17:18
*** sree has joined #openstack-nova17:18
jmlowehappens for both, seemingly any time an instance goes into error17:19
mriedembecause if is_shared_block_storage=True, then https://github.com/openstack/nova/blob/8d9eb670c75a8209901bac3b4dd94925b6b1f000/nova/virt/libvirt/driver.py#L1014 would delete instance files on disk17:19
jmlowein my case ceph backed17:19
mriedemthe instance going to error state is not a related issue for deleting the disks in the virt driver17:19
mriedemjmlowe: which release of nova are you on?17:20
jmlowenewton17:20
jmlowedon't think it was doing this in mitaka17:20
*** xyang1 has joined #openstack-nova17:20
mriedemi don't really see anything in newton when resume would fail that would destroy the disks17:20
mriedemnow, i think that _cleanup_failed_start method was new in ocata and backported to newton...17:21
*** jose-phillips has quit IRC17:21
mriedemyeah https://review.openstack.org/#/q/Iab5afdf1b5b8d107ea0e5895c24d50712e7dc7b117:21
*** sree has quit IRC17:22
*** fragatina has joined #openstack-nova17:23
mriedemhmm, destroy disks wasn't passed through at that point17:23
mriedemchecking something17:23
mriedemhttps://review.openstack.org/#/q/I38c969690fedb71c5b5ec4418c1b0dd53df733ec17:24
mriedemjmlowe: do you have this fix? https://review.openstack.org/#/c/428671/17:24
mriedemwent into newton 14.0.417:24
mriedemfixed the thing introduced in 14.0.317:25
*** jmlowe_ has joined #openstack-nova17:26
*** jose-phillips has joined #openstack-nova17:26
jmlowe_specifically I should have nova 14.0.317:27
mriedemjmlowe_: you mean that's what you're running?17:27
mriedemif so, you need https://review.openstack.org/#/c/428671/17:28
jmlowe_yeah, was just trying to determine if that commit was in 14.0.317:28
*** jmlowe has quit IRC17:28
jmlowe_and it's not17:29
*** ralonsoh has quit IRC17:30
jmlowe_*grumble* now off to go harass the rdo packagers17:31
jmlowe_thanks for the pointer, never would have found that myself17:31
jmlowe_on the other hand I was planning on going to ocata in the next week or so17:32
*** suresh12 has joined #openstack-nova17:32
*** annegentle has quit IRC17:34
*** READ10 has joined #openstack-nova17:36
*** karthiks has joined #openstack-nova17:36
*** gianpietro has quit IRC17:37
*** dklyle is now known as david-lyle17:37
*** NostawRm has joined #openstack-nova17:38
*** jose-phillips has quit IRC17:39
*** annegentle has joined #openstack-nova17:39
*** ijw has quit IRC17:40
*** mvk has joined #openstack-nova17:41
*** slaweq has quit IRC17:41
*** slaweq has joined #openstack-nova17:42
*** lyan has quit IRC17:42
*** slaweq has quit IRC17:46
*** lyan has joined #openstack-nova17:47
*** annegentle has quit IRC17:48
*** jose-phillips has joined #openstack-nova17:48
*** fragatina has quit IRC17:49
*** itlinux has quit IRC17:50
*** fragatina has joined #openstack-nova17:54
*** annegentle has joined #openstack-nova17:54
*** nicolasbock has quit IRC17:57
*** fragatina has quit IRC17:58
*** aarefiev is now known as aarefiev_afk17:59
*** gjayavelu has joined #openstack-nova18:00
dansmithedleafe: looks like maybe this https://review.openstack.org/#/c/464086 should go underneath bauzas' patch to do the claim?18:00
edleafedansmith: sure, it should go before the claim code18:01
*** NostawRm has quit IRC18:02
*** gianpietro has joined #openstack-nova18:03
*** gomarivera has joined #openstack-nova18:03
*** sambetts is now known as sambetts|afk18:04
*** annegentle has quit IRC18:05
*** nicolasbock has joined #openstack-nova18:05
*** annegentle has joined #openstack-nova18:05
dansmithedleafe: so given the conductor direction, are you going to work on the obviously half-finished claim-in-scheduler patch at the top of that series?18:07
dansmithedleafe: there is a big missing piece of handling the case where we fail to claim the first host It hink18:07
edleafeWell, I was kind of told *not* to work on that one :)18:08
dansmithoh18:08
edleafeBut yeah, I was planning on completing it18:08
edleafeuntil this morning18:08
dansmithokay.. no update since may 1118:08
dansmithI guess we're expecting something presently?18:09
edleafeI'm curious, though - where do you see the claim fail logic missing?18:09
dansmithif we fail doing _allocate_for_instance() here, we need to retry the claim for the same instance, but on the next host right? https://review.openstack.org/#/c/460455/6/nova/conductor/manager.py18:10
dansmithL957 just moves on to the next instance, right?18:10
edleafedansmith: fwiw, I was on PTO May 12, and came back in town this morning to a big fat -218:10
dansmithedleafe: I meant no update on bauzas' conductor claim one18:11
dansmithedleafe: yours have been updated more recently, AFAICT18:11
edleafedansmith: oh, you mean bauzas' series. I thought you meant mine18:11
dansmithnope18:11
edleafeah18:11
dansmithso actually,18:11
dansmithif we fail to claim on L957 do we just stop building everything?18:12
edleafebauzas did mention that he was ready with a new PS real soon18:12
dansmithI'm kinda assuming we do18:12
dansmithactually, maybe we don't explode, but just continue with no allocation?18:12
dansmitheesh18:12
edleafe...which negates the whole point of eliminating the race, no?18:12
dansmithwell, it's just completely wrong18:13
dansmithwe can't ask compute to build an instance we didn't actually claim for :P18:13
edleafeMy series didn't (yet) account for the computes not all being updated to the latest18:13
edleafewas hoping for guidance on that18:14
dansmithyeah, so what bauzas has there is how to do that18:14
dansmithchecking the service version18:14
edleafewell, yeah, but I didn't think it was right for the scheduler to know about such things18:15
dansmithI feel like we also probably want to communicate to the compute that it should expect the allocation to have been done already18:15
dansmithwell, I dunno.. scheduler knows such things in a couple places I think, but it's definitely okay for conductor to know that18:15
mriedemdansmith: we agreed a few weeks ago that the compute would look for existing allocations and if none found, assume the control layer didn't do that18:15
edleafethe compute checks already, if I'm not mistaken18:15
edleafein bauzas' series18:15
edleafemriedem: bingo18:16
dansmithmriedem: okay, well, we better be damn sure that we fix the bug here :)18:16
dansmithbecause this code will not claim and just send it to the compute, destined for certain failure18:16
dansmiththe exact opposite of what it's supposed to do18:17
mriedembecause it's not retrying?18:17
mriedemsorry i'm coming in to this halfway18:17
edleafedansmith: well, if you can look at the rest of my series and assume that the reqspec changes will be reversed and the list of UUIDs will be an RPC param, your feedback would be appreciated18:17
dansmithmriedem: it totally ignores the result of "did my claim work?"18:17
mriedemyeah18:17
mriedem-1 it18:18
dansmithedleafe: well, your set still does the claim in the scheduler18:18
dansmithmriedem: I am18:18
mriedem-1 it, faster :)18:18
edleafedansmith: 'zactly.18:18
edleafestill think that's cleaner18:18
*** READ10 has quit IRC18:18
*** annegentle has quit IRC18:18
*** krtaylor has quit IRC18:19
dansmithedleafe: well, I'd rather we not have the conversation again, but as you wish:)18:19
edleafedansmith: my hero!18:20
*** gianpiet_ has joined #openstack-nova18:21
*** gianpiet_ has quit IRC18:21
*** gianpiet_ has joined #openstack-nova18:21
*** gianpiet_ is now known as gianpietro_18:22
*** gianpietro_ has quit IRC18:22
*** nicolasbock has quit IRC18:22
*** gianpiet_ has joined #openstack-nova18:23
*** gianpietro has quit IRC18:24
openstackgerritJackie Truong proposed openstack/nova master: Add missing rootwrap filter for cryptsetup  https://review.openstack.org/46234818:24
openstackgerritJackie Truong proposed openstack/nova master: Fix decoding of encryption key passed to dmcrypt  https://review.openstack.org/46267418:24
*** brault has joined #openstack-nova18:25
*** gianpiet_ is now known as gianpietro18:26
*** nicolasbock has joined #openstack-nova18:27
*** artom__ has joined #openstack-nova18:28
*** artom_ has quit IRC18:28
*** gianpietro has quit IRC18:28
*** suresh12 has quit IRC18:28
*** artom__ is now known as artom18:28
*** suresh12 has joined #openstack-nova18:30
*** gianpietro has joined #openstack-nova18:30
*** annegentle has joined #openstack-nova18:30
*** gomarivera has quit IRC18:31
*** fragatina has joined #openstack-nova18:32
sdaguelyarwood - Merged openstack/requirements master: Revert "Update eventlet uc to 0.21.0"  https://review.openstack.org/46466418:32
sdagueI think you should be back in business, hopefully18:32
*** fragatina has quit IRC18:32
*** gomarivera has joined #openstack-nova18:32
*** fragatina has joined #openstack-nova18:33
*** armax has joined #openstack-nova18:34
*** annegentle has quit IRC18:37
openstackgerritChris Friesen proposed openstack/nova-specs master: update flavor info in server details spec  https://review.openstack.org/46472318:38
cfriesenmriedem: this look okay? ^18:38
*** gianpietro has quit IRC18:39
*** gomarivera has quit IRC18:41
*** jose-phillips has quit IRC18:41
*** crushil has quit IRC18:42
*** gomarivera has joined #openstack-nova18:43
*** brault has quit IRC18:44
mriedemcfriesen: yup, +218:46
mriedemsdague: ^ i assume you also don't want rxtx_factor in the instance.flavor response data18:47
mriedemsince if/when we remove rxtx_factor from flavors, we'd have to also remove it from that server response in the same microversion18:47
sdaguemriedem: I'm totally fine dumping anything early here that we think would go away later18:48
sdagueI really wasn't keen on extra_specs in here18:48
sdaguehow suck would it be if it wasn't in there?18:49
dansmithextra_specs? it's what a lot of people will want if they're looking for "is this thing numa?" and stuff18:50
dansmithI imagine cfriesen wants extra_specs, for example18:50
sdaguehmmm... ok18:52
mriedemyeah i'm good with extra specs18:53
mriedemthe admin wants to recreate an instance using a similar flavor18:53
mriedemso they'll want the extra specs too18:54
openstackgerritChris Friesen proposed openstack/nova master: fix InvalidSharedStorage exception message  https://review.openstack.org/46472618:55
cfriesensdague: yeah, it would suck if we didn't have extra-specs for the reasons given18:56
sdagueok, no prob18:56
sdaguealright, +218:56
sdaguecfriesen: so how do you propose the array of extra specs be documented at the api-ref level?18:57
dansmithoh18:58
dansmithso18:58
dansmithusers can't see extra specs on regular flavors by default I guess18:58
dansmithso we'd be exposing them here, which would be bad I guess18:58
*** baoli has quit IRC18:58
dansmithI still think this is almost useless if we don't, we just have to separate that out18:58
cfriesendansmith: the current proposed code uses the same policy as we have now for displaying extra-specs.  if the user doesn't have permission the extra-specs key isn't included in the dict18:59
dansmithand then the answer to the api-ref question is "however it's documented in the flavor api"18:59
dansmithcfriesen: ah, perfect18:59
*** baoli has joined #openstack-nova18:59
*** gianpietro has joined #openstack-nova19:00
sdaguedansmith: well, it's documented poorly there, but keys showing up in the servers doc typically try to have a higher bar of documentation19:00
dansmithokay19:00
*** brault has joined #openstack-nova19:00
*** jose-phillips has joined #openstack-nova19:00
cfriesensdague: dansmith: the proposed nova change is https://review.openstack.org/#/c/434514/19:00
cfriesenthat one doesn't have the rxtx_factor removal though, need to update it19:00
sdaguecfriesen: so, you are going to need to break each of those out19:01
sdagueI'll leave a comment19:01
*** vladikr has quit IRC19:01
cfriesensdague: in parameters.yaml?    I talked about it with mriedem before the summit, we weren't sure how to handle it because each of them is already defined, but we didn't know how to link to them19:02
sdagueyeh19:02
mriedemi can confirm that that conversation did in fact happen and i even have tapes of it19:03
*** karthiks has quit IRC19:04
*** djohnsto has joined #openstack-nova19:04
sdagueok, the pattern we were working on was https://review.openstack.org/#/c/434514/7/api-ref/source/servers.inc@61419:04
*** crushil has joined #openstack-nova19:04
cfriesensdague: there are three api calls in servers.inc that return this dict...does that mean I need to describe the dict three times?19:08
sdaguecfriesen: it needs to be described every time it's part of the response, but that should just be the same block of field refs19:09
cfriesensdague: and what about the existing flavor dict fields that are already in parameters.yaml?  (flavor_server, flavor_swap, etc)19:09
sdaguecfriesen: these are going to have min_microversion on them, so they'll need new entries19:09
*** NostawRm has joined #openstack-nova19:10
sdaguethe parameters.yaml is sort of 3rd normal form for this data, things aren't hyper normalized, means there is sometimes some copy paste, but it means fixes are more straight forward because they don't unrelatedly change something else you weren't expecting19:10
cfriesenokay, will give it a shot19:11
cfriesensdague: mriedem: dansmith: on a totally different note, what do you think of https://review.openstack.org/464726 ?   This is currently mostly an issue if you use horizon since it defaults to specifying "block_migration=False".19:14
sdaguecfriesen: i'm fine with that change19:15
*** devananda has quit IRC19:16
*** crushil has quit IRC19:17
*** gianpietro has quit IRC19:17
*** crushil has joined #openstack-nova19:17
*** adisky_ has quit IRC19:18
mriedemi don't know what there is not to not like about the original wording19:23
dansmithit doesn't sound unfine to me when it's not inapplicable due to zero or less other conditionals19:23
*** yamahata has quit IRC19:25
*** felipemonteiro has joined #openstack-nova19:25
*** gomarivera has quit IRC19:28
*** slaweq has joined #openstack-nova19:28
*** gomarivera has joined #openstack-nova19:28
*** gianpietro has joined #openstack-nova19:29
*** krtaylor has joined #openstack-nova19:30
*** zul has quit IRC19:32
*** gomarivera has quit IRC19:33
*** karthiks has joined #openstack-nova19:34
*** gianpietro has quit IRC19:34
*** gianpietro has joined #openstack-nova19:37
*** slaweq has quit IRC19:38
cfriesenthanks guys19:45
*** gianpietro has quit IRC19:45
*** gomarivera has joined #openstack-nova19:49
*** zul has joined #openstack-nova19:52
*** Apoorva_ has joined #openstack-nova19:58
*** devananda has joined #openstack-nova19:58
*** yamahata has joined #openstack-nova19:59
*** gianpietro has joined #openstack-nova20:01
*** Apoorva has quit IRC20:02
*** gianpietro has quit IRC20:04
*** Apoorva_ has quit IRC20:06
*** Apoorva has joined #openstack-nova20:06
*** marst has quit IRC20:06
*** marst has joined #openstack-nova20:07
*** Apoorva_ has joined #openstack-nova20:07
*** belmoreira has joined #openstack-nova20:07
*** belmoreira has quit IRC20:08
openstackgerritDan Smith proposed openstack/nova master: WIP: De-complicate some of the instance delete path  https://review.openstack.org/45863620:10
openstackgerritDan Smith proposed openstack/nova master: Move to proper target_cell calling convention  https://review.openstack.org/45863420:10
openstackgerritDan Smith proposed openstack/nova master: Make target_cell() yield a new context  https://review.openstack.org/45863520:10
openstackgerritDan Smith proposed openstack/nova master: WIP: Debug race in delete-while-booting test  https://review.openstack.org/46151320:10
openstackgerritDan Smith proposed openstack/nova master: DNM: Canary test against multi-cell devstack  https://review.openstack.org/45853720:10
openstackgerritDan Smith proposed openstack/nova master: Target cell on local delete  https://review.openstack.org/46405920:10
*** Apoorva has quit IRC20:11
*** baoli has quit IRC20:12
*** baoli has joined #openstack-nova20:12
*** baoli has quit IRC20:13
*** gianpietro has joined #openstack-nova20:14
*** baoli has joined #openstack-nova20:15
*** avolkov has quit IRC20:16
*** rnoriega has quit IRC20:17
*** belmoreira has joined #openstack-nova20:17
*** eharney has quit IRC20:18
*** pchavva has quit IRC20:19
*** jose-phillips has quit IRC20:19
*** gianpietro has quit IRC20:19
*** rnoriega has joined #openstack-nova20:20
*** suresh12 has quit IRC20:21
*** crushil has quit IRC20:24
*** annegentle has joined #openstack-nova20:30
*** gomarivera has quit IRC20:31
*** jose-phillips has joined #openstack-nova20:32
*** suresh12 has joined #openstack-nova20:34
*** smatzek has quit IRC20:34
*** Apoorva_ has quit IRC20:35
*** suresh12 has quit IRC20:35
*** jamesden_ has joined #openstack-nova20:38
*** liverpooler has quit IRC20:39
*** tjones has joined #openstack-nova20:40
*** tonygunk has quit IRC20:40
*** belmoreira has quit IRC20:41
*** gomarivera has joined #openstack-nova20:42
*** awaugama has quit IRC20:42
*** jamesden_ has quit IRC20:42
*** jamesde__ has joined #openstack-nova20:42
*** tonygunk has joined #openstack-nova20:44
*** liverpooler has joined #openstack-nova20:44
*** eharney has joined #openstack-nova20:49
*** brault has quit IRC20:50
*** dave-mccowan has quit IRC20:52
*** winston-d_ has quit IRC20:52
*** lyan has quit IRC20:55
*** lyan has joined #openstack-nova20:56
*** annegentle has quit IRC20:56
*** gomarivera has quit IRC20:59
*** smatzek has joined #openstack-nova21:02
*** Sukhdev has joined #openstack-nova21:02
*** smatzek has quit IRC21:02
*** smatzek has joined #openstack-nova21:03
*** gomarivera has joined #openstack-nova21:04
*** jose-phillips has quit IRC21:05
*** annegentle has joined #openstack-nova21:05
*** gianpietro has joined #openstack-nova21:05
*** dimtruck is now known as zz_dimtruck21:05
*** jamesde__ has quit IRC21:07
*** Apoorva has joined #openstack-nova21:08
*** smatzek has quit IRC21:10
*** Sukhdev_ has joined #openstack-nova21:11
*** larainema has quit IRC21:13
*** kaisers has quit IRC21:15
*** suresh12 has joined #openstack-nova21:16
mriedemmdbooth: https://review.openstack.org/#/c/443951/ was rebased but isn't failing on your test that checks to see if connect_volume and disconnect_volume are taking an instance parameter21:16
*** hrw has quit IRC21:17
*** edmondsw_ has quit IRC21:17
*** edmondsw has joined #openstack-nova21:18
*** jose-phillips has joined #openstack-nova21:18
*** thorst_afk has quit IRC21:18
*** jamesden_ has joined #openstack-nova21:18
*** thorst_afk has joined #openstack-nova21:19
*** edmondsw_ has joined #openstack-nova21:20
*** kaisers has joined #openstack-nova21:20
*** Kevin_Zheng has quit IRC21:21
*** edmondsw has quit IRC21:22
*** thorst_afk has quit IRC21:23
*** edmondsw_ has quit IRC21:24
*** gongysh has joined #openstack-nova21:26
*** gongysh has quit IRC21:27
*** baoli has quit IRC21:28
*** jamesden_ has quit IRC21:36
*** burt has quit IRC21:37
*** rmk has joined #openstack-nova21:40
*** rmk has quit IRC21:40
*** rmk has joined #openstack-nova21:40
*** annegentle has quit IRC21:40
*** kaisers has quit IRC21:41
*** eharney has quit IRC21:42
*** annegentle has joined #openstack-nova21:42
*** kaisers has joined #openstack-nova21:42
*** jmlowe_ has quit IRC21:43
*** kaisers has quit IRC21:47
*** krtaylor has quit IRC21:48
*** edmondsw has joined #openstack-nova21:48
*** edmondsw has quit IRC21:53
*** zz_dimtruck is now known as dimtruck21:59
*** suresh12 has quit IRC22:01
*** ijw has joined #openstack-nova22:02
*** oanson has quit IRC22:03
*** thorst_afk has joined #openstack-nova22:03
*** oanson has joined #openstack-nova22:03
*** thorst_afk has quit IRC22:08
*** gomarivera has quit IRC22:08
*** annegentle has quit IRC22:09
*** suresh12 has joined #openstack-nova22:09
*** felipemonteiro has quit IRC22:11
*** annegentle has joined #openstack-nova22:13
*** kylek3h has quit IRC22:13
*** dixiaoli has joined #openstack-nova22:17
*** marst has quit IRC22:17
*** tbachman has quit IRC22:17
*** annegentle has quit IRC22:18
*** annegentle has joined #openstack-nova22:18
openstackgerritMatt Riedemann proposed openstack/nova master: libvirt: expand checks for SubclassSignatureTestCase  https://review.openstack.org/46476522:18
mriedemmdbooth: fixed here ^22:18
*** tbachman has joined #openstack-nova22:23
*** slaweq has joined #openstack-nova22:23
*** annegentle has quit IRC22:24
*** mlavalle has quit IRC22:26
openstackgerritMatt Riedemann proposed openstack/nova master: Libvirt volume driver for Veritas HyperScale  https://review.openstack.org/44395122:28
*** slaweq has quit IRC22:33
*** annegentle has joined #openstack-nova22:34
*** slaweq has joined #openstack-nova22:35
*** slaweq has quit IRC22:35
*** gongysh has joined #openstack-nova22:39
*** gongysh has quit IRC22:40
*** morgan_ has joined #openstack-nova22:41
*** tjones1 has joined #openstack-nova22:43
*** kaisers has joined #openstack-nova22:44
*** liverpooler has quit IRC22:46
*** tjones has quit IRC22:47
*** tjones1 has quit IRC22:47
*** kaisers has quit IRC22:57
*** pratapagoutham has quit IRC22:57
*** chyka has quit IRC23:03
*** chyka has joined #openstack-nova23:03
*** liverpooler has joined #openstack-nova23:06
*** gomarivera has joined #openstack-nova23:08
*** ijw has quit IRC23:09
*** ijw has joined #openstack-nova23:11
*** annegentle has quit IRC23:11
*** lyan has quit IRC23:12
*** annegentle has joined #openstack-nova23:12
*** suresh12 has quit IRC23:12
*** suresh12 has joined #openstack-nova23:13
*** xyang1 has quit IRC23:14
*** ijw has quit IRC23:16
*** annegentle has quit IRC23:19
*** gianpietro has quit IRC23:20
*** gianpietro has joined #openstack-nova23:20
*** Sukhdev has quit IRC23:22
*** dixiaoli has quit IRC23:22
*** hongbin has quit IRC23:23
*** gianpietro has quit IRC23:31
*** gianpietro has joined #openstack-nova23:32
gmannalex_xu:  m back to work. ll update/follow up on policy doc and extension patches.23:37
*** gianpietro has quit IRC23:37
*** suresh12 has quit IRC23:39
*** takashin has joined #openstack-nova23:39
*** ijw has joined #openstack-nova23:42
*** suresh12 has joined #openstack-nova23:43
*** chyka has quit IRC23:44
*** harlowja has quit IRC23:46
*** gianpietro has joined #openstack-nova23:51
*** annegentle has joined #openstack-nova23:52
*** crushil has joined #openstack-nova23:53
*** kaisers has joined #openstack-nova23:54
*** hoonetorg has quit IRC23:54
*** suresh12 has quit IRC23:55
*** gouthamr has quit IRC23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!