Friday, 2017-02-03

melwittdansmith: yeah, that was being maximally careful, not to pollute the db with any root_gb = 000:00
dansmithyeah, gotta do it, or change a bunch of RT code00:00
mriedemedleafe: what do you mean to get safe_connect?00:00
mriedemthat's already in the report client00:00
dansmithit's just -> :/00:00
melwittyeah00:00
mriedemedleafe: if you're rebasing on https://review.openstack.org/#/c/404472/ then your fixes are effectively not making ocata00:01
edleafemriedem: hmmm... for some reason I thought we added it in that patch00:01
edleafemriedem: let me un-rebase it00:01
gmannmriedem: oh, that huge list is really difficult tasks always :)00:02
mriedemgmann: yeah i compared it one by one with the schema validation code00:02
mriedemand now my eyes are bleeding00:02
gmannmriedem: also we should mention internal attributes also along with joined table00:02
gmanni am going to do "find and search" each one00:03
mriedemgmann: these? https://github.com/openstack/nova/blob/c90acdbc55b743666baa9ef82c56a8e85475f739/nova/api/openstack/compute/schemas/servers.py#L28000:03
mriedemthe joined table ones are a 400 error, ^ are just ignored00:03
gmannno00:03
gmannhttps://github.com/openstack/nova/blob/c90acdbc55b743666baa9ef82c56a8e85475f739/nova/api/openstack/compute/schemas/servers.py#L36500:03
*** hongbin has quit IRC00:04
*** catintheroof has joined #openstack-nova00:04
gmannignore for sort only but we changed 500 -> 400 in filter and same foe _*00:04
gmannfor00:04
mriedemgmann: so mention anything with ^_.* in with this list https://review.openstack.org/#/c/428285/2/releasenotes/notes/add-whitelist-for-server-list-filter-sort-params-2ae766d03ba895e5.yaml@103 ?00:05
mriedemgmann: i'm not going to actually go and hunt those down00:05
gmannhttps://github.com/openstack/nova/blob/c90acdbc55b743666baa9ef82c56a8e85475f739/nova/api/openstack/compute/servers.py#L23400:05
mriedemi could just say, joined tables or internal data model attributes00:05
gmannyea that is enough00:06
mriedemok00:06
*** hfu has joined #openstack-nova00:12
mriedemedleafe: i was wondering if that delete RP fix was a blocker but then i remembered again that ironic basically can't schedule if you're all upgraded and have one of those things lying around00:13
mriedemso yeah i guess it's a blocker00:13
edleafemriedem:yeah, pretty much00:14
edleafemriedem: running tests locally on the un-rebased change00:14
*** nic has quit IRC00:16
*** esberglu has joined #openstack-nova00:16
*** dimtruck is now known as zz_dimtruck00:16
mriedemok00:16
mriedemi've gone through and marked everything with rc1, rc2 or defer https://etherpad.openstack.org/p/nova-ocata-rc1-todos00:16
mriedemdansmith: melwitt: jaypipes: ^00:16
jrollcan't schedule? or may schedule to a thing that doesn't exist00:16
*** esberglu has quit IRC00:17
mriedemschedule to a thing that blows up00:17
jrollright00:17
jrolla build failing here and there isn't the worst thing that could happen00:17
*** esberglu has joined #openstack-nova00:17
mriedemi don't know how to talk about scheduling to ironic compute nodes when the nodes are deleted00:17
mriedemor they are hosts, or wtf they are00:17
mriedemit's ironic00:17
jrolland how often to people actually delete machines from ironic00:18
jrolls/to/do/00:18
mriedemjroll: does it retry in this case?00:18
jrollmriedem: dunno, I didn't reproduce, I'm guessing not00:18
jrollI guess I'm just thinking as long as this gets in rc2, it isn't a big deal if it's in rc100:18
jrollI could be wrong ofc00:18
mriedemjroll: i think it will reschedule https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L201500:19
jrollmriedem: maaaybe, that's a lot of except blocks00:20
mriedemyeah we call spawn here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L192700:20
mriedemironicclient returns node not found00:20
mriedemwhich gets handled here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L200800:20
jrollhuh, I swear it said something about not rescheduling in the bug00:21
jrollI could be drunk00:21
mriedemhe said it didn't cleanup in a periodic run00:21
mriedembut i could also be drunk00:21
*** ducttape_ has joined #openstack-nova00:21
jrollyeah I guess not00:21
*** esberglu has quit IRC00:22
jrollI think it's fine00:22
*** jose-phillips has quit IRC00:22
openstackgerritmelanie witt proposed openstack/nova master: WIP Add functional test for boot-from-volume instances  https://review.openstack.org/42850500:22
*** raunak has quit IRC00:22
melwittdansmith: ^ I put the RT change into there for testing. the instance create works but then fails the resize bc it's requesting disk00:23
dansmithyeah was just looking at it00:23
dansmithI've got a little bit more time and then I'm really out00:24
dansmithmriedem: when do you cut the tag?00:24
openstackgerritEd Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted  https://review.openstack.org/42837500:26
edleafemriedem: ^^ un-rebased00:26
*** mlavalle has quit IRC00:26
*** sdague has quit IRC00:27
melwittdansmith: it doesn't look like mriedem put it on the RC1 things etherpad. maybe we could hope for RC2? since other stuff is slated for it also anyway00:28
dansmithmelwitt: I wasn't asking for this, I assume this isn't going to make it00:28
edleafejroll: and while this was noticed by ironic, it aaplies to any compute node00:28
melwittdansmith: oh, okay. yeah00:28
jrolledleafe: damnit I have draft comments00:28
* jroll moves them00:28
dansmithmelwitt: until your claim thing, I was going to say it looked pretty backportable, but..00:28
*** raunak has joined #openstack-nova00:29
edleafejroll: gerrit needs a "move to latest PS" button00:29
jrollor a "don't let someone update this patch for a sec" button00:29
jroll:P00:29
*** catintheroof has quit IRC00:30
melwittdansmith: yeah. it would still be technically, no db messings up. assuming that legacy request spec doesn't ever get saved anywhere but if it did, that would be a problem anyway00:30
edleafeAh, I don't mind commenting on a stale patch. I really hate it when I write these deep comments, and then find that someone beat me to it, and mine sound like "me too"00:31
dansmithmelwitt: but the scheduler would start sending instances to computes without knowing if they would agree to not count root_gb, if you updated scheduler first00:31
dansmithmelwitt: so no architectural blocks, but still probably not free of bumbs00:31
dansmither, bumps heh :)00:31
melwittoh, hm00:31
jrolledleafe: yeah, I'm kidding :)00:32
dansmithI'm projecting a little of our OSP backport rules here, but still00:32
openstackgerritMatt Riedemann proposed openstack/nova master: Add release note for filter/sort whitelist  https://review.openstack.org/42828500:32
jrolledleafe: I'm investigating that big comment to be sure, but have a few there for you00:32
edleafejroll: heh, but I'm not :)00:32
mriedemlaura kicked my ass into eating dinner with her so i'll be back in a bit00:32
mriedemand all questions will be answered00:32
*** bkopilov_ has quit IRC00:33
*** Swami has quit IRC00:33
* jroll hopes mriedem doesn't get drugged and taken away for the night00:33
melwittdansmith: yeah. it takes me awhile to understand the backport implications to upgrades00:33
dims_mriedem : better listen to her :)00:33
dansmithmelwitt: wish you had brought this up earlier, but the actual patch for this request to the scheduler took like four weeks longer than expected, so :?00:34
dansmither -> :/00:34
edleafejroll: doh! Yeah, the way that the merge conflicts were marked in git didn't help00:34
*** sapcc-bot has quit IRC00:34
edleafecleaning that up now00:35
*** sapcc-bot has joined #openstack-nova00:35
jrolledleafe: yeah no worries00:35
*** baoli has joined #openstack-nova00:35
jrollthe main problem is the get_by_host thing00:35
jrollI think it should likely be get_by_host_and_node00:35
melwittdansmith: yeah, I should have been paying more attention to that. I didn't know whether the bfv thing was planned to be handled this cycle. and cells v2 madness occurred, yadayada00:36
*** Swami has joined #openstack-nova00:36
*** chyka has quit IRC00:36
dansmithmelwitt: yeah00:39
edleafejroll: the get_by_host_and_node makes sense. I was using what jaypipes suggested, but I think he typed that off the top of his head.00:40
*** Swami has quit IRC00:40
jrolledleafe: the host/node thing is weird when you aren't an ironic person :)00:41
edleafejroll: and I thought it was always weird00:41
dansmithedleafe: and you would be right00:41
jrolledleafe: true that00:42
jrollI gotta bounce for the night, I'll look again tomorrow00:42
edleafejroll: thanks00:43
*** jaosorior has quit IRC00:46
*** salv-orlando has joined #openstack-nova00:49
*** salv-orlando has quit IRC00:54
*** sapcc-bot has quit IRC00:54
*** sapcc-bot has joined #openstack-nova00:54
*** dave-mccowan has joined #openstack-nova00:56
*** liusheng has joined #openstack-nova00:56
mriedemdansmith: melwitt: let's put the scheduler BFV patch or whatever at the bottom of the etherpad in the nice to haves list00:57
dansmithI still have 21 failures without getting into unshelve00:58
dansmithand I'm about to drop off for the evening00:58
dansmithbut, sure.00:58
mriedemultimately it's going to depend on if we can backport it00:59
*** salv-orlando has joined #openstack-nova01:00
mriedemgmann: ok i think https://review.openstack.org/#/c/428285/ is good to go01:02
mriedemoomichi: ^01:02
mriedemhttp://docs-draft.openstack.org/85/428285/3/check/gate-nova-releasenotes/aa1d391//releasenotes/build/html/unreleased.html#upgrade-notes01:03
oomichimriedem: terminated_at is valid, not terminate_at :)01:04
mriedem?01:04
mriedemterminate_at isn't in there01:05
mriedemi already fixed that bug yesterday01:05
oomichimriedem: yeah, by you. LGTM for the reno01:05
mriedemjust sign with your +W01:05
*** salv-orlando has quit IRC01:06
oomichiyep, done01:07
*** jamesdenton has joined #openstack-nova01:07
mriedemgreat, thanks01:07
*** ducttape_ has quit IRC01:08
*** jamesdenton has quit IRC01:09
gmannmriedem: +1, all grep and extra check good now01:13
edleafejroll: thanks01:13
edleafedoh01:13
mriedemi'm going to tag the highest for ocata microversions in the docs too, for compute and placement01:13
* gmann remember 47 filter params and 29 sort :)01:15
gmann+101:15
*** sam_nowitzki has joined #openstack-nova01:15
openstackgerritMatt Riedemann proposed openstack/nova master: Mark compute/placement REST API max microversions for Ocata  https://review.openstack.org/42852801:15
mriedemoomichi: ^01:16
oomichimriedem: so will you -2 patches which increase microversion in this cycle?01:17
oomichiif so, +2 on the patch01:17
gmannafter that branch going to be made right01:18
melwittdansmith, mriedem: what are the 21 failures? I don't see that on the jenkins run01:18
mriedemwe won't have any other microversions in ocata01:18
gmannyea01:18
oomichiok, +201:18
dansmithmelwitt: 21 failures after I've started making the other functions do the same01:19
openstackgerritDan Smith proposed openstack/nova master: WIP Request zero root disk for boot-from-volume instances  https://review.openstack.org/42848101:19
openstackgerritDan Smith proposed openstack/nova master: WIP Add functional test for boot-from-volume instances  https://review.openstack.org/42850501:19
dansmithmelwitt: that ^01:19
dansmithand .. I'm out01:19
melwittoh, gotcya01:20
melwittthat means ... gotcha and cya at the same time01:20
*** thorst_ has quit IRC01:20
*** crushil has joined #openstack-nova01:20
*** zz_dimtruck is now known as dimtruck01:21
*** thorst_ has joined #openstack-nova01:21
mriedemwe need more depends-ons https://review.openstack.org/#/c/428531/01:22
*** jaypipes has quit IRC01:24
openstackgerritMaciej Szankin proposed openstack/nova master: conf: Resolve TODOs in 'database'  https://review.openstack.org/39369501:25
*** thorst_ has quit IRC01:25
*** rdo has quit IRC01:26
*** rdo has joined #openstack-nova01:28
*** owalsh has joined #openstack-nova01:28
*** owalsh-afk has quit IRC01:28
*** raunak has quit IRC01:29
mriedemmelwitt: want to send this in https://review.openstack.org/#/c/428528/ ?01:29
openstackgerritEd Leafe proposed openstack/nova master: Delete a compute node's resource provider when node is deleted  https://review.openstack.org/42837501:30
melwittmriedem: k01:30
melwitt+W. gotta run for now01:31
edleafemriedem: jroll: ^^ fixed jroll's comments01:31
*** gcb has joined #openstack-nova01:31
*** Kevin_Zheng has joined #openstack-nova01:32
*** baoli has quit IRC01:32
*** amotoki has joined #openstack-nova01:33
*** hongbin has joined #openstack-nova01:35
*** ducttape_ has joined #openstack-nova01:37
* edleafe heads out for the evening01:40
*** kberger has quit IRC01:44
*** edmondsw has joined #openstack-nova01:45
*** Apoorva has quit IRC01:47
*** baoli has joined #openstack-nova01:48
*** edmondsw has quit IRC01:51
*** zhurong has joined #openstack-nova01:52
*** tbachman has joined #openstack-nova01:53
*** unicell has quit IRC01:53
*** browne has quit IRC01:55
openstackgerritZhenguo Niu proposed openstack/nova master: Fix breaking out of nested loops  https://review.openstack.org/42493101:55
*** anteaya has quit IRC01:57
*** thorst_ has joined #openstack-nova02:00
*** thorst_ has quit IRC02:00
*** salv-orlando has joined #openstack-nova02:02
*** dave-mccowan has quit IRC02:03
*** anteaya has joined #openstack-nova02:04
*** ducttape_ has quit IRC02:06
*** salv-orlando has quit IRC02:06
*** gouthamr has joined #openstack-nova02:09
*** gongysh has joined #openstack-nova02:11
*** dimtruck is now known as zz_dimtruck02:13
*** baoli has quit IRC02:13
*** ayogi has joined #openstack-nova02:17
*** baoli has joined #openstack-nova02:17
*** baoli has quit IRC02:18
*** zz_dimtruck is now known as dimtruck02:19
*** hongbin has quit IRC02:19
*** mdnadeem has joined #openstack-nova02:20
*** hongbin has joined #openstack-nova02:21
*** ducttape_ has joined #openstack-nova02:24
*** mtanino has quit IRC02:28
*** hongbin has quit IRC02:28
*** thorst_ has joined #openstack-nova02:32
*** thorst_ has quit IRC02:32
*** unicell has joined #openstack-nova02:40
*** abalutoiu has quit IRC02:41
*** amotoki_ has joined #openstack-nova02:44
*** amotoki has quit IRC02:46
*** brault has quit IRC02:49
*** unicell has quit IRC02:53
*** dave-mccowan has joined #openstack-nova02:53
*** dimtruck is now known as zz_dimtruck02:54
*** larainema has quit IRC02:56
*** amotoki_ has quit IRC02:57
*** mlakat has quit IRC02:57
*** mlakat has joined #openstack-nova02:58
*** baoli has joined #openstack-nova02:58
*** brault has joined #openstack-nova03:00
*** yamahata has quit IRC03:02
*** salv-orlando has joined #openstack-nova03:03
*** timello has quit IRC03:04
*** crushil has quit IRC03:05
*** ducttape_ has quit IRC03:07
*** salv-orlando has quit IRC03:07
*** ijw has quit IRC03:14
*** dave-mccowan has quit IRC03:15
mriedemedleafe: reviewed your patch, some issues, mostly minor, should be easy to deal with in the morning03:18
*** baoli has quit IRC03:19
*** hongbin has joined #openstack-nova03:21
*** diana_clarke has quit IRC03:22
*** kfarr has quit IRC03:23
*** baoli has joined #openstack-nova03:23
*** timello has joined #openstack-nova03:24
*** crushil has joined #openstack-nova03:25
*** tbachman has quit IRC03:28
*** hongbin has quit IRC03:31
*** owalsh has quit IRC03:33
*** owalsh_ has joined #openstack-nova03:33
*** edmondsw has joined #openstack-nova03:33
*** zhurong_ has joined #openstack-nova03:36
*** Kevin_Zheng has quit IRC03:37
*** baoli has quit IRC03:37
*** amotoki has joined #openstack-nova03:38
*** edmondsw has quit IRC03:38
*** zhurong has quit IRC03:38
*** diana_clarke has joined #openstack-nova03:39
*** eharney has quit IRC03:40
*** crushil has quit IRC03:41
*** mdnadeem has quit IRC03:42
*** armax has quit IRC03:45
*** amotoki has quit IRC03:45
*** ducttape_ has joined #openstack-nova03:48
mriedemEmilienM: in a tripleo experimental run on nova, which job should i be looking at? gate-tripleo-ci-centos-7-ovb-ha-tempest?03:48
mriedemi can't make heads or tails of the failures in gate-tripleo-ci-centos-7-ovb-nonha03:48
*** ducttape_ has quit IRC03:48
*** ducttape_ has joined #openstack-nova03:48
mriedemi guess the 2 hour 13 minute tempest one is the one i want http://logs.openstack.org/60/428360/2/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha-tempest/4eff29f/console.html#_2017-02-03_00_42_04_02809403:49
EmilienMmriedem: no, gate-tripleo-ci-centos-7-nonha-multinode-nv03:49
EmilienMit's the fastest we have03:49
mriedemEmilienM: that doesn't run on nova's experimental queue03:49
mriedemlooking at https://review.openstack.org/#/c/428360/03:49
EmilienMit did, gate-tripleo-ci-centos-7-nonha-multinode03:50
EmilienMthe -nv is when the job will move in check pipeline03:50
EmilienMmriedem: do you need anything else? otherwise I'm heading to sleep03:52
mriedemoh i was looking in the group of tripleo jobs03:52
mriedemno i'll poke in the morning i guess03:52
mriedemi'm gonna have a hell of a time figuring out why this job failed03:54
EmilienMmriedem: I can release my morning and debug with you on the tripleo job03:54
EmilienMI've patched our CI tonight to add more debug03:54
EmilienMif you run "check experimental" again, you should have debug = true on nova.conf03:55
mriedemok03:55
mriedemi also see the heat stack failed on instance not found http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/6ca334c/console.html#_2017-02-02_20_42_31_04425203:55
mriedemso that killed it03:55
mriedemanyway, talk to you tomorrow03:55
EmilienMso this is a pingtest03:55
EmilienMwhich means we weren't able to spawn a server03:55
EmilienMhttp://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/6ca334c/logs/subnode-2/var/log/nova/nova-api.txt.gz#_2017-02-02_20_42_17_09703:56
EmilienMrecheck the job (and check experimental), so you'll have DEBUG03:56
EmilienMmriedem: good night03:56
*** thorst_ has joined #openstack-nova03:58
*** thorst_ has quit IRC03:58
*** hfu has quit IRC04:00
*** owalsh_ is now known as owalsh-afk04:01
*** hfu has joined #openstack-nova04:02
*** armax has joined #openstack-nova04:04
*** ayogi has quit IRC04:07
*** crushil has joined #openstack-nova04:07
*** zhurong_ has quit IRC04:11
*** mdnadeem has joined #openstack-nova04:11
*** nmathew has joined #openstack-nova04:12
*** udesale has joined #openstack-nova04:13
*** esberglu has joined #openstack-nova04:17
*** gouthamr has quit IRC04:17
*** psachin has joined #openstack-nova04:20
*** thorst_ has joined #openstack-nova04:21
*** thorst_ has quit IRC04:21
*** esberglu has quit IRC04:21
*** sree has joined #openstack-nova04:24
mriedemrc1 is basically ready to go i think https://review.openstack.org/#/c/428531/04:25
*** dikonoor has joined #openstack-nova04:32
*** crushil has quit IRC04:33
*** crushil has joined #openstack-nova04:33
*** tbachman has joined #openstack-nova04:35
*** armax has quit IRC04:37
*** crushil has quit IRC04:39
*** diga has joined #openstack-nova04:40
*** ducttape_ has quit IRC04:40
*** crushil has joined #openstack-nova04:41
*** ducttape_ has joined #openstack-nova04:42
*** yamahata has joined #openstack-nova04:43
*** ijw has joined #openstack-nova04:43
*** ducttape_ has quit IRC04:46
*** jvgrant has quit IRC04:48
*** ayogi has joined #openstack-nova04:48
*** pbandark has joined #openstack-nova04:49
*** nicolasbock has quit IRC04:52
*** jvgrant has joined #openstack-nova04:57
*** sudipto has joined #openstack-nova04:58
*** sudipto_ has joined #openstack-nova04:59
*** salv-orlando has joined #openstack-nova05:04
*** vladikr has quit IRC05:07
*** salv-orlando has quit IRC05:09
*** sudipto_ has quit IRC05:13
*** sudipto has quit IRC05:13
*** crushil has quit IRC05:16
*** crushil has joined #openstack-nova05:16
*** ijw has quit IRC05:17
*** edmondsw has joined #openstack-nova05:21
*** prateek has joined #openstack-nova05:25
*** edmondsw has quit IRC05:26
*** raunak has joined #openstack-nova05:26
*** lpetrut has joined #openstack-nova05:32
*** abalutoiu has joined #openstack-nova05:35
*** Sukhdev has joined #openstack-nova05:39
*** ducttape_ has joined #openstack-nova05:43
*** ratailor has joined #openstack-nova05:43
*** yamahata has quit IRC05:43
*** ducttape_ has quit IRC05:47
*** salv-orlando has joined #openstack-nova06:05
*** gabor_antal_ has quit IRC06:10
*** salv-orlando has quit IRC06:10
*** crushil has quit IRC06:13
*** vsaienko has joined #openstack-nova06:15
*** vsaienko has quit IRC06:17
*** thorst_ has joined #openstack-nova06:22
*** raunak has quit IRC06:22
*** lpetrut has quit IRC06:26
*** armax has joined #openstack-nova06:26
*** thorst_ has quit IRC06:27
*** mjura has joined #openstack-nova06:32
*** Kevin_Zheng has joined #openstack-nova06:33
*** mvk has quit IRC06:37
*** mvk has joined #openstack-nova06:39
*** mptacekx has joined #openstack-nova06:40
*** unicell has joined #openstack-nova06:41
*** Sukhdev has quit IRC06:43
*** zhurong has joined #openstack-nova06:45
*** satyar has joined #openstack-nova06:52
*** longlq_ has joined #openstack-nova06:56
*** salv-orlando has joined #openstack-nova06:57
*** unicell1 has joined #openstack-nova06:57
*** armax has quit IRC06:58
*** unicell has quit IRC06:59
*** sam_nowitzki has quit IRC06:59
*** hieulq has joined #openstack-nova07:02
*** claudiub has joined #openstack-nova07:03
*** claudiub|2 has joined #openstack-nova07:06
*** longlq_ has quit IRC07:06
*** andreas_s has joined #openstack-nova07:06
*** sam_nowitzki has joined #openstack-nova07:06
*** tbachman has quit IRC07:06
*** Oku_OS-away is now known as Oku_OS07:07
*** claudiub has quit IRC07:08
*** brault has quit IRC07:08
*** edmondsw has joined #openstack-nova07:10
*** edmondsw has quit IRC07:14
*** brault has joined #openstack-nova07:15
*** sridharg has joined #openstack-nova07:16
openstackgerritJiri Suchomel proposed openstack/nova master: Retry few times when creating the initial connection to libvirtd.  https://review.openstack.org/41692807:17
*** tesseract has joined #openstack-nova07:20
*** nkrinner_afk is now known as nkrinner07:23
*** rcernin has joined #openstack-nova07:26
openstackgerritZhenyu Zheng proposed openstack/nova master: Should raise correct error instead of class exist in Placement API  https://review.openstack.org/42861207:28
openstackgerritZhenyu Zheng proposed openstack/nova master: Check destination_type when booting with bdm provided  https://review.openstack.org/40237207:31
openstackgerritZhenyu Zheng proposed openstack/nova master: Don't delete neutron port when attach failed  https://review.openstack.org/40356807:31
*** kaisers2 has quit IRC07:34
*** kaisers1 has quit IRC07:34
openstackgerritgecong proposed openstack/os-vif master: Delete H803 from ignore list  https://review.openstack.org/42861407:36
openstackgerritgecong proposed openstack/os-vif master: Delete H803 from ignore list  https://review.openstack.org/42861407:38
*** ratailor is now known as ratailor|Lunch07:38
*** kaisers has joined #openstack-nova07:38
*** ducttape_ has joined #openstack-nova07:43
*** lpetrut has joined #openstack-nova07:45
*** ducttape_ has quit IRC07:48
*** salv-orl_ has joined #openstack-nova07:49
*** esberglu has joined #openstack-nova07:50
*** vsaienko has joined #openstack-nova07:51
*** salv-orlando has quit IRC07:51
*** hieulq has quit IRC07:54
*** esberglu has quit IRC07:54
*** panda|zZ is now known as panda07:57
*** unicell1 has quit IRC07:57
*** aloga has quit IRC07:59
*** aloga has joined #openstack-nova07:59
*** hieulq has joined #openstack-nova08:03
*** salv-orl_ has quit IRC08:04
*** salv-orlando has joined #openstack-nova08:04
*** avolkov has joined #openstack-nova08:04
*** salv-orlando has quit IRC08:08
*** sam_nowitzki has quit IRC08:11
*** hieulq has quit IRC08:13
*** ralonsoh has joined #openstack-nova08:13
tdurakovjohnthetubaguy: hi, are you around?08:16
*** hieulq has joined #openstack-nova08:21
*** thorst_ has joined #openstack-nova08:23
*** amoralej|off is now known as amoralej08:26
*** thorst_ has quit IRC08:28
*** slaweq has joined #openstack-nova08:30
*** FL1SK has quit IRC08:32
*** wlhc has joined #openstack-nova08:36
*** wlhc has quit IRC08:37
openstackgerritHieu LE proposed openstack/nova master: Remove mox from unit/virt/libvirt/test_imagebackend.py (end)  https://review.openstack.org/33592508:40
*** hieulq has quit IRC08:45
*** sudipto has joined #openstack-nova08:48
*** sudipto_ has joined #openstack-nova08:48
*** ratailor|Lunch is now known as ratailor08:51
openstackgerritJeremy Liu proposed openstack/nova master: [3/3]Replace six.iteritems() with .items()  https://review.openstack.org/41508008:51
*** hieulq has joined #openstack-nova08:56
*** edmondsw has joined #openstack-nova08:58
openstackgerritAndrey Volkov proposed openstack/nova master: Update pci device fields from deleted object  https://review.openstack.org/42624308:58
openstackgerritSilvan Kaiser proposed openstack/nova master: Removes getfattr from Quobyte Nova driver  https://review.openstack.org/42864608:59
*** zzzeek has quit IRC09:00
*** zzzeek has joined #openstack-nova09:00
*** edmondsw has quit IRC09:02
*** liusheng has quit IRC09:07
*** liusheng has joined #openstack-nova09:07
*** derekh has joined #openstack-nova09:13
sudiptobauzas, hi, just a small query - if i have a property like cpuset, how do we represent that in the form of a inventory table data (of the new placement API). My doubt is - cpu set is a list of cpus, however the inventory field says: Total as one of the field attributes...09:15
sudiptoSo how can i represent a cpuset inventory...09:15
openstackgerritSergey Nikitin proposed openstack/nova master: Change parent of diagnostics objects to NovaObject class  https://review.openstack.org/39448009:18
openstackgerritSergey Nikitin proposed openstack/nova master: Standardization of VM diagnostics info API.  https://review.openstack.org/35554009:18
openstackgerritSergey Nikitin proposed openstack/nova master: Completed implementation of instance diagnostics for Xen  https://review.openstack.org/39961309:18
*** CristinaPauna has joined #openstack-nova09:21
*** karimb has joined #openstack-nova09:27
*** udesale__ has joined #openstack-nova09:28
*** udesale has quit IRC09:30
openstackgerritZhenyu Zheng proposed openstack/nova master: Should raise correct error instead of class exist in Placement API  https://review.openstack.org/42861209:35
*** lucas-afk is now known as lucasagomes09:38
*** haplo37 has quit IRC09:40
*** haplo37 has joined #openstack-nova09:42
*** ducttape_ has joined #openstack-nova09:45
*** udesale__ has quit IRC09:46
*** udesale has joined #openstack-nova09:46
*** udesale has quit IRC09:46
*** amotoki has joined #openstack-nova09:47
*** hfu has quit IRC09:47
*** ducttape_ has quit IRC09:49
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Remove redundant bdm serial mangling and saving during swap_volume  https://review.openstack.org/42736409:49
*** sree has quit IRC09:52
*** sree has joined #openstack-nova09:52
*** udesale has joined #openstack-nova09:52
*** zhurong has quit IRC09:55
*** erlon has joined #openstack-nova09:57
*** haplo37 has quit IRC09:58
*** satyar has quit IRC10:01
*** amotoki has quit IRC10:05
*** salv-orlando has joined #openstack-nova10:06
*** haplo37 has joined #openstack-nova10:07
bauzassudipto: sorry, missed your question10:10
sudiptobauzas, np10:11
bauzassudipto: for the moment, we only have quantitave resources represented in the Placement PAI10:11
*** salv-orlando has quit IRC10:11
bauzassudipto: which are CPU usage, RAM and disk usage10:11
sudiptobauzas, hmm ok.10:11
bauzassudipto: when saying cpu_set, which particular compute resource are you talking about ?10:12
sudiptobauzas, https://github.com/openstack/nova/blob/master/nova/objects/numa.py#L47 this one...10:13
bauzassudipto: on the cpu pinning details10:13
bauzassudipto: I mean the NUMA things10:13
*** rfolco has joined #openstack-nova10:13
bauzassudipto: so, yeah, this one is targeted to be implemented thanks to a nested-resource-provided BP10:13
sudiptobauzas, yeah i read through this: https://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/nested-resource-providers.html10:14
bauzasbasically, we'd say that that compute UUID == RP UUID10:14
bauzasand it would be parent of nested resource providers giving details on NICs or CPUs10:14
bauzassudipto: exactly, so FWIW it will be a pike thing10:14
sudiptoHowever, i could visualise how it would work for a numa_node's cpus or numa_node's memory (which would be qutitiative) - but if there's a set of cpus for a numa node, how do we represent it was my question...10:15
sudiptosince we have defined a "total" field in the inventory table.10:15
*** mkrai has joined #openstack-nova10:18
johnthetubaguytdurakov: sorry, late getting onto IRC this morning, hows things?10:18
*** gongysh has quit IRC10:19
johnthetubaguybauzas: good point on updating the allocations, that change just fixes the bleeding10:20
bauzassudipto: FWIW, I don't think cpu sets are quantitative amounts10:20
johnthetubaguybauzas: I have a feeling if you restart nova-compute, we end up writing out updated allocations, but I am not 100% sure on that, the code scares me lots10:20
bauzasjohnthetubaguy: so I asked yesterday10:20
bauzasjohnthetubaguy: and I've been told that periodic updates rewrite the whole allocations10:20
bauzasso we should be good10:21
*** carthaca_ has joined #openstack-nova10:21
*** sree has quit IRC10:21
johnthetubaguybauzas: hmm, honestly, that sounds terrible, but it fixes the bug10:21
bauzasbut I can look at code10:21
johnthetubaguybauzas: I want to do that for the ironic change anyways, I don't get how that works right now10:21
sudiptobauzas, but i am wondering that has to be accounted for in case of pinning cpusets10:22
*** thorst_ has joined #openstack-nova10:24
bauzassudipto: I think a CPU set a list of differnet RPs with an inventory of a VCPU resource class with amount of 110:24
bauzassudipto: and all those RPs are nested children of a parent RP which is compute node10:24
sudiptobauzas, ok so basically we will have to create RPs for each individual cpuset then?10:25
*** vsaienko has quit IRC10:25
bauzassudipto: that's what I'd think but I'm not the mastermind :)10:25
sudiptoand then map that to a parent RP of numa_node and root RP of the compute node10:25
sudiptook, that sounds like a lot of queries to me.10:25
johnthetubaguyI think the resource providers end up modelling the NUMA node topology, as nested resource providers10:25
bauzasjohnthetubaguy: that's what I said to sudipto10:25
bauzasjohnthetubaguy: the related question was rather how we model that in a nested RP world10:26
sudiptoyeah10:26
johnthetubaguydid that go in the spec, i remember it being nasty10:26
bauzasbut like I said, I just think each cpu pin is a RP with an inventory of 110:26
sudiptojohnthetubaguy, didn't read that in the spec. I may have missed if it was there.10:26
bauzasa NUMA cell would be a parent RP10:27
bauzasof that CPU pin10:27
johnthetubaguyThe NUMA node child providers will have inventory records populated for the NUMA_CORE, NUMA_THREAD and NUMA_MEMORY_MB resource classes.10:27
bauzasand a compute RP would be the parent of all NUMA cell RPs10:27
*** sambetts|afk is now known as sambetts10:27
johnthetubaguyhttp://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/nested-resource-providers.html#proposed-change10:27
bauzasjohnthetubaguy: right, but that doesn't mention the CPU pins10:27
sudiptoyeah10:27
johnthetubaguyah, gotcha10:27
johnthetubaguythats a qualitative problem, which I guess is what bauzas said earlier10:28
bauzasjohnthetubaguy: not really10:28
bauzasjohnthetubaguy: you need to *consume* a pin10:28
sudiptoyeah consume the pin quantitatively.10:28
bauzasjohnthetubaguy: hence my thinking of having a RP being the CPU pin itself10:28
bauzaswith an inventory of VCPU with amount of 110:29
*** thorst_ has quit IRC10:29
johnthetubaguyso there are a few sorts of pins right now, which one is this?10:29
bauzasand pinning the CPU would mean writing an allocation against that inventory10:29
johnthetubaguyreally you care about getting a dedicated pCPU on the "correct" numa node, I thought?10:29
johnthetubaguyor is this something else10:29
bauzasjohnthetubaguy: I'm considering the CPU-pinning feature we implemented in Nova10:30
bauzasthere are many ways to schedule the pinning10:30
bauzasbut AFAICT, we fully pin10:30
*** udesale__ has joined #openstack-nova10:30
sudiptowe could choose to pin it to a NUMA node or choose to just pin a cpuset (older nova behaviour)10:30
bauzaswhatever10:31
sudiptoMy problem is with the representation of the inventory data for cpusets itself with the current inventory field called 'total'10:31
*** amotoki has joined #openstack-nova10:31
bauzasand ?10:31
johnthetubaguyso the current inventroy doesn't work right, hence the need for nested, and the new resources of NUMA_CORE, etc10:31
bauzasif you consider a single pCPU being a RP10:32
sudiptoso we start considering every single pCPU as a RP and then do what bauzas suggested.10:32
bauzaswith an VCPU inventory attached to it10:32
johnthetubaguywell, no, I think a NUMA_CORE is a colleciton of pCPU, and you consume one of them10:32
johnthetubaguybut I may have this upside down10:32
bauzasjohnthetubaguy: you mean a NUMA cell10:32
johnthetubaguywell, no10:33
*** udesale__ has quit IRC10:33
bauzasI'm confused then10:33
johnthetubaguya NUMA cell, reports as a provider of NUMA_CORE10:33
johnthetubaguycontains lots of pCPUs you can claim10:33
johnthetubaguyI think10:33
bauzasrighrt10:33
bauzasthat's just another indirection10:33
johnthetubaguythe pCPUs are just the NUMA_CORE resources10:33
johnthetubaguyI think10:33
bauzasI just feel for CPU pinning that we need to have an inventory of 110:33
*** udesale has quit IRC10:33
sudiptook10:34
bauzasso you "take" that pCPU10:34
johnthetubaguyyou don't care about the particular pCPU though10:34
johnthetubaguyyou only care about the NUMA node its on relative to the others you have10:34
bauzasthat's a scheduling decision10:34
bauzasthe fact is10:34
johnthetubaguydoesn't seem to need the ironic 1:1 treatment10:34
bauzasthe current scheduler doesn't choose the ping10:34
bauzaspin10:34
bauzasit leaves the compute picking one pin10:35
johnthetubaguysure, the new one can pick the NUMA node, doesn't have to pick the pCPU10:35
bauzasit just verifies that the the cell has at least one pin left10:35
bauzasbut10:35
bauzaswe have a shit ton of corner cases in the current model10:35
bauzasand libvirt is pretty laxist on that10:35
*** owalsh-afk is now known as owalsh10:35
bauzasI just discovered that libvirt is totally okay having two VMs asking the same full excluse access to a pCPU10:36
bauzasexclusive*10:36
johnthetubaguyyep, the key thing is what do you care about, you don't care its pin 7, you care its in the same NUMA_NODE as your other resources10:36
bauzasyeah, but there are situations where we fail10:36
*** FL1SK has joined #openstack-nova10:36
bauzasI'm not a NUMA specialist but I swear having seen problems with that10:36
johnthetubaguywe do today, because we don't lock on the NUMA cell, only on the compute node10:37
bauzasmostly because we don't claim that in the scheduler10:37
johnthetubaguyat least thats my understanding of the issue10:37
bauzasright10:37
johnthetubaguyhence the change above10:37
lyarwoodjohnthetubaguy / bauzas ; apologies if it's too late to ask but would https://review.openstack.org/#/c/420146/ be something we can squeeze in ahead of the RC1 tag? ./me lost track of it this week with everything else flying around.10:37
bauzasanyway, I think sorting that problem is not trival and could be discussed at PTH10:37
bauzasPTG even10:37
johnthetubaguythe buckets are NUMA_NODEs rather than the computes10:38
bauzaslyarwood: I need to pick my daughter10:38
johnthetubaguylyarwood: lets take a look, there is always RC210:38
bauzasoh I thought RC1 sailed already10:39
bauzasbut it's not AFAICT10:39
bauzaslemme verify the releases repo10:39
*** hieulq has quit IRC10:39
lyarwoodI thought it was pushed to Monday, maybe I'm mistaken, but it can wait10:39
johnthetubaguyyeah, I don't see the tag right now10:39
bauzasyeah https://review.openstack.org/#/c/428531/10:39
*** amotoki has quit IRC10:40
johnthetubaguyoh, this is based on the we don't do releases on fridays because it breaks weekends10:40
johnthetubaguyalthough this one shouldn't impact others10:40
bauzasall the Depends-On landed10:41
bauzasso nothing prevents https://review.openstack.org/#/c/428531/ to be merged except the -W of course10:41
johnthetubaguylyarwood: I kinda like this use of os.path.ismount, seems sensible, shame out findmnt10:41
bauzasjohnthetubaguy: the only problem with tagging late is that we delay the requirements unfreeze10:42
bauzashonestly, I don't think tagging RC1 requires to wait over monday10:42
lyarwoodjohnthetubaguy: yeah the regression in findmnt is being addressed but IMHO we just don't need to use it10:42
bauzasbut I leave mriedem decide10:42
bauzasoh that reminds me, let me drop a courtesy highlight10:42
johnthetubaguybauzas: we have tagged on friday before, using the timezone of thursday on some pacific island10:43
lyarwoodlol10:43
bauzas\o/10:43
bauzasproblem solved.10:43
johnthetubaguyindeed :)10:43
bauzasanyway, I need to bail out, school time10:43
bauzasI could pretend the ISS is still on Thursday given they cross the world every 1.5h10:43
*** cdent has joined #openstack-nova10:44
johnthetubaguythat might give us a few more hours, sweet10:44
bauzasjust ask Thomas Pesquet to pull the +W on the patch10:44
*** edmondsw has joined #openstack-nova10:46
johnthetubaguylyarwood: are we sure our source is always the same format at what is listed in /proc/mounts10:46
johnthetubaguylyarwood: just wondering about nfs vs nfs3, etc10:46
lyarwoodjohnthetubaguy: hmm I'm pretty sure they are the same but let me double check10:48
johnthetubaguyI guess if it works for findmnt it kinda has to be, but just wanted to double check10:49
*** edmondsw has quit IRC10:50
cdentanything special I should poke at, or let the etherpad be my guide?10:55
johnthetubaguyI think the etherpad is still king10:55
johnthetubaguyat this point, seems like its looking for things to backport to RC210:56
cdenti'll get the fixes for https://review.openstack.org/#/c/428375/11:02
*** tbachman has joined #openstack-nova11:05
tdurakovjohnthetubaguy: got question on that one and xen-api(https://review.openstack.org/#/c/424428/) live-migration process - what is the common process upgrading xen server in openstack( 6.5 > 7.0)11:06
tdurakovany recommendations on xenapi multinode deployment?11:06
johnthetubaguytdurakov: the way we used to do it was live migrate everything off the box, re-image box. I am sure you could do in place upgrade, but all the VMs will be turned off for the duration11:07
*** salv-orlando has joined #openstack-nova11:08
johnthetubaguytdurakov: also, never attempt to use the pools feature with Nova, just have several standalone boxes11:08
johnthetubaguytdurakov: but the expert these days is BobBall, he can help you through the details11:08
tdurakovBobBall: are you around?11:09
tdurakovjohnthetubaguy: thanks for details11:09
johnthetubaguytdurakov: no worries11:10
tdurakovjohnthetubaguy: no ci for xen multinode, so it's kind of tough thing to reveiw11:10
johnthetubaguytdurakov: yeah, not sure if they have an internal one11:11
*** salv-orlando has quit IRC11:12
openstackgerritBob Ball proposed openstack/nova master: Fix live migrate with XenServer  https://review.openstack.org/42442811:12
BobBallhi tdurakov :) I am indeed around.11:13
*** hfu has joined #openstack-nova11:13
tdurakovBobBall: hi, want to make some testing of ^11:13
BobBallAnd no, no multinode CI.  Current testing is done manually.  We recently got some more hardware installed in our CI env though with the purpose of enabling multinode for live migrate testing.  Hopefully that'll be added in the next few weeks.11:13
tdurakovwhat's the best way doing multinode xen installation now?11:14
BobBallTo test I normally set up two normal devstack environments then connect them together11:14
tdurakovok11:14
*** hfu has quit IRC11:16
lyarwoodjohnthetubaguy: sorry took a while to ./stack.sh with the NFS plugin, both v3 and v4 leave the same source URI in /proc/mounts - http://paste.openstack.org/show/597531/11:17
johnthetubaguylyarwood: oh cool, thanks for digging11:17
lyarwoodjohnthetubaguy: np it was worth it tbh11:18
*** panda is now known as panda|brb11:19
openstackgerritChris Dent proposed openstack/nova master: Delete a compute node's resource provider when node is deleted  https://review.openstack.org/42837511:22
*** gongysh has joined #openstack-nova11:24
*** nicolasbock has joined #openstack-nova11:24
cdentjohnthetubaguy: responded to you about the 404 thing: it's if not 404, log11:24
cdentso everything unusual is logged11:25
*** salv-orlando has joined #openstack-nova11:25
johnthetubaguycdent: balls, I read that as ==11:25
cdenton the bugger scale where do balls fit?11:25
* cdent cringes11:25
johnthetubaguyits hard to know, lol11:25
*** tojuvone has quit IRC11:26
*** amotoki has joined #openstack-nova11:26
cdentI made the exact same mistake where the same idiom was used elsewhere (I think on deleting allocations, but not sure).11:26
jrollmorning y'all11:26
cdentjohnthetubaguy: i put up a new version of that while you were commenting. I think getting all the previous comments11:26
johnthetubaguycdent: yeah, just taking a peak11:27
jroll"It seems to be the webob pattern to make a non-2xx response False-y" webob whyyyy11:27
cdentmorning jroll, you might want to look at https://review.openstack.org/#/c/428375/ too11:27
cdentthat's request not webob doing that11:27
jrollyes, I'm already there :)11:27
jrollok requests whyyyy11:27
jroll:)11:27
cdentand I would guess they do it because, like, man, requests is http for humans, man11:27
cdentit vaguely makes sense, but it does make for some confusion, and is to me biased11:28
*** rmart04 has joined #openstack-nova11:28
cdentI think response object would only be falsey if the web server couldn't be reached11:28
jrollyeah, it's non-obvious11:29
jrolloh well, learned something new today11:29
johnthetubaguyits better than success being falsey I guess11:29
*** hfu has joined #openstack-nova11:30
*** ayogi has quit IRC11:30
* cdent feels a "it's better than a kick in the head" cascade coming on11:30
andymccrif anybody is about to give https://review.openstack.org/#/c/426163/ a review, that'd be really appreciated :)11:32
johnthetubaguycdent: should we worry about the compute nodes we already deleted, but have entries for in the placement API, I am thinking probably no?11:34
*** ratailor has quit IRC11:37
*** amotoki has quit IRC11:37
cdentjohnthetubaguy: I'm not sure how we'd figure that out in that direction.11:38
johnthetubaguycdent: good question, I guess we can't list the resource providers by host as such, only node?11:39
*** sree has joined #openstack-nova11:40
*** panda|brb is now known as panda11:40
johnthetubaguycdent: I do wonder about a nova-manage tool to "re-sync" placement, and check for any odd things, its something that could live in such a tool. Although creating that tool feels a bit too much like giving up.11:40
jrolljohnthetubaguy: left a comment, I'm thinking we throw people a release note about that, at least?11:44
jrollpatch lgtm otherwise though11:44
cdentjohnthetubaguy: yeah the host, node stuff is not something that the placement side knows. it just knows resource providers. it's up to the "nova" side to do that jiggery pokery11:46
cdentjroll: That release note seems potentially very confusing and scary for people and might lead to random deletions against the api?11:47
jrollcdent: yeah, we can add big scary warnings or something11:48
*** tbachman has quit IRC11:48
jrollwarning: this only applies if you're doing CD by the day/week11:48
jrollidk maybe it's fine11:48
jrollfeels like we should tell people somehow though11:49
* cdent looks at johnthetubaguy 11:49
*** vsaienko has joined #openstack-nova11:50
*** hfu has quit IRC11:53
*** tojuvone has joined #openstack-nova11:56
*** sdague has joined #openstack-nova11:56
*** Hosam has joined #openstack-nova11:56
*** crushil has joined #openstack-nova11:56
*** Hosam_ has joined #openstack-nova11:57
*** pabardina has quit IRC11:57
*** lucasagomes is now known as lucas-hungry11:57
johnthetubaguycdent: jroll: oh, so this is affecting ironic already, I kept thinking it wasn't11:57
johnthetubaguywires crossed in my head11:58
jrolljohnthetubaguy: yeah, since a few days ago, I believe11:58
johnthetubaguysurely since last cycle when we started reporting the inventory?11:59
*** mnestratov|2 has quit IRC12:00
johnthetubaguyor did we skip ironic till really recently?12:00
jrolljohnthetubaguy: oh. ohhhhhh.12:00
jrolldamn12:00
jrollyeah, we only noticed it when scheduling switched to inventory12:00
*** Hosam has quit IRC12:01
*** ayogi has joined #openstack-nova12:01
johnthetubaguyoh, right, its only impacting since we started caring about whats in there12:01
jrollbut I guess the real problem might be much longer eh12:01
jrollI guess we could spin up a newton install and test this easily12:01
cdentjohnthetubaguy: are you taking typing lessons from me "I was forgetting this could already an issue in production for folks, we should do that thing."12:02
johnthetubaguyheh12:03
cdentSo how should this reno express itself? Do we explain just the issue or the issue and the hack for how to deal with it? Or something else?12:03
*** kristian__ has joined #openstack-nova12:03
*** gongysh has quit IRC12:03
*** eharney has joined #openstack-nova12:04
*** ducttape_ has joined #openstack-nova12:04
johnthetubaguycdent: its probably worth saying the problem you will see with iroinc and placement if you don't clear out old deleted nodes12:04
johnthetubaguycdent: its tempting to hack up some tool to help fix that, but we good to stop the bleeding with this patch anyways12:05
* johnthetubaguy has a evil thought, is just checking...12:05
cdentJust so I'm clear on the situation how would that hack operate? How do you know which things on the placement api side to delete?12:06
johnthetubaguybecause they are not in the current compute node list? I am not 100% sure actually12:06
jrolldoesn't resource_provider.uuid == ironic_node.uuid here?12:07
*** sudipto has quit IRC12:07
*** sudipto_ has quit IRC12:07
*** vsaienko has quit IRC12:08
*** vsaienko has joined #openstack-nova12:08
johnthetubaguycdent: so compute node is soft_delete, so if they don't trim that table, we could double check from soft deleted compute nodes to double check they are all removed12:09
*** dave-mccowan has joined #openstack-nova12:10
cdentbut if they do, then they truly are orphans, unless the rp.name is useful in some fashion?12:12
cdentso the hack is for each get_all_soft_delete_compute_nodes; try_to_delete_rp_with_same_uuid()12:14
johnthetubaguyname is hypervisor_hostname I think12:14
johnthetubaguyhttps://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L63012:14
jrolland hypervisor_hostname is the ironic node uuid12:16
johnthetubaguycan you get a list of the deleted uuids though?12:16
*** bvanhav__ has joined #openstack-nova12:16
*** bvanhav__ is now known as bvanhav12:16
johnthetubaguyI mean you can get all the current ones and spot the difference12:16
johnthetubaguymaybe thats the best way12:16
jrollyeah that'd be the only way12:16
cdentblechy12:17
jrollfor cn in compute_nodes: if ironic.get_node(cn.hypervisor_hostname) gives a 404: deleted_nodes.append(cn.hypervisor_hostname)12:17
johnthetubaguybut we already deleted the compute node12:17
jrolldelete from resource_providers where name in $deleted_nodes;12:17
jrolloh right, s/compute_nodes/inventory thing12:18
jrollyou get the idea :)12:18
johnthetubaguyah, gotcha12:18
*** esberglu has joined #openstack-nova12:18
cdent"inventory thing"?12:18
cdent(hasn't had sufficient coffee to get the idea)12:18
jrollheh12:18
*** gcb has quit IRC12:19
jrollfor rp in resource_providers: if ironic.get_node(rp.name).status_code == 404: deleted_nodes.append(rp.name)12:19
jrollcdent: ^12:19
* johnthetubaguy is heading out to get some food, back in a bit12:19
johnthetubaguyyeah, that12:19
cdentisn't that proposing deleting all resource providers that are not ironic nodes?12:20
johnthetubaguyfeels like we should add a nova-manage to do that12:20
*** catintheroof has joined #openstack-nova12:20
*** catintheroof has quit IRC12:20
jrollcdent: yes, more sanity checks are needed for deployments that include not-ironic and such12:20
johnthetubaguycdent: yeha, I guess we have to mix in compute nodes too12:20
*** catintheroof has joined #openstack-nova12:21
jrollif I was going for 100% ready to ship I wouldn't be writing this in irc :)12:21
johnthetubaguylisting all compute nodes, would do it, rather than iroinc I guess12:21
cdentand any other resource providers someone has chosen to create outside the context of the resource tracker12:21
cdentwe cannot assume the list of resource providers is only things "we" have created12:21
jrolltrue12:22
johnthetubaguywell, feels liek we should add some user concepts in, when we can't assume that12:22
*** esberglu has quit IRC12:22
jrollmaybe the reno is just "if you get build failures trying to schedule to a non-existent node, check *waves hands* stuff, you might need to delete the RP"12:23
cdentI gotta break away for several minutes, perhaps we can all percolate on this for a small window of time and we'll know something after that?12:23
cdentjroll++ :)12:23
johnthetubaguyyeah, a vague reno is better than no reno or fix getting in12:24
cdentbbs12:25
*** panda is now known as panda|lch12:26
BobBalljohnthetubaguy / mriedem: Citrix XenServer CI is back healthy now - can username citrix_xenserver_ci be re-added to the voting group https://review.openstack.org/#/admin/groups/511,members?12:28
*** satyar has joined #openstack-nova12:31
*** rcernin has quit IRC12:31
*** diga has quit IRC12:32
*** vladikr has joined #openstack-nova12:33
*** edmondsw has joined #openstack-nova12:34
*** ducttape_ has quit IRC12:35
*** thorst_ has joined #openstack-nova12:37
*** CristinaPauna has quit IRC12:38
*** edmondsw has quit IRC12:38
*** daemontool has joined #openstack-nova12:38
*** edmondsw has joined #openstack-nova12:43
*** gszasz has joined #openstack-nova12:43
*** ayogi has quit IRC12:45
*** CristinaPauna has joined #openstack-nova12:47
*** mkoderer has joined #openstack-nova12:49
*** ducttape_ has joined #openstack-nova12:53
*** jvgrant has quit IRC12:58
*** jvgrant_ has joined #openstack-nova12:58
*** lucas-hungry is now known as lucasagomes12:59
*** khappone_ has quit IRC13:00
*** tbachman has joined #openstack-nova13:04
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Remove redundant bdm serial mangling and saving during swap_volume  https://review.openstack.org/42736413:06
*** zhurong has joined #openstack-nova13:09
*** timello has quit IRC13:10
johnthetubaguyBobBall: OK13:18
openstackgerritSilvan Kaiser proposed openstack/nova master: Removes getfattr from Quobyte Nova driver  https://review.openstack.org/42864613:18
johnthetubaguyBobBall: is a vote on this one coming really soon? https://review.openstack.org/#/c/424428/13:18
*** vsaienko has quit IRC13:19
BobBalljohnthetubaguy: If by soon you mean 25 minutes ago, then yes13:19
BobBallWe commented but didn't having voting rights so no vote :)13:19
*** sudipto has joined #openstack-nova13:20
*** sudipto_ has joined #openstack-nova13:20
*** timello has joined #openstack-nova13:22
*** salv-orlando has quit IRC13:24
tdurakovBobBall: in case of re-roll please add change-log comment to xen migrate data object13:25
*** jheroux has joined #openstack-nova13:26
johnthetubaguyBobBall: heh, thats soon, cool. turns out you have to click refresh13:26
*** pbandark has quit IRC13:26
BobBalltdurakov: Sure will do - what do you mean by re-roll though?13:26
tdurakovBobBall: also, what if I'm migration instances, from 6.5 to 7.0 or vice versa is this field required for both of them?13:26
tdurakovBobBall: resubmit patch13:27
tdurakovin case of ^13:27
*** panda|lch is now known as panda13:27
*** sree has quit IRC13:28
johnthetubaguytdurakov: I believe this is Nova version related, not XenServer version related13:28
BobBalltdurakov: Migrating from 7.0 to 6.5 isn't supported by XenServer - XAPI will prevent that explicitly.  6.5 to 7.0 will work as the source doesn't look at or use the vif_map, it's prepared by and used by the target.13:28
tdurakovjohnthetubaguy: from the but definiton I understood that xen version affect this as well13:28
tdurakovs/but/bug13:28
johnthetubaguyyeah, I see that detail now13:28
johnthetubaguyoops13:28
mriedemcdent: johnthetubaguy: jroll: vdrok: edleafe: on https://review.openstack.org/#/c/428375/ i think we actually should raise a ResourceProviderInUse or some kind of exception if we delete the compute node but can't delete the resource provider,13:29
BobBalljohnthetubaguy: XS 7.0 introduced the requirement of the vif map13:29
mriedembecause that means we won't reschedule13:29
cdentmriedem: but what would catch it? it's already getting logged as a warning?13:29
*** markvoelker_ has joined #openstack-nova13:29
mriedemway up in the compute manager13:30
johnthetubaguymriedem: I thought this was just getting called from some periodic task?13:30
mriedemthe thing that caught the NodeNotFound in the bug report13:30
mriedemno13:30
mriedemwell,13:30
johnthetubaguyboth I guess13:30
mriedemyeah it is13:30
mriedemthe periodic task doesn't delete the RP,13:30
mriedemand then the scheduler uses it,13:30
*** khappone has joined #openstack-nova13:30
mriedemwhich then blows up with NodeNotFound in build_and_run_instances13:30
mriedemwhich triggers a reschedule to another node13:30
mriedemso,13:30
mriedemi guess raising the exception up doesn't help13:30
mriedemfooy13:30
cdentindeed13:31
cdentthis is messy. mriedem did you see all the chatter above about how to word a reno note (I linked on the review if not)13:31
mriedemi didn't read that yet13:31
*** bhagyashris has quit IRC13:32
*** crushil has quit IRC13:32
*** markvoelker has quit IRC13:32
mriedem"btw this optional thing which you might be using might be borking your ironic scheduling but then again if you're not using it so nevermind but just in case you were and hit this thing....beware"13:32
mriedemsomething like that? ^13:32
*** jaypipes has joined #openstack-nova13:33
cdentwell yeah, exactly, that's pretty much the problem13:33
*** hfu has joined #openstack-nova13:33
cdentI'm drafting something now13:33
*** hfu has quit IRC13:33
johnthetubaguycould we use that NodeNotFound error to trigger a check of that resource provider, thinking about it13:34
openstackgerritSergey Nikitin proposed openstack/nova-specs master: PCI NUMA Policies  https://review.openstack.org/36114013:34
vdrokif the compute node was deleted but resource provider was not, the bit of deleting resource provider will not be retried by the periodic task?13:34
*** hoonetorg has quit IRC13:35
jrollvdrok: yeah, that could be a problem, maybe we do it the other way (RP first, CN second), and if RP blows up don't delete the compute node either13:36
jrollhrm13:36
cdentmriedem, johnthetubaguy, vdrok, jroll: I'll hold off on pushing up anything until this discussion resolves in some fashion, otherwise it feels like we're going to churn13:37
jrolljohnthetubaguy: oh, we totally could13:37
jrollcdent: maybe etherpad the reno draft for now, then, so we can get that handled?13:37
johnthetubaguymriedem: I am tempted to merge the current fix with a note, that stops *more* badness when stuff gets deleted, feels like we discuss the extra bits on top?13:37
cdentjroll: except that it sounds like we potentially have a more robust fix in the works, which would change the note quite a bit?13:38
mriedemjohnthetubaguy: yeah i'm not looking to get a comprehensive fix in for rc113:38
*** xyang1 has joined #openstack-nova13:38
cdentor that, okay13:38
jrollheh13:38
mriedemwe can backport something if needed13:38
johnthetubaguy+113:38
mriedemhonestly i doubt very much that anyone is using placement yet, if even on newton yet13:38
johnthetubaguyI think thats fair, based on the massive bugs we found in the last few days13:39
jrollyeah, so to trigger this, you need newton running with placement, with ironic, and delete at least one ironic node13:39
jrolland it will be triggered upon install of current master13:40
jrollright?13:40
mriedemnot just install master,13:41
mriedembut bring all of your computes up to ocata13:41
mriedemso full upgrade of all nova services13:41
*** sree has joined #openstack-nova13:41
jrollI don't think so13:42
*** annegentle has joined #openstack-nova13:42
jrollthe bug is in newton if you're running placement13:42
*** jpena is now known as jpena|lunch13:42
jrollbecause you have inventories for things that don't exist13:42
mriedemjroll: but the scheduler in newton doesn't use placement13:42
jrolloh wait, the scheduler in ocata checks compute version, doesn't it13:42
sean-k-mooneymriedem: if you have got a moment would you have any objection to me creating an experimatal job for nova to run a subset of what is currently run in the intel NFV ci? we will be moving where the NFV ci is hosted in then next 2-4 months and now seems like a good opurtnity to upstream the tests that dont require special hardware. those that do will remain in the nfv ci.13:43
jrollthe scheduler in ocata does, though, I just don't know if it waits for computes to be upgraded before using placement13:43
*** prateek_ has joined #openstack-nova13:43
cdentjroll: it does13:44
cdentrecent change13:44
jrollok cool, so builds don't break until you upgrade everything to ocata13:44
jrollthanks cdent13:44
cdenthere's a stab at the reno: https://etherpad.openstack.org/p/bug-1661258-reno13:46
*** sree has quit IRC13:46
cdentrather hand wavey13:46
*** ducttape_ has quit IRC13:46
cdentor should it be an issue instead of fixes?13:47
jrollmaybe both?13:47
mriedemsean-k-mooney: sure13:48
*** hoonetorg has joined #openstack-nova13:48
*** amotoki has joined #openstack-nova13:48
mriedemcdent: jroll: i'd just have 'issues' as in known issue13:48
cdentis leading to a reschedule the only outcome? I thought it could be more dire13:48
*** Hosam_ has quit IRC13:49
mriedemit triggers a reschedule13:50
*** vsaienko has joined #openstack-nova13:50
jrollhm, is there any way to trigger this for not ironic nodes?13:50
mriedemof which you get 3 and then NoValidHost13:50
*** annegentle has quit IRC13:50
mriedemjroll: you'd have to delete the compute node records manually for non-ironic13:50
jrollmriedem: right, because n-cpu wouldn't be running for a hyp that went away right13:51
jrolls/went away/was thrown in the dumpster/13:51
mriedemn-cpu can be running and if it is, it will recreate the compute node record in the db13:51
mriedembut the service record might be gone13:51
sean-k-mooneymriedem: ok ill start working on the job. after a while i would hope we can move it to the check pipeline as non voting and then eventually voteing once it is shown to be stable.13:51
jrollmriedem: right-o13:52
jrollokay, I think I'm happy with that reno13:53
cdentjroll, mriedem I we're happy with that content I can smooth it13:53
jrolland I'm good with the fix as-is, for now, given the reno13:53
*** crushil has joined #openstack-nova13:53
* cdent takes it13:53
jrollcdent: ++13:53
sean-k-mooneymriedem: im also going to work on some jobs for os-vif and a few other repos. i may have an update at the ptg on nfv testing in general.13:53
mriedemsean-k-mooney: splendid13:54
mriedemthanks for working on that13:54
*** esberglu has joined #openstack-nova13:54
sean-k-mooneymriedem: well its been kindof buging me that we dont have a linux bridge job for os-vif there are some other gaps in the unit testing in os-vif and nova that i want to take a look at too to tighten up the testing a  little13:56
*** kberger has joined #openstack-nova13:56
openstackgerritChris Dent proposed openstack/nova master: Delete a compute node's resource provider when node is deleted  https://review.openstack.org/42837513:58
cdent^^ adds the reno and changes one bug to partial-bug13:58
bauzasmriedem: FWIW, saw your discussion with dansmith and melwitt yesterday about BFV instances13:58
*** esberglu has quit IRC13:59
jrollcdent: trailing space on the reno, not sure if that breaks some check job13:59
bauzasmriedem: so I saw your comment on gerrit and replied to it by saying that we should still verify the disk resource in placement13:59
cdentdag nab it13:59
bauzasmriedem: unless the disk value is 013:59
*** crushil has quit IRC14:00
openstackgerritChris Dent proposed openstack/nova master: Delete a compute node's resource provider when node is deleted  https://review.openstack.org/42837514:00
bauzasmriedem: because once we will have Cinder RPs, then we will still take a allocation for the disk size14:00
*** annegentle has joined #openstack-nova14:00
*** bvanhav has quit IRC14:00
*** daemontool has left #openstack-nova14:01
jrollcdent: +1'd. now I'm wondering if we want to catch NodeNotFound and have a better error message, but it's probably fine14:01
*** ociuhandu has quit IRC14:01
mriedemjroll: you could still probably get the NodeNotFound for other reasons14:01
mriedemlike some kind of race to delete the node and schedule to it at the same time14:01
mriedemi'd just leave it14:02
jrollmriedem: good point14:02
mriedembauzas: i don't really have that all in my head, but dansmith has a WIP up for some of this14:02
mriedemhttps://review.openstack.org/#/c/428481/14:02
*** sree has joined #openstack-nova14:04
*** sridharg has quit IRC14:05
*** edleafe is now known as figleaf14:07
*** READ10 has joined #openstack-nova14:07
figleafcdent: thanks for picking up https://review.openstack.org/42837514:07
mriedem+2 from me14:08
cdentfigleaf: you're welcome, seemed like that one had some rush on it. If you're busy today I could also pick up the allocation schema stuff too (on which I left some comments about some wrongness)14:08
figleafcdent: No, not busy. Just getting started, and the caffeine hasn't yet kicked in14:09
figleafGonna look at those next14:09
cdentroger that14:09
jrollfigleaf: it's 8am already, why so late14:09
figleafjroll: had to get the rest of the house out to work/school14:10
cdentI guess that means I should think about writing a "placement update" which I've been avoiding because I have no clue, probably because caffeine14:10
figleafjroll: gotta be able to focus!14:10
jrollfigleaf: that was a joke, to be clear :P14:10
figleafjroll: so was my response :)14:10
jroll:D14:11
* cdent has no sense of humor before of https://twitter.com/TechnicallyRon/status/82692914892917964814:11
cdents/before/because/14:11
* cdent also has no sense of typing14:11
*** crushil has joined #openstack-nova14:12
*** sree has quit IRC14:14
*** gouthamr has joined #openstack-nova14:14
*** eharney has quit IRC14:15
*** sree has joined #openstack-nova14:15
*** mvk has quit IRC14:15
*** baoli has joined #openstack-nova14:16
cdentmriedem: imma fix that nit14:16
*** annegentle has quit IRC14:17
openstackgerritChris Dent proposed openstack/nova master: Delete a compute node's resource provider when node is deleted  https://review.openstack.org/42837514:17
*** bvanhav has joined #openstack-nova14:19
*** mdrabe has joined #openstack-nova14:19
*** ducttape_ has joined #openstack-nova14:19
bauzascdent: figleaf: open questions in https://review.openstack.org/#/c/428375/914:20
*** Dinesh_Bhor has quit IRC14:20
figleafcdent: regarding https://review.openstack.org/#/c/416752/8/nova/api/openstack/placement/handlers/inventory.py, what should be the value for 'max_unit' in the INVENTORY_DEFAULTS (line 109) ?14:20
cdentfigleaf: I would think since we want to set a schema minimum of 1 (since less than that is meaningless or wrong) then the default should be 1 as well?14:22
cdentbauzas: looking14:22
*** tlian has joined #openstack-nova14:22
bauzasmriedem: when are you removing -W for https://review.openstack.org/#/c/428531/ ?14:23
*** sree has quit IRC14:23
*** cleong has joined #openstack-nova14:24
mriedembauzas: probably after https://review.openstack.org/#/c/428375/ is merged now14:24
mriedemjohnthetubaguy: +2 on ^ now14:24
johnthetubaguylooking...14:25
figleafcdent: didn14:25
mriedemcdent: figleaf: a functional report client test as noted by bauzas for deleting RPs would be good, but i'd do that in a follow up14:25
figleafdoh14:25
*** owalsh is now known as owalsh-brb14:25
mriedembauzas: i already talked to tonyb about blocking requirements yesterday, it's not a big deal right now14:25
figleafcdent: didn't seem right to set the default 'max' to 114:25
mriedemdims_: was telling everyone that yesterday was a soft date for rc1 and monday would be ok14:25
*** mjura has quit IRC14:25
mriedembauzas: but i'll get rc1 out today if we get this last fix in14:26
cdentfigleaf: it is weird yeah, but is better than 014:26
* dims_ nods14:26
*** mjura has joined #openstack-nova14:26
figleafcdent: sure, but even db.MAX_INT felt better (even though that seems wrong, too)14:26
bauzasmriedem: okay, cool with me about RC114:26
mriedemi've got to get my kid on the bus and head into the office14:26
bauzascdent: mriedem: about https://review.openstack.org/#/c/428375/8/nova/scheduler/client/report.py@772 my question is rather14:27
*** mriedem has quit IRC14:27
bauzascdent: mriedem: if we're deleting a compute node, are we also deleting the nested instances elsewhere ?14:27
cdentfigleaf: hmmm. yeah. a good point. that is probably better than 1 as it results in things being left in a working state if a max is not provided14:27
cdentfigleaf: which is what DEFAULTS ought to be going for14:28
cdentfigleaf: so yeah, step_size 1, min_unit 1, max_unit MAX_INT seems right for the defaults14:28
figleafcdent: "Hi, I'd like 1 MEMORY_MB for my VM, please"14:29
cdentit's more that 640k so it's gotta be enough14:29
*** owalsh-brb has quit IRC14:29
* figleaf goes to make more coffee14:30
cdentmetoo@aol.com14:31
*** esberglu has joined #openstack-nova14:31
*** satyar has quit IRC14:31
*** esberglu has quit IRC14:31
*** esberglu has joined #openstack-nova14:32
*** liverpooler has joined #openstack-nova14:32
*** owalsh-brb has joined #openstack-nova14:32
*** eharney has joined #openstack-nova14:32
*** tbachman has quit IRC14:33
*** esberglu_ has joined #openstack-nova14:33
*** rmart04 has quit IRC14:35
*** tbachman has joined #openstack-nova14:35
bauzasjohnthetubaguy: I had a concern on https://review.openstack.org/#/c/428375/14:35
bauzasI did +1 because I didn't wanted to pollute and say it was a blocker14:36
bauzasbut I'm still wondering if all goes the right way14:36
*** esbergl__ has joined #openstack-nova14:37
*** esberglu has quit IRC14:37
johnthetubaguybauzas: most of the time the list of instances will be empty, I think14:37
johnthetubaguybauzas: its hitting the 409 from placement I guess, that would cause an issue14:37
bauzasjohnthetubaguy: I need to look at code, because I wonder that14:38
*** esberglu has joined #openstack-nova14:38
*** esberglu_ has quit IRC14:38
bauzasjohnthetubaguy: say I'm booting inst1, then deleting computeA, do I get a nova list empty?14:38
bauzasI suppose so14:38
bauzasso I'm wondering where the thing is done14:38
*** paw has quit IRC14:39
*** amoralej is now known as amoralej|lunch14:40
*** eharney has quit IRC14:40
*** jpena|lunch is now known as jpena14:40
*** esbergl__ has quit IRC14:41
*** esberglu_ has joined #openstack-nova14:41
*** zz_dimtruck is now known as dimtruck14:42
*** paw has joined #openstack-nova14:42
*** andreas_s has quit IRC14:43
*** burt has joined #openstack-nova14:43
*** esberglu has quit IRC14:44
*** mvk has joined #openstack-nova14:44
openstackgerritChris Dent proposed openstack/nova master: Add more debug logging on RP inventory update failures  https://review.openstack.org/41423014:44
openstackgerritChris Dent proposed openstack/nova master: Add placement request id to log when GET or POST rps  https://review.openstack.org/42774014:44
openstackgerritChris Dent proposed openstack/nova master: Add placement request id to log when GET aggregates  https://review.openstack.org/42687414:44
openstackgerritChris Dent proposed openstack/nova master: Add more debug logging on RP inventory delete failures  https://review.openstack.org/42629014:44
*** kaisers has quit IRC14:45
cdentjohnthetubaguy: the bottom of that stack ^ was +W (+2 from you) but failing in the gate a few times, so I've rebased but now it is all bereft of its approval14:45
*** tblakes has joined #openstack-nova14:46
openstackgerritSylvain Bauza proposed openstack/nova master: Destroy the ReqSpec object when deleting the instance  https://review.openstack.org/39106014:46
openstackgerritSylvain Bauza proposed openstack/nova master: Cover again the ReqSpec record creation testing  https://review.openstack.org/39105914:46
*** amotoki has quit IRC14:48
*** amotoki has joined #openstack-nova14:48
*** amotoki has quit IRC14:49
*** ratailor has joined #openstack-nova14:51
*** mtanino has joined #openstack-nova14:52
openstackgerritEd Leafe proposed openstack/nova master: Add check for invalid allocation amounts  https://review.openstack.org/41675214:54
figleafcdent: ^^ Addressed your concerns14:54
*** dikonoor has quit IRC14:54
*** eharney has joined #openstack-nova14:54
*** AAR has joined #openstack-nova14:57
*** AAR has left #openstack-nova14:57
johnthetubaguycdent: ah, doh, will take a peak14:57
*** mvk has quit IRC14:58
*** AAR has joined #openstack-nova14:58
*** mriedem has joined #openstack-nova14:59
*** AAR has left #openstack-nova14:59
*** sree has joined #openstack-nova15:01
*** mtanino_ has joined #openstack-nova15:01
*** mtanino has quit IRC15:02
*** eharney has quit IRC15:02
*** enki_anunnaki has quit IRC15:04
johnthetubaguycdent: I do wonder if we have too many new lines in some of the error messages in the logs, BTW15:06
johnthetubaguycdent: its a micro nit, it might be more to do with the error message omitted by the placement API and how that looks in the logs15:07
mriedemthat's the nature of the requests.response isn't it?15:07
johnthetubaguyit could be15:09
*** smatzek has joined #openstack-nova15:09
*** mtanino_ has quit IRC15:09
*** haplo37_ has quit IRC15:09
*** dimtruck is now known as zz_dimtruck15:10
*** gabor_antal has joined #openstack-nova15:11
*** pcaruana has quit IRC15:11
*** mtanino has joined #openstack-nova15:11
*** zz_dimtruck is now known as dimtruck15:11
*** mptacekx has quit IRC15:11
cdentjohnthetubaguy, mriedem: that could be the way that webob is formatting error responses. the json_error_formatter stuff subclasses something which puts newlines in here and there for no apparently reason15:12
johnthetubaguycdent: I was looking at that with pumaranikar the other day, I think it might be a bit new line happy15:12
*** haplo37_ has joined #openstack-nova15:12
cdentnew line happy15:13
cdentyes15:13
openstackgerritSarafraj Singh proposed openstack/nova-specs master: Send service token along with user tokens  https://review.openstack.org/42693315:13
cdenti'm not sure I'm keen on the decorator idea, but the current middleware idea won't work15:13
cdent(but I've mostly been ignoring that)15:13
*** sree has quit IRC15:15
cdentjohnthetubaguy: in any case, the logging on the rt side shouldn't care. we should just fix it on the api side, yeah?15:16
*** sree has joined #openstack-nova15:16
johnthetubaguycdent: yeah, certainly if the extra new lines are from the API, which I think they are15:16
johnthetubaguycdent: it was more your patches reminded my about the niggle I spotted debugging that bug the other day15:17
*** pcaruana has joined #openstack-nova15:17
*** mlavalle has joined #openstack-nova15:17
cdentlet's make pumaranikar fix it :)15:17
*** eharney has joined #openstack-nova15:18
johnthetubaguygood idea15:18
johnthetubaguypumaranikar: we found you another placement bug15:18
EmilienMmriedem: good morning, regarding the issue with Nova in WSGI in TripleO - does it make sense we stop doing that in Ocata (and revisit later)?15:19
mriedemso on the tripleo failures from yesterday, with a run on my debug patch, i see the instance that heat is trying to create fails15:19
mriedemhttp://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-conductor.txt.gz#_2017-02-03_05_17_35_82615:19
mriedemEmilienM: good timing15:19
EmilienMwow15:19
mriedemscheduler couldn't find a host15:19
EmilienMdo you want me to stop deploing nova api in wsgi with apache and see if it helps?15:19
*** amotoki has joined #openstack-nova15:20
mriedemgiven we haven't really flushed out testing with that, i think it's prudent at this point in ocata15:20
EmilienMok let me do it15:20
*** sree has quit IRC15:20
mriedemthe 404 isn't because the instance was created and we just can't find it though, it's not found because it fails to build and the instance isn't created (it looks like), but the dead instance should go into cell0 and we should pull it from cell015:21
mriedemhrm, dansmith melwitt do we even look in cell0 for compute api get()?15:21
*** marst has quit IRC15:21
cdentjohnthetubaguy, pumaranikar It can probably be built right in with the json_formatter stuff because it's all of a piece15:21
melwittmriedem: we should, I think. do we not? checking15:22
dansmithmelwitt: yes15:22
dansmither, mriedem yes15:22
dansmithmriedem: the mapping tells us what cell to look in15:22
melwittoh yeah15:22
dansmithwhich would be cell0 if it's there15:22
*** nkrinner is now known as nkrinner_afk15:23
*** prateek has quit IRC15:23
*** prateek_ has quit IRC15:23
mriedemwhat's odd is in this debug run i expect to see the 'found x cells' thing from https://review.openstack.org/#/c/428360/2/nova/compute/api.py@252615:24
mriedembut it's not in http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-api.txt.gz15:24
kashyapjohnthetubaguy: "But asking for a friendlier looking supernova seems a bit... unscientific" :-)15:24
johnthetubaguykashyap: :)15:24
kashyapMeanwhile, if there are any Virtualization maintainers of Ubuntu, you may want to triage this - https://bugs.launchpad.net/nova/+bug/164391115:25
openstackLaunchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed]15:25
kashyaplease see my last comment there.15:25
melwittmriedem: I thought we had found there weren't any 'GET .../servers' in those logs15:25
kashyap(s/lease/Please/)15:25
mriedemmelwitt: my debug patch also did the cells thing in _get_instance()15:26
melwittoh, you added logging for _get_instance15:26
*** amotoki has quit IRC15:26
mriedemmelwitt: http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-api.txt.gz#_2017-02-03_05_17_30_873 is the GET15:26
mriedembut it's not hitting the logging i added here https://review.openstack.org/#/c/428360/2/nova/compute/api.py15:27
*** owalsh-brb is now known as owalsh15:28
mriedemso i'm thinking we're trying to pull it from a cell but it's not there, which would be nuts, but i'll add some more logging15:28
mriedemoh well i guess that would make sense if we pull the instance here https://review.openstack.org/#/c/428360/2/nova/compute/api.py@226915:28
mriedembecause the service version is 015:29
melwittyeah, I was about to say, we had found bc they're running through apache they're getting service version 0, or something15:29
mriedemyup,15:31
mriedemAND15:31
mriedemhttp://logs.openstack.org/15/428415/1/experimental/gate-tripleo-ci-centos-7-nonha-multinode/4a76477/15:31
mriedemis dan's workaround patch15:32
EmilienMmriedem: https://review.openstack.org/428778 - once it pass our CI, I'll ask you you Depends-On this one, so you can continue the tests without WSGI in Apache15:33
*** amoralej|lunch is now known as amoralej15:34
mdboothmriedem: Thanks for looking at that NFS patch, btw. You found 2 bugs in it :)15:36
mriedemmdbooth: but you told me it was rock solid!15:36
*** burgerk has joined #openstack-nova15:37
mdboothmriedem: I was thinking a good limestone.15:37
mdboothYou know, pretty solid.15:37
mdbooth;)15:37
mriedemheh15:37
*** psachin has quit IRC15:37
mdboothBoth would result in mounts not being unmounted, which is the opposite of the original problem of mounts being unmounted when they shouldn't be.15:38
*** dave-mccowan has quit IRC15:40
*** jaosorior has joined #openstack-nova15:40
*** tbachman has quit IRC15:40
*** ratailor has quit IRC15:40
*** tbachman has joined #openstack-nova15:41
*** marst has joined #openstack-nova15:41
*** vsaienko has quit IRC15:42
*** raunak has joined #openstack-nova15:43
*** salv-orlando has joined #openstack-nova15:44
*** dimtruck is now known as zz_dimtruck15:48
*** dave-mccowan has joined #openstack-nova15:53
*** tbachman has quit IRC15:55
*** raunak has quit IRC15:56
*** annegentle has joined #openstack-nova15:59
*** mlakat has quit IRC16:02
*** crushil has quit IRC16:06
*** crushil has joined #openstack-nova16:06
*** stvnoyes has quit IRC16:06
*** hongbin has joined #openstack-nova16:06
EmilienMmriedem: FYI https://review.openstack.org/#/q/topic:bug/1661360+owner:%22Emilien+Macchi+%253Cemilien%2540redhat.com%253E%2216:07
*** stvnoyes has joined #openstack-nova16:07
*** mvk has joined #openstack-nova16:07
*** tbachman has joined #openstack-nova16:08
*** BlackDex has quit IRC16:09
mdboothmriedem: On second thoughts, only 1 bug :) The other 1 just looks sufficiently weird that I misread it, too.16:09
*** zz_dimtruck is now known as dimtruck16:09
*** yamahata has joined #openstack-nova16:10
mriedemEmilienM: good lord that's a lot of change16:11
mriedembut i don't know puppet or tripleo16:11
*** baoli has quit IRC16:13
pumaranikarjohnthetubaguy, cdent sry i missed initial conversation16:13
johnthetubaguypumaranikar: no worries, was going to ping you16:14
johnthetubaguypumaranikar: thinking about the errors produced by the placement API, they have lots of big new lines in them16:14
johnthetubaguypumaranikar: it would be great if you could raise a bug for that and fix it16:14
johnthetubaguypumaranikar: seems like the json formatter needs a tweak to fix that up16:15
*** BlackDex has joined #openstack-nova16:16
*** BlackDex has quit IRC16:16
*** kristian__ has quit IRC16:17
EmilienMmriedem: it was just an fyi16:17
pumaranikarjohnthetubaguy, sure i will check it.16:17
*** BlackDex has joined #openstack-nova16:17
mriedemEmilienM: i made one very constructive comment on one patch16:17
johnthetubaguypumaranikar: thanks16:17
*** Oku_OS is now known as Oku_OS-away16:18
pumaranikarjohnthetubaguy, regarding json_formatter, should we try again with middleware or go with decorated function. what you suggest ?16:19
*** baoli has joined #openstack-nova16:19
*** BlackDex has quit IRC16:20
*** armax has joined #openstack-nova16:20
*** dansmith is now known as superdan16:21
openstackgerritEd Leafe proposed openstack/nova master: Add check for invalid allocation amounts  https://review.openstack.org/41675216:21
openstackgerritEd Leafe proposed openstack/nova master: Add check for invalid inventory amounts  https://review.openstack.org/42880016:21
figleafcdent: split the inventory and allocation changes ^^16:21
cdentthanks figleaf16:22
*** jaypipes is now known as leakypipes16:22
*** panda is now known as panda|bbl16:23
*** kristian__ has joined #openstack-nova16:23
*** kristian__ has quit IRC16:24
cdentpumaranikar, johnthetubaguy I'd prefer middleware if we can make it work, but I haven't thought about it enough to know it can. We would need to make sure that it only engaged (processed the body) when it absolutely had to. The main issue is that anything happening "outside" of webob.dec.wsgify will not see an exception, only a response16:24
*** mjura has quit IRC16:24
*** tbachman has quit IRC16:24
*** nmathew has quit IRC16:25
*** BlackDex has joined #openstack-nova16:25
*** kristian__ has joined #openstack-nova16:28
*** tbachman has joined #openstack-nova16:29
*** osallou has joined #openstack-nova16:31
osallouhi, i have activated nova notifications and connected multiple subscribes to nova exchange, however a notifications.info queue is always created and is filled indefinitly is no subscribed to drain it, is it a configuration error? I need multiple subscribers on notifications, so I cannot simply listen to the queue... thanks16:33
*** bnemec is now known as beekneemech16:33
*** mjura has joined #openstack-nova16:33
*** baoli has quit IRC16:34
*** zhurong has quit IRC16:34
*** tbachman has quit IRC16:35
*** mat128 is now known as mat128|mtg16:35
*** dtp has joined #openstack-nova16:36
*** osallou has left #openstack-nova16:37
*** baoli has joined #openstack-nova16:38
*** claudiub|2 has quit IRC16:38
pumaranikarcdent, johnthetubaguy yes. I think for every resource based action we have added webob.dec.wsgify16:39
EmilienMmriedem: http://logs.openstack.org/86/428786/1/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/522b390/logs/nova/nova-api.txt.gz#_2017-02-03_16_15_23_47016:41
EmilienMmriedem: should it be ::1:8774?16:41
*** chyka has joined #openstack-nova16:42
*** tbachman has joined #openstack-nova16:44
mriedemand remove the []16:45
mriedem?16:45
mriedemidk16:45
mriedemis that ipv6?16:45
*** mdnadeem has quit IRC16:49
*** lpetrut has quit IRC16:50
*** kristia__ has joined #openstack-nova16:51
cdentpumaranikar: sorry was on the phone. yes that's right, thus the difficulty16:51
*** mtanino has quit IRC16:53
*** mriedem has quit IRC16:53
*** mtanino has joined #openstack-nova16:53
*** kristian__ has quit IRC16:53
*** mtanino has quit IRC16:53
*** mtanino has joined #openstack-nova16:53
*** mriedem has joined #openstack-nova16:54
openstackgerritMatthew Booth proposed openstack/nova master: libvirt: Fix races with nfs volume mount/umount  https://review.openstack.org/38385916:54
*** satyar has joined #openstack-nova16:55
openstackgerritMaciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs  https://review.openstack.org/42881116:56
pumaranikarcdent, i will try with middleware and update you16:58
openstackgerritMaciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs  https://review.openstack.org/42881116:58
*** mdrabe has quit IRC17:03
johnthetubaguypumaranikar: we did that yesterday right, and it failed with middlewear?17:03
johnthetubaguycdent: the exception just didn't seem to get as fair as that fault wrapper middleware17:04
johnthetubaguyit was already converted into a response with the correct error code, at least is seemed that way17:04
pumaranikarjohnthetubaguy, yes. because we were getting valid response17:04
cdentyeah, if you use middleware, it has to be body processing instead of exception inspecting17:04
johnthetubaguybut the body doesn't have the data present that was in the exception I guess?17:05
cdentif the response ststus >= 400 and <500, then if it is json, then reformat it to the structure we want17:05
johnthetubaguycdent: my suggestion to pumaranikar was to replace the wsgify with our own decorator, that ensure the exception gets the formatter added, its not ideal though17:05
johnthetubaguyit seems easier to remember that adding the formatter to the error message, but that might just be me17:06
cdentif there's a way to do that doesn't involve double decorating that might work17:06
* cdent looks at webob code17:06
johnthetubaguyyeah, its just the one decorator, I pushed that up yesterday into pumaranikar's patch17:06
cdentthat's what I mean by double decorator17:07
mriedemartom: on https://blueprints.launchpad.net/nova/+spec/fix-tag-attribute-disappearing i think i'm going to just mark that as partially completed and then we'll finish the rest in pike17:07
cdentI think it would be better to subclass webob.dec.wsgify (which is a class)17:07
johnthetubaguycdent: its not the nicest thing, but it seems to work: https://review.openstack.org/#/c/395194/5/nova/api/openstack/placement/handlers/allocation.py@27417:07
artommriedem, yep :)17:07
johnthetubaguycdent: yeah, thats nicer17:08
johnthetubaguypumaranikar: does that make sense?17:08
cdentexcept, unfortunately the part that one would want to override is smack in a tall stack of code :(17:08
artommriedem, it's debatable whether the changes to tests are parts of the spec proper, in any case17:08
johnthetubaguycdent: ah, doh, anyways my hack appeared to work, we could always make that nicer later17:09
cdenthmm, maybe override call_func17:09
johnthetubaguycdent: did you see this way: https://review.openstack.org/#/c/395194/5/nova/api/openstack/placement/handlers/allocation.py@274 ?17:09
cdentpumaranikar: have a look at the wsgify class in webob/dec.py, may be some ideas17:09
cdentjohnthetubaguy: yeah, saw that17:10
cdentI think that will work, but it feels icky for a reason I can't put my finger on17:10
johnthetubaguycdent: totally with you on that17:11
pumaranikarcdent, looking at wsgify class.17:11
*** raunak has joined #openstack-nova17:11
cdentit might, however, be less messy than what I'm thinking. won't really know until we try it17:11
johnthetubaguycdent: +117:11
cdentpumaranikar: I think overriding the call_fun method might work17:11
*** mdrabe has joined #openstack-nova17:11
*** nic has joined #openstack-nova17:13
*** mjura has quit IRC17:14
*** raunak has quit IRC17:15
johnthetubaguycdent: I was curious on your gabbi thoughts too, this seemed the simplest way to ensure the formatter was present: https://review.openstack.org/#/c/395194/5/nova/tests/functional/api/openstack/placement/gabbits/allocations.yaml@15817:15
cdentjohnthetubaguy: yeah, pretty much that's it. look for a key and value that wouldn't be there if the formatter was not swinging17:16
*** mvk has quit IRC17:17
*** kfarr has joined #openstack-nova17:17
EmilienMmriedem: yes it's IPv6. Do you test nova in ipv6?17:18
EmilienMosapi_compute_listen=[::1] doesn't seem to work17:18
mriedemEmilienM: not sure off hand17:19
mriedemsdague or sc68cal might know17:19
EmilienMI'll try with ::117:19
EmilienMmriedem: triple jobs have ipv6 coverage, fyi17:20
*** sambetts is now known as sambetts|afk17:20
openstackgerritDan Smith proposed openstack/nova master: Update the upgrades part of devref  https://review.openstack.org/42882417:21
EmilienMmriedem: btw, I saw your comment on https://review.openstack.org/#/c/428424/ - and I think we should have the job asap since things are broken for us, it would be helpful on both sides to have the job in place without having to run "check experimental".17:21
superdanmriedem: see my doc update for the upgrades.rst just now17:23
superdanmriedem: I was explaining that process to someone recently and wanted to RTFM them to that doc17:24
*** xyang1 has quit IRC17:24
superdanmriedem: but then realized it needed some serious freshening17:24
*** annegentle has quit IRC17:25
*** xyang1 has joined #openstack-nova17:25
*** annegentle has joined #openstack-nova17:28
*** browne has joined #openstack-nova17:30
*** derekh has quit IRC17:30
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: utils.get_disk return the target dev of the root device  https://review.openstack.org/40271617:31
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Make guest.BlockDevice only use the target dev of a disk  https://review.openstack.org/40216317:31
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Wait for abort_job to complete when not resizing  https://review.openstack.org/38244917:31
sc68calmriedem: EmilienM: [::1]:<port>17:32
johnthetubaguysuperdan: I will take a peak at that upgrade doc, looks interesting17:32
superdanjohnthetubaguy: thanks17:32
sc68calmriedem: EmilienM: so I guess [::1]:8774 is what you're looking for17:32
EmilienMsc68cal: yeah? we had osapi_compute_listen=host until now17:33
EmilienMand osapi_compute_listen_port for the port17:33
sc68calok, then osapi_compute_listen=::1 should work17:35
EmilienMsc68cal: http://git.openstack.org/cgit/openstack/nova/tree/nova/conf/service.py#n9617:35
EmilienMsc68cal: ok I'm trying it, thanks17:35
EmilienMsc68cal: FYI we add brackets because some other services need it17:35
sc68calhang on let me poke it with netcat17:36
EmilienMand the best practise in IPv6 is to have brackets17:36
EmilienMtbh, I think having [::1] rather than ::1 is much safer17:36
sc68caloh yes much safer17:36
*** lucasagomes is now known as lucas-afk17:36
*** raunak has joined #openstack-nova17:37
*** dave-mccowan has quit IRC17:38
*** mkrai has left #openstack-nova17:41
*** kristian__ has joined #openstack-nova17:41
*** vsaienko has joined #openstack-nova17:41
sc68calsome services want brackets, others don't17:42
*** ducttape_ has quit IRC17:42
*** kristia__ has quit IRC17:44
sc68calare we passing the correct options to eventlet.listen? there's an optional kwarg for address family17:44
sc68calit might be defaulting to v4 and trying to resolve ::1 as a dns name17:45
johnthetubaguysuperdan: I thought SIG_HUP was enough after the initial restart?17:45
johnthetubaguysuperdan: oh, missread, I see your point now17:46
superdanjohnthetubaguy: not for static configs, yeah17:47
superdanthat subtle detail could have been pretty annoying for someone :)17:47
johnthetubaguyyeah, totally17:48
*** bvanhav_ has joined #openstack-nova17:50
superdanjohnthetubaguy: I was tempted to put "read the effing release notes" as a formal step in there :)17:51
*** bvanhav has quit IRC17:52
johnthetubaguysuperdan: step 2, no really, read them.17:52
openstackgerritEd Leafe proposed openstack/nova master: Add check for invalid inventory amounts  https://review.openstack.org/42880017:53
*** Swami has joined #openstack-nova17:53
superdanjohnthetubaguy: heh, right17:53
figleafcdent: ^^ Fixed17:53
*** yamahata has quit IRC17:53
*** kristian__ has quit IRC17:53
cdentfigleaf: ossum17:54
*** lpetrut has joined #openstack-nova17:54
openstackgerritAnusha Unnam proposed openstack/nova master: Reno for additional-notification-fields-for-searchlight  https://review.openstack.org/42883517:54
*** EmilienM has quit IRC17:56
*** scottynomad has quit IRC17:56
*** Kimmo_ has quit IRC17:56
*** mrodden has quit IRC17:56
*** tonyb has quit IRC17:56
*** jamespd has quit IRC17:56
*** mikal has quit IRC17:56
*** rodrigods has quit IRC17:56
*** Tahvok has quit IRC17:56
*** gmann has quit IRC17:56
*** gibi has quit IRC17:56
*** Nakato has quit IRC17:56
*** mkoderer___ has quit IRC17:56
*** d34dh0r53 has quit IRC17:56
*** Anticimex has quit IRC17:56
*** s1rp has quit IRC17:56
*** alex_xu has quit IRC17:56
*** tonyb has joined #openstack-nova17:56
*** mikal has joined #openstack-nova17:56
*** rodrigods has joined #openstack-nova17:56
*** gibi has joined #openstack-nova17:56
*** jamespd has joined #openstack-nova17:56
*** scottynomad has joined #openstack-nova17:56
*** Kimmo_ has joined #openstack-nova17:56
*** s1rp has joined #openstack-nova17:56
*** Tahvok has joined #openstack-nova17:56
*** andreaf has quit IRC17:56
*** aries has quit IRC17:56
*** agarner has quit IRC17:56
*** shaohe_feng has quit IRC17:56
*** devananda has quit IRC17:56
*** smcginnis has quit IRC17:56
*** kevinbenton has quit IRC17:56
*** jotr has quit IRC17:56
*** Madasi has quit IRC17:56
*** amrith has quit IRC17:56
*** d34dh0r53 has joined #openstack-nova17:56
*** Anticimex has joined #openstack-nova17:56
*** mrodden has joined #openstack-nova17:56
*** EmilienM has joined #openstack-nova17:56
*** jowisz_ has quit IRC17:56
*** lyarwood has quit IRC17:56
*** jgriffith has quit IRC17:56
*** zer0c00l has quit IRC17:56
*** scottda has quit IRC17:56
*** zer0c00l has joined #openstack-nova17:56
*** lyarwood has joined #openstack-nova17:56
*** jowisz has joined #openstack-nova17:56
*** smcginnis has joined #openstack-nova17:56
*** alex_xu has joined #openstack-nova17:56
*** devananda has joined #openstack-nova17:56
*** mkoderer__ has joined #openstack-nova17:56
*** amrith has joined #openstack-nova17:56
*** dave-mccowan has joined #openstack-nova17:57
*** aries has joined #openstack-nova17:57
*** Nakato has joined #openstack-nova17:57
*** gmann has joined #openstack-nova17:57
*** kevinbenton has joined #openstack-nova17:57
*** jgriffith has joined #openstack-nova17:57
*** andreaf has joined #openstack-nova17:58
*** Madasi has joined #openstack-nova17:58
*** jose-phillips has joined #openstack-nova17:59
*** aleph1 has joined #openstack-nova17:59
*** DuncanT has quit IRC17:59
*** test222 has quit IRC17:59
sc68calEmilienM: so looks like we had a netsplit18:00
sc68calEmilienM: where did I lose you18:00
*** jotr has joined #openstack-nova18:01
*** DuncanT has joined #openstack-nova18:01
EmilienMsc68cal: better: http://logs.openstack.org/86/428786/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d244aec/logs/nova/nova-api.txt.gz#_2017-02-03_17_44_34_09718:01
EmilienMsc68cal: so using ::1 works :-/18:01
*** bvanhav_ is now known as bvanhav18:01
*** ralonsoh has quit IRC18:02
sc68calEmilienM: yep..... we have same issue with mysql vs alembic/sqlchemy18:02
aunnamgibi, if possible can you review this patch https://review.openstack.org/#/c/428835/18:02
*** shaohe_feng has joined #openstack-nova18:03
sc68calEmilienM: https://github.com/openstack-dev/devstack/blob/master/lib/database#L7718:03
*** test222 has joined #openstack-nova18:03
*** scottda has joined #openstack-nova18:04
*** Apoorva has joined #openstack-nova18:04
openstackgerritBob Ball proposed openstack/nova master: Fix live migrate with XenServer  https://review.openstack.org/42442818:05
BobBalljohnthetubaguy: ^^ Hopefully that'll fix the different network concern18:05
*** alex_xu has quit IRC18:06
*** sudipto has quit IRC18:07
*** sudipto_ has quit IRC18:07
*** alex_xu has joined #openstack-nova18:08
*** ducttape_ has joined #openstack-nova18:09
cdentmriedem, johnthetubaguy: the tagging has not happened has it?18:09
cdentwriting up some notes and wanted to know if https://review.openstack.org/#/c/428375/ was part of the now-ish release18:10
johnthetubaguyhttps://review.openstack.org/#/c/428531/ says we haven't tagged yet, looks like we can include the ironic one if we wish18:11
*** xinli has joined #openstack-nova18:16
mriedemcdent: not yet18:26
mriedembut now that https://review.openstack.org/#/c/428375/ is merged and i'm back from lunch, i'm going to rev the rc1 tag patch18:26
cdentmriedem: how very exciting18:27
*** annegentle has quit IRC18:30
mriedemi know, i'm cutting glass over here18:31
*** kristian__ has joined #openstack-nova18:31
superdanlol18:33
openstackgerritSujitha proposed openstack/nova master: Reno for additional-notification-fields-for-searchlight  https://review.openstack.org/42883518:33
*** vsaienko has quit IRC18:33
* johnthetubaguy makes smashing sounds18:33
*** yamahata has joined #openstack-nova18:34
johnthetubaguymriedem: I am guessing there is no way to backport this one, if it misses RC1 right? https://review.openstack.org/#/c/42442818:34
mriedemjohnthetubaguy: not with the object version change,18:35
mriedembut there is rc218:35
johnthetubaguymriedem: what if we just get the object change in?18:35
mriedemheh, so i thought about that yesterday18:36
johnthetubaguymriedem: but rc2 is a backport I guess18:36
mriedemyes rc2 is a backport to stable/ocata technically18:36
mriedembut it'd also be pre-release18:36
superdanjohnthetubaguy: mriedem it's okay if the versions have no gap, and I think this is sufficiently defensive18:36
superdanI mean, it's technically okay if there's no gap18:36
*** kristian__ has quit IRC18:36
johnthetubaguysuperdan: oh, true, the likely hood of moving from stable to a commit without that commit seems fairly tiny18:37
superdanjohnthetubaguy: the biggestdeal is if you happen to get conductor on rc1 and compute on rc218:37
superdanhowever,18:37
superdanI think rcN is really to make sure that rc1 was what we wanted18:38
superdanmechanically it's a backport,18:38
superdanbut I can't really think the semantics can be considered the same18:38
johnthetubaguyyeah, thats fair18:38
*** tesseract has quit IRC18:38
mriedemso..are we saying we can backport the whole thing for rc2, or do we hold up rc1 for just the object version change and then backport the rest?18:39
johnthetubaguynot hit an object change in an RC potential bug before, I guess it must have happened18:39
johnthetubaguymriedem: the first one I think18:39
mriedemsuperdan: you agree? ^18:39
superdanmriedem: yeah, all in rc218:39
johnthetubaguyits a driver specific object anyways, so the risk is lower I guess18:40
superdanthis is the thing that is not a regression in ocata though right?18:40
johnthetubaguyit might have been newton, not sure actually18:40
EmilienMmriedem: can you revisit https://review.openstack.org/#/c/428424/ please?18:41
mriedemjohnthetubaguy: superdan: BobBall said it was a regression in newton18:42
superdanyeah, thought so18:42
*** prateek_ has joined #openstack-nova18:42
*** prateek has joined #openstack-nova18:42
*** Sukhdev has joined #openstack-nova18:45
EmilienMmriedem: wow so https://review.openstack.org/#/c/428415/ passed for us?18:47
EmilienMmriedem: but fail on our baremetal jobs: http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/console.html#_2017-02-03_01_43_02_87230918:48
mriedemand fails a ton of tests in nova18:48
mriedemand we don't really know what else would break with it18:48
mriedemsince we have 0 burn in time18:48
*** Hosam has joined #openstack-nova18:49
EmilienMmriedem: this one looks strange: http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_02_02018:49
mriedemyjs18:50
mriedemthat's using ironic?18:50
EmilienMmriedem: yes18:50
*** cleong has quit IRC18:50
*** slaweq has quit IRC18:50
*** burt has quit IRC18:50
mriedemhmm, that's odd. something not working in the resource tracking, but that stuff should come in when nova-compute starts up,18:52
mriedemunless nova-compute is failing to get info from ironic18:52
mriedemTotal disk: 40 GB, used: 40.00 GB18:52
mriedemright above that18:52
EmilienMcould it be nova/wsgi again?18:52
mriedemand just before that Total disk: 40 GB, used: 0.00 GB18:53
superdanmriedem: unless nova-compute starts before any ironic nodes are provisioned18:53
superdanmriedem: also, they stop reporting resources for nodes that are in the middle of a change18:53
superdanmriedem: so they will go in and out and in and out over the course of a normal lifetime18:53
superdanit's weird18:53
EmilienMsuperdan: in the case opposite happens, nova-ocmpute is able to update nodes resources?18:53
mriedemthere is an instance created right before that one that claims all the disk http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_01_05218:53
superdanbut jroll and I were looking at that the other day18:53
EmilienMlet me look in logstash if tripleo ci had this error before18:54
mriedemEmilienM: logstash is super backed up18:54
jrollI didn't do it18:54
mriedemso probably no results there18:54
mriedemplus and http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_02_020 is debug level which we don't index18:55
EmilienMmriedem: over the last 30 days, one occurence, on 427536 changfe18:56
mriedemwe only store 10 days of logs18:56
mriedemand it's been busted for several days18:56
EmilienMmriedem: I think it's pretty clear this patch is responsible18:56
EmilienMeven 10 days18:56
mriedemwell the patch is https://review.openstack.org/#/c/427536/18:56
EmilienMmriedem: http://tripleo.org/cistatus.html we run so many jobs per day, we would have seen it before18:56
mriedem2 deps18:56
mriedemi don't see how https://review.openstack.org/#/c/427782/ would cause this18:57
mriedemso what does https://review.openstack.org/#/c/427534/ do?18:57
mriedemmy money is on ^18:57
mriedem"Ensure that the ironic nodes have been picked up by the nova resource tracker before running nova-manage cell_v2 host discovery."18:57
EmilienMowalsh: you around? ^18:57
superdan427782 is only about getting instances via the api, really nothing to do with any of that18:57
mriedemright18:57
mriedemthe latter is specifically about when to discover ironic hosts for resource tracking18:58
superdanmriedem: right, and it is important because they weren't enrolling nodes by the time they ran discover18:58
superdanwithout it they're 100% fail because they never have host mappings18:59
mriedemeven though, regardless of all those changes,18:59
mriedemhttp://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_01_04918:59
mriedemAttempting claim: memory 4096 MB, disk 40 GB, vcpus 1 CPU18:59
openstackgerritJohn Perkins proposed openstack/nova master: Integrate oslo.config validator  https://review.openstack.org/42897418:59
mriedemTotal disk: 40 GB, used: 0.00 GB18:59
mriedemso that instance claims all of the disk18:59
mriedemwhich is why the next one fails18:59
mriedemunless, there should be more nodes reporting in18:59
mriedemi.e. we should have more than 40GB total19:00
superdanor the scheduler raced19:00
cdentwhose 40GB is that? Is there a chance ironic is now trying to claim the wrong disk? (we are talking ironic here aren't we?)19:00
mriedemyes ironic19:01
mriedem2017-02-03 01:05:33.326 28176 DEBUG nova.virt.ironic.driver [req-4a6f8f71-565c-4379-8288-347b8e9da028 - - - - -] Returning 3 available node(s) get_available_nodes /usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py:61019:03
mriedemHypervisor/Node resource view: name=7c2a1870-c2f8-46d5-b1a4-c72443bfc82f free_ram=8192MB free_disk=40GB free_vcpus=4 pci_devices=None19:03
mriedemFinal resource view: name=7c2a1870-c2f8-46d5-b1a4-c72443bfc82f phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[]19:03
mriedemFinal resource view: name=aa97370e-02a2-4aa9-8833-616d30a4d589 phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[]19:04
mriedemFinal resource view: name=6a085a83-e051-480b-bf37-d092d7bbf13e phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[]19:04
mriedemso those are the 3 nodes19:04
mriedem40GB each19:04
superdanright and ironic doesn't sum the disks I think19:04
superdanto prevent you from trying to schedule a 120G instance to that compute right/19:04
mriedemso 1 instance should be placed on 1 node right?19:04
*** salv-orlando has quit IRC19:05
superdanyep19:05
superdanI might be thinking of the summing in the api bit19:05
superdananyway, three nodes, 40g each19:05
mriedemwould be nice if the "Claim successful" message told us which node the resources were claimed on19:05
mriedemin this log it's building a few instances19:05
mriedemand it looks like one hits on a node that's used19:05
*** annegentle has joined #openstack-nova19:06
openstackgerritJohn Perkins proposed openstack/nova master: Integrate oslo.config validator  https://review.openstack.org/42897419:06
superdani.e. scheduler race19:06
superdanaka that thing we say is explicitly out of scope: scheduling to 100% capacity19:06
openstackgerritEd Leafe proposed openstack/nova master: placement: RT now adds proper Ironic inventory  https://review.openstack.org/40447219:07
*** salv-orlando has joined #openstack-nova19:08
openstackgerritOpenStack Release Bot proposed openstack/nova master: Update reno for stable/ocata  https://review.openstack.org/42898319:10
*** unicell has joined #openstack-nova19:10
openstackgerritMatt Riedemann proposed openstack/nova master: Add nodename to _claim_test log messages  https://review.openstack.org/42898519:12
cdentfigleaf, johnthetubaguy: Is https://review.openstack.org/#/c/404472/ no longer targetted for some version of ocata, or is it still under consideration?19:13
mriedemcdent: i'm not considering it unless johnthetubaguy is super comfortable with it,19:14
mriedemi've been out of the loop on that one since last week19:14
mriedemas discussed a bit in here yesterday, i think we can do that in pike,19:14
mriedemand put a service version on it so that the scheduler could know when it can start making requests with custom resource classes19:15
cdentI think there are still some outstanding questions on it, maybe addressed by figleaf just now, so it does feel a bit late19:15
*** crushil has quit IRC19:15
mriedemin the ironic host manager scheduler thing19:15
johnthetubaguyyeah, the questions don't all seem answered yet19:15
johnthetubaguyI think we update the allocations more frequently than I first though, but I don't quite understand how the old allocations get refreshed19:16
*** sdague has quit IRC19:16
johnthetubaguyfigleaf: ^19:16
cfriesenI've got an idea for an enhancement, wondering if it's worth writing up a spec.  Basically the idea is that currently when booting the instance we allow "--user-data" and "--file", but when you rebuild the instance we only support "--file".  So I'd like to propose adding support for "--user-data" when doing a rebuild.19:17
johnthetubaguyit also seems odd to rebuild the inventory for the resource provider in two places19:17
cfriesenIs this a dumb idea that'd get shot down immediately, or is it worth writing up?19:17
johnthetubaguycfriesen: seems worth adding a spec, make sense I think, config drive will be recreated either way, so why not I guess19:18
figleafjohnthetubaguy: I'm not very clear on what part of this is confusing to you. I guess the fact that a lot of it is confusing is an issue in and of itself.19:19
*** eharney has quit IRC19:19
openstackgerritMaciej Szankin proposed openstack/nova master: conf: deprecate key_manager.fixed _key opt  https://review.openstack.org/42899219:19
mriedemcfriesen: didn't this come up recently-ish in the ML?19:20
mriedemcfriesen: something about injected files not persisting across a rebuild?19:20
cdent"I guess the fact that a lot of it is confusing is an issue in and of itself."++19:20
mriedemor evacuate maybe19:20
melwittyeah, I dunno what the reasons might be for not allowing it, but I have recalled mikal having the opinion that config drive is a once-at-init thing and not creating/changing it after instance boot19:21
mriedemcfriesen: which was tied to a thread i had about deprecating/removing file injection19:21
melwittso I feel like he's the one to talk to19:21
cfriesenmriedem: yes, userdata currently gets lost over a rebuild19:21
johnthetubaguyfigleaf: how do all the old instance allocations get refreshed, basically19:21
melwittit doesn't get lost, it stays on the config drive. or it should19:21
mriedemi was thinking of the injected files / evacuate thing specifically i think19:21
cfriesenmreidem: are you talking about removing --userdata from boot as well?19:21
johnthetubaguyfigleaf: I see how the new ones get the correct allocations19:21
mriedemcfriesen: no19:22
johnthetubaguyfigleaf: I guess its possible when nova-compute first starts?19:22
mriedemjust --files19:22
mriedemor personality or whatever19:22
mriedemlet me see if i can find the thread19:22
johnthetubaguymriedem: files will get lost, it only appears in the initial config drive, we don't store them19:22
mriedemjohnthetubaguy: yup i know19:23
cfriesenmriedem: right.  so currently we don't allow specifying files on an evacuate, but they are preserved19:23
mriedemit was one of the reasons for killing file injection19:23
*** amotoki has joined #openstack-nova19:23
mriedemcfriesen: http://lists.openstack.org/pipermail/openstack-dev/2016-November/107195.html19:23
*** vsaienko has joined #openstack-nova19:23
cfriesenmriedem: preserved in the config drive, right19:23
johnthetubaguycfriesen: well, depends if you are using shared storage I guess19:24
*** bvanhav has quit IRC19:24
cfriesenmriedem: yes19:24
cfriesenmridem: so if we're still going to allow --user-data on boot, I think it'd make sense to allow it on rebuild19:24
figleafjohnthetubaguy: the _init_compute_node is one place, yes19:24
*** bvanhav has joined #openstack-nova19:24
johnthetubaguyfigleaf: because tracked instances is empty to start with, so everything is treated as a new instance?19:24
mriedemcfriesen: yeah i think that's fine19:25
figleafjohnthetubaguy: then self._update_usage_from_instances keeps them in sync19:25
melwittcfriesen: I'm still stuck on the part where you say userdata gets wiped out. it didn't use to on rebuild, it should just stay on the config drive?19:25
melwittthis came up before at my last company and we tested it and userdata stayed intact across a rebuild19:25
figleafjohnthetubaguy: yeah, that happens in _init_compute_node19:25
mriedem14:20:38 <johnthetubaguy> #action deprecate localfs file injection in ocata, remove in pike19:25
cfriesenmelwitt: currently injected files get lost on evacuate19:25
mriedemdamn!19:25
melwitthmm19:26
*** big_macsz has joined #openstack-nova19:26
cfriesenmelwitt: currently userdata stays intact over a rebuild.  my usecase is for people that want to rebuild to a different image with different userdata19:26
*** vsaienko has quit IRC19:26
*** sdague has joined #openstack-nova19:26
melwittcfriesen: yeah. okay, that makes sense. I thought earlier you said something about userdata getting wiped out during a rebuild19:26
cdentfigleaf, johnthetubaguy: tracked_instances is emptied every periodic job and _init_compute_node is called every periodic job19:26
superdancfriesen: isn't that called "boot" ?19:26
figleafcdent: oh geez - really?19:27
cdentfigleaf: you read this yet: http://lists.openstack.org/pipermail/openstack-dev/2017-January/110953.html19:27
cfriesensuperdan: you would think19:27
*** dharinic is now known as dharinic|lunch19:27
EmilienMsuperdan: do you think I need to file a bug on tripleo side for the ironic thing? (wrt the patch in tripleo-common owalsh did)19:27
cfriesensuperdan: as I recall it ended up being related to heat updates19:27
figleafcdent: yes, but I don't have it committed to memory :)19:27
cdentfigleaf: oh, but you should19:28
figleafcdent: Ok, I see the tracked_instances.clear() line19:28
sdagueEmilienM: any behavior in the nova osapi_compute_listen field is because of the way oslo.service works - https://github.com/openstack/oslo.service/blob/7fa34bf8e7a37a6d9355aaf55aa85b1931567cd6/oslo_service/wsgi.py#L127-L15119:28
cdentfigleaf: it's all very bewildering and amazing19:28
*** amotoki has quit IRC19:28
superdanEmilienM: no19:28
EmilienMsdague: ok I see, thanks19:28
johnthetubaguycdent: I guess thats where all those fetch inventory calls come from, from the attempted update19:30
johnthetubaguycdent: hmm, I meant allocations19:30
cdentI'm gonna escape for the evening unless there's something needing me?19:31
johnthetubaguycdent: I need to do the same really19:31
johnthetubaguyI have the oven on19:31
figleafHave a great weekend, cdent and johnthetubaguy19:31
*** vsaienko has joined #openstack-nova19:33
cdentjohnthetubaguy: yeah, dinner calls19:33
* cdent bows19:33
*** prateek_ has quit IRC19:33
*** cdent has quit IRC19:33
*** prateek has quit IRC19:33
big_macszbon appetit19:35
cfriesenmriedem: did anyone actually start on a spec for deprecating file injection?19:35
mriedemcfriesen: no it was on my todo list at some point and then i forgot about it19:36
mriedemi blame the holidays19:36
*** vsaienko has quit IRC19:36
cfriesenheh...so are we now looking at deprecating in pike?19:36
cfriesenremoving in q?19:36
mriedemyeah19:36
* johnthetubaguy nods in shame19:36
mriedemjohnthetubaguy: it was on me19:36
cfriesenheh...okay, I can write that one up if you like.19:37
openstackgerritEmilien Macchi proposed openstack/nova master: DNM - test tripleo without nova api in WSGI/apache  https://review.openstack.org/42900819:37
johnthetubaguyfigleaf: that last question I had was about the inventory, it seems odd to need to update it when updating the instances, surely thats been done already in the RT?19:38
EmilienMmriedem: now rc1 is cut, can you look https://review.openstack.org/#/c/428424/ please?19:38
*** tbachman has quit IRC19:39
*** vsaienko has joined #openstack-nova19:39
figleafjohnthetubaguy: the only way inventory is updated there is if it doesn't match the number of allocations19:42
figleafjohnthetubaguy: so it removes everything and rebuilds everything to the "correct" way (old and new style)19:42
johnthetubaguyfigleaf: which is for new instances too right?19:42
figleafjohnthetubaguy: next pass through, it should find that len(cur_allocs) == len(inv_data), so it won't do that again19:43
johnthetubaguyfigleaf: but its not needed, if you look here: https://github.com/openstack/nova/blob/cab5dbfac7513d2cef1073787e49f6d063ec8fa7/nova/compute/resource_tracker.py#L57519:43
figleafjohnthetubaguy: well, what we might think are "new instances" is different19:43
johnthetubaguyfigleaf: we always update the inventory first19:43
figleafjohnthetubaguy: as cdent mentioned, the tracker is cleared on every pass, so they are always "new"19:43
johnthetubaguyyes, but thats a red herring here19:44
johnthetubaguy I mean if you boot an instance, every time we would now re-write the inventory19:44
johnthetubaguy(well attempt an update)19:44
figleafjohnthetubaguy: what we need is to map this all out visually. It's all too confusing otherwise19:46
johnthetubaguyfigleaf: that probably would help19:46
figleafjohnthetubaguy: I still trip myself up following the ironic vs. non-ironic paths19:46
johnthetubaguyfigleaf: so generally, the code updates the inventory, then updates the allocations19:47
johnthetubaguyfigleaf: I think this new code should do the same thing, rather than trying to re-update the inventory19:47
dtpreading the release notes.  do cells v1 deployments need to go through the cells v2 setup for Ocata?19:47
superdandtp: yes19:48
superdanI thought it said that explicitly19:48
dtpit does, but from discussions we had previously, i'm under the impression folks can still run v1 cells w/ the CELLS_ENABLED flag or whatever it was19:49
superdanwe've been through so many cells related renos I've lost track19:49
superdanthey can19:49
superdanbut they still have to do the cellsv2 stuff19:49
*** claudiub|2 has joined #openstack-nova19:49
dtpok, including creating a cell 1?  or just the cell 0?19:49
*** salv-orl_ has joined #openstack-nova19:49
mriedemboth19:49
mriedemdtp: also https://review.openstack.org/#/c/427981/19:49
figleafjohnthetubaguy: understood, but I'm not seeing where the problem is. I can only see that happening when the allocations don't match inventory for ironic19:51
johnthetubaguyfigleaf: no, its a bit different19:51
johnthetubaguy... trying to think of how to describe it19:52
johnthetubaguywe have the inventory update, so it includes the new resource calss19:52
johnthetubaguyclass19:52
figleafjohnthetubaguy: hey! how about writing a functional test! :)19:52
*** salv-orlando has quit IRC19:52
johnthetubaguywe now call the update19:52
figleaf(which is what we should have done from the outset)19:52
johnthetubaguyfigleaf: if it were not 7.50pm, sure19:52
johnthetubaguybasically, we start with updating the inventory19:53
johnthetubaguythen we have to check if the allocations match what should be in the inventory19:53
johnthetubaguythats not the case if its new, or we are hitting the upgrade bump19:53
johnthetubaguyin that case we need to fix up the allocations, thats all good19:54
johnthetubaguybut there is no need to update the inventory, its already been done19:54
*** vsaienko has quit IRC19:54
*** claudiub has joined #openstack-nova19:54
figleafjohnthetubaguy: so you're saying that they inventory always has to be correct? IOW, it's always the allocations that are out of whack?19:55
johnthetubaguyI am saying the inventory is always updated before we update the allocations19:55
*** dharinic|lunch is now known as dharinic19:55
*** vsaienko has joined #openstack-nova19:56
*** ducttape_ has quit IRC19:56
*** claudiub|2 has quit IRC19:57
* johnthetubaguy bravely runs away19:58
figleafjohnthetubaguy: that could well be the case19:58
figleafjohnthetubaguy: I think leakypipes must have trying the belts and braces approach19:58
figleafjohnthetubaguy: the part I'm not clear on is where all the calls to RT._update_usage_from_instance() can be made, and whether those paths always update inventory first20:01
*** tbachman has joined #openstack-nova20:01
cfriesenis there a writeup of the as-implemented behaviour for resource providers, or are the specs/code all there is currently?20:01
mriedemdidn't markus_z have a burndown chart somewhere for the config option cleanup work?20:03
figleafcfriesen: There have been several excellent posts to the ML by cdent20:03
cfriesenfigleaf: just saw his most recent one20:03
figleafcfriesen: but the fact that those of us working on it are still a little fuzzy about the edge cases is a good sign that we need to do something more20:04
mriedemhttp://45.55.105.55:8082/config-options.html20:05
*** salv-orl_ has quit IRC20:05
*** salv-orlando has joined #openstack-nova20:06
*** amoralej is now known as amoralej|off20:08
dtpthanks mriedem superdan.  left my comments on that review20:14
openstackgerritmelanie witt proposed openstack/nova master: Add step-by-step examples for Cells V2 setup  https://review.openstack.org/42798120:14
*** panda|bbl is now known as panda20:15
melwittdarn it, didn't see dtp's comments a minute ago20:15
dtpwe were very close :)20:15
*** sree has joined #openstack-nova20:16
*** ijw has joined #openstack-nova20:16
mriedemmacsz: sfinucan: if you have open config option cleanup changes, move them to https://blueprints.launchpad.net/nova/+spec/centralize-config-options-pike please20:17
macszmriedem: i will move every single patch that is left20:17
mriedemthanks20:18
macszmriedem can you re-assign me to it?20:18
mriedemsure20:19
macszthanks20:19
*** unicell has quit IRC20:20
*** Sukhdev has quit IRC20:20
*** raunak has quit IRC20:21
*** sree has quit IRC20:21
EmilienMmriedem: do you have a CI job that test Nova API deployed with SSL?20:21
*** ducttape_ has joined #openstack-nova20:21
EmilienMI couldn't find one20:21
EmilienMgate-grenade-dsvm-neutron-multinode-ubuntu-xenial seems to deploy non-ssl20:22
*** tbachman has quit IRC20:22
mriedemEmilienM: dane-fichter and the john hopkins gang was working on creating a security-specific dsvm job that tests with that plus all of the other security features like signed images and using barbican as the key manager20:22
mriedembut that's not done, and honestly i don't know the status of it right now20:22
*** tbachman has joined #openstack-nova20:22
EmilienMok20:22
EmilienMbecause I found a bug, I'll probably report it20:22
mriedemok20:23
mriedemEmilienM: so on your multinode nonha job, is that running in the same nodepool as the rest of infra or a special subset?20:25
EmilienMmriedem: same20:26
*** jheroux has quit IRC20:26
*** esmiurium has quit IRC20:26
mriedemok20:26
EmilienMmriedem: I really see a value of using TripleO jobs here. We are testing SSL, IPv6, (WSGI but will be removed today), and all of that for free20:26
*** READ10 has quit IRC20:27
EmilienMmriedem: is there something that you don't like in this idea?20:29
EmilienMI might be wrong but I see some resistance that we didn't have with other projects20:30
*** dtp is now known as dtp-afk20:30
mriedemEmilienM: well i'm a bit distracted with other things right now20:31
mriedembut also,20:31
mriedemand i know this isn't voting right now,20:31
mriedembut right now, devs have a reasonable shot at making changes in devstack to get things through nova that depend on devstack, but that learning curve gets higher if they have to touch devestack-gate or project-config or grenade,20:31
mriedemadding this means to get things in nova done, they now have to learn a whole new set of tools,20:32
mriedemi for one don't know puppet or ruby20:32
mriedemi've learned to monkey my way through devstack and it's bashtasticness over the last 4 years that i'm able to sort of know what i'm doing,20:32
mriedemuntil sdague hits me and tells me i'm wrong20:32
EmilienMI don't think we asked Nova team to learn about TripleO or did we?20:32
EmilienMwhat we ask is to tell us when that job is failing20:33
EmilienMbecause we think it can provide useful feedback20:33
mriedemno, but if we have changes that require deployment tooling changes to move forward, we can do that in devstack today20:33
mriedemyeah i mean there will be a lot of hand-holding at the least20:33
mriedemand it opens a door for the horde of other deployment projects (kolla, ansible, whatever else since fuel/chef/salt are dead i think)20:34
superdanmriedem: are you cool with getting some stuff like this into rc2? https://review.openstack.org/#/c/428824/20:34
superdanor do you want to hold off20:34
superdanwe could backport too, but I think it should be in the ocata packages20:34
mriedemsuperdan: i think docs are fine20:34
superdanokay, agree20:34
mriedemEmilienM: anyway you asked so those are the considerations, and which is why i'm not jumping on this right away20:35
mriedemplus it's been a hectic week and i'm getting burned out20:36
EmilienMmriedem: ok, let's talk about it next time you have free cycles.20:37
*** ducttape_ has quit IRC20:40
*** gszasz has quit IRC20:41
*** karimb has quit IRC20:42
*** karimb has joined #openstack-nova20:43
*** dimtruck is now known as zz_dimtruck20:46
*** bvanhav has quit IRC20:47
*** Apoorva has quit IRC20:47
*** karimb has quit IRC20:48
*** kristian__ has joined #openstack-nova20:50
*** eharney has joined #openstack-nova20:51
*** burt has joined #openstack-nova20:53
*** kristian__ has quit IRC20:54
*** gouthamr has quit IRC20:59
*** cmartin has joined #openstack-nova21:04
*** gouthamr has joined #openstack-nova21:05
cmartinHey quick question about the upcoming PTG in Atlanta, roughly what time will the conference hours be? particularly on friday? I am trying to book my flight home after the conference ends on friday but am unsure what times I can select for the flight21:05
*** xinli has quit IRC21:06
mriedemi assume things start at 9am?21:06
mriedemdhellmann: ^?21:06
mriedemcmartin: by friday afternoon (after lunch) people are usually pretty spent on a week like this,21:06
mriedemso not a ton happens on friday afternoon21:06
dhellmannthat's my assumption, too, but I don't know for sure. ttx would, but he's offline today due to travel.21:07
*** tbachman has quit IRC21:07
cmartinmriedem: I see, so a flight in the afternoon should be fine?21:07
mriedemcmartin: yup21:07
superdancmartin: you'll be begging to leave by then21:07
cmartinmriedem: thanks that is what I needed to know!21:07
cmartinmriedem: lol we will see, but it will be nice to go back home and sleep in my own bed21:08
*** Apoorva has joined #openstack-nova21:08
*** catintheroof has quit IRC21:11
*** kristian__ has joined #openstack-nova21:14
*** satyar has quit IRC21:14
*** tbachman has joined #openstack-nova21:16
EmilienMmriedem: so when I disable wsgi, multinode job is passing :-)21:18
EmilienMmriedem: I'll check baremetal jobs how it goes21:18
mriedemEmilienM: yeah because of the service version thing21:18
EmilienMawesome21:18
*** slaweq has joined #openstack-nova21:19
*** kristian__ has quit IRC21:19
cburgess@superdan lol21:22
openstackgerritMaciej Szankin proposed openstack/nova master: conf: resolved final todos in libvirt conf  https://review.openstack.org/42904221:22
*** smatzek has quit IRC21:23
superdancburgess: what?21:23
cburgesssuperdan It was just funny  your statement about begging to leave. I don't know why. Probably years of seeing the look on your face by Friday afternoons.21:24
*** amotoki has joined #openstack-nova21:24
superdancburgess: oh, heh, yeah21:24
cburgesssuperdan I was trying to come up with something witty about Dan wanting to stab people by Friday afternoon but I failed.21:24
cburgessWith regard to PTG, are we planning any time of team dinner like we have done in the past?21:25
superdancburgess: heh21:25
*** crushil has joined #openstack-nova21:26
openstackgerritMaciej Szankin proposed openstack/nova master: conf: deprecate key_manager.fixed _key opt  https://review.openstack.org/42899221:26
openstackgerritMaciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs  https://review.openstack.org/42881121:27
openstackgerritMaciej Szankin proposed openstack/nova master: conf: Deprecate two more nova-net options  https://review.openstack.org/41239021:28
*** amotoki has quit IRC21:28
openstackgerritMaciej Szankin proposed openstack/nova master: conf: Resolve TODOs in 'database'  https://review.openstack.org/39369521:29
openstackgerritMaciej Szankin proposed openstack/nova master: conf: add min parameter to scheduler opts  https://review.openstack.org/42835521:30
*** burgerk_ has joined #openstack-nova21:30
openstackgerritMaciej Szankin proposed openstack/nova master: conf: resolved TODOs in compute configuration  https://review.openstack.org/42679321:30
mriedemcburgess: like team dinner from the midcycles?21:30
cburgessYeah21:31
mriedemhmm, the midcycles are usually somewhere where the host person knows where to go...21:31
cburgessGranted those are semi planned anyways.21:31
*** cmartin has quit IRC21:31
mriedemright21:31
melwittsuperdan: based on dtp-afk's questions, I'm wondering what the right thing to do for cells v1 is at this point. when they map hosts, how do they pick which cell database to pull them from?21:31
mriedemit's always like last minute21:31
cburgessYeah21:31
openstackgerritMaciej Szankin proposed openstack/nova master: conf: mark glance.api_servers opt as required  https://review.openstack.org/42824121:31
mriedemcburgess: i'd like to do something like that at least one night...21:31
cburgessThere are lots of places options in and around ATL. Lack of transportation might limit us though.21:31
melwittsuperdan: assuming we're not having them create one cell per cells v1 cell yet21:31
cburgessmriedem Yeah its been good the times I've been able to go.21:32
mriedemi wonder if there is anywhere downtown near the hotels people are staying at21:32
openstackgerritMaciej Szankin proposed openstack/nova master: config options: remove libvirt imagecache  https://review.openstack.org/37849721:32
openstackgerritMaciej Szankin proposed openstack/nova master: conf: remove deprecated image url options  https://review.openstack.org/39780021:32
mriedemi'll throw it on the etherpad so i don't forget about it21:32
cburgessATL does have decent subway (MARTA) but it can still be a pain. I can ask around amongst some friends and see if they know of anything (grew up in ATL).21:32
*** burgerk has quit IRC21:33
mriedemisn't dhellmann and/or markmcclain in ATL?21:33
mriedemi have to assume they have a backyard...21:33
cburgessmarkmcclain Is yeah. I'm not sure about dhellmann, but sounds right.21:33
*** r-daneel has joined #openstack-nova21:33
mriedemand southern charm21:33
mriedem / hospitality21:34
cburgessLOL21:34
*** Jeffrey4l has quit IRC21:35
superdanmelwitt: they use their nova-cell.conf file, per cell yeah21:35
*** crushil has quit IRC21:35
*** Jeffrey4l has joined #openstack-nova21:35
superdanmelwitt: we bypass the lookups now, but that's what we were doing before and it worked other than that the timing of noticing the replicated statuses was then off21:36
superdanmelwitt: is this on the step by step thing?21:36
melwittsuperdan: okay, so at this stage we do want to tell them to go ahead and create a cell v2 per cell v1 cell21:36
superdanmelwitt: yeah and that's what we do in devstack21:36
*** liverpooler has quit IRC21:36
superdanotherwise the work to convert later will be messy21:36
melwittsuperdan: no but I'm asking to know if I should add it. sounds like I do need to. I had been thinking we were doing just a cheat of a single cell21:36
melwittyeah. okay21:37
superdanmelwitt: you mean from the api cell's database?21:37
superdanthere's no reason to do that since they'll all be wrong when they want to go use them21:37
melwittsuperdan: I guess so. but that wouldn't make sense because there's no compute hosts in there21:37
melwittI hadn't thought it through21:38
superdanyeah21:38
*** nic has quit IRC21:38
melwittokay, I'll add a new section for cells v1 people, I think that would be most clear21:39
superdanmelwitt: okay21:39
melwittthanks21:39
superdanmelwitt: maybe in another patch so we have a better chance of getting this in while we noodle over the cellsv1 stuff?21:39
melwittsuperdan: okay, if you think that's best21:40
superdanwhatever you want21:40
*** nic has joined #openstack-nova21:40
melwittI'm not opinionated about it. I hadn't thought of doing a follow up21:41
superdanI'd just like that bit to be in rc2 and I wouldn't want to miss that if we're messing around with cellsv1 stuff that's all21:41
melwittyeah21:41
superdanthe mainline bit I mean21:42
melwittright21:42
*** dtp-afk is now known as dtp21:43
*** burgerk_ has quit IRC21:45
*** salv-orlando has quit IRC21:45
melwittsuperdan: I'm trying to think of how the scheduling works with where we're at now. we're not yet pulling compute hosts from all cells or anything for choosing a host ... and cells v1 doesn't care about that since it chooses a cell first. so I guess that's how I had been thinking things could be gotten away with by setting up only one cell for cells v121:46
melwittbut doing that would be like, arbitrarily picking a cell to convert over and then later you'd have to do the rest, so not a great idea anyway21:46
superdanmelwitt: yeah, but I think we're okay because we choose a cell first, then go to a nova-scheduler that only knows about the lower levels21:47
superdanno, I don't think we'll be able to do that unfortunatel21:47
superdannot from v1 to v2 anyway21:47
mriedemhmm, if you have a cell mapping per cell v1 cell, and the scheduler was checking all cell mappings, then it could hit something in another cell v1 cell couldn't it?21:47
superdanno21:47
mriedemif that were the case, i guess we'd just do if CONF.cells.enabled21:47
superdanbecause scheduler is at the cell level21:47
superdanand there's a cells scheduler that hasn't been touched in ages that knows nothing of this stuff21:47
*** lpetrut has quit IRC21:47
mriedem"scheduler is at the cell level"21:48
mriedemcells v1? - there are multiple levels, can you clarify?21:48
superdanif cells.enabled is false in the api cell, then it will start rpc'ing to the global schedule21:48
melwittsuperdan: don't think we'll be able to what? didn't catch what part you were referring to21:48
mriedemi thought the cells scheduler was at the top level cell in cells v1?21:48
superdanmriedem: yeah, in cellsv1 the scheduler is at the lower levels21:48
superdanmriedem: no there's a special scheduler up there, regular scheduler down below21:48
*** raunak has joined #openstack-nova21:48
mriedemright21:48
superdanmelwitt: won't be able to have a cell-by-cell conversion from v1 to v221:49
superdanmelwitt: but it'll be more of a "turning stuff off" kind of procedure I think, if we get the mappings in place ahead of time21:49
superdanthese are not friday afternoon questions21:49
superdanask me shit like "is it beer o clock yet" or something21:50
melwittsuperdan: heh, sorry. I blame dtp for asking me to think about stuff by asking about cells v1 step-by-step21:50
mriedemscheduler is global in cells v2 world right?21:50
* superdan scowls at dtp 21:50
superdanmriedem: yeah21:50
mriedemso computes in multiple cells v2 cells are reporting into placement,21:50
mriedemscheduler asks placement for an RP, gets one, sends that to conductor,21:50
mriedemwhich is mapped to a cell via hostmappings,21:50
mriedemand then conductor creates the instance in that cell and builds it there21:50
superdanwith pure v2 yes, but with v1,21:51
superdanthat all happens inside a cellv1 cell21:51
melwittsuperdan: I was thinking it doesn't matter if they set up all their cells now (technically) and that they could do them later. but I think that would make it more confusing21:51
mriedemwhat's preventing the low-level filter scheduler in cells v1 not picking an RP from another v1 cell?21:51
superdanmriedem: because they don't have visibility of each other? they'll all need their own placement, just like they need their own api21:52
*** kristian__ has joined #openstack-nova21:52
mriedemok i thought placement was still global in cells v121:52
superdanmriedem: otherwise we have to teach placement about cellsv1, which would be ungood I think21:52
mriedemright21:52
mriedemwe don't want cells v1 specific garbage in placement21:52
mriedemmaybe we want a note of that somewhere so when i ask it 5 more times in 12 months, you can point me21:53
superdanplacement is kinda part of the regular scheduler, which is replicated per v1 cell, which means placement is too21:53
superdanmelwitt: are you doing a general "what to make of the v1 mess" section? you could put that ^ in there if so21:53
melwittsuperdan: I'm doing a step-by-step for cells v1 peeps. what's this note about, just telling why not to worry about scheduling working right?21:54
mriedemmelwitt: that placement is per cells v1 cell21:55
mriedemnot global in cells v121:55
superdanthat ^21:55
melwittoh ... hm21:55
mriedemin other words, placement only has knowledge of the computes in the cells v1 cell that it's running in21:55
superdanprobably needs to go into some placement doc21:55
mriedemyeah, which we have21:55
melwittyeah, I dunno anything about how to deploy placement for cells v121:56
mriedemjust like nova-api21:56
mriedemexcept in the child cell21:56
superdanmriedem: I'll add it to placement.rst21:56
mriedemok http://docs.openstack.org/developer/nova/placement.html21:56
melwittso more like nova-scheduler then?21:56
mriedemnot sure what the best place is21:56
*** kristian__ has quit IRC21:56
mriedemmelwitt: yeah21:56
superdanmelwitt: right, goes with scheduler21:57
mriedemdon't we have nova-scheduler in the top cell too?21:57
melwittk, that makes sense21:57
mriedemor is that the cells service?21:57
mriedemwith the cell scheduler21:57
*** gouthamr has quit IRC21:57
melwittit's the cells scheduler, I dunno if it's the same code as nova-scheduler or not21:57
mriedemcells-region or whatever21:57
superdanhmm, you know21:57
superdanbecause placement is all service-catalog-y, I wonder if people will be able to do that21:57
*** tbachman has quit IRC21:58
openstackgerritDan Smith proposed openstack/nova master: Clarify the deployment of placement for cellsv1 users  https://review.openstack.org/42905621:59
superdanmriedem: no it's different I think21:59
superdannova/cells/scheduler.py22:00
melwittah, cool22:01
*** edmondsw has quit IRC22:04
*** edmondsw has joined #openstack-nova22:05
*** edmondsw has quit IRC22:05
*** edmondsw has joined #openstack-nova22:05
*** tbachman has joined #openstack-nova22:11
*** dtp is now known as dtp-afk22:11
*** zackf has joined #openstack-nova22:13
*** karimb has joined #openstack-nova22:14
superdanmriedem: your rc1 todo etherpad says we have until monday to do rc1 but the announce already went out right?22:15
mriedemyeah, i'll fix22:15
superdanokay22:15
*** slaweq has quit IRC22:16
*** Sukhdev has joined #openstack-nova22:17
*** karimb has quit IRC22:17
*** sree has joined #openstack-nova22:17
*** sree has quit IRC22:21
owalshsuperdan, mriedem: hola... TL;DR what is the suspected problem with https://review.openstack.org/#/c/427534/?22:23
*** smatzek has joined #openstack-nova22:24
*** thorst_ has quit IRC22:25
superdanowalsh: none that I know of22:25
mriedemowalsh: not totally sure, i dug into the compute logs a bit to see instances not building on ironic nodes because the disk on the nodes was fully consumed by another instance22:25
mriedemwhich i think those things just get rescheduled,22:25
mriedembut i didn't trace through any failures in real detail22:26
superdanbut really, there's no reason for that patch to affect that kind of thing22:26
openstackgerritJohn Perkins proposed openstack/nova master: Integrate oslo.config validator  https://review.openstack.org/42897422:26
*** breitz has quit IRC22:26
superdanand before it, no nodes were even registered22:26
owalshok, can't see how they could be related... that change just double checks nova has the ironic nodes in the RT before running discover_hosts22:26
superdanyep22:27
*** breitz has joined #openstack-nova22:27
owalshpeeling an onion me thinks... my fix just got us to the next layer22:27
superdanyep22:27
superdanI haven't even heard that it happened more than once22:27
superdanmriedem: did you see a bunch of these failures?22:27
*** smatzek has quit IRC22:29
*** salv-orlando has joined #openstack-nova22:29
mriedemi don't think i actually saw any error messages, just that resources couldn't be claimed on a given node because the disk was fully consumed on that node at the time of the instance build22:30
mriedemlike i said, i didn't trace a job failure all the way through,22:30
mriedemwas just pointed at a debug log message and was asked what it meant, and i poked around that abit22:30
owalshok https://review.openstack.org/428974 couldn't be related I think...22:30
melwittsuperdan: sanity check me here but I don't think cells v1 peeps can use the discover_hosts periodic task because their nova-schedulers run inside cells, right?22:31
ttxmriedem: I don't have final hours yet but yes 9am start sounds like a safe bet. Up until 6pm in the afternoon I'd say. Probably wrap up slightly earlier on Friday (5pm or so)22:31
mriedemi also added this because of that debugging https://review.openstack.org/#/c/428985/22:31
mriedemttx: ok22:31
superdanmelwitt: why not?22:31
mriedemmelwitt: it would discover the computes in that cell22:31
superdanright22:31
mriedemwhich is what we want22:31
melwittsuperdan: writes to the api database? okay22:31
mriedemwrites the host mappings to the api db22:31
melwittthey just have to configure it per cell22:32
mriedemso do they have the api_database/connection in their nova-cells.conf for the low-level scheduler to do that?22:32
mriedemthey're going to have to for any of the nova-manage commands22:32
superdanyeah,22:32
mriedemwe should probably point that out somewhere too22:32
melwittI think they do in that I think mgagne found out by experience that he had to configure it per cell (the api db connection)22:32
mriedemwas just going to mention mgagne22:33
melwittokay, cool. I shall continue my writings22:33
superdanmelwitt: or did he just say it worked that way?22:33
openstackgerritJohn Perkins proposed openstack/nova master: Integrate oslo.config validator  https://review.openstack.org/42897422:33
superdanmelwitt: either way, doesn't matter that much22:33
melwittsuperdan: it matters in that the discover_hosts needs for it to be there for to write the host mappings22:33
melwittright?22:33
superdanif you run the periodic yeah22:34
melwittokay. I'll mention that in there too. I dunno if it's already required for some other reason and thus moot to mention22:34
*** tblakes has quit IRC22:36
*** baoli has quit IRC22:37
*** Sukhdev has quit IRC22:38
*** mvk has joined #openstack-nova22:38
*** owalsh is now known as owalsh-afk22:40
*** ducttape_ has joined #openstack-nova22:41
*** edmondsw has quit IRC22:41
mriedemok i'm calling it a day22:43
mriedemtalk to you all later22:43
mriedemput some salve on the wounds from the last 2 days22:43
melwittheh o/22:43
*** mriedem has quit IRC22:44
*** gouthamr has joined #openstack-nova22:45
*** ducttape_ has quit IRC22:46
*** karimb has joined #openstack-nova22:52
*** jaosorior has quit IRC22:52
openstackgerritKen'ichi Ohmichi proposed openstack/nova master: Update hacking version  https://review.openstack.org/38933822:59
*** vsaienko has quit IRC23:00
*** Swami has quit IRC23:01
*** mdrabe has quit IRC23:04
*** esberglu_ has quit IRC23:06
diana_clarkesuperdan: Do you know off the top of your head what you need to do to run the functional MySQL tests locally? Like TestNovaAPIMigrationsWalkMySQL.test_walk_versions.23:09
openstackgerritmelanie witt proposed openstack/nova master: Add Cells V1 --> Cells V2 step-by-step example  https://review.openstack.org/42907523:12
*** kfarr has quit IRC23:12
*** dtp-afk is now known as dtp23:14
diana_clarkelike setting OS_TEST_DBAPI_ADMIN_CONNECTION etc, perhaps?23:16
superdandiana_clarke: yeah, some envars to trigger it, IIRC23:16
*** burt has quit IRC23:17
*** figleaf is now known as edleafe23:17
diana_clarkesuperdan: Okie dokie, I'll try a few more things. Have a great weekend, folks!23:18
*** armax has quit IRC23:18
*** panda is now known as panda|Zz23:18
openstackgerritmelanie witt proposed openstack/nova master: Add Cells V1 --> Cells V2 step-by-step example  https://review.openstack.org/42907523:21
*** unicell has joined #openstack-nova23:24
*** amotoki has joined #openstack-nova23:24
*** Sukhdev has joined #openstack-nova23:25
*** zz_dimtruck is now known as dimtruck23:25
*** amotoki has quit IRC23:29
*** xyang1 has quit IRC23:31
*** dtp has quit IRC23:32
*** annegentle has quit IRC23:35
*** kristian__ has joined #openstack-nova23:42
*** karimb has quit IRC23:45
*** kristian__ has quit IRC23:46
*** sdague has quit IRC23:48
*** salv-orlando has quit IRC23:48
*** salv-orlando has joined #openstack-nova23:49
*** paw has quit IRC23:51
*** Swami has joined #openstack-nova23:51
*** armax has joined #openstack-nova23:54
*** salv-orlando has quit IRC23:54
openstackgerritmelanie witt proposed openstack/nova master: Add Cells V1 --> Cells V2 step-by-step example  https://review.openstack.org/42907523:56
openstackgerritDiana Clarke proposed openstack/nova master: vomiting  https://review.openstack.org/42911423:56
*** salv-orlando has joined #openstack-nova23:57
*** ducttape_ has joined #openstack-nova23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!