13:00:57 <claudiub> #startmeeting hyper-v
13:00:58 <openstack> Meeting started Wed Jun  8 13:00:57 2016 UTC and is due to finish in 60 minutes.  The chair is claudiub. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:59 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:01:01 <openstack> The meeting name has been set to 'hyper_v'
13:01:17 <claudiub> sorry for the delayed start, time flies. :)
13:01:20 <claudiub> hellou :)
13:01:24 <abalutoiu> o/
13:01:26 <lpetrut> Hi
13:01:26 <sagar_nikam> hi
13:01:32 <kvinod> hi
13:01:44 <itoader> Hi
13:02:14 <atuvenie> hello all
13:02:36 <claudiub> so, this meeting will probably be shorter. The activity log is shorter this time around, we had to attend the OpenStack CEE Day in Budapest.
13:02:48 <claudiub> and that consumed a few days. :)
13:02:55 <claudiub> #topic monasca patches
13:03:20 <claudiub> no new patch was sent. primarely replying to comments. no patch merged yet. bp still approved.
13:03:33 <claudiub> received a bunch of comments last night, didn't get to reply to all of them.
13:03:34 <sagar_nikam> ok
13:03:43 <sagar_nikam> all 14 patches reviewed ?
13:03:55 <claudiub> yep
13:03:59 <sagar_nikam> nice
13:04:15 <claudiub> I'll have to adjust the code a little bit, basically.
13:04:25 <claudiub> anyways. next topic.
13:04:33 <domi007> hi all, sorry for being late :)
13:04:43 <claudiub> #topic networking-hyperv bugs
13:04:49 <claudiub> hi domi007. :)
13:05:26 <claudiub> so this should be ready to merge: https://review.openstack.org/#/c/322068/
13:05:34 <claudiub> so, atuvenie, lpetrut, pls review.
13:05:44 <claudiub> even the ci passed.
13:06:03 <kvinod> Yes we can merge this
13:06:06 <claudiub> thanks for the contribution. :)
13:06:31 <kvinod> wanted to know thoughts on this
13:06:31 <claudiub> yep, +2'd
13:06:33 <kvinod> https://bugs.launchpad.net/networking-hyperv/+bug/1586354
13:06:33 <openstack> Launchpad bug 1586354 in networking-hyperv "Intermittent Issue seen --Associating a vm from one security group (having tcp rule) to another security group(not having tcp rule) does not stop ssh from happening" [Undecided,New]
13:06:59 <kvinod> thanks for +2
13:07:00 <claudiub> kvinod: hm, seems I've missed this.
13:07:40 <kvinod> We donot have much in out hand to fix it
13:08:03 <claudiub> kvinod: oh, so basically, you're changing a neutron port's security group, am I right?
13:08:07 <kvinod> it seems to be windows underlay issue
13:08:13 <kvinod> yes
13:08:16 <kvinod> right
13:08:43 <claudiub> kvinod: hm, interesting. haven't tried something like this before. I am curios about it.
13:08:50 <kvinod> any help on this will be good
13:08:54 <claudiub> kvinod: do you have any logs laying around?
13:08:55 <kvinod> ok
13:09:25 <kvinod> I have posted a query on http://ask.cloudbase.it/question/1233/hyperv-security-groups-update-does-not-work-correctly/
13:09:54 <kvinod> I will ask my team if they have it
13:10:06 <kvinod> Will share it in case available
13:10:12 <domi007> it feels like the bug reporter's default hyperv firewall setting is off for some reason, so the machine gets all the traffic
13:11:00 <kvinod> domi007: are you talking about https://bugs.launchpad.net/networking-hyperv/+bug/1586354
13:11:00 <openstack> Launchpad bug 1586354 in networking-hyperv "Intermittent Issue seen --Associating a vm from one security group (having tcp rule) to another security group(not having tcp rule) does not stop ssh from happening" [Undecided,New]
13:11:03 <claudiub> kvinod: sure, thanks. will try it as well.
13:11:33 <claudiub> domi007: yeah, I was thinking the same, but it does say in the bug report that it's intermittent.
13:11:39 <kvinod> ok fine, if you get to know something please do share
13:11:50 <claudiub> will do
13:12:11 <kvinod> thanks claudiub
13:12:26 <kvinod> There is one more bug
13:12:29 <claudiub> as for this one, I wasn't able to reproduce it: https://bugs.launchpad.net/networking-hyperv/+bug/1583541
13:12:30 <openstack> Launchpad bug 1583541 in networking-hyperv "Hyper-V neutron agent hangs on nova boot (with enable_security_group=true)" [Undecided,New]
13:12:31 <alexpilotti> hello folks
13:12:40 <kvinod> s://bugs.launchpad.net/networking-hyperv/+bug/1583541
13:13:01 <sagar_nikam> hi alexpilotti:
13:13:03 <kvinod> https://bugs.launchpad.net/networking-hyperv/+bug/1583541
13:13:29 <kvinod> this was found as part of our scale test with SG enabled
13:13:49 <claudiub> kvinod: yep. I followed the steps described in the bug report, but it didn't happen
13:14:20 <alexpilotti> kvinod: does it happen everytime you start it?
13:14:34 <kvinod> Monkey patching all modules fixes the issue
13:14:37 <domi007> could you guys also include what release you are running? there qas quite an overhaul of the firewall part between liberty and mitaka
13:14:48 <kvinod> yes that was seen everytime when we run scale
13:14:53 <claudiub> I've tried it on newton. bug report says it's reproducable on newton as well.
13:15:12 <sagar_nikam> domi007: we are on liberty
13:15:25 <kvinod> I saw that happening/reproduced on devstack also
13:15:31 <kvinod> but that was only once
13:15:40 <kvinod> mostly in scale it happens always
13:15:57 <kvinod> my devstack was on mitaka
13:16:20 <domi007> claudiub: I read the bugreport but found no mention of newton :) but it's your table, so nevermind
13:16:21 <sagar_nikam> alexpilotti: claudiub: we are running scale tests often now and finding some issues, one of them is what is getting discussed now
13:16:54 <claudiub> domi007: there's a comment saying: "Should be reproducible on Hyper-V latest also."
13:17:25 <domi007> oh sorry, I had still the previous bug opened :)
13:18:58 <kvinod> claudiub : I saw that on devstak mitaka once of which I uploaded the traces and logs. There was not much difference in mitaka and master, so I said should be reproducible on master also
13:19:12 <alexpilotti> sagar_nikam: I mean, if we trt to repro it, does it happen everytime?
13:19:20 <claudiub> kvinod: well, monkey patching works because then the agent will only work with greenthreads and there's no locking there. but using greenthreads instead of native threads will impact the performance negatively.
13:19:59 <sagar_nikam> alexpilotti: as mentioned by kvinod: it is intermittent
13:20:03 <claudiub> kvinod: so, if there's a deadlock somewhere, the best solution is to find it and solve it.
13:20:44 <kvinod> alexpilotti: there are two different issue
13:20:47 <kvinod> https://bugs.launchpad.net/networking-hyperv/+bug/1586354
13:20:47 <openstack> Launchpad bug 1586354 in networking-hyperv "Intermittent Issue seen --Associating a vm from one security group (having tcp rule) to another security group(not having tcp rule) does not stop ssh from happening" [Undecided,New]
13:20:49 <kvinod> and
13:21:06 <kvinod> https://bugs.launchpad.net/networking-hyperv/+bug/1586354
13:22:03 <kvinod> sorry https://bugs.launchpad.net/networking-hyperv/+bug/1583541
13:22:03 <openstack> Launchpad bug 1583541 in networking-hyperv "Hyper-V neutron agent hangs on nova boot (with enable_security_group=true)" [Undecided,New]
13:22:22 <kvinod> so monkey patching is for https://bugs.launchpad.net/networking-hyperv/+bug/1583541
13:23:30 <kvinod> and intermittent seen issue is https://bugs.launchpad.net/networking-hyperv/+bug/1586354
13:23:30 <openstack> Launchpad bug 1586354 in networking-hyperv "Intermittent Issue seen --Associating a vm from one security group (having tcp rule) to another security group(not having tcp rule) does not stop ssh from happening" [Undecided,New]
13:24:16 <alexpilotti> kvinod: we need clear steps that we can run to reproduce
13:25:00 <kvinod> for which one, I think for both bugs we have captured the steps
13:25:05 <alexpilotti> otherwise if they dont show up during our scale tests (Rally) it becomes impossible to isolate them
13:25:28 <alexpilotti> unlike you guys can give us access to your env
13:25:31 <kvinod> in case you need more info I will be happy to share it
13:26:40 <claudiub> kvinod: at the very least, can you share the rally scenario you're using?
13:27:16 <kvinod> our environment is not rally based
13:27:33 <claudiub> how are you doing scale tests?
13:27:48 <kvinod> we have some shell scripts which does the operation
13:27:59 <kvinod> in a scale environment
13:28:23 <claudiub> hm, you should really give rally a shot. it's actually very nice.
13:28:38 <claudiub> at the very least, the output is very useful.
13:29:07 <kvinod> sure lets see when we plans for this
13:29:26 <kvinod> i mean rally
13:30:04 <claudiub> anyways. I'll try to reproduce it on mitaka as well, but I'm a bit skeptic that I can encounter the issue by following the described steps.
13:30:20 <kvinod> I would say on your setup you can give a try by applying this patch https://review.openstack.org/#/c/321452/
13:30:39 <sagar_nikam> alexpilotti: claudiub: we will speak to scale team and check the feasibility of using rally. for now if we let you know the steps to reproduce, is it sufficient ?
13:31:30 <kvinod> sagar_nikam: steps are already there for https://bugs.launchpad.net/networking-hyperv/+bug/1583541
13:31:31 <openstack> Launchpad bug 1583541 in networking-hyperv "Hyper-V neutron agent hangs on nova boot (with enable_security_group=true)" [Undecided,New]
13:31:48 <sagar_nikam> kvinod: thanks
13:32:14 <kvinod> probably if more info required for bug 1586354
13:32:14 <openstack> bug 1586354 in networking-hyperv "Intermittent Issue seen --Associating a vm from one security group (having tcp rule) to another security group(not having tcp rule) does not stop ssh from happening" [Undecided,New] https://launchpad.net/bugs/1586354
13:32:21 <kvinod> we can share it
13:32:45 <claudiub> kvinod: yeah, that will cause the threading module to be monkey patched. If that happens, you can't have native threads. :)  which means poorer performance.
13:33:10 <kvinod> ok
13:33:24 <claudiub> anyways. a pip freeze and the neutron-hyperv-agent logs might be useful.
13:33:45 <kvinod> with monkeypatch, enhanced RPC and pyMi together
13:33:50 <claudiub> and the neutron-hyperv-agent's conf file.
13:34:28 <kvinod> we were easily able to boot 1000VMs in 22 node compute environment with Network node HA
13:35:12 <abalutoiu> kvinod: I have a curiosity, how do you make sure that the VM is created with error state?
13:37:09 <kvinod> abalutoiu: when I recreated the issue on devstak i stopped neutron agent such that VM goes into error state
13:37:10 <sagar_nikam> claudiub: i also have a scale issue to discuss, once we are done with this
13:37:20 <kvinod> and booted the VM
13:38:11 <kvinod> on devstack even though I hit the issue once I was not able to recreate again
13:38:20 <kvinod> but in scale its very easy to recreate
13:39:01 <abalutoiu> can you share with us more details about what your scale tests are doing?
13:39:14 <kvinod> you just have to ensure that a compute that does not has any VM booted receives provider rule update
13:40:36 <kvinod> abalutoiu: I will share the details
13:41:01 <kvinod> on what we are doing as part of scale test
13:42:11 <abalutoiu> kvinod: thanks
13:42:44 <claudiub> ok, so, we're waiting for more details on this.
13:42:53 <claudiub> #topic nova resource tracking
13:43:03 <claudiub> sagar_nikam: I think this is what you wanted, right?
13:43:12 <sagar_nikam> claudiub: yes
13:43:21 <sagar_nikam> we are hitting a issue in scale tests
13:43:36 <sagar_nikam> i sent you the screen shot as well as logs
13:43:38 <claudiub> ok, so. let me explain a little bit how resource tracking works on nova. :)
13:43:56 <sagar_nikam> on the hyperv host, we have only 1.5 GB RAM
13:43:59 <sagar_nikam> free
13:44:34 <claudiub> so, the drivers do report their actual memory, disk, cpu consumption, but the nova resource tracker only logs those values and reports to the scheduler only the 'ideal' values.
13:44:34 <sagar_nikam> whereas i see that nova-compute sends it as some 9gb free
13:45:31 <sagar_nikam> now what is happening is, scheduler thinks there is more memory and schedules a instance t it
13:45:35 <sagar_nikam> nova start fails
13:45:42 <sagar_nikam> since no memory available
13:45:56 <claudiub> basically, the resource tracker will iterate over all the instances on that host, and calculates the memory consumption / disk usage that host should have.
13:46:19 <sagar_nikam> you mean this is a issue with resource tracker ?
13:46:31 <sagar_nikam> we are not finding this issue on KVM
13:46:42 <claudiub> nova doesn't really call it an "issue".
13:47:04 <sagar_nikam> but then ... nova boot fails
13:47:10 <claudiub> i know.
13:47:17 <sagar_nikam> since the host does not have the memory
13:47:46 <claudiub> so, the resource tracker doesn't take into account the host memory or disk necesities.
13:47:46 <sagar_nikam> does this mean this is the same behavior for KVM and VMWare as well ?
13:47:55 <claudiub> but you do have some config options to help on this
13:48:11 <sagar_nikam> dynamic_memory ?
13:48:15 <sagar_nikam> or something else
13:48:32 <sagar_nikam> what are those config options ?
13:49:43 <claudiub> sagar_nikam: I don't really remember the names, and they've been moved from where they were.
13:49:57 <sagar_nikam> ok
13:50:03 <sagar_nikam> we will try to check it
13:50:15 <claudiub> sagar_nikam: https://github.com/openstack/nova/blob/stable/mitaka/nova/compute/resource_tracker.py#L45
13:50:23 <sagar_nikam> in case anybody from your team finds it, please do let us know
13:50:33 <claudiub> reserved_host_disk_mb and reserved_host_memory_mb
13:51:07 <sagar_nikam> ok, i think we have set these values on the host
13:51:14 <sagar_nikam> but i will confirm with scale team again
13:51:21 <claudiub> basically, those are the values the resource tracker considers as "reserved for the host"
13:51:28 <sagar_nikam> ok
13:52:01 <claudiub> so, a host does have its memory and disk usages as well.
13:52:02 <sagar_nikam> we will try and see how it works
13:52:08 <sagar_nikam> thanks for the info
13:52:12 <claudiub> np.
13:52:24 <sagar_nikam> one more topic from me
13:52:41 <sagar_nikam> http://specs.openstack.org/openstack/nova-specs/specs/newton/approved/convert-consoles-to-objects.html
13:52:44 <claudiub> you can also see in the nova-compute logs the "Hypervisor resource view" and the "Final resource tracker view", or something like that.
13:53:01 <sagar_nikam> we saw the final resource view
13:53:02 <claudiub> #topic open discussion
13:53:11 <sagar_nikam> and that is what is reportin 9gb free
13:53:16 <claudiub> sagar_nikam: yep.
13:53:19 <sagar_nikam> ok
13:53:21 <sagar_nikam> now
13:53:26 <sagar_nikam> have you seen this BP
13:53:27 <sagar_nikam> http://specs.openstack.org/openstack/nova-specs/specs/newton/approved/convert-consoles-to-objects.html
13:53:46 <sagar_nikam> it is worked on by team mate paul murray
13:53:55 <claudiub> i am not very familiar with it.
13:53:55 <kvinod> claudiub: any update on windows ovs logo certification
13:54:03 <domi007> I just wanted to say it was great seeing some of you in Budapest. atuvenie please let me know if you have the installer for liberty with ovs WMI security groups. Also please let us know if FreeRDP is updated :)
13:54:06 <sagar_nikam> this may impact nova consoles for hyperv using freerdp
13:54:23 <c64cosmin> domi007: yes we have a new version
13:54:37 <atuvenie> domi007: sure domi, I'll leave you all the details on skype
13:54:38 <c64cosmin> https://cloudbase.it/downloads/FreeRDPWebConnect_Beta.msi
13:54:44 <sagar_nikam> claudiub: c64cosmin: please check if that BP affects freerdp
13:54:44 <c64cosmin> https://cloudbase.it/downloads/FreeRDPWebConnect_Beta.zip
13:54:53 <domi007> cool :) thank you for both of you!
13:55:06 <c64cosmin> it's x64 now
13:55:16 <sagar_nikam> c64cosmin: can this be available in stable release as well
13:55:30 <sagar_nikam> our QA picks up from stable release
13:55:51 <claudiub> sagar_nikam: I will read the spec, but from what I can briefly read, it shouldn't. It will just standardize / object-ify some items related to remote consoles.
13:55:52 <c64cosmin> sagar_nikam: not yet
13:56:01 <claudiub> sagar_nikam: it shouldn't change the behaviour.
13:56:10 <claudiub> but, I'll read it in more detail.
13:56:20 <sagar_nikam> claudiub: i think UUID needs to be passed in the URL
13:56:32 <sagar_nikam> atleast that change is required
13:56:44 <sagar_nikam> may be you can chat with paul murray ?
13:56:50 <sagar_nikam> he can provide more info
13:57:05 <sagar_nikam> c64cosmin: when is it planned for stable ?
13:57:15 <sagar_nikam> also keystrokes issue fixed ?
13:57:32 <sagar_nikam> what we discussed in last IRC meeting
13:57:51 <c64cosmin> sagar_nikam: inside Horizon right?
13:57:57 <sagar_nikam> yes
13:58:09 <sagar_nikam> does not work ... i have hit it many times
13:58:14 <c64cosmin> sagar_nikam: I have not encountered those
13:58:29 <claudiub> domi007: yeah, thanks for saying hello in Budapest. :) I don't think we met there though.
13:58:29 <kvinod> claudiub: any update on windows logo certification for ovs. When is it planned.
13:58:43 <sagar_nikam> c64cosmin: we hit it many times
13:58:52 <c64cosmin> sagar_nikam: but to be sure I will make such that when entering the console, the key state is reset
13:59:03 <c64cosmin> maybe there is some RDP command to that one
13:59:05 <c64cosmin> not sure
13:59:10 <claudiub> kvinod: It's still underway. I guess it takes a while. :(
13:59:15 <sagar_nikam> maybe i will pick the latest MSI whenever it is available in stable and test and check it
13:59:23 <c64cosmin> usually happens when a key press is sent but not the key release
13:59:27 <claudiub> sagar_nikam: sure, I'll ask him. He's a nice guy. :)
13:59:32 <kvinod> claudiub: ok thanks
13:59:45 <sagar_nikam> claudiub: thanks
14:00:04 <sagar_nikam> c64cosmin: next debian is planned or it requires some more time ?
14:00:36 <c64cosmin> sagar_nikam: I will have an incoming PR, and will leave after that passes and is merged
14:00:43 <c64cosmin> I still need to do some tests
14:00:48 <claudiub> I think we'll have to end it here. Other people will get upset otherwise. :) We can continue for a bit on #openstack-hyper-v if needed.
14:00:53 <c64cosmin> also make sure it works well with mitaka
14:01:03 <sagar_nikam> sure, lets continue there
14:01:06 <sagar_nikam> i will join
14:01:07 <claudiub> so, thanks folks for joining!. :)
14:01:10 <claudiub> #endmeeting