Friday, 2019-10-18

*** beekneemech has joined #openstack-nova00:06
*** bnemec has quit IRC00:08
*** takashin has joined #openstack-nova00:11
*** beekneemech has quit IRC00:12
*** bnemec has joined #openstack-nova00:14
openstackgerritMerged openstack/nova stable/queens: Fix rebuild of baremetal instance when vm_state is ERROR  https://review.opendev.org/68087300:18
*** markvoelker has joined #openstack-nova00:49
*** dpawlik has joined #openstack-nova00:53
*** dpawlik has quit IRC00:58
*** Liang__ has joined #openstack-nova01:02
*** nanzha has joined #openstack-nova01:06
*** sapd1 has joined #openstack-nova01:09
*** nweinber has quit IRC01:11
*** Liang__ is now known as LiangFang01:19
*** tbachman_ has joined #openstack-nova01:21
*** tbachman has quit IRC01:22
*** tbachman_ is now known as tbachman01:22
*** mkrai_ has joined #openstack-nova01:24
*** igordc has quit IRC01:25
*** sapd1 has quit IRC01:39
*** mkrai_ has quit IRC01:55
*** guozijn has joined #openstack-nova01:58
*** artom has quit IRC01:59
*** artom has joined #openstack-nova01:59
*** artom has joined #openstack-nova02:00
*** gbarros has joined #openstack-nova02:02
*** Garyx_ has joined #openstack-nova02:17
*** Garyx has quit IRC02:19
*** sapd1 has joined #openstack-nova02:37
*** ricolin_ has joined #openstack-nova02:51
*** dpawlik has joined #openstack-nova02:54
*** dpawlik has quit IRC02:58
*** sapd1 has quit IRC03:06
*** dave-mccowan has quit IRC03:21
*** jangutter has joined #openstack-nova03:22
*** jangutter has quit IRC03:26
*** dviroel has quit IRC03:28
*** psachin has joined #openstack-nova03:37
*** gbarros has quit IRC04:07
*** guozijn has quit IRC04:33
*** igordc has joined #openstack-nova04:35
*** igordc has quit IRC04:35
*** igordc has joined #openstack-nova04:35
openstackgerritMerged openstack/nova master: Add image caching API for aggregates  https://review.opendev.org/68714004:41
*** ratailor has joined #openstack-nova04:48
*** Luzi has joined #openstack-nova04:50
*** dpawlik has joined #openstack-nova04:55
*** guozijn has joined #openstack-nova04:55
*** adriant has joined #openstack-nova04:57
*** dpawlik has quit IRC04:59
*** igordc has quit IRC05:05
openstackgerritMerged openstack/nova master: Fix up some feedback on image precache support  https://review.opendev.org/68817205:06
*** dpawlik has joined #openstack-nova05:15
*** awalende has joined #openstack-nova05:18
*** tetsuro has quit IRC05:18
*** awalende has quit IRC05:22
*** mdbooth has quit IRC05:35
*** mdbooth has joined #openstack-nova05:36
*** maciejjozefczyk has joined #openstack-nova05:50
*** tetsuro has joined #openstack-nova06:00
*** tetsuro has quit IRC06:01
*** jawad_axd has joined #openstack-nova06:09
*** jawad_axd has quit IRC06:13
*** jawad_axd has joined #openstack-nova06:13
*** tetsuro has joined #openstack-nova06:15
*** dpawlik has quit IRC06:20
*** dpawlik has joined #openstack-nova06:27
*** slaweq has joined #openstack-nova06:30
*** ociuhandu has joined #openstack-nova06:31
*** nanzha has quit IRC06:31
*** ociuhandu has quit IRC06:35
*** nanzha has joined #openstack-nova06:37
*** ratailor has quit IRC06:49
*** sapd1 has joined #openstack-nova06:50
*** trident has quit IRC07:00
*** trident has joined #openstack-nova07:04
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] To create single NUMA node instance in function '_get_numa_topology_auto'  https://review.opendev.org/68893207:05
*** lpetrut has joined #openstack-nova07:05
*** udesale has joined #openstack-nova07:08
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] Assign and track instance pinning cpu through 'cpu_pinning' field  https://review.opendev.org/68893307:09
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] Add a new instance CPU allocation policy: mixed  https://review.opendev.org/68893407:11
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] virt/libvirt: Get host pin cpuset according instance cpu_pinning  https://review.opendev.org/68893507:11
*** ratailor has joined #openstack-nova07:12
*** tetsuro has quit IRC07:13
*** tetsuro has joined #openstack-nova07:20
*** tesseract has joined #openstack-nova07:22
*** awalende has joined #openstack-nova07:25
*** pcaruana has joined #openstack-nova07:26
bauzasmorning fellows07:28
*** ricolin_ is now known as ricolin07:30
openstackgerritBrin Zhang proposed openstack/nova master: Pass RequestContext to oslo_policy  https://review.opendev.org/67403807:41
openstackgerritBrin Zhang proposed openstack/nova master: Add test coverage of existing os-services policies  https://review.opendev.org/66918107:43
openstackgerritBrin Zhang proposed openstack/nova master: Add test coverage of existing os-agents policies  https://review.opendev.org/66957807:43
openstackgerritBrin Zhang proposed openstack/nova master: Introduce scope_types in os-services  https://review.opendev.org/64542707:44
openstackgerritBrin Zhang proposed openstack/nova master: Add new default roles and mapping in policy base class  https://review.opendev.org/64545207:44
*** tssurya has joined #openstack-nova07:45
*** brinzhang has joined #openstack-nova07:47
*** ralonsoh has joined #openstack-nova07:51
*** jangutter has joined #openstack-nova07:54
*** jangutter_ has joined #openstack-nova07:54
*** brinzhang_ has joined #openstack-nova07:56
*** jangutter has quit IRC07:58
openstackgerritBrin Zhang proposed openstack/nova master: Pass RequestContext to oslo_policy  https://review.opendev.org/67403807:59
openstackgerritBrin Zhang proposed openstack/nova master: Add test coverage of existing os-services policies  https://review.opendev.org/66918107:59
openstackgerritBrin Zhang proposed openstack/nova master: Introduce scope_types in os-services  https://review.opendev.org/64542707:59
openstackgerritBrin Zhang proposed openstack/nova master: Add new default roles and mapping in policy base class  https://review.opendev.org/64545207:59
openstackgerritBrin Zhang proposed openstack/nova master: Add policy deprecation fixture and Suppress warnings in tests  https://review.opendev.org/67667007:59
*** brinzhang has quit IRC07:59
openstackgerritBrin Zhang proposed openstack/nova master: Pass RequestContext to oslo_policy  https://review.opendev.org/67403808:01
*** rpittau|afk is now known as rpittau08:01
*** tetsuro has quit IRC08:02
openstackgerritBrin Zhang proposed openstack/nova master: Add test coverage of existing os-services policies  https://review.opendev.org/66918108:03
openstackgerritBrin Zhang proposed openstack/nova master: Introduce scope_types in os-services  https://review.opendev.org/64542708:03
openstackgerritBrin Zhang proposed openstack/nova master: Add new default roles in os-services API policies  https://review.opendev.org/64848008:04
openstackgerritBrin Zhang proposed openstack/nova master: Add new default roles and mapping in policy base class  https://review.opendev.org/64545208:07
*** xek__ has joined #openstack-nova08:07
*** brinzhang has joined #openstack-nova08:16
*** tssurya has quit IRC08:17
*** brinzhang_ has quit IRC08:19
*** ttsiouts has joined #openstack-nova08:23
*** ratailor_ has joined #openstack-nova08:26
*** ratailor has quit IRC08:28
*** tssurya has joined #openstack-nova08:30
*** takashin has left #openstack-nova08:30
openstackgerritBrin Zhang proposed openstack/nova master: Add policy deprecation fixture and Suppress warnings in tests  https://review.opendev.org/67667008:30
*** damien_r has joined #openstack-nova08:33
*** priteau has joined #openstack-nova08:36
*** hamzy_ has joined #openstack-nova08:41
*** hamzy has quit IRC08:43
*** guozijn has quit IRC08:44
*** guozijn has joined #openstack-nova08:46
*** brinzhang_ has joined #openstack-nova08:49
*** brinzhang has quit IRC08:53
*** ileixe has joined #openstack-nova08:58
*** ileixe has left #openstack-nova08:59
*** sapd1 has quit IRC09:01
*** brinzhang_ has quit IRC09:02
*** dansmith has quit IRC09:07
openstackgerritBalazs Gibizer proposed openstack/nova master: Refactor rebuild_instance  https://review.opendev.org/68841909:14
openstackgerritBalazs Gibizer proposed openstack/nova master: Allow evacuating server with port resource request  https://review.opendev.org/68838709:14
openstackgerritBalazs Gibizer proposed openstack/nova master: Enable evacuation with qos ports  https://review.opendev.org/68868809:14
*** LiangFang has quit IRC09:19
*** dansmith has joined #openstack-nova09:25
*** dtantsur|afk is now known as dtantsur09:31
*** dpawlik has quit IRC09:47
*** ociuhandu has joined #openstack-nova09:47
*** ttsiouts has quit IRC09:51
*** ttsiouts has joined #openstack-nova09:51
*** sapd1 has joined #openstack-nova09:53
openstackgerritMatthew Booth proposed openstack/nova master: Functional test for UnexpectedDeletingTaskStateError  https://review.opendev.org/68927809:55
openstackgerritMatthew Booth proposed openstack/nova master: Unplug VIFs as part of cleanup of networks  https://review.opendev.org/66338209:55
openstackgerritMatthew Booth proposed openstack/nova master: Fix incorrect vm and task state after build failure race  https://review.opendev.org/68938809:55
mdboothstephenfin sean-k-mooney: ^^^09:55
*** ttsiouts has quit IRC09:56
mdboothsean-k-mooney: The bottom of that stack is the new test base we discussed, btw, which also has a +1 from zuul. It would be useful to do the rounds on that one sooner rather than later. If you could rebase your patch on top of it so we can show it's useful that would be even better.10:00
sean-k-mooneyya i can do that10:02
*** xek__ has quit IRC10:08
*** dpawlik has joined #openstack-nova10:09
*** xek has joined #openstack-nova10:11
*** priteau has quit IRC10:13
*** bnemec has quit IRC10:13
*** bnemec has joined #openstack-nova10:15
*** xek_ has joined #openstack-nova10:15
*** xek has quit IRC10:16
*** ygk_12345 has joined #openstack-nova10:28
*** guozijn has quit IRC10:34
*** cdent has joined #openstack-nova10:34
*** xek_ has quit IRC10:38
*** xek has joined #openstack-nova10:40
ygk_12345hi all10:40
ygk_12345I am going thru the nova code on github. Van anyone point me to the exact code which says the libvirt to use qemu-kvm binary to launch a new process for the vm ?10:41
ygk_12345*Can anyone10:41
openstackgerritMerged openstack/nova master: Add support for 'initenv' elements  https://review.opendev.org/66797510:41
openstackgerritMatthew Booth proposed openstack/nova master: Unplug VIFs as part of cleanup of networks  https://review.opendev.org/66338210:43
openstackgerritSurya Seetharaman proposed openstack/nova master: Include removal of ephemeral backing files in the image cache manager  https://review.opendev.org/68942210:47
*** tbachman has quit IRC10:48
*** zigo has quit IRC10:54
*** ttsiouts has joined #openstack-nova10:59
*** zigo has joined #openstack-nova11:04
*** ratailor_ has quit IRC11:09
*** dpawlik has quit IRC11:09
*** tssurya has quit IRC11:15
*** tkajinam has quit IRC11:16
*** xek has quit IRC11:16
*** tbachman has joined #openstack-nova11:16
artomygk_12345, you mean you want to see a os.exec('qemu') in the code? You won't find it11:17
artomygk_12345, that's in the libvirt code itself11:17
ygk_12345@artom where exactly ?11:17
artomygk_12345, Nova just defines a domain with some XML11:17
artomygk_12345, I dunno, use grep, or ask in #libvirt11:17
artomBut mostly grep11:18
cdent"But mostly grep" is my mantra11:18
ygk_12345@artom I have installed libvirt-python and used recursive grep but could not find the qemu-kvm pattern11:18
artomMostly Grep is my rap name11:18
cdentoh, that's a good one11:19
artomygk_12345, that's because those are just the Python bindings for libvirt11:19
artomygk_12345, you want libvirtd11:19
artomWe form wht Unix Krew with Oh Snap it's Sed, Aw man it's Awk, and Who Let Perl In?11:21
artom*the Unix11:21
ygk_12345@artom things at the OS level are weird but interesting11:24
cdentygk_12345: out of curiosity, why are you looking for the exec call? are you trying to solve a problem, exploring, something else?11:26
ygk_12345@cdent I am trying to understand and trace the vm create event through nova code :)11:27
cdentI think what artom said is the key bit: nova creates some xml and gives it to libvirt, libvirt does the rest11:27
ygk_12345@cdent so its libvirt API internally11:28
* cdent nods11:28
ygk_12345@cdent awesome11:28
ygk_12345@cdent u see no limits to curiosity until we hit the hardware11:29
cdent:)11:29
cdentplenty of room for curiosity on the hardware too11:29
ygk_12345@cdent appreciate Linus Torvalds for his first version of Linux11:30
* cdent does11:31
*** cdent has quit IRC11:37
*** ygk_12345 has quit IRC11:38
*** awalende has quit IRC11:46
*** awalende has joined #openstack-nova11:47
*** awalende has quit IRC11:47
*** awalende has joined #openstack-nova11:47
*** cdent has joined #openstack-nova11:52
*** dpawlik has joined #openstack-nova11:56
*** nanzha has quit IRC12:02
*** awalende has quit IRC12:03
*** bbowen has quit IRC12:06
*** cdent has quit IRC12:07
*** mgariepy has joined #openstack-nova12:07
*** awalende has joined #openstack-nova12:07
*** awalende has quit IRC12:09
*** nanzha has joined #openstack-nova12:10
*** ygk_12345 has joined #openstack-nova12:10
*** awalende has joined #openstack-nova12:11
*** udesale has quit IRC12:13
*** udesale has joined #openstack-nova12:16
ygk_12345HI all ?12:17
ygk_12345which wsgi framework is used in nova ?12:17
sean-k-mooneywe kind of have our own https://github.com/openstack/nova/blob/master/nova/api/wsgi.py12:19
ygk_12345ok12:19
ygk_12345does it have a standalone web server  ?12:20
sean-k-mooneyit can run in several different modes12:20
sean-k-mooneyit can run on the built in python webserver with eventlets12:21
sean-k-mooneyor you can run the wsgi service via mod_wsgi or uwsgi12:21
ygk_12345as of now when we install nova-api,  in which mode deso it erun ?12:21
ygk_12345*does it12:22
sean-k-mooneythe nova api console script uses https://github.com/openstack/nova/blob/master/nova/cmd/api.py12:22
sean-k-mooneywhich uses eventlet12:22
sean-k-mooneybut12:22
sean-k-mooneyif you use devstack12:22
sean-k-mooneyor most installers12:22
sean-k-mooneythe nova-api service is not calling that script12:22
*** tbachman has quit IRC12:22
*** ociuhandu has quit IRC12:22
sean-k-mooneyits running the service under uwsgi i think12:22
ygk_12345oh12:23
*** tbachman has joined #openstack-nova12:23
*** ociuhandu has joined #openstack-nova12:23
*** dtantsur is now known as dtantsur|brb12:24
*** macz has joined #openstack-nova12:24
sean-k-mooneyygk_12345: cdent know more about this then i but we moved to testing with extrenal wsgi server implemenation a few release ago12:25
ygk_12345oh ok12:25
*** larsks has quit IRC12:25
*** larsks has joined #openstack-nova12:32
*** cdent has joined #openstack-nova12:33
*** eharney has joined #openstack-nova12:44
*** Luzi has quit IRC12:46
*** bbowen has joined #openstack-nova12:53
*** ttsiouts has quit IRC12:53
*** ttsiouts has joined #openstack-nova12:54
*** dviroel has joined #openstack-nova12:55
*** ttsiouts has quit IRC12:58
*** larsks has quit IRC13:00
*** mgariepy has quit IRC13:00
*** nweinber has joined #openstack-nova13:01
*** larsks has joined #openstack-nova13:01
*** ttsiouts has joined #openstack-nova13:10
*** awalende has quit IRC13:12
*** jangutter_ is now known as jangutter13:13
*** jawad_axd has quit IRC13:15
*** jawad_axd has joined #openstack-nova13:16
*** mriedem has joined #openstack-nova13:16
*** bbowen has quit IRC13:19
*** ygk_12345 has quit IRC13:20
*** jawad_ax_ has joined #openstack-nova13:21
*** jawad_axd has quit IRC13:21
*** ociuhandu has quit IRC13:25
*** jawad_ax_ has quit IRC13:26
*** ociuhandu has joined #openstack-nova13:26
*** ociuhandu has quit IRC13:31
*** bbowen has joined #openstack-nova13:31
*** eharney has quit IRC13:33
*** rcernin has quit IRC13:40
KeithMnemonicmriedem Good day, what is the best was to get some reviews on the patch you submitted for me ? https://review.opendev.org/#/c/683008/13:42
*** ircuser-1 has quit IRC13:42
*** ricolin has quit IRC13:43
mriedemKeithMnemonic: umm13:43
mriedemyou know we won't release that right?13:43
mriedemsince pike is in extended maintenance mode. we can merge stuff to pike but it won't be released.13:43
mriedemso if you really need it you could probably just cherry pick it downstream13:44
*** larsks has quit IRC13:44
*** larsks has joined #openstack-nova13:46
*** mgariepy has joined #openstack-nova13:46
KeithMnemonicok, i can do that, i was hoping it would get merged at least , is that not an option anymore even it if it is not released13:46
*** dave-mccowan has joined #openstack-nova13:46
mriedemit could be merged upstream but finding people to care about pike at this point upstream is hard, and that patch isn't exactly trivial13:47
mriedemwith all of the conflicts13:47
dansmithmriedem: api patch landed, can you kick this in? https://review.opendev.org/#/c/687141/13:48
KeithMnemonicok let me try and cherry pick it locally13:48
mriedemdansmith: were you going to reply to andrey again before doing so?13:49
mriedemhe is a core so i feel sort of bad if i just overrule him without a reply13:50
dansmithokay13:51
*** xek has joined #openstack-nova13:52
*** mlavalle has joined #openstack-nova13:53
*** pcaruana has quit IRC13:53
*** ab-a has quit IRC13:53
mdboothOh, that's evil13:54
*** psachin has quit IRC13:54
*** ab-a has joined #openstack-nova13:54
mdboothIn python2, if your module is in nova.tests.functional and you 'import fixtures', you get nova.tests.functional.fixtures. In python3 you get 'fixtures'.13:54
mdboothApart from renaming nova.tests.functional.fixtures, I'm not immediately sure wtf to do about that.13:55
*** dave-mccowan has quit IRC13:56
*** munimeha1 has joined #openstack-nova13:57
mdboothFun fact 2: emojis in review comments elicit a 500 from gerrit.13:57
sean-k-mooneymdbooth: that to do with relitive imports and what happens if you have a module in the local director that alisas a module in your path13:58
mdboothsean-k-mooney: Any idea how to disable in py2 in a manner which doesn't break py3?13:58
sean-k-mooneythere might be a futre thing for this. absolute import or something like htat13:58
melwittKeithMnemonic, mriedem: I started reviewing that patch yesterday but not done yet. it's taking forever bc of all the conflicts. I'll finish reviewing today13:59
KeithMnemonicthanks!13:59
sean-k-mooneyhttps://www.python.org/dev/peps/pep-0328/13:59
sean-k-mooneyfrom __future__ import absolute_import13:59
sean-k-mooneybut that shoudl be enabled by default in py 2.714:00
mdboothApparently not14:00
bauzasmdbooth: to make it clear, nobody should just import any module like "import <module>" but rather doing "from <where> import <module>"14:05
bauzasif I'm reviewing any change like this, I'd -1 it14:06
melwittmdbooth: there shouldn't be a module nova.tests.functional.fixtures module, there's one in nova.tests.fixtures though14:09
melwittI think that's how this hasn't come up before14:09
*** jawad_axd has joined #openstack-nova14:09
*** jawad_axd has quit IRC14:10
*** jawad_ax_ has joined #openstack-nova14:11
*** jawad_ax_ has quit IRC14:11
efried_afkmdbooth: you need14:11
efried_afkfrom __future__ import absolute_import14:11
*** efried_afk is now known as efried14:11
efriedoh, I'm late to the party.14:12
efriedif we could just merge the "kill py2" patch, we could stop doing this ick14:12
sean-k-mooneyyes14:14
sean-k-mooneyalthough we shoudl still keep it in mind if we backport things14:14
*** macz has quit IRC14:17
*** dpawlik has quit IRC14:18
*** jangutter has quit IRC14:19
melwittbauzas: well, if you need to import the base fixtures module (https://pypi.org/project/fixtures/) there's no other way than 'import fixtures', right?14:27
*** larsks has quit IRC14:29
*** larsks has joined #openstack-nova14:30
mriedemmelwitt: nova.tests.functional.fixtures is where the placement fixture is14:34
mriedemit got moved there after ripping placement out of nova14:35
melwittoh, I see. I forgot it was just 'fixtures' there14:36
melwittmah bad14:36
openstackgerritEric Fried proposed openstack/nova master: Always trait the compute node RP with COMPUTE_NODE  https://review.opendev.org/68897914:43
openstackgerritEric Fried proposed openstack/nova master: ItemsMatcher: mock call list arg in any order  https://review.opendev.org/68948714:43
efriedgibi: You'll like ^ :)14:44
*** sapd1 has quit IRC14:47
*** bnemec has quit IRC14:49
*** sapd1 has joined #openstack-nova14:52
bauzasmelwitt: sorry was afk, I meant to not import any module by relative14:52
bauzasany import should be absolute14:53
bauzasthat's it14:53
melwittbauzas: np. I know, I'm saying how do you do the "from <where> import <module>" if you need https://pypi.org/project/fixtures ?14:53
bauzasmelwitt: "import fixtures" is absolute for it :)14:54
bauzasI meant for packages14:54
bauzasfixtures is both the package and the module you wanna import14:54
bauzasnova.tests.unit.fixtures isn't14:54
*** bnemec has joined #openstack-nova14:55
* bauzas would love sam&max to be English14:55
bauzashttp://sametmax.com/les-imports-en-python/14:55
melwittright, but if your test is in nova/tests/functional/my_test.py and you need 'import fixtures' and there is a nova/tests/functional/fixtures.py it would pick up the latter, we now know14:56
melwittanyway, just saying I don't see a way to avoid the problem other than using 'from __future__ import absolute_import' as has been mentioned earlier14:56
*** psachin has joined #openstack-nova14:57
*** dtantsur|brb is now known as dtantsur14:57
mdboothThanks, all! I'll update the test.14:58
*** gbarros has joined #openstack-nova14:58
* mdbooth notes that the following did not work:14:59
mdboothimport fixtures as fucking_fixtures14:59
mdbooth__future__ works :)15:00
*** tbachman has quit IRC15:00
*** FlorianFa has quit IRC15:01
openstackgerritMatthew Booth proposed openstack/nova master: Unplug VIFs as part of cleanup of networks  https://review.opendev.org/66338215:02
*** bnemec is now known as beekneemech15:02
*** ttsiouts has quit IRC15:03
*** mgariepy has quit IRC15:03
*** ttsiouts has joined #openstack-nova15:03
mdboothefried: I split out the regression test from the new bugfix, btw. Turned it into a double regression test with 2 separate fixes: https://review.opendev.org/#/c/689278/15:03
efriedmdbooth: I was thinking that might be appropriate, but I was too fried to make a stink about it yesterday. Thanks.15:04
mdboothefried: Yeah, np. Thanks for looking.15:05
efriedcdent, gibi, bauzas, mriedem: I think all the ducks are lined up for: Always trait the compute node RP with COMPUTE_NODE  https://review.opendev.org/68897915:06
mdboothefried: I was also fried, and extremely annoyed at finding a new bug I wasn't looking for.15:06
efriedand btw, "if you build it, they will come" -- I came across another use case for it yesterday https://review.opendev.org/#/c/676522/21/nova/compute/resource_tracker.py@182315:06
*** gyee has joined #openstack-nova15:07
*** sapd1 has quit IRC15:08
bauzasefried: cool, I'll just honestly leave it for monday15:08
*** ociuhandu has joined #openstack-nova15:08
*** ttsiouts has quit IRC15:08
bauzas(and with no kids at home, my productivity will dramatically increase)15:08
efriedbauzas: are they "not at home" like moved out, or just on vacation?15:09
bauzasboth, grand-parents care15:10
bauzasfor the week \o/15:10
efriedwoot15:10
bauzasindeed15:10
mdboothbauzas: Party at your house :)15:10
*** udesale has quit IRC15:10
bauzastechnically, our friends and us have no kids at same time15:11
bauzasI don't wanna rush into details about the fact we already planned the whole week15:11
*** udesale has joined #openstack-nova15:11
mdboothbauzas: Why haven't you taken the week off?15:11
bauzasbecause I still love my job and despite the fact it looks like, I'm not on perpetual PTO (c) mriedem15:12
*** ociuhandu has quit IRC15:12
mdboothHe's just jealous15:13
bauzasI'm pretty sure France still provides work permits for people who'd like to :p15:14
bauzasthere is one drawback tho15:14
bauzasyou have to leave in France15:14
*** TxGirlGeek has joined #openstack-nova15:14
* mdbooth would like a little British enclave in the alps where everybody speaks english and you can get normal tea. Something like the Costa Del Alps.15:16
*** mgariepy has joined #openstack-nova15:16
cdent"normal tea"15:17
cdentsuch a british thing to say :P15:17
bauzasmdbooth: it exists, it's called Courchevel15:17
bauzaswell, you also have to speak russian too15:17
mdboothcdent: You know, tea that's just normal. Like PG Tips or Yorkshire Tea. Not fancy. Just... normal.15:19
bauzasmdbooth: do you think British MPs would be more inclined to accept the deal if we would add a clause for selling Courchevel ? Honestly, it's not like we miss anything15:19
bauzas(sorry, Friday here)15:19
mdboothbauzas: I tried to put myself in the mind of a British MP once. I awoke from a coma 2 months later. Haven't tried again.15:20
*** gbarros has quit IRC15:20
cdentonly 2 months?15:23
* cdent steps outside15:23
sean-k-mooneycdent: my mother refers to barrys or lyons tea as normal tea15:24
sean-k-mooneycdent: so it translate to ireland too15:24
*** tbachman has joined #openstack-nova15:30
*** eharney has joined #openstack-nova15:30
*** igordc has joined #openstack-nova15:32
*** beekneemech has quit IRC15:34
*** macz has joined #openstack-nova15:38
*** ociuhandu has joined #openstack-nova15:43
*** ociuhandu has quit IRC15:43
*** ociuhandu has joined #openstack-nova15:44
mriedemefried: i'm going to have to take a different approach on https://review.opendev.org/#/c/689049/ where the caller has the context, like this: http://paste.openstack.org/show/784726/15:46
mriedemand pretty much leave move_allocations as-is15:46
mriedemif we had consumer types i could determine from move_allocations if the instance still exists or not...15:46
mriedemor just pass the types into move_allocations but that kind of sucks15:47
efriedmriedem: don't we still have a race in that situation?15:48
mriedemas in when i refresh the instance is still exists but is gone by the time move_allocations does the POST?15:48
efriedyeah15:49
efriedSmall window -- but the window you're closing is pretty durn small too.15:49
mriedemi guess that's possible, but i i'm not sure how to detect that when consumers in placement are ephemeral15:49
mriedemmeaning when we get the consumer allocs we pass the generation if we have one otherwise we just pass None if it's a new consumer15:50
efriediow you need to know *before* you decide what to do in placement?15:50
efriedInstead of refreshing the instance, you could GET its allocations from placement.15:51
efriedthen the window is n/a because when you push back if the generation doesn't match you'll blow up.15:51
mriedemwhich is what move_allocations does15:51
*** damien_r has quit IRC15:52
efriedmriedem: let me stare at it a bit after my meeting here.15:52
efried(by which time you'll have probably figured it out anyway)15:52
mriedemdealing with a head cold so my brain isn't really functioning15:53
*** dtantsur is now known as dtantsur|afk16:01
*** panda has quit IRC16:01
*** panda has joined #openstack-nova16:03
*** rpittau is now known as rpittau|afk16:04
*** ociuhandu has quit IRC16:11
*** nanzha has quit IRC16:11
openstackgerritDan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines  https://review.opendev.org/68714116:16
openstackgerritDan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query  https://review.opendev.org/68953716:16
*** ociuhandu has joined #openstack-nova16:19
*** gbarros has joined #openstack-nova16:23
*** ociuhandu has quit IRC16:24
*** markvoelker has quit IRC16:26
*** markvoelker has joined #openstack-nova16:27
*** dlbewley has quit IRC16:35
*** igordc has quit IRC16:40
*** gbarros has quit IRC16:51
*** gbarros has joined #openstack-nova16:57
*** lpetrut has quit IRC17:09
efriedmriedem: when an instance is in CONFIRM_RESIZE, is the migration allocation still on the dest?17:09
eanderssonWould enabling randomize_allocation_candidates help with race conditions (RescheduleException)?17:15
eanderssonWe heavily favor stacking, which tends to cause race conditions when deploying a lot of small flavors.17:15
eanderssonEspecially bad when deploying 1k+ VMs17:16
*** ociuhandu has joined #openstack-nova17:18
efriedeandersson: are you deploying them with a single command (max_instances=N or whatever that thing is) or multiple concurrent/quick operations?17:18
*** tesseract has quit IRC17:18
*** tbachman has quit IRC17:19
eanderssonmultiple concurrent using terraform or ansible17:20
eanderssonif I do a single command it works fine17:20
dansmithefried: the migration one is yeah17:20
eanderssonwe have a lot of custom weighers and filters that probably make things.. slower (more likely to end in a race condition)17:20
dansmithefried: the instance one is on the dest17:20
dansmithefried: sorry, the migration allocation is on the _source_, the instance is on the _dest_17:21
dansmithassume that's what you meant17:21
eanderssonWe used host_subset_size and max_attempts to counter it17:21
*** ociuhandu has quit IRC17:22
*** cdent has quit IRC17:25
efriedmriedem: Correct me if I'm wrong, move_allocations needs to deal with these scenarios:17:34
efried- (live/cold) migrate execute: source is instance, target is migration.17:34
efried- revert, including from a failed migration: source is migration, target is instance.17:34
efried    - In either case, if instance disappears, we want the operation fail, but that's something the caller needs to be aware of, not move_allocations itself.17:34
efried    - If an oob operation updates the instance's allocs (which only applies to the execute case; I assume you can't resize an instance that's in CONFIRM_RESIZE or ERROR, right?) we also want the migration to fail.17:34
efried      However, just like the delete thing that started you down this rathole, we can't close that window effectively while doing the GET from within move_allocations17:34
efriedSo here's what I think needs to happen:17:34
efried- move_allocations should always use 'allocations': {} to delete, never DELETE.17:34
efried- move_allocations should *not retry on a consumer 409*; it should raise AllocationMoveFailed. It *should* retry on a *provider* 409 though. (This is a little complicated, and we have seen this issue elsewhere -- gibi L@@K.)17:34
efried- Eventually (probably not now) the calling flows should be responsible for pulling the instance consumer (at least the generation, if not the whole allocation record) *early*, before entering the "critical path", and funneling that information into move_allocations. This is the only way we're going to effectively detect the race.17:34
efriedTL;DR the only thing that should change right now is: don't retry on consumer 409.17:34
*** markvoelker has quit IRC17:34
efriedeandersson: randomize sounds like it would help there, yes.17:35
efriedeandersson: when you said you favor "stacking", did you mean "packing" (as opposed to "spread")?17:36
efriedIf you're doing that ^ via your weighers/filters, then randomize probably won't do you a lot of good.17:37
efriedbecause your weigher will trawl down the list, random or not, and pick the fullest host anyway.17:38
melwitteandersson: what version are you running again? as of claims in placement, the rescheduling due to racing parallel requests shouldn't be a thing17:38
efriedmelwitt: he's using separate parallel processes17:39
melwittI know17:39
efriedso each is doing a GET /a_c at the same time, getting the same answers, and all thinking they can hit the same host at the same time.17:39
efriedeven with three retries, that'll race easily.17:39
melwittthe whole point of claims in placement was to stop that though, that was the "multiple schedulers" problem of old. am I missing something?17:40
efriedmore likely I am17:40
eanderssonefried, correct, packing17:41
efriedeandersson: If you did randomize and then fiddled around with a small ?limit= you might be able to reduce a little bit. But none of this sounds like a really good approach.17:41
eanderssonwe have a lot of game servers that, and our goal is to fit as many game servers as possible17:42
eanderssongame servers take up one numa17:42
eanderssonbut usually there are 2-4 vcpus left17:42
eanderssonso our filter/weighers tries to put small vms on top of that17:42
eanderssonwithout taking up enough to prevent a "game server" from fitting17:43
eanderssone.g. if small vms take up 6 cores of a numa, no large server could fit17:43
efriedare the races happening on "small vm" or "game server"?17:43
eanderssonsmall vms only17:43
efriedwell, you said "numa", which could be the crux of the problem.17:43
efriedplacement doesn't know about numa yet.17:43
eanderssonbecause I think what is happening is that we have 2-3 computes  with the "perfect fit" so they all try to land at those17:43
melwittplacement claims were added in pike https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/placement-claims.html17:43
eanderssonthis is rocky btw17:44
melwittoh, numa isn't checked until after it lands on the compute, so that's why you're hitting this17:44
melwittyeah17:44
efriedSo if you have 1vcpu in each of two numas and you ask for 2VCPU, placement will give you what it thinks is a viable candidate17:44
melwitt(as efried said earlier)17:44
efriedbut the numa topo filter will punt it.17:44
efriedeandersson: does your "small vm" have a numa topo specified?17:45
*** eharney has quit IRC17:45
eanderssonyea17:45
efrieddoes it need to? :P17:45
efriedif you allow it to float across numas, I suspect your race will go away.17:45
eanderssonI would assume so, but not 100% sure17:45
eanderssonIf just ranzomizing the results a bit fixes it I probably prefer that route17:46
eanderssonbut if that isn't enough I could look into doing that17:46
melwittfwiw I think you're already doing all you can to mitigate, by increasing the subset size and increasing retries. there's another option that could help, https://docs.openstack.org/nova/rocky/configuration/config.html#filter_scheduler.shuffle_best_same_weighed_hosts17:47
melwitt(assuming you can't change anything about the numa)17:48
eanderssoninteresting, didn't know about that setting17:48
melwittyeah, that's the only other one involved in all this that you haven't looked at already17:48
efriedeandersson: if the problem is numa as described, randomizing won't help at all.17:49
openstackgerritDan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query  https://review.opendev.org/68953717:49
openstackgerritDan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines  https://review.opendev.org/68714117:49
eanderssonI see - I think what melwitt just linked might be the issue17:49
melwitteandersson: the randomize_allocation_candidates won't help you I don't think because it returns things in a deterministic order (so as not to necessarily break scheduling stacking)17:49
openstackgerritDan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query  https://review.opendev.org/68953717:50
openstackgerritDan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines  https://review.opendev.org/68714117:50
efriedmelwitt: "it returns things" <== "it" being the filter?17:51
melwittno, placement17:51
efriedwith randomize_allocation_candidates placement does *not* return things in deterministic order.17:52
melwittoh, lol, sorry I read the help backwards17:52
efriedyeah, with it *off* the order is deterministic (but inscrutable, based on natural db order of some nonobvious kind)17:53
melwittright17:53
melwitteandersson: ^ ignore what I said earlier about randomize_allocation_candidates17:54
eanderssonrandomize_allocation_candidates = good?17:54
*** udesale has quit IRC17:56
efriedeandersson: if the problem is the numa thing, randomize_allocation_candidates = irrelevant.17:56
*** TxGirlGeek has quit IRC17:57
melwitteandersson: what efried said. it won't help you because your problem is the numa fitment. I was just correcting what I said about it being a deterministic order, it would be the opposite of that, a random order17:57
eanderssonI see - I assumed so, but wanted to make sure17:57
melwittI read the help backwards and... I don't have much of an excuse17:57
efriedThe only way it would be at all likely to help you is if $number_of_concurrent_deploys < $number_of_possible_hosts *and* you use a ?limit < $number_of_possible_hosts17:58
efriedbut like I said, that's a pretty hacky thing to try to figure out17:58
melwitteandersson: the shuffle_best_same_weighed_hosts might help you, I think. numa stuff might be why it was added in the first place17:59
*** TxGirlGeek has joined #openstack-nova17:59
eanderssonYea - sounds super promising.17:59
melwitt+1 efried. eandersson: the randomize_allocation_candidates use case is for very large deployments where the default limit for returned candidates is < the total number of compute hosts in the deployment. think CERN with multiple cells18:02
eanderssonJust fyi we have ~1k nodes in each deployment18:02
melwittack18:03
eanderssonper cell18:04
melwittoh, ok that is a lot18:06
eanderssonWe use regions instead of cells at the moment, but might want to change that in the future18:06
melwitthere's the background on the option https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/allocation-candidates-limit.html18:08
eanderssonWe limit ourself to 1k nodes today because we don't know if neutron can handle it18:08
eandersson(used to be nova + neutron, but doubt nova will have a problem today)18:09
eanderssonwhen we upgraded to rocky neutron used up like 500GB memory lol18:09
eanderssoneach neutron process peaked at 8.2GB memory used18:10
melwittgotcha. belmiro from CERN would be the one to chat with about neutron scaling18:10
eanderssonWe work with them a lot18:10
melwittI don't remember if they are doing something special re: neutron18:10
eanderssonThey use one big Layer 2 network18:10
melwittok cool18:10
melwittand that's... a lot of memory O.o18:11
eanderssonYea it's nuts. I tend to ping the neutron channel but no one ever replies18:11
eanderssonNova channel is way better :p18:11
melwittah yeah, I think they might be mostly EU timezone18:12
eanderssonSome awesome people here, melwitt efried sean-k-mooney mnaser mriedem to name a few ;)18:12
melwitt:)18:12
mnaser\o/18:13
mnasereandersson: will you be at shanghai?18:13
eanderssonI will miss shanghai :'(18:13
eanderssonWill be attending the next one for sure18:13
*** psachin has quit IRC18:15
eanderssonWhich is a bummer, especially since Train has been the largest contribution from us yet :P18:15
eandersson90% of that was to Designate hehe18:16
*** igordc has joined #openstack-nova18:17
openstackgerritMerged openstack/nova stable/queens: Drop source node allocations if finish_resize fails  https://review.opendev.org/68272218:17
mnasereandersson: awesome18:20
eanderssonbtw some more shameless pr, but if you know anyone in South California or Austin Texas, we are hiring OpenStack people (systems and/or software)18:25
eandersson*that is looking for work18:25
*** tbachman has joined #openstack-nova18:26
*** nsherry4 has joined #openstack-nova18:32
*** nsherry4 has quit IRC18:32
*** ircuser-1 has joined #openstack-nova18:43
efriedeandersson: not that I'm looking, who's "we"?18:50
efriedI'm in Austin, but have a cushy wfh gig. Wouldn't consider anything with a commute longer than eleven seconds.18:51
dansmithagree, although the traffic on my commute is getting ridiculous18:52
dansmithearlier this week, I had to wait for the cat to pass before I made a left at the stairs18:52
dansmithmajor road rage situation18:52
*** trident has quit IRC18:57
*** eharney has joined #openstack-nova18:58
*** TxGirlGeek has quit IRC18:59
*** trident has joined #openstack-nova19:00
mordredefried: I believe eandersson is at blizzard19:02
efriedthat makes sense19:03
efriedAre game companies still as crazy as they were in the 90s? That's the last time I paid attention.19:04
efriedpeople living in their office for a month at a time, kind of thing19:05
mordredI live in my office for a month at a time19:06
efriedaaand you're crazy, what's your point?19:09
efriedoh, that things other than game companies can be crazy, got it.19:09
*** dave-mccowan has joined #openstack-nova19:10
*** ircuser-1 has quit IRC19:15
*** ircuser-1 has joined #openstack-nova19:17
openstackgerritEric Fried proposed openstack/nova master: ItemsMatcher: mock call list arg in any order  https://review.opendev.org/68948719:18
openstackgerritEric Fried proposed openstack/nova master: Always trait the compute node RP with COMPUTE_NODE  https://review.opendev.org/68897919:18
* efried procrastinates ^19:18
mordredefried: yeah. I mostly meant, you know - like, my house, which is also my office19:23
mordredefried: I'm maybe not as funny or cleve as I sometimes think19:24
efriedOh, you are, but probably not for the *reasons* you think.19:24
*** osmanlicilegi has joined #openstack-nova19:24
efriedSee, my funniness is inversely proportional to how much I'm trying. There's no surprise funny from me.19:25
*** markvoelker has joined #openstack-nova19:36
*** mdbooth has quit IRC19:37
*** markvoelker has quit IRC19:38
*** markvoelker has joined #openstack-nova19:38
*** mdbooth has joined #openstack-nova19:39
*** gyee has quit IRC19:39
*** TxGirlGeek has joined #openstack-nova19:39
*** gyee has joined #openstack-nova19:39
*** ociuhandu has joined #openstack-nova19:45
*** mgariepy has quit IRC19:47
*** ociuhandu has quit IRC19:50
*** igordc has quit IRC19:54
*** ralonsoh has quit IRC20:08
*** igordc has joined #openstack-nova20:23
*** TxGirlGeek has quit IRC20:35
eanderssonefried, yea unfortunately still all about that office20:50
eanderssonhttps://careers.blizzard.com/en-us/openings/o6vmbfwS (Austin) https://careers.blizzard.com/en-us/openings/ohdYafwm (Irvine)20:53
*** nweinber has quit IRC21:03
*** lpetrut has joined #openstack-nova21:06
*** lpetrut has quit IRC21:13
*** markvoelker has quit IRC21:14
*** maciejjozefczyk has quit IRC21:19
*** dklyle has quit IRC21:30
mriedemgmann: remember how we thought host_status would be returned for GET /servers/detail with servers from down cells? i found that won't happen21:38
mriedembecause of this filtering https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/views/servers.py#L59021:38
mriedemefried: sorry about the late reply, i've been not around so much today - i took your comments and threw them into my patch so i can process them on monday21:39
efriedsaw dat, get some rest and feel better.21:39
efried(How) does nova (via oslo.config?) vet permissions of nova.conf?21:53
efriedor is that completely the responsibility of the admin?21:54
sean-k-mooneyefried: i think its up to the admin/installer22:00
sean-k-mooneye.g. we dont check that the config is 600 or 660 as far as i am aware22:01
efriedso if /etc/nova and contents were 5777 and owned by $random we would truck happily along?22:01
mriedemgmann: melwitt: a couple of questions in the host_status_unknown patch - mostly for gmann about policy rule naming and the docs on the policy rule since 2.75+ https://review.opendev.org/#/c/679181/22:01
sean-k-mooneyefried: i think so22:01
efriedo...kay.22:02
sean-k-mooneynova does not create the file and we never write to it22:02
mriedemefried: likely a question for bnemec22:02
sean-k-mooneyso its really up to the install to make sure that files with password in them are not world readable22:02
mriedemi thought there was some talk of some how vaultifying that stuff but i might be dreaming that up22:03
efriedsean-k-mooney: well, that and if they're writable in any way a malicious user could make nova do horrible things to the system22:03
efriedmriedem: I think sdk does something like that.22:03
efriedor at least supports it.22:03
efriedfor clouds.yaml22:03
efriedmordred: ^22:04
efriednot that that's relevant here22:04
efriedI'll ML this sucker.22:04
sean-k-mooneyefried: i mean programs like ssh refuse to work if you dont set the right permisions in your ssh .config or key22:04
sean-k-mooneyso its not unresonable22:04
mordredaroo?22:04
efriedright, swhat I'm sayin, it wouldn't be unreasonable for oslo and/or nova to enforce something like that.22:05
efriedbut that doesn't mean we do22:05
mordredoh - yeah - so - sdk supports clouds.yaml being split into clouds.yaml and secure.yaml22:05
mordredwe do not enforce that secure.yaml is actually secure - although I agree, it would be a reasonably sane thing to do22:05
efriedmordred: before I blast the ML, do you know off the top whether oslo.config does any such validation either?22:06
mriedemhttps://specs.openstack.org/openstack/oslo-specs/specs/stein/secret-management-store.html ?22:06
sean-k-mooneyit looks like kolla atleast makes it 660 https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/config.yml#L5722:07
mordredefried: I do not know anything about oslo.config22:07
sean-k-mooneythey will be own by the kolla:kolla user and group by default22:07
sean-k-mooneywell on the host22:08
sean-k-mooneythey will be own by nova in the containers22:08
mriedemefried: that spec looks pretty close but i don't see any code ever showed up for it22:08
efriedeven if it was coded per the spec, though, it would only mean that you *could* secure your configs; it'd still be n/a for anyone using nova.conf22:09
sean-k-mooneymriedem: i do think your recalection is right we did talk about vaultifing this at some point22:09
sean-k-mooneyim not sure it was that oslo spec specificaly but seam likely that was related22:10
*** dklyle has joined #openstack-nova22:10
mriedemanywho, i'm going to drop now, have a good weekend o/22:13
efriedo/22:13
*** mriedem has quit IRC22:13
efrieddrink lots of fluids. Suck on zinc.22:13
sean-k-mooneyo/22:13
sean-k-mooneyefried: looks like kolla locks it down to 600 in the containers https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-cell/templates/nova-compute.json.j2#L822:14
efriedgtk22:14
sean-k-mooneyyep it means that at least implcitly there is nothing that requires the group to be able to read nova.conf22:15
sean-k-mooneythat makes sense however nothing out side of nova should read nova's config22:15
*** dklyle has quit IRC22:15
*** david-lyle has joined #openstack-nova22:15
sean-k-mooneymordred: by the way is there any reason not to use the cloud.yml vs the old rc files22:17
sean-k-mooneyi assume no but just said i woudl ask22:17
efriedyou mean in nova?22:18
mordredsean-k-mooney: you should never use old rc files they are evil and horrible22:18
efriedor in sdk?22:18
efriedoh, I don't know what an "old rc file" is, /me stfu22:18
sean-k-mooneyi mean when im runing the openstack cli22:18
mordredefried: the files with env vars set in them22:18
sean-k-mooneyya those22:18
sean-k-mooneythe openrc file that devstack has22:18
mordredsean-k-mooney: yeah - definitely not - clouds.yaml is much betterer than env var files22:18
mordredthe only reason to keep those openrc files around are the legacy cli clients22:19
sean-k-mooneyright which im trying not to use anymore22:19
sean-k-mooneywe should file a bug with horizon to not default to the RC files22:19
mordred++22:20
sean-k-mooneyyou can download the cloud.yaml too but its not the default22:20
mordredsean-k-mooney: I've also been meaning to write something like "openstack login" or similar that would let you slurp in a clouds.yaml you downloaded and maybe prompt you for your password and write it to secure.yaml or something like that22:21
mordredbut - you know - ENOTIME22:21
sean-k-mooneyi totaly dont just edit to add my password...22:21
sean-k-mooneyyou know what i would love22:21
sean-k-mooneya way to use keystoen with an ssh key22:21
sean-k-mooneyjust let me upload my public key as an alternitive to a password then never ask me again.22:22
mordredsean-k-mooney: ++22:23
*** slaweq has quit IRC22:24
mordredsean-k-mooney: I truly do not understand why this is not an actual option22:24
sean-k-mooneytime22:24
sean-k-mooneyalso i dont think we ever asked the keystone folks22:24
mordredI guess that's our fault22:25
sean-k-mooneythat would also be a way to secure the secure.yml i guess22:26
sean-k-mooneyanyway i shoudl actully finish up. enjoy the weekend o/22:27
*** tbachman has quit IRC22:28
mordredyou too!22:29
*** gbarros has quit IRC22:49
*** macz has quit IRC22:50
*** gbarros has joined #openstack-nova22:51
efriedI'm outta here as well o/23:01
*** slaweq has joined #openstack-nova23:11
*** markvoelker has joined #openstack-nova23:15
*** slaweq has quit IRC23:15
*** gbarros has quit IRC23:16
*** markvoelker has quit IRC23:19
*** tbachman has joined #openstack-nova23:35
*** gyee has quit IRC23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!