Tuesday, 2014-01-14

*** boris-42 has quit IRC00:32
*** sdake has quit IRC00:34
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Make it possible to customise overcloud names.  https://review.openstack.org/6646700:38
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Use the JSON description to probe the seed VM.  https://review.openstack.org/6624800:38
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Use the test env JSON to detect remote VM hosts.  https://review.openstack.org/6624900:38
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Start to split devtest_setup out more.  https://review.openstack.org/6616500:38
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Populate devtest_variables from devtest_setup.  https://review.openstack.org/6618000:38
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Cleanup use of SEED_IP.  https://review.openstack.org/6625600:38
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Allow passed-in TE_DATAFILE environments.  https://review.openstack.org/6624300:38
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Create and use a JSON testenvironment file.  https://review.openstack.org/6618300:38
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: Fix typo.  https://review.openstack.org/6624500:38
openstackgerritlifeless proposed a change to openstack/tripleo-incubator: No longer export DEPLOY_IMAGE_ELEMENT  https://review.openstack.org/6615700:38
*** newell has quit IRC00:42
*** e0ne has quit IRC00:46
*** e0ne has joined #tripleo00:48
*** CaptTofu has quit IRC00:51
StevenKlifeless: That was rebasing the remaining stack?01:01
lifelessStevenK: fixing the install-dependencies conflict01:02
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Update apache2 to use the os-apply-config dir.  https://review.openstack.org/6647001:07
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Update cinder to use the os-apply-config dir.  https://review.openstack.org/6647101:07
lifelessI wonder if I've found another bad machine01:10
*** nosnos has joined #tripleo01:13
*** e0ne has joined #tripleo01:15
*** e0ne_ has joined #tripleo01:16
*** vkozhukalov has joined #tripleo01:17
*** e0ne_ has quit IRC01:18
*** e0ne has quit IRC01:20
lifelessah yes, it's marked bust by clint01:23
lifelessbnut it's still registered01:24
*** xuhanp has joined #tripleo01:44
SpamapSlifeless: perhaps it got re-reg'd somehow?01:45
SpamapSlifeless: the one I took out seemed to not be on the network01:46
lifelessSpamapS: dunno. I have changed the service host to 'disabled' which I hope will be self documenting01:46
lifelessSpamapS: better than unregistering01:46
SpamapSthere's a disabled?01:46
lifelessupdate bm_nodes set service_host='disabled' where uuid='....';01:46
SpamapSI recall looking for that and failing to find it01:46
SpamapSah so not in the API01:46
lifelessif service_host doesn't match the nova-compute service hostname, then the node won't be advertised to the scheduler01:47
openstackgerritA change was merged to openstack/tripleo-incubator: Start to split devtest_setup out more.  https://review.openstack.org/6616501:49
openstackgerritA change was merged to openstack/tripleo-incubator: Populate devtest_variables from devtest_setup.  https://review.openstack.org/6618001:50
openstackgerritA change was merged to openstack/tripleo-incubator: No longer export DEPLOY_IMAGE_ELEMENT  https://review.openstack.org/6615701:51
StevenK\o/01:51
openstackgerritA change was merged to openstack/tripleo-incubator: Create and use a JSON testenvironment file.  https://review.openstack.org/6618301:51
openstackgerritA change was merged to openstack/tripleo-incubator: Allow passed-in TE_DATAFILE environments.  https://review.openstack.org/6624301:53
openstackgerritA change was merged to openstack/tripleo-incubator: Fix typo.  https://review.openstack.org/6624501:53
lifelessctlplane=10.10.16.177, 10.10.16.17701:54
lifelessodd01:54
StevenKlifeless: Is that the full stack that we did on Sunday?01:56
lifelessStevenK: I think so01:56
StevenK\o/01:56
SpamapShttps://review.openstack.org/#/c/66248/ <-- not sure why we need jq there02:01
lifelessSpamapS: to update the JSON02:02
SpamapSlifeless: oh jq is write? hm02:04
lifelessits both02:05
lifelessbut the thing we needed was write02:05
lifelessthat sets the attribute to the given value02:05
lifelessand outputs the entire JSON again02:05
lifelesswhich I then write to a file02:05
SpamapSthat is not at all clear from the code.. but I didn't bother to read the manpage of jq either, so thats my bad. :)02:06
SpamapSwell the "then write to file" is clear :)02:06
lifelessthe -m stuff I'd like to leave, it's kinda meh02:06
SpamapSI was just noting it. :)02:08
SpamapS+2'd02:08
lifeless| 38ad3976-1c81-45b1-a840-97e52a5ff88a | ci-overcloud | CREATE_IN_PROGRESS | 2014-01-14T01:53:38Z |02:11
lifelesshurray up02:11
*** sdake has joined #tripleo02:13
*** sdake has joined #tripleo02:13
lifelessand tada02:21
lifelessa new cloud02:21
SpamapSh02:29
SpamapShm02:29
SpamapSthis haproxy element.. we really should be writing a Neutron LBaaS driver that configures it.02:29
lifelessmaybe02:32
lifelessI think thats worth exploring, certainly02:32
SpamapSI think it would be a ton simpler02:33
SpamapSWe have API's for these things for a reason :)02:33
lifelessOne word: bootstrapcycles02:34
SpamapSWe have neutron-dhcp-agent on the undercloud right? If we have that, we can have neutron-lbaas-agent too02:36
SpamapSlifeless: not sure what that word is?02:36
lifelessSpamapS: does neutron-lbaas depend on any APIs02:36
SpamapSneutron-server rpc certainly02:37
SpamapSand they've talked about having a driver that spins up VMs for haproxy02:37
lifelessremember that heat can't orchestrate the API for us yet02:37
SpamapSHm, I think it can... this is 100% undercloud02:38
lifelessSpamapS: no, it's seed02:38
SpamapSseed? no02:38
SpamapSFor the undercloud bootstrap, yes02:38
lifelessSpamapS: you want the haproxy in the undercloud to be configured by undercloud metadata, which the seed provides.02:38
SpamapSbut that is the "scale out the undercloud" story that we have a ton of weirdness to overcome on.02:38
lifelessSpamapS: right, but we can either a) not have haproxy until we do that, or b) not have LBaaS until we do that.02:39
SpamapSthe haproxy in the undercloud that I'm talking about will be proxying to overcloud machines behind it.02:39
SpamapSlifeless: I want HA in overcloud more than I want HA in undercloud.02:39
SpamapSand would rather not do it two different ways02:40
lifelessSpamapS: I get that; lets deconstruct it a little02:41
lifelessSpamapS: for HAproxy to do it's job, we need N HAProxies02:41
lifelesswith VIP themselves02:41
lifelessso the LBaaS in neutron can lean on floating ip implementations02:42
lifelesswe don't have one02:42
lifelessSpamapS: ahahahahaha 'unexpected poweroff event' on this server.02:50
* lifeless STABS the hardware02:50
SpamapSlifeless: right, but that seems like work we can do in Neutron the same way we made ports work.02:50
lifelessSpamapS: it took 7 months to get ports to work.02:50
SpamapSow02:51
lifelessSpamapS: I'm super super worried about purity over pragmatism here.02:51
SpamapSThis is pragmatism02:51
SpamapSIt will almost "just work" for the overcloud.02:52
SpamapSanyway02:52
SpamapSI get that there are hurdles02:52
lifelessI don't think it will; I think floating ips are going to be a much bigger consideration than you do.02:52
SpamapSand I'm fine with bypassing them for a short term "get 'er done" .. :-P02:52
*** coolsvap has quit IRC02:53
SpamapSanyway.. kids.. family time.. bbl02:53
lifelessciao02:54
lifeless1611-Chassis Fan7 Failure02:54
lifeless1611-Chassis Fan15 Failure02:54
lifelessFan Solution Not Sufficient02:54
lifelessCritical Failure Detected - System Shutting Down in 1 Seconds!02:54
lifelessahahahahahahahahahahahahahahahahahahahahahahahahahahahaha02:54
StevenKHah02:56
*** vipul is now known as vipul-away02:59
*** AaronGr is now known as AaronGr_Zzz03:02
*** vipul-away is now known as vipul03:19
lifelessSpamapS: how do you get heat to use an existing floating IP ?03:22
*** ccrouch has joined #tripleo03:35
*** weshay has quit IRC04:02
*** ramishra has joined #tripleo04:03
*** ramishra_ has joined #tripleo04:05
SpamapSlifeless: you pass the ID in here http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Neutron::FloatingIPAssociation04:08
*** ramishra has quit IRC04:08
StevenKlifeless, devananda: WRT what we discussed with mirror sizes at LCA: http://paste.openstack.org/show/61159/04:09
*** edmund has quit IRC04:32
*** coolsvap has joined #tripleo04:39
*** akuznetsov has joined #tripleo05:10
*** tserong has quit IRC05:17
*** tserong has joined #tripleo05:20
*** tserong has joined #tripleo05:20
*** ramishra_ has quit IRC05:36
*** hewbrocca has quit IRC05:49
*** hewbrocca has joined #tripleo05:49
*** ramishra has joined #tripleo05:57
*** coolsvap has quit IRC06:07
*** boris-42 has joined #tripleo06:18
*** rpodolyaka1 has joined #tripleo06:29
*** coolsvap has joined #tripleo06:32
*** rlandy has joined #tripleo06:46
*** akuznets_ has joined #tripleo06:49
*** akuznetsov has quit IRC06:49
*** mrunge has joined #tripleo06:51
*** vkozhukalov has quit IRC06:53
*** e0ne has joined #tripleo06:57
*** marun has quit IRC07:01
*** boris-42 has quit IRC07:04
*** akuznets_ has quit IRC07:32
*** akuznetsov has joined #tripleo07:33
*** ifarkas has joined #tripleo07:37
*** rpodolyaka1 has quit IRC07:41
*** jtomasek has joined #tripleo07:47
*** jcoufal has quit IRC07:48
*** ramishra has quit IRC07:54
*** jistr has joined #tripleo08:01
lifelessNg: so - new cloud is setup, creds for it are in ~/tripleo-ci-* and ~/os-asserted-users on the undercloud08:07
lifelessNg: it has SSL08:07
*** ifarkas has quit IRC08:07
lifelessNg: and the tripleo-bm-test network setup, in principle08:07
*** xuhanp has quit IRC08:07
lifelessNg: infra should be able to spin up and build nodes on it from the nodepool tenant08:08
* lifeless done for the day08:08
*** jcoufal has joined #tripleo08:08
lifelessNg: https://trello.com/c/Z0WFQaFg/64-dedicated-non-updated-overcloud-until-we-have-nodowntime-deploys08:08
*** bauzas has joined #tripleo08:12
*** e0ne has quit IRC08:17
*** xuhanp has joined #tripleo08:19
rpodolyakamorning tripleo08:20
*** vkozhukalov has joined #tripleo08:27
*** rdopieralski has joined #tripleo08:35
*** kpavel has joined #tripleo08:42
*** kpavel_ has joined #tripleo08:43
*** jprovazn has joined #tripleo08:46
*** pblaho has joined #tripleo08:46
*** kpavel has quit IRC08:47
*** kpavel_ is now known as kpavel08:47
*** rushiagr has joined #tripleo08:53
*** xuhanp has quit IRC08:56
*** jrist has quit IRC09:00
Ngmorning09:03
Nglifeless: nice!09:03
openstackgerritVictor Sergeyev proposed a change to openstack/python-tuskarclient: Use HTTPClient from common apiclient code  https://review.openstack.org/6374909:04
*** coolsvap has quit IRC09:05
*** e0ne has joined #tripleo09:07
*** e0ne_ has joined #tripleo09:08
*** e0ne has quit IRC09:12
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Add Nodes Overview page  https://review.openstack.org/6652209:14
*** ifarkas has joined #tripleo09:18
*** ifarkas has quit IRC09:19
*** ifarkas has joined #tripleo09:19
*** xuhanp has joined #tripleo09:20
*** ifarkas has quit IRC09:31
*** athomas has joined #tripleo09:32
*** derekh has joined #tripleo09:36
*** tserong has quit IRC09:36
*** tserong has joined #tripleo09:38
*** coolsvap has joined #tripleo09:38
*** lsmola_ has quit IRC09:39
*** ifarkas has joined #tripleo09:42
*** markmc has joined #tripleo09:48
SpamapSjprovazn: btw ignore my -1 on haproxy... lifeless set me straight.. reviewing it for real now09:49
jprovaznSpamapS: cool, thanks!09:50
SpamapSjprovazn: though I think I'm going to -1 the approach..09:51
*** lsmola_ has joined #tripleo09:51
SpamapSjprovazn: I think it would be a bazillion times simpler to just have default ports (proxy and frontend) in the heat templates than in the images.09:51
SpamapSjprovazn: that way we can just configure things the way we usually do.. with Heat metadata.09:52
jprovaznSpamapS: well, I defined both port and proxy port in heat template in the previous patchset (check prev version), the problem is that knowledge of proxy port is needed on other places than haproxy config file too: when configuring iptables, it will also be needed when setting up custom endpoints09:55
jprovaznSpamapS: and I don't think that it's needed to have proxy port cinfgurable, choosing just some suitable set of ports should work fine09:56
SpamapSjprovazn: its not so much the need, but the fact that we're now using new config tools, and I'm not sure why.09:59
SpamapSjprovazn: all of those other places will have access to the ports..09:59
SpamapS?09:59
jprovaznSpamapS: what new config tools do you mean? save-proxy-setting script?10:00
SpamapSjprovazn: os-apply-config --key haproxy.services.keystone.port  no?10:00
SpamapSjprovazn: And the python script to generate the template10:00
jprovaznSpamapS: usage of above commands comes from need to combine knowledge of metadata info and installed elements10:02
jprovaznSpamapS: so elements can say "hey, I'm able to by proxied" and if heat template says "proxy this service" then it's really proxied10:03
SpamapSjprovazn: and what about remote elements?10:03
jprovaznSpamapS: what do you mean by remote element?10:03
SpamapSwhat about load balancing from the controller to a dedicated set of nova-api boxes?10:04
jprovaznSpamapS: still not sure if I understand, but you define set of nodes by haproxy.nodes in heat metadata10:05
jprovaznI mean set of target real nodes10:06
rlandyderekh: thanks for your help - devtest completed :)10:07
derekhrlandy: cool, no problem10:07
SpamapSjprovazn: ok but nova-api wouldn't be on the controller at all at that point10:07
jprovaznSpamapS: you can define any set of nodes (doesn't have to be local) - https://review.openstack.org/#/c/61374/4/elements/haproxy/README.md10:08
SpamapSjprovazn: what I'm getting at is that "installed elements" has nothing to do with "services to provide"10:08
jprovaznah, I see10:09
SpamapSjprovazn: if I just built an haproxy image how would I tell it to listen on all the keystone/etc. etc. ports?10:09
jprovaznmmt, let me check current patchset10:10
*** phschwartz has quit IRC10:10
jprovaznSpamapS: so https://review.openstack.org/#/c/61374/4/elements/haproxy/os-config-applier/etc/haproxy/haproxy.cfg would require an update to match this requirement10:11
*** phschwartz has joined #tripleo10:12
jprovaznSpamapS: is it real use case?10:12
jprovaznSpamapS: and https://review.openstack.org/#/c/61374/4/elements/haproxy/os-refresh-config/pre-configure.d/98-haproxy-iptables has to be updated too10:13
*** jistr has quit IRC10:14
*** xuhanp has quit IRC10:16
SpamapSjprovazn: being able to load balance to nodes that don't have the actual software on them is a real use case yes. It is not our use case today, but it is a pretty common setup.10:16
SpamapSjprovazn: It seems rather narrow minded to rule it out entirely. What was the problem with just passing the ports in via heat templates? (and I'm sorry if I was the one pushing back.. my perspective changes.. often..)10:17
jprovaznSpamapS: getting ports directly of heat template involves more complex iptables rule updating which I didn't like (though I have to admit that I didn't count with "having proxy nodes only w/o services" use case)10:20
jprovaznSpamapS: but there is no technical blocker I guess - os-apply-config --key can be used to get it from heat template10:20
jprovaznSpamapS: and if there will be more configuration of related service requried - all will have to be done in os-collect-config phase10:24
jprovaznSpamapS: e.g. current iptables rules (in boot-stack IIRC) should be then fixed - there is not reason to open default ports for proxy-only nodes10:25
SpamapSjprovazn: it doesn't have to be this 100% flexible, I just want to make sure it will be able to handle real changes in topology.10:26
*** coolsvap has quit IRC10:27
jprovaznSpamapS: another alternative to fetch ports from heat templat ewould be to create ports mapping file on image10:27
SpamapSjprovazn: Lets let others weigh in. I am very tired, and I may be asking too much. :p10:27
jprovaznSpamapS: one file installed as part of some basic element10:27
jprovaznSpamapS: sure10:27
jprovaznSpamapS: it's pretty late in your TZ10:27
jprovazngood night10:28
SpamapSjprovazn: I dislike having the ports for a remote application come from the local image. I think it may be too rigid, that is all.10:28
openstackgerritA change was merged to openstack/diskimage-builder: Install traceroute on Fedora.  https://review.openstack.org/6641110:30
jprovaznSpamapS: well, I meant only proxy ports, not target ports of real application10:30
*** akuznetsov has quit IRC10:31
*** akuznetsov has joined #tripleo10:32
*** lsmola_ has quit IRC10:40
*** max_lobur_afk is now known as max_lobur10:44
*** rpodolyaka has quit IRC10:52
*** lsmola_ has joined #tripleo10:53
openstackgerritA change was merged to openstack/tripleo-heat-templates: Add a wait condition to undercloud stack  https://review.openstack.org/6632310:54
*** jistr has joined #tripleo10:57
openstackgerritA change was merged to openstack/tripleo-image-elements: Install os-collect-config from release into a venv  https://review.openstack.org/6635110:58
openstackgerritA change was merged to openstack/tripleo-image-elements: Install os-refresh-config from release into a venv  https://review.openstack.org/6635210:59
openstackgerritA change was merged to openstack/tripleo-image-elements: Don't install python-pip  https://review.openstack.org/6635310:59
openstackgerritA change was merged to openstack/tripleo-image-elements: Don't install git-core  https://review.openstack.org/6635410:59
*** lucasagomes has joined #tripleo11:00
*** rpodolyaka has joined #tripleo11:07
*** boris-42 has joined #tripleo11:07
openstackgerritA change was merged to openstack/diskimage-builder: Removing the config-applier element  https://review.openstack.org/6633711:09
openstackgerritA change was merged to openstack/diskimage-builder: Retry link check up to 10 times.  https://review.openstack.org/6639711:12
*** martyntaylor has joined #tripleo11:17
*** jcoufal has quit IRC11:47
*** jcoufal has joined #tripleo11:54
*** jtomasek has quit IRC12:10
*** morazi has joined #tripleo12:13
*** jtomasek has joined #tripleo12:13
*** coolsvap has joined #tripleo12:24
derekhis it just me or is network access to the cd-undercloud gone flaky ?12:30
openstackgerritA change was merged to openstack/diskimage-builder: Add package support to source-repositories  https://review.openstack.org/6575412:42
*** coolsvap has quit IRC12:46
*** mrunge has quit IRC12:48
*** rushiagr has quit IRC12:56
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Add Nodes Overview page  https://review.openstack.org/6652213:02
openstackgerritA change was merged to openstack/diskimage-builder: Add new modprobe-blacklist element.  https://review.openstack.org/6642513:05
openstackgerritPetr Blaho proposed a change to openstack/python-tuskarclient: [WIP] Adds help for subcommands  https://review.openstack.org/5625713:07
lsmola_SpamapS: hello13:08
*** jtomasek has quit IRC13:19
*** jtomasek has joined #tripleo13:20
lsmola_SpamapS: can I ask when we are planning to get rid of merge.py and make it via HOT nested resources ?13:22
lsmola_SpamapS: or are there any blockers with this approach?13:23
*** jdob has joined #tripleo13:30
*** e0ne has joined #tripleo13:37
*** pblaho has quit IRC13:40
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Updates cinder element  https://review.openstack.org/6269313:40
*** e0ne_ has quit IRC13:41
*** jayg|g0n3 is now known as jayg13:41
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Updates iptables rule for MySQL element  https://review.openstack.org/6656913:47
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Updates boot-stack iptables file  https://review.openstack.org/6657113:51
*** weshay has joined #tripleo13:51
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Updates gearman iptables file  https://review.openstack.org/6657213:53
*** akuznetsov has quit IRC13:55
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Updates iptables rule for MySQL element  https://review.openstack.org/6656913:57
*** rushiagr has joined #tripleo14:01
openstackgerritMark McLoughlin proposed a change to openstack/tripleo-heat-templates: Avoid echoing comment from makefile recipe  https://review.openstack.org/6657514:03
openstackgerritMark McLoughlin proposed a change to openstack/tripleo-heat-templates: Add all and clean targets to makefile  https://review.openstack.org/6657614:03
*** dprince has joined #tripleo14:04
rbrady Looking for additional reviews for https://review.openstack.org/#/c/62735/14:04
*** lblanchard has joined #tripleo14:05
openstackgerritA change was merged to openstack/tuskar-ui: Add Nodes Overview page  https://review.openstack.org/6652214:11
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Updates boot-stack iptables file  https://review.openstack.org/6657114:12
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Updates iptables rule for MySQL element  https://review.openstack.org/6656914:15
*** ftcjeff has joined #tripleo14:18
*** julim has joined #tripleo14:22
*** ftcjeff has quit IRC14:30
*** ftcjeff has joined #tripleo14:30
derekhIs anybody else having trouble with nova client in images? http://paste.openstack.org/show/61191/14:31
derekhlooks like the system version of six is being used instead of the venv one14:32
*** jprovazn has quit IRC14:33
*** spzala has joined #tripleo14:33
*** matty_dubs|gone is now known as matty_dubs14:48
openstackgerritRyan Brady proposed a change to openstack/tripleo-image-elements: Updates nova notty file  https://review.openstack.org/6659314:49
*** nosnos has quit IRC14:50
*** jrist has joined #tripleo14:51
*** e0ne_ has joined #tripleo14:59
*** slagle has quit IRC15:02
*** e0ne has quit IRC15:02
*** sdake has quit IRC15:03
*** coolsvap has joined #tripleo15:04
*** vkozhukalov has quit IRC15:04
openstackgerritMark McLoughlin proposed a change to openstack-infra/tripleo-ci: Fix some nitpicks in testenv-work element README  https://review.openstack.org/6659515:06
*** sdake has joined #tripleo15:15
openstackgerritRoman Podoliaka proposed a change to openstack/tripleo-incubator: Use wait condition when waiting for undercloud  https://review.openstack.org/6660015:18
openstackgerritRoman Podoliaka proposed a change to openstack/tripleo-incubator: Preserve USE_CACHE value in devtest_setup.sh  https://review.openstack.org/6660115:18
openstackgerritRoman Podoliaka proposed a change to openstack/tripleo-incubator: Cleanup docs  https://review.openstack.org/6660215:18
*** lucasagomes is now known as lucas-hungry15:27
*** rushiagr is now known as rushiagr_away15:28
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Move templates for nodes.overview  https://review.openstack.org/6660615:34
*** kpavel has quit IRC15:41
*** akrivoka has joined #tripleo15:44
openstackgerritRoman Podoliaka proposed a change to openstack/tripleo-incubator: Cleanup docs  https://review.openstack.org/6660215:44
*** rushiagr_away is now known as rushiagr15:46
rpodolyakaderekh: I can confirm six version issue (ubuntu saucy, image built 1 hour ago)15:48
derekhrpodolyaka: looks like nova client isn't using the python-six installed into the venv instead using the globally installed one15:49
derekhrpodolyaka: this used to be ok because os-refresh-config was installing globally15:49
derekhrpodolyaka: but yesterday I moved it to a venv, so python-six is no longer upgraded by the install of os-refresh-config (or may its os-collect-config)15:50
rpodolyakaderekh: hmm, but doesn't this mean that novaclient has six version specified incorrectly in its requirements.txt? or it's just installed without -U?15:52
rpodolyakaso it must be missing -U option, requirements.txt has six>=1.4.115:54
rpodolyakaas well as site-packages=True15:54
*** newell has joined #tripleo15:55
openstackgerritImre Farkas proposed a change to openstack/tripleo-image-elements: Add iptables rule for Ironic in Fedora  https://review.openstack.org/6661015:55
derekhrpodolyaka: well it's installing 1.4.1 into the venv15:56
derekhbut using the global one http://paste.openstack.org/show/61193/15:56
rpodolyakaoh, nice15:56
*** mrunge has joined #tripleo15:59
*** akuznetsov has joined #tripleo16:14
*** AaronGr_Zzz is now known as AaronGr16:16
*** jcoufal has quit IRC16:16
*** e0ne_ has quit IRC16:19
SpamapSderekh: network on cd-undercloud is definitely flaky16:20
*** e0ne has joined #tripleo16:20
SpamapSlsmola_: we are not going to use nested resources in HOT, because that is not composable enough.16:20
SpamapSlsmola_: we are going to use some of that, but we will also use the software config stuff which will let us combine multiple metadata sections in one server.16:21
SpamapSlsmola_: it will also free us of explicit user/accesskey/policy definition16:21
*** boris-42 has quit IRC16:21
SpamapSlsmola_: oh and waitcondition definition too :)16:21
*** lucas-hungry is now known as lucasagomes16:23
*** jistr has quit IRC16:23
derekhSpamapS: I was kinda hoping it was a problem where ever their hosted but if any of us have changed anything in there recently its probably more likely16:23
*** e0ne has quit IRC16:24
SpamapSderekh: that box has a history of flaky network16:24
derekhSpamapS: ahh yes, didn't something need to be restarted periodically16:26
SpamapSderekh: we actually have to rmmod/insmod the network driver.16:28
SpamapSderekh: but lately we haven't needed to so I was hoping maybe it was handled elsewhere. :-P16:28
*** bauzas has quit IRC16:36
derekhSpamapS: hmm, I can't ssh in anymore16:36
SpamapSderekh: I'll ssh to our ilo bastion host and see if we can resolve there16:38
*** mrunge has quit IRC16:38
derekhSpamapS: ok, thanks, no hurry (for me at least), I'll pick up on the ci stuff tomorrow16:39
* derekh just got ssh'd in for a moment but lost it again16:41
SpamapSderekh: ok I kicked the mellanox driver .. should be more reliable now16:42
SpamapSthough inevitably I'll break something else. :-P16:43
*** slagle has joined #tripleo16:44
derekhSpamapS: thanks, still feels a little like packets are doing a lap of the moon but I think its better16:44
*** viktors has left #tripleo16:45
SpamapSyeah it does16:46
SpamapSI think this may be something else16:46
*** rdopieralski has quit IRC16:47
derekhSpamapS: must be something else, terminal unresponsive again16:49
SpamapSno I did that.. I think something is bong on the bridge setup16:49
SpamapSok actually that was the local bridges needing to flush cache16:50
*** beekneemech is now known as bnemec16:50
*** blamar has quit IRC16:53
*** gothicmindfood has joined #tripleo16:54
*** blamar has joined #tripleo16:55
*** gothicmindfood has left #tripleo16:57
*** gothicmindfood_ has joined #tripleo17:01
*** vkozhukalov has joined #tripleo17:06
*** akuznetsov has quit IRC17:06
*** matty_dubs is now known as matty_dubs|lunch17:09
openstackgerritA change was merged to openstack/tripleo-image-elements: Updates cinder element  https://review.openstack.org/6269317:09
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Wait for tgtd socket to be available  https://review.openstack.org/6663917:11
SpamapShmmmm17:12
SpamapSlooking at https://review.openstack.org/62693 ...17:12
SpamapSI wonder if /var/run is correct..17:12
SpamapSI _think_ that means on every reboot the iptables rules will be duplicated.17:12
* SpamapS should think before writing those comments.. doh17:12
anteayalifeless mikal for when you are awake, please note there is another 3rd party testing meeting scheduled: http://lists.openstack.org/pipermail/openstack-dev/2014-January/024297.html17:16
anteayawould be great if you could attend and share your thoughts17:16
anteayasdague too ^^17:17
anteayaplus anyone else who wants to attend17:17
openstackgerritA change was merged to openstack/tripleo-image-elements: Updates nova notty file  https://review.openstack.org/6659317:17
*** akuznetsov has joined #tripleo17:18
sdagueanteaya: just remind me in the room, I'll join it :)17:18
NgSpamapS: hmm, that's an interesting point actually17:18
slagleSpamapS: hmm, good point. /var/run is tmpfs mounted17:18
slagleso it should be something under /mnt/state17:19
Ngyeah, I suspect a fedora-iptables element is needed, to collect all these .ok files17:19
Ngdoes fedora not have some kind of idempotent service-level firewall tool like ufw?17:20
SpamapSI think it would make sense to simplify that logic into a little tool17:21
Ngyeah, that also makes sense17:22
SpamapSNg: oh.. ufw.. why aren't we using that?17:22
SpamapSI sometimes forget it exists :)17:22
NgSpamapS: does that even on fedora?17:22
SpamapSNg: I see no reason why it wouldn't.17:22
SpamapSOther than politics.17:22
slaglefedora has firewalld, but not in the cloud images17:22
slagletoo big a footprint17:22
NgI don't see any immediate google juice for ufw being available on fedora17:24
openstackgerritA change was merged to openstack/tripleo-image-elements: Updates gearman iptables file  https://review.openstack.org/6657217:25
Ngdammit17:25
Ngtoo late to unapprove that one17:25
anteayasdague: #openstack-meeting-alt and thank you17:25
*** boris-42 has joined #tripleo17:26
slagleactually there is a iptables-services rpm for fedora that has this same save/restore logic17:27
slagleso we should probably just use that. have each element just open the ports it needs, then iptables-services that runs save at the very end17:28
*** max_lobur is now known as max_lobur_afk17:28
openstackgerritA change was merged to openstack/tripleo-image-elements: Update apache2 to use the os-apply-config dir.  https://review.openstack.org/6647017:31
derekhrlandy: ^ see iptables conversation above, related to the iptables script I mentioned earlier.17:37
*** digambar_ has joined #tripleo17:42
*** matty_dubs|lunch is now known as matty_dubs17:44
openstackgerritA change was merged to openstack/tripleo-heat-templates: Avoid echoing comment from makefile recipe  https://review.openstack.org/6657517:44
openstackgerritA change was merged to openstack/tripleo-heat-templates: Add all and clean targets to makefile  https://review.openstack.org/6657617:44
*** akuznetsov has quit IRC17:46
* SpamapS is always happy and horrified to see merge.py related things land17:48
NgI am never happy to review makefiles ;)17:49
*** akuznetsov has joined #tripleo17:49
Ngit's 2014!17:49
rlandyderekh: ok  - I was about to ask what happened to service firewalld but I see slagle answered that question.17:49
SpamapSNg: all debian packages are make files17:49
SpamapSSo...17:49
SpamapSsuch is life17:49
NgSpamapS: sure, but debian is the 1990s making the best of what the 1980s left them with. perl and make.17:50
*** akuznetsov has quit IRC17:50
digambar_Hi17:50
digambar_Getting error17:50
digambar_ronic node-create -d fake_ssh -i ssh_address=192.168.122.1 -i ssh_username=root -i ssh_password=openstack -i ssh_virt_type=virsh17:50
digambar_HTTPForbidden (HTTP 403)17:50
digambar_This is authentication error17:51
SpamapSNg: that _might_ end up on twitter if I can navigate the distraction maze to get there17:51
NgSpamapS: I will retoot it, if you do :)17:51
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Wait for tgtd socket to be available  https://review.openstack.org/6663917:53
*** jcoufal-m has joined #tripleo17:53
*** derekh has quit IRC17:54
openstackgerritJames Slagle proposed a change to openstack/tripleo-incubator: Update write-tripleorc  https://review.openstack.org/6665518:02
*** bauzas has joined #tripleo18:04
openstackgerritJay Dobies proposed a change to openstack/tuskar: Tuskar domain model updates for wireframes  https://review.openstack.org/6606218:05
openstackgerritA change was merged to openstack/tripleo-heat-templates: Add undercloud-vm-ironic-source yaml file  https://review.openstack.org/6611518:06
* Ng breaks for dinner18:08
*** jcoufal has joined #tripleo18:09
*** mrunge has joined #tripleo18:10
*** rushiagr has quit IRC18:15
*** spzala_ has joined #tripleo18:19
*** spzala has quit IRC18:21
*** spzala_ is now known as spzala18:21
*** bauzas has quit IRC18:21
*** akrivoka has quit IRC18:23
lifelessmorning18:25
*** martyntaylor has quit IRC18:28
*** mrunge has quit IRC18:29
lifelessSpamapS: FWIW I'm in complete agreement that haproxy shouldn't care whats installed locally18:32
SpamapSlifeless: Yeah I am worried that I'm just not understanding why the heat template isn't the place to configure this.18:34
lifelessanteaya: sdague: I would, but I have a medical thing on then, sorry.18:35
*** vipul is now known as vipul-away18:35
lifelessslagle: Ng: SpamapS: FWIW, something that asserts iptables is likely better than save-restore. IMNSHO.18:36
lifelessSpamapS: [haproxy] it was, in an earlier iteration.18:37
SpamapSlifeless: ufw basically works that way. Packages ship service definitions to make it easy to poke holes in the firewall.18:38
SpamapSand then ufw runs and asserts that those holes are there, and that the default rules are in place18:38
*** rushiagr has joined #tripleo18:40
lifelessright18:41
lifelessbetter than incremental editing18:41
SpamapS293 packets transmitted, 134 received, 54% packet loss, time 293170ms18:45
SpamapSsomething is not right with cd-undercloud18:45
*** ccrouch has quit IRC18:45
*** martyntaylor has joined #tripleo18:45
SpamapS10 packets transmitted, 10 received, 0% packet loss, time 9011ms18:45
mordredlifeless, SpamapS: speaking of iptables - any ideas on how to start supporting nftables from 3.13?18:45
SpamapSbastion host is fine18:45
SpamapSmordred: looks like just start replacign all the other ones with nftables calls.18:46
SpamapSreplacing even18:46
SpamapS" The nftables project provides a backward compatibility layer that allows you run iptables/ip6tables (using the same syntax) over the nftables infrastructure"18:46
*** ccrouch has joined #tripleo18:47
SpamapSmordred: so it can be incremental looksl ike18:47
mordredexcellent18:47
*** ccrouch has quit IRC18:48
lifelessmordred: I thought there was a compat layer18:50
lifelessmordred: so, step a) don't, until b) we don't have iptables on a supported platform18:50
mordredlifeless: just forward looking to see if there was anything needed doing18:51
*** jcoufal-m has quit IRC18:51
lifelessmordred: ah, so with kernel stuff you can rely on backwards compat from userspace :)18:52
lifelessmordred: however a bigger question is when we want to take advantage of it's new features18:52
mordredlifeless: yah18:53
*** vipul-away is now known as vipul18:53
*** rpodolyaka1 has joined #tripleo18:56
*** jprovazn has joined #tripleo18:57
*** e0ne has joined #tripleo18:57
*** gothicmindfood_ has left #tripleo18:58
lifelessmeeting time y'all18:59
*** jistr has joined #tripleo18:59
*** pblaho has joined #tripleo19:00
*** ccrouch has joined #tripleo19:05
anteayalifeless: okay thanks19:08
*** rushiagr has quit IRC19:10
*** spzala has quit IRC19:18
*** digambar_ has quit IRC19:21
*** bauzas has joined #tripleo19:23
*** vkozhukalov has quit IRC19:29
*** athomas has quit IRC19:34
*** athomas has joined #tripleo19:48
openstackgerritLucas Alvares Gomes proposed a change to openstack/diskimage-builder: Fix deploy-ironic init script  https://review.openstack.org/6667719:48
*** vipul is now known as vipul-away19:52
*** vipul-away is now known as vipul19:52
*** CaptTofu has joined #tripleo19:56
*** lucasagomes has quit IRC19:57
*** jcoufal has quit IRC20:00
SpamapSand after that we need to fix Heat to not delete "the last server added" when a scaling group is scaled down20:00
*** sdake has quit IRC20:01
SpamapSotherwise we'll have to push merge.py harder to generate templates and then manually delete things from said templates.20:01
*** pblaho has quit IRC20:01
*** sdake has joined #tripleo20:01
*** sdake has joined #tripleo20:01
rlandydprince: could I also get on that CI/testing call tomorrow?20:02
*** jcoufal has joined #tripleo20:02
dprincerlandy: sure.20:03
ccrouch(2:00:01 PM) lsmola: SpamapS, so we are keepig merge.py in I ?20:03
ccrouchis a good question20:03
SpamapSlifeless: anyway, my point is, there seems to be this "only SpamapS can do Heat" thing that is not going to scale well.20:03
*** e0ne has quit IRC20:03
SpamapSccrouch: I have no idea.20:03
shadowerSpamapS: I'm going to be more involved with Heat if that helps20:04
*** e0ne has joined #tripleo20:04
ccrouchSpamapS: fair enough. its not a goal i'm aware we've articulated20:04
SpamapSI'd love to get rid of it. It would make our dev more smooth to just write templates and not have to shove them through merge.py. But we have something that works so meh if it stays.20:04
SpamapS"bigger fish to fry"20:04
SpamapSI think I'm more worried about scaling out now20:05
SpamapSthe retry thing and updatable wait condition thing are important, but we need to switch to using instancegroup or resource group and that is going to cause issues.20:05
*** e0ne has quit IRC20:06
SpamapSthe other option is to just scale out by making a stack-per-compute.20:06
*** e0ne has joined #tripleo20:06
lifelessSpamapS: so, I think the spamaps-only-heat is a result of surfacing work20:06
lifelessSpamapS: till now most of our work hasn't been heat, and now - because we've finally pushed hard on update etc, it will be for a while20:06
lifelessSpamapS: so I think what you're doing - surfacing that there is heat work to do - is good20:07
lifelessSpamapS: but lets not lose focus at the same time; so keep the surfaced work fairly narrow, and collaborate on patches20:07
lifelessSpamapS: our current story, which AIUI we haven't delivered yet, is being able to do an upgrade, with downtime.20:07
lifelessSpamapS: we're close right? but not -there-20:08
*** vipul is now known as vipul-away20:08
*** notq has joined #tripleo20:08
lifelessdid we have a heat template for running up the broker?20:10
SpamapSlifeless: I think we're there but we're not waiting long enough because the wait conditions don't wait.20:10
lifelessSpamapS: so we're not there, but the fix is in heat :)20:10
lifelessSpamapS: so once thats done, I think the next thing is graceful deploys w/out downtime - so HA control and HA compute, live migration and rolling upgrades.20:11
SpamapSyeah, so I'll whip up a second attempt at that patch today20:11
lifelessthats a bunch of fodder and we may want to have a bit of divergence on what folk tackle there, because it's still fairly narrow.20:11
SpamapSyeah, HA means "n" things.. and we don't have "n" things possible right now, we have static servers defined20:11
*** CaptTofu has quit IRC20:12
SpamapSanyway, in all of that we're also going to need retrying of updates to work20:12
SpamapSbecause OpenStack is non-deterministic :)20:12
*** vipul-away is now known as vipul20:12
ftcjeffSo I'm trying to run devtest.sh, but I get this:20:17
ftcjeffdistutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pbr')20:17
lifelessSpamapS: right, but I guess I'm saying lets be guided by what breaks first20:17
ftcjeffAnyone know what mess I've created for myself?  I've seen this before doing horizon code, but I can't remember how I got out of it20:18
lifelessSpamapS: e.g. lets get 3 control plane and 2 compute20:18
lifelessSpamapS: HA != any-N20:18
lifelessftcjeff: thats a cascade errror, the actual failure will be higher up20:18
lifelessftcjeff: e.g. a SSL error or some such20:18
ftcjeffDownload error on https://pypi.python.org/simple/pbr/: [Errno 1] _ssl.c:509: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol -- Some packages may not be found!20:19
ftcjeff?20:19
lifelessyup20:19
lifelesscaused by http proxies with older pip IIRC20:19
ftcjeffokay, I'll try a different proxy20:19
ftcjeffthanks :-)20:19
*** rlandy has quit IRC20:20
*** gothicmindfood has joined #tripleo20:21
*** gothicmindfood has left #tripleo20:21
*** vipul is now known as vipul-away20:27
*** jtomasek has quit IRC20:39
*** jtomasek has joined #tripleo20:41
*** jistr has quit IRC20:42
lifelesswe don't have /var/log/upstart persisted yet either20:43
lifelessmaybe we shouldn't; just pull straight into logstash.20:43
lifelessthought for another day20:44
SpamapSlifeless: Ok well what has broken already meany times is update fails for abitrary reason and we just have to kick it to retry.20:45
lifelessSpamapS: so maybe that should be next20:45
SpamapSit is 33% done20:46
SpamapSCREATE FAILED works fine.. we delete/create20:46
lifelessSpamapS: either in MVP or the decreasefragility20:46
SpamapSDELETE FAILED just needs to retry DELETE20:46
SpamapSand UPDATE FAILED needs to retry UPDATE20:46
SpamapSlifeless: I don't think this is fragility. It is part of the story to keep it running "somehow"20:47
SpamapSfragility is it breaks on every update20:47
lifelessSpamapS: I think this is splitting hairs; we both want the work done20:47
SpamapSRight. Anyway, I need to run some errands. bbiab20:48
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Fix broken iptables command in cinder  https://review.openstack.org/6668420:50
openstackgerritJay Dobies proposed a change to openstack/tuskar: Tuskar domain model updates for wireframes  https://review.openstack.org/6606220:52
rbradydprince: ping20:57
*** lblanchard has quit IRC20:58
dprincerbrady: hi20:59
dprincerbrady: gotta run in a minute but what is up?21:00
rbradyre: your cinder change.  I just posted the comments21:01
lifelessyay ova secgroup-list21:01
lifelessERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-3d2dd2a7-8b23-416d-bd96-975c3b586919)21:01
rbradydprince: I'm not sure which way is the "only one obvious way" to do this21:02
*** lblanchard has joined #tripleo21:03
*** lblanchard has quit IRC21:03
*** AaronGr is now known as AaronGr_afk21:03
rbradydprince: or if it matters after Spamaps review21:03
lifelesshmm, working now21:04
dprincerbrady: I would rather land this quick fix and then move towards using a common iptables element for this.21:04
lifelessand holy cow, 3 seconds to live. Thats a cloud.21:04
rbradydprince: rgr21:04
dprincerbrady: agree that we should create /var/run/cinder in install.d... but I'm not sure that will be the eventual location we use for iptables stuff so lets get this in and move it to a common place21:05
rbradydprince: updated review21:05
dprincerbrady: feel free to overwrite my patch if you have a better solution though, just trying to keep things working quickly21:05
dprincerbrady: gotta go. thanks for looking into this21:06
*** dprince has quit IRC21:06
*** julim has quit IRC21:06
*** rpodolyaka1 has quit IRC21:09
ftcjeffI got a little further this time:21:22
ftcjeffWaiting for keystone to initialise...21:22
ftcjeff<attribute 'message' of 'exceptions.BaseException' objects> (HTTP Unable to establish connection to http://192.0.2.1:35357/v2.0/tenants)21:22
ftcjeffand then it finished with:21:23
ftcjeff/home/jwalls/tripleo/tripleo-incubator/scripts/devtest_seed.sh: line 95: os-apply-config: command not found21:23
*** jprovazn has quit IRC21:23
*** e0ne has quit IRC21:24
openstackgerritA change was merged to openstack/tuskar-ui: Move templates for nodes.overview  https://review.openstack.org/6660621:26
*** vipul-away is now known as vipul21:26
*** newell_ has joined #tripleo21:33
*** newell has quit IRC21:36
*** ifarkas has quit IRC21:45
*** markmc has quit IRC21:49
*** mrunge has joined #tripleo21:50
*** AaronGr_afk is now known as AaronGr21:53
*** jdob has quit IRC21:54
*** athomas has quit IRC21:59
*** lblanchard has joined #tripleo21:59
*** jtomasek has quit IRC22:01
*** jtomasek has joined #tripleo22:01
*** modi_vnc has joined #tripleo22:17
*** julim has joined #tripleo22:29
*** e0ne has joined #tripleo22:35
openstackgerritJames Slagle proposed a change to openstack/diskimage-builder: Symlink correct element install type  https://review.openstack.org/6672422:38
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Install nova based on source installation type  https://review.openstack.org/6580622:38
*** e0ne has quit IRC22:39
SpamapSftcjeff: os-apply-config is installed in a virtualenv and then symlinked in tripleo-incubator/scripts ... so you should have it on your path if scripts is on your path22:40
SpamapSftcjeff: devtest_variables.sh should handle htat22:41
SpamapSthat22:41
ftcjeffspamaps: I don't see os-apply-config in tripleo-incubator/scripts :(  I'll keep looking for what might have gone wrong22:43
*** jcoufal has quit IRC22:43
*** mrunge has quit IRC22:44
*** CaptTofu has joined #tripleo22:49
SpamapSftcjeff: the symlink is setup in setup-clienttools22:51
SpamapSftcjeff: which is called in install-dependencies and refresh-env22:51
SpamapSftcjeff: so devtest_setup.sh should have handled that (it is run from devtest.sh)22:52
*** julim has quit IRC22:52
ftcjeffspamaps: so it's linked from tripleo-incubator/openstack-tools/bin; but it doesn't exist there either22:53
*** lblanchard has quit IRC22:55
SpamapSftcjeff: I wonder if we're not detecting a failure to install that22:56
ftcjeffspamaps: unfortunately, my scroll buffer doesn't go back very far :(22:57
SpamapSftcjeff: ah-hah!22:57
SpamapSbug22:57
SpamapShttp://paste.openstack.org/show/6122022:57
SpamapSftcjeff: I don't have os-apply-config either..22:57
SpamapSftcjeff: the dir is never cleared out22:58
SpamapSftcjeff: so we skip if we've already installed before.. but new tools have been added22:58
SpamapSftcjeff: I know you were looking for each bugs but this is a one liner. :)22:59
ftcjeffspamaps: cool22:59
SpamapSftcjeff: you can work around it by rm -r'ing openstack-tools23:00
ftcjeffspamaps: want me to try to fix that?  :-)23:02
openstackgerritClint "SpamapS" Byrum proposed a change to openstack/tripleo-incubator: Refresh openstack-tools virtualenv every time  https://review.openstack.org/6672723:02
SpamapSftcjeff: too late :)23:02
ftcjeffdarn!23:02
*** CaptTofu has quit IRC23:04
openstackgerritA change was merged to openstack/tripleo-image-elements: Fix broken iptables command in cinder  https://review.openstack.org/6668423:04
*** ccrouch1 has joined #tripleo23:09
*** CaptTofu has joined #tripleo23:10
*** ccrouch has quit IRC23:10
*** jtomasek has quit IRC23:10
*** modi_vnc has quit IRC23:12
*** CaptTofu has quit IRC23:18
*** CaptTofu has joined #tripleo23:19
*** CaptTofu has quit IRC23:19
*** ftcjeff has quit IRC23:25
*** CaptTofu has joined #tripleo23:36
*** ccrouch1 has quit IRC23:45
*** martyntaylor has quit IRC23:46
StevenKlifeless: Oh, we should bump the 64bit docs in _variables, now that I remember. Double everything, you think?23:49
lifelessStevenK: I'm not sure; the difference really is the python footprint23:50
StevenKRight, it's about 30% bigger or so23:50
lifelessStevenK: data would be a good thing23:50
*** spzala has joined #tripleo23:53

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!