*** mahito has joined #openstack-ansible | 00:04 | |
daneyon_ | I added swift to my v10 deployment, but it doesn;t appear that Keystone is adding the required swift user/service/endpoint, etc.. I used this playbook: https://github.com/stackforge/os-ansible-deployment/blob/juno/etc/rpc_deploy/conf.d/swift.yml | 00:12 |
---|---|---|
daneyon_ | ^ In looking at the keystone-add-all-services.yml, I don't see Swift info. | 00:13 |
*** vmtrooper has joined #openstack-ansible | 00:28 | |
*** vmtrooper has quit IRC | 00:33 | |
*** stevemar has joined #openstack-ansible | 00:43 | |
*** jmccrory has joined #openstack-ansible | 00:50 | |
openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Enable fatal_deprecations configuration per OpenStack service https://review.openstack.org/164911 | 00:51 |
openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Enable fatal_deprecations configuration per OpenStack service https://review.openstack.org/164911 | 00:53 |
*** jmccrory has quit IRC | 00:55 | |
cloudnull | daneyon_ in v10 adding swift requires a swift config and rerun all of the swift plays. the keystone service entry is added here: https://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/playbooks/openstack/swift-proxy.yml#L31 | 00:57 |
cloudnull | make sure you have something similar to https://github.com/stackforge/os-ansible-deployment/blob/juno/etc/rpc_deploy/conf.d/swift.yml.example | 00:57 |
cloudnull | the absolute minimum config required can be seen here: https://github.com/stackforge/os-ansible-deployment/blob/juno/etc/rpc_deploy/conf.d/swift.yml | 00:58 |
*** britthouser has quit IRC | 01:04 | |
*** britthouser has joined #openstack-ansible | 01:07 | |
openstackgerrit | Merged stackforge/os-ansible-deployment: Simplify and improve bootstrap/gate/run scripts https://review.openstack.org/163837 | 01:49 |
*** MentalRay_ has joined #openstack-ansible | 02:00 | |
*** vmtrooper has joined #openstack-ansible | 02:17 | |
Sam-I-Am | MentalRay_: hey | 02:17 |
*** galstrom_zzz is now known as galstrom | 02:18 | |
MentalRay_ | hey :) | 02:18 |
Sam-I-Am | MentalRay_: didnt know you hung in here | 02:18 |
MentalRay_ | first visit :p | 02:19 |
Sam-I-Am | MentalRay_: welcome! | 02:19 |
MentalRay_ | thanks thanks | 02:19 |
MentalRay_ | still working? | 02:19 |
Sam-I-Am | not... really. | 02:20 |
Sam-I-Am | maybe helping someone in #openstack | 02:20 |
Sam-I-Am | probably need to catch up on some networking docs work | 02:20 |
Sam-I-Am | spent the day in training, so i'm really behind on stuff | 02:20 |
MentalRay_ | hehehe ok | 02:22 |
*** vmtrooper has quit IRC | 02:22 | |
*** sigmavirus24_awa is now known as sigmavirus24 | 02:24 | |
*** sdake__ has quit IRC | 02:32 | |
openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs https://review.openstack.org/164933 | 02:56 |
cloudnull | Please review that as soon as possible. this is what commit to get our specs repo going. ^ | 02:57 |
sigmavirus24 | cloudnull: why do we have testr in there? | 03:03 |
sigmavirus24 | what tests will be run? | 03:03 |
cloudnull | it runs a build test against the specs | 03:04 |
cloudnull | it makes sure all of the sections are present, and that the spec is line wrapped at 79 | 03:04 |
cloudnull | and that the spec has been created in a directory that has an index | 03:04 |
sigmavirus24 | hm | 03:06 |
sigmavirus24 | yeah I just happened upon the actual test files | 03:06 |
sigmavirus24 | interesting | 03:06 |
cloudnull | the setup was copied from other spec repos. i simply tweaked it for our use case. | 03:06 |
sigmavirus24 | :thumbsup: | 03:07 |
openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Enable fatal_deprecations configuration per OpenStack service https://review.openstack.org/164911 | 03:08 |
sigmavirus24 | stevelle: so we're just going to break the gate today? That's what we're going to do? | 03:10 |
stevelle | confirmed. | 03:10 |
stevelle | because apparently I was hungry and distracted | 03:10 |
sigmavirus24 | stevelle: http://weknowmemes.com/wp-content/uploads/2012/06/red-do-you-think-im-smart.jpg | 03:10 |
cloudnull | hahahah | 03:11 |
stevelle | At least I found why I failed the gate. :thumbsup: to the improved ansible logging | 03:11 |
cloudnull | http://docs-draft.openstack.org/33/164933/1/check/gate-os-ansible-deployment-specs-docs/7875a01//doc/build/html/specs/template.html | 03:13 |
* sigmavirus24 points finger at other guy | 03:13 | |
cloudnull | draft doc, all pretty like | 03:13 |
sigmavirus24 | thank you cloudnull | 03:13 |
sigmavirus24 | I was going to pull that down | 03:13 |
openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs https://review.openstack.org/164933 | 03:14 |
cloudnull | i just updated the template to show an example. | 03:14 |
sigmavirus24 | damnit | 03:14 |
* sigmavirus24 shakes fist | 03:15 | |
*** galstrom is now known as galstrom_zzz | 03:17 | |
*** markvoelker has quit IRC | 03:29 | |
*** vmtrooper has joined #openstack-ansible | 03:49 | |
*** britthouser has quit IRC | 04:02 | |
*** stevemar has quit IRC | 04:22 | |
*** stevemar has joined #openstack-ansible | 04:23 | |
*** sigmavirus24 is now known as sigmavirus24_awa | 04:24 | |
openstackgerrit | Merged stackforge/os-ansible-deployment: Revise gate script library to report correctly https://review.openstack.org/163914 | 04:32 |
*** vmtrooper has quit IRC | 04:37 | |
*** MentalRay_ has quit IRC | 04:47 | |
*** ishant|2 has joined #openstack-ansible | 05:53 | |
*** ishant|2 has quit IRC | 05:54 | |
*** ishant has joined #openstack-ansible | 05:54 | |
openstackgerrit | Merged stackforge/os-ansible-deployment: Remove 'holland' package from the wheel repository https://review.openstack.org/164822 | 06:11 |
*** vmtrooper has joined #openstack-ansible | 06:26 | |
*** vmtrooper has quit IRC | 06:31 | |
*** sdake has joined #openstack-ansible | 06:44 | |
*** stevemar has quit IRC | 06:44 | |
*** sdake has quit IRC | 06:48 | |
*** markvoelker has joined #openstack-ansible | 07:04 | |
*** markvoelker has quit IRC | 07:08 | |
*** MentalRay_ has joined #openstack-ansible | 07:54 | |
*** MentalRay_ has quit IRC | 07:54 | |
*** markvoelker has joined #openstack-ansible | 08:05 | |
*** markvoelker has quit IRC | 08:10 | |
*** vmtrooper has joined #openstack-ansible | 08:15 | |
*** vmtrooper has quit IRC | 08:21 | |
*** mahito has quit IRC | 08:35 | |
*** markvoelker has joined #openstack-ansible | 09:06 | |
*** markvoelker has quit IRC | 09:11 | |
*** vmtrooper has joined #openstack-ansible | 10:04 | |
*** markvoelker has joined #openstack-ansible | 10:07 | |
*** vmtrooper has quit IRC | 10:09 | |
*** markvoelker has quit IRC | 10:12 | |
*** markvoelker has joined #openstack-ansible | 11:08 | |
*** markvoelker has quit IRC | 11:14 | |
*** britthouser has joined #openstack-ansible | 11:24 | |
*** markvoelker has joined #openstack-ansible | 11:38 | |
*** galstrom_zzz is now known as galstrom | 11:51 | |
*** vmtrooper has joined #openstack-ansible | 11:53 | |
*** vmtrooper has quit IRC | 11:58 | |
*** KLevenstein has joined #openstack-ansible | 12:03 | |
*** stevemar has joined #openstack-ansible | 12:05 | |
*** jaypipes has joined #openstack-ansible | 12:20 | |
*** galstrom is now known as galstrom_zzz | 13:07 | |
*** sdake has joined #openstack-ansible | 13:25 | |
*** sdake__ has joined #openstack-ansible | 13:27 | |
*** sdake has quit IRC | 13:31 | |
openstackgerrit | Merged stackforge/os-ansible-deployment: Ensure return code passes through output trimming https://review.openstack.org/164480 | 13:41 |
*** vmtrooper has joined #openstack-ansible | 13:42 | |
*** vmtrooper has quit IRC | 13:47 | |
*** sdake has joined #openstack-ansible | 13:48 | |
*** sdake__ has quit IRC | 13:51 | |
*** Mudpuppy has joined #openstack-ansible | 14:00 | |
*** Mudpuppy has quit IRC | 14:01 | |
*** Mudpuppy has joined #openstack-ansible | 14:02 | |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Adjust method for trimming ansible output debug information https://review.openstack.org/163809 | 14:05 |
*** sigmavirus24_awa is now known as sigmavirus24 | 14:07 | |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Ensure return code passes through output trimming https://review.openstack.org/165085 | 14:09 |
*** ishant has quit IRC | 14:15 | |
*** galstrom_zzz is now known as galstrom | 14:56 | |
*** jaypipes has quit IRC | 15:06 | |
*** erikmwilson is now known as Guest78757 | 15:06 | |
*** erikmwilson has joined #openstack-ansible | 15:06 | |
*** erikmwilson_ has joined #openstack-ansible | 15:06 | |
palendae | odyssey4me, cloudnull: Looks like the checksum changes didn't take care of it: http://logs.openstack.org/93/161893/1/check/os-ansible-deployment-dsvm-check-commit/4241128/console.html#_2015-03-17_15_05_18_959 | 15:07 |
*** erikmwilson has quit IRC | 15:08 | |
*** erikmwilson has joined #openstack-ansible | 15:08 | |
palendae | Apsu as well ^ | 15:09 |
palendae | That's the latest check for https://review.openstack.org/#/c/161893/ | 15:09 |
Apsu | looking | 15:10 |
palendae | Apsu: I was trying to add the checksum rules to the utility container to see if the intermittent failures went away | 15:10 |
palendae | Evidently it doesn't | 15:11 |
palendae | I would hope it's not a resource thing | 15:11 |
odyssey4me | palendae Apsu yeah, it could be a resource thing - but it could also be something along the lines of many other things - the db not committing in time (the AIO has a 3 node DB cluster), neutron spinning on something (as it does), an IP conflict of some sort? | 15:13 |
palendae | Yeah.. | 15:13 |
odyssey4me | it may be worth digging into the logs we now collect to see if there're any other bits that show what's going on | 15:13 |
palendae | Lots of corner cases to chase down. I think it's going to take doing a checkout on an AIO and adding --debug to the runner in addition to log spelunking | 15:14 |
Apsu | Looks that way | 15:14 |
odyssey4me | the trouble is the intermittency | 15:14 |
palendae | Yep | 15:14 |
palendae | Exactly | 15:14 |
palendae | Which was insanely hard to catch earlier, too, but that was because the --serial change hadn't landed | 15:15 |
odyssey4me | an option is also to just modify your existing patch into a WIP test patch, and add --debug to the TEMPEST_OPTS/TESTR_OPTS (whichever is appropriate) | 15:15 |
palendae | I think it is WIP | 15:15 |
palendae | Or do you mean a patch against testing | 15:15 |
odyssey4me | but yes, if you can get a fail on a cloudserver AIO you'll have a better tool to diagnose properly - using openstack-infra to figure this out is harder | 15:16 |
palendae | Yeah | 15:16 |
d34dh0r53 | palendae: IIRC the gate scripts do an iptables -F which may be wiping out your rules | 15:16 |
hughsaunders | daneyon_: cinder_service_port is defaulted to 8776 in group_vars/cinder_all, have you overridden that anywhere? | 15:17 |
palendae | odyssey4me: The --debug option for tester would be to drop a pdb in to the code and look in the middle of the test | 15:17 |
d34dh0r53 | not 100% sure of the ordering there | 15:17 |
palendae | d34dh0r53: My current patch is after that | 15:17 |
d34dh0r53 | ok | 15:17 |
d34dh0r53 | just making sure of the obvious | 15:17 |
odyssey4me | palendae ah, cloud server test then :) | 15:17 |
palendae | odyssey4me: Yep | 15:17 |
palendae | Yeah, my intention was to inspect the environment the scenario sets up | 15:18 |
odyssey4me | palendae Apsu rackertom if you guys could try to urgently figure out the root cause and find a solution for consistant results it'd be appreciated... the network tests are a serious problem in our gating right now | 15:21 |
Apsu | odyssey4me: Aye | 15:22 |
palendae | I'll help where I can, split my brain between class and this | 15:23 |
Apsu | Yeah, I'm going to setup a box | 15:23 |
cloudnull | can we get some core reviewers on: https://review.openstack.org/#/c/164714/ | 15:24 |
cloudnull | and this one too: https://review.openstack.org/#/c/164933/ | 15:28 |
Sam-I-Am | there are network problems? | 15:28 |
palendae | Sam-I-Am: Yeah, some tempest network tests are getting timeouts | 15:28 |
cloudnull | randomly . | 15:28 |
Sam-I-Am | the best kind | 15:29 |
cloudnull | it cycles between, no ping, server fault, and success. | 15:29 |
palendae | Oh, server faults still? | 15:29 |
palendae | Crap | 15:29 |
rackertom | Cool! Do we have any of the setup logs anywhere | 15:29 |
palendae | rackertom: https://review.openstack.org/#/c/161893/, logs for latest failure at http://logs.openstack.org/93/161893/1/check/os-ansible-deployment-dsvm-check-commit/4241128/ | 15:30 |
Sam-I-Am | wonder if its slight timing issues | 15:30 |
*** Bjoern__ has joined #openstack-ansible | 15:30 | |
palendae | Sam-I-Am: A good guess. This is all on AIOs | 15:30 |
palendae | But haven't proven it yet | 15:30 |
*** Bjoern__ is now known as BjoernT | 15:31 | |
palendae | We certainly had timing issues with it prior to getting all the tests serialized | 15:31 |
*** vmtrooper has joined #openstack-ansible | 15:31 | |
cloudnull | palendae: http://logs.openstack.org/09/163809/7/check/os-ansible-deployment-dsvm-check-commit/11000bf/console.html#_2015-03-17_15_12_44_710 | 15:31 |
Sam-I-Am | 172.29.248.9 is a fip? | 15:31 |
cloudnull | Sam-I-Am yes, that should be a float from a flat net | 15:31 |
palendae | cloudnull: Oof - that's befor ethe test even | 15:32 |
Sam-I-Am | did anyone try adding 1 or 2 sheps here? | 15:32 |
Apsu | Sam-I-Am: It's tempest that's failing in its looping tests. So no we haven't modified tempest yet | 15:32 |
Sam-I-Am | havent looked at the code, but sometimes i've seen fips take longer to actually get assigned | 15:32 |
Apsu | I'm digging into it | 15:32 |
*** vmtrooper has quit IRC | 15:35 | |
cloudnull | palendae and it doesnt look like a resource issue either: http://logs.openstack.org/09/163809/7/check/os-ansible-deployment-dsvm-check-commit/11000bf/console.html#_2015-03-17_15_12_44_718 | 15:36 |
Apsu | Yeah, didn't appear to be when we were running it manually and it was still failing | 15:37 |
Apsu | So I'm going to try to debug the tests | 15:37 |
Apsu | Or recreate them manually | 15:37 |
Apsu | So I can investigate | 15:37 |
daneyon_ | hughsaunders: I have not overridden the cinder port | 15:40 |
rackertom | Am I reading that right? It's trying to get .9 running, fails the test, deletes it, and tries again? | 15:40 |
hughsaunders | daneyon_: could you put -debug: var=internal_vip_address -debug: var=cinder_service_port before the cinder api content check, and check that the ip and port match your service catalog? | 15:42 |
daneyon_ | Will Swift work using the default swift1/2/3.img instead of actual drives? I am getting the following error: proxy-server: ERROR Insufficient Storage 192.168.222.42:6002/swift3.img (txn: tx18697e8e53ff45b1a9370-005507acdb) (client_ip: 192.168.223.195) | 15:43 |
Sam-I-Am | daneyon_: is this a test env? | 15:44 |
daneyon_ | Sam-I-Am: yes | 15:44 |
palendae | rackertom: Correct - Neutron is provisioning all of this stuff, doing the test, then tearing it all down for the next test | 15:46 |
palendae | So it gets torn down on success or failure | 15:46 |
rackertom | So if it's a timing problem then it's only made worse by bringing down what was just set up. | 15:46 |
palendae | For our debugging purposes, yeah. | 15:47 |
Sam-I-Am | daneyon_: it probably works enough for tempest | 15:47 |
Sam-I-Am | so, whatever test it runs | 15:47 |
Sam-I-Am | i'd imagine those loopback mounts are small | 15:47 |
palendae | When we had tempest tests running in parallel, I think tests were finishing at different times and tearing stuff down in the middle of other tests | 15:48 |
palendae | Sorry, Neutron's not provisioning this...Tempest is | 15:48 |
palendae | Tempest is exercising openstack | 15:48 |
daneyon_ | Sam-I-Am: Surprising it would pass Tempest. | 15:50 |
Sam-I-Am | daneyon_: what were you trying to do? | 15:50 |
daneyon_ | Sam-I-Am: I was just trying to do a swift stat... show that Swift works. It would be nice to store 1 glance image. I'm currently hacking around it by copying my 1 image across my 3 controller nodes file store backend... just a cirros image | 15:51 |
cloudnull | andymccr ^ ping | 15:52 |
Sam-I-Am | yeah, swift stat should work | 15:52 |
andymccr | reading up | 15:53 |
Sam-I-Am | i cant find where those loopbacks are created, so i dont know their size | 15:53 |
andymccr | daneyon_: if you want to test the swift plays using .img files it should work fine as long as they are mounted in the correct location | 15:54 |
andymccr | the swift plays dont actually do any of the mounting/formatting etc | 15:54 |
daneyon_ | ya, I don;t see any info re: a loopback in fstab/mtab | 15:56 |
cloudnull | bug triage meeting in this channel in a couple of minutes: cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, mancdaz, dolphm, _shaps_, BjoernT, claco | 15:57 |
cloudnull | daneyon_ our AIO scripts do all that for us when were testing. if this is a deployment on gear you'd have to do that outside of ansible. | 15:58 |
cloudnull | and then rerun the swift plays. | 15:58 |
daneyon_ | cloudnull: OK | 15:59 |
daneyon_ | From what I could tell, it didn't seem that the loopback gets setup, but I wanted to double check with the group | 15:59 |
b3rnard0 | hello | 16:00 |
cloudnull | for sure. | 16:00 |
rromans | \o | 16:01 |
b3rnard0 | bug triage notes are going to be here: https://etherpad.openstack.org/p/openstack_ansible_bug_triage.2015-03-17-16.00 | 16:01 |
andymccr | daneyon_: we probably need better docs in the repo to explain the swift side better | 16:01 |
cloudnull | first bug from launchpad: https://bugs.launchpad.net/openstack-ansible/+bug/1430872 | 16:02 |
openstack | Launchpad bug 1430872 in openstack-ansible "Add token dhcp_domain to dhcp_agent.ini template" [Undecided,New] | 16:02 |
cloudnull | this is an issue that we can address in master. | 16:03 |
cloudnull | do we want to address this in Juno? | 16:03 |
BjoernT | yes juno | 16:03 |
cloudnull | IMO Icehouse is no. | 16:03 |
d34dh0r53 | 0/ | 16:03 |
BjoernT | we have active deployments | 16:03 |
BjoernT | no icehouse | 16:03 |
odyssey4me | seems reasonable and fairly simple to do | 16:04 |
cloudnull | yup its a simple variable add to the dhcp.ini | 16:04 |
cloudnull | who wants it | 16:04 |
odyssey4me | I'm happy to do it. | 16:05 |
cloudnull | ok. | 16:05 |
odyssey4me | I think we're missing a lot of people in this triage today. | 16:05 |
palendae | openstack: We are | 16:05 |
palendae | Er | 16:05 |
palendae | Yeah, a bunch of people here not paying attention | 16:05 |
palendae | I'd be willing to do this too, but it'll have to wait from me | 16:05 |
cloudnull | ha. | 16:05 |
cloudnull | miguelgrinberg, alextricity : https://bugs.launchpad.net/openstack-ansible/+bug/1432742 <- what say you ? | 16:06 |
openstack | Launchpad bug 1432742 in openstack-ansible "SoftwareDeployment Heat resource can not poll internal heat API" [Undecided,New] - Assigned to Miguel Alejandro Cantu (miguel-cantu) | 16:06 |
cloudnull | to change the vip address or not to change? this is the question. | 16:06 |
odyssey4me | hmm, there's already a patch for this I think: https://review.openstack.org/164785 | 16:06 |
Sam-I-Am | palendae: neutron consumes 100% brain cpu | 16:07 |
miguelgrinberg | I think alextricity is off this week, not sure if he'll be online to comment | 16:07 |
cloudnull | i saw him yesterday . | 16:07 |
d34dh0r53 | he's out the rest of the week | 16:07 |
cloudnull | ok. | 16:07 |
miguelgrinberg | cloudnull: I'd like to understand his setup, I never needed to change the metadata server | 16:07 |
cloudnull | ok. we should also note that this is now a default that can be changed per the user configuration as needed. so its no longer hard coded. | 16:08 |
cloudnull | maybe we leave it as is, and note that he can set heat_metadata_server_url: "{{ heat_service_proto }}://{{ external_lb_vip_address }}:{{ heat_cfn_serv | 16:09 |
cloudnull | ice_port }}" in his user_variables.yml | 16:09 |
cloudnull | thoughts? | 16:09 |
odyssey4me | cloudnull that's for master - right? | 16:09 |
cloudnull | yup | 16:10 |
odyssey4me | at this point in time it would seem that we don't have enough information to work from | 16:10 |
cloudnull | so then im going to mark the bug as incomplete. | 16:10 |
odyssey4me | so add a comment about that for master | 16:10 |
odyssey4me | but miguelgrinberg's questions need answering | 16:10 |
miguelgrinberg | cloudnull: incomplete is fine, we can resume the discussion then he's back | 16:11 |
cloudnull | yesir . | 16:11 |
cloudnull | done. | 16:13 |
cloudnull | https://bugs.launchpad.net/openstack-ansible/+bug/1432670 | 16:13 |
openstack | Launchpad bug 1432670 in openstack-ansible "ldappool required for keystone AD backend" [Undecided,New] | 16:13 |
cloudnull | that issue is inprogress hughsaunders is that correct? | 16:13 |
odyssey4me | yup: https://review.openstack.org/164715 | 16:13 |
hughsaunders | cloudnull: yeah | 16:13 |
cloudnull | are we backporting to Juno ? | 16:14 |
hughsaunders | I think that would be sensible | 16:14 |
hughsaunders | tag added | 16:14 |
BjoernT | I like Juno, because we have an active deployment on juno using this | 16:15 |
cloudnull | done. | 16:15 |
cloudnull | https://bugs.launchpad.net/openstack-ansible/+bug/1432438 | 16:15 |
openstack | Launchpad bug 1432438 in openstack-ansible "Update README notes to reflect the upstream ansible-lxc module merge" [Low,New] | 16:15 |
palendae | cloudnull: That could probably be pushed way out, maybe even wishlisted | 16:15 |
palendae | For when it actually lands in an ansible release | 16:15 |
cloudnull | palendae do you think you can bang out that readme update? | 16:15 |
cloudnull | ok. | 16:16 |
daneyon_ | andymccr: Might not be a bad idea to do something like this: https://github.com/stackforge/puppet-swift/blob/master/manifests/storage/loopback.pp | 16:16 |
palendae | Sure; I can say it's merged but not released, too | 16:16 |
palendae | Either way | 16:16 |
cloudnull | up to you palendae . | 16:16 |
palendae | I could get that out pretty quickly today though | 16:16 |
palendae | Yeah, I'll do it | 16:16 |
cloudnull | done. | 16:17 |
daneyon_ | andymccr: The puppet-swift loopback setup works well for test environments | 16:17 |
cloudnull | next: https://bugs.launchpad.net/openstack-ansible/+bug/1432873 | 16:17 |
openstack | Launchpad bug 1432873 in openstack-ansible "Add FDB bridge entry fails if old entry not removed" [Undecided,New] | 16:17 |
hughsaunders | daneyon_: the AIO scripts do a similar thing, good for testing: https://github.com/stackforge/os-ansible-deployment/blob/master/scripts/bootstrap-aio.sh#L184-L199 | 16:17 |
BjoernT | just added the reference to openstack-ansible | 16:18 |
odyssey4me | cloudnull that looks like an upstream bug - not sure if we can do much about that within os-ansible-deployment? | 16:18 |
daneyon_ | hughsaunders: thx | 16:18 |
BjoernT | This issue is new and caused by neutron l2pop plugin | 16:18 |
Apsu | It is an upstream bug. | 16:19 |
Apsu | They're not handling the exit code appropriately to make the add call idempotent, or clearing old ones first. | 16:19 |
BjoernT | odyssey4me: Yes you are right. We did this on other bugs to have a reference. I can remove it if needed | 16:19 |
odyssey4me | for this particular kind of bug, unless there is a workaround, I really don't think os-ansible-deployment can do anything - this should be handled in upstream and hopefully will make it into Kilo/Liberty | 16:20 |
*** Mudpuppy has quit IRC | 16:21 | |
Apsu | yep | 16:21 |
odyssey4me | I doubt this would be eligible for a stable backport | 16:21 |
*** Mudpuppy has joined #openstack-ansible | 16:21 | |
*** Mudpuppy has quit IRC | 16:21 | |
odyssey4me | So I'd vote this as invalid for openstack-ansible. | 16:21 |
cloudnull | odyssey4me i would suggest that we look into running updated 2014.2.2 with the stable juno release for our bits, it may not be "fixed" but there was some work that was backported for juno that pertained to ml2 + l2 pop. | 16:22 |
cloudnull | our juno is running 2014.2.1 | 16:22 |
odyssey4me | cloudnull I would support that, but that's a minor release update and not specifically to solve this patch | 16:22 |
cloudnull | for sure. we'd need to do to 10.1.0 | 16:23 |
cloudnull | sorry 10.2.x | 16:23 |
odyssey4me | it may improve the behaviour though, so I'd say that perhaps it could be a 'related' bug | 16:23 |
cloudnull | that sounds sensible to me. other thoughts? | 16:24 |
d34dh0r53 | +1 | 16:25 |
*** jaypipes has joined #openstack-ansible | 16:27 | |
cloudnull | done. | 16:27 |
cloudnull | https://bugs.launchpad.net/openstack-ansible/+bug/1432269 | 16:28 |
openstack | Launchpad bug 1432269 in openstack-ansible "Adding a new Cinder storage node fails api service check" [Undecided,New] | 16:28 |
cloudnull | this was an issue that daneyon_ reported. | 16:28 |
cloudnull | hughsaunders have you had any luck reproducing this issue? | 16:28 |
hughsaunders | cloudnull: no, I have tested extensively and can't get the check to fail, even with the exact response tat daneyon_ is getting :( | 16:29 |
daneyon_ | cloudnull: I will be rebuilding my env tomorrow and I will see if i run into the same error | 16:29 |
cloudnull | h... | 16:29 |
hughsaunders | daneyon_: thanks for helping with the debug, will be interesting to see if you hit it again | 16:29 |
cloudnull | BjoernT have you seen anything like this with your deployments ? | 16:29 |
daneyon_ | hughsaunders: will do | 16:29 |
BjoernT | No i did not see this behavior in our environments | 16:30 |
cloudnull | ok, so ill leave that issue open for the time being. | 16:31 |
cloudnull | next: https://bugs.launchpad.net/openstack-ansible/+bug/1431324 | 16:31 |
openstack | Launchpad bug 1431324 in openstack-ansible "gate: change ansible to run in virtual environment" [Undecided,New] | 16:31 |
*** sigmavirus24 is now known as sigmavirus24_awa | 16:31 | |
cloudnull | this seems like a sensible change and something that the bootstrap script should be able to take care of. | 16:32 |
cloudnull | thoughts? | 16:32 |
odyssey4me | cloudnull the ansible bootstrap or the aio bootstrap? | 16:32 |
cloudnull | bootstrap-ansible.sh | 16:32 |
cloudnull | i mean that we can do that as a rule and not as the exception . | 16:33 |
odyssey4me | I would think that perhaps we could make it optional | 16:33 |
cloudnull | or maybe make a tunable ? | 16:33 |
cloudnull | default sys install, tunable venv? | 16:33 |
odyssey4me | if deployers have a dedicated deployment server, then they should be able to deploy without the virtualenv | 16:33 |
odyssey4me | yep, I think that'd be best | 16:34 |
odyssey4me | the gate-check-commit can use the tunable | 16:34 |
cloudnull | sounds good to me. | 16:34 |
d34dh0r53 | +1 to the tunable | 16:34 |
cloudnull | bp to juno ? | 16:34 |
odyssey4me | I think that'd be wise. This issue was actually discovered when doing a multi-node juno set of gate tests. | 16:35 |
cloudnull | ok. | 16:35 |
odyssey4me | ideally we should have this before we tag 10.1.3 | 16:35 |
cloudnull | typie typie make it go | 16:36 |
cloudnull | and we're done here. with regard to new issues. | 16:37 |
cloudnull | do we have anything thats open that people want to lobby for ? | 16:37 |
BjoernT | let me check | 16:37 |
BjoernT | https://bugs.launchpad.net/openstack-ansible/+bug/1416880 probably should be moved into rpcextras? | 16:38 |
openstack | Launchpad bug 1416880 in openstack-ansible juno "Iteration over ansible device for disk_utilisation maas check include sr0 device" [Medium,New] | 16:38 |
cloudnull | yesir | 16:38 |
odyssey4me | +1 | 16:38 |
BjoernT | ok. | 16:38 |
BjoernT | https://bugs.launchpad.net/openstack-ansible/+bug/1411260 | 16:38 |
openstack | Launchpad bug 1411260 in openstack-ansible trunk "Redirect neutron-ha-tool logs to a file" [Wishlist,New] | 16:38 |
b3rnard0 | thanks BjoernT | 16:38 |
BjoernT | id like this one added for 10.1.3 as well and 9.0.7 ? | 16:39 |
odyssey4me | I disagree with doing this the way it's been suggested. | 16:39 |
odyssey4me | The tool logs to syslog already, which logs to a file. | 16:39 |
cloudnull | +1 | 16:39 |
BjoernT | -1 | 16:39 |
BjoernT | it's nightmare to debug | 16:40 |
odyssey4me | We've used the tool verbatim from the source and would rather not be changing it. | 16:40 |
cloudnull | +2 | 16:40 |
cloudnull | lol | 16:40 |
hughsaunders | my comment on that bug is out of date | 16:40 |
BjoernT | lolm the syslog is infected with dhcp messages | 16:40 |
odyssey4me | in that case perhaps it'd be better to implement some sort of syslog/dnsmasq configuration to manage that. | 16:41 |
BjoernT | either way is fine | 16:41 |
odyssey4me | alternatively, make the att neutron-ha-tool log to a specific syslog facility so that it can be diverted | 16:42 |
odyssey4me | but, quite honestly, I don't see why grep -v isn't easy :p | 16:42 |
odyssey4me | I would vote for icehouse - no fix | 16:43 |
cloudnull | https://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/neutron_l3_ha/templates/neutron_ha_tool.sh.j2#L34 - so we can change that to a file of sorts. | 16:43 |
BjoernT | not on millions of lines the logfiles growing fast | 16:43 |
cloudnull | but im with odyssey4me : grep -v dhcp -e neutron /var/log/message ... done. | 16:44 |
odyssey4me | cloudnull yeah the tool itself is relatively easy to change to use python debugging and thus configurable files.... but is it worth the effort? | 16:44 |
BjoernT | cloudnull: right. Exposing it to a file makes it available to /openstack/log/<container> | 16:45 |
BjoernT | something we use everyday | 16:45 |
BjoernT | also we have more diskspace available than /var/log/syslog | 16:45 |
odyssey4me | an alternative is to log it to local0-9 and configurs syslog to push that to /openstack/log/whatever | 16:46 |
odyssey4me | don't those logs get pushed into elasticsearch? | 16:47 |
BjoernT | yes they do. But elasticsearch is not working still | 16:48 |
odyssey4me | well, the simplest and the least moving parts would be to make the tool output to stdout and make the cron job redirect that output to a file | 16:49 |
BjoernT | +1 | 16:49 |
BjoernT | and don't forget logrotate | 16:49 |
BjoernT | Who's working on https://bugs.launchpad.net/openstack-ansible/+bug/1428833, if we are done with neutron-ha-tool logfile | 16:50 |
openstack | Launchpad bug 1428833 in openstack-ansible trunk "Add novnc console support in favor of spice" [High,Triaged] - Assigned to Andy McCrae (andrew-mccrae) | 16:50 |
BjoernT | ? | 16:50 |
odyssey4me | BjoernT andymccr has logged a bug upstream for it | 16:51 |
odyssey4me | if spice won't do it, we'll try and get it into horizon | 16:51 |
BjoernT | hmm, as I stated there are many other ossie and web sockify 0.6 still creates zombies | 16:51 |
BjoernT | I meant issues | 16:51 |
cloudnull | BjoernT: thats a no go in juno | 16:52 |
BjoernT | why that ? | 16:52 |
cloudnull | because its a large change to the stack | 16:52 |
cloudnull | we can see about doing it in trunk but theres not a lot of interest. | 16:52 |
BjoernT | it still is supported right ? I don't think it is large when we add additional containers and flip the config setting and adding a different F5 pool | 16:53 |
odyssey4me | uh, what're we talking about here? | 16:54 |
BjoernT | That said, RedHat for instance moved back to novnc. We are talking about spice vs novnc | 16:54 |
cloudnull | adding novnc instead of spice | 16:54 |
odyssey4me | ah, I'm happy to do this for master, and it may backport to juno but will have to only make it in for 10.2.0 - if that | 16:55 |
cloudnull | if that | 16:55 |
odyssey4me | it's a feature change, so requires a feature release according to the rules | 16:55 |
cloudnull | +1 | 16:55 |
odyssey4me | (the tagging rules in the release process) | 16:55 |
odyssey4me | that said, I see that Andy did volunteer to do it - the priority just hasn't been there yet | 16:56 |
BjoernT | You can call it what you want we have a broken console right now in RPC>9 and I don't see upstream can fix it fast enough. | 16:56 |
odyssey4me | it's already marked high for master, so ideally it'll make it into 11.0.0 | 16:56 |
BjoernT | Personally I don't care which console, if we can fix it enough | 16:56 |
BjoernT | fast enough | 16:57 |
odyssey4me | BjoernT so as we discussed last round, if it really is that bad for RPC then lobby for it within RPC | 16:57 |
BjoernT | lol, that issue will affect all users | 16:58 |
BjoernT | so what's next to discuss? | 16:59 |
odyssey4me | agreed, but os-ansible-deployment has yet to find it as a priority as the resource allocations have been focused elsewhere by its stakeholders | 16:59 |
b3rnard0 | looks like we are out of time | 17:00 |
b3rnard0 | notes are here: https://etherpad.openstack.org/p/openstack_ansible_bug_triage.2015-03-17-16.00 | 17:00 |
BjoernT | ok | 17:01 |
cloudnull | ok so we're done here. | 17:04 |
cloudnull | time for the foods | 17:04 |
cloudnull | thanks everyone :) | 17:04 |
*** vmtrooper has joined #openstack-ansible | 17:20 | |
*** vmtrooper has quit IRC | 17:25 | |
*** sigmavirus24_awa is now known as sigmavirus24 | 17:28 | |
*** Mudpuppy has joined #openstack-ansible | 17:37 | |
*** Mudpuppy has quit IRC | 17:39 | |
*** Mudpuppy has joined #openstack-ansible | 17:40 | |
palendae | Apsu: Any progress on the network stuff? | 17:40 |
*** Mudpuppy has quit IRC | 17:40 | |
*** Mudpuppy has joined #openstack-ansible | 17:41 | |
openstackgerrit | Nolan Brubaker proposed stackforge/os-ansible-deployment: Update README to reflect state of the lxc module https://review.openstack.org/165169 | 18:08 |
*** sdake__ has joined #openstack-ansible | 18:14 | |
cloudnull | core peoples please review https://review.openstack.org/#/c/164714/ and https://review.openstack.org/#/c/164933/ | 18:16 |
*** sdake has quit IRC | 18:18 | |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Rename test sets and include more api tests https://review.openstack.org/165173 | 18:19 |
cloudnull | andymccr hughsaunders odyssey4me d34dh0r53 mattt git-harry meow please > https://review.openstack.org/#/c/164714/ and https://review.openstack.org/#/c/164933/ <- i'd like to see our current blueprints become specs starting today if at all possible. | 18:21 |
* odyssey4me is looking | 18:22 | |
hughsaunders | updated gerrit watched projects to include the specs repo | 18:30 |
hughsaunders | should help not missing specs reviews | 18:31 |
odyssey4me | still work to be done in that review IMO | 18:31 |
d34dh0r53 | merged the rsyslog-client review, that looks very good to me | 18:32 |
hughsaunders | odyssey4me: yeah, I removed my +2 after seeing your comments | 18:32 |
openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs https://review.openstack.org/164933 | 18:39 |
cloudnull | responded to comments inline and fixed the couple of nits | 18:39 |
odyssey4me | also needs work in https://review.openstack.org/164714 | 18:44 |
*** sdake has joined #openstack-ansible | 18:45 | |
*** sdake__ has quit IRC | 18:48 | |
*** galstrom is now known as galstrom_zzz | 18:49 | |
openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Adds rsyslog client role and enables it in all plays https://review.openstack.org/164714 | 18:52 |
cloudnull | replied to inline comments and fixed nits ^ | 18:52 |
odyssey4me | cloudnull see published reply in patch set 2 for the rsyslog client | 18:54 |
cloudnull | say what ? | 18:55 |
odyssey4me | sorry - patch set 2 of specs template change | 18:55 |
cloudnull | odyssey4me: "'Make sure to document how gate testing will be impacted, indicating any changes which will need to be made to scripts, configurations, the AIO bootstrapping, etc. All changes made must be compatible with the AIO gate-check-commit and with a multi-node external CI gate check." ? | 18:57 |
odyssey4me | yup | 18:57 |
cloudnull | what if its a change in tempest only? | 18:57 |
cloudnull | IE "It is assumed that if the change is OpenStack related it should be accompanied by some kind of a test in Tempest" | 18:57 |
odyssey4me | if it's just tempest, then the gate check commit scripts are not impacted and that should be noted. | 18:58 |
cloudnull | so wouldn't that be covered with "Please discuss how the change will be tested" | 18:58 |
odyssey4me | essentially I'd like an explicit consideration of how an AIO and the gate checks will be impacted by whatever change. | 18:58 |
odyssey4me | I think that's only one part - the other part is how gate checks will be impacted. | 18:59 |
cloudnull | it seems a bit too specific, imo | 18:59 |
d34dh0r53 | odyssey4me: cloudnull: after reading through the comments in os_aggregate_storage.js on patch set 5 of rsyslog-client I'm thinking that we should set rotate and maxage to configurable vars and default them to 14 and 60 respectively. | 18:59 |
odyssey4me | if a bunch of tempest tests are added, will the gate check actually run those extra tests and does it have the required resources to do so? | 18:59 |
cloudnull | i would assume so as it would be a change to our tempest plays/runner script. | 19:00 |
cloudnull | but we're not testing everything in the aio as is | 19:01 |
cloudnull | so it may be a smoke test ? | 19:01 |
cloudnull | or something else ? | 19:01 |
odyssey4me | cloudnull that's the point - if someone is adding something, gate testing must be deliberately thought through and a note added about how the test are impacted - if it's not being tested in the commit test, then is it tested elsewhere or is it going to be an orphan of some sort | 19:02 |
cloudnull | do we provide language for all of the potential scenarios ? | 19:02 |
cloudnull | because thats the road we're going down . | 19:02 |
odyssey4me | I'm not asking for specialised language, just deliberate mention of gate test impacts | 19:02 |
cloudnull | ok. | 19:02 |
odyssey4me | d34dh0r53 I would think so, but if it's never used then it's just another thing to manage and maintain. So perhaps leaving it as-is for now is ok and it can be made configurable later if that is expressed as a need. | 19:04 |
odyssey4me | I'm happy with it as-is. | 19:05 |
d34dh0r53 | odyssey4me: yeah, that's fair. I had that consideration as well. Kind of a do it now or do it later question. | 19:05 |
d34dh0r53 | odyssey4me: ok, I'm good with it then | 19:06 |
*** vmtrooper has joined #openstack-ansible | 19:08 | |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Rename test sets and include more api tests https://review.openstack.org/165173 | 19:12 |
openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs https://review.openstack.org/164933 | 19:13 |
*** vmtrooper has quit IRC | 19:13 | |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Ensure return code passes through output trimming https://review.openstack.org/165085 | 19:14 |
openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs https://review.openstack.org/164933 | 19:14 |
odyssey4me | <---- happy with changes cloudnull | 19:16 |
cloudnull | hahaha | 19:16 |
openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs https://review.openstack.org/164933 | 19:19 |
cloudnull | i wasn't, it failed tox, for formatting. so i fixed it ^ | 19:19 |
cloudnull | line wrap over 79 ... | 19:19 |
*** KLevenstein_ has joined #openstack-ansible | 19:20 | |
odyssey4me | cloudnull you are a bad man :p | 19:20 |
*** KLevenstein has quit IRC | 19:20 | |
*** KLevenstein_ is now known as KLevenstein | 19:20 | |
*** KLevenstein has quit IRC | 19:38 | |
*** KLevenstein has joined #openstack-ansible | 19:40 | |
*** sdake__ has joined #openstack-ansible | 20:21 | |
*** sdake has quit IRC | 20:25 | |
*** sigmavirus24 is now known as sigmavirus24_awa | 20:44 | |
*** Mudpuppy has quit IRC | 20:44 | |
*** sdake has joined #openstack-ansible | 20:45 | |
cloudnull | this has now passed , cores please go review it https://review.openstack.org/#/c/164933/ | 20:46 |
openstackgerrit | Merged stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs https://review.openstack.org/164933 | 20:48 |
*** sdake__ has quit IRC | 20:49 | |
*** Mudpuppy has joined #openstack-ansible | 20:56 | |
*** vmtrooper has joined #openstack-ansible | 20:58 | |
d34dh0r53 | Is this: https://blueprints.launchpad.net/nova/+spec/io-ops-weight something we'd like to look at for kilofication of nova? | 21:02 |
*** vmtrooper has quit IRC | 21:02 | |
*** sigmavirus24_awa is now known as sigmavirus24 | 21:03 | |
BjoernT | d34dh0r53 possibly, customers would like that. Aren't there any libvirt QOS feature supported yet ? Something more interesting ;-) | 21:04 |
d34dh0r53 | :) not in the blueprints that I'm looking at, but I haven't been through all of them yet | 21:05 |
d34dh0r53 | cloudnull: ping | 21:11 |
cloudnull | pong | 21:11 |
d34dh0r53 | cloudnull: are we creating kilo/approved/<bp-name> for specs? | 21:11 |
cloudnull | just kilo/<bp-name> | 21:12 |
d34dh0r53 | 👍 | 21:12 |
cloudnull | and if its the first one make sure you add it to an index | 21:12 |
d34dh0r53 | ?? | 21:12 |
sigmavirus24 | d34dh0r53: eg. index.rst | 21:12 |
d34dh0r53 | ahh | 21:13 |
d34dh0r53 | does index.rst go in specs/ or specs/kilo/? | 21:16 |
cloudnull | yea we have specs/kilo/* | 21:22 |
cloudnull | in the index | 21:23 |
cloudnull | or rather we should have | 21:23 |
openstackgerrit | Merged stackforge/os-ansible-deployment: Add ldappool to keystone packages https://review.openstack.org/164715 | 21:24 |
*** KLevenstein has quit IRC | 21:28 | |
palendae | Does anyone know if we can make jobs skip gating if they're to specific files? Like, https://review.openstack.org/#/c/165169/ is failing on the gate tests | 21:28 |
palendae | Ideally - changes that include only READM.rst could skip a full gate commit check | 21:28 |
sigmavirus24 | palendae: don't think so | 21:29 |
Apsu | You get a gate! And You get a gate! Everybody gets a gate! | 21:30 |
palendae | Hm | 21:30 |
*** KLevenstein has joined #openstack-ansible | 21:31 | |
sigmavirus24 | Apsu: you get a jenkins-job-builder and you get an out of date project-config | 21:31 |
Apsu | yey! | 21:31 |
palendae | I'm assuming we get the changeset in the gate job, wonder if we could check for that | 21:31 |
Apsu | There really should be a nightly job that runs tox to generate new sample configs | 21:31 |
sigmavirus24 | palendae: git show HEAD? | 21:31 |
Apsu | Or a pre-merge hook | 21:31 |
sigmavirus24 | Apsu: new jenkins job that fails when it's out of date | 21:32 |
cloudnull | palendae we're waiting on an upstream commit from infra which will no-op rst and docs | 21:32 |
sigmavirus24 | propose that and watch how angry everyone gets | 21:32 |
Apsu | lolz | 21:32 |
palendae | cloudnull: Ok, I thought I saw that, but wasn't sure on the status | 21:32 |
sigmavirus24 | Apsu: you'll make TC with that moxie | 21:32 |
palendae | cloudnull: Cool, that's basically what I wantd, thanks | 21:32 |
Apsu | sigmavirus24: I'd be down with Moxie Marlinspike on TC | 21:33 |
* sigmavirus24 too | 21:33 | |
palendae | cloudnull: Ah, found it. https://review.openstack.org/#/c/164271/ I added a +1 | 21:36 |
*** Mudpuppy_ has joined #openstack-ansible | 21:41 | |
*** Mudpuppy has quit IRC | 21:44 | |
*** erikmwilson is now known as Guest84604 | 21:59 | |
*** erikmwilson has joined #openstack-ansible | 21:59 | |
*** sigmavirus24 is now known as sigmavirus24_awa | 22:14 | |
openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Enable fatal_deprecations configuration per OpenStack service https://review.openstack.org/164911 | 22:15 |
*** Mudpuppy has joined #openstack-ansible | 22:20 | |
*** Mudpuppy_ has quit IRC | 22:23 | |
*** Mudpuppy has quit IRC | 22:24 | |
*** erikmwilson has quit IRC | 22:25 | |
daneyon_ | Even if I deploy multiple haproxy load-balancers, keepalived is not part of the code for HA between the haproxy nodes... correct? | 22:37 |
daneyon_ | Also, any reason why haproxy is not containerized? | 22:39 |
mattt | daneyon_: there is no keepalived iirc | 22:40 |
daneyon_ | mattt: ya, i don't see it anywhere. I just wanted to make sure | 22:40 |
*** erikmwilson_ is now known as erikmwilson | 22:43 | |
mattt | daneyon_: i can't give you a conclusive reason behind us not putting it in a container, but most of our testing is done on cloud instances etc. and we use the routable IP from the instance itself for the haproxy VIP | 22:43 |
*** KLevenstein has quit IRC | 22:44 | |
*** vmtrooper has joined #openstack-ansible | 22:46 | |
*** vmtrooper has quit IRC | 22:51 | |
*** stevemar has quit IRC | 22:52 | |
daneyon_ | mattt: Thx for the feedback | 22:56 |
*** daneyon has joined #openstack-ansible | 23:18 | |
*** daneyon_ has quit IRC | 23:21 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!