*** d0ugal has joined #oooq | 00:06 | |
*** dsneddon has joined #oooq | 00:34 | |
*** dsneddon has quit IRC | 00:39 | |
*** dsneddon has joined #oooq | 01:07 | |
*** dsneddon has quit IRC | 01:12 | |
*** dsneddon has joined #oooq | 01:40 | |
*** dsneddon has quit IRC | 01:44 | |
*** bhagyashris has joined #oooq | 01:48 | |
*** dsneddon has joined #oooq | 02:13 | |
*** dsneddon has quit IRC | 02:18 | |
*** redrobot has quit IRC | 02:23 | |
*** Guest90568 has joined #oooq | 02:29 | |
*** Guest90568 is now known as redrobot | 02:32 | |
*** dsneddon has joined #oooq | 02:51 | |
*** dsneddon has quit IRC | 02:56 | |
*** surpatil has joined #oooq | 03:10 | |
*** dsneddon has joined #oooq | 03:27 | |
*** dsneddon has quit IRC | 03:31 | |
*** ykarel has joined #oooq | 03:51 | |
*** dsneddon has joined #oooq | 04:04 | |
*** dsneddon has quit IRC | 04:09 | |
*** saneax has quit IRC | 04:09 | |
*** ratailor has joined #oooq | 04:14 | |
*** udesale has joined #oooq | 04:30 | |
*** dsneddon has joined #oooq | 04:41 | |
*** dsneddon has quit IRC | 04:46 | |
*** ykarel has quit IRC | 05:09 | |
*** bhagyashris has quit IRC | 05:09 | |
*** skramaja has joined #oooq | 05:09 | |
*** ykarel has joined #oooq | 05:10 | |
*** suraj_ has joined #oooq | 05:12 | |
*** surpatil has quit IRC | 05:12 | |
*** ykarel is now known as ykarel|afk | 05:14 | |
*** suraj_ is now known as surpatil | 05:14 | |
*** surpatil has joined #oooq | 05:15 | |
*** surpatil has joined #oooq | 05:17 | |
*** kopecmartin|off is now known as kopecmartin | 05:18 | |
*** surpatil has joined #oooq | 05:18 | |
*** dsneddon has joined #oooq | 05:20 | |
*** raukadah is now known as chkumar|ruck | 05:23 | |
*** dsneddon has quit IRC | 05:25 | |
*** dsneddon has joined #oooq | 05:45 | |
*** ykarel|afk is now known as ykarel | 05:49 | |
*** jaosorior has joined #oooq | 05:51 | |
*** marios has joined #oooq | 05:52 | |
*** jbadiapa has joined #oooq | 06:05 | |
*** skramaja_ has joined #oooq | 06:07 | |
*** skramaja has quit IRC | 06:07 | |
*** marios has quit IRC | 06:18 | |
*** matbu has joined #oooq | 06:19 | |
*** ykarel is now known as ykarel|afk | 06:21 | |
*** marios has joined #oooq | 06:25 | |
*** saneax has joined #oooq | 06:27 | |
*** ykarel|afk is now known as ykarel | 06:38 | |
*** dsneddon has quit IRC | 06:48 | |
*** ksambor has quit IRC | 06:50 | |
*** ksambor has joined #oooq | 06:51 | |
*** jtomasek has joined #oooq | 06:58 | |
*** amoralej|off is now known as amoralej | 07:05 | |
*** udesale has quit IRC | 07:05 | |
*** udesale has joined #oooq | 07:06 | |
*** jaosorior has quit IRC | 07:14 | |
*** ykarel is now known as ykarel|afk | 07:16 | |
*** apetrich has joined #oooq | 07:17 | |
*** jtomasek has quit IRC | 07:21 | |
*** dsneddon has joined #oooq | 07:21 | |
*** ykarel|afk has quit IRC | 07:23 | |
*** dsneddon has quit IRC | 07:26 | |
*** yolanda has quit IRC | 07:42 | |
*** yolanda__ has joined #oooq | 07:43 | |
*** udesale has quit IRC | 07:43 | |
*** jpena|off is now known as jpena | 07:44 | |
*** udesale has joined #oooq | 07:44 | |
*** ykarel|afk has joined #oooq | 07:46 | |
*** ykarel|afk is now known as ykarel | 07:46 | |
*** dsneddon has joined #oooq | 07:59 | |
*** dsneddon has quit IRC | 08:04 | |
*** jaosorior has joined #oooq | 08:05 | |
*** chem has joined #oooq | 08:06 | |
*** matbu has quit IRC | 08:12 | |
*** skramaja has joined #oooq | 08:12 | |
*** skramaja_ has quit IRC | 08:12 | |
*** ykarel is now known as ykarel|afk | 08:23 | |
chkumar|ruck | sshnaidm: Hello | 08:26 |
---|---|---|
chkumar|ruck | sshnaidm: on this patch https://review.opendev.org/#/c/677063/ all the multi node jobs got failed with following error Data could not be sent to remote host "10.4.70.109". Make sure this host can be reached over ssh: ssh: connect to host 10.4.70.109 port 22: Network is unreachable | 08:27 |
chkumar|ruck | is it seen? | 08:27 |
*** yolanda__ is now known as yolanda | 08:28 | |
*** derekh has joined #oooq | 08:34 | |
*** dsneddon has joined #oooq | 08:35 | |
sshnaidm | chkumar|ruck, if it's limestone cloud, I wrote it in last mail | 08:35 |
sshnaidm | chkumar|ruck, but I didn't submit a patch for it | 08:37 |
sshnaidm | chkumar|ruck, s/patch/bug/ | 08:37 |
sshnaidm | it should be discussed with infra | 08:38 |
chkumar|ruck | sshnaidm: yes, all are limestone related | 08:39 |
*** dsneddon has quit IRC | 08:40 | |
sshnaidm | great.. ovh cloud fails because of unathorized errors, limestone fails in all jobs.. where should we run jobs then? :D | 08:41 |
sshnaidm | seems like rdo cloud is island of stability now | 08:43 |
chkumar|ruck | sshnaidm: we have issue with rrcockpit also | 08:44 |
chkumar|ruck | sshnaidm: here http://rhos-release.virt.bos.redhat.com:3030/rhosp stein promotion is 2d ago but in cockpit it is 11 days | 08:45 |
chkumar|ruck | I am not sure why it is showing like this | 08:47 |
*** ykarel|afk is now known as ykarel | 08:51 | |
*** matbu has joined #oooq | 09:02 | |
*** surpatil has quit IRC | 09:04 | |
*** dsneddon has joined #oooq | 09:10 | |
*** dsneddon has quit IRC | 09:14 | |
*** bhagyashris has joined #oooq | 09:17 | |
*** surpatil has joined #oooq | 09:30 | |
sshnaidm | chkumar|ruck, seems like something changed in dlrn api | 09:35 |
*** surpatil has quit IRC | 09:39 | |
chkumar|ruck | sshnaidm: nope https://softwarefactory-project.io/r/#/q/project:dlrnapi_client | 09:40 |
*** surpatil has joined #oooq | 09:40 | |
chkumar|ruck | changes are in progress | 09:40 |
chkumar|ruck | https://tree.taiga.io/project/morucci-software-factory/us/2648 | 09:40 |
chkumar|ruck | sshnaidm: do we have specific error? | 09:41 |
sshnaidm | chkumar|ruck, yeah, weird.. script doesn't return repo_hash | 09:41 |
*** pierreprinetti has joined #oooq | 09:43 | |
*** dsneddon has joined #oooq | 09:46 | |
*** pierreprinetti has quit IRC | 09:48 | |
*** pierreprinetti has joined #oooq | 09:48 | |
*** dsneddon has quit IRC | 09:50 | |
panda | marios: all test jobs in https://tree.taiga.io/project/tripleo-ci-board/task/1232?kanban-status=1447276 are failing, but I think they are passing the critical point of your card, right ? | 09:50 |
marios | panda: yeah, i was goign to add a depends-on to sagi patch in a sec to see how far it can get | 09:52 |
marios | panda: but for your question yes commented in https://review.rdoproject.org/r/#/c/21787/6/zuul.d/standalone-jobs.yaml@3302 | 09:53 |
panda | marios: ok, call me when you're satisfied and we can merge this. | 09:54 |
panda | marios: 1-800-panda | 09:54 |
*** apetrich has quit IRC | 09:56 | |
marios | panda: thanks for looking. zbr volunteered to baby sit that when i'm out | 09:56 |
panda | marios: when are you out ? | 09:57 |
marios | panda: (see pvt) & will email later | 09:58 |
sshnaidm | CI is mostly down, limestone cloud fails all centos jobs, ovh cloud fails with unauthorized errors | 09:59 |
sshnaidm | marios, panda ^^ | 10:00 |
sshnaidm | only thing that works is rdo cloud :D | 10:00 |
sshnaidm | never believed I'd say that | 10:00 |
chkumar|ruck | sshnaidm: if it possible to hold one of the limestone centos node and debug there? | 10:02 |
panda | we are having only two providers ? | 10:02 |
chkumar|ruck | *is it | 10:02 |
marios | panda: please also check https://review.rdoproject.org/r/#/c/21832/ when you have time for the layout | 10:04 |
marios | sshnaidm: thanks | 10:05 |
sshnaidm | chkumar|ruck, no idea, better ask on #openstack-infra | 10:06 |
panda | marios: each extra review is going to cost you a day of PTO. | 10:14 |
marios | panda: i am considering entering witness protection | 10:15 |
*** dsneddon has joined #oooq | 10:17 | |
panda | marios: you can't, you lack the wit. | 10:17 |
marios | damn foiled again ! | 10:18 |
panda | marios: it's a nice way to get used to aluminium. | 10:21 |
*** dsneddon has quit IRC | 10:22 | |
marios | panda: does wonders for your hair too and helps to keep indoor lighting requirements to a minimum as long as you sit near a window | 10:23 |
zbr | marios: can you explain me why you added a variable that gets assigned "false" string? | 10:29 |
zbr | trying to figureout what made you dothat | 10:29 |
marios | zbr: maybe but it depends on if you can point me to where you are referring to :D | 10:30 |
marios | zbr: or even better comment there and point to it here | 10:30 |
marios | zbr: and i'll update if there is somehting missing | 10:30 |
zbr | ahh. is was not your change. it was sshnaidm https://review.opendev.org/#/c/676497/5 | 10:30 |
marios | zbr: k | 10:30 |
sshnaidm | zbr, what is the problem? | 10:32 |
zbr | sshnaidm: ^ can you help on that? if possible would like to avoid use of strings for boolean vars, it's a road to pain. if no special reason I can fix it. | 10:32 |
zbr | both python and yaml support booleans. | 10:33 |
sshnaidm | zbr, booleans don't work well in heat template + jinja + yaml + godknowswhatelsethere | 10:34 |
sshnaidm | zbr, but you can try | 10:34 |
*** skramaja has quit IRC | 10:39 | |
*** bhagyashris has quit IRC | 10:50 | |
*** pierreprinetti has quit IRC | 10:53 | |
*** dsneddon has joined #oooq | 10:55 | |
*** dsneddon has quit IRC | 11:00 | |
* marios food biab | 11:12 | |
*** marios has quit IRC | 11:12 | |
*** udesale has quit IRC | 11:15 | |
*** tesseract has joined #oooq | 11:15 | |
chkumar|ruck | sshnaidm: as per donnnyd on #openstack-infra, there is no issue with centos image side | 11:16 |
sshnaidm | chkumar|ruck, cool, then problem is in different place of limestone :) | 11:17 |
*** pierreprinetti has joined #oooq | 11:20 | |
*** dsneddon has joined #oooq | 11:33 | |
chkumar|ruck | sshnaidm: do we have a filed bug for rhel8 tempest issue? | 11:36 |
*** apetrich has joined #oooq | 11:36 | |
sshnaidm | chkumar|ruck, I don't think so | 11:37 |
sshnaidm | chkumar|ruck, will submit | 11:37 |
*** dsneddon has quit IRC | 11:38 | |
*** jpena is now known as jpena|lunch | 11:39 | |
sshnaidm | chkumar|ruck, https://bugs.launchpad.net/tripleo/+bug/1840645 | 11:40 |
openstack | Launchpad bug 1840645 in tripleo "Tempest network scenario test fails on rhel8 host because "No such file or directory: 'ping': 'ping'"" [High,Triaged] | 11:40 |
sshnaidm | chkumar|ruck, ok, got it.. " OK for 11 days " it means the alarm is OK for 11 days, not "last promotion was 11 days before" | 11:45 |
sshnaidm | chkumar|ruck, http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?panelId=263&fullscreen&edit&tab=alert - for last 11 days no alarms for stein promotions | 11:46 |
chkumar|ruck | sshnaidm: ok got it, thanks :-) | 11:47 |
*** rlandy has joined #oooq | 11:47 | |
*** rlandy is now known as rlandy|rover | 11:48 | |
rlandy|rover | chkumar|ruck: hi - are we switching today? | 11:49 |
chkumar|ruck | rlandy|rover: yes | 11:49 |
rlandy|rover | k | 11:49 |
*** chkumar|ruck is now known as chkumar|rover | 11:49 | |
*** rlandy|rover is now known as rlandy|ruck | 11:49 | |
rlandy|ruck | chkumar|rover: https://bugs.launchpad.net/tripleo/+bug/1840451 - is this something we fix or a dfg? | 11:51 |
openstack | Launchpad bug 1840451 in tripleo "Master promotions are failing tempest test (fs020) tempest.api.volume.admin.test_volume_quotas.VolumeQuotasAdminTestJSON" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 11:51 |
chkumar|rover | rlandy|ruck: we need to bug storage dfg | 11:52 |
rlandy|ruck | I am editing the card on escalation board | 11:52 |
rlandy|ruck | k | 11:52 |
chkumar|rover | rlandy|ruck: I have left all my finding on the bug itself | 11:52 |
rlandy|ruck | I see that - thanks | 11:52 |
rlandy|ruck | do you know who is our storage contact? | 11:52 |
chkumar|rover | might be eric harney | 11:52 |
rlandy|ruck | k | 11:53 |
rlandy|ruck | chkumar|rover: what's the story with https://trello.com/c/ZvV5ul81/1029-cixlp1836046tripleociproa-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-on-queens - queens is promoting now | 11:56 |
chkumar|rover | rlandy|ruck: I am trying with config_drive there in my patch | 11:57 |
rlandy|ruck | chkumar|rover: the card is still open but queens has been promoting successfully | 11:58 |
chkumar|rover | rlandy|ruck: it comes every twice in aprox 5-6 runs | 11:58 |
rlandy|ruck | ha - and master promoted http://rhos-release.virt.bos.redhat.com:3030/rhosp | 11:58 |
* rlandy|ruck is confused | 11:58 | |
*** weshay_pto is now known as weshay | 12:00 | |
weshay | 0/ | 12:00 |
*** marios has joined #oooq | 12:05 | |
*** ykarel is now known as ykarel|afk | 12:05 | |
*** dsneddon has joined #oooq | 12:06 | |
chkumar|rover | sshnaidm: https://review.opendev.org/#/c/677191/ might fix the tempest issue | 12:07 |
chkumar|rover | sshnaidm: does kolla patches can be tested in ovb jobs? | 12:09 |
sshnaidm | chkumar|rover, no, we don't build containers in ovb job | 12:09 |
sshnaidm | chkumar|rover, actually we use kolla in containers-build jobs only | 12:09 |
chkumar|rover | sshnaidm: do adding the same in tripleo-common kolla override will also not help? | 12:10 |
sshnaidm | chkumar|rover, what do you mean? | 12:11 |
*** dsneddon has quit IRC | 12:11 | |
chkumar|rover | sshnaidm: I think i got the answer adding the tempest_packages_append here https://github.com/openstack/tripleo-common/blob/master/container-images/tripleo_kolla_template_overrides.j2 is also not going to help | 12:12 |
chkumar|rover | I need to find another way of testing it | 12:12 |
*** jaosorior has quit IRC | 12:14 | |
sshnaidm | chkumar|rover, maybe we can do like that: 1) build the tempest container with your modifications, 2) upload it somewhere on dockerhub, 3) in tripleo-common replace path with name_prefix of tempest container to yours, 4) depends on this patch and run ovb job | 12:16 |
*** amoralej is now known as amoralej|lunch | 12:16 | |
chkumar|rover | sshnaidm: sounds like a good plan | 12:17 |
sshnaidm | chkumar|rover, but in general we need to develop such option - to build a container and test it in a job | 12:17 |
sshnaidm | for future | 12:17 |
*** apetrich has quit IRC | 12:20 | |
ykarel|afk | sshnaidm, running both jobs in check pipeline with depends-on should also work | 12:25 |
ykarel|afk | ovb depend on container push | 12:25 |
ykarel|afk | and patch depend on kolla | 12:26 |
sshnaidm | ykarel|afk, ovb job takes containers from dockerhub, it doesn't build them | 12:26 |
ykarel|afk | sshnaidm, periodic one don't take from trunk registry? | 12:26 |
ykarel|afk | i think they do, let me check again | 12:26 |
sshnaidm | ykarel|afk, yes, periodic does take them from trunk | 12:27 |
sshnaidm | ykarel|afk, but we talk about check job | 12:27 |
ykarel|afk | sshnaidm, ohhk | 12:27 |
ykarel|afk | sshnaidm, so you facing issue only in check? | 12:27 |
sshnaidm | if building containers job could hold these containers and pass it anyhow to ovb job, it would work | 12:27 |
sshnaidm | ykarel|afk, we need to check container modification in ovb job, that's the problem | 12:28 |
*** apetrich has joined #oooq | 12:28 | |
*** ykarel|afk is now known as ykarel | 12:28 | |
rlandy|ruck | sshnaidm: after scrum, can we sync for re: sova changes? | 12:28 |
*** rfolco has joined #oooq | 12:28 | |
sshnaidm | ykarel|afk, we need option to build a specific container like we build a specific rpm.. | 12:29 |
sshnaidm | rlandy|ruck, sure | 12:29 |
ykarel | sshnaidm, okk got it, currently we either build or update all containers | 12:29 |
sshnaidm | yep | 12:29 |
*** jpena|lunch is now known as jpena | 12:31 | |
chkumar|rover | sshnaidm: can we do a force push for this patch with the help of openstack-infra root admins? | 12:32 |
chkumar|rover | check jobs are broken | 12:32 |
chkumar|rover | sshnaidm: https://review.opendev.org/#/c/677063/ | 12:32 |
chkumar|rover | sshnaidm: the issue while doing gathering facts on subnode 2 is known https://github.com/ansible/ansible/issues/57542 | 12:36 |
sshnaidm | chkumar|rover, not sure I understand | 12:37 |
*** jaosorior has joined #oooq | 12:38 | |
chkumar|rover | sshnaidm: about this issue Data could not be sent to remote host "10.4.70.30". Make sure this host can be reached over ssh: ssh: connect to host 10.4.70.30 port 22: Network is unreachable | 12:38 |
chkumar|rover | sshnaidm: https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/logs_82/677182/1/check/tripleo-ci-centos-7-containers-multinode-stein/0b4a29b/job-output.txt | 12:38 |
sshnaidm | chkumar|rover, yeah? | 12:39 |
chkumar|rover | sshnaidm: as per this warning [WARNING]: Unhandled error in Python interpreter discovery for host subnode-2 -> it appears it is related to this ansible issue https://github.com/ansible/ansible/issues/57542 | 12:40 |
weshay | chkumar|rover congrats!!! | 12:42 |
weshay | chkumar|rover rlandy|ruck anything I can do for you? | 12:42 |
chkumar|rover | weshay: thanks :-) | 12:42 |
*** dsneddon has joined #oooq | 12:42 | |
sshnaidm | chkumar|rover, hmm.. idk | 12:42 |
chkumar|rover | brb for 20 mins! | 12:43 |
rlandy|ruck | weshay: there a bunch of stuff going on | 12:46 |
rlandy|ruck | weshay: we can sync after team scrum | 12:46 |
weshay | k | 12:46 |
*** ykarel is now known as ykarel|away | 12:47 | |
*** dsneddon has quit IRC | 12:47 | |
rlandy|ruck | log moves, rrcockpit updates, gate failure lots of fun | 12:47 |
sshnaidm | ykarel|away, chkumar|rover congrats to you both :) | 12:48 |
rlandy|ruck | oh wow - congrats chkumar|rover and ykarel|away - well deserved | 12:50 |
weshay | ykarel|away wooooot!!! congrats | 12:51 |
*** ykarel|away has quit IRC | 12:52 | |
chkumar|rover | sshnaidm: rlandy|ruck thanks :-) | 12:57 |
weshay | chkumar|rover rlandy|ruck arxcruz I see the periodic reproducer jobs are all red, in post_failure.. anyone know why? | 12:57 |
rlandy|ruck | weshay: yep - saw that - will check | 12:58 |
rlandy|ruck | weshay: do we plan to add redhat8-master jobs to http://rhos-release.virt.bos.redhat.com:3030/rhosp? | 12:58 |
rlandy|ruck | we are not tracking that progress anywhere public | 12:58 |
weshay | rlandy|ruck it would just be lumped in w/ master | 12:59 |
chkumar|rover | weshay: and do we plan to have RDO phase 1 jobs also on rhel-8? | 12:59 |
chkumar|rover | *have plan | 12:59 |
weshay | so if standalone rhel 8 is stable.. we could add that to criteria | 12:59 |
*** pierreprinetti has quit IRC | 12:59 | |
rlandy|ruck | no redhat nodes | 12:59 |
rlandy|ruck | on ci.centos | 12:59 |
weshay | that's fine | 12:59 |
rlandy|ruck | we would have ti add those jobs somewhere else | 12:59 |
weshay | those jobs are almost worthless | 12:59 |
*** pierreprinetti has joined #oooq | 12:59 | |
rlandy|ruck | phase 1? | 13:00 |
weshay | ya | 13:00 |
rlandy|ruck | in general? | 13:00 |
rlandy|ruck | k - so we should add that to our sync discussions | 13:00 |
*** dsneddon has joined #oooq | 13:00 | |
rlandy|ruck | along with other fun | 13:00 |
*** pierreprinetti has quit IRC | 13:00 | |
rlandy|ruck | no rest for the ruck/rover this sprint | 13:00 |
*** ratailor has quit IRC | 13:00 | |
rfolco | ping scrum zbr | 13:01 |
rfolco | sshnaidm, ping scrum | 13:02 |
*** pierreprinetti has joined #oooq | 13:06 | |
rlandy|ruck | hmmm .. redhat8 has not promoted in a while | 13:07 |
chkumar|rover | rlandy|ruck: /me has seen check jobs green last week | 13:07 |
rlandy|ruck | why were the build jobs skipped | 13:08 |
rlandy|ruck | last promoted on August 16 | 13:08 |
rlandy|ruck | 2019-08-19 12:34:29.975352 | TASK [prepare-node : Dump hostvars on the host to pass it to inner ansible] | 13:08 |
rlandy|ruck | 2019-08-19 12:34:30.038849 | primary | ERROR | 13:08 |
rlandy|ruck | 2019-08-19 12:34:30.039207 | primary | { | 13:08 |
rlandy|ruck | 2019-08-19 12:34:30.039321 | primary | "msg": "The task includes an option with an undefined variable. The error was: 'featureset' is undefined\n\nThe error appears to have been in '/var/opt/rh/rh-python35/lib/zuul/builds/dddf778627ee4c51af594ed4516b80b9/untrusted/project_3/opendev.org/openstack/tripleo-ci/roles/prepare-node/tasks/main.yaml': line 40, column 3, but may\nbe elsewhere in the file depending on the exact | 13:08 |
rlandy|ruck | syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Dump hostvars on the host to pass it to inner ansible\n ^ here\n" | 13:08 |
rlandy|ruck | 2019-08-19 12:34:30.039387 | primary | } | 13:08 |
chkumar|rover | rlandy|ruck: it is a known issue | 13:09 |
chkumar|rover | rlandy|ruck: https://review.opendev.org/#/c/677063 | 13:09 |
chkumar|rover | got stuck in gate issue | 13:09 |
chkumar|rover | thanks to sshnaidm for fixing on weekend | 13:09 |
chkumar|rover | rlandy|ruck: line https://etherpad.openstack.org/p/ruckroversprint14 30 to 33 | 13:10 |
rlandy|ruck | it broke over the weekend | 13:11 |
*** aakarsh has quit IRC | 13:18 | |
*** amoralej|lunch is now known as amoralej | 13:25 | |
weshay | https://mojo.redhat.com/groups/omega-prime/overview | 13:26 |
chkumar|rover | arxcruz: congrats :-) | 13:30 |
arxcruz | chkumar|rover:thanks :) | 13:36 |
*** surpatil has quit IRC | 13:36 | |
*** ykarel|away has joined #oooq | 13:43 | |
*** ykarel|away is now known as ykarel | 13:45 | |
*** aakarsh has joined #oooq | 13:54 | |
weshay | chkumar|rover a lot of these user stories are in parked afaict | 13:55 |
chkumar|rover | weshay: /me is looking at the board, I will ask the team to update the status or will do that, | 13:58 |
chkumar|rover | rlandy|ruck: is it possible to put this in fs020 CinderLVMLoopDeviceSize itself? | 13:59 |
rlandy|ruck | chkumar|rover: the master bug? | 14:00 |
chkumar|rover | rlandy|ruck: yes | 14:00 |
chkumar|rover | rlandy|ruck: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/cinder/cinder-volume-container-puppet.yaml#L95 | 14:01 |
chkumar|rover | rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1840451 | 14:01 |
openstack | Launchpad bug 1840451 in tripleo "Master promotions are failing tempest test (fs020) tempest.api.volume.admin.test_volume_quotas.VolumeQuotasAdminTestJSON" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 14:01 |
rlandy|ruck | chkumar|rover: if that is passed to the deploy, yes | 14:01 |
rlandy|ruck | we have a number of files that pass parameters | 14:01 |
rlandy|ruck | if that is passed directly to tempest, not sure | 14:02 |
rlandy|ruck | chkumar|rover: getting files | 14:02 |
*** dsneddon has quit IRC | 14:03 | |
chkumar|rover | rlandy|ruck: we need to increase CinderLVMLoopDeviceSize value in deployment itself during overcloud deploy | 14:03 |
rlandy|ruck | chkumar|rover: ack ... sorry having concurrent conversation on sf-ops re: reproducer failures | 14:04 |
rlandy|ruck | we have these options | 14:04 |
ykarel | chkumar|rover, rlandy|ruck just noticed in stein full tempest job, all tests are not running | 14:05 |
rlandy|ruck | https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/1ea6dd7/logs/undercloud/home/zuul/overcloud-deploy.sh | 14:05 |
ykarel | chkumar|rover, rlandy|ruck tempest plugins are not installed in tempest container | 14:05 |
rlandy|ruck | will look in a bit | 14:06 |
chkumar|rover | ykarel: we install openstack-tempest-all there in contauner | 14:06 |
ykarel | chkumar|rover, nope it's not installed in stein | 14:06 |
rlandy|ruck | chkumar|rover: k - I'll try add that option | 14:06 |
ykarel | so no plugins | 14:06 |
ykarel | chkumar|rover, https://review.opendev.org/#/c/651601/5/docker/tempest/Dockerfile.j2 | 14:07 |
ykarel | need to fix it in stein ^^ | 14:07 |
chkumar|rover | ykarel: ok fix that | 14:07 |
ykarel | chkumar|rover, /me propose or u will take care? | 14:08 |
chkumar|rover | ykarel: taking care | 14:09 |
weshay | marios want to sync? | 14:09 |
arxcruz | i have a doctor appointment brb | 14:09 |
ykarel | chkumar|rover, ack | 14:09 |
weshay | marios I don't hvave much, so your call | 14:09 |
marios | weshay: ack pvt | 14:09 |
weshay | marios have anything you want to discuss? | 14:10 |
marios | weshay: not particularly will summarize my stuff and sold most of it already | 14:10 |
zbr | rlandy|ruck: are you aware about an error The task includes an option with an undefined variable. The error was: 'featureset' is undefined ? | 14:12 |
sshnaidm | zbr, arxcruz are you sure ansible-version is inherited via different zuul systems? | 14:12 |
sshnaidm | I'd check that on #zuul | 14:12 |
rfolco | panda, weshay: please give me a time to discuss staging env today or tomorrow. | 14:12 |
chkumar|rover | zbr: https://review.opendev.org/#/c/677063/ | 14:12 |
zbr | sshnaidm: almost sure is just a variable, so it will be inhereted. also very easy to discover. we only need to define it in a job. | 14:13 |
rfolco | arxcruz, congrats man | 14:13 |
*** pierreprinetti has quit IRC | 14:13 | |
rlandy|ruck | zbr: yes | 14:14 |
panda | rfolco: all the time in the world | 14:14 |
*** pierreprinetti has joined #oooq | 14:14 | |
sshnaidm | zbr, arxcruz I think it's worth to switch one rhel8 job to 2.8 in zuul, to remove hostvars and see if it helps, before doing any global changes | 14:14 |
* panda raises both his hands to the sky | 14:14 | |
rlandy|ruck | zbr: if you mean https://github.com/openstack/tripleo-ci/commit/84018c747d5a1a1da7ad4fcbc31c2c7ff3b5963e change | 14:14 |
sshnaidm | zbr, I believe it is, but just to be sure.. | 14:14 |
panda | rfolco: here take my life energy | 14:15 |
zbr | rlandy|ruck: not really, is about this error https://45da25d3022f8ec0f64e-bc6452672c10420faa7f3759f0d39f73.ssl.cf1.rackcdn.com/677059/1/check/tripleo-buildimage-overcloud-full-centos-7/b577b12/job-output.txt -- which has nothing to do with the change being tested. | 14:15 |
*** sshnaidm is now known as sshnaidm|bbl | 14:16 | |
rlandy|ruck | https://review.opendev.org/#/c/677063 | 14:16 |
rlandy|ruck | zbr: ^^ | 14:16 |
zbr | damn, containers already failed. | 14:16 |
rlandy|ruck | that is the fix | 14:17 |
chkumar|rover | ykarel: rlandy|ruck https://review.opendev.org/#/c/677212/ | 14:17 |
rfolco | weshay, need to discuss staging env, the earlier the better... you seem to be free 40 min from now. Is it a good time ? | 14:17 |
rfolco | panda, ^ | 14:17 |
rlandy|ruck | chkumar|rover: thanks | 14:17 |
* panda gives rfolco his bleeding heart | 14:18 | |
ykarel | chkumar|rover, ack | 14:18 |
ykarel | +1 | 14:18 |
panda | rfolco: just extracted, take it, it's yours to use. | 14:18 |
rlandy|ruck | ugh - everything on fire today | 14:18 |
chkumar|rover | weshay: rlandy|ruck sshnaidm|bbl https://review.opendev.org/#/c/677063/ can we get this patch force merged with the help of openstack-infra roots | 14:19 |
ykarel | chkumar|rover, one comment | 14:19 |
chkumar|rover | it will help to bring back the ci green except limestone 401 issues | 14:19 |
ykarel | wrong link | 14:20 |
rlandy|ruck | as long as the errors are not legit | 14:20 |
zbr | why we do have this double compression and broken file type on undercloud-install archive? https://6484f2cfee4024eb280c-d3a3c82ce3aa1074eb73e263d71495ca.ssl.cf5.rackcdn.com/677063/8/check/tripleo-ci-centos-7-undercloud-containers/e860eac/logs/undercloud/home/zuul/ | 14:21 |
rfolco | panda, weshay: staging env call starts in 38 min | 14:22 |
rfolco | invitation sent | 14:22 |
rfolco | zbr, added you and others as optional to attend ^ | 14:22 |
panda | rfolco: I'm sorry, I can't | 14:25 |
rfolco | panda, :( | 14:26 |
rfolco | panda, you hurt my feelings | 14:26 |
panda | rfolco: I have judo lessons in 15 minutes | 14:26 |
rfolco | panda, you gave my heart | 14:26 |
rfolco | your hear | 14:26 |
rfolco | t | 14:26 |
panda | rfolco: it was a pig heart | 14:26 |
rfolco | you broke my heart | 14:26 |
weshay | rfolco ya.. add me to a call for sure | 14:26 |
rfolco | weshay, panda can't moving to tomorrow | 14:27 |
panda | rfolco: weshay just joking | 14:27 |
panda | rfolco: weshay I'm here | 14:27 |
rfolco | damnn | 14:27 |
panda | rfolco: stop taking me seriously | 14:27 |
rfolco | panda, you broke my heart and laugh on my face | 14:27 |
rfolco | panda, what kind of human being you are ? | 14:27 |
zbr | rlandy|ruck: it is normal to get "authentication required" when pulling from dockerhub? https://6484f2cfee4024eb280c-d3a3c82ce3aa1074eb73e263d71495ca.ssl.cf5.rackcdn.com/677063/8/check/tripleo-ci-centos-7-undercloud-containers/e860eac/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 14:28 |
rlandy|ruck | zbr: https://bugs.launchpad.net/tripleo/+bug/1839532 | 14:30 |
openstack | Launchpad bug 1839532 in tripleo "tripleo gate jobs are failing to pull containers when running on ovh provider with "UNAUTHORIZED" error" [Critical,Triaged] | 14:30 |
rlandy|ruck | going to bring that up with weshay | 14:30 |
rlandy|ruck | we have no fix and it's a major roblem | 14:30 |
rlandy|ruck | problem | 14:30 |
weshay | rlandy|ruck cix it | 14:30 |
rlandy|ruck | just fixing two other bugs atm | 14:30 |
zbr | yeah, i see it. thanks for the link. | 14:30 |
rlandy|ruck | in a few | 14:30 |
*** yolanda has quit IRC | 14:31 | |
*** dtrainor has joined #oooq | 14:31 | |
*** yolanda has joined #oooq | 14:36 | |
*** dsneddon has joined #oooq | 14:39 | |
*** dsneddon has quit IRC | 14:43 | |
*** saneax has quit IRC | 14:49 | |
weshay | rlandy|ruck chkumar|rover I have 10 min if you guys need something | 14:50 |
rlandy|ruck | in the middle of bug fixing - will ping later | 14:50 |
*** surpatil has joined #oooq | 14:52 | |
*** ykarel has quit IRC | 14:57 | |
rlandy|ruck | chkumar|rover: let's see what this does ... https://review.opendev.org/677227 Add overcloud deploy parameter to set CinderLVMLoopDeviceSize | 15:00 |
rlandy|ruck | https://review.rdoproject.org/r/#/c/21851/ rocky should promote | 15:02 |
rlandy|ruck | queens is in trouble | 15:02 |
*** jaosorior has quit IRC | 15:04 | |
chkumar|rover | rlandy|ruck: what is the new queens issue? except tempest fs01 | 15:13 |
rlandy|ruck | chkumar|rover: https://review.rdoproject.org/r/#/c/21862/ rerun | 15:14 |
rlandy|ruck | all failed | 15:14 |
rlandy|ruck | will look at it after fixing reproducer failures | 15:14 |
*** dsneddon has joined #oooq | 15:14 | |
chkumar|rover | checking | 15:15 |
chkumar|rover | rlandy|ruck: it is tempest | 15:15 |
chkumar|rover | for queens | 15:15 |
chkumar|rover | let me try with my fix https://review.opendev.org/#/c/676439/ | 15:17 |
*** saneax has joined #oooq | 15:17 | |
chkumar|rover | rlandy|ruck: https://review.rdoproject.org/r/#/c/21862/ updated here, we will know in another 3 hrs | 15:18 |
rlandy|ruck | chkumar|rover: k - thanks - will watch | 15:18 |
*** dsneddon has quit IRC | 15:19 | |
chkumar|rover | rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1840616 needs attension | 15:20 |
openstack | Launchpad bug 1840616 in tripleo "Master/stein check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged] | 15:20 |
chkumar|rover | ansibel timed out during overcloud deploy | 15:20 |
chkumar|rover | I have seen last week and today also | 15:20 |
rlandy|ruck | chkumar|rover: we can increase the deploy timeout | 15:21 |
chkumar|rover | rlandy|ruck: it is related to docker pull at some places | 15:21 |
rlandy|ruck | 2019-08-14 03:57:14 | >f+++++++++ fOvercloud configuration failed. | 15:21 |
chkumar|rover | rlandy|ruck: weshay I am logging out, please get this merged https://review.opendev.org/#/c/677063/ if possible, I will take over from tomorrow | 15:24 |
chkumar|rover | morning | 15:24 |
chkumar|rover | see ya | 15:24 |
rlandy|ruck | chkumar|rover: ack - thanks | 15:24 |
*** chkumar|rover is now known as raukadah | 15:24 | |
rlandy|ruck | weshay: let me know when you have time to sync | 15:29 |
weshay | k.. need a few | 15:31 |
marios | panda: feauxmotion | 15:36 |
marios | please that ^ | 15:36 |
rfolco | marios, need a 5 min handoff if you can | 15:37 |
marios | rfolco: ack | 15:38 |
marios | rfolco: as soon as we finish on this call? | 15:38 |
rfolco | yep | 15:38 |
panda | marios: faux | 15:41 |
panda | marios: also foomotion could be good | 15:42 |
rlandy|ruck | 2019-08-19 14:23:18,259 773 INFO promoter Promoting the container images for dlrn hash aef1c3ef1ed640e269fbb45e067934e7cdbad697 on rocky to current-tripleo | 15:42 |
rlandy|ruck | rocky promoting - one step | 15:42 |
*** dsneddon has joined #oooq | 15:50 | |
*** dsneddon has quit IRC | 15:55 | |
* marios almost out | 15:56 | |
weshay | rlandy|ruck k.. ready whenever you are | 15:57 |
rlandy|ruck | weshay: k - joing your bj | 15:58 |
*** marios is now known as marios|out | 16:03 | |
*** Vorrtex has joined #oooq | 16:08 | |
*** jpena is now known as jpena|off | 16:11 | |
weshay | rlandy|ruck https://wiki.centos.org/About/Building_8 | 16:20 |
weshay | raukadah fyi ^.. they are getting close.. no action required atm | 16:20 |
weshay | just fyi | 16:20 |
raukadah | weshay: /me will wait for the announcements. | 16:21 |
*** dsneddon has joined #oooq | 16:23 | |
*** surpatil has quit IRC | 16:25 | |
*** dsneddon has quit IRC | 16:27 | |
rlandy|ruck | lunch - brb\ | 16:28 |
weshay | rlandy|ruck sorry.. my computer overheated | 16:31 |
weshay | shutdown | 16:31 |
*** marios|out has quit IRC | 16:33 | |
*** tesseract has quit IRC | 16:35 | |
*** dsneddon has joined #oooq | 16:39 | |
rlandy|ruck | weshay: np - finished my tales of woe | 16:41 |
weshay | :) | 16:41 |
zbr | sshnaidm|bbl: rlandy|ruck can we do https://review.opendev.org/#/c/676474/3 ? | 16:43 |
*** derekh has quit IRC | 17:03 | |
rlandy|ruck | back looking | 17:04 |
rlandy|ruck | weshay, https://review.rdoproject.org/r/21856 Remove Fedora28 stein/master and add RedHat 8 master (rrcockpit updates) | 17:05 |
rlandy|ruck | pls see if that works for you | 17:05 |
weshay | k.. thanks | 17:05 |
*** amoralej is now known as amoralej|off | 17:08 | |
zbr | weshay: brilliant design: i change one job and I see >20 running, none of them based on the one I touched: https://review.opendev.org/#/c/677256/ | 17:11 |
rlandy|ruck | https://review.opendev.org/#/c/677063 - still failing - may be legit | 17:11 |
weshay | tripleo-build-containers-centos-7-buildahFAILURE in 1h 13m 44s | 17:17 |
weshay | that one? | 17:17 |
weshay | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_63/677063/8/gate/tripleo-build-containers-centos-7-buildah/e459824/logs/containers-build-errors.log.txt.gz | 17:19 |
weshay | SystemError: The following jobs were incomplete: set([<Future at 0x7f927d9fdb10 state=finished returned NoneType>]) | 17:22 |
weshay | 2019-08-19 03:43:50 | The following jobs were incomplete: set([<Future at 0x7f927d9fdb10 state=finished returned NoneType>]) | 17:22 |
weshay | 2019-08-19 03:43:50 | ova-serialproxy/nova-serialproxy-build.log -t 127.0.0.1:8787/tripleomaster/centos-binary-nova-serialproxy:6cd9e55d69e90ea73219abced4e9a3f2372b204e_10e135ca /tmp/kolla-SRgmFH/docker/nova/nova-serialproxy | 17:22 |
weshay | 2019-08-19 03:43:50 | Building nova-api image with: sudo buildah bud --tls-verify=False --logfile /tmp/kolla-SRgmFH/docker/nova/nova-api/nova-api-build.log -t 127.0.0.1:8787/tripleomaster/centos-binary-nova-api:6cd9e55d69e90ea73219abced4e9a3f2372b204e_10e135ca /tmp/kolla-SRgmFH/docker/nova/nova-api | 17:22 |
*** kopecmartin is now known as kopecmartin|off | 17:33 | |
*** pierreprinetti has quit IRC | 18:15 | |
*** Vorrtex has quit IRC | 18:15 | |
rlandy|ruck | weshay: what column does this card move to? https://trello.com/c/rPZGwJmU/1081-cixlp1839532tripleociproa-tripleo-gate-jobs-are-failing-to-pull-containers-when-running-on-ovh-provider-with-unauthorized-error | 18:29 |
rlandy|ruck | weshay: slow - pls vote on https://review.rdoproject.org/r/#/c/21875/ - ci results are looking promising - will merge when ci completes | 18:30 |
rlandy|ruck | thanks | 18:32 |
rlandy|ruck | saw it move | 18:32 |
weshay | np | 18:34 |
*** aakarsh has quit IRC | 18:35 | |
rlandy|ruck | zbr: thanks for merging https://review.rdoproject.org/r/#/c/21856/ - did break the rrcockpit :) | 18:36 |
rlandy|ruck | weshay: ^^ no worries to test | 18:36 |
rlandy|ruck | already merged | 18:36 |
weshay | cool.. the prod cockpit looks good.. nice job rlandy|ruck zbr | 18:37 |
*** aakarsh has joined #oooq | 18:39 | |
*** saneax has quit IRC | 18:47 | |
*** pierreprinetti has joined #oooq | 18:55 | |
rlandy|ruck | weshay: zbr: what do you think of increasing deploy timeout for https://bugs.launchpad.net/tripleo/+bug/1840616? | 19:15 |
openstack | Launchpad bug 1840616 in tripleo "Master/stein check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy) | 19:15 |
weshay | rlandy|ruck I don't think we can make a case for a timeout bug w/o trying it locally | 19:16 |
rlandy|ruck | weshay: yeah - I know it's a workaround | 19:17 |
rlandy|ruck | failure is sporadic | 19:17 |
weshay | rlandy|ruck I would add alert so it's on our radar | 19:18 |
weshay | but that's it.. wouldnt cix it yet | 19:18 |
rlandy|ruck | ci and alert? | 19:18 |
rlandy|ruck | will look a little more into it | 19:19 |
*** sshnaidm|bbl is now known as sshnaidm | 19:19 | |
*** dsneddon has quit IRC | 19:20 | |
*** dtrainor has quit IRC | 19:21 | |
sshnaidm | zbr, please don't change anything in rhel containers patches, I'm testing the current version | 19:21 |
*** dtrainor has joined #oooq | 19:23 | |
rlandy|ruck | https://review.opendev.org/#/c/677063 will kill us | 19:23 |
rlandy|ruck | cant get it through zuul :( | 19:23 |
*** dsneddon has joined #oooq | 19:26 | |
rlandy|ruck | zbr: pls vote on https://review.rdoproject.org/r/#/c/21875/ so I don't just self merge | 19:29 |
rlandy|ruck | anybody: ^^ | 19:31 |
*** pierreprinetti has quit IRC | 19:33 | |
*** saneax has joined #oooq | 19:36 | |
rlandy|ruck | weshay: rfolco: ^^ anyone else vote here? required change to get rid of periodic post failure | 19:43 |
weshay | k..s ec | 19:44 |
weshay | sec | 19:44 |
weshay | rlandy|ruck done | 19:44 |
weshay | thanks | 19:44 |
rlandy|ruck | thank you | 19:46 |
rfolco | wes hit the button first | 19:46 |
rlandy|ruck | one less failure hopefully | 19:47 |
rlandy|ruck | rfolco: know who has +2 on openstack/kolla- https://review.opendev.org/#/c/677212/ | 19:48 |
*** sanjayu_ has joined #oooq | 19:53 | |
*** saneax has quit IRC | 19:56 | |
*** sanjayu_ has quit IRC | 19:57 | |
jjoyce | rlandy|ruck: Do you happen to have a link handy to the docker register setup playbook? | 20:00 |
rlandy|ruck | jjoyce: on the undercloud? | 20:01 |
rfolco | rlandy|ruck, I don't know | 20:01 |
rlandy|ruck | or a regular public registry? | 20:01 |
jjoyce | Just a regular registry | 20:02 |
*** pierreprinetti has joined #oooq | 20:02 | |
rlandy|ruck | jjoyce: we have a couple pieces of it depending on deployment ... https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/container-prep , https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/containers-prepare-parameter.yaml.j2 | 20:06 |
rlandy|ruck | then there is if you are working standalone | 20:06 |
rlandy|ruck | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/tasks/main.yml-> overcloud | 20:07 |
rlandy|ruck | I am not sure I am answering the questions | 20:07 |
rlandy|ruck | mostly this is done in the product itself | 20:08 |
rlandy|ruck | we just pass parameters | 20:08 |
jjoyce | I was looking for something that just installs all the pieces needed for a local registry | 20:08 |
jjoyce | I know Wes said that their were some playbooks that did that. | 20:09 |
*** pierreprinetti has quit IRC | 20:09 | |
rlandy|ruck | weshay: we do? | 20:10 |
weshay | jjoyce rlandy|ruck ya.. the container check job installs a registry.. /me gets | 20:11 |
rlandy|ruck | oh that | 20:11 |
* rlandy|ruck was looking in deployment | 20:11 | |
weshay | jjoyce rlandy|ruck https://github.com/openstack/ansible-role-container-registry | 20:12 |
jjoyce | weshay: Perfect! Just what I was looking for. | 20:14 |
weshay | jjoyce I've been looking for you for all my life too!!! | 20:15 |
weshay | I found my soul mate | 20:15 |
jjoyce | weshay: <3 | 20:15 |
jjoyce | :-) | 20:15 |
weshay | :) | 20:15 |
weshay | rlandy|ruck https://review.rdoproject.org/r/#/c/21884/ | 20:39 |
rlandy|ruck | weshay: oh - bless you - finally | 20:55 |
rlandy|ruck | ugh - stein - one stupid tempest test - so close | 20:59 |
*** aakarsh has quit IRC | 21:10 | |
*** sshnaidm is now known as sshnaidm|afk | 21:24 | |
rlandy|ruck | ugh - we have t get this patch through or disable image builds | 22:01 |
rlandy|ruck | tripleo-buildimage-overcloud-full-centos-7 | 22:03 |
*** aakarsh has joined #oooq | 22:35 | |
rlandy|ruck | https://review.opendev.org/#/c/677063/ keeps failing https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_63/677063/8/check/tripleo-ci-centos-7-containers-multinode/dc1abe2/ | 22:43 |
rlandy|ruck | weshay: hello | 22:43 |
rlandy|ruck | https://review.opendev.org/#/c/677063/ keeps failing some check/gate | 22:43 |
rlandy|ruck | this is supposed to clear up the image build failure | 22:44 |
rlandy|ruck | also this error is failing other jobs on gates | 22:44 |
rlandy|ruck | we have to option to get the patch pushed through by infra cores | 22:44 |
rlandy|ruck | thoughts? | 22:44 |
*** rlandy|ruck is now known as rlandy|ruck|bbl | 22:52 | |
*** d0ugal has quit IRC | 23:25 | |
*** d0ugal has joined #oooq | 23:41 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!