| *** goldyfruit_ has joined #openstack-infra | 00:00 | |
| *** markvoelker has quit IRC | 00:02 | |
| *** aaronsheffield has quit IRC | 00:07 | |
| *** xenos76 has quit IRC | 00:07 | |
| *** gyee has quit IRC | 00:14 | |
| *** jamesmcarthur has joined #openstack-infra | 00:21 | |
| *** jamesmcarthur has quit IRC | 00:23 | |
| *** jamesmcarthur has joined #openstack-infra | 00:23 | |
| *** rkukura has quit IRC | 00:25 | |
| *** whoami-rajat has joined #openstack-infra | 00:27 | |
| openstackgerrit | Merged opendev/system-config master: Fix files matcher and bazel for gerrit base image https://review.opendev.org/682449 | 00:42 |
|---|---|---|
| *** masayukig has joined #openstack-infra | 00:47 | |
| *** hongbin has joined #openstack-infra | 01:03 | |
| *** jamesmcarthur has quit IRC | 01:05 | |
| *** igordc has quit IRC | 01:08 | |
| *** mattw4 has quit IRC | 01:09 | |
| *** darvon has quit IRC | 01:23 | |
| *** darvon has joined #openstack-infra | 01:29 | |
| *** Goneri has quit IRC | 01:31 | |
| *** markvoelker has joined #openstack-infra | 02:03 | |
| *** markvoelker has quit IRC | 02:08 | |
| *** apetrich has quit IRC | 02:11 | |
| *** lxkong has joined #openstack-infra | 02:16 | |
| lxkong | hi there, i'm adding a new CI job (based on publish-openstack-artifacts), when I use `check experimental` an error occurs: `Unable to freeze job graph: Pre-review pipeline experimental does not allow post-review job ...`, can anybody give me some hints please? | 02:18 |
| lxkong | an unrelated question, given we have several jobs in the experimental, how can i just trigger specific one of them? | 02:19 |
| clarkb | for the first thing the job is marked as post revuew likely because it uses a secret | 02:25 |
| clarkb | this meansit can inly be triggered afterbeing approved by humans | 02:25 |
| clarkb | to avoid exposing the secret | 02:26 |
| clarkb | for the second thing you can push a patchset that temporarily removes the other experimental jobs | 02:26 |
| clarkb | but in general there is no way to trigger a specific job.It is basedon the pipeline selection and file matchers | 02:26 |
| lxkong | clarkb: thanks for the answer. So do you mean only after a patch is merged that the job could be triggered manually? | 02:27 |
| *** rfolco has quit IRC | 02:28 | |
| *** ykarel|away has joined #openstack-infra | 02:28 | |
| *** whoami-rajat has quit IRC | 02:29 | |
| clarkb | no it means that a job that uses secrets like that can only run post review. In our case the gate, promote, post, tag, release pipelines are all post review | 02:29 |
| clarkb | otherwise you would be able to push a change that exposes the secret then run the job yo do so | 02:30 |
| clarkb | *to do so | 02:30 |
| lxkong | clarkb: cool, i see, thanks | 02:33 |
| *** roman_g has quit IRC | 02:33 | |
| *** jamesmcarthur has joined #openstack-infra | 02:38 | |
| *** persia has quit IRC | 02:50 | |
| *** rajinir has quit IRC | 02:50 | |
| *** gsantomaggio has quit IRC | 02:50 | |
| *** rajinir has joined #openstack-infra | 02:56 | |
| *** persia has joined #openstack-infra | 02:56 | |
| *** gsantomaggio has joined #openstack-infra | 02:56 | |
| *** jamesmcarthur has quit IRC | 02:57 | |
| *** dave-mccowan has quit IRC | 03:00 | |
| *** armax has joined #openstack-infra | 03:05 | |
| *** persia has quit IRC | 03:18 | |
| *** rajinir has quit IRC | 03:18 | |
| *** gsantomaggio has quit IRC | 03:18 | |
| *** setuid is now known as zzZzzZZetuid | 03:19 | |
| *** psachin has joined #openstack-infra | 03:22 | |
| *** ricolin has joined #openstack-infra | 03:22 | |
| *** rajinir has joined #openstack-infra | 03:24 | |
| *** persia has joined #openstack-infra | 03:24 | |
| *** gsantomaggio has joined #openstack-infra | 03:24 | |
| *** ramishra has joined #openstack-infra | 03:33 | |
| *** jamesmcarthur has joined #openstack-infra | 03:56 | |
| *** jamesmcarthur has quit IRC | 03:56 | |
| *** jamesmcarthur has joined #openstack-infra | 03:57 | |
| *** rmcallis has quit IRC | 03:59 | |
| *** hongbin has quit IRC | 04:01 | |
| *** rh-jelabarre has quit IRC | 04:04 | |
| *** armax has quit IRC | 04:04 | |
| *** hongbin has joined #openstack-infra | 04:08 | |
| *** hongbin has quit IRC | 04:10 | |
| *** njohnston has quit IRC | 04:30 | |
| *** ociuhandu has joined #openstack-infra | 04:31 | |
| *** ociuhandu has quit IRC | 04:35 | |
| *** lajoskatona has joined #openstack-infra | 04:43 | |
| AJaeger | config-core, please review these changes to convert some more jobs to promote, it will make moving from static to AFS easier: https://review.opendev.org/#/q/status:open++topic:promote-project-config | 04:48 |
| *** ykarel|away is now known as ykarel | 04:49 | |
| *** jamesmcarthur has quit IRC | 04:51 | |
| *** udesale has joined #openstack-infra | 05:01 | |
| *** soniya29 has joined #openstack-infra | 05:13 | |
| *** pcaruana has joined #openstack-infra | 05:16 | |
| *** ykarel has quit IRC | 05:18 | |
| *** goldyfruit_ has quit IRC | 05:19 | |
| *** jamesmcarthur has joined #openstack-infra | 05:21 | |
| *** kjackal has joined #openstack-infra | 05:22 | |
| *** pcaruana has quit IRC | 05:29 | |
| *** sshnaidm|afk is now known as sshnaidm|pto | 05:35 | |
| *** dciabrin_ has joined #openstack-infra | 05:38 | |
| *** zhurong has joined #openstack-infra | 05:38 | |
| *** ykarel has joined #openstack-infra | 05:38 | |
| openstackgerrit | Sundar Nadathur proposed openstack/project-config master: Retiring project os-acc. https://review.opendev.org/682544 | 05:40 |
| *** dciabrin has quit IRC | 05:41 | |
| openstackgerrit | Sundar Nadathur proposed openstack/project-config master: Retiring project os-acc: Step 4/5. https://review.opendev.org/682544 | 05:51 |
| *** AJaeger has quit IRC | 05:52 | |
| *** Tengu has joined #openstack-infra | 05:53 | |
| *** iokiwi has quit IRC | 05:54 | |
| *** adriant has quit IRC | 05:54 | |
| *** spsurya has joined #openstack-infra | 05:55 | |
| openstackgerrit | Sundar Nadathur proposed openstack/project-config master: Retiring project os-acc. https://review.opendev.org/682544 | 05:55 |
| *** AJaeger has joined #openstack-infra | 05:56 | |
| *** diablo_rojo has joined #openstack-infra | 06:04 | |
| AJaeger | config-core, https://review.opendev.org/#/c/676331/ is ready for review, please | 06:08 |
| *** larainema has joined #openstack-infra | 06:09 | |
| *** jamesmcarthur has quit IRC | 06:10 | |
| *** kjackal has quit IRC | 06:21 | |
| *** jtomasek has joined #openstack-infra | 06:23 | |
| *** pgaxatte has joined #openstack-infra | 06:24 | |
| *** slaweq has joined #openstack-infra | 06:26 | |
| *** eandersson has quit IRC | 06:28 | |
| *** kopecmartin|off is now known as kopecmartin | 06:31 | |
| *** jamesmcarthur has joined #openstack-infra | 06:32 | |
| *** jamesmcarthur has quit IRC | 06:37 | |
| *** eandersson has joined #openstack-infra | 06:38 | |
| *** jamesmcarthur has joined #openstack-infra | 06:45 | |
| *** markvoelker has joined #openstack-infra | 06:47 | |
| *** markvoelker has quit IRC | 06:52 | |
| *** xek has joined #openstack-infra | 06:57 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: [dnm] testing python3 ansible https://review.opendev.org/682556 | 06:57 |
| *** efried has quit IRC | 07:00 | |
| *** efried has joined #openstack-infra | 07:01 | |
| *** xarses has quit IRC | 07:06 | |
| *** xarses has joined #openstack-infra | 07:07 | |
| *** xek has quit IRC | 07:08 | |
| *** jbadiapa has quit IRC | 07:08 | |
| *** kjackal has joined #openstack-infra | 07:10 | |
| *** diablo_rojo has quit IRC | 07:10 | |
| *** kjackal has quit IRC | 07:11 | |
| *** jamesmcarthur has quit IRC | 07:17 | |
| *** diablo_rojo has joined #openstack-infra | 07:20 | |
| *** kjackal has joined #openstack-infra | 07:20 | |
| *** trident has quit IRC | 07:22 | |
| *** jbadiapa has joined #openstack-infra | 07:24 | |
| *** tosky has joined #openstack-infra | 07:27 | |
| *** diablo_rojo has quit IRC | 07:27 | |
| *** kjackal_v2 has joined #openstack-infra | 07:28 | |
| *** rpittau|afk is now known as rpittau | 07:28 | |
| *** kjackal has quit IRC | 07:28 | |
| *** trident has joined #openstack-infra | 07:31 | |
| *** ralonsoh has joined #openstack-infra | 07:33 | |
| *** trident has quit IRC | 07:36 | |
| *** apetrich has joined #openstack-infra | 07:40 | |
| *** priteau has joined #openstack-infra | 07:43 | |
| *** slaweq has quit IRC | 07:43 | |
| *** jamesmcarthur has joined #openstack-infra | 07:44 | |
| *** pkopec has joined #openstack-infra | 07:45 | |
| *** trident has joined #openstack-infra | 07:46 | |
| *** jpena|off is now known as jpena | 07:46 | |
| yoctozepto | hey infra, have you ever mirrored something from obs repos like http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/ | 07:46 |
| *** rcernin has quit IRC | 07:47 | |
| yoctozepto | we would like to mirror this particular repo because it timeouts for us | 07:47 |
| *** ykarel is now known as ykarel|lunch | 07:48 | |
| *** iurygregory has joined #openstack-infra | 07:48 | |
| *** jamesmcarthur has quit IRC | 07:50 | |
| *** jamesmcarthur has joined #openstack-infra | 07:52 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks and static dirs https://review.opendev.org/678875 | 07:52 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 07:52 |
| *** slaweq has joined #openstack-infra | 07:52 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Fedora 30 : add python-path for Python 3 https://review.opendev.org/682569 | 07:58 |
| ianw | yoctozepto: could add it to https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/opensuse-mirror-update ? | 08:00 |
| yoctozepto | ianw: thanks, just hit the same file myself :D | 08:01 |
| yoctozepto | must be the right one | 08:01 |
| openstackgerrit | Merged openstack/project-config master: End gating for os-acc as that project is about to be retired. https://review.opendev.org/676331 | 08:02 |
| *** tkajinam has quit IRC | 08:03 | |
| *** iurygregory has quit IRC | 08:04 | |
| *** kaiokmo has quit IRC | 08:04 | |
| *** lpetrut has joined #openstack-infra | 08:07 | |
| *** lpetrut has quit IRC | 08:08 | |
| frickler | ianw: IIUC https://review.opendev.org/682569 should also fix the devstack issues with F30, if it passes tests I'm going to fast-approve so it can make it into today's image build. Or maybe AJaeger can take a look, too | 08:11 |
| *** dchen has quit IRC | 08:13 | |
| AJaeger | ianwm, frickler, approved 682569 | 08:13 |
| *** kaiokmo has joined #openstack-infra | 08:13 | |
| frickler | AJaeger: oh, zuul verified in sync with my vote, I wouldn't have noticed that in a while, thx. | 08:15 |
| *** jamesmcarthur has quit IRC | 08:15 | |
| *** iurygregory has joined #openstack-infra | 08:16 | |
| *** jamesmcarthur has joined #openstack-infra | 08:16 | |
| AJaeger | mordred, frickler , want to review https://review.opendev.org/680901 and https://review.opendev.org/680703 as well, please? | 08:17 |
| mgoddard | AJaeger: all backports now proposed. Rocky & earlier weren't clean backports as CI code has changed, and had to attend to $life. Thanks again | 08:18 |
| *** zhangfei has joined #openstack-infra | 08:18 | |
| *** igordc has joined #openstack-infra | 08:20 | |
| openstackgerrit | Merged openstack/project-config master: Fedora 30 : add python-path for Python 3 https://review.opendev.org/682569 | 08:24 |
| openstackgerrit | Radosław Piliszek proposed opendev/system-config master: Mirror OBS ha-clustering repo for CentOS 7 https://review.opendev.org/682576 | 08:26 |
| yoctozepto | ianw: ^ | 08:27 |
| *** igordc has quit IRC | 08:28 | |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Fix release ACL for Kayobe https://review.opendev.org/682581 | 08:30 |
| AJaeger | mgoddard: great, thanks! | 08:31 |
| openstackgerrit | Merged opendev/irc-meetings master: Update UC meeting time and day https://review.opendev.org/682422 | 08:31 |
| AJaeger | mgoddard: FYI, see 682581 ^ | 08:31 |
| mgoddard | AJaeger: ack | 08:33 |
| *** derekh has joined #openstack-infra | 08:33 | |
| *** xenos76 has joined #openstack-infra | 08:38 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add a netconsole role https://review.opendev.org/680901 | 08:38 |
| *** e0ne has joined #openstack-infra | 08:41 | |
| *** gfidente has joined #openstack-infra | 08:44 | |
| *** slaweq has quit IRC | 08:52 | |
| *** kjackal_v2 has quit IRC | 09:00 | |
| mgoddard | Hello, I'd like to set up a redirect from #openstack-kayobe to #openstack-kolla. I was following https://docs.openstack.org/infra/system-config/irc.html#renaming-an-irc-channel but don't have the necessary on #openstack-kolla | 09:03 |
| *** pkopec has quit IRC | 09:04 | |
| *** ykarel|lunch is now known as ykarel | 09:08 | |
| *** slaweq has joined #openstack-infra | 09:10 | |
| *** pcaruana has joined #openstack-infra | 09:19 | |
| openstackgerrit | Mark Goddard proposed openstack/project-config master: Remove accessbot from #openstack-kayobe https://review.opendev.org/682591 | 09:20 |
| *** uberjay has quit IRC | 09:28 | |
| *** uberjay has joined #openstack-infra | 09:28 | |
| *** roman_g has joined #openstack-infra | 09:30 | |
| *** jamesmcarthur has quit IRC | 09:31 | |
| *** jamesmcarthur has joined #openstack-infra | 09:32 | |
| *** zhangfei has quit IRC | 09:46 | |
| frickler | mgoddard: I have executed the mlock command for you, IIUC you should be able to do everything else with your access to the old channel | 09:48 |
| mgoddard | frickler: thanks, I'll give it a try | 09:48 |
| AJaeger | frickler: we need to remove the channel from system-config first AFAIK! | 09:49 |
| AJaeger | mgoddard: can you send a patch for opendev/system-config as well, please? | 09:49 |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: [dnm] testing python3 ansible https://review.opendev.org/682556 | 09:50 |
| mgoddard | AJaeger: +1 - computer says no | 09:50 |
| mgoddard | I'll send a patch | 09:51 |
| AJaeger | thanks | 09:53 |
| openstackgerrit | Mark Goddard proposed opendev/system-config master: Remove #openstack-kayobe IRC channel https://review.opendev.org/682595 | 09:54 |
| frickler | infra-root: we seem to have issues with ubuntu mirrors again, see e.g. https://4714dd3447917085c579-aa35bee35130c30199865e40021a7f50.ssl.cf2.rackcdn.com/682593/1/check/openstack-tox-pep8/529f37b/job-output.txt | 09:54 |
| frickler | lots of gate failures, too | 09:54 |
| *** armax has joined #openstack-infra | 09:58 | |
| *** armax has quit IRC | 10:02 | |
| *** zhangfei has joined #openstack-infra | 10:02 | |
| ianw | frickler: it looks like we're synced (as of an hour ago) so is it upstream? | 10:05 |
| *** ociuhandu has joined #openstack-infra | 10:05 | |
| *** ramishra_ has joined #openstack-infra | 10:07 | |
| *** jamesmcarthur has quit IRC | 10:07 | |
| *** pcaruana has quit IRC | 10:07 | |
| *** dtantsur|afk is now known as dtantsur | 10:07 | |
| *** ramishra has quit IRC | 10:08 | |
| frickler | ianw: I would be thinking that reprepo would rebuild the index, so that type of error shouldn't be able to be transferred from upstream | 10:10 |
| *** pgaxatte has quit IRC | 10:13 | |
| dtantsur | hi folks, do you know what this means https://c34066b9a144c60378b7-045c6253c589ccba85304c02bb42bfce.ssl.cf1.rackcdn.com/680404/2/gate/ironic-standalone/16afa43/job-output.txt ? | 10:14 |
| dtantsur | [Zuul] Log Stream did not terminate | 10:14 |
| jrosser | looks like the suse mirror might be broken too https://a3fd8c27c0382f01ec6b-02618f010e74d581319c83aa0d27e1a8.ssl.cf5.rackcdn.com/682587/2/check/openstack-ansible-deploy-aio_metal-opensuse-15/bbe3f6b/job-output.txt | 10:17 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Install jeepyb into the gerrit images https://review.opendev.org/682601 | 10:23 |
| frickler | ianw: it looks like an afs mirror issue. /afs/openstack.org/mirror/ubuntu/dists/bionic-updates/Release has old content on e.g. mirror01.bhs1.ovh.openstack.org, although /afs/openstack.org/mirror/ubuntu/timestamp.txt is up to date | 10:23 |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: [dnm] testing python3 ansible https://review.opendev.org/682556 | 10:27 |
| ianw | frickler: hrrrm, in recent history did clarkb have to do some cache flushing at some point? | 10:28 |
| *** zzZzzZZetuid is now known as setuid | 10:29 | |
| frickler | ianw: not that I'd have noticed. also nothing to be found in infra status afaict | 10:30 |
| *** udesale has quit IRC | 10:34 | |
| *** udesale has joined #openstack-infra | 10:34 | |
| ianw | frickler: around http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-06-05.log.html#t2019-06-05T00:01:29 | 10:34 |
| ianw | talking about "fs flush" and "fs flushvolume" i think | 10:35 |
| ianw | sorry have to head off, but i think that was similar out-of-sync issues ... | 10:35 |
| frickler | ianw: thx, goodnight to you | 10:36 |
| yoctozepto | confirming mirror.iad.rax.openstack.org and mirror.bhs1.ovh.openstack.org have issues with the ubuntu repo | 10:37 |
| yoctozepto | killing my precious contribution to mirror new repo and many kolla jobs right now | 10:38 |
| *** jamesmcarthur has joined #openstack-infra | 10:40 | |
| *** prometheanfire has quit IRC | 10:41 | |
| *** prometheanfire has joined #openstack-infra | 10:43 | |
| *** armax has joined #openstack-infra | 10:46 | |
| frickler | so I copied clarkb's "fs flush" commands from mirror.dfw.rax and executed them on the ovh mirror and at least the Release file looks correct now | 10:46 |
| frickler | I then tried to the same one iad.rax but the "fs" command wasn't available. I tried to "apt install openafs-client" but that got stuck with some dkms build, so I may have broken that mirror even more now. the fs command now exists but claims flushing a file not to be implemented :-/ | 10:49 |
| *** zhangfei has quit IRC | 10:50 | |
| frickler | that's probably because that's a kafs node | 10:50 |
| *** larainema has quit IRC | 10:54 | |
| *** pcaruana has joined #openstack-infra | 10:55 | |
| mordred | frickler: oh good - I was just reading scrollback | 10:55 |
| mordred | frickler: oh right - the kafs. should we pull iad.rax out of nodepool for now? then maybe we should reboot that node? /me isn't sure what the right things to do on the kafs node are | 10:57 |
| openstackgerrit | Pierre Riteau proposed openstack/project-config master: Enable Review-Priority flag for Blazar https://review.opendev.org/682613 | 10:57 |
| AJaeger | mordred, frickler , can we get https://review.opendev.org/682595 (system-config: remove #openstack-kayobe channel) os that it gets decommissioned? | 10:58 |
| mordred | AJaeger: do we need to do something to put in a redirect after we do that? | 10:59 |
| AJaeger | mordred: I think frickler was helping mgoddard with that - but this change should be fine as is AFAIK | 11:00 |
| mordred | kk | 11:01 |
| *** goldyfruit_ has joined #openstack-infra | 11:07 | |
| frickler | mordred: me neither, I was fearing it might come up non-working after a reboot. I was trying to check more logs to see what files are affected there | 11:07 |
| *** lucasagomes has joined #openstack-infra | 11:08 | |
| mgoddard | One of my reviews seems to have disappeared: https://review.opendev.org/#/c/681700/ | 11:12 |
| mgoddard | 11:12 | |
| mgoddard | The page you requested was not found, or you do not have permission to view this page. | 11:12 |
| *** jamesmcarthur has quit IRC | 11:12 | |
| frickler | mgoddard: that page works fine for me, maybe you need to login again to gerrit? | 11:13 |
| frickler | mordred: I installed the redirect for the kayobe channel earlier, so with the above review merged everything should be fine in that regard | 11:14 |
| mgoddard | oh excellent, thanks frickler | 11:17 |
| *** jpena is now known as jpena|lunch | 11:17 | |
| mgoddard | frickler: logout/in didn't help | 11:17 |
| yoctozepto | https://review.opendev.org/682576 <- anyone to give w+1? :-) | 11:20 |
| frickler | mgoddard: hmm, but it only affects that review? | 11:20 |
| yoctozepto | mgoddard: it works for me (TM) | 11:21 |
| mgoddard | frickler: that I know of. I can see others fine | 11:21 |
| AJaeger | mgoddard: I'm really puzzled, there shouldn't be any access restrictions | 11:27 |
| mgoddard | :) | 11:27 |
| AJaeger | mgoddard: it works for me both logged in and logged out (two different browser) | 11:28 |
| mordred | I can also see mgoddard's change | 11:28 |
| * AJaeger adds a recheck as proof ;) | 11:28 | |
| mgoddard | I can see it when logged out :/ | 11:28 |
| mordred | WEIRD | 11:28 |
| mordred | frickler: so maybe we should just disable iad until we can figure out its mirror? | 11:29 |
| mgoddard | weirder: I logged back in from that page, and can see it now | 11:29 |
| openstackgerrit | Merged openstack/project-config master: Remove accessbot from #openstack-kayobe https://review.opendev.org/682591 | 11:29 |
| frickler | mordred: I don't see any further failures currently, the outdated file now seems current on the iad mirror, too. so I'd tend to keep things running unless someone sees further errors after I did the fs flush, i.e. after 11:00 | 11:30 |
| mordred | frickler: oh! good | 11:32 |
| frickler | mgoddard: hmm, possibly some transient error in gerrit that got cached by your browser | 11:33 |
| yoctozepto | thanks frickler | 11:36 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Install jeepyb into the gerrit images https://review.opendev.org/682601 | 11:37 |
| openstackgerrit | Merged opendev/system-config master: Remove #openstack-kayobe IRC channel https://review.opendev.org/682595 | 11:39 |
| *** panda is now known as panda|ruck | 11:41 | |
| *** takamatsu has joined #openstack-infra | 11:52 | |
| openstackgerrit | Merged opendev/system-config master: Mirror OBS ha-clustering repo for CentOS 7 https://review.opendev.org/682576 | 11:58 |
| *** armax has quit IRC | 11:58 | |
| *** goldyfruit_ has quit IRC | 12:02 | |
| *** armax has joined #openstack-infra | 12:03 | |
| *** rfolco has joined #openstack-infra | 12:04 | |
| *** gfidente has quit IRC | 12:05 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 12:07 |
| *** jamesmcarthur has joined #openstack-infra | 12:10 | |
| *** jamesmcarthur has quit IRC | 12:10 | |
| *** jamesmcarthur_ has joined #openstack-infra | 12:10 | |
| *** markvoelker has joined #openstack-infra | 12:11 | |
| *** rlandy has joined #openstack-infra | 12:18 | |
| *** rh-jelabarre has joined #openstack-infra | 12:22 | |
| *** dave-mccowan has joined #openstack-infra | 12:25 | |
| *** jamesmcarthur_ has quit IRC | 12:26 | |
| *** soniya29 has quit IRC | 12:26 | |
| *** pcaruana has quit IRC | 12:27 | |
| *** openstackstatus has quit IRC | 12:28 | |
| *** openstack has joined #openstack-infra | 12:29 | |
| *** ChanServ sets mode: +o openstack | 12:29 | |
| *** aaronsheffield has joined #openstack-infra | 12:30 | |
| *** xenos76 has joined #openstack-infra | 12:30 | |
| *** gfidente has joined #openstack-infra | 12:32 | |
| *** AJaeger has quit IRC | 12:34 | |
| *** AJaeger has joined #openstack-infra | 12:36 | |
| *** jpena|lunch is now known as jpena | 12:37 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Install jeepyb into the gerrit images https://review.opendev.org/682601 | 12:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks and static dirs https://review.opendev.org/678875 | 12:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 12:39 |
| *** Goneri has joined #openstack-infra | 12:39 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/diskimage-builder master: Update gentoo test to current system profile https://review.opendev.org/682635 | 12:43 |
| frickler | prometheanfire: ^^ we also need to update project-config according to https://review.opendev.org/677290 as builds with 17.0 are failing. do you want to change the name to gentoo-17-1-systemd everywhere or maybe go for a generic gentoo-17-systemd instead? | 12:46 |
| frickler | see https://nb01.openstack.org/gentoo-17-0-systemd-0000081223.log for the failing build | 12:47 |
| *** fdegir has quit IRC | 12:47 | |
| *** georgk has quit IRC | 12:47 | |
| *** georgk has joined #openstack-infra | 12:48 | |
| *** fdegir has joined #openstack-infra | 12:48 | |
| frickler | suse ppl: opensuse-423 builds are also failing https://nb01.openstack.org/opensuse-423-0000061022.log | 12:54 |
| frickler | and debian-buster https://nb01.openstack.org/debian-buster-0000025017.log | 12:54 |
| *** jcoufal has joined #openstack-infra | 12:55 | |
| frickler | I'm also probably not up to date with our build schedule, I was assuming one build per image being started around 12 UTC-ish, but all the above failing builds are happening in very frequent succession | 12:56 |
| *** redrobot has joined #openstack-infra | 12:56 | |
| *** lajoskatona has quit IRC | 12:57 | |
| *** pcaruana has joined #openstack-infra | 12:59 | |
| AJaeger | frickler: dirk and evrardjp have been trying to stop using openSUSE 42.3 - but it should still build, dirk recently fixed a issue AFAIR | 12:59 |
| frickler | AJaeger: the latest 8 builds that I can find, all of them ran today in roughly hourly intervals, are showing the same failure | 13:01 |
| *** pgaxatte has joined #openstack-infra | 13:03 | |
| frickler | tumbleweed doesn't look much better, either https://nb02.openstack.org/opensuse-tumbleweed-0000087972.log | 13:03 |
| *** mriedem has joined #openstack-infra | 13:05 | |
| fungi | mordred: frickler: i'm still catching up and not sure if you've hit on this detail yet, but there are no kafs mirrors in use. check the other domain name | 13:05 |
| AJaeger | frickler: hope dirk or evrardjp can help. Thanks for the info | 13:05 |
| evrardjp | hey. I am still removing 42.3 in some places when I have time indeed. Tumbleweed is more a dirk thing :p | 13:06 |
| evrardjp | I will query him and see what's wrong with SuSEfirewall2. | 13:08 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/project-config master: Update gentoo image to current profile https://review.opendev.org/682639 | 13:08 |
| frickler | prometheanfire: ^^ that would be the minimal change to get the builds working, but leaves the "wrong" image name | 13:08 |
| *** goldyfruit_ has joined #openstack-infra | 13:09 | |
| *** sthussey has joined #openstack-infra | 13:13 | |
| frickler | fungi: ah, good point, I was looking on opendev.org instead of openstack.org. but I also haven't seen any fresh failures, so either the "fs flush" on ovh fixed things for rax, too, or the problem went away after some time on its own | 13:19 |
| *** yamamoto has joined #openstack-infra | 13:20 | |
| frickler | *sigh* es also is not quite up to date: Last Elastic Search Index Update: Sat Sep 14 2019 14:00:09 GMT+0000 (UTC) | 13:21 |
| *** eharney has joined #openstack-infra | 13:24 | |
| *** pcaruana has quit IRC | 13:28 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - add support for git.tag.creation event https://review.opendev.org/679938 | 13:33 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - add support for git.tag.creation event https://review.opendev.org/679938 | 13:37 |
| *** goldyfruit_ has quit IRC | 13:39 | |
| *** psachin has quit IRC | 13:43 | |
| *** armax has quit IRC | 13:43 | |
| *** jamesmcarthur has quit IRC | 13:47 | |
| *** xek has joined #openstack-infra | 13:56 | |
| *** ykarel is now known as ykarel|afk | 13:56 | |
| *** armax has joined #openstack-infra | 13:58 | |
| *** goldyfruit_ has joined #openstack-infra | 13:59 | |
| *** pcaruana has joined #openstack-infra | 13:59 | |
| *** michael-beaver has joined #openstack-infra | 14:00 | |
| clarkb | frickler: I think the problem does go away eventually on its own :/ | 14:03 |
| clarkb | re afs | 14:03 |
| clarkb | also I think we thought part of the problem was ubuntus older afs package but we install from our ppa now J guess it still affects that version | 14:04 |
| clarkb | mgoddard: my guess is you cached data in your browser for that change that pointed at old stuff pre rename | 14:07 |
| clarkb | mgoddard: everyone that hasnt cached that data loads it up cleanly with new name data | 14:07 |
| clarkb | might try clearing your browser caches? | 14:07 |
| mgoddard | clarkb: sounds plausible | 14:08 |
| mgoddard | it worked after a second logout/in, possibly I didn't do a full refresh the first time | 14:08 |
| *** larainema has joined #openstack-infra | 14:09 | |
| *** rpittau is now known as rpittau|afk | 14:10 | |
| *** whoami-rajat has joined #openstack-infra | 14:12 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Disable rsh synchronize rsync_opts https://review.opendev.org/682657 | 14:12 |
| mordred | mgoddard: have you tried turning it off and back on again? | 14:17 |
| *** jamesmcarthur has joined #openstack-infra | 14:17 | |
| *** goldyfruit___ has joined #openstack-infra | 14:17 | |
| mgoddard | mordred: I see you have attended computer school | 14:17 |
| *** goldyfruit_ has quit IRC | 14:20 | |
| mordred | mgoddard: sadly I'm also old enough to have been admin on a flaky sun box that had to be reinstalled if it powered off. | 14:21 |
| mordred | talk about stateless compute ... | 14:22 |
| mgoddard | diskless ftw | 14:22 |
| fungi | mordred: best part about those was that they'd drop to the firmware prompt if you issued a break over the serial line | 14:22 |
| *** jamesmcarthur has quit IRC | 14:22 | |
| * fungi accidentally stopped two full racks of them once by rebooting a serial concentrator | 14:23 | |
| mordred | whoops | 14:24 |
| fungi | all my best data center stories are about the disasters i've inadvertently triggered | 14:25 |
| *** lbragstad_ has joined #openstack-infra | 14:26 | |
| fungi | you could issue a `go` command at the firmware prompt and sunos (or later, solaris) would resume, but a lot of the software didn't really recover | 14:26 |
| *** jamesmcarthur has joined #openstack-infra | 14:27 | |
| fungi | didn't help that the power-on selftest for those took an age to complete | 14:28 |
| *** ykarel|afk is now known as ykarel | 14:28 | |
| *** lbragstad has quit IRC | 14:28 | |
| *** jamesmcarthur has quit IRC | 14:32 | |
| efried | I don't suppose there's any way to reorder patches waiting in the check pipeline, for our project? | 14:35 |
| AJaeger | in check? No | 14:37 |
| efried | openstack/nova | 14:37 |
| efried | 679640,30 | 14:37 |
| efried | ^ has had just horrible horrible luck, somewhere over a dozen rechecks. | 14:37 |
| efried | It's the last patch in a big feature; and it's underneath another big feature with 15-ish patches. | 14:37 |
| efried | Right now it's sitting 10th in the queue afaict; but it's the most important thing we have going. | 14:37 |
| efried | AJaeger: okay, sigh, as I feared. | 14:38 |
| *** jamesmcarthur has joined #openstack-infra | 14:38 | |
| AJaeger | efried: an admin can bypass check and enqueue directly into gate | 14:39 |
| AJaeger | efried: that might be the best way here | 14:39 |
| efried | AJaeger: ooo, that would do the trick. Will that en-gate all the patches already +V'd behind it? | 14:39 |
| AJaeger | efried: those that have a +1 by Zuul | 14:39 |
| efried | AJaeger: that would be wonderful. Who should I beg (and send the gift basket to)? | 14:40 |
| AJaeger | efried: ask infra-root and somebody might react | 14:41 |
| *** lajoskatona has joined #openstack-infra | 14:41 | |
| clarkb | usually we only do that for bug fixes | 14:42 |
| *** jamesmcarthur has quit IRC | 14:43 | |
| * efried peels off another C-note | 14:43 | |
| *** FlorianFa has quit IRC | 14:44 | |
| *** jamesmcarthur has joined #openstack-infra | 14:48 | |
| clarkb | fwiw I don't think a zuul enqueue of that change will automatically enqueue the others. Though they can be explicitly enqueued too. The reason for my hesitation here is in the past we've found that flaky changes commonly introduce new bugs that make the gate flakier. This is why clean check exists in openstack | 14:49 |
| clarkb | looks like the last check hit the mirror problems and this particular change is fairly simple so risk seems low | 14:49 |
| clarkb | however I haven't had a chance to look at the other 15 changes in the stack | 14:49 |
| *** kaiokmo has quit IRC | 14:51 | |
| clarkb | a few of teh child changes look quite a bit more complicated. efried do you just need the one change or the whole stack? | 14:51 |
| *** dtantsur is now known as dtantsur|afk | 14:52 | |
| efried | clarkb: Ultimately the whole stack. Most of them are already +V (which should help assuage fears about introducing regressions). There are two or three others that are in the check pipeline right now. | 14:52 |
| efried | clarkb: All of the failures thus far have been due to issues unrelated to the series. | 14:52 |
| efried | we've merged fixes for like three of those. One we're still working on (the notorious test_walk_versions). And then there was the mirror snafu early this morning. | 14:53 |
| *** pcaruana has quit IRC | 14:55 | |
| clarkb | I think I'd be ok with the first in the stack due to its simplicity, but I have no way of evaluating the next one as it is much larger and touches libvirt driver internals | 14:55 |
| clarkb | infra-root ^ anyone else want to weigh in? maybe mriedem too? | 14:56 |
| clarkb | (mriedem tends to do the debugging and fixing of gate issues so am happy to defer to mriedem on the concern of "this might make the gate worse") | 14:57 |
| corvus | AJaeger: *theoretically* we may be able to promote in check. i don't think anyone has tried it. :) | 14:57 |
| mriedem | sorry what's the question? if https://review.opendev.org/#/q/topic:bp/virtual-persistent-memory+status:open should be promoted? | 14:59 |
| mriedem | to unblock this https://review.opendev.org/#/q/topic:bp/cpu-resources+status:open | 14:59 |
| mriedem | oh god | 14:59 |
| clarkb | mriedem: if https://review.opendev.org/#/c/679640/30 and its children should be direct enqueued to the gate | 14:59 |
| clarkb | I don't think we would promote them, just enqueue them to the gate to bypass check | 14:59 |
| corvus | my read is that we would not normally handle them specially since they're not fixing gate stability issues | 15:00 |
| mriedem | given that series after it was just dealing with fixing intermittent test failures yesterday and also yesterday we found a new regression due to a series that merged last week b/c it was recheck ground through the gate, i'm a bit loathe to enqueue these things to the gate | 15:01 |
| mriedem | so the only benefit in enqueing these things is to find out if any of them introduce new regressions faster, | 15:01 |
| mriedem | but imo the better way to have dealt with that was not rush them all in last week right at FF....but that ship has sailed | 15:02 |
| mriedem | iow i'm a soft -.5 on enqueue | 15:02 |
| fungi | what, don't procrastinate? how very un-openstack of you | 15:02 |
| clarkb | ok sounds like we probably shouldn't bypass check in this case. (it would go against long standing policy and the change stack has had a bit of churn) Should we attempt a check queue promote | 15:07 |
| clarkb | corvus: ^ any idea on what the risk to trying that is? | 15:07 |
| fungi | i would be surprised if it does anything at all in check since each change is in its own queue and promote generally just reorders the queue for the indicated change | 15:08 |
| fungi | but can't hurt to test (probably) | 15:08 |
| openstackgerrit | Merged zuul/zuul master: Disable rsh synchronize rsync_opts https://review.opendev.org/682657 | 15:08 |
| *** ykarel is now known as ykarel|away | 15:08 | |
| *** udesale has quit IRC | 15:08 | |
| *** iurygregory has quit IRC | 15:11 | |
| efried | fungi: worst case I guess it goes to the back of the line? I'm willing to take the chance. | 15:15 |
| donnyd | so this is interesting... I don't see anything in logstash for FN for a couple of days | 15:15 |
| clarkb | donnyd: ya frickler pointed out we have an issue there earlier. I've not had a chance to look at it though | 15:15 |
| donnyd | issue with FN or logstash? | 15:16 |
| clarkb | with logstash | 15:16 |
| fungi | sounded like logstashing indexing across the board | 15:16 |
| clarkb | we have a logstash job queue of >350k events | 15:16 |
| clarkb | 62 workers currently processing events | 15:16 |
| corvus | clarkb, fungi, efried: give me a second to do a thought experiment to see whether it's worth trying | 15:17 |
| clarkb | ES is "red" status likely explains it | 15:19 |
| corvus | clarkb, fungi, efried: yeah, i think fungi's right -- it would probably be a no-op or worse; we shouldn't try it. | 15:19 |
| efried | ack | 15:19 |
| fungi | clarkb: oof, lost some es cluster members i guess? | 15:19 |
| clarkb | fungi: we had a shard that was failing to initialize in the index for september 14 | 15:21 |
| clarkb | I've never been able to sort out how to fix that other than deleting the index | 15:21 |
| fungi | ugh | 15:22 |
| clarkb | I've done that now and chances are we may lose all the logstash events too if we don't end up catching back up again | 15:22 |
| clarkb | so we might have a hole from the 14-nowish | 15:22 |
| clarkb | lets see how it does with the bad index gone first | 15:22 |
| clarkb | the inded from the 13th is 850GB large | 15:23 |
| clarkb | possible we are running up against disk limits | 15:23 |
| *** lbragstad_ is now known as lbragstad | 15:23 | |
| clarkb | might have to reduce retention to 7 days instead of 10 if so | 15:24 |
| donnyd | clarkb: fwiw I usually use logstash to T/S realtime issues.. and typically don't go back more than 48H | 15:25 |
| paladox | corvus: awesome! | 15:25 |
| clarkb | donnyd: ya that tends to be the main use case and due to volume its always been best effort so not the end of the world to reduce retention. The biggest downside is we lose ~30% of our tracking data for elastic-recheck bug tracking | 15:27 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add scheduler max_hold_age config option. https://review.opendev.org/682675 | 15:28 |
| *** gyee has joined #openstack-infra | 15:32 | |
| clarkb | we are ~800 events ahead of where we were when I first started debugging this | 15:33 |
| clarkb | I think we check it in an hour or two and see if the trend holds. Then math out how long to catch up based on the trend | 15:34 |
| clarkb | 325449 events is the starting point at about 15:25 UTC | 15:34 |
| donnyd | :) | 15:36 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 15:38 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 15:38 |
| *** piotrowskim has quit IRC | 15:41 | |
| *** jcoufal has quit IRC | 15:42 | |
| *** jcoufal has joined #openstack-infra | 15:44 | |
| *** e0ne has quit IRC | 15:48 | |
| *** ykarel_ has joined #openstack-infra | 15:48 | |
| *** lajoskatona has quit IRC | 15:49 | |
| AJaeger | clarkb: could you put the changes from https://review.opendev.org/#/q/status:open++topic:promote-project-config on your review list, please? Those will make the move from static.o.o publishing to AFS based publishing easier with this intermediate step to use promote jobs. I can merge them one by one and test but need a second +2 on the stack. | 15:49 |
| corvus | infra-root: i'm setting up a google account for zuul to use to talk to gerrit-review.googlesource.com | 15:50 |
| *** goldyfruit_ has joined #openstack-infra | 15:50 | |
| clarkb | AJaeger: will do | 15:50 |
| *** ykarel|away has quit IRC | 15:50 | |
| AJaeger | thanks, clarkb | 15:52 |
| *** goldyfruit___ has quit IRC | 15:53 | |
| rajinir | Zuul2 to Zuul3 ( For thirdparty CI), is this being worked on ? Any ETA? https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-3rd-party-ci.html | 15:53 |
| *** ykarel_ has quit IRC | 15:54 | |
| clarkb | rajinir: no one is assigned to it or working on it as far as I know | 15:54 |
| clarkb | rajinir: the spec was written to give guidance to people that may watn to work on it | 15:55 |
| rajinir | clarkb: looks like we need someone to guide us | 15:55 |
| clarkb | well the spec is the start of that. If someone wants to volunteer to start working on that I'm sure we can help as questions come up | 15:56 |
| *** mattw4 has joined #openstack-infra | 15:56 | |
| rajinir | clarkb: Thirparty CI is so behind. We need someknow who knows to volunteer I guess | 15:56 |
| clarkb | rajinir: https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-3rd-party-ci.html#work-items should be a fairly complete list if you want to advertise that to third party ci operators | 15:57 |
| mordred | rajinir, clarkb: I feel like there's someone using zuulv3 for 3pci - so hopefully someone who has figured it out can help with content | 15:57 |
| fungi | headed out to grab lunch, back in a bit | 15:58 |
| rajinir | mordred> Any idea who that would be? From time to time I see email on the openstack-discuss but no response | 15:58 |
| mordred | rajinir: no - sadly. I remember discussing things with people at PTGs, but not specifically who | 16:04 |
| mordred | I think they may be doing 3pci of ironic things .,.. dtantsur|afk TheJulia do you know? | 16:05 |
| *** armax has quit IRC | 16:06 | |
| *** mriedem is now known as mriedem_afk | 16:09 | |
| clarkb | looks like centos 7.7 release(s|d) today. Heads up that we may start to see centos job fail in unexpected ways as a result | 16:11 |
| *** yamamoto has quit IRC | 16:14 | |
| *** pgaxatte has quit IRC | 16:26 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold delete/info commands to web API https://review.opendev.org/679057 | 16:28 |
| *** rlandy has quit IRC | 16:30 | |
| *** gfidente has quit IRC | 16:33 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Remove outdated TODO https://review.opendev.org/682421 | 16:34 |
| *** rkukura has joined #openstack-infra | 16:34 | |
| *** rlandy has joined #openstack-infra | 16:35 | |
| *** goldyfruit___ has joined #openstack-infra | 16:40 | |
| *** derekh has quit IRC | 16:41 | |
| *** iurygregory has joined #openstack-infra | 16:42 | |
| *** goldyfruit_ has quit IRC | 16:43 | |
| *** diablo_rojo has joined #openstack-infra | 16:48 | |
| *** rfolco is now known as rfolco|dentist | 16:51 | |
| *** e0ne has joined #openstack-infra | 16:55 | |
| *** pcaruana has joined #openstack-infra | 16:55 | |
| *** diablo_rojo has quit IRC | 16:55 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add enqueue reporter action https://review.opendev.org/681132 | 16:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add no-jobs reporter action https://review.opendev.org/681278 | 16:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add report time to item model https://review.opendev.org/681323 | 16:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add Item.formatStatusUrl https://review.opendev.org/681324 | 16:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add support for the Gerrit checks plugin https://review.opendev.org/680778 | 16:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Update gerrit pagination test fixtures https://review.opendev.org/682114 | 16:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Support HTTP-only Gerrit https://review.opendev.org/681936 | 16:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add autogenerated tag to Gerrit reviews https://review.opendev.org/682473 | 16:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Use robot_comments in Gerrit https://review.opendev.org/682487 | 16:58 |
| clarkb | AJaeger: I've approved those changes (well the first 4 at least) | 16:59 |
| clarkb | AJaeger: can you help keep an eye on them? I'll also keep my eye out for needing to reapprove due to the depends on and checking the indexes after | 16:59 |
| *** jpena is now known as jpena|off | 16:59 | |
| AJaeger | clarkb: yes, will monitor and recheck etc. Thanks! | 17:00 |
| *** e0ne has quit IRC | 17:00 | |
| *** e0ne has joined #openstack-infra | 17:01 | |
| *** e0ne has quit IRC | 17:05 | |
| clarkb | 318403 is the current gearman logstash queue size | 17:05 |
| *** kopecmartin is now known as kopecmartin|off | 17:06 | |
| clarkb | its been ~100 minutes | 17:06 |
| clarkb | at this rate we'll catch up in 3 days? | 17:07 |
| clarkb | anyone else want to doulbe check that math? | 17:07 |
| clarkb | if that math is correct I think we should dump the queue and prefer new data to old data | 17:07 |
| *** ramishra_ has quit IRC | 17:08 | |
| openstackgerrit | Merged openstack/project-config master: Switch project-config to promote jobs [1]: Disable publish https://review.opendev.org/682211 | 17:10 |
| *** iurygregory has quit IRC | 17:12 | |
| *** e0ne has joined #openstack-infra | 17:13 | |
| *** ralonsoh has quit IRC | 17:16 | |
| *** e0ne has quit IRC | 17:17 | |
| *** armax has joined #openstack-infra | 17:19 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Switch project-config to promote jobs [2]: Update build jobs https://review.opendev.org/682213 | 17:25 |
| *** ociuhandu has quit IRC | 17:25 | |
| *** gfidente has joined #openstack-infra | 17:29 | |
| *** priteau has quit IRC | 17:32 | |
| *** e0ne has joined #openstack-infra | 17:33 | |
| *** gfidente is now known as gfidente|afk | 17:36 | |
| *** yamamoto has joined #openstack-infra | 17:40 | |
| *** michael-beaver has quit IRC | 17:40 | |
| openstackgerrit | Merged openstack/project-config master: Switch project-config to promote jobs [3]: Update building https://review.opendev.org/682212 | 17:42 |
| *** e0ne has quit IRC | 17:43 | |
| *** ociuhandu has joined #openstack-infra | 17:44 | |
| dirk | frickler: AJaeger: evrardjp: reading the backlog right now.. the build failiures looks like the mirroring is broken. | 17:44 |
| dirk | is this a repeating problem? where do we store the logfiles of the mirroring? are those accessible? | 17:45 |
| openstackgerrit | Merged zuul/zuul-jobs master: Update the base-roles test to use prepare-workspace-git https://review.opendev.org/680703 | 17:47 |
| openstackgerrit | Merged zuul/zuul-jobs master: Clean non-bare remote repos https://review.opendev.org/680689 | 17:47 |
| clarkb | dirk: they are accessivle I just have to find them | 17:47 |
| *** yamamoto has quit IRC | 17:48 | |
| *** ociuhandu has quit IRC | 17:48 | |
| clarkb | dirk http://files.openstack.org/mirror/logs/rsync-mirrors/ | 17:48 |
| dirk | thanks! | 17:51 |
| dirk | thats very useful | 17:51 |
| dirk | so it looks like mirroring was finished 16:40:xx UTC, and the build was missing the files at 17:41:xx | 17:52 |
| dirk | so assuming that there isn't an off by one in timezones, is it really normal that it takes an hour to propagate? | 17:52 |
| *** mriedem_afk is now known as mriedem | 17:53 | |
| clarkb | when does the log show vos releasesucceeding | 17:54 |
| clarkb | once that is done the data should be available | 17:54 |
| fungi | #status log booted ze01 2019-09-17 17:45z after it entered a shutoff state for unidentified reasons around 2019-09-16 13:10z | 17:56 |
| openstackstatus | fungi: finished logging | 17:56 |
| dirk | clarkb: not sure, http://mirror.dfw.rax.openstack.org/opensuse/timestamp.txt says 16:40x | 17:57 |
| dirk | the last timestamp says Starting ForwardMulti from 536871010 to 536871010 on afs02.dfw.openstack.org (as of Tue Sep 17 10:10:18 2019). | 17:57 |
| dirk | I assume that is some other timezone other than UTC | 17:58 |
| clarkb | no it should be utc | 17:58 |
| dirk | but thats ~ 6 hours before the previous log line | 17:59 |
| clarkb | that may be afs reporting the last update to the volume | 17:59 |
| clarkb | from 6 hours ago to currwnt data | 17:59 |
| clarkb | dirk: that vos release is still running since 16:40 | 18:07 |
| mordred | clarkb: I thought I was going to be able to just work through to the infra meeting, but I've got to go get some dinner. mucho apologies | 18:08 |
| clarkb | I think that is why we don't see the updated files yet | 18:08 |
| clarkb | mordred: I just finished eating an early lunch for this reason :) | 18:08 |
| mordred | clarkb: yeah - that's what I should have done | 18:08 |
| clarkb | infra-root I'm going to restart the gearman server that we run for logstash event processing. The queue is long and doesn't seem to be speeding up. This will dump events for the last ~3 days in order to process them for current jobs | 18:11 |
| clarkb | let me know if you think we should wait it out longer | 18:11 |
| *** weshay has joined #openstack-infra | 18:12 | |
| *** whoami-rajat has quit IRC | 18:12 | |
| fungi | sounds like a reasonable compromise to me | 18:13 |
| fungi | better to find out what bugs are hitting builds now rather than wait 3 days | 18:13 |
| clarkb | yup | 18:13 |
| *** igordc has joined #openstack-infra | 18:16 | |
| dirk | clarkb: so it is ongoing sync? | 18:19 |
| clarkb | dirk: yes I believe so | 18:19 |
| dirk | hmm, so that explains why things are faling | 18:19 |
| auristor | There are two vos release operations in flight at the moment. mirror.fedora and mirror.opensuse. afs01.dfw.openstack.org is serving the latest version of the mirror.fedora.readonly and mirror.opensuse.readonly. | 18:19 |
| auristor | The afs02.dfw.openstack.org server has those volumes offline while they being updated. | 18:20 |
| frickler | dirk: clarkb: iiuc the rsync was killed with a timeout earlier in that log, so we are publishing an unfinished version | 18:20 |
| clarkb | frickler: ah | 18:20 |
| auristor | the clients should be reading from afs01.dfw.openstack.org | 18:20 |
| fungi | i thought the script avoided running a vos release if rsync exited nonzero? | 18:20 |
| clarkb | fungi: ya me too | 18:21 |
| clarkb | but that could explain some of the weirdness if that isn't working | 18:21 |
| frickler | I think we changed that to ignore errors on tumbleweed | 18:21 |
| frickler | because it help up releases too often | 18:21 |
| frickler | held | 18:21 |
| fungi | oh, is it only jobs on tumbleweed getting bad package updates? | 18:21 |
| auristor | The timestamp listed in the "Starting ForwardMulti" line is the last update time of the source volume | 18:21 |
| frickler | I think the error on 423 was different | 18:22 |
| frickler | yeah, that's a pkg conflict. it may also be due to broken mirroring, but not clear to me https://nb01.openstack.org/opensuse-423-0000061039.log | 18:24 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update infra and specs indices https://review.opendev.org/682711 | 18:28 |
| *** ricolin has quit IRC | 18:28 | |
| AJaeger | config-core, here's a change to test that the promote jobs work which is usefull - reviews welcome ^ | 18:29 |
| openstackgerrit | Merged openstack/project-config master: Switch project-config to promote jobs [4]: Add promote jobs https://review.opendev.org/682214 | 18:30 |
| *** goldyfruit___ has quit IRC | 18:31 | |
| fungi | clarkb: we're due to generate another openstack release signing key, so today might be a good opportunity to toss around ideas for future iterations of that during the opendev portion of the meeting | 18:33 |
| clarkb | fungi: ok | 18:34 |
| clarkb | fungi: its currently managed as a secret right? Seems like the openstack release team could generate it themselves then attest to it? | 18:34 |
| * clarkb throws initiall thoughts out into the ether | 18:34 | |
| fungi | yeah, at this point there's no longer hiera involvement so it could be project-specific quite easily | 18:35 |
| fungi | or we could decide we just want it to be an infrastructure-wide feature | 18:35 |
| AJaeger | mgoddard: all Zuul config errors disappeared - thanks for fixing! | 18:35 |
| fungi | clarkb: the biggest gotcha is the need to have the people involved in generating the new keys able to initially have somewhere to share them with each other, reducing the bus factor. we accomplish it with a directory on bridge.o.o | 18:37 |
| fungi | because our root sysadmins can all ssh in and gain access to co-manage those private files | 18:38 |
| fungi | a group like the release team lacks a safe coordination space like that, i think | 18:38 |
| fungi | which is not to say that they couldn't create one, obviously | 18:38 |
| *** hemna_ is now known as hemna_afk | 18:43 | |
| AJaeger | fungi, could you review 682214 , please? | 18:43 |
| AJaeger | then I know all is fine ;) | 18:44 |
| *** dims has quit IRC | 18:45 | |
| clarkb | fungi: would a system of ptl create key and sign it, work as long as there is a web of trust among the release team? I suppose that presumes that all team members have some way of trustign each other in that manner ahead of time | 18:50 |
| ianw | fricker / mordred : fyi none of the kafs servers are in rotation ATM | 18:50 |
| cmurphy | summit keysigning party? | 18:51 |
| fungi | clarkb: yeah, though also for best practice they probably need to do something like e-mail a copy of the revocation certificate to all the teammates | 18:52 |
| *** dims has joined #openstack-infra | 18:52 | |
| *** ociuhandu has joined #openstack-infra | 18:52 | |
| fungi | cmurphy: it'd be just like old times! | 18:52 |
| fungi | AJaeger: 682214 merged at 18:30z, did you mean a different change? | 18:54 |
| AJaeger | fungi: https://review.opendev.org/682711 | 18:54 |
| AJaeger | sorry | 18:54 |
| *** xek_ has joined #openstack-infra | 18:54 | |
| fungi | no trouble, thanks | 18:54 |
| AJaeger | wow, efried's 679640 (and some others) is waiting since more than 7 hours for nodes... | 18:55 |
| efried | indeed | 18:55 |
| *** ociuhandu has quit IRC | 18:56 | |
| fungi | AJaeger: yep, nova has a lot of changes in flight so the later ones get prempted to give changes from other projects a turn | 18:56 |
| fungi | er, preempted | 18:56 |
| efried | The one at the front of the queue just started after ~9h | 18:57 |
| *** xek has quit IRC | 18:57 | |
| clarkb | fungi: and we have had a lot of gate resets today and we have reduced node counts with inap out | 18:59 |
| *** e0ne has joined #openstack-infra | 18:59 | |
| fungi | yup | 18:59 |
| clarkb | meeting time | 19:00 |
| *** lucasagomes has quit IRC | 19:01 | |
| *** armax has quit IRC | 19:03 | |
| *** armax has joined #openstack-infra | 19:05 | |
| *** rkukura has quit IRC | 19:34 | |
| *** armax has quit IRC | 19:36 | |
| *** e0ne has quit IRC | 19:38 | |
| openstackgerrit | Amy Marrich (spotz) proposed opendev/irc-meetings master: Updated the Diversity WG to monthly https://review.opendev.org/682727 | 19:40 |
| *** goldyfruit___ has joined #openstack-infra | 19:40 | |
| *** jcoufal has quit IRC | 19:45 | |
| *** priteau has joined #openstack-infra | 19:46 | |
| *** denaitre has joined #openstack-infra | 19:47 | |
| *** spsurya has quit IRC | 19:48 | |
| *** e0ne has joined #openstack-infra | 19:50 | |
| *** armax has joined #openstack-infra | 19:56 | |
| *** larainema has quit IRC | 19:59 | |
| fungi | #status log all zuul executors restarted on "zuul==3.10.2.dev79 # git sha e4f3f48" | 20:01 |
| openstackstatus | fungi: finished logging | 20:01 |
| ianw | AJaeger: sorry, just had my head in other things. will come up with, as mentioned, a bit more of a plan in the etherpad and we can see who we can get interested from there | 20:01 |
| AJaeger | ianw: no urgency on this, no worries | 20:02 |
| *** armax has quit IRC | 20:02 | |
| ianw | well we would like to get that server retired :) | 20:02 |
| AJaeger | ok ;) | 20:02 |
| AJaeger | fungi, still time for 682711 , please? | 20:03 |
| fungi | in positive news, the /srv/static/logs volume on static.o.o is only 4% full, so we can probably start freeing up pvs | 20:03 |
| fungi | should be able to pvmove everything else onto a single pv | 20:04 |
| fungi | and then shrink the vg off the remainder of them and nova detach/cinder delete them all | 20:04 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Restore setup-org.yaml https://review.opendev.org/682732 | 20:05 |
| clarkb | infra-root ^ that is my chagne to restore that file. I just checked it out from the commit prior to the pythonification and committed that | 20:05 |
| clarkb | fungi: maybe turn it into a raid 1? (though maybe we can't do that with lvm as is) | 20:08 |
| AJaeger | thanks, fungi | 20:10 |
| *** jtomasek has quit IRC | 20:11 | |
| openstackgerrit | Merged openstack/project-config master: Update infra and specs indices https://review.opendev.org/682711 | 20:15 |
| *** goldyfruit_ has joined #openstack-infra | 20:15 | |
| *** sean-k-mooney has quit IRC | 20:17 | |
| *** goldyfruit___ has quit IRC | 20:18 | |
| AJaeger | \o/ http://specs.openstack.org/ is updated | 20:20 |
| AJaeger | and https://docs.openstack.org/infra/ as well! | 20:20 |
| AJaeger | config-core, https://review.opendev.org/#/c/682215/ and https://review.opendev.org/682216 should be ready for review, please | 20:22 |
| * AJaeger won't be around to monitor them but can do tomorrow | 20:23 | |
| AJaeger | good night! | 20:23 |
| *** sean-k-mooney has joined #openstack-infra | 20:25 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Add zuul-registry project https://review.opendev.org/682743 | 20:27 |
| *** priteau has quit IRC | 20:28 | |
| corvus | clarkb: i belatedly responded to your email on zuul-discuss about that ^ at http://lists.zuul-ci.org/pipermail/zuul-discuss/2019-September/001017.html -- take a look at that and let me know if you think that hacking on crane and improving the docker registry swift support is a better alternative than zuul-registry | 20:29 |
| *** panda|ruck is now known as panda|ruck|off | 20:29 | |
| clarkb | corvus: having poked around a little bit myself I think a simple zuul registry that solve's zuul's needs may be simplest. In particular harbor seems to need a postgres and a redis and a bunch of microservices | 20:31 |
| clarkb | which seems like overkill for copying data around | 20:31 |
| corvus | agreed | 20:31 |
| *** mriedem has quit IRC | 20:32 | |
| corvus | i really dig pulp's crane. and heck, it's in python -- even if we do make zuul-registry, there's no reason we can't then talk to the pulp folks about merging the two later if we like | 20:32 |
| *** mriedem has joined #openstack-infra | 20:32 | |
| *** weshay is now known as weshay|ruck | 20:32 | |
| corvus | but given that it'd probably be a 100% expansion in the size and scope, i don't want to presume :) | 20:33 |
| *** bdodd_ has joined #openstack-infra | 20:41 | |
| *** bdodd has quit IRC | 20:42 | |
| fungi | clarkb: lvcreate -m/--mirrors will do that, yes, but we'd need to copy data into it... i don't know of a way to change the number of volume mirrors for an existing volume | 20:43 |
| fungi | alternatively we could use mdraid across two cinder volumes and then pvcreate on the raid device | 20:44 |
| fungi | either way, without some sort of volume anti-affinity, i'm not sure mirroring there makes a ton of sense | 20:44 |
| corvus | at least shrinking makes it faster to recover i guess | 20:45 |
| fungi | yes, well, recovery time is mainly the fsck, so deleting the logs mount entirely accomplishes the same | 20:45 |
| *** Goneri has quit IRC | 20:46 | |
| *** trident has quit IRC | 20:48 | |
| fungi | then again, fsck of a very large volume with no files is also probably quite fast | 20:48 |
| fungi | s/volume/filesystem/ | 20:49 |
| *** gfidente|afk has quit IRC | 20:53 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: [dnm] testing python3 ansible https://review.opendev.org/682556 | 20:53 |
| *** ociuhandu has joined #openstack-infra | 20:53 | |
| fungi | the main reason we might consider shrinking the vg is we can give back 13tb of cinder quota we're currently using in dfw | 20:53 |
| fungi | and not having to scan the additional 13 devices to assemble the volume group might save a bit of boot time | 20:54 |
| *** eharney has quit IRC | 20:55 | |
| clarkb | fungi: yup I think shrinking is a great idea | 20:58 |
| *** pcaruana has quit IRC | 20:58 | |
| corvus | our account on gerrit's gerrit should have verify+1 perms now | 20:59 |
| *** trident has joined #openstack-infra | 21:00 | |
| corvus | it's unclear whether we also have the right perms to use the checks api; i'm assuming so, but we may have to either test it or wait until tomorrow for confirmation | 21:00 |
| corvus | anyway, we should have the pre-reqs ready for when we land the gerrit stack | 21:00 |
| *** slaweq has quit IRC | 21:10 | |
| *** slaweq has joined #openstack-infra | 21:11 | |
| mordred | corvus: there are some test unhappies in the gerrit stack | 21:14 |
| corvus | oops forgot gitadd | 21:14 |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: [dnm] testing python3 ansible https://review.opendev.org/682556 | 21:15 |
| mordred | corvus: any reason to hold off on +A for the things that are ready for it? | 21:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add support for the Gerrit checks plugin https://review.opendev.org/680778 | 21:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Update gerrit pagination test fixtures https://review.opendev.org/682114 | 21:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Support HTTP-only Gerrit https://review.opendev.org/681936 | 21:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add autogenerated tag to Gerrit reviews https://review.opendev.org/682473 | 21:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Use robot_comments in Gerrit https://review.opendev.org/682487 | 21:15 |
| *** slaweq has quit IRC | 21:16 | |
| *** rkukura has joined #openstack-infra | 21:16 | |
| *** e0ne has quit IRC | 21:19 | |
| corvus | mordred: Shrews wanted to look at it, which i think is a fine idea, though perhaps since it's an experimental feature anyway (and we can undo anything we want without breaking backwards compat), Shrews would you mind if we go ahead and start +Wing them and we can address any issues you find in a followup? i'd like to keep moving on testing this, and it'll probably take a while to land them anyway | 21:19 |
| corvus | with the backlog. | 21:19 |
| corvus | er i meant to say "we can do anything we want without worrying about breaking backwards compat" | 21:20 |
| fungi | words to live by! ;) | 21:20 |
| Shrews | corvus: mordred: go ahead. i can review post merge and am already involved in dinner things | 21:22 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use the gerrit driver for the googlesource connection https://review.opendev.org/682749 | 21:22 |
| *** rfolco|dentist is now known as rfolco | 21:23 | |
| corvus | mordred: ^ strictly speaking, i think we probably should have called that "gerrit-googlesource" or something. i think i finally understand their hostname construction. but it's probably not worth changing now :) | 21:24 |
| paladox | corvus \o/ | 21:26 |
| *** e0ne has joined #openstack-infra | 21:28 | |
| *** e0ne has quit IRC | 21:28 | |
| *** aaronsheffield has quit IRC | 21:29 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Add the gerrit checks plugin repo to Zuul https://review.opendev.org/682752 | 21:29 |
| *** slaweq has joined #openstack-infra | 21:32 | |
| *** ociuhandu has quit IRC | 21:33 | |
| *** markvoelker has quit IRC | 21:35 | |
| *** slaweq has quit IRC | 21:36 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Add upstream Gerrit projects to OpenDev Zuul tenant https://review.opendev.org/682754 | 21:48 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Add system-config and zuul to OpenDev tenant for jobs https://review.opendev.org/682755 | 21:48 |
| *** prometheanfire has quit IRC | 21:50 | |
| *** prometheanfire has joined #openstack-infra | 21:51 | |
| corvus | remote: https://review.opendev.org/682756 Add a third-party check pipeline to OpenDev | 21:52 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Announce opendev/project-config changes in channels https://review.opendev.org/682757 | 21:54 |
| corvus | okay, i believe "topic:gerrit-zuul" is ready -- anything that isn't WIP or Verified:-1 can safely merge now; the rest are after we restart with the gerrit changes for zuul | 21:55 |
| corvus | that will get us basic connectivity; next step i think is to add a checker and some noop jobs | 21:56 |
| corvus | oops, made one change in the wrong repo | 21:58 |
| corvus | remote: https://review.opendev.org/682758 Add a third-party check pipeline to OpenDev | 21:59 |
| corvus | there we go | 21:59 |
| clarkb | we need to restart the scheduler too right? | 22:03 |
| corvus | clarkb: only for 682749 which is WIP and 682758 which depend-on it; the others can merge now. | 22:04 |
| corvus | (basically, anything that can merge can be merged) | 22:05 |
| clarkb | gotcha | 22:05 |
| EmilienM | hey infra folks | 22:11 |
| EmilienM | FYI I'm clearing a bunch of tripleo jobs in check/gate (abandon) - we have identified an issue in how we prepare the container registry in our undercloud, which causes a bunch of timeouts | 22:12 |
| EmilienM | currently investigating and working on a proper fix | 22:12 |
| *** jamesmcarthur has quit IRC | 22:13 | |
| clarkb | EmilienM: thanks | 22:14 |
| EmilienM | I'm also really concerned by docker.io reliability in general | 22:14 |
| clarkb | EmilienM: we should be caching those in our proxy if you use it | 22:15 |
| clarkb | that should improve reliability | 22:15 |
| *** xek_ has quit IRC | 22:15 | |
| EmilienM | we use the proxies | 22:15 |
| * EmilienM digs a bit | 22:16 | |
| clarkb | if this is still the issue with authentication that has to do with your custom python client and the same error is reproduceable with curl | 22:16 |
| clarkb | direlcty to docker hub | 22:16 |
| openstackgerrit | Merged zuul/zuul master: Add enqueue reporter action https://review.opendev.org/681132 | 22:23 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Fix the zuul main.yaml config checks https://review.opendev.org/682759 | 22:31 |
| clarkb | corvus: I think https://review.opendev.org/#/c/682755/ needs ^ | 22:31 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Move reference pipelines out of the quickstart https://review.opendev.org/682760 | 22:36 |
| corvus | clarkb: ah thanks! | 22:37 |
| corvus | clarkb: i guess we only used that form for non-gerrit connections so far | 22:37 |
| clarkb | ya | 22:38 |
| clarkb | like github | 22:38 |
| corvus | clarkb: quick suggestion on that | 22:38 |
| clarkb | corvus: thanks will update | 22:39 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Fix the zuul main.yaml config checks https://review.opendev.org/682759 | 22:40 |
| corvus | clarkb: lgtm. you'll be happy to know we only support that one level deep :) | 22:41 |
| corvus | i just got spam offering me "OpenStock Users List" | 22:44 |
| paladox | lol | 22:44 |
| *** exsdev0 has joined #openstack-infra | 22:45 | |
| *** exsdev has quit IRC | 22:46 | |
| *** exsdev0 is now known as exsdev | 22:46 | |
| fungi | i get those on an almost daily basis | 22:48 |
| fungi | i find it hilarious every time | 22:49 |
| fungi | oh! openstOck | 22:49 |
| fungi | no, haven't had one of those yet | 22:49 |
| fungi | i could get behind some opensnack right about now though | 22:49 |
| *** mriedem has quit IRC | 22:52 | |
| spotz | ooohh snacks! | 23:02 |
| *** tkajinam has joined #openstack-infra | 23:04 | |
| *** tosky has quit IRC | 23:11 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** bdodd_ has quit IRC | 23:13 | |
| *** slaweq has quit IRC | 23:15 | |
| *** goldyfruit_ has quit IRC | 23:17 | |
| *** calebb has quit IRC | 23:22 | |
| clarkb | fungi: ianw have a quick moment for https://review.opendev.org/#/c/682759/ ? will address testing bug in the gerrit gerrit zuul stack | 23:22 |
| *** markvoelker has joined #openstack-infra | 23:22 | |
| *** jamesmcarthur has joined #openstack-infra | 23:24 | |
| openstackgerrit | Merged zuul/zuul master: Move reference pipelines out of the quickstart https://review.opendev.org/682760 | 23:25 |
| *** dchen has joined #openstack-infra | 23:34 | |
| *** mattw4 has quit IRC | 23:34 | |
| *** adriant has joined #openstack-infra | 23:35 | |
| *** igordc has quit IRC | 23:39 | |
| openstackgerrit | Merged openstack/project-config master: Fix the zuul main.yaml config checks https://review.opendev.org/682759 | 23:41 |
| *** rcernin has joined #openstack-infra | 23:44 | |
| *** sthussey has quit IRC | 23:44 | |
| *** yamamoto has joined #openstack-infra | 23:46 | |
| *** jamesmcarthur has quit IRC | 23:50 | |
| *** rlandy has quit IRC | 23:51 | |
| *** yamamoto has quit IRC | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!