*** Tengu has quit IRC | 00:20 | |
*** Tengu has joined #oooq | 00:22 | |
*** dsneddon has quit IRC | 00:40 | |
*** dsneddon has joined #oooq | 00:45 | |
*** ChanServ sets mode: +o panda | 00:46 | |
*** jmasud has joined #oooq | 00:47 | |
*** panda changes topic to "Docs: https://docs.openstack.org/tripleo-quickstart/latest/ || 5th Mar promoter patch to review https://review.rdoproject.org/r/24721 || next promoter patch in the chain https://review.rdoproject.org/r/24774" | 00:47 | |
*** panda sets mode: -o panda | 00:47 | |
*** jmasud has quit IRC | 00:54 | |
*** jmasud has joined #oooq | 00:56 | |
*** Goneri has quit IRC | 02:24 | |
*** sanjayu_ has joined #oooq | 02:42 | |
*** sanjayu_ has quit IRC | 02:53 | |
*** raukadah is now known as chandankumar | 03:02 | |
*** dsneddon has quit IRC | 03:51 | |
*** apetrich has quit IRC | 04:00 | |
*** udesale has joined #oooq | 04:09 | |
*** rlandy|bbl is now known as rlandy | 04:29 | |
*** skramaja has joined #oooq | 04:29 | |
rlandy | chandankumar: hey ... https://review.rdoproject.org/r/#/c/25618/ should be in a mergeable state now. Note the change to scenario010. Had mixed results with scenario007 | 04:51 |
---|---|---|
rlandy | putting up a testproject to retest that | 04:52 |
chandankumar | rlandy, will take care of that | 04:52 |
rlandy | https://review.opendev.org/#/c/708599/ is in gate now | 04:53 |
rlandy | going to sleep - back in 7 hours | 04:53 |
*** rlandy has quit IRC | 04:53 | |
*** ykarel|away is now known as ykarel | 04:55 | |
*** udesale has quit IRC | 05:31 | |
*** udesale has joined #oooq | 05:39 | |
*** dsneddon has joined #oooq | 06:12 | |
chandankumar | ykarel, can you check this job https://logserver.rdoproject.org/12/25612/7/check/distgit-tripleo-master-testing-centos-8-standalone-scenario010/1efbfb9/logs/undercloud/var/log/extra/package-list-installed.txt.gz | 06:25 |
chandankumar | I am not sure packages from temp repo got installed or not | 06:25 |
ykarel | chandankumar, looking | 06:28 |
ykarel | chandankumar, u mean gating-repo or temp repo? | 06:28 |
*** marios has joined #oooq | 06:29 | |
chandankumar | ykarel, temp repo | 06:29 |
ykarel | chandankumar, in logs i see @temp-distgit | 06:29 |
ykarel | so should be used | 06:29 |
ykarel | any specific example u looking for? | 06:29 |
ykarel | which is not updated | 06:29 |
chandankumar | ykarel, I want to make sure unpinned packages get used | 06:30 |
ykarel | those worked in my tests both on host and containers | 06:30 |
ykarel | but if u doubt on some specific package i can check again | 06:30 |
chandankumar | ok coming to office ttyl | 06:31 |
ykarel | ack | 06:31 |
*** weshay|ruck has quit IRC | 06:36 | |
*** saneax has joined #oooq | 06:36 | |
*** weshay has joined #oooq | 06:36 | |
*** weshay has quit IRC | 06:40 | |
*** weshay has joined #oooq | 06:40 | |
*** jfrancoa has joined #oooq | 06:52 | |
*** dsneddon has quit IRC | 07:01 | |
*** ccamacho has quit IRC | 07:02 | |
*** weshay has quit IRC | 07:13 | |
*** weshay_ has joined #oooq | 07:16 | |
*** jbadiapa has joined #oooq | 07:24 | |
*** weshay_ has quit IRC | 07:34 | |
*** weshay_ has joined #oooq | 07:34 | |
*** ykarel is now known as ykarel|lunch | 07:35 | |
*** weshay_ has quit IRC | 07:39 | |
*** migi has quit IRC | 07:40 | |
*** tesseract has joined #oooq | 07:52 | |
*** weshay_ has joined #oooq | 07:56 | |
*** ccamacho has joined #oooq | 08:11 | |
*** weshay_ has quit IRC | 08:24 | |
*** weshay has joined #oooq | 08:25 | |
*** jpena|off is now known as jpena | 08:26 | |
*** saneax has quit IRC | 08:29 | |
*** saneax has joined #oooq | 08:30 | |
*** ykarel|lunch is now known as ykarel | 08:32 | |
*** amoralej|off is now known as amoralej | 08:45 | |
*** bogdando has joined #oooq | 08:46 | |
*** marios has quit IRC | 08:56 | |
*** pojadhav is now known as pojadhav|lunch | 09:01 | |
*** pojadhav|lunch is now known as pojadhav | 09:01 | |
panda | arxcruz|rover: can't stop thinking now about a parody itaalian song from about the same period. It goes "I suffer from stress, yeah I suffer from stress // I'm tired and out of shape // I play in a boy band, I play in a boy band // There must be some mistake" | 09:05 |
arxcruz|rover | panda: where can I find this preciousity? | 09:06 |
*** irclogbot_2 has quit IRC | 09:08 | |
panda | arxcruz|rover: Yuo asked for it, you've got it. https://www.youtube.com/watch?v=Z1UO91TmOsY | 09:08 |
arxcruz|rover | OMG, I can watch youtube video directly from glowing-bear | 09:13 |
arxcruz|rover | thanks dmellado for the tip :D | 09:13 |
dmellado | xD | 09:13 |
dmellado | \o/ | 09:13 |
dmellado | glad you liked it | 09:14 |
arxcruz|rover | panda: I have to say, I only know Laura Pausini italian | 09:14 |
panda | arxcruz|rover: .... | 09:14 |
arxcruz|rover | I mean, I only understand italian from Laura Pausini songs :D | 09:15 |
arxcruz|rover | and Renato Russo in Italian, which is pretty good | 09:15 |
*** apetrich has joined #oooq | 09:24 | |
*** tosky has joined #oooq | 09:25 | |
*** holser has joined #oooq | 09:28 | |
*** sshnaidm|afk is now known as sshnaidm | 09:37 | |
arxcruz|rover | apetrich: ping | 09:39 |
arxcruz|rover | apetrich: are you aware of this mistral error? https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/e7bbed3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 09:40 |
apetrich | arxcruz|rover, heya | 09:40 |
apetrich | arxcruz|rover, how big is that file? Still loading | 09:41 |
arxcruz|rover | apetrich: not sure... | 09:41 |
*** irclogbot_2 has joined #oooq | 09:43 | |
apetrich | arxcruz|rover, what is that file? still loading | 09:43 |
arxcruz|rover | apetrich: really? | 09:43 |
arxcruz|rover | apetrich: http://paste.openstack.org/show/790326/ | 09:44 |
apetrich | just finished | 09:44 |
amoralej | arxcruz|rover, i've finished moving deps for centos8-master, let me know if you find any issue | 09:44 |
arxcruz|rover | amoralej: ok i'll | 09:45 |
apetrich | 1.6Mb is not that big. weird | 09:45 |
*** openstackstatus has quit IRC | 09:45 | |
apetrich | arxcruz|rover, so. kind of. We seen those but I'm not sure we have a patch for it | 09:49 |
arxcruz|rover | apetrich: I see this on the job yesterday, the same job today fails in another way, but I'm not sure yet, if was before or after mistral run | 09:50 |
sshnaidm | chandankumar, hi | 09:50 |
chandankumar | sshnaidm, Hello | 09:51 |
chandankumar | sshnaidm, https://review.rdoproject.org/r/#/c/25689/ | 09:53 |
sshnaidm | chandankumar, are 007 and 010 working on c8? | 09:55 |
chandankumar | sshnaidm, yes | 09:55 |
sshnaidm | cool | 09:55 |
*** bogdando has quit IRC | 09:55 | |
chandankumar | sshnaidm, https://review.opendev.org/711287 and https://review.opendev.org/711330 | 09:55 |
*** bogdando has joined #oooq | 09:56 | |
arxcruz|rover | apetrich: so, it fails today before the mistral, so not sure if it was fixed, I'm running on testproject right now to check | 10:01 |
*** derekh has joined #oooq | 10:13 | |
sshnaidm | panda, can you take a look please at http://38.102.83.109/centos7_queens.log - there is "Exception: Named Hashes Changed!" | 10:17 |
*** marios has joined #oooq | 10:17 | |
arxcruz|rover | apetrich: can you check if there's a patch already? Otherwise i'll open a bug | 10:22 |
arxcruz|rover | marios: regarding https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/e7bbed3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz did you open a bug? | 10:22 |
apetrich | arxcruz|rover, sure. one minute | 10:22 |
arxcruz|rover | i see it in some runs yesterday, today it fails in a different step, before mistral runs so, not sure if it's persistent, i'm running the job again in testproject to check | 10:23 |
*** dtantsur|afk is now known as dtantsur | 10:24 | |
marios | arxcruz|rover: looking | 10:28 |
marios | arxcruz|rover: (but if it isn't on the etherpad no) | 10:29 |
apetrich | arxcruz|rover, yeah I think we need a bug | 10:29 |
marios | arxcruz|rover: yeah it looks like that one | 10:29 |
apetrich | lol marios | 10:29 |
marios | arxcruz|rover: https://bugs.launchpad.net/tripleo/+bug/1866031 | 10:29 |
openstack | Launchpad bug 1866031 in tripleo "periodic centos7 fs2 upload/fs1 master fails overcloud deploy pcs "create constraint failed"" [Critical,Fix released] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 10:29 |
marios | apetrich: :) which bit | 10:31 |
arxcruz|rover | marios: apetrich sshnaidm okay, so, I did not see this today, however, the job fails before this step, so I'm running right now in testproject | 10:31 |
marios | apetrich: arxcruz|rover: confirmed same i see 2020-03-03 03:31:09 | Error: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Constraint::Order[ip-192.168.24.16-with-ovndb_servers]/Pcmk_constraint[order-ovn-dbs-bundle-ip-192.168.24.16]/ensure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200303-10-1ku4ixs create constraint failed: Error: | 10:31 |
marios | Resource 'ip-192.168.24.16' does not exist | 10:31 |
marios | apetrich: arxcruz|rover: fix from sshnaidm https://review.opendev.org/#/c/711201/ | 10:31 |
panda | sshnaidm: it's a check to be use taht nothing modified the dlrnapi while we are promoting. It's either a) a known bug I thought was fixed, b) something really chenged dlrnapi while you were promoting. Investigating. | 10:36 |
panda | sshnaidm: s/use/sure | 10:36 |
panda | sshnaidm: yep, bug, it was fixed while I was refactoring, so maybe I did not ported the fix correctly. The previous promotion did not update the global promotions status. It's generally harmless, if you rerun, the status will be consiste again, since it's per-invocation. I'll try to fix it. | 10:46 |
*** udesale has quit IRC | 10:49 | |
chandankumar | sshnaidm, good to go https://review.opendev.org/#/c/710961/ minion one | 10:55 |
sshnaidm | panda, ack, thanks | 11:01 |
*** jmasud has quit IRC | 11:04 | |
*** jmasud has joined #oooq | 11:04 | |
*** chem has quit IRC | 11:35 | |
*** tesseract-RH has joined #oooq | 11:40 | |
*** tesseract has quit IRC | 11:43 | |
chandankumar | sshnaidm, where we are testing image build changes | 11:54 |
chandankumar | ? | 11:54 |
chandankumar | may be job link would be helpful | 11:55 |
chandankumar | or is it current ovb wip patch in rdo-jobs? | 11:55 |
sshnaidm | chandankumar, just remove here to_build: false https://review.rdoproject.org/r/#/c/25332/36/zuul.d/ovb-jobs.yaml | 11:55 |
chandankumar | sshnaidm, ok will get to that | 11:56 |
chandankumar | brb | 11:58 |
*** marios is now known as marios|call | 12:01 | |
marios|call | arxcruz|rover: weshay: we have sync? | 12:01 |
arxcruz|rover | marios|call: yes, we are waiting for you | 12:01 |
marios|call | wut, did i join wrong one? from calendar meet.google.com/ixv-xbhp-xzg | 12:01 |
arxcruz|rover | marios|call: https://meet.google.com/ixv-xbhp-xzg | 12:01 |
*** rlandy has joined #oooq | 12:02 | |
rlandy | chandankumar: hey ... merging https://review.rdoproject.org/r/#/c/25689/ | 12:05 |
*** amoralej is now known as amoralej|lunch | 12:06 | |
rlandy | oh we should remove the old jobs | 12:07 |
rlandy | I'll fix that | 12:08 |
*** jpena is now known as jpena|lunch | 12:09 | |
*** dpawlik has quit IRC | 12:14 | |
*** dpawlik has joined #oooq | 12:15 | |
rlandy | chandankumar: pls review https://review.opendev.org/#/c/711287/ | 12:15 |
rlandy | ie: vote | 12:15 |
*** dpawlik has quit IRC | 12:22 | |
chandankumar | rlandy, done | 12:23 |
chandankumar | rlandy, ah, I forgot | 12:23 |
rlandy | chandankumar: https://review.rdoproject.org/r/#/c/25700/ | 12:23 |
rlandy | chandankumar: I checked with weshay about fs002 - he said it could be removed but I just want to confirm that | 12:24 |
rlandy | ^^ patch can merge | 12:24 |
chandankumar | rlandy, ok | 12:24 |
chandankumar | sshnaidm, ^^ | 12:24 |
chandankumar | rlandy, sshnaidm https://review.rdoproject.org/r/#/c/25699/ | 12:24 |
rlandy | chandankumar: ^^ let's use that review to add new jobs to criteria | 12:25 |
rlandy | add all the ones we added | 12:25 |
chandankumar | except ovb we got all the jobs I think | 12:26 |
rlandy | chandankumar: fs002? yeah so I don;t know if we are still promoting anything | 12:26 |
rlandy | ie: changing the hash | 12:26 |
rlandy | if yes, I would leave fs002 - will confimr | 12:26 |
chandankumar | rlandy, arxcruz|rover filed a bug related to fs02 | 12:27 |
rlandy | chandankumar: yep, according to meeting, sshnaidm fixed it | 12:27 |
rlandy | if it's the same one | 12:27 |
arxcruz|rover | chandankumar: there is already a bug... | 12:27 |
chandankumar | rlandy, not yet | 12:28 |
chandankumar | rlandy, https://review.opendev.org/#/c/711081/ going to work on it tonight to make compatile to c8 | 12:28 |
ykarel | chandankumar, need to hold https://review.opendev.org/#/c/711287/3..5/zuul.d/standalone-jobs.yaml | 12:29 |
ykarel | wrong parent | 12:29 |
chandankumar | one min fixing it | 12:29 |
ykarel | ack | 12:30 |
chandankumar | ykarel, done | 12:30 |
chandankumar | ykarel, https://review.opendev.org/#/c/711287/ | 12:31 |
chandankumar | sshnaidm, https://review.opendev.org/#/c/711287/ +2 and +w | 12:31 |
ykarel | ok now | 12:31 |
chandankumar | rlandy, rlandy https://review.rdoproject.org/r/#/c/25698/1/zuul.d/projects.yaml is this one correct way of testing component jobs? | 12:32 |
rlandy | chandankumar: you don't want force_periodic | 12:33 |
rlandy | it should use current tripleo for other components | 12:34 |
rlandy | also promote_source should be defined | 12:34 |
chandankumar | promote_source: component-ci-testing will work? | 12:34 |
*** dpawlik has joined #oooq | 12:36 | |
chandankumar | rlandy, https://review.opendev.org/#/c/711408/ good to go | 12:46 |
rlandy | sec - adding jobs | 12:47 |
chandankumar | arxcruz|rover, weshay rlandy https://bugs.launchpad.net/tripleo/+bug/1866184 | 13:05 |
openstack | Launchpad bug 1866184 in tripleo "centos-8 multinode overcloud deploy failed with heat_resource_tree_params = heat_resource_tree['parameters'] KeyError: 'parameters'" [Critical,Confirmed] | 13:05 |
chandankumar | ykarel, ^^ | 13:05 |
weshay | chandankumar, please join | 13:06 |
weshay | panda, please join | 13:06 |
ykarel | chandankumar, ack | 13:06 |
arxcruz|rover | chandankumar: thanks, do you have a patch already ? just to add on the rr hackmd | 13:08 |
chandankumar | arxcruz|rover, nope | 13:08 |
*** udesale has joined #oooq | 13:14 | |
*** cgoncalves has quit IRC | 13:41 | |
weshay | rlandy, check if container-builds are usng ceph repos from rdo | 13:44 |
*** cgoncalves has joined #oooq | 13:45 | |
weshay | https://hackmd.io/HrQd03c9SxOMtFPFrq50tg?both | 13:48 |
weshay | chandankumar, https://review.opendev.org/711322 | 13:49 |
chandankumar | weshay, https://bugs.launchpad.net/tripleo/+bug/1866184 | 13:52 |
openstack | Launchpad bug 1866184 in tripleo "centos-8 multinode overcloud deploy failed with heat_resource_tree_params = heat_resource_tree['parameters'] KeyError: 'parameters'" [Critical,Confirmed] | 13:52 |
chandankumar | weshay, https://review.rdoproject.org/r/25612 | 13:59 |
chandankumar | rlandy, add the require jobs definition here https://review.rdoproject.org/r/#/c/25659/ and add the respective jobs here by editing .zuul.yaml file in this review https://review.rdoproject.org/r/#/c/25612/ and you might also need to rebase this review https://review.rdoproject.org/r/#/c/25551/ | 14:04 |
chandankumar | and we are done | 14:04 |
chandankumar | rlandy, please update the review only when jobs finishes | 14:05 |
chandankumar | it takes approx 6 hours to complete | 14:05 |
rlandy | ok- will check it out - thanks | 14:05 |
rlandy | chandankumar: can you take a look at the standalone tempest review ... jobs runs 6 hours - no logs | 14:17 |
rlandy | job times out in three places | 14:17 |
chandankumar | rlandy, sure, link please | 14:17 |
rlandy | https://review.rdoproject.org/r/#/c/25325/ | 14:17 |
chandankumar | rlandy, log collection is also timing out sudo unbound-control dump_cache | 14:19 |
chandankumar | sudo unbound-control dump_cache | 14:19 |
chandankumar | 2020-03-03 21:43:11.398921 | TASK [warn when collect logs timed out (SIGTERM or SIGKILL used)] | 14:19 |
chandankumar | 2020-03-03 21:43:11.476915 | ERROR: Collect logs timed out | 14:19 |
*** marios has joined #oooq | 14:25 | |
*** marios|call has quit IRC | 14:25 | |
rlandy | chandankumar: job times out in three places | 14:25 |
*** ykarel is now known as ykarel|away | 14:27 | |
chandankumar | arxcruz|rover, rlandy I have updated the concurrency let's see how it goes | 14:27 |
chandankumar | moved to 4 | 14:27 |
chandankumar | https://review.rdoproject.org/r/#/c/25325/ | 14:28 |
rlandy | chandankumar: thanks - will watch it | 14:28 |
* chandankumar headed home | 14:28 | |
*** jpena|lunch is now known as jpena | 14:29 | |
*** amoralej|lunch is now known as amoralej | 14:31 | |
*** Goneri has joined #oooq | 14:31 | |
*** TrevorV has joined #oooq | 14:32 | |
*** chem has joined #oooq | 14:33 | |
*** cgoncalves has quit IRC | 14:39 | |
*** cgoncalves has joined #oooq | 14:40 | |
rlandy | weshay: you moving the taiga tasks or should I ? | 14:49 |
weshay | rlandy, it's tricky.. there's a button for it.. let's do it together later so you see how | 14:50 |
rlandy | weshay: ack | 14:50 |
weshay | sshnaidm++ re: vexx perf email | 14:51 |
arxcruz|rover | weshay: what's the rhel8 issue you're talking about ? | 14:51 |
weshay | arxcruz|rover, aye. ./me looks sec | 14:51 |
arxcruz|rover | i see the rados one, that is waiting to get promoted | 14:51 |
rlandy | sshnaidm: what's the deal with scenario012? https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario012-standalone.yaml - still relevant for master? | 14:52 |
rlandy | with novaless deploy | 14:52 |
sshnaidm | rlandy, this is a poor job | 14:52 |
sshnaidm | rlandy, tried to make it working, but there is some bad stuff in heat templates probably | 14:53 |
rlandy | sshnaidm: ok - so we'll leave it for the baremetal component now ... | 14:53 |
rlandy | when there is some value, we can test it there | 14:53 |
sshnaidm | rlandy, I think it might be fine to have it, but somebody more familiar with tht should fix it finally | 14:53 |
rlandy | that's where we'll get the biggest value | 14:53 |
rlandy | sshnaidm: stevebaker? | 14:53 |
sshnaidm | rlandy, I think more about EmilienM :) | 14:54 |
sshnaidm | rlandy, I think he was already looking into it once, EmilienM right? (about standalone012 job) | 14:54 |
rlandy | it's been failing for a while - so really we are no worse off | 14:55 |
EmilienM | hi | 14:55 |
rlandy | but it would be great to have that in the baremetal component pipeline | 14:55 |
rlandy | EmilienM: hi ... | 14:55 |
rlandy | scenario012 | 14:55 |
rlandy | is there any worth in moving that to centos-8? | 14:56 |
weshay | arxcruz|rover, checking the latest deployments.. looks like we get through the deployment and just fail on the known bug.. | 14:56 |
weshay | arxcruz|rover, so I guess we're good | 14:56 |
rlandy | with the new noveless deploy? | 14:56 |
EmilienM | I haven't looked at standalone012 for a long time | 14:56 |
weshay | arxcruz|rover, it will fail until we promote.. which will be a little while | 14:56 |
rlandy | EmilienM: so it's been non-voting forever - so we really haven't had coverage there | 14:57 |
arxcruz|rover | weshay: ack | 14:57 |
rlandy | EmilienM: it's a test we considered adding to the baremetal component pipeline | 14:58 |
weshay | help w/ https://review.opendev.org/#/q/topic:centos-8-jobs+(status:open+OR+status:merged) | 14:58 |
rlandy | but it's not essential if we have OVB there | 14:58 |
weshay | sshnaidm, https://review.opendev.org/#/c/711320/ | 15:00 |
weshay | sshnaidm, rlandy https://review.opendev.org/#/c/710810/ | 15:03 |
*** d0ugal has joined #oooq | 15:07 | |
rlandy | weshay: comments added https://review.opendev.org/#/c/710810 | 15:08 |
*** jaosorior has quit IRC | 15:20 | |
weshay | rhanks | 15:32 |
weshay | thanks | 15:32 |
weshay | sshnaidm, bug on fs001 coming up? | 15:33 |
sshnaidm | weshay, yeah | 15:33 |
weshay | thanks | 15:33 |
sshnaidm | weshay, https://bugs.launchpad.net/tripleo/+bug/1866202 | 15:37 |
openstack | Launchpad bug 1866202 in tripleo "OVB on centos8 fails because of networking failures" [Critical,Triaged] | 15:37 |
marios | rlandy: so how does the coordination look like on https://hackmd.io/HrQd03c9SxOMtFPFrq50tg?both do you have sync calls? /me wondering where /how to get involved | 15:55 |
weshay | marios, we have nothing for scenario010-ovn-provider-standalone | 15:56 |
weshay | https://hackmd.io/HrQd03c9SxOMtFPFrq50tg?view#Standalone | 15:56 |
rlandy | marios: we were syncing every morning | 15:56 |
rlandy | I'll set that meeting up again | 15:56 |
marios | weshay: rlandy: ack | 15:58 |
rlandy | marios: I can also take you through the component pipeline and you can help there | 15:58 |
marios | rlandy: ack let me look at scen10 and i can join your next sync call and take from there. | 15:59 |
rlandy | marios: ack | 16:00 |
rlandy | weshay: setting up 1Pm UTC daily sync calls again - know that clashes with some of your other meetings. If you want to change that time, let me know | 16:01 |
*** Trevor_V has joined #oooq | 16:01 | |
*** dsneddon has joined #oooq | 16:01 | |
marios | sshnaidm: minor addition to README about ssh tunnel (I wasted some time on that today might be useful for others, like use RDO vm for grafana development) https://review.rdoproject.org/r/25706 | 16:04 |
*** TrevorV has quit IRC | 16:05 | |
marios | weshay: but scen10 ovn not for periodic right just check? | 16:06 |
marios | weshay: currently no periodic centos7 afaics | 16:06 |
weshay | marios, any voting check job requires a periodic | 16:06 |
rlandy | marios: chandankumar: weshay: panda: sync meeting invite sent for this sprint | 16:06 |
marios | weshay: its non voting so i assumed it was not in gate | 16:07 |
weshay | marios, k.. let's convert to centos-8 and add a periodic | 16:07 |
marios | weshay: and so i was not suprised to not find a periodic for it | 16:07 |
marios | weshay: ack so we do want it in gate & periodic for c8 | 16:07 |
weshay | marios, add it to criteria but # it out as non-voting atm | 16:08 |
marios | weshay: ack when i get there ^ | 16:08 |
weshay | marios, we want it in check and periodic, not gate | 16:08 |
marios | weshay: voting in check but not in gate? | 16:08 |
weshay | is it voting or non-voting in check atm? | 16:08 |
marios | weshay: non | 16:08 |
weshay | ok.. so leave it non-voting | 16:09 |
sshnaidm | marios, ack, btw you can just remove "127.0.0.1" here: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/docker-compose.yml#L50:L50 | 16:09 |
marios | weshay: oooci-jobs.sh: *** COLLECTED URLs *** http://zuul.openstack.org/job/tripleo-ci-centos-7-scenario010-ovn-provider-standalone http://codesearch.openstack.org/?q=tripleo-ci-centos-7-scenario010-ovn-provider-standalone https://github.com/openstack/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L595 http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-ovn-provider-standalone | 16:09 |
marios | weshay: sorry wrong paste | 16:09 |
marios | weshay: that https://github.com/openstack/tripleo-ci/blob/7c68cda8b4390a58b4bdbbfed2a2fd87584d25f6/zuul.d/standalone-jobs.yaml#L596 | 16:09 |
marios | (non voting) | 16:09 |
sshnaidm | marios, and then go to http://public_ip:8080 | 16:09 |
marios | sshnaidm: oh nice, that would have been easier ;) | 16:10 |
marios | sshnaidm: but i thin kyou'd still need the tunnel for the port from floating ip | 16:10 |
weshay | marios, is it clear or need clarification? | 16:10 |
sshnaidm | marios, no, just connect to floating ip, 8080 need to be opened by sec group | 16:11 |
marios | weshay: well :D some conflicting info, but i guess you want it in check only? | 16:11 |
marios | weshay: as is currently? | 16:11 |
weshay | in nonvoting-check, in periodic, no-gate job, in criteria but remarked out w/ a #non-voting atm comment | 16:12 |
marios | weshay: or non voting in check, like curently but with periodic not in creiteria | 16:12 |
marios | ack weshay ok ^ | 16:12 |
weshay | marios, people add jobs .. on or off the team.. and fail to add something to periodic | 16:12 |
weshay | fucks the system up :) | 16:12 |
marios | weshay: hmm... the job is in pretty bad shape by the way no green at all http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-ovn-provider-standalone | 16:14 |
marios | maybe thats why its not gating/voting | 16:14 |
weshay | it's because there is no periodic | 16:15 |
weshay | this happens all the time | 16:16 |
weshay | almost 100% when someone not on our team adds jobs | 16:16 |
chandankumar | rlandy, weshay panda I tested the emit release patch here https://review.rdoproject.org/r/#/c/25694/ | 16:16 |
chandankumar | but periodic-tripleo-ci-centos-8-singlenode-featureset027-masterTIMED_OUT in 4h 56m 16s | 16:17 |
chandankumar | periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-masterSUCCESS in 1h 47m 14s | 16:17 |
chandankumar | sshnaidm, can you take a look at collect logs https://logserver.rdoproject.org/94/25694/1/check/periodic-tripleo-ci-centos-8-singlenode-featureset027-master/782dec9/job-output.txt | 16:18 |
chandankumar | after timeout logs are not getting collected | 16:18 |
rlandy | ok | 16:18 |
chandankumar | warn when collect logs timed out (SIGTERM or SIGKILL used | 16:18 |
chandankumar | seen in full tempest and these jobs | 16:18 |
weshay | chandankumar, that is the fastest fs10/containers-multinode I've ever seen | 16:19 |
weshay | I hope that happens upstream | 16:19 |
weshay | chandankumar, so we clear to merge https://review.opendev.org/#/c/710441/ | 16:20 |
weshay | if hte timeout was logs.. I would say yes | 16:20 |
chandankumar | ok | 16:20 |
weshay | chandankumar, please vote | 16:20 |
weshay | wf if you have the votes | 16:21 |
weshay | panda, ^ | 16:21 |
rlandy | https://review.opendev.org/#/c/711305/ - should that be w+'ed? | 16:27 |
rlandy | chandankumar: ^^ | 16:28 |
chandankumar | rlandy, yes, as this is a dependent on emit release | 16:28 |
chandankumar | rlandy, will I not? | 16:28 |
weshay | chandankumar, fyi http://codesearch.openstack.org/?q=tripleo-ci-centos-7-containers-undercloud-minion&i=nope&files=&repos= | 16:29 |
weshay | chandankumar, want me to update those repos? | 16:29 |
chandankumar | weshay, I will take care of that tomorrow | 16:29 |
weshay | chandankumar, I'll do it now | 16:29 |
chandankumar | weshay, ok | 16:29 |
sshnaidm | chandankumar, this jobs is messed up completely | 16:30 |
chandankumar | sshnaidm, sorry which one minion? | 16:30 |
marios | rlandy: i see you're working on the scen10 standalone... https://review.opendev.org/#/c/711287/6/zuul.d/standalone-jobs.yaml any idea what the difference with the ovn-provider one is ? | 16:30 |
* marios looking at definitions cant see it | 16:30 | |
sshnaidm | chandankumar, seems like failed in sudo unbound-control dump_cache, if it doesn't happen again just ignore it | 16:30 |
rlandy | chandankumar: https://review.opendev.org/#/c/710441/ is not in | 16:30 |
marios | rlandy: same featureset and vars afaics https://github.com/openstack/tripleo-ci/blob/7c68cda8b4390a58b4bdbbfed2a2fd87584d25f6/zuul.d/standalone-jobs.yaml#L575-L595 | 16:31 |
marios | only the tempest differs | 16:31 |
rlandy | the depends-on is the wrong way around | 16:31 |
marios | weshay: do you know (difference between scen10 and scen10 ovn) | 16:31 |
rlandy | chandankumar: pls check that | 16:31 |
rlandy | marios: getting | 16:31 |
weshay | marios, brent put up that job... I don't have much background on it | 16:32 |
chandankumar | rlandy, un workflowed that | 16:32 |
rlandy | chandankumar: we put the depends-on the wrong way :( | 16:32 |
marios | weshay: k i can try reach out to him too.. but afaics they have same featureset, same standalone environment file. only tempest differs | 16:33 |
chandankumar | fixing that | 16:33 |
marios | weshay: https://github.com/openstack/tripleo-ci/blob/7c68cda8b4390a58b4bdbbfed2a2fd87584d25f6/zuul.d/standalone-jobs.yaml#L575-L595 | 16:33 |
rlandy | marios: take a look at ^^ | 16:34 |
chandankumar | rlandy, fixed | 16:34 |
marios | rlandy: yeah? | 16:34 |
* marios playing spot the difference | 16:35 | |
marios | tempest differs yes but something else i missed rlandy ? | 16:35 |
rlandy | marios: yep - seems like just tempest | 16:36 |
marios | rlandy: ack thanks for sanity check i can reach out to beagles | 16:36 |
marios | beagles: (we are trying to understand difference between scen10 standalone and scen10 ovn-provider standalone.. looks like just tempest difference? do we really need both jobs? | 16:36 |
chandankumar | rlandy, is it about scenario 10? | 16:37 |
marios | chandankumar: yes | 16:37 |
chandankumar | rlandy, there we run validate-tempest | 16:37 |
rlandy | marios: those jobs are a bit of a mystery - if you compare rdo and check - they are actually different | 16:37 |
chandankumar | so stats might vary | 16:37 |
rlandy | you'll note I changed the centos-8 one | 16:37 |
beagles | marios, yes we need both because the tempest configs are distinct and afaik we aren't able to run tempest twice in the same CI job | 16:37 |
marios | rlandy: chandankumar: k i'll post the centos8 see what happens and same time try to understand the difference or if we can combine the two jobs eventually | 16:37 |
chandankumar | rlandy, only fs I thinj | 16:37 |
marios | beagles: ack thanks for confirming | 16:37 |
rlandy | chandankumar: yeah ... | 16:38 |
marios | well there we go then | 16:38 |
rlandy | I changed it to 062 for two reasons | 16:38 |
rlandy | 1. carlos commented | 16:38 |
rlandy | 2. ykarel noticed the jobs was missing pieces | 16:38 |
rlandy | 062 seemed more complete but old | 16:38 |
rlandy | marios: ^ may need an update in general | 16:39 |
marios | rlandy: ack gonna start with job definition + testproject run and see where it breaks ;) | 16:40 |
rlandy | cool | 16:41 |
rlandy | checking out the component fails - no idea what hit there | 16:41 |
*** udesale has quit IRC | 16:44 | |
*** Trevor_V has quit IRC | 16:47 | |
* marios hometime | 16:47 | |
marios | panda: ack on the invite for tomorrow | 16:48 |
*** ccamacho has quit IRC | 16:48 | |
*** TrevorV has joined #oooq | 16:50 | |
chandankumar | weshay, rlandy based on today's result on this review https://review.rdoproject.org/r/#/c/25612/ is it ok, unpin few packages and move it to a seperate review and keep testing with others? | 16:57 |
weshay | chandankumar, NOOO | 16:57 |
weshay | chandankumar, you mean merge? | 16:57 |
weshay | or just split the tests? | 16:58 |
rlandy | I think chandankumar means unpin a few in a review | 16:58 |
rlandy | not upin and let it fly | 16:58 |
weshay | that's fine | 16:58 |
weshay | chandankumar, don't freak me out man | 16:58 |
weshay | :) | 16:58 |
rlandy | waiting on https://review.opendev.org/#/c/711156/ | 16:58 |
chandankumar | weshay, based on job results we can break this giant review https://review.rdoproject.org/r/#/c/25612/ in small chunck | 16:58 |
weshay | chandankumar++ | 16:58 |
weshay | that's fine.. | 16:59 |
*** marios is now known as marios|out | 16:59 | |
chandankumar | then if anythign got missed, we have componenet pipeline to check | 16:59 |
rlandy | weshay: pls check https://review.rdoproject.org/r/#/c/25702/ | 16:59 |
weshay | just don't merge until we have it planned :) | 16:59 |
chandankumar | otherwise we are delaying the pining | 16:59 |
chandankumar | *unpinning | 16:59 |
rlandy | yep - https://review.rdoproject.org/r/#/c/25702/ will merge as soon as depends-on does | 16:59 |
rlandy | need tempest and ovb | 17:00 |
*** tesseract-RH has quit IRC | 17:01 | |
weshay | rlandy++ | 17:02 |
*** bogdando has quit IRC | 17:06 | |
chandankumar | rlandy, https://logserver.rdoproject.org/25/25325/14/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-master/333fe90/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz | 17:14 |
chandankumar | with concurreny 4 job ended in less than 3 hours | 17:14 |
chandankumar | but we have 2 failures | 17:14 |
rlandy | chandankumar++ | 17:15 |
rlandy | that job was driving me crazy before with no logs | 17:16 |
rlandy | Details: {'type': 'IpAddressGenerationFailure', 'message': 'No more IP addresses available on network 2a5727e8-73b3-4191-82dd-51ed66fb8d3e.', 'detail': ''} | 17:16 |
rlandy | ^^ does that look like a temp failure? | 17:16 |
chandankumar | rlandy, may be race condition or resource leakage also | 17:17 |
rlandy | chandankumar: ok - going to spin more | 17:17 |
rlandy | but merge jobs review | 17:17 |
rlandy | chandankumar: thoughts about a component skip list? | 17:17 |
weshay | rlandy, ya.. that's harder than you think | 17:18 |
weshay | chandankumar, rlandy let's section out a part in the current skip list file for component failures... | 17:18 |
chandankumar | rlandy, skip list is a big topic | 17:18 |
weshay | but do not add that feature yet | 17:18 |
chandankumar | I will drop a mail tomorrow what is my thoughts on that | 17:18 |
rlandy | chandankumar: ack that | 17:18 |
weshay | but we can diliniate in text in case we have time in future sprints | 17:18 |
rlandy | I'll spin again to see if that failure is consistent | 17:19 |
weshay | let's just add these to the normal skip list and bug it, cix etc | 17:19 |
chandankumar | weshay, rlandy for current situation , replace all concurrency to 3 or 4 | 17:19 |
weshay | rlandy, aye.. twice is good | 17:19 |
rlandy | but it'd good enough to add the job definition | 17:19 |
weshay | chandankumar, I would think 4 | 17:19 |
chandankumar | less test 3 and more tests keep it 4 | 17:19 |
weshay | rlandy++++ | 17:19 |
weshay | yes | 17:19 |
rlandy | weshay: that's chandankumar | 17:19 |
rlandy | s ++ | 17:19 |
rlandy | not mine :) | 17:19 |
weshay | yes.. chandankumar+++++++++++++++++++++++ | 17:19 |
weshay | as well. | 17:19 |
weshay | rlandy, but add it to the component pipeline now | 17:19 |
weshay | and we'll add the skips | 17:20 |
weshay | if needed | 17:20 |
rlandy | weshay: ack - merging job definition and spinning | 17:20 |
chandankumar | rlandy, trick is here https://review.rdoproject.org/r/#/c/25325/14/.zuul.yaml | 17:20 |
weshay | chandankumar, we could make the skip per component a seperate file and set the var in zuul | 17:20 |
chandankumar | jsut append 2 lines you are done | 17:20 |
rlandy | we can assess the skip thing tomorrow | 17:20 |
weshay | but we'll cross that bridge | 17:20 |
weshay | chandankumar, we'll run into trouble if components have overlapping temest tests | 17:21 |
weshay | which they will | 17:21 |
rlandy | spin 2 in progress | 17:21 |
weshay | we closing out work faster than we can plan | 17:21 |
weshay | ++++ | 17:21 |
rlandy | don't worry upstream zuul is slowing us down bit | 17:21 |
weshay | ha | 17:21 |
weshay | rlandy, so far the zuul forecast today is pretty sunny | 17:22 |
weshay | we just need to get contianers-multinode.. fixed and voting asap | 17:22 |
weshay | because devels LOVE to break that job | 17:22 |
rlandy | weshay: sat a small prayer for https://review.opendev.org/#/c/711156/ | 17:22 |
rlandy | say | 17:22 |
rlandy | which one id the fix review? | 17:23 |
rlandy | is | 17:23 |
weshay | rlandy, we'll know in 3hours | 17:24 |
weshay | 711480 | 17:24 |
weshay | fyi.. my list gets longer https://review.opendev.org/#/q/topic:centos-8-jobs+(status:open+OR+status:merged) | 17:24 |
chandankumar | weshay, pass it to me I will take care in morning, sorry not in a mood to smash the patches | 17:25 |
weshay | chandankumar, no prob man | 17:25 |
weshay | chandankumar, I have minion added | 17:25 |
chandankumar | weshay, minions merged na? | 17:25 |
weshay | yes | 17:25 |
chandankumar | I am watching this https://review.rdoproject.org/zuul/stream/3b94973fe3a64bc48cd4109549b70dc2?logfile=console.log | 17:26 |
weshay | Change has been successfully merged by Zuul | 17:26 |
chandankumar | weshay, after this patch, we need to coordinate to deal with neutron unpinning and ovn repo mess | 17:26 |
weshay | k | 17:27 |
chandankumar | weshay, rlandy for neutron we need to get all secnario in component which we run in check | 17:27 |
chandankumar | network is always a problem | 17:28 |
rlandy | chandankumar: please comment on the hackmd in the networking component line and I'll fix it | 17:28 |
chandankumar | rlandy, ok | 17:28 |
rlandy | chandankumar: thanks - just for tracking here | 17:28 |
rlandy | as we will share that with the dfs | 17:29 |
*** marios|out has quit IRC | 17:35 | |
*** holser has quit IRC | 17:36 | |
*** dtantsur is now known as dtantsur|afk | 17:40 | |
*** derekh has quit IRC | 18:00 | |
chandankumar | rlandy, thanks for pushing the criteria | 18:05 |
rlandy | sure | 18:05 |
rlandy | list updated in hackmd | 18:06 |
rlandy | chandankumar: dude, don't you sleep?? it's late | 18:07 |
chandankumar | rlandy, will be sleeping soon | 18:07 |
rlandy | :) | 18:07 |
chandankumar | last night didnot got too much sleep due to early morning Pycon U.S. Code of Conduct Incident Response workshop it was at 4:30 AM | 18:08 |
rlandy | wow | 18:09 |
rlandy | there is no Pycon India Code of Conduct Incident Response workshop? | 18:09 |
chandankumar | rlandy, nope, we are going to create that and taking it from US pycon | 18:10 |
chandankumar | for PyCon India 2020 | 18:10 |
chandankumar | Thanks to Sage sharp for the workshop, It was very hard to record the report when you hear something wrong has happened to someone in confernce and very hard to take measures to follow up ad take proper action so that it does not happen again without revealing the identity of reporter | 18:12 |
chandankumar | see ya | 18:17 |
chandankumar | weshay, rlandy drop comments, will take over tomorrow | 18:18 |
*** chandankumar is now known as raukadah | 18:18 | |
rlandy | raukadah: ack - will have tempest review for you | 18:19 |
rlandy | thanks | 18:19 |
*** ccamacho has joined #oooq | 18:20 | |
*** Goneri has quit IRC | 18:23 | |
*** jbadiapa has quit IRC | 18:23 | |
*** jpena is now known as jpena|off | 18:26 | |
*** Goneri has joined #oooq | 18:27 | |
*** amoralej is now known as amoralej|off | 18:35 | |
raukadah | weshay, rlandy amoralej|off ykarel|away Here is the status report on upinning https://review.rdoproject.org/r/25612 standalone plain 2,3,4 ,7,10 undercloud passed, | 18:44 |
raukadah | full tempest still running | 18:45 |
raukadah | fs01 and fs010 known failure https://logserver.rdoproject.org/12/25612/10/check/distgit-tripleo-master-testing-centos-8-multinode-1ctlr-featureset010/7aee69a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 18:45 |
raukadah | so we are good there | 18:45 |
weshay | very cool | 18:45 |
weshay | raukadah, don't work on https://review.opendev.org/#/c/711081/ | 18:45 |
weshay | btw | 18:45 |
raukadah | weshay, aye | 18:46 |
weshay | rlandy, raukadah sshnaidm we're going to "to_build: false" in centos-8 for now for ovb jobs | 18:46 |
raukadah | weshay, but I need to find a way to enjict current role in tq | 18:46 |
weshay | raukadah, it's easy. I'll wip a patch | 18:46 |
weshay | raukadah, but really a distraction atm | 18:47 |
raukadah | weshay, ok, I will take it tomorrow | 18:47 |
raukadah | rlandy, on emit release https://logserver.rdoproject.org/94/25694/2/check/periodic-tripleo-ci-centos-8-singlenode-featureset027-master/b8de8cd/job-output.txt | 18:49 |
raukadah | still timeout at tempest | 18:49 |
raukadah | something wrong there needs to be investigated | 18:50 |
weshay | raukadah, don't worry about images for now.. focus on the pinning | 18:50 |
weshay | and other centos8 work | 18:50 |
raukadah | weshay, we got awesome mate rlandy so no worry and marios joining the club also | 18:51 |
raukadah | and sshnaidm & panda in background | 18:51 |
panda | weshay: raukadah all green again in https://review.rdoproject.org/r/24774, and code coverage on the promoted (not the stage) is above 80%. I'll remove the WIP tomorrow and we are ready for reviews. | 18:53 |
*** jaosorior has joined #oooq | 19:08 | |
rlandy | nice | 19:10 |
rlandy | raukadah: I think I know what's wrong with images | 19:11 |
rlandy | checking if fix went through | 19:12 |
rlandy | https://review.opendev.org/#/c/711408/ | 19:17 |
rlandy | raukadah: ^^ should fix image build - checking | 19:18 |
sshnaidm | weshay, I think we can stop running 039 on master, right? | 19:27 |
weshay | sshnaidm, nope.. those fuckers got it working | 19:27 |
weshay | :) | 19:27 |
weshay | lolz | 19:27 |
sshnaidm | :( | 19:27 |
weshay | leave it for now.. just switch it | 19:27 |
weshay | if it doesn't work.. fine.. but let's keep it | 19:27 |
sshnaidm | ok | 19:27 |
raukadah | rlandy, thanks | 19:28 |
rlandy | tripleo-ci-centos-7-containers-multinode-steinFAILURE in 3h 14m 55s | 19:28 |
rlandy | UGH | 19:28 |
rlandy | reall??? | 19:29 |
rlandy | https://review.opendev.org/#/c/711156/ | 19:29 |
raukadah | fs039 is still a thing, i think we are dropping in favor of tls standalone | 19:29 |
rlandy | back again in check | 19:29 |
weshay | rlandy, 2020-03-05 19:08:34 | 2020-03-05 19:08:17Z [overcloud-Controller-uzibk2hhweqt.0]: UPDATE_FAILED resources[0]: Parameter 'ControllerImage' is invalid: Error validating value 'overcloud-full': No images unique match found for {'name': u'overcloud-full'}. | 19:30 |
rlandy | weshay: where did ^^ come from? | 19:31 |
weshay | rlandy, ur failing job on stein | 19:31 |
rlandy | weshay: same job passed in check | 19:31 |
weshay | aye.. it would have to | 19:31 |
rlandy | tripleo-ci-centos-7-containers-multinode-stein SUCCESS in 2h 30m 21s | 19:32 |
rlandy | so now?? | 19:32 |
weshay | rlandy, I know.. gates happen | 19:38 |
weshay | sorry? | 19:38 |
weshay | tripleo is phunny sometimes/ | 19:39 |
weshay | ? | 19:39 |
rlandy | weshay: not your fault :) | 19:40 |
weshay | ya.. but frustrating.. | 19:41 |
rlandy | going to work on the updates job - will forget about that for a few hours | 19:41 |
weshay | DAM https://review.opendev.org/#/c/711480/ just failed in check :) | 19:41 |
*** saneax has quit IRC | 19:42 | |
rlandy | weshay: check it out ... periodic-tripleo-ci-centos-8-standalone-full-tempest-mastertestprojectmastercheck25325,14106002020-03-05T17:22:58SUCCESS | 20:23 |
rlandy | https://review.rdoproject.org/zuul/build/0085ea6c14c7469f80149f4b94aa0463 | 20:24 |
rlandy | is that legit passing??? | 20:24 |
rlandy | merging https://review.rdoproject.org/r/#/c/25717/ | 20:24 |
weshay | rlandy, LEGIT! Status: Pass 1524 Skip 172 | 20:26 |
rlandy | ok - component pipeline here we come | 20:27 |
* rlandy is spending too much time texting with my nephew ... picking up his favorite word 'legit' | 20:28 | |
weshay | too legit to quit | 20:43 |
*** TrevorV has quit IRC | 20:44 | |
*** sshnaidm is now known as sshnaidm|afk | 20:48 | |
*** cgoncalves has quit IRC | 20:53 | |
*** cgoncalves has joined #oooq | 20:56 | |
*** jfrancoa has quit IRC | 20:57 | |
*** cgoncalves has quit IRC | 21:10 | |
rlandy | weshay: https://review.rdoproject.org/r/#/c/25416/ | 21:10 |
*** cgoncalves has joined #oooq | 21:11 | |
*** dpawlik has quit IRC | 21:18 | |
*** dsneddon has quit IRC | 21:48 | |
*** jmasud has quit IRC | 22:26 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!