Thursday, 2019-12-19

*** irclogbot_3 has quit IRC02:56
*** irclogbot_0 has joined #oooq03:08
*** irclogbot_0 has quit IRC03:09
*** irclogbot_1 has joined #oooq03:14
*** rlandy|bbl is now known as rlandy03:20
*** rlandy has quit IRC03:25
*** dsneddon has quit IRC03:27
*** dsneddon has joined #oooq03:28
*** dsneddon has quit IRC03:36
*** holser has joined #oooq03:43
*** ykarel|away has joined #oooq04:05
*** holser has quit IRC04:16
*** apetrich has quit IRC04:29
*** ykarel|away has quit IRC04:30
*** ykarel has joined #oooq04:45
*** whoami-rajat has joined #oooq04:47
*** dsneddon has joined #oooq04:52
*** holser has joined #oooq04:57
*** bhagyashris has joined #oooq04:59
*** whoami-rajat has quit IRC05:03
*** whoami-rajat__ has joined #oooq05:04
*** epoojad1 has joined #oooq05:14
*** bhagyashris has quit IRC06:00
*** raukadah is now known as chandankumar06:06
*** bhagyashris has joined #oooq06:06
*** holser has quit IRC06:09
*** soniya29 has joined #oooq06:12
*** surpatil has joined #oooq06:17
*** surpatil has quit IRC06:18
*** surpatil has joined #oooq06:19
*** whoami-rajat__ has quit IRC06:28
*** marios has joined #oooq06:29
*** bhagyashris has quit IRC06:31
*** saneax has joined #oooq06:31
*** bhagyashris has joined #oooq06:38
*** jfrancoa has joined #oooq06:53
*** ratailor has joined #oooq07:18
*** jbadiapa has joined #oooq07:20
*** SurajPatil has joined #oooq07:23
*** surpatil has quit IRC07:26
*** whoami-rajat has joined #oooq07:26
*** saneax has quit IRC07:43
*** irclogbot_1 has quit IRC07:44
*** irclogbot_1 has joined #oooq07:44
*** ratailor has quit IRC07:45
*** ratailor has joined #oooq07:45
*** ykarel is now known as ykarel|lunch07:48
*** surpatil has joined #oooq07:49
*** SurajPatil has quit IRC07:52
*** ratailor has quit IRC07:58
*** ratailor has joined #oooq07:58
*** kopecmartin|off is now known as kopecmartin07:58
*** dsneddon has quit IRC08:00
*** ccamacho has joined #oooq08:13
*** tosky has joined #oooq08:14
*** dsneddon has joined #oooq08:20
*** saneax has joined #oooq08:20
*** bhagyashris has quit IRC08:24
*** dsneddon has quit IRC08:26
*** SurajPatil has joined #oooq08:26
*** surpatil has quit IRC08:29
*** surpatil has joined #oooq08:31
*** SurajPatil has quit IRC08:34
*** jpena|off is now known as jpena08:38
mariosbiab08:40
*** marios has quit IRC08:40
*** amoralej|off is now known as amoralej08:44
*** ratailor_ has joined #oooq08:47
*** apetrich has joined #oooq08:47
*** ratailor has quit IRC08:49
*** ykarel|lunch is now known as ykarel08:50
*** dsneddon has joined #oooq09:00
*** jbadiapa has quit IRC09:13
*** jbadiapa has joined #oooq09:13
*** dsneddon has quit IRC09:14
*** bhagyashris has joined #oooq09:16
*** ykarel is now known as ykarel|afk09:16
*** SurajPatil has joined #oooq09:18
*** surpatil has quit IRC09:20
*** epoojad1 has quit IRC09:24
*** dsneddon has joined #oooq09:29
*** dsneddon has quit IRC09:39
*** derekh has joined #oooq09:41
*** ykarel|afk is now known as ykarel09:42
*** marios has joined #oooq09:51
*** holser has joined #oooq10:04
*** holser has quit IRC10:05
*** tosky_ has joined #oooq10:07
*** holser has joined #oooq10:07
*** tosky is now known as Guest9772710:07
*** tosky_ is now known as tosky10:07
*** Guest97727 has quit IRC10:08
chandankumarmarios: \o/10:15
chandankumarmarios: How are you?10:15
*** chem has joined #oooq10:15
*** dsneddon has joined #oooq10:15
marioschandankumar: \o/10:16
marioschandankumar: better :) you?10:16
chandankumarmarios: yes doing good10:17
*** dsneddon has quit IRC10:26
*** d0ugal has quit IRC10:28
*** d0ugal has joined #oooq10:31
chandankumarmarios: needs +w on this https://review.rdoproject.org/r/#/c/24261/10:36
marioschandankumar: minor comments there (commit message not that important but found it really hard to parse sorry) but mainly 'check-github' vs 'github-check'10:46
marioschandankumar: if you disagree with that ^^^ then we can merge i won't block on it10:52
*** dsneddon has joined #oooq10:53
chandankumarcheck-github sounds fine10:58
*** surpatil has joined #oooq11:02
*** dsneddon has quit IRC11:04
*** SurajPatil has quit IRC11:04
*** SurajPatil has joined #oooq11:10
*** surpatil has quit IRC11:12
*** sshnaidm|afk is now known as sshnaidm11:21
*** surpatil has joined #oooq11:32
*** SurajPatil has quit IRC11:35
*** dsneddon has joined #oooq11:39
*** dsneddon has quit IRC11:48
*** epoojad1 has joined #oooq11:49
*** epoojad1 is now known as epoojad1|appoint11:51
*** epoojad1|appoint is now known as epoojad111:51
*** epoojad1 is now known as epoojad1|appoint11:51
*** epoojad1|appoint is now known as epoojad1|apointm11:51
*** epoojad1|apointm is now known as epoojad1|away11:51
*** epoojad1|away has quit IRC11:58
*** jpena is now known as jpena|lunch11:58
*** dsneddon has joined #oooq11:59
*** amoralej is now known as amoralej|lunch12:00
*** dsneddon has quit IRC12:04
*** ratailor_ has quit IRC12:05
*** epoojad1 has joined #oooq12:09
chandankumarsshnaidm: can we merget https://review.rdoproject.org/r/#/c/24260/ and https://review.rdoproject.org/r/#/c/24261/12:20
chandankumarhttps://review.rdoproject.org/r/#/c/23785/ https://review.rdoproject.org/r/#/c/23798/ https://review.rdoproject.org/r/#/c/23799/12:20
*** bhagyashris_ has joined #oooq12:21
sshnaidmchandankumar, don't you want "recheck" here? https://review.rdoproject.org/r/#/c/24261/5/zuul.d/github.yaml12:22
*** bhagyashris has quit IRC12:23
chandankumarsshnaidm: nope12:24
chandankumarsshnaidm: recheck will auto triggering jobs at two places12:24
chandankumarthen12:24
sshnaidmchandankumar, recheck-github?12:25
chandankumarsshnaidm: recheck-github, let me add that also12:25
sshnaidmchandankumar, or it will just a dup?12:25
sshnaidmwhen you run check-github, it should recheck also, isn't it?12:25
chandankumarsshnaidm: yes, it also recheck, just like check-rdo12:26
sshnaidmchandankumar, ok, then no need12:26
chandankumarI think check-github is enough12:26
*** bhagyashris_ has quit IRC12:27
*** bhagyashris has joined #oooq12:28
*** SurajPatil has joined #oooq12:29
*** surpatil has quit IRC12:31
*** surpatil has joined #oooq12:31
*** SurajPatil has quit IRC12:34
*** ykarel is now known as ykarel|afk12:36
*** dsneddon has joined #oooq12:38
chandankumarsshnaidm: thanks,12:39
* chandankumar relocates12:39
*** ykarel|afk has quit IRC12:41
*** dsneddon has quit IRC12:42
*** rlandy has joined #oooq12:57
*** bhagyashris has quit IRC12:59
*** Goneri has quit IRC13:00
rlandyare we meeting?13:03
sshnaidmrlandy, only and marios were there, so we dropped13:04
chandankumarI am here now13:04
mariosk lets rejoin13:04
rlandychandankumar: lol - now I dropped13:04
rlandyk - rejoining13:04
marioshaha13:04
marioschandankumar: no you're not13:05
marios:D13:05
*** jpena|lunch is now known as jpena13:11
chandankumarrlandy: https://review.opendev.org/#/c/699438/13:14
*** epoojad1 has quit IRC13:18
*** dsneddon has joined #oooq13:18
*** dsneddon has quit IRC13:23
*** skramaja has quit IRC13:26
chandankumarsshnaidm: rlandy https://bugs.launchpad.net/tripleo/+bug/185582613:31
openstackLaunchpad bug 1855826 in tripleo "RHEL 8 master fs01 fails while getting image expected checksum after multiple retry" [Critical,Fix released]13:31
*** amoralej|lunch is now known as amoralej13:34
chandankumarrlandy:https://review.opendev.org/#/c/698211/ and https://review.opendev.org/#/c/698214/ are also good to go, when feel comfortable merge it13:39
rlandychandankumar: w+'ed https://review.opendev.org/#/c/698211/13:40
rlandywill w the other when the gate queue clears13:40
chandankumarmarios: rlandy: no cix call on monday and wed fyi13:41
rlandychandankumar: ack13:41
marioschandankumar: yah saw it thanks13:41
rlandychandankumar: so this is the failure I saw yesterday ... deployment failed in fs001 rhel master ... http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/6d05252/job-output.txt13:43
rlandybut we ave no underloud logs13:43
rlandyhttp://logs.rdoproject.org/53/21853/19/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/c1e73a2/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz13:44
rlandythat is the prep-images timeout13:45
rlandywe had a bug for that13:45
rlandyhttps://bugs.launchpad.net/tripleo/+bug/184679113:46
openstackLaunchpad bug 1846791 in tripleo "race, openstack overcloud node introspect --all-manageable --provide may hang" [Medium,Triaged]13:46
*** Goneri has joined #oooq13:46
*** fmount has quit IRC13:46
chandankumaropenstack: yes13:49
chandankumarrlandy: yes13:49
rlandywe see that every now and again13:49
chandankumarrlandy: we made some fixes on selinux side13:49
*** dsneddon has joined #oooq13:50
chandankumarnow we have only dbus related denials13:51
chandankumarTengu: please have a look at this denials logs.rdoproject.org/53/21853/19/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/c1e73a2/logs/undercloud/var/log/extra/denials.taxt.gz13:51
chandankumarif around then13:51
Tenguchandankumar: lemme check13:52
Tengualready known13:52
Tenguchandankumar: https://bugzilla.redhat.com/show_bug.cgi?id=178394713:52
openstackbugzilla.redhat.com bug 1783947 in openstack-selinux "Do not log "send_msg" denials from container_t" [Medium,Closed: wontfix] - Assigned to jpichon13:52
Tenguit's most likely linked to collectd doing some unholy things with journalctl|systemctl in order to get healthcheck status for the containers.13:53
chandankumarrlandy: ^^13:53
Tenguchandankumar: It's linked to my current work: https://review.opendev.org/#/q/topic:healthcheck/SyslogIdentifier13:53
Tengurlandy: -^^13:53
chandankumarTengu: thanks :-)13:53
Tengunp :)13:53
Tengucontainer_t was allowed to access host dbus service (connectto), due to the collectd thing some time ago.13:54
Tenguand since this is a security issue, I'm working in order to close it while providing a better way - so that send_msg should disapear once we get the 3 patches merged + collectd "check" changed (crossing fingers)13:55
Tenguso probably beginning next year.13:55
rlandychandankumar: Tengu: thanks - will watch that13:55
*** dsneddon has quit IRC13:55
Tengu:)13:56
*** surpatil has quit IRC14:10
*** ykarel|afk has joined #oooq14:15
*** fmount has joined #oooq14:18
*** ykarel|afk is now known as ykarel14:18
*** dsneddon has joined #oooq14:22
*** tosky_ has joined #oooq14:23
*** tosky is now known as Guest2789014:23
*** tosky_ is now known as tosky14:23
*** TrevorV has joined #oooq14:25
*** Guest27890 has quit IRC14:26
*** dsneddon has quit IRC14:27
*** soniya29 has quit IRC14:34
*** rfolco has joined #oooq14:43
rfolcorlandy, o/14:44
rfolcorlandy, can we merge https://review.rdoproject.org/r/#/c/24250/14:44
rlandydone14:46
rlandymigi: hi ...14:50
rlandymigi: downstream component compute and keystone pipelines are up14:51
*** derekh has quit IRC14:54
rfolcorlandy, thx14:55
migirlandy: wow15:01
migirlandy: that is great, I am having troubles building RPMs on rhel8.1 atm due to some rhpkg incompability15:02
rlandymigi: k - I am here today, tomorrow and monday of you want my help15:03
*** dsneddon has joined #oooq15:03
rlandymigi, you can follow the promote-to-tripleo-ci-testing and dlrn reporting15:03
rlandyyour jobs will need those pieces15:03
*** jpena is now known as jpena|off15:04
*** Trevor_V has joined #oooq15:08
*** dsneddon has quit IRC15:08
*** TrevorV has quit IRC15:11
*** jpena|off is now known as jpena15:13
Tenguok folks, signing off for the year ;). Try to not break SELinux durig the next two weeks :]15:25
*** dsneddon has joined #oooq15:37
*** dsneddon has quit IRC15:44
chandankumarTengu: Happy Holidays, see ya next year :-)15:45
rlandymarios: sshnaidm: added readme section for sanitize_logs ... https://review.opendev.org/#/c/69199415:55
chandankumarrlandy: any thing need from my side?15:56
rlandychandankumar: I don't think so - have a good vacation15:57
rlandyjust rerunning the train image build that failed15:57
chandankumarrlandy: we need to wait for 2 hours, the tripleo gate queue will drop15:57
rlandychandankumar: gate?15:57
chandankumarrlandy: http://zuul.openstack.org/status15:58
rlandyyeah - watching that15:58
*** dsneddon has joined #oooq15:59
mariosrlandy: thanks can i look forward to it for breakfast?16:09
mariosrlandy: (will review tomorrow if still around thanks)16:10
mariosrlandy: thanks for the readme addition16:10
mariosrlandy: nm lgtm but zuul didn't vote yet16:12
mariosttyl folks16:20
*** marios is now known as marios|out16:20
chandankumarmarios|out: Have a  great days head, please enjoy the vacation and finish stocked * 2 mozito from my side16:20
marios|outu 2 chandankumar16:21
* marios|out throws 2 mojito back 16:21
* chandankumar grabs it16:23
chandankumarcheers marios|out16:23
rlandythanks16:25
*** marios|out has quit IRC16:29
*** ykarel is now known as ykarel|away16:32
chandankumarrlandy: http://logs.rdoproject.org/58/23558/10/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train/9b22f5e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz we are now able to reproduce train fs01 rhel 8 bug16:42
chandankumarrlandy: https://bugs.launchpad.net/tripleo/+bug/185397816:43
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged]16:43
rfolcorlandy, next fix https://review.rdoproject.org/r/#/c/24273/16:45
rlandywell it's repeatable16:45
chandankumarrlandy: may be weshay didnot created train release tags16:46
chandankumarrlandy: above class is removed in master from puppet-tripleo16:46
rlandysec - reading back on bug16:47
chandankumarrlandy: https://review.opendev.org/#/q/owner:%22wes+hayutin+%253Cwhayutin%2540redhat.com%253E%22+project:openstack/releases may be we need final tags for train16:48
rlandythis has been going a long time16:48
rlandyhttps://review.opendev.org/#/c/681897/16:48
chandankumarEmilienM: please have a look at this bug https://bugs.launchpad.net/tripleo/+bug/1853978 http://logs.rdoproject.org/58/23558/10/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train/9b22f5e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz16:51
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged]16:51
rlandyoh my gosh16:51
chandankumarEmilienM: may be again missing release tags16:51
rlandyhow long has this test been broken?16:51
chandankumarrlandy: earlier broken on various reasons16:51
chandankumarselinux was one16:51
chandankumarrlandy: marios and I looked for 4 weeks to get it resolved but got tired16:52
rlandychandankumar: so how come we got a pass yesterday?16:53
rlandyhow is that possible16:53
chandankumarrlandy: may be job comparison will help16:53
EmilienMchandankumar: I can't look at it16:54
rlandychandankumar: yeah - catching up here16:54
EmilienMmy day is already super busy16:54
chandankumarrlandy: https://github.com/rdo-infra/ci-config/tree/master/jcomparison16:54
EmilienMask someone else plz16:54
rlandychandankumar: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train16:54
rlandyhttp://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train/71adc35/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz16:55
rlandycomplete deploy16:55
chandankumarrlandy: logreduce will help http://paste.openstack.org/show/787801/16:56
rlandychandankumar: so the current run skip because image build failed .. rerun that16:56
rlandyreran that and it passed16:56
chandankumarrlandy: jqcomparison will tell what rpms got changes, it will give a good insight16:56
chandankumarfrom passed and failed one16:57
chandankumarrlandy: https://etherpad.openstack.org/p/ruckroversprint19  from line 28-50 current open bugs16:59
chandankumarrlandy: this one needs your helphttps://trello.com/c/fdP2q3Yc/1274-cixlp1856830tripleociproa-master-train-promote-consistent-to-tripleo-ci-testing-times-out-blocks-mastertrain-pipelinehttps://trello.com/c/fdP2q3Yc/1274-cixlp1856830tripleociproa-master-train-promote-consistent-to-tripleo-ci-testing-times-out-blocks-mastertrain-pipeline17:00
rlandychandankumar: one sec - reading logs here17:00
rlandychandankumar: that card should be closed out right?17:00
chandankumarrlandy: yes17:01
chandankumarrlandy: let me take care of the card17:01
chandankumarrlandy: done17:02
rlandythank you17:02
rlandychandankumar: so let's narrow this down ...17:02
rlandywe are looking at two errors here17:02
rlandyone: train fs00117:02
rlandyand the mast fs00117:02
rlandyboth on rhel17:02
rlandyunrelated17:02
chandankumarrlandy: yes17:04
rlandytripleo::profile::base::neutron::ovn_metadata_agent_wrappers17:04
rlandy^^ looking for that17:04
chandankumarrlandy: it is train17:04
rlandyyep17:04
chandankumarrlandy: as per weshay we need to make sure RHEL8 master should be green17:04
chandankumarand train standlone rhel8 should be green that's it17:05
chandankumarrlandy: we might revisit in new year17:05
rlandychandankumar: ok ... well weshay may be available in the afternoon17:05
chandankumaryup17:05
*** whoami-rajat has quit IRC17:09
rlandy' "overcloud_image_url": "http://38.145.34.141/rcm-guest/images/redhat8/train/rdo_trunk/tripleo-ci-testing/overcloud-full.tar17:14
rlandyfuxed17:14
rlandyfixed17:14
*** ykarel|away has quit IRC17:14
rlandy"overcloud_image_url": "http://38.145.34.141/rcm-guest/images/redhat8/train/rdo_trunk/9e17b93b7b6c04b078c30db184f65f7242dedb99_e4dedca9/overcloud-full.tar"17:15
chandankumargo ahead!17:15
rlandychandankumar ok so now if it's pulling the right image - then we are only missing the backport/train version pinning17:17
rlandybut again confused at the pass yesterday17:18
rlandyand the return of the failure today17:18
chandankumarrlandy: can we record that in bug, may be I will take alook when I will nbe at airport17:19
rlandychandankumar: k - still digging through logs17:19
rlandyand changes17:19
chandankumarrlandy: use jqcomparison17:19
chandankumarpull sshnaidm container and it will work17:19
*** holser has quit IRC17:26
chandankumarkopecmartin: EmilienM: rlandy: weshay rfolco panda marios sshnaidm arxcruz|pto zbr Thank you learned so many things this year, Hope to break more and fix in new year, see ya Happy Holidays :-)17:30
sshnaidmchandankumar, happy holidays, man!17:30
rfolcohappy holidays chandan17:31
chandankumartime to delete this instance, see ya in new year wirh new experiments :-)17:31
*** chandankumar has quit IRC17:31
rfolcodon't drink and code17:31
rlandyenjoy your time off!17:31
*** TrevorV has joined #oooq17:32
*** Trevor_V has quit IRC17:36
*** jpena is now known as jpena|off17:44
rlandyweshay: ping if you are around17:51
weshayrlandy, what's up17:51
rlandyweshay: yay ... ok - train rhel 817:52
rlandyhttps://bugs.launchpad.net/tripleo/+bug/185397817:52
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged]17:52
rlandyhas been going on for weeks17:52
weshayhrm... k.. so Chandra mentioned I probably need to do a tripleo release for train17:53
rlandychandan suggested that we need to pin version17:53
weshayI'm going to go after that tomorrow17:53
rlandyyep https://review.opendev.org/#/q/owner:%22wes+hayutin+%253Cwhayutin%2540redhat.com%253E%22+project:openstack/releases17:53
rlandyok - so may I'll prepare review17:53
rlandyhttps://review.opendev.org/#/c/681897/7/deliverables/train/puppet-tripleo.yaml17:53
rlandy^^ what was wrong with that?17:53
* weshay will look17:54
weshayrlandy, /me looks17:55
rlandyweshay: doesn't totally make sense to me as we had a green run yesterday17:56
rlandyand this bug is weeks old17:56
weshayrlandy, the release team ( upstream ) doesn't always agree w/ the way tripleo sets up the releases.. but we're usually able to work it out17:56
rlandyI just kick ed a new run17:57
weshayrlandy, usually it's done so that the rdo team can update the dependencies for a particular repo17:57
rlandyrock is promoting17:58
rlandyrocky17:59
*** udesale has joined #oooq18:03
rlandymaster may promote18:06
*** panda has quit IRC18:08
*** panda has joined #oooq18:13
*** rfolco is now known as rfolco|bbl18:18
*** udesale has quit IRC18:37
*** amoralej is now known as amoralej|off18:54
*** ccamacho has quit IRC19:22
*** sshnaidm is now known as sshnaidm|afk20:00
*** saneax has quit IRC20:00
*** irclogbot_1 has quit IRC20:06
*** irclogbot_3 has joined #oooq20:09
*** Trevor_V has joined #oooq20:30
*** holser has joined #oooq20:30
*** apetrich has quit IRC20:31
*** TrevorV has quit IRC20:33
*** jfrancoa has quit IRC21:15
*** Goneri has quit IRC21:35
rlandyweshay: you around?21:52
*** Trevor_V has quit IRC22:05
*** rfolco|bbl has quit IRC22:06
*** holser has quit IRC22:07
*** jbadiapa has quit IRC23:13
*** tosky has quit IRC23:22
*** Goneri has joined #oooq23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!