Friday, 2019-12-20

rlandysshnaidm|afk: you still around?00:13
sshnaidm|afkrlandy, yep00:13
rlandysshnaidm|afk: can you help me clean up promoter?00:13
rlandyfailed: [localhost] (item=rsyslog) => {"ansible_loop_var": "item", "changed": false, "item": "rsyslog", "msg": "Error pulling trunk.registry.rdoproject.org/tripleotrain/centos-binary-rsyslog - code: None message: failed to register layer: Error processing tar file(exit status 1): write /var/lib/rpm/Packages: no space left on device"}00:13
rlandyfailing pulling containers00:13
rlandysshnaidm|afk: what did you delete before?00:14
sshnaidm|afklemme look00:14
rlandydev/vdb1        80G   69G   12G  86% /var/lib/docker00:15
sshnaidm|afkrlandy, yeah, there are 2 months old images, removing them00:16
rlandysshnaidm|afk: from where ? so I know what to do next time00:17
sshnaidm|afkdocker image ls00:17
sshnaidm|afkin the bottom you can see '2 months ago'00:18
sshnaidm|afkwe don't need them00:18
rlandyyeah I see00:19
sshnaidm|afkfor i in $(docker image ls | grep ab34c90c15f5cb5efb7236fdf4b22020e6f3cd04_67a09fe9  | awk {'print $1'}); do docker rmi ${i}:ab34c90c15f5cb5efb7236fdf4b22020e6f3cd04_67a09fe9; done00:19
sshnaidm|afkso we remove this tag only00:20
rlandy 6bd5835cd379bcf9824bba7e9f81f2489d19e506_bada63cf   4537d04ac5a7        6 days ago00:20
rlandynah - ^^ that can go as well00:20
rlandyif we need it, it will download again00:20
*** rfolco|bbl has joined #oooq00:21
sshnaidm|afkyeah00:21
rlandythanks - master, train, stein,rocky could all promote today00:22
sshnaidm|afkwow00:23
rlandysee the logs, they all tried00:23
rlandyI figure this may have been going on for a few days00:23
sshnaidm|afka big promotion day00:23
rlandyrhel is bad shape though00:24
rlandycan't get fs001 on train or master through00:24
rlandycollectd/healthcheck hang00:24
rlandygit one tht patch that may clear train eventually00:24
rlandygot00:25
sshnaidm|afkwhich one?00:25
rlandyhttps://review.opendev.org/#/c/699438/00:27
sshnaidm|afkk, we'll see00:29
sshnaidm|afkI'll keep an eye next week, at least first days :)00:29
rlandysshnaidm|afk: thanks01:10
*** Goneri has quit IRC01:36
*** Goneri has joined #oooq01:38
*** ysandeep has joined #oooq01:43
*** rlandy has quit IRC01:51
*** ysandeep has quit IRC02:05
*** rfolco|bbl has quit IRC03:31
*** rfolco|bbl has joined #oooq03:32
*** rfolco|bbl has quit IRC03:42
*** ykarel|away has joined #oooq03:45
*** d0ugal has quit IRC03:47
*** d0ugal has joined #oooq03:48
*** ykarel|away has quit IRC04:24
*** dsneddon has quit IRC04:24
*** ykarel|away has joined #oooq04:24
*** bhagyashris has joined #oooq04:37
*** whoami-rajat has joined #oooq04:40
*** ykarel|away has quit IRC04:51
*** dsneddon has joined #oooq04:57
*** dsneddon has quit IRC05:09
*** ykarel|away has joined #oooq05:10
*** ykarel|away is now known as ykarel05:14
*** holser has joined #oooq05:19
*** whoami-rajat has quit IRC05:32
*** holser has quit IRC05:32
*** bhagyashris has quit IRC05:34
*** dsneddon has joined #oooq05:46
*** dsneddon has quit IRC05:51
*** bhagyashris has joined #oooq05:59
*** surpatil has joined #oooq06:01
*** surpatil has quit IRC06:16
*** surpatil has joined #oooq06:17
*** dsneddon has joined #oooq06:20
*** dsneddon has quit IRC06:25
*** skramaja has joined #oooq06:31
*** dsneddon has joined #oooq06:54
*** holser has joined #oooq06:57
*** dsneddon has quit IRC07:08
*** holser has quit IRC07:38
*** apetrich has joined #oooq07:41
*** dsneddon has joined #oooq07:42
*** marios has joined #oooq07:43
*** saneax has joined #oooq07:48
*** dsneddon has quit IRC07:51
*** dsneddon has joined #oooq07:54
*** bhagyashris has quit IRC07:57
*** dsneddon has quit IRC08:05
*** amoralej|off is now known as amoralej08:08
*** dsneddon has joined #oooq08:12
*** dsneddon has quit IRC08:16
*** tosky has joined #oooq08:19
*** dsneddon has joined #oooq08:22
*** holser has joined #oooq08:38
*** jpena|off is now known as jpena08:57
marios2019-12-20 04:29:44,459 30179 INFO     promoter Promoting the container images for dlrn hash 99d58382abb0d6ad8ee126835cd40edf3ec07d6c on master to current-tripleo08:58
*** dsneddon has quit IRC08:58
*** ccamacho has joined #oooq09:03
*** Tengu has quit IRC09:04
*** bhagyashris has joined #oooq09:08
*** yolanda has quit IRC09:10
*** SurajPatil has joined #oooq09:21
*** surpatil has quit IRC09:24
*** yolanda has joined #oooq09:28
*** dsneddon has joined #oooq09:31
*** derekh has joined #oooq09:42
*** dsneddon has quit IRC09:46
*** surpatil has joined #oooq10:10
*** SurajPatil has quit IRC10:12
*** dsneddon has joined #oooq10:13
*** dsneddon has quit IRC10:18
*** ykarel is now known as ykarel|afk10:25
*** dsneddon has joined #oooq10:42
*** Tengu has joined #oooq10:47
*** Tengu has quit IRC10:47
*** bhagyashris has quit IRC10:48
*** Tengu has joined #oooq10:49
*** dsneddon has quit IRC11:20
*** bhagyashris has joined #oooq11:22
*** ykarel|afk is now known as ykarel11:26
*** dsneddon has joined #oooq11:39
*** dsneddon has quit IRC11:46
*** rfolco|bbl has joined #oooq12:11
*** rfolco|bbl is now known as rfolco12:12
*** SurajPatil has joined #oooq12:15
*** surpatil has quit IRC12:15
*** dsneddon has joined #oooq12:18
*** bhagyashris_ has joined #oooq12:20
*** bhagyashris has quit IRC12:22
*** bhagyashris_ is now known as bhagyashris12:32
*** dsneddon has quit IRC12:36
rfolcomarios, panda weshay need this fix https://review.rdoproject.org/r/#/c/24273/ for the component promotion job12:36
*** jpena is now known as jpena|lunch12:38
*** dsneddon has joined #oooq12:38
*** surpatil has joined #oooq12:40
mariosrfolco: k let me know if you want merge12:40
rfolcoI do12:40
mariosand so it was done12:42
mariosthe rfolco willed it so12:42
mariosand it was12:42
marios# coolstorybro12:42
*** SurajPatil has quit IRC12:42
*** SurajPatil has joined #oooq12:44
*** surpatil has quit IRC12:47
rfolcomarios, thank you. Do I have more 2 wishes, my lamp genius12:47
rfolco?12:47
mariosno12:49
*** bhagyashris has quit IRC12:53
weshaymorning12:58
*** rlandy has joined #oooq12:58
*** dsneddon has quit IRC13:00
rlandyhey weshay13:01
rlandywelcome back13:01
*** dsneddon has joined #oooq13:01
marioso/13:01
mariosweshay: rlandy: have you ever had to add swap to the promoter13:02
mariosweshay: rlandy: i added 4gb but not really seeing it be used at all13:02
mariosi think i should disable it13:02
rlandymarios: no - but it keeps running out of space on device13:02
rlandyneeded clean up yesterday13:03
mariosrlandy: yeah i saw that... so i thought it was completely borked. everything, i mean all the things promoted today13:03
mariosor trying to...13:03
rlandyand then all releases promoted13:03
rlandyand now it's out of space again13:03
rlandyto pormote phase 113:03
rlandycleaning13:03
mariosrlandy: so i saw a no space for stein. master was stuck for a veeeery long time13:03
mariosrlandy: k going to remove that swap then if it isn't used then no point taking space13:03
rlandymarios: yep - figured that out late my time13:03
mariosrlandy: maybe i should leave 2 gb? wdyt?13:04
rlandyprobably harmless13:04
mariosrlandy: but really i've been looking at top and i dno't see swap being used. but memory is maxed out13:04
rlandyit's the docker containers13:04
rlandythey are not being cleaned up13:04
rlandydev/vdb1        80G   53G   28G  67% /var/lib/docker13:04
weshaymarios, ah.. seeing those traces?13:06
mariosweshay: which bit13:06
rlandymarios: queens is promoting now13:06
weshaypromoter13:06
weshaysec in tempest call13:06
rlandyso leaving those containers13:06
weshaySurajPatil, Sorin ( zbr )13:07
rlandymarios: also looks like there is a legit issue with the weirdo jobs13:07
mariosrlandy: k made it 2 gb swap13:08
mariosrlandy: see https://etherpad.openstack.org/p/ruckroversprint19 i added logs there13:08
mariosrlandy: they're all promoting13:08
rlandyykarel: ^^ https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/13:08
rlandythe weirdo jobs have been failing on master for the last few runs13:08
mariosrlandy: train and stein left queens already promoted curre-tripleo maybe you mean queens promoting phase113:09
rlandyweirdo-master-promote-puppet-openstack-scenario00{1-4}13:09
ykarelrlandy, it need promotion of current-tripleo13:09
ykarelthere was a fix recently in glance13:09
rlandypromoter Promoting the container images for dlrn hash 2c6d1578fef7087aa440544293190eca80ae4f9b on queens to current-tripleo-rdo13:09
mariosweshay: basically all the things are promoting and it is straining the promoter server.13:09
rlandyykarel: any action needed from us or the fix will work its way through?13:09
*** dsneddon has quit IRC13:09
marioskind of a nice problem to have but rlandy do we need to move the server to new node13:09
mariosbut never on a friday ... :/13:10
ykarelrlandy, with next master promotion to current-tripleo , phase1 should clear13:10
rlandymarios: containers should be cleaned up13:10
ykarelrlandy, any known blockers for master promotion13:10
rlandyykarel: k - thanks13:10
mariosrlandy: having said that, i thought master was borked earlier but then it managed to finish container push13:10
rlandyit takes a long time13:10
mariosrlandy: but next time lets get a bigger node. i mean why don't we just max that out13:10
mariosweshay: ^^13:10
rlandybecause it had not promoted in a while13:10
mariosrlandy: went though and closed bunch of things out and removed them so we have a shorter list at https://etherpad.openstack.org/p/ruckroversprint19 for ongoing things13:11
rlandymarios: also, as you say, not usual that all releases promote both phases on the same day13:12
rlandybut that was my goal13:12
rlandyto clear the board13:12
mariosit's a christmas miracle13:12
marios\o/13:12
marios;)13:12
rlandymarios: rhel is still in trouble13:12
rlandyboth train and master13:12
rlandylooking through current issues on the etherpad13:12
mariosi man we've been chasing train for a looong time. i think its close13:12
rlandywe merged the supposed fix for train13:12
mariosif we got train today/monday rlandy it would be icing on cake13:12
rlandyI didn;t see a pass yesterday13:12
mariosrlandy: train failed in different way i saw one today sec13:12
marios * http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train/4d09b8c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz13:13
marios            * 2019-12-19 18:32:45 | 2019-12-19 18:27:53 131238 INFO tripleo_common.image.image_uploader [  ]   stderr: 'Error: Unknown repo: ''gating-repo'''13:13
weshaySurajPatil, https://github.com/pwittchen/learn-python-the-hard-way#unit-testing13:13
rlandymarios: oh gee13:14
rlandythat sounds legit13:14
mariosweshay: relevant bug is there https://bugs.launchpad.net/tripleo/+bug/1853978/comments/27 i mean 'the rhel8 train bug'13:14
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Fix released]13:14
rlandygating-repo is the update13:14
rlandyI think the promoter is running now13:15
rlandyspace is at 61%13:15
rlandywhich should be functional13:15
rlandywill watch if queens containers go away after promotion13:15
mariosrlandy: k but for christmas i want a bigger promoter server please13:15
rlandymarios: I'll ping santa13:16
marios\o/13:16
*** jfrancoa has joined #oooq13:16
rlandymarios: it's actually within our power13:16
rlandyour infra node13:16
rlandywe can add more storage13:17
* marios not sure he likes the way this is heading.13:17
mariosrlandy: i'd just like to remind you today is friday13:17
rlandymarios; I don;t have a death wish13:17
mariosrlandy: wait its just an rdo vm though right?13:18
rlandyyes13:18
mariosrlandy: i see what you mean. but we have to pause it first no?13:18
mariosrlandy: or we can just resize it13:18
mariosrlandy: well attach a new bigger volume?13:18
rlandyperiodic-tripleo-ci-rhel-8-scenario001-standalone-master - passed13:18
mariosrlandy: but still means disruption13:18
rlandyand13:18
rlandyperiodic-tripleo-ci-rhel-8-scenario001-standalone-train13:18
mariosrlandy: nice13:18
mariosand nice13:19
rlandyso we got one bug down13:19
weshayk..13:19
rlandythat is the fix we merged13:19
* weshay says hi to my family for a bit13:19
rlandynow we have fs001 bugs13:19
weshayah NICE on rhel 8 :)13:19
rlandymarios: ack we can13:19
mariosrlandy: yeah i closed the train bug out earlier https://bugs.launchpad.net/tripleo/+bug/1853978/comments/27 based on the 18th green run13:19
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Fix released]13:19
rlandycool13:19
rlandydid we move the CIX?13:19
mariosrlandy: oh fs1 master excellent cos https://bugs.launchpad.net/tripleo/+bug/1853652/comments/3413:20
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner)13:20
rlandymarios: no, no13:20
rlandyhttps://bugs.launchpad.net/tripleo/+bug/185627813:20
openstackLaunchpad bug 1856278 in tripleo "RHEL8 scenario 1 standalone deployment failed with The following containers failed validations and were not started: collectd" for master and train" [Critical,Fix released] - Assigned to Cédric Jeanneret (cjeanner)13:20
mariosrlandy: so looks like the very latest run included the selinux fixes13:20
rlandymarios: fs001 master and train are still failing13:21
rlandythis https://trello.com/c/r8g1gG41/1270-cixlp1856278tripleociproa-rhel8-scenario-1-standalone-deployment-failed-with-the-following-containers-failed-validations-and-wer is fixed13:21
mariosrlandy: ah scen1 sorry mixed them ack with you now13:21
rlandyok - moving that card to done13:21
rlandythen let's look at the two fs00113:21
mariosrlandy: i even commented there earlier https://bugs.launchpad.net/tripleo/+bug/1856278/comments/1413:22
openstackLaunchpad bug 1856278 in tripleo "RHEL8 scenario 1 standalone deployment failed with The following containers failed validations and were not started: collectd" for master and train" [Critical,Fix released] - Assigned to Cédric Jeanneret (cjeanner)13:22
mariosrlandy: (standalone i mean )13:22
mariosrlandy: so fs1 still failing on timeout13:22
mariosrlandy: i commented there https://bugs.launchpad.net/tripleo/+bug/1853652/comments/3413:22
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner)13:22
mariosrlandy: still neutron dhcp agent issue but not selinux this time cos definitely permissive13:22
mariosrlandy: so we need to ping back to slaweq/neutron folks but13:22
mariosholidays13:22
rlandymarios: I'm confused :)13:22
rfolcorlandy, please https://review.rdoproject.org/r/2428413:23
rlandydid I move the wrong card?13:23
mariosrlandy: periodic rhel-8-ovb-3ctlr_1comp-featureset001-master13:23
rlandyrfolco: done13:23
mariosrlandy: want to bluejeans?13:23
rlandymarios: I closed the one about scenario00113:23
mariosrlandy: yes right13:23
mariosrlandy: scen1 done13:23
mariosrlandy: now i am talking about fs113:23
marios15:21 < rlandy> then let's look at the two fs00113:24
rlandymarios: yep - let's chat on meet13:24
marios15:22 < marios> rlandy: so fs1 still failing on timeout13:24
marios15:22 < marios> rlandy: i commented there https://bugs.launchpad.net/tripleo/+bug/1853652/comments/3413:24
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner)13:24
mariosrlandy: k plugging headphones13:24
mariosimpromptu scrum welcome all https://meet.google.com/evp-nyur-dhb13:24
mariosrlandy: ^13:24
rlandyjoined13:25
*** amoralej is now known as amoralej|lunch13:35
*** jpena|lunch is now known as jpena13:36
marios2019-12-20 13:26:41,621 3382 INFO     promoter Promoting the container images for dlrn hash 80cc8ed4b1d2e10e7c4a518dd4ca1a5305ed8ede on train to current-tripleo-rdo13:38
marioshttps://bugs.launchpad.net/tripleo/+bug/1853652/comments/3413:42
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner)13:42
*** dsneddon has joined #oooq13:43
*** ykarel has quit IRC13:48
weshayback13:54
weshaysorry .. so tired.. got it at 2am last night13:55
weshayanything I can help w/?13:56
marioshttps://bugs.launchpad.net/tripleo/+bug/1853978/comments/2713:56
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Fix released]13:56
weshayah ya13:57
* weshay put up a puppet-tripleo review.. and then I'll work the release13:57
mariosweshay: sorry that was for rlandy13:57
rlandyweshay: train rhel 8 is still legit failure13:57
mariosweshay: not asking for help13:57
weshayrlandy, marios ok.. so this merged https://review.opendev.org/#/c/700045/13:58
weshaymarios, oh.. I thought train needed a release for rdo for fixing that13:58
rlandyweshay: so looking at latest run13:59
mariosweshay: perhaps i am really not sure13:59
mariosweshay: rlandy: we are looking at latest i think its a new issue but we aren't sure ... :/13:59
rlandyhttp://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train/253a697/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz13:59
rlandylatest logs13:59
*** skramaja has quit IRC14:00
mariosweshay: btw we don't know how but the promoter is promoting all the things at the same time14:00
mariospromoter on steroids14:00
rlandyweshay: the promoter is outsourcing14:00
mariosweshay: we decided to let it finish and then do a reboot/clear all the services (might be duplicate services)14:01
weshayya.. that's been going on for the last couple weeks14:01
weshaymarios, rlandy has anyone tried killing all the python processes, stopping the service and restarting14:01
weshay?14:01
rlandyweshay: no - because it's actually promoting14:01
mariosweshay: we don't want to interrupt it. cos it is like a christmas tree its promoting EVEERYTHING14:01
rlandyonce it stops14:01
weshaylolz14:01
weshaylike right now?14:02
mariosweshay: whole day14:02
weshaylolz14:02
weshayk14:02
mariosweshay: we've had master queens rocky already14:02
mariostrain happening14:02
marios#happening14:02
rlandyon both phases14:02
rlandyand let us say amen14:02
weshay#happy_xmas_from_your_local_promoter14:02
weshayfrom our promoter to g-d's ears?14:03
weshayok.. I'll work on it later14:03
mariosweshay: so yeah we want to reboot that at some point and maybe add some more storage14:03
rlandyoprah special ... you get a promotion, and you get a promotion, you all get a promotion14:03
mariosweshay: but maybe very late in your day14:04
weshaylolz14:04
weshayrlandy++14:04
weshaymarios, not as late as yours14:04
mariosoprah: "promotioooooooooooons "14:04
mariosweshay: i mean 'later on in your day before you leave maybe please reboot the promoter"14:05
rlandyweshay: marios: ^^ ack in our afternoon14:05
weshayaye.. not sure if I'll reboot, but I'll def kill everything and restart the service14:05
mariosrlandy: yes "once it stops"14:05
marios;)14:05
weshayit's not windoze14:05
rlandymaybe we should just stop it now14:05
rlandytake the hit14:06
*** dsneddon has quit IRC14:06
rlandyit keeps running out of space14:06
weshayoh14:06
weshaythat's not great14:06
* weshay checks it out14:06
*** dsneddon has joined #oooq14:06
rlandyweshay: see docker image ls14:06
rlandyyou will see two sets of containers14:07
rlandyand ps -u cenos14:07
rlandycentos14:07
weshaydisk space is ok right now14:07
rlandywill show you two ansible-playbook processes14:07
rlandyweshay: ack because we cleaned up14:07
rlandyit was at 80%14:07
rlandyit's parallel processing14:07
rlandywhich could be a neat trick or a disaster14:07
weshayya.. not good.. but I think what happens is that a new push starts and kills the running one14:09
weshayit looks like both are running in ps14:09
weshaybut it's only one14:09
weshaynot sure14:10
weshayI saw this happen before I left for nyc14:10
*** d0ugal has quit IRC14:10
weshaywow.. 95% pass rate upstream14:11
*** dsneddon has quit IRC14:11
rlandy2019-12-20 13:26:41,621 3382 INFO14:11
rlandyso the train run just started14:11
weshayoh ha.. that my local dev env14:11
*** dsneddon has joined #oooq14:12
weshayk.. /me is on the tmux14:13
*** d0ugal has joined #oooq14:13
rlandymarios: kicked a rerun on rhel8 train14:14
rlandyweshay: you on the promoter?14:15
rlandyweshay: maybe we should just kill and restart14:16
rlandyand clean up these containers images14:16
mariosrlandy: ack thanks14:16
weshayrlandy, ya14:16
*** dsneddon has quit IRC14:17
*** SurajPatil has quit IRC14:17
rlandy2019-12-20 10:06:08,242 787714:17
rlandyqueens one has been going for ages14:17
rlandyweshay: sorry ya to which part14:17
weshayI'm on the promoter14:18
weshayI'm sorry.. very sluggish14:18
weshaymarios, rlandy sshnaidm|afk fyi.. I booked my flight etc14:22
mariosweshay: fyi _manifests looks good updated that https://tree.taiga.io/project/tripleo-ci-board/task/1315 with some logs moving to ready-for-review though we're not merging anything there14:22
weshayk14:22
weshaywaiting on the pandalorian14:22
mariosweshay: looks like it deploys the _manifest 'containers' fine14:23
mariosweshay: even though they're manifests14:23
mariosweshay: ack on flight i will likely do that next week14:23
weshayvery cool14:23
rlandyweshay: yeah - got to get on that14:23
weshaymarios, I'll check out those test jobs in a bit14:23
*** TrevorV has joined #oooq14:24
mariosweshay: ack if you mean _manifests would be nice to have a sanity check i added pointers to logs there in taiga. Once we switch to the new code it will be easier we can also test on master too for example14:24
weshayvery cool.. thanks for pushing on that14:25
mariosweshay: ack nice to finally close that card out14:26
rlandyweshay: on your  tmux14:26
*** epoojad1 has joined #oooq14:26
rlandyi can't operate with these multiple windows :(14:27
weshaymarios, rlandy ok.. usually tox fails but.. https://review.opendev.org/70017914:29
rlandyand now stein is also promoting14:29
rlandyweshay: thanks14:30
rlandyweshay: pls join me on https://meet.google.com/xnz-rpyd-tfq14:31
rlandymarios: ^^ feel free to join us if you want14:31
mariosrlandy: sure plugging in14:32
*** amoralej|lunch is now known as amoralej14:39
*** dsneddon has joined #oooq14:46
*** ccamacho has quit IRC14:57
*** dsneddon has quit IRC15:11
*** EmilienM is now known as EvilienM15:16
*** epoojad1 has quit IRC15:25
*** dsneddon has joined #oooq15:47
*** derekh has quit IRC15:51
weshayrlandy, the component pipeline is very green15:53
*** dsneddon has quit IRC15:54
rlandyweshay: downstream or upstream?15:54
weshayrlandy, I only have eyes on upstream atm15:54
mariosweshay: rlandy: getting ready to go need something before i do?15:55
rlandyweshay: ok - so downstream standalone compute fails tempest15:55
rlandymario: nah - have some time off15:56
rlandymarios: ^^15:56
mariosrlandy: ack around monday anyway will take time after that :D15:56
weshayrlandy, ya.. let migi handle that :)15:56
rlandyweshay: wrt upstream, it should track standalone master15:56
rlandywhich is also very green15:56
rlandyweshay: the downstream failure is legit15:57
rlandyosp17 component with rhos-16 product base15:57
weshayaye..15:59
*** marios is now known as marios|out15:59
rlandyweshay: interestingly enough the mix keystone job works16:05
rlandytrain promotion completed16:08
rlandystein in progress16:09
rlandyweshay: ^^ cleaning up  train container images16:10
rlandydev/vdb1        80G   50G   31G  62% /var/lib/docker16:10
weshayk16:14
*** epoojad1 has joined #oooq16:15
*** ykarel has joined #oooq16:21
*** dsneddon has joined #oooq16:24
rlandydev/vdb1        80G   36G   45G  45% /var/lib/docker16:27
rlandybetter16:27
rlandyimages area just stein and queens now16:27
rlandy2019-12-20 10:36:42 |     ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500"16:37
rlandyshoot16:37
rlandytenant cleanup may be needed16:38
*** dsneddon has quit IRC16:38
*** Goneri has quit IRC16:41
*** marios|out has quit IRC16:45
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master16:48
rlandywow = can see how many times master promoted from BM logs16:48
*** dsneddon has joined #oooq17:00
*** Trevor_V has joined #oooq17:00
*** TrevorV has quit IRC17:04
rfolcorlandy, https://review.rdoproject.org/r/#/c/24287/ - tested this one now locally, with / it works as expected17:04
weshayrlandy, ya.. lolz17:15
weshayrlandy, I asked Sagi to stand up an internal cockpit on some hardware I gave him17:15
rlandyweshay: thanks17:16
rlandydone17:16
weshayrlandy, rfolco check it out :) https://trunk.rdoproject.org/centos8-master/component/17:16
rfolcocool17:16
rlandyok17:17
weshayrfolco, rlandy as we're standing up centos-8 next year.. we'll have to chat about if it makes sense to build out all the component jobs and have them feed the integration jobs17:18
rlandywe're ready for that17:18
rlandyweshay: who has +2 on https://review.opendev.org/#/c/700179?17:19
rlandyperiodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train still fails17:20
rlandyperiodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master - that is the job to watch now17:20
rlandyweshay: if cockpit is internal - where should the logs post?17:21
*** holser has quit IRC17:21
weshayrlandy, the folks in #openstack-release17:22
weshayrlandy, want to chat about that for 5?17:22
rlandyok17:22
weshayhttps://meet.google.com/iiq-zhpf-msq17:23
*** ykarel is now known as ykarel|away17:33
*** Goneri has joined #oooq17:37
weshayrlandy,  https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds17:37
weshayrlandy, https://sf.hosted.upshift.rdu2.redhat.com/zuul/api/tenant/tripleo-ci-internal/builds17:43
*** ykarel|away has quit IRC17:45
weshayrlandy, just to document the path https://review.rdoproject.org/r/#/c/24291/1/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/telegraf.d/zuulv3_job_builds.conf17:48
rlandyack17:49
*** jpena is now known as jpena|off17:51
*** jpena|off has quit IRC17:52
*** amoralej has quit IRC17:57
rlandygreat - stein done17:57
rlandypromoter is looking healthy again17:58
weshayrfolco, https://review.rdoproject.org/zuul/builds?pipeline=github-check18:05
weshayhttps://review.rdoproject.org/zuul/builds?pipeline=github-manual18:06
weshayrfolco, saying we closed it out.. when we don't yet have the job that runs the tests done is probably not as accurate as we want to be :)18:06
*** jfrancoa has quit IRC18:07
rfolcoweshay, ok, almost closed it out, the poc I meant18:12
rfolcook18:12
weshayrfolco, ya.. I just don't have a good handle on the actual check jobs yet18:13
weshayhow much more work we have there18:13
weshay3rd party deps was a hot topic at the manager meetup18:13
rfolcofor the poc we just need to fix the job that promotes, I fix one issue, another shows up18:13
weshayit's really nice we have this stuff going18:14
rfolcolatest issue - http://logs.rdoproject.org/99/24199/5/check/periodic-tripleo-centos-7-master-component-compute-promote-to-current-tripleo/4bf6b95/job-output.txt18:14
weshayrfolco, ?18:14
weshayrfolco, we're not talking about components18:14
weshaywe're talking deps18:14
weshaypodman, ceph-ansible18:14
rfolcoah18:15
rfolcosorry18:15
rfolcowell, need to rephrase that then18:16
*** saneax has quit IRC18:20
*** rfolco is now known as rfolco|dentist18:21
rlandyweshay: looked into the gate failures18:37
rlandywe are running at 91% pass18:37
rlandybut it record 15 failures18:37
weshayya.. for check and gate18:37
weshayaye18:37
rlandyonly one though18:37
rlandyin the last couple hours18:37
rlandyso I think it's ok18:37
weshay:)18:39
weshaythanks for checking18:39
*** hamzy_ has quit IRC18:42
*** tosky has quit IRC18:45
rlandystill out n rhel fs00119:01
rlandy2019-12-20 13:47:31 |     ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500"19:01
rlandyugh19:01
rlandycan't get enough resources to run these tests19:01
rlandyweshay: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master19:03
rlandy^^ has not passed since 12/1119:03
rlandymaybe even before19:04
rlandyshould we try promote w/o it?19:04
rlandychecking cockpit stats19:04
*** hamzy has joined #oooq19:07
rlandysince 12/0719:13
*** brault has quit IRC19:17
rlandyNo such file or directory: '/var/log/pcsd/pcsd.log'19:35
rlandywe have a pacemaker issue19:36
*** rfolco|dentist has quit IRC19:48
*** rfolco has joined #oooq19:49
weshayrlandy, hrm that master job?19:58
rlandyweshay: yes19:59
rlandynot the latest run19:59
rlandythat one dies on resources19:59
rlandyI am rerunning with https://review.opendev.org/#/c/699318/19:59
rlandysee if it helps19:59
weshayk.. thanks.. I guess that job is busted anyway20:01
rlandyrocky promoted20:03
rlandyand queens is up20:03
rlandygetting there20:03
rlandyweshay: finally http://rhos-release.virt.bos.redhat.com:3030/rhosp20:20
rlandyone round of mass promoting done20:20
*** rlandy has quit IRC21:13
*** Trevor_V has quit IRC21:26
weshaynice22:25
*** EvilienM is now known as EmilienM22:36
*** dtrainor has quit IRC23:32
*** Goneri has quit IRC23:40
*** dsneddon has quit IRC23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!