Wednesday, 2019-12-18

*** tosky has quit IRC00:00
*** saneax has quit IRC00:20
*** rlandy has quit IRC00:29
*** rodrigods has quit IRC00:39
*** apetrich has quit IRC03:09
*** gouthamr has quit IRC03:14
*** gouthamr has joined #oooq03:24
*** ykarel|away has joined #oooq03:46
*** bhagyashris has joined #oooq04:05
*** ykarel|away has quit IRC04:18
*** skramaja has joined #oooq04:26
*** ykarel|away has joined #oooq04:35
*** bhagyashris has quit IRC04:47
*** bhagyashris has joined #oooq04:47
*** epoojad1 has joined #oooq04:49
*** holser has joined #oooq04:54
*** ykarel|away is now known as ykarel05:00
*** whoami-rajat__ has joined #oooq05:03
*** holser has quit IRC05:34
*** soniya29 has joined #oooq05:55
*** raukadah is now known as chkumar|rover05:58
*** surpatil has joined #oooq05:59
*** whoami-rajat__ has quit IRC06:07
*** SurajPatil has joined #oooq06:09
*** surpatil has quit IRC06:12
*** ksambor has quit IRC06:17
*** ksambor has joined #oooq06:17
*** saneax has joined #oooq06:24
*** SurajPatil has quit IRC06:25
*** surpatil has joined #oooq06:26
*** marios|ruck has joined #oooq06:36
*** whoami-rajat__ has joined #oooq06:37
*** jfrancoa has joined #oooq06:52
*** TrevorV has joined #oooq07:23
*** TrevorV has quit IRC07:23
marios|ruckchkumar|rover: \o/07:25
chkumar|rovermarios|ruck: \o/07:25
marios|ruck:)07:25
chkumar|rovermarios|ruck: Added notes on etherpad, consistent to toci testing broken07:27
marios|ruckchkumar|rover: again? so we need a bug for it. but the test job was green07:27
chkumar|rovermarios|ruck: it is timing out twice07:27
marios|ruckchkumar|rover: on the good news we got at least one good run yesterday (with https://review.rdoproject.org/r/24211 for full-tempest rerun) & promoted07:27
chkumar|roverawesome07:28
marios|ruckchkumar|rover: (master promoted)07:28
marios|ruckchkumar|rover: so https://bugs.launchpad.net/tripleo/+bug/1856723 is not consistent looks like :/07:28
openstackLaunchpad bug 1856723 in tripleo "periodic-tripleo-ci-centos-7-standalone-full-tempest-master fails during tempest run ImagesTestJSON:test_delete_saving_image" [Critical,Triaged]07:28
marios|rucki mean it passed on test07:28
marios|ruckchkumar|rover: i'll update those in a bit going though ether now07:28
chkumar|roverit is a timeout issue, but the test needs improvement, I asked soniya29 to work with glance people to get  aproper fix07:29
marios|ruckchkumar|rover: k, like we tested it yesterday there rembmer https://bugs.launchpad.net/tripleo/+bug/185672307:29
openstackLaunchpad bug 1856723 in tripleo "periodic-tripleo-ci-centos-7-standalone-full-tempest-master fails during tempest run ImagesTestJSON:test_delete_saving_image" [Critical,Triaged]07:29
marios|ruckoops wrong paste :/07:29
marios|ruckchkumar|rover: there https://review.rdoproject.org/r/#/c/24224/07:29
chkumar|roversweet :-)07:30
marios|ruckchkumar|rover: yeah but just means its inconcistent if it happens again today07:30
chkumar|roveryes, it might happen07:30
marios|ruckchkumar|rover: something merged last few days maybe rlandy knows we can ask later (she was merging stuff last night too)07:30
chkumar|roverhttps://review.rdoproject.org/r/#/c/24236/ merged right now, checking whether it fixes or not07:31
*** holser has joined #oooq07:31
*** ccamacho has joined #oooq07:38
marios|ruckchkumar|rover: man still not a single green on that m*****r f*****r https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset00107:40
marios|ruckchkumar|rover: lets see i rechecked that https://review.rdoproject.org/r/#/c/23919/ based on https://bugs.launchpad.net/tripleo/+bug/1853652/comments/3207:45
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Fix committed] - Assigned to Cédric Jeanneret (cjeanner)07:45
chkumar|roverok07:45
chkumar|roverbrb for food07:45
marios|ruckchkumar|rover: take your time07:45
*** ykarel is now known as ykarel|lunch07:47
*** whoami-rajat__ is now known as whoami-rajat|lun07:48
*** kopecmartin|off is now known as kopecmartin07:53
marios|ruckchkumar|rover: when back can you check https://bugs.launchpad.net/tripleo/+bug/1853978/comments/2408:05
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged]08:05
*** soniya29 has quit IRC08:09
*** amoralej|off is now known as amoralej08:14
*** tosky has joined #oooq08:18
*** jpena|off is now known as jpena08:22
*** SurajPatil has joined #oooq08:29
*** surpatil has quit IRC08:30
*** surpatil has joined #oooq08:31
*** tosky has quit IRC08:32
*** SurajPatil has quit IRC08:34
chkumar|rovermarios|ruck: I think I got the issue https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/promote-secondary-distro.yaml#L708:36
chkumar|roverbash -xe {{ ci_config_repo }}/ci-scripts/tripleo-upstream/promote-hash.sh is not there in https://github.com/rdo-infra/ci-config/tree/master/ci-scripts08:36
marios|ruckack chkumar|rover thanks, gonna file a bug for the consistent issue now cos it completely blew the master pipeline08:37
marios|rucklucky we got master promoted yesterday ;)08:37
chkumar|roverhttps://review.rdoproject.org/r/#/c/18184/ got removed long time ago need to check what is happening there08:38
chkumar|roverok08:39
marios|ruckchkumar|rover: agree on skip that then? https://bugs.launchpad.net/tripleo/+bug/1856723/comments/308:52
openstackLaunchpad bug 1856723 in tripleo "periodic-tripleo-ci-centos-7-standalone-full-tempest-master fails during tempest run ImagesTestJSON:test_delete_saving_image" [Critical,Triaged]08:52
chkumar|roveryes agreed08:57
*** soniya29 has joined #oooq09:11
chkumar|rovermarios|ruck: seen this issye OSError: [Errno 13] Permission denied: '/dlrnapi_venv'09:14
*** tosky has joined #oooq09:16
marios|ruckchkumar|rover: https://bugs.launchpad.net/tripleo/+bug/1856830 for the consistent issue  fyi09:16
openstackLaunchpad bug 1856830 in tripleo "master-promote-consistent-to-tripleo-ci-testing times out blocks master pipeline" [Critical,Triaged]09:16
marios|ruckchkumar|rover: hmm not sure about that seems familiar but i don't have somethign to point to for the venv thing didn't we see soemthign like that yesterday ? checking the etherpad09:16
soniya29chkumar|rover, marios|ruck, this bug:- https://bugs.launchpad.net/tripleo/+bug/1856723 may be a network issue as well, since it is not failing consistently.09:23
openstackLaunchpad bug 1856723 in tripleo "periodic-tripleo-ci-centos-7-standalone-full-tempest-master fails during tempest run ImagesTestJSON:test_delete_saving_image" [Critical,Triaged]09:23
marios|rucksoniya29: ack thanks please add note on the bug if you dig there09:24
marios|rucksoniya29: chkumar|rover: either way since this blocks master promotion we should skip with https://review.opendev.org/#/c/699624/ until the test is more stable or improved depending on the root09:24
soniya29chkumar|rover, marios|ruck, according to glance people the issue isn't with glance.09:25
marios|rucksoniya29: yeah see https://bugs.launchpad.net/tripleo/+bug/1856723/comments/1 and https://bugs.launchpad.net/tempest/+bug/1713163/comments/509:27
openstackLaunchpad bug 1856723 in tripleo "periodic-tripleo-ci-centos-7-standalone-full-tempest-master fails during tempest run ImagesTestJSON:test_delete_saving_image" [Critical,Triaged]09:27
openstackLaunchpad bug 1713163 in tempest "test_delete_saving_image fails because image hasn't transitioned to SAVING" [Medium,Confirmed]09:27
*** apetrich has joined #oooq09:27
marios|ruckchkumar|rover: so... lets do it ourselves :/ posted master just now lets see https://review.rdoproject.org/r/2424109:35
chkumar|rovermarios|ruck: ok, i am still looking into it why it is timing out09:36
chkumar|roverconsistent to tripleo ci promotoon09:36
marios|ruckchkumar|rover: yeah maybe we get lucky09:36
marios|ruckchkumar|rover: on the testproject but forgot sthing updating it now09:36
*** derekh has joined #oooq09:37
marios|ruckchkumar|rover: (consistent thing was green yesterday at https://review.rdoproject.org/r/#/c/24211/ for exmple09:39
chkumar|roveryes it is broken tonight09:40
marios|rucksounds like a love song09:40
* marios|ruck re-coffee brb09:40
*** bhagyashris has quit IRC09:41
marios|ruckchkumar|rover: https://review.rdoproject.org/zuul/stream/65444c6f81a04d719b8452b04b77d66c?logfile=console.log09:43
marios|ruckchkumar|rover: gonna time out :/09:43
marios|ruckbooooo09:44
*** ykarel|lunch is now known as ykarel09:44
chkumar|roveryes09:44
chkumar|roverI donot know what is going on there09:44
*** whoami-rajat|lun is now known as whoami-rajat09:44
*** fmount has quit IRC09:53
*** apetrich has quit IRC09:58
*** fmount has joined #oooq10:02
*** tosky has quit IRC10:02
*** tosky has joined #oooq10:03
chkumar|rovermarios|ruck: rdo current is not consistent that's why it is happening10:24
*** bhagyashris has joined #oooq10:24
chkumar|rovermarios|ruck: we need to improve the tooling which promotes from consisitent to current10:25
chkumar|rover*toci testing10:25
ykarelchkumar|rover, which release is not consistent?10:26
ykarelit looks like all realeases are consistent to me10:26
ykareli checked centos7 one10:26
ykarelohhk rhel8-master is not consistent10:27
chkumar|roverykarel: it is for RHEL810:27
marios|ruckykarel: see https://bugs.launchpad.net/tripleo/+bug/185683010:27
openstackLaunchpad bug 1856830 in tripleo "master + train promote-consistent-to-tripleo-ci-testing times out blocks master+train pipeline" [Critical,Triaged]10:27
marios|ruckthanks chkumar|rover10:27
ykarelchkumar|rover, yup got it10:27
ykareli doubted on it10:27
chkumar|roverykarel: i wasted my whole morning and afternoon in that figyring out what is going on10:28
ykarel:)10:28
ykarelit should have straight forward, chkumar|rover logs were missing?10:28
chkumar|rovermarios|ruck: may be it would be a good rfe for rlandy or foloco10:28
chkumar|roverykarel: the job got timed out10:28
chkumar|roverykarel: http://logs.rdoproject.org/41/24241/2/check/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/65444c6/job-output.txt10:29
ykarelchkumar|rover, and no logs?10:29
chkumar|roveryes no logs after at that step10:29
ykarelchkumar|rover, ack ^^ is bad, log should contain some info10:29
ykarelotherwise manual check is needed10:29
ykarellooks like there are some recent changes in that job10:30
ykareliirc earlier it used to print details10:30
ykarelthat commit is not found10:30
chkumar|roverykarel: the script is still same, just moved the script to role10:31
ykarelor it's possible timeout of job < timeout of comparison check10:31
chkumar|rovermarios|ruck: rechecking this patch https://review.rdoproject.org/r/#/c/24241/10:32
marios|ruckack chkumar|rover10:39
*** holser has quit IRC10:50
*** holser__ has joined #oooq10:50
ykarelchkumar|rover, http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-promote-consistent-to-tripleo-ci-testing/4a91560/job-output.txt10:51
ykareland job timedout is 2000, while that check for hash is 40 * 60=240010:52
ykarelhttps://github.com/rdo-infra/review.rdoproject.org-config/commit/b24bc86af75611dddd3654d4e3f1a97f6d1f9d8b10:52
ykarelhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/adb546793c1ccc627a61faa531b52c1e74a68d9b/zuul.d/tripleo-rdo-base.yaml#L24010:52
*** soniya29 has quit IRC10:53
*** bhagyashris has quit IRC10:54
chkumar|roveradded10:56
*** bhagyashris has joined #oooq10:58
*** soniya29 has joined #oooq10:58
*** ykarel is now known as ykarel|afk11:20
*** holser__ has quit IRC11:21
*** holser has joined #oooq11:22
marios|ruckrfolco: you might want to push the retro back 1 hour then for the meet&greet11:22
*** ykarel|afk has quit IRC11:30
marios|ruckso vote chkumar|rover please ? https://review.opendev.org/#/c/699624/11:31
marios|ruckanyone else ^^11:31
*** jfrancoa has quit IRC11:32
chkumar|rovermarios|ruck: done11:33
chkumar|roversshnaidm: marios|ruck https://review.rdoproject.org/r/#/c/24244/ vote please11:33
marios|ruckchkumar|rover: so do we know what's blocking the rdo consistent build11:38
marios|ruckchkumar|rover: for https://bugs.launchpad.net/tripleo/+bug/185683011:39
openstackLaunchpad bug 1856830 in tripleo "master + train promote-consistent-to-tripleo-ci-testing times out blocks master+train pipeline" [Critical,Triaged]11:39
marios|ruckchkumar|rover: checking the review you point to now11:39
marios|ruckchkumar|rover: you think timeout will help?11:39
marios|ruckchkumar|rover: merged11:39
marios|ruckjpena: is there something known blocking the consistent build wrt  https://bugs.launchpad.net/tripleo/+bug/185683011:41
openstackLaunchpad bug 1856830 in tripleo "master + train promote-consistent-to-tripleo-ci-testing times out blocks master+train pipeline" [Critical,Triaged]11:41
marios|ruckjpena: we've seen that timeout for more than a day now11:41
jpenamarios|ruck: yes, there was a build in rhel8-master that failed (oslo-rootwrap) due to a unit test timeout. It was fixed earlier today11:42
*** apetrich has joined #oooq11:42
marios|ruckthank you jpena11:44
*** jfrancoa has joined #oooq11:47
*** bhagyashris has quit IRC11:52
*** bhagyashris has joined #oooq11:58
*** epoojad1 is now known as epoojad1|afk11:59
*** sshnaidm has quit IRC12:04
*** epoojad1|afk has quit IRC12:04
chkumar|roverjpena: still not picked byt the job12:09
jpenachkumar|rover: hm, which hashes is it trying to pick?12:09
*** bhagyashris_ has joined #oooq12:09
chkumar|roverjpena: http://logs.rdoproject.org/41/24241/2/check/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/a9439f4/job-output.txt12:10
chkumar|roverjpena: 9de2e41f64593017fa9ec21902356abbc88d67af for centos12:10
chkumar|roverjpena: for RHEL it is not showing anything12:10
jpenachkumar|rover: ok, that hash was not consistent just yet. If you check https://trunk.rdoproject.org/rhel8-master/report.html, you will see that oslo-rootwrap is just the following one12:12
jpenaso the next commit in master will be ok12:12
*** bhagyashris has quit IRC12:12
chkumar|roverjpena: ok12:12
jpena(once rhel8-master builds python-hardware)12:13
*** bhagyashris_ has quit IRC12:13
chkumar|roverjpena: train is also not consistent https://trunk.rdoproject.org/rhel8-train/report.html12:14
chkumar|roverhttps://trunk.rdoproject.org/rhel8-train/a4/f6/a4f6185f76e130b2f61f840050467bf37199d141_4450dbca/rpmbuild.log12:14
jpenachkumar|rover: it is. There's a later openstack-selinux commit that was successfully built12:14
chkumar|roveryes saw that12:14
rfolcomarios|ruck, I just saw meet&greet now. I'll move one hour later, retro should be shorter this time.12:15
rfolcomarios|ruck, done.12:16
rfolcopushed 1h later and shrunk it to 2h12:17
chkumar|roverrfolco: Hello12:17
rfolcochkumar|rover, its me12:17
marios|ruckrfolco: acketty12:17
chkumar|roverrfolco: I merged one of your patch to fix one fo the timeout issye12:17
chkumar|roverrfolco: https://bugs.launchpad.net/tripleo/+bug/185683012:18
openstackLaunchpad bug 1856830 in tripleo "master + train promote-consistent-to-tripleo-ci-testing times out blocks master+train pipeline" [Critical,Triaged]12:18
rfolcochkumar|rover, my patch ? which one? or you mean rlandy?12:19
chkumar|roverrfolco: https://review.rdoproject.org/r/#/c/24236/12:20
*** ykarel|afk has joined #oooq12:21
marios|ruckchkumar|rover: ok now? https://review.rdoproject.org/r/#/c/24211/12:35
marios|ruckchkumar|rover: sorry12:35
marios|ruckchkumar|rover: that is from yesterday12:35
* marios|ruck going job crazy 12:35
chkumar|rovermarios|ruck: not yet12:36
marios|ruckchkumar|rover: rechecked https://review.rdoproject.org/r/#/c/24241/12:36
chkumar|roverok12:36
marios|ruckchkumar|rover: if the pipeline starts running i can worfklow -A to kill it assuming it works12:36
chkumar|roveryes12:37
chkumar|roverfew hours left now12:37
marios|ruckchkumar|rover: T - 23 mins and counting12:37
marios|ruck\o/12:37
marios|ruckchkumar|rover: we will enter calls and emerge with new nicks12:37
marios|ruck\o/12:37
chkumar|roveryup yup12:37
weshaygood morning / afternoon .. hey folks :) is there an issue w/ the upstream gate?12:39
marios|rucknot that am aware of weshay12:40
weshayk k12:40
marios|ruckchkumar|rover: ? 14:39 < weshay> good morning / afternoon .. hey folks :) is there an issue w/ the upstream gate?12:40
marios|ruckweshay: why you see something?12:40
*** jpena is now known as jpena|lunch12:41
weshayjust noticed a lot of gate failures and the queue is at 15hr which is unusual these days12:41
chkumar|rovermarios|ruck: in the morning, upstream gate queue was very high 9 hrs12:41
marios|rucko.k.12:41
chkumar|roverI thought it went done, people still approving patches :-(12:42
chkumar|rover*down12:42
weshaymeet and greet in 18 min.. you can add questions to https://docs.google.com/document/d/1mXCT4N4hpjPXKZwuGoArMuD0C_vZV5ij0dkawDYK9zg/edit#heading=h.gtig1khwjqzx12:42
marios|ruck2019-12-18 09:27:39,826 6410 INFO     promoter Promoting the container images for dlrn hash 80dcc19695f84819aec46fb3a600eca2bd90861e on stein to current-tripleo12:44
marios|ruckchkumar|rover: \o/ stein12:44
marios|ruckstein is very stable lately12:44
chkumar|rover\o/12:45
marios|ruckchkumar|rover: rocky should follow https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-wednesday-weekend12:46
marios|ruckchkumar|rover: but for queens it is blocked until we activate the new promoter code we should discuss on call today/handover12:46
chkumar|roverok12:46
chkumar|rovermarios|ruck: increasing timedout worked https://review.rdoproject.org/r/#/c/24208/12:53
chkumar|roverand i think we have a bug there12:53
chkumar|rover "msg": "Unsupported parameters for (command) module: environment Supported parameters include: _raw_params, _uses_shell, argv, chdir, creates, executable, removes, stdin, warn, zuul_log_id"12:54
marios|ruckchkumar|rover: cool... but still what changed since yesterday. well.. don't see how timeout has to do with consistent build delay12:54
chkumar|rovermarios|ruck: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/promote-secondary-distro.yaml#L26 and https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/promote-secondary-distro.yaml#L4612:55
chkumar|roverretries and delay12:55
*** rlandy has joined #oooq12:57
*** sshnaidm has joined #oooq12:58
*** Goneri has quit IRC12:58
*** marios|ruck is now known as marios|ruck|call13:00
weshayrfolco, can you help round folks up13:01
weshayplease13:01
*** sshnaidm has quit IRC13:02
rfolcoweshay, joining13:02
*** sshnaidm has joined #oooq13:03
rfolcorlandy, sshnaidm marios|ruck|call panda arxcruz|pto zbr13:04
rfolcohttps://bluejeans.com/4113567798?src=calendarLink13:04
rfolcoCI greet & meet ^13:04
*** amoralej is now known as amoralej|lunch13:09
*** udesale has joined #oooq13:11
*** holser has quit IRC13:17
*** jbadiapa has joined #oooq13:17
*** holser has joined #oooq13:18
*** ssbarnea has quit IRC13:22
*** ykarel|afk is now known as ykarel13:36
*** jpena|lunch is now known as jpena13:45
*** Goneri has joined #oooq13:46
*** amoralej|lunch is now known as amoralej13:51
*** fmount has quit IRC13:53
chkumar|rovermarios|ruck|call: How to pull all the jobs which are timing out from graphana?13:53
*** fmount has joined #oooq13:54
marios|ruck|callchkumar|rover: dont know13:56
sshnaidmrfolco, weshay I will join mtg in 40 mins, sorry, need to go offline14:01
rfolcosshnaidm, ack14:02
*** sshnaidm has quit IRC14:02
*** udesale has quit IRC14:02
rfolcoretro starts now at https://meet.google.com/mce-cnhy-gmk14:06
*** marios|ruck|call is now known as marios14:07
*** marios is now known as marios|call14:07
*** chkumar|rover is now known as chandankumar14:07
chandankumartimes up!14:08
marios|callchandankumar: (^^ insert coin to continue? )14:08
chandankumarrfolco: rlandy https://bugs.launchpad.net/tripleo/+bug/185683014:08
openstackLaunchpad bug 1856830 in tripleo "master + train promote-consistent-to-tripleo-ci-testing times out blocks master+train pipeline" [Critical,Triaged]14:08
rfolcohttps://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both14:10
rfolcohttps://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:UniSprint1914:10
*** soniya29 has quit IRC14:17
marios|callrfolco: chandankumar: rlandy: just one question what about the cix calls then?14:28
marios|call;)14:28
marios|callthere is one in an hour. i will join it rlandy rfolco chandankumar14:28
*** surpatil has quit IRC14:29
rfolcomarios|call, ack14:29
chandankumarrlandy: http://logs.rdoproject.org/08/24208/3/check/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/d625814/job-output.txt14:38
chandankumarrfolco: https://review.rdoproject.org/r/#/c/24208/14:39
chandankumarsorry14:39
chandankumarrlandy: https://review.rdoproject.org/r/#/c/24208/14:39
chandankumarrlandy: https://review.rdoproject.org/r/#/c/24244/14:40
chandankumarmarios|call: now it's time to open the mozito and beer14:42
chandankumarmarios|call: thank man!14:42
* chandankumar headed home14:42
marios|callchandankumar: thanks to you too14:43
marios|callchandankumar: have a good one14:43
*** jrist has joined #oooq14:44
*** marios|call is now known as marios14:45
rlandyhttps://review.rdoproject.org/r/24246 Fix spacing on secondary promote14:47
rlandymarios: ^^ chandankumar14:47
rlandyrfolco: ^^14:49
rfolco+114:49
rfolcochandankumar, ^14:49
rfolcoor marios ^14:49
mariosrfolco: want me to merge that?14:50
mariosrlandy: rather ^14:50
rlandymarios: sure14:52
rlandychandankumar: going to rerun your testproject job14:52
rlandyafter merge14:52
*** sshnaidm has joined #oooq14:53
*** TrevorV has joined #oooq14:55
rlandymarios: chandankumar: rfolco: promote secondary is fixed ... http://logs.rdoproject.org/08/24208/3/check/periodic-tripleo-centos-7-train-promote-consistent-to-tripleo-ci-testing/a891981/job-output.txt15:05
rlandysorry about that15:06
rfolconice15:06
rfolco\o/15:06
mariosthanks rlandy15:06
*** jfrancoa has quit IRC15:08
*** ykarel is now known as ykarel|away15:11
rlandymarios: chandankumar: can I close the CIX out?15:12
mariosrlandy: which one15:13
mariosrlandy: for consistent?15:13
mariosrlandy: would like to see it run first15:13
mariosrlandy: did we have green already15:13
rlandymarios: just in testproject - will wait until we have a pipeline run15:14
mariosrlandy: ah on my test in fact https://review.rdoproject.org/r/#/c/24241/15:14
mariosrlandy: didn't report yet but green15:14
mariosrlandy: ack15:14
mariosrlandy: too soon to close cix but worth a comment if you wantt15:14
rlandymarios: on https://review.rdoproject.org/r/#/c/24208/15:15
rlandybut you can recheck your patch15:15
mariosrlandy: already did it is still running but that one finished15:15
mariosrlandy: http://logs.rdoproject.org/41/24241/2/check/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/9ccab39/15:15
mariosgreen ^15:15
rlandymarios: deceptive - still a timeout in your patch15:16
rlandy2019-12-18 13:18:36.804647 | rdo-centos-7 |   "msg": "Unsupported parameters for (command) module: environment Supported parameters include: _raw_params, _uses_shell, argv, chdir, creates, executable, removes, stdin, warn, zuul_log_id"15:17
rlandywhatever - the next run should be fine15:17
*** udesale has joined #oooq15:17
*** whoami-rajat has quit IRC15:19
chandankumarrlandy: wat was the fix?15:23
rlandychandankumar: I messed up the spacing on secondary promote role15:23
chandankumarrlandy: cool15:23
rlandyhttps://review.rdoproject.org/r/24246 Fix spacing on secondary promote15:23
rlandyfix ^^15:23
rlandychandankumar: marios: am I attending the CIX call now or you guys?15:24
chandankumarrlandy: I think marios is attending the call?15:24
*** jfrancoa has joined #oooq15:24
rlandyI will join anyways so I know what's happening for monday15:24
*** udesale has quit IRC15:25
*** udesale has joined #oooq15:25
rlandyif I'm going, just need a heads up on status of https://trello.com/c/r8g1gG41/1270-cixlp1856278tripleociproa-rhel8-scenario-1-standalone-deployment-failed-with-the-following-containers-failed-validations-and-wer15:25
chandankumarsshnaidm: rlandy: please donot approve patches, gates in bad state15:28
rlandychandankumar: something wrong or just backed up?15:29
chandankumarsshnaidm: timeout15:29
chandankumarrlandy: timeout, random jobs15:29
*** marios is now known as marios|call15:29
sshnaidmchandankumar, example?15:29
chandankumarsshnaidm: minion ones check #tripleo channel15:30
*** panda has quit IRC15:31
*** panda has joined #oooq15:33
chandankumarsshnaidm: rlandy https://bugs.launchpad.net/tripleo/+bug/185686415:34
openstackLaunchpad bug 1856864 in tripleo "Not all firewall rules are applied, leading to timeouts" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)15:34
sshnaidmso it's still firewall module..15:37
EmilienMno15:40
*** holser has quit IRC15:40
EmilienMit's THT15:40
*** holser has joined #oooq15:40
EmilienMfirewall module brought it up because https://review.opendev.org/699692 is needed I think15:40
rlandyok collect logs people ...15:48
rlandychandankumar: ^^  I eventually managed to sanitize the logs for redhat.com and download links ...15:49
rlandyhttps://review.opendev.org/#/c/691994/15:50
rlandyit's WIP right now15:50
rlandyI'll change that15:50
rlandythink it's ready to go - see sanitized logs in https://sf.hosted.upshift.rdu2.redhat.com/logs/61/187961/1/check/tripleo-ci-rhel-8-standalone-rhos-16/f985d1a/logs/undercloud/etc/yum.repos.d/rhos-release-16-trunk.repo.txt.gz15:51
rlandysshnaidm: rfolco: marios|call: ^^ fyi15:51
chandankumarrlandy: cool, will check that15:51
rlandychandankumar: thanks, no major rush15:52
chandankumartomorrow will be my last day of year here, anything to pick up?15:52
rlandydlrnapi --url https://trunk.rdoproject.org/api-rhel8-train --username review_rdoproject_org repo-promote --commit-hash 80cc8ed4b1d2e10e7c4a518dd4ca1a5305ed8ede --distro-hash a18d663b61419140a86d912c4c9545e8f76ff36c --promote-name tripleo-ci-testing15:52
rlandyworking now15:53
chandankumar\o/15:53
chandankumarwe might see master and train promotion tomorrow15:53
*** marios|call is now known as marios15:54
rlandyhopefully15:54
chandankumarfor master it will be needed https://review.opendev.org/#/c/699624/15:54
chandankumarrlandy: ^^, but gate is in bad state15:54
rlandychandankumar: well I'll +2 it in the mean time15:55
rlandyno w+15:55
chandankumaraye15:55
rlandytripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020 still fails15:55
*** udesale has quit IRC15:56
chandankumarrlandy: may be race issue15:56
chandankumartesttools.matchers._impl.MismatchError: <bound method type.create_image_from_server of <class 'tempest.api.compute.images.test_images_oneserver_negative.ImagesOneServerNegativeTestJSON'>> returned {}15:57
marioschandankumar: yes it is a race16:04
marioschandankumar: its on the bug16:04
marioschandankumar: like the state didn't finish transition to dlete yet or something16:05
marioschandankumar: per https://bugs.launchpad.net/tripleo/+bug/1856723/comments/3 i mean or you talking about sthing else16:05
openstackLaunchpad bug 1856723 in tripleo "periodic-tripleo-ci-centos-7-standalone-full-tempest-master fails during tempest run ImagesTestJSON:test_delete_saving_image" [Critical,Triaged]16:05
chandankumarmarios: it is the same test but negative16:06
marioschandankumar: so add it to https://review.opendev.org/#/c/699624/16:06
chandankumaryup16:06
*** ksambor has quit IRC16:07
chandankumarok time's up see ya tomorrow16:10
chandankumarmarios: take some good sleep please !16:10
marioschandankumar: have a good one me almost out too16:10
marioschandankumar: you too mate16:10
*** chandankumar is now known as raukadah16:10
raukadahrlandy: thank you for taking the charge16:10
*** marios is now known as marios|out16:15
rlandyk - I'll add that test16:19
*** marios|out has quit IRC16:26
sshnaidmrlandy, what does sanitize_lines valie look like?16:39
rlandysshnaidm: see example https://code.engineering.redhat.com/gerrit/#/c/184172/11/config/release/rhos-16.yml16:40
rlandycould really be anything16:40
rlandywas just experimenting with the ability to sanitize log strings16:40
sshnaidmrlandy, ack, can you please add it to readme?16:40
rlandyyep - will do16:40
rlandyraukadah: updated skip list to add second failing test ...  https://review.opendev.org/69962416:44
*** kopecmartin is now known as kopecmartin|off16:58
*** dtrainor has quit IRC17:08
*** dtrainor has joined #oooq17:08
rfolcorlandy, https://review.rdoproject.org/r/2425017:16
rfolcorlandy, fix yaml criteria loading17:16
rfolcofor promote to current-tripleo17:16
rlandyneed to run out for a few - will look when I get back17:17
rlandyrfolco: ^^17:17
*** rlandy is now known as rlandy|brb17:17
rfolcok17:18
*** ykarel|away has quit IRC17:26
*** jpena is now known as jpena|off17:36
*** panda has quit IRC17:36
*** panda has joined #oooq17:39
raukadahrfolco: can you point me the standlone job running in component promotion?17:39
*** ssbarnea has joined #oooq17:40
rfolcoraukadah, https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-compute17:41
rfolcoraukadah, https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-keystone17:41
raukadahrfolco: ah good, we need to fix the tempest tests there17:43
rfolcoraukadah, cool17:44
*** rlandy|brb is now known as rlandy17:46
rlandyraukadah: pls check https://review.opendev.org/#/c/699624/17:47
raukadahrlandy: done17:48
rlandyrerunning the train failures17:49
rlandyhopefully we get a rhel train promotion17:49
raukadahrlandy: yes17:49
raukadahrlandy: if fs01 is excluded, fs01 master /train was a nightmare for me and marios17:50
rlandyraukadah: let's see how that one goes17:50
rlandyperiodic-tripleo-ci-rhel-8-scenario001-standalone-train17:50
rlandy^^ rerunning17:50
raukadahrdo people added c8 based jobs for packstack, we will be picking in new year17:51
rlandy "The following containers failed validations and were not started: collectd"17:51
rlandy2019-12-18 16:56:25 |     ]17:51
raukadahrlandy: awesome17:51
rlandy^^ that may be real error17:51
rlandyfrom CIX17:52
raukadahrlandy: https://review.opendev.org/#/c/699438/17:52
raukadahis the fix hunted down due to gate issue17:52
rlandyraukadah: k - let's recheck that17:52
raukadahour friend Tengu is fixing a lot of selinux issue, he is our selinux expert after daniel walsh in tripleo land17:53
rlandyraukadah: restoring https://review.opendev.org/#/c/699438/17:53
raukadahcool17:53
rlandythe scenario 1 failure is https://trello.com/c/r8g1gG41/1270-cixlp1856278tripleociproa-rhel8-scenario-1-standalone-deployment-failed-with-the-following-containers-failed-validations-and-wer17:54
rlandyok - so no promotion17:55
rlandysorry17:55
*** chem_ has quit IRC17:55
raukadahfs01 train rhel8 is a circle17:56
rlandywhen we get that fix, it should be ok17:57
raukadahno idea17:57
rlandyat least it's not my fault this time17:58
raukadahrfolco: rlandy sshnaidm : Did you stocked mozitoo and beer for holiday celebration?17:58
sshnaidmraukadah, for Hanukkah?17:59
rlandyraukadah: not much of a drinker, sorry17:59
rlandyraukadah: it's more like an eating holiday for us17:59
raukadahI am emptying my freeze today as I will not drink at home as my sister will kill me17:59
rlandylots of oil17:59
raukadahsshnaidm: yes17:59
raukadahrlandy: oil?18:00
raukadahyou mean in fish or chicken or veggy?18:00
sshnaidmwe drink from HeBrewery18:01
rlandytraditionally olive oil18:01
rlandybut these days mostly donuts18:02
raukadahI love oil when it comes to this https://en.wikipedia.org/wiki/Puri_(food)18:02
raukadahI love a lot18:02
raukadahsshnaidm: HeBrewery ?18:02
raukadahDoes brewery have a gender?18:03
sshnaidmraukadah, hebrew brewery :)18:03
raukadahsshnaidm: :-)18:03
rlandyha - fs01 rhel train is on tempest tests18:04
raukadahsshnaidm: it is a beer, not wine18:04
*** derekh has quit IRC18:04
sshnaidmraukadah, yeah18:04
* raukadah will ask for the same in wine shop tomorrow evening18:04
sshnaidmrlandy, one more reason to celebrate..18:04
sshnaidmnes hanukkah :)18:05
raukadahrlandy: does not say tempest failed on fs0118:05
rlandyit's still running18:05
*** rfolco has quit IRC18:07
raukadahsshnaidm: by the way, contributing to podman is fun, they believe a lot on contributors18:07
raukadahlast systemd does not fixed the issye, there is an issue with makefile find string still18:07
raukadahI realized late18:08
sshnaidmraukadah, is it issue with building rpm?18:08
raukadahthere is no issue just the message18:09
sshnaidmI see18:09
raukadahsshnaidm: https://github.com/containers/libpod/blob/master/Makefile#L4518:09
rlandywaiting for log collection is like watching a pot boil18:10
raukadahmay be above makefile issue18:10
raukadahrlandy: write a async bot to deal with finger protocol18:10
raukadahidea is nice, /me thinking how to parse finger stdout in runtime to remind at particular failure18:12
raukadahsshnaidm: this holiday I want to work on this https://github.com/containers/libpod/issues/713 may be it will solve rlandy rfolco and marios issue18:14
rlandywhich issue?18:15
sshnaidmraukadah, nice18:16
sshnaidmraukadah, it's weird no gerrit plugins with job finish notifications18:16
raukadahrlandy: manifest one https://github.com/containers/libpod/issues/713 buildah has support for that18:16
raukadahsshnaidm: tweet gerrrit review they are quite responsive18:16
*** yolanda__ is now known as yolanda18:18
rlandywhat happened to rfolco?18:18
rlandyhe run away18:19
rlandyright when I am ready to merge his work18:19
raukadahsshnaidm: regarding docker compose review on reproducer , you are using kompose to comnvert docker-compose.yaml to k8s yaml to work with podman as I saw some shell scripts18:19
raukadah?18:19
sshnaidmraukadah, we tried kompose, didn't work in our case :( so we use podman pods18:19
sshnaidmraukadah, and podman pod config you can dump to k8s  with "generate-kube" or kind of18:20
raukadahsshnaidm: I used to know few kompose devs but nolonger work in RH18:20
sshnaidmraukadah, there is also podman-compose, but it doesn't work as well..18:20
* raukadah not looked at podman-compose, will poke on that18:21
raukadahrlandy: regarding current promote hash shell script I was thinking to use this https://nox.thea.codes/en/stable/18:22
raukadahactivate dlrn venv part basically18:23
*** tosky has quit IRC18:23
rlandyraukadah: I am not breaking promotions again this year - nice idea for next time18:23
raukadahsshnaidm: thanks :-)18:23
raukadahrlandy: sshnaidm next year is going to be fun as zuul + jenkins stuff18:24
raukadahfrom downstream CI18:25
rlandyraukadah: periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train success18:25
raukadahw00t18:25
sshnaidmraukadah, I hope we'll use our downstream Zuul more and more18:25
rlandywaiting on the master run18:25
raukadahrlandy: Tengu will be happy by seeing this18:26
rlandysshnaidm: just fyi - wrote this over thanksgiving to get rid of rhel containers on docker https://review.rdoproject.org/r/#/c/23894/18:27
rlandytox failure due to my commenting out18:27
rlandybeen waiting for weshay to approve it18:27
*** Goneri has quit IRC18:28
rlandyok - we have a gate queue again18:28
raukadahsshnaidm: rlandy I got the permission from dwalsh, let;s implement and use it and then curse podman more18:28
weshayrlandy,  this one? https://review.rdoproject.org/r/#/c/23894/2/ci-scripts/infra-cleanup/container-trimming.py18:28
raukadahmanifest one18:28
rlandyoh hey weshay is alive18:28
weshay0/18:28
raukadahweshay: \o/18:28
rlandyweshay: git more urgtent stuff for you to look at18:28
rlandyurgent18:28
weshayrlandy, point at what you need18:28
sshnaidmrlandy, nice! did you run it?18:28
raukadahweshay: I will be spending some cycle on this https://github.com/containers/libpod/issues/713 to get rid of docker18:29
rlandysshnaidm: yeah - without the delete  so it just prints out what it deletes18:29
sshnaidmraukadah, implement what?18:29
raukadahsshnaidm: https://github.com/containers/libpod/issues/71318:29
rlandyweshay: downstream pipeline in action18:29
weshayrlandy, oh nice!18:29
rlandyweshay: got the log sanitize working18:29
rlandyso we are ready to work on reporting18:30
rlandyie: downstream cockpit18:30
rlandyweshay: you back on friday?18:30
weshayaye18:30
rlandyweshay: ok - will chat then18:30
weshayah cool!18:30
rlandyneed to confirm which strings to get rid of18:30
rlandyand then add reports18:31
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-component-compute18:31
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-component-keystone18:31
raukadahweshay: I like podman, we as a team can make to impact there and make tripleo and friends awesome, if you allow18:31
weshaynice work rlandy++18:32
weshayraukadah, /me reads through that18:32
raukadahrlandy: is it possible to send downstream job status to upstream cockpit?18:34
rlandyraukadah: that is my next move18:34
rlandywe need to get the log sanitize patch through first18:35
rlandyworking on it18:35
*** amoralej is now known as amoralej|off18:44
*** rfolco has joined #oooq18:52
*** sshnaidm is now known as sshnaidm|afk18:58
*** chem_ has joined #oooq19:12
*** chem_ has quit IRC19:18
*** chem_ has joined #oooq19:21
*** holser has quit IRC19:24
*** chem_ has quit IRC19:26
*** chem_ has joined #oooq19:27
*** jbadiapa has quit IRC19:31
*** Goneri has joined #oooq19:37
*** chem_ has quit IRC19:52
*** chem_ has joined #oooq19:57
Tenguraukadah: no way I can be compared to Dan for SELinux o_O20:12
Tenguraukadah: I'm just playing around and gnawing some dry bones ^^'20:13
*** rf0lc0 has joined #oooq20:15
*** rfolco has quit IRC20:17
*** rf0lc0 has quit IRC20:41
rlandyTengu: we appreciate those willing to gnaw at dry selinux bones to fix bugs20:52
*** holser has joined #oooq20:54
*** rf0lc0 has joined #oooq21:26
*** rf0lc0 has quit IRC21:26
rlandytrain is promoting21:27
weshaywoot21:31
*** saneax has quit IRC21:55
*** TrevorV has quit IRC22:10
*** jfrancoa has quit IRC22:10
*** chem_ has quit IRC22:11
*** saneax has joined #oooq22:54
*** holser has quit IRC23:09
*** rlandy is now known as rlandy|bbl23:11
*** ccamacho has quit IRC23:23
*** saneax has quit IRC23:24

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!