Tuesday, 2019-10-29

*** rlandy has quit IRC00:43
*** ccamacho has quit IRC00:51
*** sanjayu_ has joined #oooq01:18
*** sanjayu_ has quit IRC01:53
*** jschlueter has quit IRC04:27
*** akahat has joined #oooq04:30
*** dtrainor_ has quit IRC05:56
*** dtrainor has joined #oooq06:01
*** marios has joined #oooq06:09
*** marios is now known as marios|ruck06:24
*** sanjayu_ has joined #oooq07:00
*** akahat has quit IRC07:05
*** ccamacho has joined #oooq07:33
*** jtomasek has joined #oooq07:37
*** akahat has joined #oooq07:42
*** tesseract has joined #oooq07:49
*** kopecmartin|off is now known as kopecmartin08:05
*** bogdando has joined #oooq08:20
*** akahat has quit IRC08:27
*** akahat has joined #oooq08:49
*** amoralej|off is now known as amoralej08:50
*** jpena|off is now known as jpena08:51
*** dtantsur|afk is now known as dtantsur08:55
*** holser has joined #oooq09:05
*** soniya29 has joined #oooq09:18
*** akahat has quit IRC09:26
zbrsshnaidm: can you help me wf https://review.opendev.org/#/c/691671/ ?09:56
marios|rucksshnaidm: wdyt rfolco|ruck  https://review.rdoproject.org/r/#/c/23503/ bump 2.5 hours based on comment 1 https://bugs.launchpad.net/tripleo/+bug/1850188 until we have the root problem/fix and unblock train10:08
openstackLaunchpad bug 1850188 in tripleo "periodic container build (train) timing out" [Critical,Triaged]10:08
sshnaidmzbr, done10:12
sshnaidmmarios|ruck, fine, +210:12
sshnaidmand +w10:12
marios|rucksshnaidm: thanks10:12
zbrthanks10:12
*** jfrancoa has joined #oooq10:39
*** jfrancoa has quit IRC10:40
*** rfolco|ruck has quit IRC11:04
*** rfolco|ruck has joined #oooq11:04
*** jschlueter has joined #oooq11:04
*** soniya29 has quit IRC11:12
*** matbu has quit IRC11:29
*** soniya29 has joined #oooq11:30
*** soniya29 is now known as soniya29|afk11:30
*** soniya29|afk has quit IRC11:35
*** arxcruz is now known as arxcruz|lunch11:37
weshay|rovermarios|ruck, do you have some time for some tripleo technical support for me about 15min probably once you are in context of the issue11:39
weshay|roveror the problem I'm trying to poke at11:39
marios|ruckweshay|rover: o/11:40
marios|ruckweshay|rover: sure lemme just look at /note this stein thing gimme couple mins11:41
weshay|roverk k11:41
marios|ruckweshay|rover: fs20 killed stien promote https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-24hr11:41
weshay|rovermarios|ruck, k.. I'll go waive on the prom server11:42
weshay|roverI see 2 failures and the next fs20 passed11:42
marios|ruckweshay|rover: tempest http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/b4e0e28/logs/tempest.html.gz11:45
weshay|roveraye11:46
weshay|rover2 fail11:46
marios|ruckweshay|rover: thanks... train is the real concern and we need luck on our side for a promotion by tomorrow hoping the next run will be ok with https://review.rdoproject.org/r/#/c/23503/ debug https://review.rdoproject.org/r/#/c/23502/11:46
marios|ruckhttps://bugs.launchpad.net/tripleo/+bug/1850188 weshay|rover11:46
openstackLaunchpad bug 1850188 in tripleo "periodic container build (train) timing out" [Critical,Triaged]11:46
weshay|rovermarios|ruck, passes here http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset027-stein/a8baf13/logs/tempest.html.gz11:46
marios|ruckweshay|rover: anyway what should i look at to help you11:47
weshay|rovermarios|ruck, see pm11:47
*** Goneri has quit IRC11:54
*** jaosorior has joined #oooq11:56
*** amoralej is now known as amoralej|lunch12:13
*** rfolco|ruck is now known as rfolco|rover12:17
*** rlandy has joined #oooq12:20
*** Goneri has joined #oooq12:23
rfolco|rovermarios|ruck, hope I followed the standard with the etherpad :)12:24
marios|ruckrfolco|rover: thanks for the updates folco /me quick food brb12:25
rfolco|rover:)12:25
*** Goneri has quit IRC12:28
*** Goneri has joined #oooq12:29
*** jpena is now known as jpena|lunch12:29
*** Goneri has quit IRC12:35
*** jaosorior has quit IRC12:39
marios|ruckweshay|rover: added some notes hope it helps a bit for your digging12:52
*** Goneri has joined #oooq12:56
*** arxcruz|lunch is now known as arxcruz12:56
*** Goneri has quit IRC13:04
*** Goneri has joined #oooq13:15
*** jpena|lunch is now known as jpena13:19
*** jfrancoa has joined #oooq13:26
rfolco|roverci community call starting in 3 min at https://meet.google.com/bqx-xwht-wky13:27
rfolco|roveragenda https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both13:27
rfolco|roverping weshay|rover rlandy zbr arxcruz ^13:30
weshay|roverrfolco|rover, I'm in manager meetings for the next 3-4 hours13:30
rfolco|roverweshay|rover, ack13:30
*** amoralej|lunch is now known as amoralej13:44
*** jaosorior has joined #oooq13:47
amoralejhi wrt https://review.opendev.org/#/c/691698/ the errors in third party are known issue?13:50
marios|ruckbeagles: fyi some intro info on the pipeline and check/gate/periodics voting etc https://docs.openstack.org/tripleo-docs/latest/ci/ci_primer.html13:51
beaglesmarios|ruck, ack, thanks!13:52
*** jaosorior has quit IRC13:59
* chandankumar is back!14:01
mjturekamoralej: FWIW there were three runs, across those three runs there were successes for all of the jobs14:02
*** jaosorior has joined #oooq14:03
*** TrevorV has joined #oooq14:04
*** weshay|rover is now known as weshay14:09
weshayrfolco|rover, did you submit those patches for scenario012-multinode ?14:20
rfolco|roverweshay, yes14:21
rfolco|roverweshay, https://review.opendev.org/69173914:21
weshaythanks14:22
arxcruzzbr: https://review.rdoproject.org/r/#/c/23508/ this adds the rhel8 version of fs01614:35
arxcruzi did not remove the centos07 because it would cause circular dependence14:36
arxcruzzbr: i'll submit another patch once this is fixed, changing the ceph-ansible job to this one, and then another to remove the centos-714:36
zbrrlandy: is TripleoFirewall a required service?14:56
rlandyzbr: context?  ceph-ansible? idk14:57
zbrrlandy: i am trying to figure-out what i need to put on https://review.opendev.org/#/c/690960/1/ci/environments/scenario000-standalone.yaml14:57
zbri also found that it failed to install openvswitch on c8 on http://logs.rdoproject.org/23/23423/2/check/tripleo-ci-centos-8-scenario000-standalone/e6349d5/ -- and it seems to run multi-node related stuff, but i have no idea what made it decide to do multi-node stuff.15:05
zbri think i found: common_featureset: featureset-multinode-common.yml15:07
zbrrlandy: maybe you can explain me these two lines: https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml#L131-L13215:08
rlandyzbr: sec - just closing out a prod-chain discussion15:09
chandankumararxcruz: Hello15:19
chandankumararxcruz: Nice to meet you in person :-)15:20
chandankumararxcruz: can we create a seperate pipeline and job name for third party gating ?15:20
chandankumardistgit jobs are meant for spec file and rdo stuff15:21
arxcruzchandankumar: ok, i'll check that, can you please comment that on the taiga card?15:22
chandankumararxcruz: sure!15:22
chandankumararxcruz: I have travelled a lot and visited so many places in last 3 days15:23
chandankumarit was kind of fun , tired and new experience15:23
arxcruzlucky you :)15:23
chandankumararxcruz: that was all of because of china visa mess!15:24
arxcruzso, unlucky you :P15:24
chandankumarhehe15:24
chandankumarrlandy: hey, How are you ?15:24
rlandychandankumar: busy :)- happy to hear you got your visa15:25
rlandyfaster than weshay :)15:25
weshaymuch15:25
chandankumarrlandy: is it? I got in 2 day. but paid a lot of money!15:25
chandankumarbut spent 4-5 hrs standing in queue for 2 days15:27
chandankumarI will share some nice clicks tomorrow!15:27
arxcruzi spend more time in the queue for the us visa, than to get the visa, 3 hours in the queue, against 5 minutes interview15:29
chandankumararxcruz: funny part was that, there is no interview, they just checked and said you work for RH, you will get visa easily15:30
rlandyzbr: sorry - reading back15:30
chandankumareven on photos on visa15:31
chandankumar*no15:31
chandankumarpanda|pto: zbr: rlandy arxcruz Thank you for solving puzzle around third party gating :-)15:31
chandankumarwe are finding issues for them15:32
rlandyzbr: environment_type refers to https://github.com/openstack/tripleo-ci/blob/master/toci-quickstart/config/testenv/singlenode.yml15:33
rlandysee other environment_types in https://github.com/openstack/tripleo-ci/tree/master/toci-quickstart/config/testenv15:33
rlandycommon_featureset: featureset-multinode-common.yml refers to ...15:34
rlandyhttps://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset-multinode-common.yml15:34
rlandyzbr: what else?15:36
marios|ruckchandankumar: rfolco|rover lets merge that? https://review.rdoproject.org/r/#/c/23260/15:42
marios|ruckrlandy: sshnaidm ^^^ ? merging...15:42
chandankumarmarios|ruck: thanks for checking on that, I completely forget about that15:43
marios|ruckchandankumar: ack just doing some reviews before eod15:43
chandankumarmarios|ruck: :-)15:44
rlandythanks15:45
* chandankumar missed Diwali for first time. But as a expense got lots of fun!15:47
chandankumarrlandy: any thing I can help tomorrow morning on ceph-ansible gating stuff?15:49
chandankumar*thing15:49
rlandychandankumar: haven't really done anything more on that15:50
rlandyneed to read through arxcruz latest comments15:50
rlandyif we are confirmed on the dependency triggering, we could start designing jobs15:50
arxcruzrlandy: talked with fultonj and gfident this morning to fix the tags on ceph-ansible, they were supposed to start a thread on ceph and cc me15:51
arxcruzi'll ping them tomorrow15:51
rlandyarxcruz: thanks15:51
arxcruzthe patch to fix the rpm, was approved, waiting for merge15:51
chandankumarrlandy: I need a set a meeting with the team on status and conclusion, we need sshnaidm eyes on that also!15:52
rlandychandankumar: need to work on a log sanitizing patch for downstream today - so where ever you want to pick pup15:52
rlandychandankumar: ack - been a lot going on15:52
chandankumarrlandy: distro zuul jobs stuffs I will pick tomorrow morning15:52
rlandychandankumar: ack - I thought we were not using those jobs?15:53
chandankumarrlandy: nope, we are not using it yet15:53
rlandywill touch base tomorrow and rethink now that we have the repos passing15:53
sshnaidmchandankumar, yeah, feel free to ping me15:54
chandankumarJust a learning, collaboration is not hard, unless we find a intersection point!15:54
chandankumarsshnaidm: welcome back, How was the vacation ?15:54
sshnaidmchandankumar, great, how is your bootcamp?15:55
*** Goneri has quit IRC15:56
chandankumarsshnaidm: It was awesome, I copied some technices from cloundnull talk and applied to tempest, I hope it went well,15:56
sshnaidmchandankumar, great15:56
chandankumarsshnaidm: rlandy marios|ruck rfolco|rover zbr https://gitlab.cee.redhat.com/chkumar/tripleo-bootcamp-tempest/blob/master/tempest_workflow.md we are tried to make learning curve easier, please go through it, and shoot queries to arxcruz and kopecmartin I hope it will help debugging team issues15:58
chandankumar*item15:59
arxcruzyup15:59
rlandycool - thanks for that15:59
chandankumar+ tempest recording so many stuff we tried to share, what we do , bad thing is that we are bad at advertizing our stuff16:00
marios|ruckchandankumar: nice maybe can go to tripleo-docs or somewhere else upstream?16:01
chandankumarmarios|ruck: there is an AI on us to conslodiate stuff on tripleo and tempest16:02
chandankumarin upstream docs, recent doc migration broke the stuff16:02
marios|ruckchandankumar: ack thanks16:02
chandankumarfs021 part was awesome, thanks to weshay for highlighting that :-)16:03
weshaywas a great presentation16:03
arxcruzrfolco|rover: chandankumar weshay: there's a prod chain meeting tomorrow, in the middle of our sprint end, how to proceed?16:06
weshayarxcruz, you don't normally go to that :P16:07
arxcruzweshay: Prod Chain Realignment all-hands16:07
arxcruzweshay: looks important16:07
weshayarxcruz, chandankumar sshnaidm rlandy rfolco|rover marios|ruck ok.. on me for a minute if you can... https://meet.google.com/_meet?authuser=116:08
weshayzbr, ^16:08
arxcruzweshay: meet code ?16:08
weshayor if you are interested in hearing what's going to happen16:08
weshayhttps://meet.google.com/ndd-zvjj-frh16:08
weshayrfolco|rover, chandankumar just 5 min16:10
weshayrfolco|rover, need for a sec16:11
marios|ruckweshay: sorry was afk joined now16:17
*** jaosorior has quit IRC16:24
chandankumarweshay: one min joning16:27
marios|ruckrfolco|rover: /me gonna go soon after the call need me for sthing?16:32
rfolco|rovermarios|ruck, don't think so :)16:32
marios|ruckrfolco|rover: t-24 ;)16:33
rfolco|rover\o/16:33
*** TrevorV has quit IRC16:34
*** tesseract has quit IRC16:38
*** jpena is now known as jpena|brb16:45
*** amoralej is now known as amoralej|off16:48
marios|ruckrfolco|rover: rlandy sshnaidm weshay zbr chandankumar please review + merge in your day https://review.opendev.org/#/q/topic:scenario9rocky for rocky gate please thanks16:54
*** kopecmartin is now known as kopecmartin|off16:59
*** bogdando has quit IRC17:00
*** d0ugal has quit IRC17:00
marios|ruckrfolco|rover: honolulu booked17:02
rfolco|rovermarios|ruck, :)17:02
marios|ruckbuckets of hummus17:11
*** d0ugal has joined #oooq17:15
*** Goneri has joined #oooq17:17
marios|ruckrfolco|rover: hoping the timeout bump helps us on train looks good at test review comment #2 https://bugs.launchpad.net/tripleo/+bug/185018817:18
openstackLaunchpad bug 1850188 in tripleo "periodic container build (train) timing out" [Critical,Triaged]17:18
marios|ruckrfolco|rover: would be nice for prog call :/17:18
*** jpena|brb is now known as jpena17:24
*** marios|ruck is now known as marios|out17:26
*** sshnaidm is now known as sshnaidm|afk17:31
*** dtantsur is now known as dtantsur|afk17:41
*** marios|out has quit IRC17:42
arxcruzrlandy: weshay  chandankumar https://review.opendev.org/#/c/689106 when you guys have time17:59
weshayzbr, send me that link again arxcruz18:02
weshayah I still have it :)18:03
rlandylooking18:03
*** jaosorior has joined #oooq18:24
*** jaosorior has quit IRC18:26
rfolco|roverweshay, fyi new promoter bug https://bugs.launchpad.net/tripleo/+bug/185052718:27
openstackLaunchpad bug 1850527 in tripleo "new promoter fails to push to registry" [Critical,Triaged]18:27
*** jpena is now known as jpena|off18:30
*** holser has quit IRC19:18
*** jbadiapa has quit IRC19:26
*** rlandy has quit IRC19:49
*** rlandy has joined #oooq19:50
*** panda|pto has quit IRC19:55
*** sanjayu_ has quit IRC19:58
*** panda has joined #oooq19:58
*** aakarsh has joined #oooq20:04
weshayrfolco|rover, did we remove the rhel8 periodic jobs from train?20:19
rfolco|roverweshay, we only have in master afaik20:19
weshayoh right. k20:20
rfolco|roverweshay, we did not add to train after branching20:20
weshayrfolco|rover, because rel-del hasn't added the packages20:20
rfolco|roveryep that20:20
weshayrfolco|rover,  looks like it's up though https://trunk.rdoproject.org/rhel8-train/report.html20:20
*** holser has joined #oooq20:28
weshayrfolco|rover, looks like they are done, can you please create a review to add rhel8 to train promotion jobs https://tree.taiga.io/project/morucci-software-factory/us/301120:31
rfolco|roverweshay, ok will do20:31
weshaythanks20:31
rfolco|roverweshay, w+ this one pls https://review.opendev.org/#/c/691739/20:46
weshayrfolco|rover, looks like we'll get a train promotion today :)20:47
weshayit's always in the nic of time20:47
weshayrfolco|rover, aye looking20:47
rfolco|roverweshay, nice, I can feel good vibes20:47
rfolco|roverthis one is not that urgent but I believe its a good idea: https://review.opendev.org/#/c/690467/20:48
rfolco|roverrlandy, weshay ^20:48
weshayrfolco|rover, k.. will watch it20:49
weshayrfolco|rover, while ur on rhel8 train.. it would be nice to run the rhel8 ovb job against tripleo-ci changes as well.. now that it's working again :)20:50
rfolco|roverweshay, hmm ok I see20:50
rlandyweshay: have no access to ilo or those machines - logging that with the request20:58
weshayrlandy, perfect20:58
rfolco|roverweshay, we currently do not run branchful jobs for rhel8 in tripleo-ci... extra load/resources21:05
weshayrfolco|rover, ya.. all we've had is master right?21:06
weshayor am I missing the context21:06
weshayrfolco|rover, so w/o team discussion I think we can safely say.. two things about rhel821:07
weshaywe want rhel8-train in promotion21:07
weshayand we want ovb fs001 rhel8 against ci repos21:07
weshaythe later being just master21:07
rfolco|roverweshay, correct, look what we run today https://review.opendev.org/#/c/669978/21:07
rfolco|roverfs001 rhel8 is not run, ok.21:08
weshayrfolco|rover, OH you are saying we're not running train on centos7 branchful21:08
rfolco|roverweshay, also that21:08
weshayya.. I think we can replace any 3rd party job in train w/ rhel821:08
weshaybut let's start w/ promotion jobs w/o criteria21:09
weshayand see how they go21:09
weshayand in master keep rhel8 working :)21:09
rfolco|roverok, that requires a more careful inspect21:09
rfolco|rovershutdown sequence for today21:09
weshayrfolco|rover, my ask atm.. .for tomorrow21:10
rfolco|roverweshay, thx21:10
weshaywould be.. rhel8 promotion jobs in train21:10
rfolco|roverok21:10
weshayand get the rhel8 - master fs001 running against ci repos21:10
weshayso we can better track it21:10
weshayand not break it :)21:10
rfolco|roverk master fs001... we discuss centos replacement tomorrow21:11
rfolco|rovergot it21:11
*** rfolco|rover is now known as rfolco|off21:12
*** Goneri has quit IRC21:27
*** rfolco|off has quit IRC21:31
*** jfrancoa has quit IRC21:34
*** pierreprinetti has left #oooq22:18
rlandyweshay: sanity check, collect_logs would use vars set on the job definition, right?22:21
weshayrlandy, should use what's in the env settings22:22
weshaybut22:22
weshaygiven what we talked about22:22
rlandyenv_settings is public22:22
weshayya.. thinking22:23
weshayrlandy, so riddle me this22:24
rlandychecking files passed to collect logs22:24
rlandyextra_vars: ('@/home/zuul/workspace/.quickstart/config/release/tripleo-ci/RedHat-8/promotion-testing-hash-rhos-16.yml'22:25
rlandy^^ think I can use the release file22:26
weshayrlandy, ya.. but that's coming from internal22:26
rlandythat's good22:26
weshayrlandy, you can put it in internal22:26
rlandylet's see if it works22:27
weshaythe beauty of setup.cfg22:27
rlandyhttps://github.com/openstack/tripleo-ci/blob/master/roles/run-test/templates/toci_quickstart.sh.j2#L5422:27
rlandypromising ,,,22:27
rlandyarxcruz: around by any chance?23:17
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/6838bb9/job-output.txt23:17
rlandy^^ got a pretty consistent tempest ping error on master23:17
rlandymissing a setting?23:17
*** Goneri has joined #oooq23:29
*** Goneri has quit IRC23:38
*** holser has quit IRC23:43

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!