Monday, 2020-01-06

*** tosky has quit IRC00:05
*** holser has quit IRC00:52
*** holser has joined #oooq00:58
*** holser has quit IRC01:09
*** holser has joined #oooq01:13
*** ysandeep has joined #oooq01:27
*** holser has quit IRC01:40
*** holser has joined #oooq03:09
*** sanjayu_ has joined #oooq03:55
*** udesale has joined #oooq03:55
*** skramaja has joined #oooq04:30
*** ykarel|away has joined #oooq04:37
*** chandankumar has joined #oooq04:50
*** sanjayu__ has joined #oooq04:52
*** sanjayu_ has quit IRC04:55
*** holser has quit IRC05:00
*** ykarel|away is now known as ykarel05:09
*** bhagyashris has joined #oooq05:19
*** chandankumar has quit IRC05:32
*** chandankumar has joined #oooq05:53
*** akahat has joined #oooq05:55
*** surpatil has joined #oooq06:05
*** chandankumar has quit IRC06:14
*** SurajPatil has joined #oooq06:16
*** chandankumar has joined #oooq06:17
*** surpatil has quit IRC06:18
*** surpatil has joined #oooq06:19
*** SurajPatil has quit IRC06:21
*** soniya29 has joined #oooq06:40
*** akahat has quit IRC06:42
*** chandankumar has quit IRC06:42
*** marios has joined #oooq06:51
*** ratailor has joined #oooq06:56
*** chandankumar has joined #oooq07:05
*** akahat has joined #oooq07:17
chandankumarmarios, Good morning :-)07:34
chandankumarmarios, Happy new year :-)07:34
*** jtomasek has joined #oooq07:37
marioschandankumar: o/ \o/07:40
*** ykarel is now known as ykarel|lunch07:40
*** tesseract has joined #oooq08:11
*** d0ugal has joined #oooq08:25
arxcruzgood morning people08:26
*** d0ugal has joined #oooq08:28
*** ykarel|lunch is now known as ykarel08:31
arxcruzzbr: weshay|ruck so, for libpod, shall we use scenario000 ? since we just need to test the rpm intallation and a small set of services ? this would be faster (i believe 30 minutes)08:41
arxcruzsoniya29: https://review.opendev.org/#/c/701016/4/config/general_config/featureset020.yml is an example you can use to run only the tests you want on featureset02008:42
sshnaidmo/08:50
arxcruzmarios: just checked the tempest timeout, and we are already using the defaults (60 seconds) on featureset020, so I don't think this will help08:51
arxcruzmarios: what might help if we want to keep only one job, is increase the resources, but I don't know if that's a good strategy or not08:51
mariosarxcruz: o/ ? not following fully though... i am not doing somethign to timeout08:52
mariosarxcruz: the goal is to get some timing for the tests... then decide how to split across two jobs08:52
mariosarxcruz: i haven't checked the results from friday yet (just api) i will be digging more there today08:53
soniya29arxcruz, thanks08:54
arxcruzmarios: yeah sorry, i though i talked with you about the timeout, but it was with weshay|ruck08:54
arxcruzwe were wondering if we could reduce the timeout08:54
arxcruzso tests would fail faster08:55
arxcruz:D08:55
mariosarxcruz: but that is not the problem08:55
mariosarxcruz: i mean, tests pass08:55
mariosarxcruz: it just takes too damn long08:55
mariossometimes they fail/timeout etc08:55
mariosbut...08:55
mariosi think just too many tests08:55
mariosarxcruz: typically fs20 fails across all branches in periodic, and if you re-run with testproject 8/10 you will get a green run08:55
arxcruzhmmmmm08:57
arxcruzmarios: weird the fs020 job didn't trigger on your patch...08:58
ykarelis fs020 consistent timeout issue is across all branches?08:58
mariosarxcruz: well its on purpose apparently08:58
mariosarxcruz: sec08:58
mariosykarel: not consistent08:58
ykareli thought it's not in all releases08:58
mariosykarel: but pretty often. and not always timeout08:58
mariosykarel: but usually a failed tempest08:58
mariosykarel: which if you re-run with testproject 8/10 you will get a green run08:58
ykarelfailed tempest is accepted08:58
*** ratailor has quit IRC08:59
ykarelbut timeout can be a different issue if it's affecting all releases08:59
mariosarxcruz: https://review.rdoproject.org/r/2433808:59
mariosykarel: no we are not talking about something specific today08:59
mariosykarel: more like the pattern.08:59
mariosykarel: in periodic pipelines it is very often fs20 which fails08:59
ykarelmarios, and that pattern is in all releases?08:59
*** ratailor has joined #oooq08:59
ykarelor just some specific releases08:59
mariosykarel: yeah we have seen it across all of them09:00
mariosarxcruz: i posted that https://review.rdoproject.org/r/24338 but after discussion with weshay its on purpose09:00
ykarelok strange then, i might have missed that then09:00
mariosarxcruz: which is why i used that test @ https://review.rdoproject.org/r/24339 with depends-on09:00
ykarelas f2020 rocky/queens have a known issues09:00
mariosykarel: right now they do yes09:00
ykarelmaster last i saw was consistent timeout09:00
mariosykarel: but even before it was the same09:01
ykarelmarios, ack, anywhere u tracking in detail what/where etc, i can look there09:01
ykarelwithout stats it will be difficult to understand09:01
mariosykarel: not really we have that https://tree.taiga.io/project/tripleo-ci-board/task/138309:02
mariosykarel: ack on stats... so i was staring at logs to get some sense of which tempest tests take longest but couldn't fine sane way09:02
mariosykarel: ara tells me it took 2.27 hours to run all of the tests09:02
mariosykarel: but not individual09:02
ykarelmarios, iirc we log individual tests timeout too09:03
ykarelat least before os-tempest09:03
mariosykarel: ack well if you feel like digging there let me know if you find something09:03
mariosykarel: or feel free to just update the taiga09:03
ykarelmarios, sure i can check but later, currently i missing full context09:04
mariosykarel: ack09:05
ykarelmain thing is to find root cause, when/why it started09:05
mariosykarel: i think root cause is just tempest.api.* and tempest.scenario.* & tempest plugins09:05
ykarelmarios, but why it used to run fine earlier09:05
ykarelare there more tests added recently?09:06
mariosykarel: well it was always the longest running job. not sure if it was more stable earlier, at least i always remember it as being a 'likely' candidate when you are checking the periodic pipleines09:06
mariosykarel: i don't know if there have been a bunch more tests added recently09:06
mariosarxcruz: ? ^09:06
arxcruzykarel: marios I believe rocky runs on old validate-tempest, this is also a point...09:07
arxcruzchandankumar: right? ^09:07
chandankumararxcruz, yes09:07
arxcruzmarios: let me check that for you09:07
mariosarxcruz: k cool that is a good point we can use it to compare the runtimes09:07
chandankumararxcruz, marios train and master uses os_tempest only09:07
mariosarxcruz: well... it isn't running the same tests so not really a good comparison though09:07
ykarelyes we can find the failure pattern09:07
chandankumarfor os_tempest jobs we can compare the tempest_run.log file09:07
mariosarxcruz: but worth having in mind09:07
ykarelfrom zuul api atleast for last few runs, to filter out the issue further09:08
chandankumarfrom /var/log/tempest09:08
*** ratailor_ has joined #oooq09:08
chandankumarsshnaidm, arxcruz Happy New Year :-)09:08
sshnaidmchandankumar, thanks, man09:08
arxcruzmarios: from the commit messages there are a few tests changed/added but nothing that might sugest the increase of our timeouts09:09
arxcruzchandankumar: happy new year man, since last decade we don't talk :)09:09
mariosarxcruz: thanks09:09
ykarelmarios, so i see https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens&result=TIMED_OUT&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master09:10
ykareland from this i see Stable release didn't TIMED_OUT from long09:10
*** ratailor has quit IRC09:10
ykarelmost timeouts are in master/than train09:11
mariosykarel: 10:58 < marios> ykarel: but pretty often. and not always timeout09:12
marios10:58 < marios> ykarel: but usually a failed tempest09:12
mariosykarel: not always timeout it also fails tests and they then pass on rerun09:12
ykarelmarios, yes but failed tempest is acceptable, and can be a valid/known issue09:12
marioschandankumar: can confirm we saw it often on our last ruck|rovering09:12
mariosykarel: ok no one is contesting that09:13
mariosykarel: i am not saying remove tests09:13
mariosykarel: we want to split job into two09:13
chandankumarmarios, yes, test failed, but it passed on re-run09:13
ykarelmarios, /me not objecting in solution, my point was mainly finding root cause09:13
ykarelas if we have root cause, solutions can be better09:14
mariosykarel: ack09:14
zbrmorning!09:19
zbrarxcruz: re libpod, i have some bad news09:20
zbrmarios: please give me a +2 on https://review.rdoproject.org/r/#/c/24348/ so i can merge it w/o abuse09:20
zbrfor libpod - also need a +2 on https://review.rdoproject.org/r/#/c/24349/09:21
zbrmainly before being able to consume the podman rpms, we need to sort https://github.com/containers/libpod/issues/466509:25
*** derekh has joined #oooq09:25
sshnaidmzbr, why do you insist to run with "set -ex" in https://review.rdoproject.org/r/#/c/24349/8..9/playbooks/podman/run.yaml09:26
sshnaidmzbr, and why do you need a "pipefail" if no pipe is there09:27
zbrbecause that line is what is called the bash strict mode. I use it everywhere. we have no pipe now, but one may add later, and forget it.09:28
zbrx is optional based on needs09:28
zbrhttp://redsymbol.net/articles/unofficial-bash-strict-mode/09:28
zbrx can prove spammy in some cases but not in this, where it adds only 3 lines.09:29
zbrif you really hate it, i can remove it.09:29
zbrwe also have no variables in that snippet, but this does not mean using the entire strict line is wrong09:29
zbri think is much easier to make a habit of adding this line to any shell block.09:30
zbr(not command, shell)09:30
sshnaidmzbr, you have linter rule to not forget about it09:32
sshnaidmzbr, this strict line is great for pure bash, but not for ansible shell task09:33
sshnaidmzbr, -x in shell task doesn't give any value, "cmd" will be printed by ansible09:34
zbrsshnaidm: i removed, x aspect does not worth more debate time, we have more important things to address.09:36
arxcruzzbr: bad news in the first monday of the year, what else I would ask for?09:36
zbrarxcruz: depends how you look at it, more work! no need to worry that we run out of things to do.09:37
sshnaidmzbr, I mention it because I see it in your other patches too, just curios why we can need it in ansible09:37
arxcruzzbr: always looking to the bright side of life09:38
zbras soon we merge the libpod change I am going to start pinging their channel for help fixing the conman issue.09:38
arxcruzand now you will stay with this music on your head09:38
arxcruzsurpatil: hey man, did you made progress on the mock ?09:38
surpatilarxcruz, not yet09:39
marioszbr: ack but would be nice to have some more context/ i can +2 later if you don't get more votes09:39
arxcruzsurpatil: let me know if you need help09:39
surpatilok arxcruz09:39
zbrmarios: mainly i am trying to zuul testing with few other 3rd party projects we use: sellinux, python-molecule and molecule itself after the first two.09:40
sshnaidmzbr, why did you remove podman version in new patchset?09:41
zbrforgot to sync, adding it now.09:42
zbrgit-review is a bit stupid in that aspect09:42
zbrmarios: re question about selinux/pytest-molecule/molecule projects, that is needed in order to migrate our jobs to centos-809:45
zbrthese projects are currently tested only on travis where only ubuntu can be used09:45
mariosthanks zbr copying that into the review, i can revisit in a while09:46
zbri am now adding a sprint task about it, it could save me time explaining it there.09:47
zbrmy mistake to do it without explaining it09:47
sshnaidmzbr, why do we need it for c8?09:48
*** SurajPatil has joined #oooq09:49
zbrbecause the plan is to migrate all tox-like jobs to centos-8, due to at least two factore: py27 reached EOL and is already dropped from most openstack projects, and 2nd: podman is not usable on centos-7.09:50
zbrin fact for these jobs it would also be ok to use a recent fedora, but clearly not c7.09:50
*** surpatil has quit IRC09:52
sshnaidmzbr, it doesn't explain though why we need this module09:56
zbrsshnaidm: what module?09:57
zbrAdded https://tree.taiga.io/project/tripleo-ci-board/task/143909:58
*** ysandeep has quit IRC09:58
zbrsshnaidm: https://review.rdoproject.org/r/#/c/24349/ passed, please revote.09:59
*** surpatil has joined #oooq10:01
sshnaidmzbr, are you talking about pycontribs/selinux module?10:02
zbrsshnaidm: yes.10:02
zbri will put links, is purely related to molecule use.10:02
sshnaidmzbr, this module ^^ :)10:03
zbrin order to be able to use ansible from inside a virtualenv, (python-)selinux is needed.10:03
zbri will explain it better in the task, with links.10:03
zbrand yeah, i have zero plans to build the real selinux myself.10:04
sshnaidmzbr, doesn't it work in c8 as it works in c7?10:04
zbrsshnaidm: yeah it works but by chance, not that we have CI to validate it.10:04
*** SurajPatil has quit IRC10:04
sshnaidmzbr, for using this module we need to have it in RDO10:05
zbrsshnaidm: the real advantage is to test  molecule project itself but that is bit trickier as I need ansible team to enable the zuul integration, and that would be easier if we show them the first two.10:05
sshnaidmzbr, I think ansible tead does have zuul, we have ansible jobs in openstack zuul10:06
zbrsshnaidm: that is a python module that is needed only for virtualenvs, is is distributed only as wheel and it makes zero sense to be distributed as rpm, as you cannot install a rpm inside a venv10:07
zbrmainly that module replace the selinux cp hack we used to have in quickstart, if you remember.10:07
sshnaidmzbr, https://dashboard.zuul.ansible.com/tenants10:07
zbrsshnaidm: yeah, the plan is to move molecule job to that tenant later, but pabelanger is busy with other stuff. we will transition when he is ready.10:08
sshnaidmzbr, I mean RDO not for RPM, but for gating more. Having uncontrolled modules in our jobs is last thing we need now..10:08
zbrsshnaidm: python-selinux module started to be used in our jobs ~april 2019 if it remember well. we lots of deps we do not control10:11
sshnaidmzbr, we control deps via rdo jobs when pin to specific versions10:11
sshnaidmzbr, we control everything in delorean repos at least..10:12
zbrfor example tox, pytest, .... they are not controlled, block pypi and find out how many stuff we consume from outside. not everything is pinned and that is not necessarly bad. risk is assered on case by case. there is a very big difference between QA tools that we never ship and the deps that end-up in the product.10:13
zbrnobody made a pre-commit rpm, and also nobody died because we didn10:14
zbrsshnaidm: maybe you are not aware that key modules like: pip, tox, setuptools and virtualenv are overridden by zuul with latest versions from PYPI, even if the rpms are installed. That is what openstack-infra decided long time ago. In fact the reality is that I bet that molecule or ansible-lint at least may not even install with the pip version we ship as rpm.10:17
*** chem has joined #oooq10:20
*** SurajPatil has joined #oooq10:26
*** surpatil has quit IRC10:28
*** sanjayu__ has quit IRC10:33
*** saneax has joined #oooq10:33
*** surpatil has joined #oooq10:53
*** SurajPatil has quit IRC10:56
matbumarios: o/ hey man best wishes11:01
matbumarios: maybe i'm missing something but this one is a promotion blocker https://bugs.launchpad.net/tripleo/+bug/1853012 ?11:01
openstackLaunchpad bug 1853012 in tripleo "Standalone upgrade job is failing - On ovn upgrade" [Critical,In progress] - Assigned to mathieu bultel (mat-bultel)11:01
matbumarios: i'm catching up thing11:01
matbuCI seems all green (update & upgrade i mean)11:01
mariosmatbu: o/ looking11:07
mariosmatbu: happy new year ;) \o/11:07
mariosmatbu: not sure... the bug is pretty old but there is a recent comment by weshay there fromfriday11:08
mariosmatbu: and the job is in pretty bad shape https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-upgrade-train11:08
mariosmatbu: but looks like the job isn't in promotion criteria https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/train.ini11:10
*** surpatil is now known as surpatil|afk11:11
matbumarios: hmm ack, i'll take a look11:12
matburight it's totally red11:12
mariosmatbu: i think it should be in promotion criteria anyway11:12
mariosmatbu: so it isnt technicall blocker but should be11:13
*** akahat has quit IRC11:15
*** holser has joined #oooq11:15
*** udesale has quit IRC11:16
*** surpatil|afk has quit IRC11:36
*** surpatil|afk has joined #oooq11:58
* matbu lunch12:00
*** matbu has quit IRC12:05
*** rfolco has joined #oooq12:10
*** akahat has joined #oooq12:16
*** holser has quit IRC12:18
*** beagles is now known as beagles|afk12:27
weshay|ruck0/12:28
weshay|ruckzbr, arxcruz ah.. yikes http://dashboard-ci.tripleo.org/d/jiglzyLWk/third-party-dependency-check?orgId=112:28
weshay|ruckarxcruz, 20-01-06 12:04:4912:29
weshay|ruckceph/ceph-ansible12:29
weshay|ruckmaster12:29
weshay|ruck488312:29
weshay|rucktripleo-ceph-integration-rhel-8-scenario001-standalone12:29
zbrweshay|ruck: yes, is known and result of improving the job. no reason to worry, sometimes red is good.12:29
mariosrfolco: o/ no scrum today? *not in calendart12:29
weshay|ruckthat triggered12:29
weshay|ruckzbr++12:29
rfolcono marios12:29
mariosack thanks rfolco12:29
rfolcowe'll have planning mtg tomorrow12:30
zbri am working on it, mainly to fix https://github.com/containers/libpod/pull/4795 -- may take 1-2 days I think.12:30
*** surpatil|afk is now known as surpatil12:31
*** matbu has joined #oooq12:33
zbrweshay|ruck: if you wonder I fixed podman locally for centos, now I only need to deliver my changes.12:33
zbrsshnaidm: at least on my system running `podman info` as non-root gives me a ERRO[0000] open /proc/17575/ns/user: no such file or directory -- but as root works.12:34
zbrnot sure if this is expected, local config issue, or a new problem.12:35
*** ratailor_ has quit IRC12:36
arxcruzweshay|ruck: so, the issue with the check-rdo was because zbr changed the pipeline, and the command now is check-github so it's triggering properly12:39
arxcruzweshay|ruck: still failing though12:39
zbrarxcruz: so the change of magic comment was a side-effect from using a different pipeline? i did not know that we had different magic comments for different pipelines. I wonder if that is really needed for some magic use-case.12:40
weshay|ruckarxcruz, ah k..12:40
weshay|ruckarxcruz, can you upodate the cockpit for us?12:40
arxcruzzbr: your pipeline you add comment to check-github on github-manual, on github-check pipeline it was check-rdo12:41
arxcruzbut that's fine12:41
sshnaidmzbr, what is your system?12:41
weshay|ruckalso I noticed the link to the patch is going to openstack gerrit12:41
arxcruzweshay|ruck: update what exactly? i'm preparing the environment to do some changes12:41
weshay|ruckwhich it should be going to project/pull_request_id12:41
arxcruzweshay|ruck: yeah, i'll check that12:41
weshay|ruckarxcruz, if the pipeline name has changed.. then the cockpit needs to be updated to search the new pipeline name12:42
arxcruzweshay|ruck: it's already as you can see http://dashboard-ci.tripleo.org/d/jiglzyLWk/third-party-dependency-check?orgId=1&fullscreen&panelId=33612:42
weshay|ruckmaybe I'm not in full context yet12:42
weshay|ruckaye.. cool cool12:42
weshay|ruckthanks12:42
arxcruzweshay|ruck: so, remember last week, we try to trigger a manual job and it wasn't working?12:42
weshay|ruckyes12:43
arxcruzbecause i was using check-rdo comment12:43
arxcruznow it's check-github12:43
weshay|ruck:)12:43
arxcruzwith check-github it was triggered properly12:43
arxcruztook me a while to figure out though :)12:43
chandankumararxcruz, I have changed that12:46
arxcruzchandankumar: change what ?12:47
chandankumararxcruz, https://review.rdoproject.org/r/#/c/24261/12:48
chandankumarcheck-github12:48
arxcruzoh, i've been blaming the wrong person then12:48
arxcruzsorry zbr12:48
arxcruz:P12:48
* arxcruz removing all zbr curses12:49
chandankumarweshay|ruck, \o/, I am back now on work.12:50
weshay|ruckchandankumar, hooo ahhhh!12:50
chandankumarIRC is not yet persistent, I need to set it up properly.12:50
*** holser has joined #oooq12:54
*** rlandy has joined #oooq12:58
*** ratailor has joined #oooq12:59
*** SurajPatil has joined #oooq13:01
mariospanda: around?13:03
*** surpatil has quit IRC13:04
pandamarios: yes13:04
mariospanda: happy new year panda \o/13:05
mariospanda: wondering if we should have a sync about the promoter status/planning for this week? me you rlandy sshnaidm weshay|ruck anyone else that is interested?13:05
mariosrfolco: ^T13:05
chandankumarrfolco, weshay|ruck if free, Can we do the pre-planning meeting early?13:05
marioschandankumar: ^13:05
weshay|ruckmarios, hold for planning please13:05
mariosah sounds like they're busy13:06
mariosweshay|ruck: ack13:06
mariosweshay|ruck: what time planning?13:06
mariosweshay|ruck: oh you mean tomorrow13:06
weshay|ruckmarios, ask the tc if you don't see it13:06
mariosweshay|ruck: yeah but just sync call not actually do anything?13:06
weshay|ruckrfolco, ^^13:06
weshay|ruckcome on man13:06
mariosweshay|ruck: ok we can hold till tomorrow13:06
mariosweshay|ruck: rfolco: i see the planning call tomorrow13:06
weshay|ruckchandankumar, sec.. checking13:07
mariosweshay|ruck: i thought you meant 'hold cos i am in another meeting now13:07
rfolcochandankumar, I am ok to push pre-planning mtg earlier, how early you need?13:07
*** akahat has quit IRC13:07
mariospanda: well tomorrow it is then13:07
weshay|ruckchandankumar, moved it to 2pm utc13:07
mariospanda: i look forward to it !13:07
marios;)13:07
chandankumarrfolco, pushed already, will work, thanks :-)13:08
rfolcook13:08
weshay|ruckrfolco, planning will take more than an hour I think as well13:10
rfolcoweshay|ruck, done! you have one more wish13:11
sshnaidmweshay|ruck, we have 1:1 in 2pm utc tomorrow iirc13:13
weshay|rucksshnaidm, 12pm13:15
weshay|rucksshnaidm, sync up w/ rlandy today about some changes to the cockpit.. sshnaidm please make sure you have the internal cockpit lots of resources :)13:16
sshnaidmweshay|ruck, oh, right13:16
sshnaidmweshay|ruck, I did :)13:16
weshay|ruckniiiice13:17
rlandyweshay|ruck: sshnaidm: k - can we then removed the commented out ansible pull?13:18
rlandyand get the new data showing13:18
rlandydownstream cockpit only13:18
arxcruz45199113:19
arxcruzdamn yubikey13:20
arxcruzweshay|ruck: https://review.opendev.org/#/c/701173/13:23
weshay|ruckarxcruz, ah thanks.. I spaced that13:24
sshnaidmrlandy, where?13:24
weshay|ruckchandankumar, zbr please review .. need a way to track the bugs on 3rd party dep ci failures https://review.opendev.org/#/c/701173/13:24
sshnaidmrlandy, downstream cockpit can't run ansible pull right now, it requires some changes13:25
sshnaidmrlandy, will submit them this week13:25
zbrweshay|ruck: any reason for not using just "dependency" instead of "depcheck" ?13:25
sshnaidmrlandy, and we need to decide how we combine both upstream and downstream code in one13:25
*** ratailor has quit IRC13:26
weshay|rucksshnaidm, aye13:27
weshay|ruckthink about that and we'll chat soon13:27
*** surpatil has joined #oooq13:29
*** SurajPatil has quit IRC13:32
*** ratailor has joined #oooq13:32
weshay|ruckmatbu, pretty sure that standalone upgrade failure is networking related.. and I think I saw some work around that but I need to refresh myself on the issue13:33
*** holser has quit IRC13:35
rlandysshnaidm: we merged https://review.rdoproject.org/r/#/c/24341/13:39
rlandysmall changes to collect downstream data - only when running within RH13:39
sshnaidmrlandy, why?13:40
rlandysshnaidm: why did we merge? why only within RH?13:40
*** soniya29 has quit IRC13:41
sshnaidmrlandy, it's going to break existing telegrad13:41
sshnaidm*telegraf13:41
rlandysshnaidm: what part is broken?13:41
sshnaidmrlandy, it won't resolve the host13:42
sshnaidmrlandy, also I don't think it should be in this file at all13:42
rlandysshnaidm: we check for that13:42
sshnaidmrlandy, it's not how I imagine combining repos13:42
sshnaidmrlandy, weshay|ruck we should have discuss our design how to do it before merging solutions13:43
chandankumararxcruz, few nit picks on this review https://review.opendev.org/#/c/701173/13:43
rlandysshnaidm: it's not broken now afaict13:43
*** beagles|afk is now known as beagles13:43
sshnaidmrlandy, I see a check, but it just prints error message, doesn't stop the script13:43
sshnaidmrlandy, anyway, mixing downstream and upstream things in one file is a bad idea13:44
rlandysshnaidm: weshay|ruck: k - let's meet to discuss this13:44
sshnaidmrlandy, I think we need to revert it, did you check telegraf logs on cockpit?13:44
matbuweshay|ruck: i will reproduce it. My first attempt failed :/13:45
*** holser has joined #oooq13:46
rlandysec- finding meeting time13:46
*** ykarel has quit IRC13:47
rlandyweshay|ruck: calendar says you are busy until 5pm utc13:47
rlandyweshay|ruck: sshnaidm: invite sent - if you can make it earlier - we can move that up13:49
chandankumaris someone looking at fs020 timing out?13:49
sshnaidmrlandy, that's fine13:52
sshnaidmrlandy, and I can make it earlier too :)13:52
rlandysshnaidm: I'm happy to revert it - but many people merge things w/o full design :)13:54
rlandyI have concerns about separating files13:54
rlandyand downstream only gets ignored13:54
rlandyby everyone except me13:54
sshnaidmrlandy, I hope to keep upstream and downstream things separated13:54
rlandyand lands up being my problem only13:54
rlandynever the less13:55
rlandylet's discuss at the meeting13:55
sshnaidmrlandy, sure13:55
*** bhagyashris has quit IRC13:57
weshay|ruckrfolco, chandankumar having trouble joining the meet13:58
weshay|ruckcan anyone confirm?13:58
* weshay|ruck reboots13:59
chandankumarweshay|ruck, joined13:59
rfolcoweshay|ruck, joined too13:59
*** tosky has joined #oooq14:01
*** Goneri has quit IRC14:03
sshnaidmrlandy, http://paste.openstack.org/show/788080/14:04
rlandyyep - we print out that error14:04
rlandywe cannot and just go by it14:05
rlandythat is what the cockpit did before14:05
rlandyfail w/o any trace14:05
sshnaidmrlandy, it's not printed error, it's error of telegraf14:05
sshnaidmrlandy, E! Error in plugin [inputs.exec]:14:05
*** holser has quit IRC14:05
*** ykarel has joined #oooq14:08
*** jtomasek has quit IRC14:13
*** jtomasek has joined #oooq14:15
sshnaidmrlandy, currently it blocks all builds metrics collection14:15
rlandysshnaidm: I +2'ed the revert14:15
rlandyaand W+14:16
rlandyall gone14:16
*** ratailor has quit IRC14:21
zbrif would be really nice if someone would add some unittests to our python scripts from rrcockpit role14:25
rlandychandankumar: hi ... the standalone rhos-16 test is failing with 'Invalid regex: * in provided whitelist file'. I'm not overriding the whitelist14:30
rlandyshould I be?14:30
chandankumarrlandy, log link?14:30
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/periodic/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/tripleo-ci-rhel-8-standalone-rhos-16/c419aea/job-output.txt14:30
rlandyscenario004, where I do override the list is ok14:30
rlandychandankumar: here is the job defn: http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d/standalone-jobs.yaml#n11314:32
*** surpatil has quit IRC14:34
chandankumarweshay|ruck, rfolco https://review.opendev.org/#/c/692368/14:43
arxcruzchandankumar: can you -W https://review.opendev.org/#/c/701158/ ?14:45
arxcruzweshay|ruck: already +2 and we agree to not run barbican on fs020 ;)14:46
*** ykarel is now known as ykarel|away14:46
chandankumarrlandy, give me few mins14:47
*** TrevorV has joined #oooq14:47
chandankumarrlandy, https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset052.yml#L28 not plyaing nicely14:48
chandankumarmake be jyst use smoke there14:48
rlandylooking14:50
chandankumarrlandy, https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/tripleo-ci-rhel-8-standalone-rhos-16/c419aea/logs/undercloud/home/zuul/tempest/etc/tempest_whitelist.txt.gz14:51
rlandychandankumar: weird - downstream uses fs05214:52
rlandyit should pick that var up14:52
rlandychandankumar: I can define it in the job but afaict, that should not be needed14:53
rlandy --extra-vars @/home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/general_config/featureset052.yml14:59
rlandy"test_white_regex": "\\[.*\\bsmoke\\b.*\\]",15:00
chandankumarrlandy, tempest_whitelist: 'smoke'15:03
chandankumarin the job definition itself it will work15:04
zbrchandankumar: a hotfix https://review.rdoproject.org/r/#/c/24378/ please.15:05
chandankumarzbr, done15:06
zbrtx15:06
zbrin ~1h need to go, GP appt.15:09
*** Goneri has joined #oooq15:10
rlandyok15:11
chandankumarrlandy, does it works?15:11
rlandychandankumar: will find out when the test gets there15:12
zbrchandankumar: another one https://review.rdoproject.org/r/#/c/24380/15:18
zbrfollowed by https://review.rdoproject.org/r/#/c/24381/ -- but that one needs double check.15:19
weshay|ruckkopecmartin, are you available for the ci planning meeting tomorrow15:20
zbr...and i hope the boss does not ask me to copy/paste link to yaml anchors docs, but  just in case: http://blogs.perl.org/users/tinita/2019/05/reusing-data-with-yaml-anchors-aliases-and-merge-keys.html15:21
kopecmartinweshay|ruck: yes15:25
weshay|ruckkopecmartin, k.. we'll be trying to figure out who best to join you on collect-logs15:25
weshay|ruckkopecmartin, you can probably drop after that discussion15:25
kopecmartinweshay|ruck: ok, sounds good15:26
zbrchandankumar: in fact I think that there is no need to use different magic comments between pipelines because a job cannot exist in both.15:27
*** chandankumar has quit IRC15:27
zbrif I am correct we can avoid that, which is very useful for UX as I find hard to remember one magic comment, but having different ones would be really confusing, especially as there is no way to know which one may apply by lookin at a change.15:28
mariosarxcruz: looks like its gonna timeout again at fs20 master https://review.rdoproject.org/r/#/c/24339/ and this is JUST api tests so must be some legit issue there15:42
marioshttps://review.rdoproject.org/zuul/stream/aebe746b0b8f4437862a5e6bac3658fe?logfile=console.log15:42
arxcruzmarios: any chance you have an environment up so we can play around?15:48
arxcruzthis is odd that it takes so long with api tests15:48
mariosarxcruz: i do not15:49
mariosarxcruz: but we can request a hold on recheck of the https://review.rdoproject.org/r/#/c/24339/15:49
mariosarxcruz: maybe tomorrow ^15:49
mariosweshay|ruck: fyi fs20 maybe some legit master issue15:50
weshay|ruckk.. I'll check it out15:50
zbrarxcruz: https://review.rdoproject.org/r/#/c/24376/15:51
arxcruzmarios: yes, please, let's get a hold15:51
mariosweshay|ruck: basically while restricting to just api tests to get a feel for timing, at https://review.rdoproject.org/r/#/c/24339/ & its depends-on, i am seeing timeouts there15:51
arxcruzzbr: I have some doubts why you're doing that, but we can discuss tomorrow, i'll not block you because of that ;)15:53
zbrarxcruz: mainly because same test will be done by "make package-install" from libpod itself.15:54
zbrso they can also fix it when it gets broken.15:55
arxcruzzbr: ok15:56
arxcruzthen i think i sould add it on ceph-ansible as well15:57
arxcruz:D15:57
zbralso there is somethign very strange which I seen when we called ` podman info  --log-level debug` -- is like someone did remove one of the dashes.15:57
arxcruzbecause i'm just getting the buildset url and using to download the .repo file15:57
zbrarxcruz: first you need to check with them if they have anything against, usualy by adding a PR and seeing how open they are about it.15:57
arxcruzalthough, the scenario001 will only start if the rpm build pass15:57
arxcruzzbr: weshay|ruck shall we use scenario000 for libpod ?15:58
arxcruzit will reduce the job time to 30 minutes +-15:58
zbrarxcruz: no idea, but cannot beat your argument.15:59
weshay|ruckarxcruz, that's fine w/ me, we'll just look for input from the interested parties after we have it up and adjust later15:59
arxcruzscenario000 have only keystone service, and it will be executed with podman15:59
arxcruzso that's what we want right?16:00
arxcruzif so, it would be better for us16:00
arxcruzunless we require more16:00
zbri may need some help to figure out why yum decides to install podman from system instead my override repo, even if I put priorty=1 on it, http://logs.rdoproject.org/96/4796/1c23e371bd908b33a02a843be90dd2af24a83379/github-check/podman-package-centos-8/8e538ba/job-output.txt16:00
arxcruzzbr: on ceph-ansible was the version16:00
arxcruzthe rpm i was building have a lower version than the one used in rdo repo16:00
arxcruzbut it was fixed now16:01
* marios shutdown sequence16:02
zbrsomehow my code from https://github.com/containers/libpod/pull/4796/files fails to enforce using the custom podman repo16:02
*** chandankumar has joined #oooq16:02
zbrand I cannot disable system repo as it contains other dependencies.16:02
weshay|ruckmarios, oh look at that16:03
weshay|ruckarxcruz, with regards to tempest timing out...16:03
weshay|ruckarxcruz, do you think it's possible to have a task in collect logs that tries to construct the tempest results if they are not present like in the case of a timeout?16:04
mariosweshay|ruck: ? 18:03 < weshay|ruck> marios, oh look at that16:04
marioswhich bit should i look at16:04
zbrmaybe someone finds out what is wrong with:  sudo ${PKG_MANAGER} -y install --best --allowerasing --nogpgcheck podman podman-remote16:04
weshay|ruckmarios, timeout on just api16:04
mariosweshay|ruck: ah16:04
mariosweshay|ruck: yeah16:04
mariosweshay|ruck: but i think must be a new issue16:05
arxcruzweshay|ruck: not sure if the streaming is created with the timeout, if so i think it's possible16:05
mariosweshay|ruck: i will revisit tomorrow anyway and we can get a hold with arxcruz16:06
arxcruzi can also create a script and collect the logs from what we have, before the timeout16:06
arxcruzmarios: ack16:06
weshay|ruckk.. arxcruz I'll pick your brain on that a bit over the next week or so16:06
arxcruzweshay|ruck: ok16:06
*** bhagyashris has joined #oooq16:07
*** arxcruz is now known as arxcruz|off16:11
*** marios is now known as marios|out16:13
*** marios|out has quit IRC16:15
*** soniya29 has joined #oooq16:37
*** dtrainor has joined #oooq16:46
*** tosky has quit IRC16:46
chandankumarrlandy, does tempest fix worked downstream?16:59
sshnaidmrlandy, where do we meet?17:00
rlandysshnaidm: weshay|ruck: https://meet.google.com/cum-jyme-bra?pli=1&authuser=017:00
rlandychandankumar: idk yet17:01
rlandyhad to fix some config errors dosntream17:01
rlandyjob just started running17:01
*** chkumar246 has joined #oooq17:01
*** chandankumar has quit IRC17:04
*** chkumar246 has quit IRC17:07
*** saneax has quit IRC17:10
*** ykarel|away has quit IRC17:11
*** bhagyashris has quit IRC17:17
*** sshnaidm is now known as sshnaidm|afk17:31
*** soniya29 has quit IRC17:46
rfolcocan we please merge this https://review.rdoproject.org/r/#/c/24325/17:56
rfolcozbr, weshay|ruck rlandy panda ^ anyone around to approve this one ?17:56
weshay|ruckrfolco,  cool :) http://logs.rdoproject.org/25/24325/7/check/rdo-tox-molecule/f2740d0/tox/reports.html17:59
rfolco:)18:00
*** derekh has quit IRC18:00
weshay|ruckanyone have time to look at https://review.rdoproject.org/r/#/c/24103/ w/ me?18:00
rlandyrfolco: happy that the final component promote is in18:04
rfolcorlandy, hopefully it works now18:05
rfolcothe best I could do was adding molecule18:05
*** tesseract has quit IRC18:38
rfolcorlandy, weshay|ruck finally job is green on https://review.rdoproject.org/r/#/c/24199/18:41
zbrrfolco: nice work with promote-hash!18:48
rfolcozbr, thank you for helping with the molecule bit18:55
rfolcozbr++18:55
rlandyweshay|ruck: I'm creating the new repo in codeeng19:10
rlandyname: openstack/rrcockpit.git?19:10
rlandywhat ldap group should I used?19:10
weshay|ruckopenstack ldap19:10
weshay|rucknaming is the hardest part of engineering19:11
weshay|ruckrlandy, I don't have a better name atm19:11
rlandyweshay|ruck: k - initial description: This repo contains the configurations for the internal ruck/rover cockpit.19:13
weshay|ruckrfolco, so what pipeline are the promote jobs in19:14
weshay|ruck?19:14
weshay|ruckrlandy, ack19:14
weshay|ruckah see it now19:14
rlandyweshay|ruck: last question ... do I get the system to create the initial commit?19:14
weshay|rucksorry?19:14
weshay|ruckdon't understand19:14
rlandythere is a question where to create the initial commit19:15
rlandyhttps://redhat.service-now.com/help?id=sc_cat_item&sys_id=8a97fe6113356600196f7e276144b01d&pnt_redirection19:16
weshay|ruckopenstack-promote-component rfolco got it.. thanks19:16
weshay|ruckrlandy, doesn't matter19:16
rlandyok - here we go19:17
rlandyCreated PNT0749394: Create New Gerrit Project - openstack/rrcockpit.git19:17
weshay|ruckcool19:17
rlandysshnaidm|afk: ^^ fyi19:17
sshnaidm|afkrlandy, ack19:18
rlandygoing to dr's appt - bbl19:43
*** rlandy is now known as rlandy|afk19:43
zbrwhy new cockpit project?20:23
zbrahh, i see. still not convinced that we really need a separated project. what prevents us from having a single cockpit for both?20:25
zbrmaybe someone can ping me to a way to debug how yum is working with repos, i used createrepo to create one but once installed in yum, it always reports 0 packages in. even if the /repodata folder appears to contain valid information (more than zero packages)20:28
*** rfolco has quit IRC20:34
*** tosky has joined #oooq20:55
*** zbr has quit IRC21:05
*** zbr has joined #oooq21:07
*** holser has joined #oooq21:09
weshay|ruckzbr, context and firewalls21:11
weshay|ruckthere is a larger scope to this in the long term21:11
*** zbr has quit IRC21:14
*** zbr has joined #oooq21:22
*** zbr has quit IRC21:26
*** zbr has joined #oooq21:30
*** jfrancoa has joined #oooq21:36
*** zbr_ has joined #oooq21:37
*** zbr has quit IRC21:39
*** jfrancoa has quit IRC22:01
*** zbr_ has quit IRC22:02
*** zbr has joined #oooq22:02
*** zbr has quit IRC22:25
*** zbr has joined #oooq22:25
*** holser has quit IRC22:29
*** zbr has quit IRC22:30
weshay|ruckrlandy|afk, fyi when ur back https://review.opendev.org/70129122:30
*** holser has joined #oooq22:30
*** Goneri has quit IRC22:46
*** TrevorV has quit IRC22:53
*** zbr has joined #oooq23:04
*** zbr_ has joined #oooq23:08
*** zbr has quit IRC23:09
*** Goneri has joined #oooq23:10
*** rlandy|afk is now known as rlandy23:11
*** zbr_ has quit IRC23:14
*** holser has quit IRC23:20
*** zbr has joined #oooq23:21
*** tosky has quit IRC23:23
*** holser has joined #oooq23:38
rlandyinternal zuul is back in business23:38
weshay|ruckrlandy, nice23:38
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds23:39
rlandybm is back to passing23:39
weshay|rucknice23:42
weshay|ruckrlandy, can we trigger a f020 you think?23:42
weshay|ruckon bm23:43
weshay|ruckso we have the box?23:43
weshay|ruckf020 master23:43
weshay|ruckI guess it needs two computes.. so maybe not23:43
* weshay|ruck has been trying to reproduce that for a bit today23:43
rlandyweshay|ruck: sure23:44
rlandyfor master?23:44
weshay|ruckya.. the others are fine23:44
weshay|ruckmaster only23:44
rlandyon baremetal or ovb?23:44
weshay|ruckI was thinking bm.. so we have the box for sure.. but I guess I can try the old script as well23:45
* weshay|ruck hasn't done that yet23:45
rlandyweshay|ruck: ok - I'l try it on one of the bm envs we are not using23:45
weshay|ruckrlandy, k.. great.. thank you. send me the review so I can see how you did it..23:46
rlandyI expect it will timeout23:47
*** holser has quit IRC23:47
rlandybm deploys take a long time23:47
rlandybut we can see23:47
weshay|ruckwe basically need no timeout on this23:48
weshay|ruckbecause something is spinning tempest into a tizzy23:48
weshay|rucksome api test23:48
weshay|ruckfor example https://review.rdoproject.org/r/#/c/24339/23:49
rlandyweshay|ruck: the tenant itself has a max timeout23:55
weshay|ruckok.. I guess we can always clean up and rerun tempest manually23:55
weshay|ruckso that's fine23:55
weshay|ruckas long as the env does not get redeployed23:56
weshay|ruckwe're good23:56
rlandyk- let's see how far the test gets23:56
weshay|ruckwe can't get any logs to diagnose23:56
weshay|ruckso bm is a good fit23:56
rlandyright23:57
rlandythe machine stays up23:57
weshay|ruckaye23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!