Tuesday, 2020-12-08

*** TrevorV has quit IRC00:10
*** Goneri has quit IRC00:28
*** rlandy has quit IRC00:42
*** tosky has quit IRC00:54
*** rfolco has joined #oooq01:10
*** rfolco has quit IRC01:28
*** ysandeep|away is now known as ysandeep02:54
*** saneax has joined #oooq02:57
*** saneax has quit IRC03:17
*** apetrich has quit IRC03:20
*** fuzzball81 has joined #oooq03:53
*** udesale has joined #oooq04:59
*** ratailor has joined #oooq05:03
chkumar|ruckhttps://bugs.launchpad.net/tripleo/+bug/190719305:36
openstackLaunchpad bug 1907193 in tripleo "Problem: package tripleo-ansible-1.5.2-0.20201124210415.633348b.el8.noarch requires python3-paunch, but none of the providers can be installed" [High,Triaged]05:36
*** marios has joined #oooq06:16
chkumar|ruckmarios: Good morning06:30
chkumar|ruckmarios: https://bugs.launchpad.net/tripleo/+bug/1907193 gate is busted in rdo as well as upstream06:30
openstackLaunchpad bug 1907193 in tripleo "package podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 is filtered out by modular filtering" [High,Triaged] - Assigned to Alex Schultz (alex-schultz)06:30
chkumar|ruckfix is in place will take some time to land on rdo side06:30
marioschkumar|ruck: morning06:30
marioschkumar|ruck: thanks06:31
*** marios is now known as marios|rover06:31
*** saneax has joined #oooq06:31
chkumar|ruckmarios|rover: https://review.opendev.org/q/topic:%22release-aware-gate%22+(status:open%20OR%20status:merged) fyi06:39
marios|roverchkumar|ruck: nice thanks will chek them in a sec06:40
chkumar|ruckweshay|ruck: rlandy and sshnaidm worked on that, once gates are back, I will be updating the patches06:41
marios|roverchkumar|ruck: added some comments for when you update07:19
marios|rovergerrit really slow right now or just me08:13
marios|roverchkumar|ruck: well.. so much for avoiding the long queue this week08:14
marios|roverchkumar|ruck: the gate issues are going to back us up nicely08:14
chkumar|ruckmarios|rover: yes08:14
marios|roverchkumar|ruck: gerrit ok for you?08:14
chkumar|ruckmarios|rover: same here08:15
marios|roverchkumar|ruck: thanks08:15
chkumar|rucktaking time to load08:15
chkumar|ruckgerrit is taking holiday i think08:16
marios|roverchkumar|ruck: yeah it is in opendev chan others are saying same08:16
*** apetrich has joined #oooq08:25
zbryeah, and i did not see any wipe message on it08:27
zbroops, is back,08:28
marios|roverk gerrit back now08:28
marios|roverzbr: ianw restarted08:28
zbrsurvey: did any of new needed to re-login on gerrit after the last restart? (~30min ago)09:02
*** rfolco has joined #oooq09:03
chkumar|ruckzbr: nope09:03
sshnaidmchkumar|ruck, wrt https://review.opendev.org/c/openstack/tripleo-ci/+/765834 - left comments09:06
*** rfolco has quit IRC09:07
sshnaidmchkumar|ruck, I'll handle it09:22
chkumar|rucksshnaidm: ok09:23
*** derekh has joined #oooq09:35
chkumar|rucksshnaidm: small nit picks09:43
sshnaidmchkumar|ruck, do you want just to add loop_var?09:47
chkumar|rucksshnaidm: ok, will do that09:47
sshnaidmchkumar|ruck, I'll add it, np09:47
chkumar|rucksshnaidm: thanks!09:47
*** rfolco has joined #oooq09:51
*** gchamoul has quit IRC10:29
*** gchamoul has joined #oooq10:31
*** udesale_ has joined #oooq10:39
*** udesale has quit IRC10:40
*** dtantsur|afk is now known as dtantsur11:00
* zbr f*** my NAS died again, i bet is again the PSU, apparently they do not live more than year.11:08
marios|roverchkumar|ruck: did you file a bug for the pacemaker deps issue?12:02
marios|roverchkumar|ruck: am filing it now just asking in case you did12:02
chkumar|ruckmarios|rover: nope12:03
chkumar|ruckmarios|rover: I am still working on pauch one12:03
marios|roverchkumar|ruck: ack thanks12:03
marios|roverchkumar|ruck: https://bugs.launchpad.net/tripleo/+bug/1907245 fyi12:09
openstackLaunchpad bug 1907245 in tripleo "periodic victoria overcloud image and container builds fail for missing deps pacemaker-libs" [Critical,Triaged]12:09
chkumar|ruck marios|rover ack12:27
chkumar|ruckmarios|rover: stepping out will back in an hour12:27
marios|roverchkumar|ruck: k12:28
*** ratailor has quit IRC12:29
*** frenzyfriday has joined #oooq12:37
*** frenzyfriday has quit IRC12:38
*** dtantsur has quit IRC12:50
weshay|ruckchkumar|ruck, marios|rover 0/12:51
marios|rovero/12:51
weshay|ruckanything to sync on?  I left some notes in the hackmd12:52
weshay|ruckmarios|rover, chkumar|ruck ussuri is really f'd12:52
marios|roverweshay|ruck: what are you referring to in particular12:53
marios|roverweshay|ruck: do you mean the periodic pipeline?12:53
weshay|ruckupstream man12:53
marios|roverweshay|ruck: yeah it is cos of https://bugs.launchpad.net/tripleo/+bug/190719312:54
openstackLaunchpad bug 1907193 in tripleo "package podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 is filtered out by modular filtering" [High,Triaged] - Assigned to Alex Schultz (alex-schultz)12:54
marios|roverweshay|ruck: content provider is hitting it then everything else skips12:55
marios|roverweshay|ruck: also hitting the ussuri periodic https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable212:56
weshay|ruckchkumar|ruck, marios|rover  k.. thanks.. stay focused on upstream this week.. don't worry about promotions12:56
weshay|ruckchkumar|ruck, marios|rover master is also failing in the gate too much.. at 86% needs to be well over 9012:58
chkumar|ruckweshay|ruck: one bug blocking all stuff12:58
chkumar|ruckweshay|ruck: above one which marios passed12:59
weshay|ruckk.. ping me if needed.. I'm locked in mtgs for the next few hours12:59
*** TrevorV has joined #oooq13:00
*** rlandy has joined #oooq13:03
*** dtantsur has joined #oooq13:11
weshay|ruckrfolco, bhagyashris I have a conflict for the community13:20
*** owalsh has quit IRC13:20
rfolcoweshay|ruck, ack13:20
*** rfolco has quit IRC13:22
weshay|ruckhttps://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah+&job_name=openstack-tox-lower-constraints+&branch=stable%2Ftrain13:22
*** rfolco has joined #oooq13:22
weshay|ruckzbr, ^ and updates on lower-constraints?13:23
weshay|ruckchkumar|ruck, marios|rover zbr https://bugs.launchpad.net/tripleo/+bug/190723613:26
openstackLaunchpad bug 1907236 in tripleo "lower-constraints jobs across tripleo projects broken" [Critical,In progress]13:26
weshay|ruckERROR: No matching distribution found for hacking<1.2.0,>=1.1.013:26
weshay|ruckhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_45c/765970/1/check/openstack-tox-lower-constraints/45c4dd9/job-output.txt13:26
weshay|ruckhttp://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&var-influxdb_filter=job_name%7C%3D%7Copenstack-tox-lower-constraints13:29
weshay|ruckwho's got that one?13:30
rfolcocommunity call https://meet.google.com/acd-fxac-rgf?authuser=113:30
rfolcoarxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar, frenzy_friday13:31
zbrweshay|ruck: if you want, i can take care of it, reqs are usually easy to fix.13:32
weshay|ruckzbr, ya.. this is infra.. you or rhos-ops https://bugs.launchpad.net/tripleo/+bug/190723613:33
openstackLaunchpad bug 1907236 in tripleo "lower-constraints jobs across tripleo projects broken" [Critical,In progress]13:33
weshay|ruckthanks13:33
zbrweshay|ruck: tbh, almost sure is not infra.13:33
zbrunless, the mirror is missing files13:34
weshay|ruckzbr, not caused by infra.. but in their realm.. imho13:34
zbri bet there is a pinning down somewhere an == or a <= which is causing it.13:34
zbrlooking now into it, failed to install locally too.13:36
weshay|ruckthanks13:38
zbrweshay|ruck: clone tripleo-common locally, master and run tox -e lower-constraints13:38
zbrit will fail ugly and has nothing to do with infra, 100% inside the repo.13:38
zbrnot sure who keeps track of this pin file, but I do not want to be me. 109 pinned deps will guarantee failures.13:40
zbrhttps://review.opendev.org/dashboard/self13:43
*** tosky has joined #oooq13:44
weshay|ruckzbr, ya.. that's fine just looking for your help in these pip dep issues, you are pretty tuned in to that field of work13:55
ysandeepzbr++ thanks for sharing!13:58
zbrkeep in mind to change the default WIP value, helps keeping the "incoming reviews" list clean.13:59
zbrand look at old reviews owned by you those with WIP/DNM/POC and make them WIP.13:59
zbrbest part about WIP is that flapping it does not trigger jobs.14:00
zbrweshay|ruck: you want to chat regarding lower-constraints?14:01
zbrmainly we need to use tools like pip-tools to produce valid lower-constraints files.14:04
zbri am already doing something similar on some projects of mine but I am not sure it will work here.14:05
*** Goneri has joined #oooq14:39
weshay|ruckzbr, ah.. sorry missed ur ping.. sure14:40
weshay|ruckzbr, https://meet.google.com/quj-nyij-kgm?authuser=114:41
chkumar|rucksshnaidm: weshay|ruck https://review.opendev.org/c/openstack/tripleo-quickstart/+/766009 not sure it will unblock ussuri14:47
weshay|ruckmarios|rover, chkumar|ruck please join https://meet.google.com/quj-nyij-kgm?authuser=1 re: lower constraints14:49
weshay|rucksorry14:49
sshnaidmchkumar|ruck, if you install it, it usually installs the latest version, isn't it?14:49
chkumar|rucksshnaidm: yes that is what needed, https://review.rdoproject.org/r/#/c/31309/7/python-paunch.spec14:50
marios|roverweshay|ruck: joining14:55
*** marios|rover is now known as marios|call14:57
rlandyysandeep: downstream zuul still out?15:00
weshay|ruckrlandy, looks like we have passing jobs there now15:05
*** marios|call is now known as marios|rover15:06
chkumar|rucksshnaidm: weshay|ruck marios|call rlandy https://review.opendev.org/q/topic:%22podman_paunch%22+(status:open%20OR%20status:merged) and https://review.rdoproject.org/r/31316 if passes, merge those15:07
marios|roverchkumar|ruck: ack15:07
rlandygot a node failure15:08
ysandeeprlandy, weshay|ruck ovb jobs are still failing15:14
rlandyyeaj - asked on rhos-ops15:14
weshay|ruckya.. ovb is down15:14
ysandeeplast i checked.. upgrades were still pending on some compute nodes15:14
ysandeeplets wait till upgrade completes15:14
chkumar|ruckweshay|ruck: rlandy sshnaidm marios|rover https://blog.centos.org/2020/12/future-is-centos-stream/15:14
marios|rovertom barron was just talking about that in tripleo15:15
marios|rovers/talking/asking15:15
marios|roverchkumar|ruck: 16:59 < tbarron> https://lists.centos.org/pipermail/centos-devel/2020-December/075451.html15:15
marios|roverchkumar|ruck: 16:52 < tbarron> marios|rover: are there plans to migrate tripleo to CentOS Stream?15:15
chkumar|ruckah, cool tanks!15:15
chkumar|ruckthanks!15:15
*** ysandeep is now known as ysandeep|away15:16
sshnaidmcentos bugs "stream", great15:17
sshnaidmIf you are using CentOS Linux 8 in a production environment, and are concerned that CentOS Stream will not meet your needs, we encourage you to contact Red Hat about options.15:18
zbrchkumar|ruck: weshay|ruck: look at "Unable to freeze job graph:" from https://review.opendev.org/c/openstack/tripleo-common/+/76601815:18
sshnaidmlet's contact15:18
sshnaidmzbr, weshay|ruck handled in https://review.opendev.org/c/openstack/tripleo-ci/+/76599615:18
weshay|ruckmarios|rover, chkumar|ruck any hand off items as your day starts to close?15:37
marios|roverweshay|ruck: will do before i go15:37
weshay|ruckk.. thanks15:37
marios|roverchkumar|ruck: new one for ovb 3rd party sshnaidm pointed me to it earlier filed it just now https://bugs.launchpad.net/tripleo/+bug/190727215:42
openstackLaunchpad bug 1907272 in tripleo "rdo ovb featureset001 fails baremetal provisioning Permission denied: '/var/lib/ironic/images/" [Critical,Triaged]15:42
*** tosky has quit IRC15:45
weshay|ruckhow is that even running?  https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001&job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset00115:50
weshay|rucksshnaidm, is vex back?15:51
sshnaidmweshay|ruck, according to infra yes15:51
weshay|ruckdon't see shit on rdo-list15:51
sshnaidmweshay|ruck, saw in #rhos-ops15:53
marios|roverweshay|ruck: possibly back but we haven't had confirm yet15:53
marios|roverweshay|ruck: as in, master ran OK, but ussuri/victoria had other issues15:53
marios|roverweshay|ruck: https://bugs.launchpad.net/tripleo/+bug/1907049/comments/4 and https://bugs.launchpad.net/tripleo/+bug/1907049/comments/515:53
openstackLaunchpad bug 1907049 in tripleo "Multiple node_failure on rdo jobs blocking rdo check jobs" [Critical,Triaged]15:53
rlandydonwstream is for sure still out15:53
sshnaidmrlandy, I had one job running today on ds15:54
sshnaidmmaybe just occasional15:54
rlandykeep getting node failures15:56
weshay|ruckhttp://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&var-influxdb_filter=type%7C%3D%7Crdo&var-influxdb_filter=job_name%7C%3D~%7C%2F.*featureset001.*%2F&from=now-6h&to=now15:57
weshay|ruckrocky, queens pass15:57
weshay|ruckstein pass15:57
weshay|ruckthe rest is red15:57
weshay|ruckmarios|rover, chkumar|ruck I would still focus on upstream15:57
weshay|ruckunless it's just totally fubar'd and there waiting time15:58
marios|roverweshay|ruck: i was looking there so we could close that bug15:58
marios|roverweshay|ruck: wasn't checking promotions, per se15:58
weshay|ruckaye.. no worries.. f.. all things other than upstream15:59
weshay|ruckperhaps time to pull in another ruck/rover?16:01
weshay|ruckmarios|rover, chkumar|ruck ^16:05
* marios|rover puts fingers in ears and listens to sean cohen talking about success :)16:05
weshay|ruckbah16:05
marios|roverweshay|ruck: not sure if it will help yet16:05
marios|roverweshay|ruck: i mean16:05
marios|roverweshay|ruck: its not like we aren't catching enough issues... more like we have a bunch open. not sure the bottleneck is us on fixing all of those16:06
weshay|ruckagree.. just you have my attention.. let me know what you think can help16:07
*** rlandy is now known as rlandy|bbl16:11
weshay|ruckzbr, so.. are we abandoning https://review.opendev.org/c/openstack/tripleo-common/+/766018 for https://review.opendev.org/c/openstack/tripleo-ci/+/76599616:11
weshay|ruckah.. we need deps on16:12
weshay|ruckzbr, sshnaidm perhaps we can combine the two reviews16:13
weshay|ruckin hopes of landing it16:13
chkumar|ruckzbr: weshay|ruck need help on this https://review.opendev.org/c/openstack/python-tripleoclient/+/76601216:14
chkumar|ruckfailing here https://logserver.rdoproject.org/16/31316/5/check/periodic-tripleo-centos-8-buildimage-ironic-python-agent-ussuri/dd0a1f8/job-output.txt16:15
chkumar|ruckParse error at line 12, column 7: expected EOF. trail: [version comment rule]16:15
chkumar|ruckmarios|rover: weshay|ruck I think we should turn on tempest tests in upgrade jobs before and after upgrade in periodic jobs?16:22
marios|roverchkumar|ruck: we do it on all of them already afaik16:23
marios|roverchkumar|ruck: well, we removed only from standalone-upgrade recently16:23
marios|roverchkumar|ruck: standalone upgrade runs it only after upgrade16:23
marios|roverchkumar|ruck: because we had timeouts16:24
marios|roverchkumar|ruck: but the general pattern is deploy temptest upgrade tempest16:24
chkumar|ruckmarios|rover: ah, ok, i thought we disabled for all, thanks :-)16:25
chkumar|ruckweshay|ruck: we solved the paunch issue16:28
chkumar|ruckmarios|rover: ^^16:28
chkumar|ruckpatch is merging soon16:28
marios|roverchkumar|ruck: great16:28
weshay|ruckthanks!!!16:28
weshay|ruckone at a time :)16:28
mjturekmarios|rover: sorry to bother, where should I watch for the thread to discuss re-enabling manifests? openstack-discuss, rdo-dev? Or are we just going to have a private email thread?16:36
marios|rovermjturek: yeah i already emailed you check inbox16:37
marios|rovermjturek: private thread16:37
marios|rovermjturek: hoping i got the right address sec, pvt16:38
mjturekcool thanks marios|rover ! I'll check that inbox now!16:40
*** kopecmartin has quit IRC16:43
*** kopecmartin has joined #oooq16:48
*** udesale_ has quit IRC16:55
marios|roverchkumar|ruck: weshay|ruck: me getting out in few mins need something before i do?17:03
weshay|ruckmarios|rover, just what you want me to watch / help w/17:04
weshay|ruckin your evening17:04
marios|roverweshay|ruck: don't have something from today weshay|ruck other than https://review.opendev.org/c/openstack/tripleo-ci/+/765834 https://review.opendev.org/c/openstack/tripleo-quickstart/+/765835 for the gating repo thing17:07
marios|roverweshay|ruck: hoping i get more time to dig into the open bugs instead of just filing new things and updating existing ones tomorrow17:07
weshay|ruckk 0/17:07
marios|roverweshay|ruck: btw, can you merge that?  https://review.rdoproject.org/r/#/c/3120017:08
marios|roveri totally thought we already merged it and only noticed today17:08
marios|roverweshay|ruck: we have new bugs from today if you haven't already noticed https://bugs.launchpad.net/tripleo/+bug/1907245 (deps/pacemaker-libs)17:09
openstackLaunchpad bug 1907245 in tripleo "periodic victoria overcloud image and container builds fail for missing deps pacemaker-libs" [Critical,Triaged]17:09
marios|roverweshay|ruck: also ovb issue? dig/collect logs & filed new https://bugs.launchpad.net/tripleo/+bug/190727217:09
openstackLaunchpad bug 1907272 in tripleo "rdo ovb featureset001 fails baremetal provisioning Permission denied: '/var/lib/ironic/images/" [Critical,Triaged]17:09
marios|roverweshay|ruck: k if you can sort all of those things out that would be great ! ^^^17:10
marios|roverthanks weshay|ruck17:10
marios|rover!!17:10
openstackmarios|rover: Error: "!" is not a valid command.17:10
marios|rover:D17:10
weshay|ruckheh17:10
marios|roverweshay|ruck: in https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0&fvid=1778061576 would be nice to have filter view for cix that includes the new/triage column17:12
marios|roverweshay|ruck: i added one today for 'open launchpad'17:12
weshay|rucksaw that.. ++17:12
weshay|ruckya.. I'll add a new17:12
* weshay|ruck trying to get this thing to work w/ cron17:12
marios|roverweshay|ruck: well, or include it in the 'open cix ' view is more what i meant17:13
weshay|rucknot sure why I'm struggling17:13
weshay|ruckOH17:13
weshay|ruckya17:13
weshay|ruckk17:13
*** owalsh has joined #oooq17:13
weshay|ruckmarios|rover, it is17:13
marios|roverweshay|ruck: yes it is17:13
marios|roverweshay|ruck: i guess it was the delay between things added to the board then the cron job running17:14
marios|roverweshay|ruck: and me refreshing my view17:14
weshay|ruckdoes suck that you have to turn it on/off for it to work properly17:14
marios|roverweshay|ruck: did you do something now? because earlier i couldn't select the new/triage int he results for the cix view17:14
marios|roverweshay|ruck: but now i see it in the things to filter on17:15
marios|roverk me off bai17:15
*** marios|rover is now known as marios|out17:15
weshay|ruckrfolco, are you testing the container build report upstream?17:20
weshay|ruckah.. see it here https://review.rdoproject.org/r/#/c/29840/17:20
rfolcoweshay|ruck, I am using a testproject to get quick results17:20
rfolcoweshay|ruck, moved report playbook to post, so we get a report even on a timeout17:21
rfolco(and this caused me some issues, dealing with undefined vars etc)17:21
*** tosky has joined #oooq17:26
*** marios|out has quit IRC17:30
pojadhavweshay|ruck, when you will be free please have a look : https://review.rdoproject.org/r/#/c/31318/17:46
weshay|ruckpojadhav, k17:47
weshay|ruckpojadhav, that looks good.. going to add a comment re: depends-on17:48
pojadhavweshay|ruck, okay17:48
weshay|ruckpojadhav, add depends-on: https://review.rdoproject.org/r/#/c/28014/17:49
pojadhavweshay|ruck, sure :)17:49
weshay|ruckpojadhav, please add it now17:51
weshay|ruckI need to forward this to another team17:51
pojadhavadding17:51
pojadhavweshay|ruck, done17:54
weshay|ruckpojadhav, thanks17:54
weshay|ruckpojadhav, any unit test reviews?17:54
pojadhavposting.. basic check is done17:55
*** derekh has quit IRC18:00
pojadhavweshay|ruck, Work in progress posted : https://review.rdoproject.org/r/#/c/30492 (tomorrow will dig more about checking keys from both lists and compare both objects)18:00
*** pojadhav is now known as pojadhav|afk18:03
*** dtantsur is now known as dtantsur|afk18:05
*** saneax has quit IRC18:27
weshay|ruckzbr, you still around?18:31
weshay|ruckah phew..18:32
weshay|rucksorry.. recheck was not working there for a few min18:32
zbrnot really, just returned for few secs to reply on an infra msg. why?18:42
weshay|ruckzbr, recheck wasn't working for a few min18:43
weshay|ruckzbr, sorry to disturb18:44
weshay|ruckbut made me nervous18:44
*** Goneri has quit IRC19:18
*** Goneri has joined #oooq19:37
*** TrevorV has quit IRC19:56
*** rlandy|bbl is now known as rlandy|pto20:26
*** rfolco has quit IRC23:06
*** rfolco has joined #oooq23:21
*** rlandy|pto has quit IRC23:35
*** rfolco has quit IRC23:53
*** rfolco has joined #oooq23:53
*** rfolco has quit IRC23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!