Tuesday, 2021-10-12

Clark[m]ianw I kept the LE stuff overridden because it has the hostname in it (cert name for review02) and the testing file doesn't supplement the prod file it completely replaces it aiui00:00
ianwhrm ok.  maybe should have called it review99 in the testing; that would have avoided confusion over what is mixing in from production00:04
Clark[m]We can still do that if we like00:07
Clark[m]Maybe as another follow-up?00:07
ianwyep definitely a follow up.  does seem like a good idea to avoid mixing things00:12
fungiwe've done it elsewhere, so yes great idea01:06
*** ykarel|away is now known as ykarel03:34
*** ysandeep|out is now known as ysandeep04:32
opendevreviewIan Wienand proposed opendev/system-config master: Refactor infra-prod jobs for parallel running  https://review.opendev.org/c/opendev/system-config/+/80767204:36
opendevreviewMerged zuul/zuul-jobs master: build-container-image: improve the reliabilty  https://review.opendev.org/c/zuul/zuul-jobs/+/81320305:49
gkskHi Team05:53
*** ksambor_ is now known as ksambor06:02
opendevreviewDong Zhang proposed zuul/zuul-jobs master: Implement role for limiting zuul log file size  https://review.opendev.org/c/zuul/zuul-jobs/+/81303407:13
*** ykarel_ is now known as ykarel07:47
opendevreviewDong Zhang proposed zuul/zuul-jobs master: WIP, only for test if build works  https://review.opendev.org/c/zuul/zuul-jobs/+/81359308:00
*** ysandeep is now known as ysandeep|lunch08:16
opendevreviewDong Zhang proposed zuul/zuul-jobs master: wip: only for test  https://review.opendev.org/c/zuul/zuul-jobs/+/81359608:26
*** ysandeep|lunch is now known as ysandeep08:37
sshnaidmianw, can we talk please about https://review.opendev.org/c/openstack/diskimage-builder/+/806819 ? Currently it blocks a big chunk of work related to CentOS/RHEL 9 and I'd like to clear things up, because I think there is a misunderstanding11:28
opendevreviewJeremy Stanley proposed zuul/zuul-jobs master: DNM: Exercise tox role on CentOS 7  https://review.opendev.org/c/zuul/zuul-jobs/+/81362012:27
*** thelounge94 is now known as redrobot13:02
*** redrobot is now known as thelounge9413:04
*** thelounge94 is now known as redrobot13:04
clarkbsshnaidm: there is a dib channel, but I think ianw wants a single centos9 stream/rhel9 support change that adds testing. That seems reasonable. There are plenty of examples for adding test builds to dib too14:39
sshnaidmclarkb, there was no any testing for any image before, I don't think it's reasonable to block a patch just because c9 doesn't have it. Testing that he mentioned is for nodepool images, and not related to the patch14:40
sshnaidmclarkb, actually we build images in tripleo check jobs every day with dib, so it's the best testing14:41
clarkbsshnaidm: there is testing for many images, I don't understand why you would say there isn't. And the nodepool jobs are just using nodepool as the engine to drive dib there is nothing really nodepool specific about them. Don't get confused14:42
clarkband opendev builds images every day with dib too, but we still put the testing in dib so that dib can be developed independently14:42
sshnaidmclarkb, all testing there is nodepool related as I see14:42
sshnaidmclarkb, I don't touch any nodepool related stuff, this patch is used in tripleo testing14:43
clarkbsshnaidm: again that testing only uses nodepool to drive dib. It is a test of dib elements. Nodepool is just a tool14:43
sshnaidmclarkb, yeah, and I don't touch dib elements, it's base url for centos 914:43
clarkbsshnaidm: that is a dib element modification14:43
sshnaidmI'm definitely not familiar with nodepool image testing and have no idea how it's tested, but I can add there tripleo job that builds c9 images14:44
clarkbplease don't add the truipleo jobs. I don't understand why asking for standard testing that dib has used for a while now is such a crazy ask14:44
clarkbif you look at the nodepool jobs they are well templated and should be super simple to make a centos-9-stream variant14:45
clarkbsshnaidm: https://opendev.org/openstack/diskimage-builder/src/branch/master/.zuul.d/jobs.yaml#L172-L183 you make a copy of that and update the values to use centos 9 instead14:46
sshnaidmclarkb, thanks, I'll give a try14:46
*** ykarel is now known as ykarel|away14:48
opendevreviewShnaidman Sagi (Sergey) proposed openstack/diskimage-builder master: Improve DIB for building CentOS 9 stream  https://review.opendev.org/c/openstack/diskimage-builder/+/80681914:48
sshnaidmclarkb, like this? ^14:48
clarkbsshnaidm: you also need to add the job to the check and gate in  https://opendev.org/openstack/diskimage-builder/src/branch/master/.zuul.d/project.yaml14:49
opendevreviewShnaidman Sagi (Sergey) proposed openstack/diskimage-builder master: Improve DIB for building CentOS 9 stream  https://review.opendev.org/c/openstack/diskimage-builder/+/80681914:52
ykarel|awaysshnaidm, wrt ^ while reading comments from ianw i got impression that he might be asking for coverage to dib-functests job atleast for 9-stream14:58
ykarel|awayjust like Alfredo attempted in his patch, although seems he too didn't called those tests14:58
ykarel|awaybut ianw can confirm, ^ just what i interpreted14:59
clarkbykarel|away: sshnaidm: I supect both sets of tests are desireable.15:00
ykarel|awayyes both should be better15:00
sshnaidmykarel|away, sorry, which job is it?15:00
clarkbI think what is important is that dib runs its own tests for the distros it cares about. Having third party tests doesn't hurt but dib can't be expected to debug and fix those which is why it has dedicated tests15:00
ykarel|awaysshnaidm, dib-functests15:00
ykarel|awayex. log https://6bce7fce9eeed9c90eb6-1a783ef91f61ec7475bcc10015912dcc.ssl.cf5.rackcdn.com/811392/4/check/dib-functests-bionic-python3/fb371bf/15:01
sshnaidmthey are running on ubuntu15:02
sshnaidmdo you mean these parts? https://github.com/openstack/diskimage-builder/blob/master/.zuul.d/jobs.yaml#L47-L4915:03
clarkbsshnaidm: dib runs on many platforms but it can build other distros from various platforms15:03
sshnaidmwell, that what I meant when I said I have no resources to dive deep into nodepool image testing..15:04
clarkbagain I think you are getting confused because it says "ubuntu" but that is just where the job is running not what elements are tested15:04
clarkbsshnaidm: right but it is well templated and you shouldn't really have to care about nodepool. Just debug any element failures and fix them before landing your element updates15:04
clarkbnodepool is just a tool to drive dib to make an image and have devstack boot it15:04
clarkbwhat you care about is an element failure or failure to boot15:05
sshnaidmtripleo jobs build those images and test them after, why not reuse it? would be nice coverage15:05
clarkbbeacuse dib has nothing to do with tripleo at a fundamental level it makes VM images regardless of who uses. Again oepndev does this too, but we don't run oepndev testing against dib15:06
clarkbthe tripleo jobs are also some of the most difficult jobs our entire CI system runs. They are very difficult to undersatnd and debug. It is difficult to point people at logs for them and have them understand how to correct a problem15:07
clarkbif you are familiar with tripleo this is fine but for everyone else it is problematic15:07
sshnaidmclarkb, I mean job that builds image only, nothing more15:07
sshnaidmand ensures it really built qcow image or whatever15:08
clarkbsshnaidm: that is what the job you just added to your update does. And it does it in a standard way for all dib images. Is tripleo going to start building suse and debian and ubuntu images? If not having a standard method is preferable15:08
sshnaidmclarkb, yeah, I doubt we start building ubuntu..15:08
fungialso gentoo, openeuler, ...15:09
fungipossibly alma15:09
sshnaidmdoes someone use gentoo for openstack deploy?15:13
clarkbsshnaidm: I'm not sure, but I think gentoo may use or did use dib to produce their cloud images for booting in clouds15:14
clarkbagain dib isn't a tool specific to deploying openstack like with tripleo. It can produce generic disk images to boot VMs or even baremetal servers15:15
sshnaidmI see15:19
clarkbit wouldn't surprise me if OSUOSL actually does run their oepnstack stuff on gentoo though15:21
fungiwe build gentoo images and upload them to our providers so we can test things like roles in zuul-jobs15:25
opendevreviewyatin proposed openstack/diskimage-builder master: [WIP] Add support for CentOS Stream 9 in DIB  https://review.opendev.org/c/openstack/diskimage-builder/+/81139215:25
*** marios is now known as marios|out15:35
clarkbfungi: I think https://review.opendev.org/c/opendev/system-config/+/813534 is ready for review now as I've largely addressed ianw's concerns. Does deserve careful review though. I'm going to work on the followup(s) we talked about yesterday next16:17
fungigreat, thanks16:18
*** ysandeep is now known as ysandeep|out16:18
opendevreviewClark Boylan proposed opendev/system-config master: Switch test gerrit hostname to review99.opendev.org  https://review.opendev.org/c/opendev/system-config/+/81367116:28
opendevreviewClark Boylan proposed opendev/system-config master: Replace testing group vars with host vars for review02  https://review.opendev.org/c/opendev/system-config/+/81353416:49
opendevreviewClark Boylan proposed opendev/system-config master: Switch test gerrit hostname to review99.opendev.org  https://review.opendev.org/c/opendev/system-config/+/81367116:49
opendevreviewClark Boylan proposed opendev/system-config master: Remove the gerrit group in favor of the review group  https://review.opendev.org/c/opendev/system-config/+/81367516:49
clarkbI ended up finding a bug in the first change of that stack when writing the last chagne of that stack. A minor thing but decided since we hadn't approved it yet to go ahead and fix it16:49
fungicorvus: not exactly sure what the cause is (seems to be comment-related), but doing an interdiff between patchsets 4 and 5 of 813534 in gertty crashes consistently, i think it may be expecting a line number for a comment and getting None instead, possibly related to the gerrit 3.3 upgrade17:42
fungi2021-10-12 18:39:08,972 Expand chunk <DiffContextChunk old lines 1-19 / new lines 1-19> 17 None18:41
fungi2021-10-12 18:39:08,981 Expand chunk <DiffContextChunk old lines 1-117 / new lines 1-117> 126 None18:42
fungithat's the last thing in the debug log before it falls over18:42
fungidoesn't seem related as that's logged in a different part of the function18:44
fungilooks like the comment_lists passed to handleUndisplayedComments() has at least one item where splitting the key on - yields a None second element but is expected to always be a string recastable as an integer18:47
fungiprobably a new comment type in 3.318:48
fungiif i can figure out which one, i'll add a filter similar to the one for /PATCHSET_LEVEL18:49
clarkbyou should be able to request the list of comments from the api? I ended up doing similar when debugging that zuul post job enqueue issue18:56
fungiyeah, i'm about there18:57
opendevreviewIan Wienand proposed opendev/system-config master: Refactor infra-prod jobs for parallel running  https://review.opendev.org/c/opendev/system-config/+/80767219:18
sshnaidmianw, clarkb please take a look at https://review.opendev.org/c/openstack/diskimage-builder/+/806819 again, there are failures which are not related to my patch, I think it's still early to create a testing for CentOS 9. My patch only starts to fix this, it's not an intention to solve all existing problems with CentOS 919:22
fungiokay, turns out we're not properly filtering out /PATCHSET_LEVEL and that's the problem. the comment key it's choking on is old-None-/PATCHSET_LEVEL19:41
fungiaha, fixed! the previous patch for that only did half the job19:44
clarkbsshnaidm: it looks related to me? https://zuul.opendev.org/t/openstack/build/8fb313ccb7514c6e8c78c1464b57ac1d/log/nodepool/builds/test-image-0000000001.log#363-36419:47
opendevreviewJeremy Stanley proposed ttygroup/gertty master: More skip patchset-level comments in diff view  https://review.opendev.org/c/ttygroup/gertty/+/81369919:50
fungicorvus: ^ that should fix it for good19:50
ianwclarkb: i really thought 807672 would run noop jobs ...20:01
clarkbianw: hrm that is curious20:01
ianwfrickler: to follow up, do you think we should rm things on the wheel mirrors?20:03
ianw(i have to admit i haven't even checked wheel releases lately ...)20:04
clarkbwe optimized to reduce mirror build time and file copies iirc20:04
fricklerianw: I'm not sure what the best solution is yet. I'll have a look at the spec first I think20:05
clarkbbut the only way to know a package doesn't build is to attempt to build it20:05
clarkbthere are definitely a few competing needs here which might make this annoying, but should be solvable20:05
ianwhttps://grafana.opendev.org/d/T5zTt6PGk/afs?orgId=1 ... no surprises, it's sad20:05
fricklermaybe a refresh job that rebuilds wheels, replaces them with new ones, deletes ones for which builds fail?20:05
fricklerat first I though of only doing something like a periodic devstack/tempest job that verifies running without our wheels still works20:07
ianwrefreshing would help with the PEP503 index generation20:07
fricklerbut then I saw that also nova unit tests are failing locally20:07
fricklerso maybe better tackle this at the root - our wheel build process20:08
ianwthe issue there was that it includes a hash, and reading all of AFS is sloooowwww20:08
frickleranyway, it's getting late here, will have another look tomorrow20:09
ianwit would definitely help to increase observability of these jobs.  every time i'm back to grepping to remember all the job names and finding logs20:09
ianwfrickler: thanks; the "scorched earth" refresh idea i hadn't really considered, but i can see it might work.  i'll also think on it20:10
opendevreviewMerged opendev/system-config master: borg-backup-server: wait for lock in verify  https://review.opendev.org/c/opendev/system-config/+/81262220:11
opendevreviewIan Wienand proposed opendev/system-config master: [dnm] testing what readme triggers, if anything  https://review.opendev.org/c/opendev/system-config/+/81370020:12
opendevreviewIan Wienand proposed opendev/system-config master: [dnm] testing syntax error  https://review.opendev.org/c/opendev/system-config/+/81370020:22
clarkbfungi I noticed you didn't approve the gerrit 3.2 image cleanup chagne. Any reason for me not to approve it now?20:34
clarkbthinking on the setuptools thing more its annoying that the decision to kill 2to3 was made. It seems perfectly reasonably that dormant code bases rely on taht for python3 support20:45
clarkbits probably in everyone's best interest to have proper python3 code bases at this point but from the user's perspective as long as it runs on the python3 interpreter that is good enough for a lot of stuff20:46
corvusfungi: thanks! +320:47
opendevreviewMerged ttygroup/gertty master: More skip patchset-level comments in diff view  https://review.opendev.org/c/ttygroup/gertty/+/81369920:47
fungiclarkb: mainly just in case anyone else was reviewing it after the meeting20:49
fungihappy to go ahead and approve now20:50
clarkbfungi: cool, thanks20:50
opendevreviewMerged opendev/system-config master: Remove Gerrit 3.2 images  https://review.opendev.org/c/opendev/system-config/+/81307420:59
clarkbPTG email sent with a link to the etherpad and time location details22:16
opendevreviewClark Boylan proposed opendev/gerritlib master: Add python39 testing  https://review.opendev.org/c/opendev/gerritlib/+/81371022:24
clarkbthats a super simple test update to add python3.9 testing to help cover the gerritlib use via jeepyb on our gerrit images22:25
opendevreviewClark Boylan proposed opendev/system-config master: Clean up our gerrit config  https://review.opendev.org/c/opendev/system-config/+/81371623:37
clarkbtotally non urgent but one of the things I noticed when doing all this gerrit stuff is we've got a few stale things in our gerrit config23:37

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!