| Clark[m] | ianw I kept the LE stuff overridden because it has the hostname in it (cert name for review02) and the testing file doesn't supplement the prod file it completely replaces it aiui | 00:00 |
|---|---|---|
| ianw | hrm ok. maybe should have called it review99 in the testing; that would have avoided confusion over what is mixing in from production | 00:04 |
| Clark[m] | We can still do that if we like | 00:07 |
| Clark[m] | Maybe as another follow-up? | 00:07 |
| ianw | yep definitely a follow up. does seem like a good idea to avoid mixing things | 00:12 |
| fungi | we've done it elsewhere, so yes great idea | 01:06 |
| *** ykarel|away is now known as ykarel | 03:34 | |
| *** ysandeep|out is now known as ysandeep | 04:32 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Refactor infra-prod jobs for parallel running https://review.opendev.org/c/opendev/system-config/+/807672 | 04:36 |
| opendevreview | Merged zuul/zuul-jobs master: build-container-image: improve the reliabilty https://review.opendev.org/c/zuul/zuul-jobs/+/813203 | 05:49 |
| gksk | Hi Team | 05:53 |
| *** ksambor_ is now known as ksambor | 06:02 | |
| opendevreview | Dong Zhang proposed zuul/zuul-jobs master: Implement role for limiting zuul log file size https://review.opendev.org/c/zuul/zuul-jobs/+/813034 | 07:13 |
| *** ykarel_ is now known as ykarel | 07:47 | |
| opendevreview | Dong Zhang proposed zuul/zuul-jobs master: WIP, only for test if build works https://review.opendev.org/c/zuul/zuul-jobs/+/813593 | 08:00 |
| *** ysandeep is now known as ysandeep|lunch | 08:16 | |
| opendevreview | Dong Zhang proposed zuul/zuul-jobs master: wip: only for test https://review.opendev.org/c/zuul/zuul-jobs/+/813596 | 08:26 |
| *** ysandeep|lunch is now known as ysandeep | 08:37 | |
| sshnaidm | ianw, can we talk please about https://review.opendev.org/c/openstack/diskimage-builder/+/806819 ? Currently it blocks a big chunk of work related to CentOS/RHEL 9 and I'd like to clear things up, because I think there is a misunderstanding | 11:28 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: DNM: Exercise tox role on CentOS 7 https://review.opendev.org/c/zuul/zuul-jobs/+/813620 | 12:27 |
| *** thelounge94 is now known as redrobot | 13:02 | |
| *** redrobot is now known as thelounge94 | 13:04 | |
| *** thelounge94 is now known as redrobot | 13:04 | |
| clarkb | sshnaidm: there is a dib channel, but I think ianw wants a single centos9 stream/rhel9 support change that adds testing. That seems reasonable. There are plenty of examples for adding test builds to dib too | 14:39 |
| sshnaidm | clarkb, there was no any testing for any image before, I don't think it's reasonable to block a patch just because c9 doesn't have it. Testing that he mentioned is for nodepool images, and not related to the patch | 14:40 |
| sshnaidm | clarkb, actually we build images in tripleo check jobs every day with dib, so it's the best testing | 14:41 |
| clarkb | sshnaidm: there is testing for many images, I don't understand why you would say there isn't. And the nodepool jobs are just using nodepool as the engine to drive dib there is nothing really nodepool specific about them. Don't get confused | 14:42 |
| clarkb | and opendev builds images every day with dib too, but we still put the testing in dib so that dib can be developed independently | 14:42 |
| sshnaidm | clarkb, all testing there is nodepool related as I see | 14:42 |
| sshnaidm | clarkb, I don't touch any nodepool related stuff, this patch is used in tripleo testing | 14:43 |
| clarkb | sshnaidm: again that testing only uses nodepool to drive dib. It is a test of dib elements. Nodepool is just a tool | 14:43 |
| sshnaidm | clarkb, yeah, and I don't touch dib elements, it's base url for centos 9 | 14:43 |
| clarkb | sshnaidm: that is a dib element modification | 14:43 |
| sshnaidm | I'm definitely not familiar with nodepool image testing and have no idea how it's tested, but I can add there tripleo job that builds c9 images | 14:44 |
| clarkb | please don't add the truipleo jobs. I don't understand why asking for standard testing that dib has used for a while now is such a crazy ask | 14:44 |
| clarkb | if you look at the nodepool jobs they are well templated and should be super simple to make a centos-9-stream variant | 14:45 |
| clarkb | sshnaidm: https://opendev.org/openstack/diskimage-builder/src/branch/master/.zuul.d/jobs.yaml#L172-L183 you make a copy of that and update the values to use centos 9 instead | 14:46 |
| sshnaidm | clarkb, thanks, I'll give a try | 14:46 |
| *** ykarel is now known as ykarel|away | 14:48 | |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/diskimage-builder master: Improve DIB for building CentOS 9 stream https://review.opendev.org/c/openstack/diskimage-builder/+/806819 | 14:48 |
| sshnaidm | clarkb, like this? ^ | 14:48 |
| clarkb | sshnaidm: you also need to add the job to the check and gate in https://opendev.org/openstack/diskimage-builder/src/branch/master/.zuul.d/project.yaml | 14:49 |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/diskimage-builder master: Improve DIB for building CentOS 9 stream https://review.opendev.org/c/openstack/diskimage-builder/+/806819 | 14:52 |
| ykarel|away | sshnaidm, wrt ^ while reading comments from ianw i got impression that he might be asking for coverage to dib-functests job atleast for 9-stream | 14:58 |
| ykarel|away | just like Alfredo attempted in his patch, although seems he too didn't called those tests | 14:58 |
| ykarel|away | but ianw can confirm, ^ just what i interpreted | 14:59 |
| clarkb | ykarel|away: sshnaidm: I supect both sets of tests are desireable. | 15:00 |
| ykarel|away | yes both should be better | 15:00 |
| sshnaidm | ykarel|away, sorry, which job is it? | 15:00 |
| clarkb | I think what is important is that dib runs its own tests for the distros it cares about. Having third party tests doesn't hurt but dib can't be expected to debug and fix those which is why it has dedicated tests | 15:00 |
| ykarel|away | sshnaidm, dib-functests | 15:00 |
| ykarel|away | ex. log https://6bce7fce9eeed9c90eb6-1a783ef91f61ec7475bcc10015912dcc.ssl.cf5.rackcdn.com/811392/4/check/dib-functests-bionic-python3/fb371bf/ | 15:01 |
| sshnaidm | they are running on ubuntu | 15:02 |
| sshnaidm | do you mean these parts? https://github.com/openstack/diskimage-builder/blob/master/.zuul.d/jobs.yaml#L47-L49 | 15:03 |
| ykarel|away | yes | 15:03 |
| clarkb | sshnaidm: dib runs on many platforms but it can build other distros from various platforms | 15:03 |
| sshnaidm | well, that what I meant when I said I have no resources to dive deep into nodepool image testing.. | 15:04 |
| clarkb | again I think you are getting confused because it says "ubuntu" but that is just where the job is running not what elements are tested | 15:04 |
| clarkb | sshnaidm: right but it is well templated and you shouldn't really have to care about nodepool. Just debug any element failures and fix them before landing your element updates | 15:04 |
| clarkb | nodepool is just a tool to drive dib to make an image and have devstack boot it | 15:04 |
| clarkb | what you care about is an element failure or failure to boot | 15:05 |
| sshnaidm | tripleo jobs build those images and test them after, why not reuse it? would be nice coverage | 15:05 |
| clarkb | beacuse dib has nothing to do with tripleo at a fundamental level it makes VM images regardless of who uses. Again oepndev does this too, but we don't run oepndev testing against dib | 15:06 |
| clarkb | the tripleo jobs are also some of the most difficult jobs our entire CI system runs. They are very difficult to undersatnd and debug. It is difficult to point people at logs for them and have them understand how to correct a problem | 15:07 |
| clarkb | if you are familiar with tripleo this is fine but for everyone else it is problematic | 15:07 |
| sshnaidm | clarkb, I mean job that builds image only, nothing more | 15:07 |
| sshnaidm | and ensures it really built qcow image or whatever | 15:08 |
| clarkb | sshnaidm: that is what the job you just added to your update does. And it does it in a standard way for all dib images. Is tripleo going to start building suse and debian and ubuntu images? If not having a standard method is preferable | 15:08 |
| sshnaidm | clarkb, yeah, I doubt we start building ubuntu.. | 15:08 |
| fungi | also gentoo, openeuler, ... | 15:09 |
| fungi | possibly alma | 15:09 |
| sshnaidm | does someone use gentoo for openstack deploy? | 15:13 |
| clarkb | sshnaidm: I'm not sure, but I think gentoo may use or did use dib to produce their cloud images for booting in clouds | 15:14 |
| clarkb | again dib isn't a tool specific to deploying openstack like with tripleo. It can produce generic disk images to boot VMs or even baremetal servers | 15:15 |
| sshnaidm | I see | 15:19 |
| clarkb | it wouldn't surprise me if OSUOSL actually does run their oepnstack stuff on gentoo though | 15:21 |
| fungi | we build gentoo images and upload them to our providers so we can test things like roles in zuul-jobs | 15:25 |
| opendevreview | yatin proposed openstack/diskimage-builder master: [WIP] Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 15:25 |
| *** marios is now known as marios|out | 15:35 | |
| clarkb | fungi: I think https://review.opendev.org/c/opendev/system-config/+/813534 is ready for review now as I've largely addressed ianw's concerns. Does deserve careful review though. I'm going to work on the followup(s) we talked about yesterday next | 16:17 |
| fungi | great, thanks | 16:18 |
| *** ysandeep is now known as ysandeep|out | 16:18 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch test gerrit hostname to review99.opendev.org https://review.opendev.org/c/opendev/system-config/+/813671 | 16:28 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Replace testing group vars with host vars for review02 https://review.opendev.org/c/opendev/system-config/+/813534 | 16:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch test gerrit hostname to review99.opendev.org https://review.opendev.org/c/opendev/system-config/+/813671 | 16:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove the gerrit group in favor of the review group https://review.opendev.org/c/opendev/system-config/+/813675 | 16:49 |
| clarkb | I ended up finding a bug in the first change of that stack when writing the last chagne of that stack. A minor thing but decided since we hadn't approved it yet to go ahead and fix it | 16:49 |
| fungi | corvus: not exactly sure what the cause is (seems to be comment-related), but doing an interdiff between patchsets 4 and 5 of 813534 in gertty crashes consistently, i think it may be expecting a line number for a comment and getting None instead, possibly related to the gerrit 3.3 upgrade | 17:42 |
| fungi | 2021-10-12 18:39:08,972 Expand chunk <DiffContextChunk old lines 1-19 / new lines 1-19> 17 None | 18:41 |
| fungi | 2021-10-12 18:39:08,981 Expand chunk <DiffContextChunk old lines 1-117 / new lines 1-117> 126 None | 18:42 |
| fungi | that's the last thing in the debug log before it falls over | 18:42 |
| fungi | doesn't seem related as that's logged in a different part of the function | 18:44 |
| fungi | looks like the comment_lists passed to handleUndisplayedComments() has at least one item where splitting the key on - yields a None second element but is expected to always be a string recastable as an integer | 18:47 |
| fungi | probably a new comment type in 3.3 | 18:48 |
| fungi | if i can figure out which one, i'll add a filter similar to the one for /PATCHSET_LEVEL | 18:49 |
| clarkb | you should be able to request the list of comments from the api? I ended up doing similar when debugging that zuul post job enqueue issue | 18:56 |
| fungi | yeah, i'm about there | 18:57 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Refactor infra-prod jobs for parallel running https://review.opendev.org/c/opendev/system-config/+/807672 | 19:18 |
| sshnaidm | ianw, clarkb please take a look at https://review.opendev.org/c/openstack/diskimage-builder/+/806819 again, there are failures which are not related to my patch, I think it's still early to create a testing for CentOS 9. My patch only starts to fix this, it's not an intention to solve all existing problems with CentOS 9 | 19:22 |
| fungi | okay, turns out we're not properly filtering out /PATCHSET_LEVEL and that's the problem. the comment key it's choking on is old-None-/PATCHSET_LEVEL | 19:41 |
| fungi | aha, fixed! the previous patch for that only did half the job | 19:44 |
| clarkb | sshnaidm: it looks related to me? https://zuul.opendev.org/t/openstack/build/8fb313ccb7514c6e8c78c1464b57ac1d/log/nodepool/builds/test-image-0000000001.log#363-364 | 19:47 |
| opendevreview | Jeremy Stanley proposed ttygroup/gertty master: More skip patchset-level comments in diff view https://review.opendev.org/c/ttygroup/gertty/+/813699 | 19:50 |
| fungi | corvus: ^ that should fix it for good | 19:50 |
| ianw | clarkb: i really thought 807672 would run noop jobs ... | 20:01 |
| clarkb | ianw: hrm that is curious | 20:01 |
| ianw | frickler: to follow up, do you think we should rm things on the wheel mirrors? | 20:03 |
| ianw | (i have to admit i haven't even checked wheel releases lately ...) | 20:04 |
| clarkb | we optimized to reduce mirror build time and file copies iirc | 20:04 |
| frickler | ianw: I'm not sure what the best solution is yet. I'll have a look at the spec first I think | 20:05 |
| clarkb | but the only way to know a package doesn't build is to attempt to build it | 20:05 |
| clarkb | there are definitely a few competing needs here which might make this annoying, but should be solvable | 20:05 |
| ianw | https://grafana.opendev.org/d/T5zTt6PGk/afs?orgId=1 ... no surprises, it's sad | 20:05 |
| frickler | maybe a refresh job that rebuilds wheels, replaces them with new ones, deletes ones for which builds fail? | 20:05 |
| frickler | at first I though of only doing something like a periodic devstack/tempest job that verifies running without our wheels still works | 20:07 |
| ianw | refreshing would help with the PEP503 index generation | 20:07 |
| frickler | but then I saw that also nova unit tests are failing locally | 20:07 |
| frickler | so maybe better tackle this at the root - our wheel build process | 20:08 |
| ianw | the issue there was that it includes a hash, and reading all of AFS is sloooowwww | 20:08 |
| frickler | anyway, it's getting late here, will have another look tomorrow | 20:09 |
| ianw | it would definitely help to increase observability of these jobs. every time i'm back to grepping to remember all the job names and finding logs | 20:09 |
| ianw | frickler: thanks; the "scorched earth" refresh idea i hadn't really considered, but i can see it might work. i'll also think on it | 20:10 |
| opendevreview | Merged opendev/system-config master: borg-backup-server: wait for lock in verify https://review.opendev.org/c/opendev/system-config/+/812622 | 20:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] testing what readme triggers, if anything https://review.opendev.org/c/opendev/system-config/+/813700 | 20:12 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] testing syntax error https://review.opendev.org/c/opendev/system-config/+/813700 | 20:22 |
| clarkb | fungi I noticed you didn't approve the gerrit 3.2 image cleanup chagne. Any reason for me not to approve it now? | 20:34 |
| clarkb | thinking on the setuptools thing more its annoying that the decision to kill 2to3 was made. It seems perfectly reasonably that dormant code bases rely on taht for python3 support | 20:45 |
| clarkb | its probably in everyone's best interest to have proper python3 code bases at this point but from the user's perspective as long as it runs on the python3 interpreter that is good enough for a lot of stuff | 20:46 |
| corvus | fungi: thanks! +3 | 20:47 |
| opendevreview | Merged ttygroup/gertty master: More skip patchset-level comments in diff view https://review.opendev.org/c/ttygroup/gertty/+/813699 | 20:47 |
| fungi | clarkb: mainly just in case anyone else was reviewing it after the meeting | 20:49 |
| fungi | happy to go ahead and approve now | 20:50 |
| clarkb | fungi: cool, thanks | 20:50 |
| opendevreview | Merged opendev/system-config master: Remove Gerrit 3.2 images https://review.opendev.org/c/opendev/system-config/+/813074 | 20:59 |
| clarkb | PTG email sent with a link to the etherpad and time location details | 22:16 |
| opendevreview | Clark Boylan proposed opendev/gerritlib master: Add python39 testing https://review.opendev.org/c/opendev/gerritlib/+/813710 | 22:24 |
| clarkb | thats a super simple test update to add python3.9 testing to help cover the gerritlib use via jeepyb on our gerrit images | 22:25 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Clean up our gerrit config https://review.opendev.org/c/opendev/system-config/+/813716 | 23:37 |
| clarkb | totally non urgent but one of the things I noticed when doing all this gerrit stuff is we've got a few stale things in our gerrit config | 23:37 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!