Tuesday, 2023-09-05

opendevreviewDr. Jens Harbott proposed openstack/project-config master: Update regexes in zuul config to the new style  https://review.opendev.org/c/openstack/project-config/+/89370206:51
opendevreviewDr. Jens Harbott proposed openstack/project-config master: DNM: testing zuul config error  https://review.opendev.org/c/openstack/project-config/+/89370306:51
opendevreviewDr. Jens Harbott proposed openstack/project-config master: DNM: testing zuul config error  https://review.opendev.org/c/openstack/project-config/+/89370307:01
gthiemongethanks for fixing that issue folks!07:05
gthiemongeit seems that "branches: ^(?!stable)" triggers a warning now ("The RE2 syntax error is: invalid perl operator: (?!"), don't we have such regexs in many places?07:09
fricklergthiemonge: yes, I counted > 200 yesterday. see 89370207:16
fricklerehm ... see that patch for an example how I expect this to be fixed07:16
fricklerbut for now it should be possible to ignore these warnings, please let us know if that is not the case07:16
gthiemongefrickler: thanks07:17
*** ralonsoh_ is now known as ralonsoh08:41
SvenKieskenot sure if it's the right channel - I suppose so - is it on purpose, that the "download-logs.sh" script on zuul is downloading the logs sequentially in order to reduce load on the systems?09:00
SvenKieskeI currently don't find the git source for this file, it seems to be only mentioned in tempest docs.09:02
*** dmitriis is now known as Guest187109:12
SvenKieskebecause if this is not intentional I'd like to tweak this, because I find the time spent waiting for it really annoying, it's also a weird mix of bash and python3 and probably could use a rewrite in python, as python is already used there.09:13
SvenKieskefound it: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/local-log-download/templates/download-logs.sh.j209:40
fricklerSvenKieske: if you really want to modify that script, likely #zuul on matrix.org is better suited for discussion11:42
frickleralso in my experience downloading all the logs is almost never needed, IMO we could rather consider dropping this role in opendev11:44
fungiSvenKieske: i believe the reason it's python wrapped in bash is to support a cut-n-paste curl|bash workflow (which i'm not a fan of, but i should be able to find those early discussions if you want)12:01
SvenKieskefrickler: please don't drop this script, I'm actually using it.12:41
SvenKieskefungi: it's still a weird mix of things done in bash and in python. I can understand the curl | bash workflow, but that also works with directly calling python, whatever12:42
fungiindeed it does12:47
opendevreviewMerged zuul/zuul-jobs master: roles/ensure-python: Fix 'python_use_stow' option  https://review.opendev.org/c/zuul/zuul-jobs/+/87182214:52
opendevreviewMerged openstack/project-config master: Update regexes in zuul config to the new style  https://review.opendev.org/c/openstack/project-config/+/89370214:53
*** dasm is now known as Guest192914:57
*** Guest1929 is now known as dasm15:02
clarkbIv'e discovered that lenovo puts a memtest type checker in its uefi system that you can select at boot time. I've got that running on the laptop now16:00
clarkbseems like more devices should have tools like this built into firmware16:01
corvusclarkb: you said you have display artifacts?  checking memory because it's shared between cpu/gpu?16:06
clarkbyup exactly16:07
clarkbits an amd APU with the gpu built into the cpu and then they share main memory16:08
clarkbno errors yet. I'm beginning to wonder if it could be a software issue. I should probably boot a live version of gnome or kde16:18
fungithese days, the memtest86+ package in debian installs /boot/memtest86+x64.efi16:19
clarkbfungi: looks like opensuse ships a memtest86+ package that probably does the same thing. I don't want to trust that though because if memory is bad then package installs can be iffy16:20
clarkbbut probably a good idea for me to start isntalling that everywhere for the potential need later16:21
fungioh, lenovo bakes one into rom or something? i expected uefi to just read from the drive anyway16:21
clarkbyes its not on disk as far as I can tell its in firmware somewhere16:22
clarkbtypically the first thing I do with these laptops is completely wipe the harddrive and get rid of the special partitions like the windows install recovery partition16:24
fungiyeah, same16:25
yoctozeptomorning; could you help me refresh my memory? (relatively) simple question: does zuul execute ansible-playbook on the first node from the nodeset or does it use an independent executor? (and/or does it depend on zuul's config?)16:59
clarkbyoctozepto: it is executed on the zuul executor which is "localhost" relative to the ansible inventory in the execution env17:00
yoctozeptothanks clarkb17:00
yoctozeptosecond question - would it be possible to run in opendev's CI a custom image for the nodeset?17:01
yoctozeptoe.g., to avoid having to download and install k8s each time and having it preloaded instead?17:02
yoctozeptoany past experiences?17:02
clarkbyoctozepto: we cache some things like git repos and cirros imges. But we've been trying to move away from that because we can't cache everything everyone might need17:04
yoctozeptook, so definitely no plans to add more custom images?17:04
yoctozeptoI'm actually trying to optimise the resource usage by avoiding the network and vm time use for that repeated activity17:05
clarkbcorrect instead we try to cache things at the network layer for common resources. For example container images hosted by docker and quay17:06
clarkbwhat we found is that the more we add to the images the more assumptiosn we have to make that will end up breaking someones testing. Additionally, we can't easily delete things from those images so they just grow larger and more unwieldy over time17:06
yoctozeptoI see; yeah, the ecosystem is huge, everyone wants something17:07
yoctozeptodo I benefit from the network-layer caching in opendev automatically when running the ensure-docker/podman?17:07
clarkbI think they get enabled if you use the intermediate registry/buildset registry but I don't think ensure-docker is sufficient17:10
clarkboh wait use-docker-mirror sets it up and is called when installing docker from upstream (not distro) packages17:11
clarkbI don't see similar for podman though17:12
clarkbinfra-root I'm working on updating the meeting agenda (late) today. Anything else need to be added or edited?17:13
clarkboh fungi filed an issue with rax so we can followup on where that ended17:14
fricklerfwiw github seems to be having recurring issues, since yesterday they've reported a new incident every couple of hours17:26
corvusFYI: remote:   https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/893792 Use re2 compatible regexes in branch matchers [NEW]17:32
fricklerfungi: seem indeed there was an infinite loop initially fixed by https://review.opendev.org/c/openstack/project-config/+/706257 and https://review.opendev.org/c/openstack/project-config/+/717277/1/zuul.d/pipelines.yaml is where the \S came from17:38
fricklerso all in all I'm pretty convinced that the current solution does what we want. I'll let the release team know to watch out for any unexpected events anyway17:39
fricklercorvus: there's two rocky cleanup patches that will likely conflict with your patch, do you want to stack them on top of yours or the other way round (not sure which way will be easier to resolve) https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/891628/ and 89162917:49
fricklerit also seems the gerrit has gotten rather bad at noticing merge conflicts17:50
corvusfrickler: when should those merge?17:52
corvus(like, "any time now"  or  "after some specified date in the future")?17:53
fricklercorvus: I need to check on that blocker, but rocky has been eoled a long time ago, so asap17:54
corvusok.  also, looks like maybe we need a zuul-sphinx change...17:54
fricklerseem blazar@pike hasn't been updated yet, may need to force-merge a patch there. I can check that tomorrow, but likely we should go ahead with your patch first, then17:56
fricklerelodilles: stable/pike in blazar is another case that predates adoption by release automation. did we ever come up with a solution to eol-ing those other than "nag elod until he gives in"? ;)18:01
frickleriirc there a also some stable/newton branches still pending18:01
clarkbmemory checked out on the laptop. So I went ahead and did system updates just in case it is a software bug that got fixed. No dice. I took a screenshot to see if it is rendering problem only but the artifacts show up in the screenshot too so the problem seems upstream of the display. IceWM does the same thing so not xfce4. I'm probably going to need to rma arg18:20
elodillesfrickler: :) well, it's still somewhere deep down on my TODO list o:)18:28
elodillesfrickler: if there are .zuul.yaml on those repos then i suggest to force-merge some .zuul.yaml removals as a quick solution18:28
elodillesfrickler: like you did with some zuul config error fixes18:29
elodillesfrickler: i don't want to manually abuse the branch deletion, so it will need some general/scripted solution :/18:30
fricklerelodilles: well I'm not convinced (ab)using force-merge powers is a better solution, but I can take a look at that18:33
clarkbthe internets say linux 6.4 and amdgpu have been having troubles, but the major workaround people point to doesn't help me so may not be related18:48
clarkband I was on 6.4 for a bit before this happened. I should stop doing rubber ducky gpu debugging in this channel though18:48
fungithere are times when i'm happy i stick to intel video controllers18:50
clarkbinfra-root ianychoi[m]: https://paste.opendev.org/show/bNUzKY3RVlV4VKs4mBtN/ this is the error produced by zanata when I request the example url ianychoi[m] provided20:54
clarkb/opt/wildfly/standalone/log/server.log is the file path20:54
clarkbit does seem related to sql, but in a way that makes me wonder if this ever worked?20:54
clarkbit looks like that error originates in mysql itself not the java side.20:56
fungia web search on that suggests this is a behavior change in mysql 5.7, so may have worked in older versions21:03
fungi"strict mode"21:04
fungiapparently disabling strict_mode will allow such (nondeterministic) queries to be process21:05
fungiskimming various discussions, it looks like strict mode in 5.7 was a little too overzealous, and later mysql versions walked that back by checking whether the query was actually nondeterministic rather than the crude heuristic used initially21:06
clarkbI guess we can try disabling strict mode then21:08
clarkbinfra-root how does this look for the fedora image removal: https://etherpad.opendev.org/p/PGyJUS1O0kJYXi1nIvmN21:09
fungifor a transitional migration period, disabling strict mode is probably an acceptable workaround21:09
fungior even just temporarily in order to get the export to run, if we're worried about leaving it that way21:09
fungiclarkb: i suppose the big takeaway is that removing the nodeset definitions can move this from "some jobs are failing" to "this project has a broken zuul configuration" and in the past we saw nodeset removal block projects from merging changes to remove their use of the missing nodesets21:11
fungiat one point we worked around that with a broken nodeset definition of some sort, though now i forget the details21:12
fungior label removal similarly, i guess21:12
clarkbfungi: yes, but projects can define their own nodesets too21:14
clarkband in this case anyone still using fedora is doing so with a different ndoeset since I looked for cases of the one we removed and bindep was theo nly example21:15
clarkbfungi it looks like there are many mysql strict mode options. Is STRICT_TRANS_TABLES the one we are running into?21:19
clarkboh no its probably ONLY_FULL_GROUP_BY21:19
clarkbso we could do something lik `set global SQL_MODE="NO_ENGINE_SUBSTITUTION";` in the running server and see if that helps21:20
clarkbI don't know if restarting the server after doing that will persist the setting or not21:20
clarkbhttps://www.percona.com/blog/upgrading-to-mysql-5-7-beware-of-the-new-strict-mode/ for reference21:20
clarkbhttps://dev.mysql.com/doc/refman/8.0/en/sql-mode.html#sqlmode_only_full_group_by currently mysql docs for that ONLY_FULL_GROUP_BY setting21:21
clarkbwe can query the current value with `SELECT @@GLOBAL.sql_mode;`21:23
clarkbso ya that seems reasonable. Query the current value and record it for potential revert purposes then set it to the 5.6 default which is what the string above would do21:23
clarkbshould I update it with `set global SQL_MODE="NO_ENGINE_SUBSTITUTION";` and then refresh ianychoi[m]'s rest url?21:25
fungii have no objection, that sounds like a reasonable approach21:32
clarkbok proceeding with that now21:32
clarkbI don't have permissions to do this apparently21:33
clarkbI'm guessing we may need to dig up server admin credentials from trove?21:33
clarkbya the connection details we have on the translate server are for the service account21:34
clarkbfungi: do you know if trove allows us to do that?21:34
fungioh, so it's a trove instance. i didn't think about that possibility21:34
clarkbyes it appears to be so since this is an old style setup21:34
fungiyou can create custom trove configs and apply them, though it requires a restart of the trove instance to switch between them21:35
clarkbfungi: do you knwo if we can just get the root account connection info instead?21:35
clarkbthat is probably easier in this instance if it is supported21:35
clarkbBut I have a hunch we can't do that for security reasons21:35
fungii've not tried21:35
fungii've only created trove config objects (and via rackspace's webui for that matter)21:36
clarkblooks like our openstackclient install on bridge doesn't have the database command stuff installed either so ya web ui seems required21:37
clarkbthere is a root user but they don't give you access to it unless you jump through api hoops first. https://docs.rackspace.com/docs/enabling-the-root-user-on-cloud-databases The root user stuff is greyed out in the web ui with a link to those docs21:42
clarkbI see the configuration stuff. Unfortunately there isn't an easy way to determine if a configuration is used by more than one server21:44
clarkbI think for this reason I will need to make a copy of the configuration and then apply it that way?21:44
clarkbI think only the two translate dbs are using this particular config though21:46
clarkbI'll try setting it this way I guess21:48
clarkbthe mode has been changed but I still get the error loading ianychoi[m]'s url. I suspect the issue there is the existing connection doesn't chagne modes and we need to restart zanata21:52
clarkbnote: both the rax web ui and the sql query to check sql_mode show the new value as updating globally21:52
clarkbI need to pause here. But I think if someone wants to figure out how to restart zanata that is likely the next thing to check21:53
clarkbfungi: any objections to that email I drafted re fedora removal?21:54
fungiclarkb: yes, is there a custom configuration already applied to the instance? either way, copying it and editing and applying the copy is the safest path21:58
fungidraft fedora label removal announcement seems fine to me21:59
clarkbfungi: I didn't find a way to copy it so I edited it in place after confirming that this config was only used by the two translate dbs22:05
clarkbfungi: that is all done now but same error from zanata. I think we need to restart zanata so that it creates a new session with the new sql_mode setting associuated to that session22:05
clarkbemail sent22:07
clarkbI need to pop out and enjoy some sunshien before its gone. I can look at zanata restarts more closely later. I'm half worried it won't come up again if we restart it given how old and stale it is...22:08
fungiclarkb: you restarted the trove instance after updating the config, right? (i think it forces you to anyway)22:09
clarkbfungi: no, it said "setting updated successfully" (or similar) and then I queried `SELECT @@GLOBAL.sql_mode;` and it showed me the new value I set22:10
clarkba restart for that particular config item doesn't appear to be required22:10
fungioh, huh. maybe it's just swapping configs that forces a restart22:10
clarkbI think existing sessions will keep their old setting though22:11
clarkbwhcih is why I suspect we need to restart zanata to create a new sql connection and session with the new value22:11
fungimakes sense22:12
corvushttps://review.opendev.org/893792 is ready to go: re2 update for openstack-zuul-jobs22:23

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!