| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Update regexes in zuul config to the new style https://review.opendev.org/c/openstack/project-config/+/893702 | 06:51 |
|---|---|---|
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: DNM: testing zuul config error https://review.opendev.org/c/openstack/project-config/+/893703 | 06:51 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: DNM: testing zuul config error https://review.opendev.org/c/openstack/project-config/+/893703 | 07:01 |
| gthiemonge | thanks for fixing that issue folks! | 07:05 |
| gthiemonge | it seems that "branches: ^(?!stable)" triggers a warning now ("The RE2 syntax error is: invalid perl operator: (?!"), don't we have such regexs in many places? | 07:09 |
| frickler | gthiemonge: yes, I counted > 200 yesterday. see 893702 | 07:16 |
| frickler | ehm ... see that patch for an example how I expect this to be fixed | 07:16 |
| frickler | but for now it should be possible to ignore these warnings, please let us know if that is not the case | 07:16 |
| gthiemonge | frickler: thanks | 07:17 |
| *** ralonsoh_ is now known as ralonsoh | 08:41 | |
| SvenKieske | not sure if it's the right channel - I suppose so - is it on purpose, that the "download-logs.sh" script on zuul is downloading the logs sequentially in order to reduce load on the systems? | 09:00 |
| SvenKieske | I currently don't find the git source for this file, it seems to be only mentioned in tempest docs. | 09:02 |
| *** dmitriis is now known as Guest1871 | 09:12 | |
| SvenKieske | because if this is not intentional I'd like to tweak this, because I find the time spent waiting for it really annoying, it's also a weird mix of bash and python3 and probably could use a rewrite in python, as python is already used there. | 09:13 |
| SvenKieske | found it: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/local-log-download/templates/download-logs.sh.j2 | 09:40 |
| frickler | SvenKieske: if you really want to modify that script, likely #zuul on matrix.org is better suited for discussion | 11:42 |
| frickler | also in my experience downloading all the logs is almost never needed, IMO we could rather consider dropping this role in opendev | 11:44 |
| fungi | SvenKieske: i believe the reason it's python wrapped in bash is to support a cut-n-paste curl|bash workflow (which i'm not a fan of, but i should be able to find those early discussions if you want) | 12:01 |
| SvenKieske | frickler: please don't drop this script, I'm actually using it. | 12:41 |
| SvenKieske | fungi: it's still a weird mix of things done in bash and in python. I can understand the curl | bash workflow, but that also works with directly calling python, whatever | 12:42 |
| fungi | indeed it does | 12:47 |
| opendevreview | Merged zuul/zuul-jobs master: roles/ensure-python: Fix 'python_use_stow' option https://review.opendev.org/c/zuul/zuul-jobs/+/871822 | 14:52 |
| opendevreview | Merged openstack/project-config master: Update regexes in zuul config to the new style https://review.opendev.org/c/openstack/project-config/+/893702 | 14:53 |
| *** dasm is now known as Guest1929 | 14:57 | |
| *** Guest1929 is now known as dasm | 15:02 | |
| clarkb | Iv'e discovered that lenovo puts a memtest type checker in its uefi system that you can select at boot time. I've got that running on the laptop now | 16:00 |
| clarkb | seems like more devices should have tools like this built into firmware | 16:01 |
| corvus | clarkb: you said you have display artifacts? checking memory because it's shared between cpu/gpu? | 16:06 |
| clarkb | yup exactly | 16:07 |
| clarkb | its an amd APU with the gpu built into the cpu and then they share main memory | 16:08 |
| clarkb | no errors yet. I'm beginning to wonder if it could be a software issue. I should probably boot a live version of gnome or kde | 16:18 |
| fungi | these days, the memtest86+ package in debian installs /boot/memtest86+x64.efi | 16:19 |
| clarkb | fungi: looks like opensuse ships a memtest86+ package that probably does the same thing. I don't want to trust that though because if memory is bad then package installs can be iffy | 16:20 |
| clarkb | but probably a good idea for me to start isntalling that everywhere for the potential need later | 16:21 |
| fungi | oh, lenovo bakes one into rom or something? i expected uefi to just read from the drive anyway | 16:21 |
| clarkb | yes its not on disk as far as I can tell its in firmware somewhere | 16:22 |
| clarkb | typically the first thing I do with these laptops is completely wipe the harddrive and get rid of the special partitions like the windows install recovery partition | 16:24 |
| fungi | yeah, same | 16:25 |
| yoctozepto | morning; could you help me refresh my memory? (relatively) simple question: does zuul execute ansible-playbook on the first node from the nodeset or does it use an independent executor? (and/or does it depend on zuul's config?) | 16:59 |
| clarkb | yoctozepto: it is executed on the zuul executor which is "localhost" relative to the ansible inventory in the execution env | 17:00 |
| yoctozepto | thanks clarkb | 17:00 |
| yoctozepto | second question - would it be possible to run in opendev's CI a custom image for the nodeset? | 17:01 |
| yoctozepto | e.g., to avoid having to download and install k8s each time and having it preloaded instead? | 17:02 |
| yoctozepto | any past experiences? | 17:02 |
| clarkb | yoctozepto: we cache some things like git repos and cirros imges. But we've been trying to move away from that because we can't cache everything everyone might need | 17:04 |
| yoctozepto | ok, so definitely no plans to add more custom images? | 17:04 |
| yoctozepto | I'm actually trying to optimise the resource usage by avoiding the network and vm time use for that repeated activity | 17:05 |
| clarkb | correct instead we try to cache things at the network layer for common resources. For example container images hosted by docker and quay | 17:06 |
| clarkb | what we found is that the more we add to the images the more assumptiosn we have to make that will end up breaking someones testing. Additionally, we can't easily delete things from those images so they just grow larger and more unwieldy over time | 17:06 |
| yoctozepto | I see; yeah, the ecosystem is huge, everyone wants something | 17:07 |
| yoctozepto | do I benefit from the network-layer caching in opendev automatically when running the ensure-docker/podman? | 17:07 |
| clarkb | I think they get enabled if you use the intermediate registry/buildset registry but I don't think ensure-docker is sufficient | 17:10 |
| clarkb | oh wait use-docker-mirror sets it up and is called when installing docker from upstream (not distro) packages | 17:11 |
| clarkb | I don't see similar for podman though | 17:12 |
| clarkb | infra-root I'm working on updating the meeting agenda (late) today. Anything else need to be added or edited? | 17:13 |
| clarkb | oh fungi filed an issue with rax so we can followup on where that ended | 17:14 |
| frickler | fwiw github seems to be having recurring issues, since yesterday they've reported a new incident every couple of hours | 17:26 |
| corvus | FYI: remote: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/893792 Use re2 compatible regexes in branch matchers [NEW] | 17:32 |
| frickler | fungi: seem indeed there was an infinite loop initially fixed by https://review.opendev.org/c/openstack/project-config/+/706257 and https://review.opendev.org/c/openstack/project-config/+/717277/1/zuul.d/pipelines.yaml is where the \S came from | 17:38 |
| frickler | so all in all I'm pretty convinced that the current solution does what we want. I'll let the release team know to watch out for any unexpected events anyway | 17:39 |
| frickler | corvus: there's two rocky cleanup patches that will likely conflict with your patch, do you want to stack them on top of yours or the other way round (not sure which way will be easier to resolve) https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/891628/ and 891629 | 17:49 |
| frickler | it also seems the gerrit has gotten rather bad at noticing merge conflicts | 17:50 |
| corvus | frickler: when should those merge? | 17:52 |
| corvus | (like, "any time now" or "after some specified date in the future")? | 17:53 |
| frickler | corvus: I need to check on that blocker, but rocky has been eoled a long time ago, so asap | 17:54 |
| corvus | ok. also, looks like maybe we need a zuul-sphinx change... | 17:54 |
| frickler | seem blazar@pike hasn't been updated yet, may need to force-merge a patch there. I can check that tomorrow, but likely we should go ahead with your patch first, then | 17:56 |
| frickler | elodilles: stable/pike in blazar is another case that predates adoption by release automation. did we ever come up with a solution to eol-ing those other than "nag elod until he gives in"? ;) | 18:01 |
| frickler | iirc there a also some stable/newton branches still pending | 18:01 |
| frickler | +re | 18:01 |
| clarkb | memory checked out on the laptop. So I went ahead and did system updates just in case it is a software bug that got fixed. No dice. I took a screenshot to see if it is rendering problem only but the artifacts show up in the screenshot too so the problem seems upstream of the display. IceWM does the same thing so not xfce4. I'm probably going to need to rma arg | 18:20 |
| elodilles | frickler: :) well, it's still somewhere deep down on my TODO list o:) | 18:28 |
| elodilles | frickler: if there are .zuul.yaml on those repos then i suggest to force-merge some .zuul.yaml removals as a quick solution | 18:28 |
| elodilles | frickler: like you did with some zuul config error fixes | 18:29 |
| elodilles | frickler: i don't want to manually abuse the branch deletion, so it will need some general/scripted solution :/ | 18:30 |
| frickler | elodilles: well I'm not convinced (ab)using force-merge powers is a better solution, but I can take a look at that | 18:33 |
| clarkb | the internets say linux 6.4 and amdgpu have been having troubles, but the major workaround people point to doesn't help me so may not be related | 18:48 |
| clarkb | and I was on 6.4 for a bit before this happened. I should stop doing rubber ducky gpu debugging in this channel though | 18:48 |
| fungi | there are times when i'm happy i stick to intel video controllers | 18:50 |
| clarkb | infra-root ianychoi[m]: https://paste.opendev.org/show/bNUzKY3RVlV4VKs4mBtN/ this is the error produced by zanata when I request the example url ianychoi[m] provided | 20:54 |
| clarkb | /opt/wildfly/standalone/log/server.log is the file path | 20:54 |
| clarkb | it does seem related to sql, but in a way that makes me wonder if this ever worked? | 20:54 |
| clarkb | it looks like that error originates in mysql itself not the java side. | 20:56 |
| fungi | a web search on that suggests this is a behavior change in mysql 5.7, so may have worked in older versions | 21:03 |
| fungi | "strict mode" | 21:04 |
| fungi | apparently disabling strict_mode will allow such (nondeterministic) queries to be process | 21:05 |
| fungi | ed | 21:05 |
| fungi | skimming various discussions, it looks like strict mode in 5.7 was a little too overzealous, and later mysql versions walked that back by checking whether the query was actually nondeterministic rather than the crude heuristic used initially | 21:06 |
| clarkb | I guess we can try disabling strict mode then | 21:08 |
| clarkb | infra-root how does this look for the fedora image removal: https://etherpad.opendev.org/p/PGyJUS1O0kJYXi1nIvmN | 21:09 |
| fungi | for a transitional migration period, disabling strict mode is probably an acceptable workaround | 21:09 |
| fungi | or even just temporarily in order to get the export to run, if we're worried about leaving it that way | 21:09 |
| fungi | clarkb: i suppose the big takeaway is that removing the nodeset definitions can move this from "some jobs are failing" to "this project has a broken zuul configuration" and in the past we saw nodeset removal block projects from merging changes to remove their use of the missing nodesets | 21:11 |
| fungi | at one point we worked around that with a broken nodeset definition of some sort, though now i forget the details | 21:12 |
| fungi | or label removal similarly, i guess | 21:12 |
| clarkb | fungi: yes, but projects can define their own nodesets too | 21:14 |
| clarkb | and in this case anyone still using fedora is doing so with a different ndoeset since I looked for cases of the one we removed and bindep was theo nly example | 21:15 |
| clarkb | fungi it looks like there are many mysql strict mode options. Is STRICT_TRANS_TABLES the one we are running into? | 21:19 |
| clarkb | oh no its probably ONLY_FULL_GROUP_BY | 21:19 |
| clarkb | so we could do something lik `set global SQL_MODE="NO_ENGINE_SUBSTITUTION";` in the running server and see if that helps | 21:20 |
| clarkb | I don't know if restarting the server after doing that will persist the setting or not | 21:20 |
| clarkb | https://www.percona.com/blog/upgrading-to-mysql-5-7-beware-of-the-new-strict-mode/ for reference | 21:20 |
| clarkb | https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html#sqlmode_only_full_group_by currently mysql docs for that ONLY_FULL_GROUP_BY setting | 21:21 |
| clarkb | we can query the current value with `SELECT @@GLOBAL.sql_mode;` | 21:23 |
| clarkb | so ya that seems reasonable. Query the current value and record it for potential revert purposes then set it to the 5.6 default which is what the string above would do | 21:23 |
| clarkb | `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION` is the current value | 21:25 |
| clarkb | should I update it with `set global SQL_MODE="NO_ENGINE_SUBSTITUTION";` and then refresh ianychoi[m]'s rest url? | 21:25 |
| fungi | i have no objection, that sounds like a reasonable approach | 21:32 |
| clarkb | ok proceeding with that now | 21:32 |
| clarkb | I don't have permissions to do this apparently | 21:33 |
| clarkb | I'm guessing we may need to dig up server admin credentials from trove? | 21:33 |
| clarkb | ya the connection details we have on the translate server are for the service account | 21:34 |
| clarkb | fungi: do you know if trove allows us to do that? | 21:34 |
| fungi | oh, so it's a trove instance. i didn't think about that possibility | 21:34 |
| clarkb | yes it appears to be so since this is an old style setup | 21:34 |
| fungi | you can create custom trove configs and apply them, though it requires a restart of the trove instance to switch between them | 21:35 |
| clarkb | fungi: do you knwo if we can just get the root account connection info instead? | 21:35 |
| clarkb | that is probably easier in this instance if it is supported | 21:35 |
| clarkb | But I have a hunch we can't do that for security reasons | 21:35 |
| fungi | i've not tried | 21:35 |
| fungi | i've only created trove config objects (and via rackspace's webui for that matter) | 21:36 |
| clarkb | looks like our openstackclient install on bridge doesn't have the database command stuff installed either so ya web ui seems required | 21:37 |
| clarkb | there is a root user but they don't give you access to it unless you jump through api hoops first. https://docs.rackspace.com/docs/enabling-the-root-user-on-cloud-databases The root user stuff is greyed out in the web ui with a link to those docs | 21:42 |
| clarkb | I see the configuration stuff. Unfortunately there isn't an easy way to determine if a configuration is used by more than one server | 21:44 |
| clarkb | I think for this reason I will need to make a copy of the configuration and then apply it that way? | 21:44 |
| clarkb | I think only the two translate dbs are using this particular config though | 21:46 |
| clarkb | I'll try setting it this way I guess | 21:48 |
| clarkb | the mode has been changed but I still get the error loading ianychoi[m]'s url. I suspect the issue there is the existing connection doesn't chagne modes and we need to restart zanata | 21:52 |
| clarkb | note: both the rax web ui and the sql query to check sql_mode show the new value as updating globally | 21:52 |
| clarkb | I need to pause here. But I think if someone wants to figure out how to restart zanata that is likely the next thing to check | 21:53 |
| clarkb | fungi: any objections to that email I drafted re fedora removal? | 21:54 |
| fungi | clarkb: yes, is there a custom configuration already applied to the instance? either way, copying it and editing and applying the copy is the safest path | 21:58 |
| fungi | draft fedora label removal announcement seems fine to me | 21:59 |
| clarkb | fungi: I didn't find a way to copy it so I edited it in place after confirming that this config was only used by the two translate dbs | 22:05 |
| clarkb | fungi: that is all done now but same error from zanata. I think we need to restart zanata so that it creates a new session with the new sql_mode setting associuated to that session | 22:05 |
| clarkb | email sent | 22:07 |
| clarkb | I need to pop out and enjoy some sunshien before its gone. I can look at zanata restarts more closely later. I'm half worried it won't come up again if we restart it given how old and stale it is... | 22:08 |
| fungi | clarkb: you restarted the trove instance after updating the config, right? (i think it forces you to anyway) | 22:09 |
| clarkb | fungi: no, it said "setting updated successfully" (or similar) and then I queried `SELECT @@GLOBAL.sql_mode;` and it showed me the new value I set | 22:10 |
| clarkb | a restart for that particular config item doesn't appear to be required | 22:10 |
| fungi | oh, huh. maybe it's just swapping configs that forces a restart | 22:10 |
| clarkb | I think existing sessions will keep their old setting though | 22:11 |
| clarkb | whcih is why I suspect we need to restart zanata to create a new sql connection and session with the new value | 22:11 |
| fungi | makes sense | 22:12 |
| corvus | https://review.opendev.org/893792 is ready to go: re2 update for openstack-zuul-jobs | 22:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!