Monday, 2023-09-25

opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS optimiser repo
opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS optimiser-utility-evaluator repo
opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS optimiser-utility-evaluator repo
yoctozepto <- any project creation heroes out there? :-)10:46
opendevreviewMerged openstack/project-config master: Add NebulOuS optimiser-utility-evaluator repo
yoctozeptoaand this time my thanks go to... fungi13:03
opendevreviewMerged opendev/git-review master: Warn rather than fail if HEAD already exists on the remote
fungiany time13:07
opendevreviewMerged opendev/git-review master: feat(cmd): add hashtag implementation
ildikovHello OpenDev team! I have a question about the mm3 migration, specifically regarding the starlingx-discuss ML.14:08
ildikovI'm a list admin for that mailing list, and received an email that there is a new message in the moderation queue. The notification email doesn't have a link to the admin page, even though the previous version provided that.14:09
ildikovI already had a mm3 account, since I'm on other mailing lists that were migrated before. But, the starlingx-discuss mailing list does not show up under that account, even though all the email addresses are listed that I'm signed up with.14:10
ildikovDo I need to create a new account on top of the one that I already have? Or is there any other way to have starlingx-discuss to show up on my existing account?14:11
fungiildikov: which lists are displayed will depend on which domain you visit. are you clicking the login link at 14:26
fungithe account is shared across all our mailman sites, but you have to log into each one independently (when we get an sso linked to it that may get simpler)14:27
ildikovfungi: this is the first time I see that link, prolly missed it in other emails before. I used this one:
fungioh, sorry. is the url for the starlingx mailing lists (still the same as it was before). logging in anywhere on that site should hopefully get you to what you need14:28
fungi is the url for the opendev mailing lists, is for the airship mailing lists, and so on14:29
fungieach one is filtered by the lists which are associated with their domain14:29
fungi is another14:29
fungias is
fungiand too14:30
fungithe only mailman site we haven't migrated yet is but that should be done in a few weeks, just wanted to wait until after the openstack release14:33
ildikovah, ok, got it14:35
ildikovso I will have one account across all domains, but I can't access the MLs on different domains in one place14:36
ildikovis that an accurate way to sum it up?14:36
fungiwe had a configuration error early on where the zuul and opendev mailing lists all showed up on each others management interfaces, but that's been fixed14:37
ildikovwell, I would be happy to still have that error, lol15:15
ildikovfungi: thank you for the pointer and clarifications!15:15
slittleplease add me as first member for starlingx-app-node-interface-metrics-exporter-core.  As usual I'll add the rest.  Thanks15:19
fungislittle: done15:29
clarkbI'm working on local software updates and after that I'15:53
clarkber I'll recycle my gerrit hold to grab the latest patchset I just pushed upstream to the replication plugin and test that15:53
clarkbdecided i didn't need to test code i knew would be replaced almost immediately15:53
clarkbinfra-root: etherpad made a new release if anyone has time to whip up a change for that. Also, gitea has new RCs for the next release we can start looking at16:04
fungii can work on the etherpad update in a sec16:14
clarkbcorvus: the zuul deployment on Friday should've pulled in the token fallback for github? So now we can create a token using an account (do we have one, maybe the one that created the zuul app?) and add it to the github config?16:31
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing replication plugin
clarkbour explicit checkouts for plugin tags override my upstream changesi n master that I was trying to test. I was really confused why nothing seemed to work the way I expected it on my test machine until I sorted that out17:34
clarkbbut now I have a working replication config for local replication that was able to reproduce the issues we have in production and I should hopefully see those go away with that latest patchset17:34
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade Etherpad to 1.9.3
opendevreviewRadosÅ‚aw Piliszek proposed openstack/project-config master: Add nebulous/activemq repo
fungiguilhermesp: do you happen to know if anyone ever looked into the ipv6 routing loop in ca-ymq-1? i'm still seeing it, fwiw17:42
corvusclarkb: that's all generally correct except that we already have that api_token in the config file, so no action should be needed17:46
clarkbcorvus: oh nice! I didn't realize it was already handled17:49
opendevreviewMerged openstack/project-config master: Add nebulous/activemq repo
fungiinfra-root: the etherpad 1.9.3 change seems to be passing tests. changelog is minimal seems to only affect deployment choices other than the ones we make. any preference for whether we hold and manually exercise a job node for this one?18:12
clarkbI tend to prefer doing a hold mostly because it is hard to test what etherpad does automatically given its highly interactive nature18:15
fricklerI won't have time to check the hold and looking at the changelog would be fine to proceed without one, but up to you18:17
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM force etherpad failure to hold node
clarkbfwiw I just looked at the gitea rc release and there is no changelog yet so not super urgent18:30
fricklerinfra-root: seems down, nothing listening on localhost:8000, maybe related to new image deployed this morning?18:38
clarkbfrickler: whcih image was that?18:39
fricklerclarkb: ptgbot? or is something else doing the web backend?18:41
clarkb that one?18:41
clarkbya I think ptgbot does it. I just wanted aware of changes but that does appear to be a likely candidate18:41
clarkbassumiong we have to apply similar to the dockerfile?18:41
fricklerthat's only a change to the readme, so the real change behind that must be older18:43
fricklerbut then maybe the page has been down for longer18:44
clarkbdiablo_rojo: ^ was the website checked after the python3.11 update?18:45
clarkbfwiw there don't appear to be any logs indicating a crash but I agree the web server process does not seem to be working18:45
clarkbthe ircbot side is running and logging18:45
diablo_rojoclarkb: ehhh I am not sure. I looked Friday. When was the py3.11 update? 18:46
clarkbdiablo_rojo: wednesday ish18:47
fungithere were two ptgbot changes merged yesterday, btw18:47
clarkbboth of those changes don't appear to affect hte web server though18:48
clarkbbut they would have triggered rebuilds which may have pulled in newer deps of something18:48
fungiyeah, i approved them but didn't expect them to break anything given they weren't touching any actual code, so it's possible something external influenced the build18:48
fungimy point was we got a new image in the past 24 hours18:49
clarkb logs for most recent build18:49
clarkb logs for the python3.11 update on wednesday18:49
diablo_rojoclarkb: I am pretty sure I saw the website up and fine since then. 80% sure I was looking at it Friday. 18:50
fungiConnection refused: AH00957: HTTP: attempt to connect to (localhost) failed18:50
fungii'm able to connect to localhost:8000 and get content back from cli18:51
clarkbfungi: the process is running now but was not before18:52
clarkbthe docker container has not restarted. Maybe frickler started it manually in the container?18:52
fungii think i may have just crashed it18:52
fungii can't connect any more18:53
clarkbya it isn't runnin anymore18:53
fricklerI tried running it manually, yes, that worked18:53
clarkbmaybe something else crashed it like fungi did18:53
fricklerbut I stopped it again so we can check why it didn't run automatically or stopped in between18:53
clarkbI see18:53
clarkbit doesn't look like the web server logs a log file like the ircbot does18:55
fungiit may log to the container output18:55
clarkbfungi: ya it does18:56
fungidocker-compose logs has some content18:56
clarkbbut nothing useful for an indication of why the process stopped18:56
funginothing useful in syslog or dmesg either18:57
fricklermy next idea would be to restart the docker container and see what happens18:58
fungiyeah, i think that's probably where we're at. and look at adding some logging to the process18:59
fricklerwell, we already run with --debug, I don't think there is much more we can do? "usage: ptgbot-web [-h] [-d] [-p PORT] [--debug] configfile"19:01
clarkbfrickler: someone (maybe diablo_rojo?) would need to improve the webserver to have better logging19:02
fricklerok, so that's longer term then. any objection to just restarting now?19:04
fungiat a minimum, it looks like it should be logging the string 'Debugging on' at start if the debugging option is working, and i don't see that19:04
diablo_rojoI think there are a lot of changes that need to happen to the bot in general to accommodate pivoting to making meetpad the default (which I unfortunately think will have to happen after this PTG; I really don't have the time to get that done and tested etc before the PTG). 19:04
fungialso its RequestHandler class is supposed to be logging every single request it receivs19:05
diablo_rojoAnd in super meta form I thought about having a ptgbot session towards the end of the week to plan some of that out, but I haven't planned that. 19:05
diablo_rojofrickler: no objection from me19:05
fungiso i agree there seems to be the intent of debug logging but it doesn't appear to actually be happening?19:06
fungioh, actually that is getting logged19:09
fungiptgbot_1  | 2023-09-25T01:18:08.201924174Z DEBUG:root:Debugging on19:09
fricklerfungi: I see "DEBUG:root:Debugging on" in the docker log19:09
fungiso maybe it's just that RequestHandler.log_message() isn't being called by anything19:09
fricklerjust nothing after that, even when it is serving requests19:09
fricklerdiablo_rojo: making meetpad the default is something I would very much like to see happen, so I'd volunteer to join any effort supporting that19:11
fungialso it's not clear to me whether what's there is sufficient to capture exceptions/tracebacks19:11
diablo_rojofrickler: that would be amazing. I definitely think it needs to happen too. Especially now that people have more familiarity and experience and the scale of the meetings is smaller so meetpad can definitely handle it. 19:12
fungifrickler: yeah, i had a think about meetpad-as-default and basically instead of requiring the room urls to be supplied in the default config blob, it shouldn't be too hard to infer them from the etherpads for the teams as they're booked (just call the same thing that the url command does whenever the etherpad url is set or changed)19:13
fungimost of what would be needed is already there, because the url command overrides the default room link on any room the team is booked into with one that's specific to the team19:14
frickleralright so I see you're way ahead of me, I didn't look at any of the implementation things so far19:17
fungias for logging, the reason i thought it wasn't working is that i don't see 'Debugging on' in syslog, even though the compose file sets logging to syslog19:17
fungiso whatever logging is being captured by docker/docker-compose doesn't seem to be replicating into syslog19:18
frickleranyway, it's late here, so I'll step out for now and leave that debugging to you, will check back tomorrow19:19
fungiunless the "syslog" logging driver doesn't do what i expect it to19:19
fungithanks frickler!19:19
fungiokay, i'll go ahead and start the container back up and see what happens19:20
fungiand yeah, it's up and responding but still doesn't log individual requests19:22
clarkbfungi: syslog means write logs using the syslog protocol but not necessarily to the syslog file. In this case those contents go to /var/log/containers files using syslogd tagging rules iirc19:43
clarkbthe infrastructure logging is fine as far as I can tell. The problem is htep rocess isn't logging much19:43
clarkbI've just opushed what I hope is a reviewable version of the replication plugin fix. I rechecked and recycled my hold goign to eat lunch while I wait for that to finish19:44
fungiyeah, i can confirm /var/log/containers/docker-ptgbot.log has the same things that docker-compose logs outputs19:44
fungiheld etherpad 1.9.3 has ip address
fungii need to go grab early dinner, but can try it out when i get back19:47
clarkbI've just marked ready for review and left notes about the testing I did there if anyone is curious21:06
clarkbfungi: I'm using the held etherpad in the clarkb-test pad21:47
clarkbfungi: it works in firefox but breaks in chrome21:48
clarkbReferenceError: require is not defined21:49
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Deprecate mirror-workspace-git-repos
clarkbif I create a random new pad it seems to not break?21:49
clarkband now clarkb-test works. I wonder if I had old js cached21:50
clarkbhrm then it started reconnecting when I tried to use chat but now works. Weird21:50
clarkbbut ya best guess is caches. Maybe you want to clear all your caches for before you test and see if you can reproduce? if not we are probably fine21:51
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Deprecate mirror-workspace-git-repos
clarkbI suppose it is possible that it is trying to hit the prod server because chrome isn't respecting /etc/hosts properly21:52
clarkbwow reconnectiong is very persistent :/21:52
clarkbya icognito seems more stable so almost certainly a local state problem. Of course we risk having problems on upgrade when everyone's local state is wonly21:53
fungii'll test in an ff container21:55
fungiseems to work fine for me. no disconnects21:58
clarkbcan you try chromium?21:58
clarkbmy main concern is that maybe chrom* have some problem with evicting cached state so when we upgrade people will be sad for a short time. In FF it definitely seems to just work with no problems21:59
clarkbas well as in incognito mode21:59
fungii don't often use chromium, so probably nothing to clear in my case22:00
fungibut it seems fine (not incognito, but it's the only tab i have open)22:00
clarkbin my case I only use it for testing this stuff 22:00
clarkbwhich I guess means it is possible I've cached some development type stuff for etherpad in chrome22:00
fungii'll test from qutebrowser on my netbook, it tends to struggle with etherpad's widget layouts for some reason22:01
clarkbso it seems like it is probably ok to upgrade. maybe if we can find another person to test that might be good but otherwise don't worry too much about it.22:09
fungiyeah, i see no problems22:09
clarkbrelated to the gerrit replication plugin testing `gerrit logging set-level all` is super useful22:09
fungiother than my netbook's networking is fighting me, but i don't think that's the test server's problem22:10
fungioh, because i can't type. wrong ip address22:16
fungiyeah etherpad's still not great in qutebrowser but this version is no worse than the last22:16
clarkbtime to edit the meeting agenda. I can put etherpad and gitea and gerrit things on there. Anything else we want to discuss? The openstack release I suppose as well as ptgbot web server crash?22:35
fungisounds good. i don't have anything else other than mm3 recap and planning22:40
clarkbfungi: how does the updated agenda look?22:51
fungisorry, looking23:20
clarkbno worries I just sent it out23:20
clarkbwe can always add stuff in during the meeting if necessary23:20
fungiclarkb: looks good to me23:22
clarkbfungi: one thought re bindep and mysql/mariadb woes. The client is almost certainly there for test environment setup not direct integration with the software23:22
fungii'm interested to dig deeper on the zuul registry topic and find out won't work with openssl v323:22
fungiyeah, so the metapackage is probably fine23:23
clarkbfungi: the thing that doesn't work with openssl v3 is python's rehash library that allows for pickling/saving hash function state on data so that you don't have to recompute hashes from scratch as you add more data to files. This is improtant to the registry due to how you add in data over time to images23:23
fungiis rehash part of stdlib or external?23:23
clarkbit does mangling of C things as well as memory so is sensitive to the openssl apis and their changes23:24
fungi that thing23:24
fungigot it23:26
fungiclarkb: so, on debian sid where there is no longer any libssl1.1 and hasn23:32
fungi't been for a while, i build openssl from source and link my python source builds to that23:32
fungiin theory we could build old openssl in those images?23:32
clarkbin this case we would need to link rehash to openssl not stdlib ? or maybe it is both? that gets complicated with tehse images beacuse we're consuming a prebuild python from upstream but is theoretically doable23:32
fungiyeah, i dunno if rehash built for libssl1.1 would work with a stdlib linked against libssl323:33
clarkbit checks the version using from ssl import OPENSSL_VERSION so ya we'd need to rebuild python on the image23:33
fungimakes sense23:34
clarkbbutthen we'd stop using the upstream python and potentially have differences. Thats a lot of work for a single app23:35
clarkbprobably easier to just keep using bullseye for now23:35
clarkbthen trim all the bullseye builds for python that aren't that specific one23:35
fungifwiw, cpython itself doesn't even touch openssl, the only thing from stdlib that links it is the ssl module23:36
fungilooks like i can probably drop my custom openssl linking for cpython 3.9 and later now23:37
clarkbrereading the issue in rehash the problem is more fundamental than simply updating the use of C apis23:37
fungilooks like it relies on internal implementation details of old openssl23:38
clarkbthat seems like a big flaw in openssl 3.0 since hashing is expensive being able to resume from a function state is useful a number of reasons23:38
clarkbfungi: well it seems old openssl had a specific interface for capturing this state and now openssl does not23:38
fungithere are probably other cryptographic primitives that are preferable to incremental hashing, but i don't know enough about the problem domain to say what exactly23:39
clarkbwe need to dangle "resumable sha256sum + rust" in front of the internt and wait for takers. Shouldn't take long :)23:39
clarkbfungi: we're limited by the docker image protocol. I'm not sure we have alternatives23:39
fungioh, right if docker cares about a specific hash construction then that's that23:40
clarkbdocker image protocol uses sha256 for everything and different objects get uploaded at different times and "combined" into a single thing that needs hashing23:40
fungi is disappointingly brief23:44
clarkblooks like golang stdlib has resumable hashes built in23:44
clarkbwe could do a hacky export of Go method to C and call it from python that way assuming we can serialize the necessary state in a meaningful way23:45
fungiIf a library context is needed then all EVP_* digest functions that return a const EVP_MD * such as EVP_sha256() should be replaced with a call to EVP_MD_fetch(3). See "ALGORITHM FETCHING" in crypto(7).23:48
fungiThe algorithm implementation that is fetched can then be used with other diverse functions that use them. For example the EVP_DigestInit_ex(3) function takes as a parameter an EVP_MD object which may have been returned from an earlier call to EVP_MD_fetch(3).23:49
fungiso that looks like maybe you can resume it by passing the structure along from iteration to iteration?23:50
fungibut this is pretty far out of my wheelhouse23:50
fungii guess the underlying problem is that while you can repeatedly call EVP_DigestUpdate() on a context to add more data, it's not clear that there's a way to export the internal state of that context so that a later process can continue adding data23:54
fungiso you're stuck holding that context in process memory23:55

Generated by 2.17.3 by Marius Gedminas - find it at!