Wednesday, 2021-09-15

melwittdoes anyone know whether has been released yet? asking bc blueprints aren't getting updated when expected. if it's been released can someone please share any error messages that may have been logged?00:27
ianwthat is and so should be incorporated00:29
fungihave we updated our gerrit container image and restarted into it since new jeepyb would have ended up in it?00:33
Clark[m]We have restarted since that change landed00:34
Clark[m]Multiple times00:34
ianwyeah the hook is there, i'm just poking at the container now00:36
ianwdo we have an example change so i can try replaying the hook?00:37
ianwthe command "/usr/local/bin/update-blueprint patchset-created --change 809020 --change-url '' --commit a5e78811c65d72870d4b758d9038398a3a1b11f7 --project openstack/nova-specs"00:45
ianwappears to complete without failure00:45
ianwhrm, i just replayed for
ianwit parsed and saved and update to the whiteboard correctly01:22
ianwthe problem is the last update to that was mid july when possibly the fix wasn't in01:28
ianwmelwitt: so i see updates to 809020 correct, and i replayed an old one (787458) and it seemed to update.  so a priori i don't see anything wrong01:29
*** odyssey4me is now known as Guest729004:14
*** ysandeep|out is now known as ysandeep04:18
*** bhagyashris is now known as bhagyashris|off05:35
*** jpena|off is now known as jpena07:26
kopecmartinhi, doesn't properly renderer .rst tables - (rst syntax should be ok based on an online rst editor), should I file a bug somewhere?08:04
*** dviroel|out is now known as dviroel11:08
*** ykarel_ is now known as ykarel11:11
*** ykarel_ is now known as ykarel11:28
*** jpena is now known as jpena|lunch11:28
*** arxcruz is now known as arxcruz|pto11:55
*** ysandeep is now known as ysandeep|brb12:12
*** odyssey4me is now known as Guest733512:17
*** jpena|lunch is now known as jpena12:21
*** odyssey4me is now known as Guest733812:31
ttxIt's probably a gitea bug / limitation12:44
fungiwell, limitation in the rst renderer it's using anyway13:04
fungii'll see if i can jog my memory as to which one it is13:05
*** ysandeep|brb is now known as ysandeep13:08
fungikopecmartin: ttx: it's using `pandoc -f rst` according to
ttxthumbsup emoji13:16
*** ykarel is now known as ykarel|afk13:18
kopecmartinthanks, i'll try a few experiments, let's see if i can find a syntax which will be accepted by pandoc and will render a proper table13:20
fungikopecmartin: alternatively, if pandoc is just outright limited in its capability, we could fairly trivially substitute any other renderer which can be called in a similar fashion13:33
fungimodulo increases in resource consumption and so on, of course13:33
*** artom_ is now known as artom13:35
*** lukas is now known as Guest734813:43
*** ykarel|afk is now known as ykarel14:07
*** odyssey4me is now known as Guest735214:07
fungiclarkb: one more upcoming nail in the lists.o.o pv instance's coffin:
fungithat just popped up this morning when i was upgrading some of my systems14:13
fungioh, though i guess that's 32-bit only, not 64-bit, so *maybe* not relevant?14:15
fungiif anything, it's an indication that support for pv xen is falling away14:16
fungibit by bit14:16
clarkbya certainly seems like it isn't a platform anyone expects to really exist out there in the wild anymore14:40
melwittianw: thank you for checking the update_blueprint script! it appears that I wrongly assumed that bc I did not receive a notification from launchpad, that the blueprint did not get updated. sorry for that 😓14:58
fungimelwitt: well, thanks for helping us confirm it's actually working now! ;)14:59
melwittnp 😂14:59
clarkbfungi: did you catch my bits of research on using node exporter from ubuntu packages? tldr is that prior to the 1.0 reelase every release was changing metric names around which would probably get very clunky to manage with our heterogenous deployments15:03
clarkbfungi: I think if the packages were >=1.0 it wouldn't be a problem15:04
fungiclarkb: yep, that makes sense as a reason to go with the containerized approach if we were to decide on node_exporter. however with snmp_exporter i guess we don't need to worry about it since that gets installed centrally on the prometheus host, and snmp is already standardized since decades anyway15:14
*** ykarel is now known as ykarel|away15:20
*** ysandeep is now known as ysandeep|away15:23
opendevreviewLajos Katona proposed openstack/project-config master: Add neutron-dynamic-routing-stable-maint group
kopecmartinfungi: rst syntax is the culprit at the end, it turned out that pandoc is less forgiving than rst online tools , the issue was a multiline cell -
kopecmartinfungi: thank you15:58
*** marios is now known as marios|out16:01
fungikopecmartin: awesome, thanks for testing that out!16:05
fungikopecmartin: it looks like you had a stray + compared to the multi-row span cell examples at
fungicould that have been the culprit?16:10
fungiseems to me the + to the left of PTG should have been a | instead16:11
*** jpena is now known as jpena|off17:04
funginow that lists.o.o has been running on focal for a bit, the memory utilization actually looks a lot better than it was on xenial:
fungieven its swap utilization isn't all that bad by comparison:
clarkbthat might be the first time in history where running newer software used less memory :)17:20
fungiwell, it's still somewhat early, but looks promising at least17:23
clarkbinfra-root heads up there appears to be a lot of openstack release activity today. Probably a good day to be very slushy17:36
kopecmartinfungi: i thought so too, however, it didn't help - seems like pandoc can't handle multiline cells, it didn't process correctly even the example tables with multiline cells from the documentation17:36
fungiclarkb: good, i was feeling very slushy anyway17:37
fungikopecmartin: good to know. i wonder whether there's a bug open against pandoc to support that17:37
fungikopecmartin: indeed, web search for the win...
kopecmartinwell, luckily we don't really need multiline cells17:38
clarkbalso gitea probably shouldn't be considered your primary publication location17:39
kopecmartinclarkb: i was thinking about that .. any suggestions except for wiki pages and
fungikopecmartin: looks like pandoc's html writer supports rowspan and colspan as of 2.11 (almost a year ago) so we may not have new enough pandoc to support it anyway17:42
fungikopecmartin: we also have docs.opendev.org17:42
* kopecmartin checking that out17:42
clarkbfungi: our gitea images are still buster iirc but we have bullseye available now we could update too. I'm not sure if pandoc is new enough on bullseye though17:43
fungikopecmartin: but i think the board of directors is looking at having a cms of some sort to provide a web presence for foundation board level efforts, so that may also fit the interop wg17:43
clarkbI guess pushing up a change to see if everything runs on bullseye in testing is worthwhile. I'll do that17:43
fungiclarkb: looks like bullseye has pandoc 2.9 still, so no17:44
fungi2.11 isn't in debian at all yet17:44
clarkbok. Still worth making that change and seeing what breaks17:45
fungiabsolutely, just won't address the rst table rendering limitations17:45
fungithough as far as pandoc is concerned, it will take us from 2.2.1-3+b2 to
fungiso we may see some rendering improvements at least (or regressions!)17:47
opendevreviewClark Boylan proposed opendev/system-config master: Update our gitea images to bullseye
clarkbsomething like that I Think17:52
clarkbhrm that change is only building images and not running the tests against them. If images build successfully I guess I can add a noop edit to testinfra to run the tests17:53
fungiinteresting that we don't include the test jobs when the image configs change17:58
clarkboh wait we do run the job I'm just not capable of reading18:11
clarkbmy brain wants to treat the job list as ordered by dependencies and the system-config-run-gitea job is listed first18:11
clarkbthinking out loud here maybe we want to start writing changes like that in bulk so that we can work through issues and then start landing them all after the openstack release (and probably sooner for certain services)18:14
fungicorvus: looks like alembic 1.7 breaks gertty on startup with "AttributeError: module 'alembic' has no attribute 'migration'"18:24
fungirolling alembic back to 1.6.5, everything's gine18:24
fungii'll see if i can find a fix18:25
clarkbfungi: zuul had that issue, it is a missing import18:25
fungiahh, thanks for the hint!18:26
fungiindeed, zuul side fix was so i'll propose the same18:27
fungiyeah, that does the trick18:28
fungiunfortunately makes no mention of that18:31
opendevreviewJeremy Stanley proposed ttygroup/gertty master: Import alembic.migration
fungicorvus: ^ that got me working again, fyi18:33
clarkbfungi: ya the blame is more on the user side than the alembic side18:35
clarkbfungi: a name was being used that was present as an import side effect then they refactored and the side effect no longer happened18:35
fungiaha, well, overheating spacebar regressions and all, but if a module imports other modules into its namespace it's not surprising when consumers inadvertently treat them as objects declared within the module18:37
opendevreviewClark Boylan proposed opendev/system-config master: Build our gerrit images on Bullseye
clarkbIn that one we use python-builder on our side which is buster and openjdk:11 currently. openjdk:11 is buster I think but will in theory become bullseye in the future. By being explicit about it we ensure we don't end up with a msimatch between those images that are used.20:46
fungiyeah, bullseye has openjdk-11 packages as well20:48
clarkbyup they have images for bullseye it is just a matter of being explicit about it which the above chagne does20:48
*** dviroel is now known as dviroel|out20:48
fungithere doesn't seem to be an openjdk-12 yet, or at least not one carried in debian20:49
clarkbfungi: they just released 17 actually. 11 is a sort of defacto standard old version aiui and is the latest that gerrit supports20:49
clarkbI think 12 - 14 are EOL?20:49
clarkbI don't really keep up with that ecosystem anymore20:50
fungiaha, yeah there's also an openjdk-17 in nullseye20:53
clarkbya so whenever gerrit adds new java support we'll be ready :)20:54
clarkbsupposedly its a fair bit quicker than java 11 at certain types of GC tasks20:54
*** timburke_ is now known as timburke20:59
fungithat's be nice, assuming it translates to gerrit not getting as bogged down on garbage collection21:18
*** osmanlicilegi is now known as Guest021:20
*** prometheanfire is now known as Guest121:20
opendevreviewMarco Vaschetto proposed openstack/diskimage-builder master: Allowing use local image
*** promethe- is now known as prometheanfire21:53
clarkbBoth of the bullseye updates I pushed today pass testing which is good to see. But in both cases we don't really rely on the operating system for much so shouldn't be surprised either22:56
opendevreviewClark Boylan proposed opendev/system-config master: Set a gerrit replication timeout of 15 minutes
clarkbinfra-root ^ I think we should consider that as we appear to have some leaky replication tasks again23:10
clarkbI'm going to manually clean up the queue again but hopefully setting a timeout like that will make things happier23:10
ianwclarkb: am i missing something or does that need to also update to write the timeout value?23:22
clarkboh you're right. For some reason I thought it iterated over all the values in the dict and wrote them if present23:23
clarkbI'll push an update shortly23:23
opendevreviewClark Boylan proposed opendev/system-config master: Set a gerrit replication timeout of 15 minutes
clarkbwe'll need to restart gerrit to pick that up after it lands and applies23:26
clarkbI can't remember why we set autoReload to false anymore but there aws a good reason for it iirc23:27
ianwclarkb: i'm just reading now, should we maybe be setting the ssh timeouts in
ianwgerrit.sshConnectionTimeout : Timeout for SSH connections. If 0, there is no timeout and the client waits indefinitely. By default, 2 minutes23:31
ianwso we're clearly waiting longer than 2 minutes, which seems to mean23:31
ianwit must be in23:31
ianwgerrit.sshCommandTimeout : Timeout for SSH command execution. If 0, there is no timeout and the client waits indefinitely. By default, 0.23:31
clarkbya I kind of figured that the network timeout would cover both of those bases since in theory if there is no network transmission it would cover both of those?23:33
clarkbit isn't claer to me what ssh command execution covers. Is that the entire replication over ssh?23:33
clarkbor just establishing the ssh connection ?23:34
clarkbianw: I guess we could set sshCommandTimeout to 15 minutse as well?23:35
clarkbit shouldnt hurt to be at least as long as the network transmission timeout23:35
ianwgit grep sshCommandTimeout doesn't seem to give me anything23:38
clarkbianw: I'm going to context switch to the space x launch here, feel free to update the change if you can decipher the replication plugin better. Or I'll see if RTFSing helps23:39
ianwi'll have a poke and see if anything becomes clear, but in general ++ for setting something here23:39
clarkblooks like src/main/java/com/googlesource/gerrit/plugins/replication/ uses the ssh command timeout23:40
clarkbProcess proc = ssh.exec(cmd, commandTimeout);23:40
clarkbsshSessionFactoryProvider.get().getSession(uri, null, FS.DETECTED, connectionTimeout);23:41
clarkbso ya it seems connection timeout is the syn ack synack + maybe authorizing and all the to establish the connection then the command timeout is everything that happens after?23:41
ianwahh, of course so it's not in my gerrit source tree23:42
clarkbits in that repo if you clone that23:42
opendevreviewMarco Vaschetto proposed openstack/diskimage-builder master: Allowing use local image
opendevreviewMarco Vaschetto proposed openstack/diskimage-builder master: Allowing use local image
ianwyeah too much java to parse :)  i say go with the simple thing and work from there23:46
opendevreviewMarco Vaschetto proposed openstack/diskimage-builder master: Allowing use local image

Generated by 2.17.2 by Marius Gedminas - find it at!