Friday, 2022-09-16

opendevreviewMerged zuul/zuul-jobs master: linters: lint that library files don't start with #!
corvusthe jaeger tracing server change is passing its test now and is ready for review:
corvusit'd be cool if we can go ahead and merge that, since i think the first few changes in zuul are about ready to merge too.  we can get that set up and evaluate things as we go00:19
corvus is the successful build00:19
fungii don't know much about what jaeger itself needs configured, but the surrounding boilerplate for deployment and cert management looks like what i expect00:29
fungiand at least there's a test that it came up00:29
opendevreviewIan Wienand proposed zuul/zuul-jobs master: configure-mirrors: make each compontent in 9-stream configurable
ianwfungi: ^ i've also dropped a comment in for context on that00:52
ianwcorvus: it might be nice to do an actual wget, and maybe even a screenshot ... with the cert setup it's as easy as cmd ='curl')00:56 would have examples; it's just a few lines but really does validate end-to-end working from apache->service00:57
*** rlandy|ruck|bbl is now known as rlandy|ruck01:24
*** ysandeep|out is now known as ysandeep01:35
corvusianw: i guess we could perform a get on / and assume a 200 is okay.  i don't think anything past that is worthwhile at this point -- i don't know what we would do to check it.  to be honest, i think the current evaluation is sufficient for a brand new server...01:46
opendevreviewJames E. Blair proposed opendev/system-config master: Add Jaeger tracing server
corvusianw: fungi ^ that adds the curl01:53
opendevreviewJames E. Blair proposed opendev/system-config master: Add Jaeger tracing server
corvusi guess we need to assert cmd.succeeded for that02:22
corvusthat passes now02:52
opendevreviewIan Wienand proposed opendev/system-config master: run-selenium: save container logs
*** ysandeep is now known as ysandeep|afk04:05
opendevreviewIan Wienand proposed opendev/system-config master: Run jobs with a focal
opendevreviewIan Wienand proposed opendev/system-config master: testinfra: Update selenium calls
opendevreviewIan Wienand proposed opendev/system-config master: testinfra: Update selenium calls
opendevreviewIan Wienand proposed opendev/system-config master: afs-release: better info when can not get lockfile
opendevreviewIan Wienand proposed opendev/system-config master: afs-release: better info when can not get lockfile
*** pojadhav|out is now known as pojadhav06:18
ianwdon't worry about reviewing just yet.  i think i have a handle on everything from installing ansible in a venv -> upgrading bridge.o.o to focal but i just need to bash a few more bits into shape.  probably ready for review end of my monday06:32
ianwsorry -> upgrading bridge to jammy06:33
ianwthe selenium issues weren't too bad -- ultimately it was just that it had tacitly pinned itself to a lower version of selenium as they must have dropped python3.6 support.  so updating the python to 3.10 brought in a much later version that had of course removed apis etc.06:34
opendevreviewBin Yang proposed openstack/project-config master: Add oran-o2 app to StarlingX
*** ysandeep|afk is now known as ysandeep06:37
*** jpena|off is now known as jpena06:38
opendevreviewIan Wienand proposed opendev/system-config master: afs-release: better info when can not get lockfile
opendevreviewIan Wienand proposed opendev/system-config master: Run jobs with a jammy
opendevreviewIan Wienand proposed opendev/system-config master: testinfra: Update selenium calls
opendevreviewMerged openstack/diskimage-builder master: added elrepo element
*** frenzyfriday is now known as frenzyfriday|doc11:05
*** dviroel|out is now known as dviroel11:23
*** frenzyfriday|doc is now known as frenzyfriday11:56
*** dasm|off is now known as dasm11:57
opendevreviewMerged openstack/diskimage-builder master: changed release check logic in lvm element
opendevreviewMerged zuul/zuul-jobs master: configure-mirrors: make each compontent in 9-stream configurable
*** ysandeep is now known as ysandeep|away15:02
*** dviroel is now known as dviroel|lunch15:07
clarkbfungi: I just responded to an old thread on openstack-discuss and it made me wonder if mm3 would handle that properly. It must since the mailing lists use header tag data to track that stuff? Do you think it is worth testing explicitly?15:12
clarkbinfra-root today is the day we said we'd try to test the meetpad jvb colibri update, but the change doesn't have any reviews yet
clarkbWould be great if I could get some feedback even if it means I need to work on new patchsets today :)15:14
fungiclarkb: we can test, but it should handle it just fine. i don't see why it shouldn't at least15:15
clarkbfungi: ya seems like a use case they would've considered very important right?15:15
fungiyes. the archive import, from what i understand, makes the imported messages essentially the same as if they'd been handled by mm3 directly when originally sent15:17
Guest681fungi: clarkb can I get some eyes on this when you have a moment?
clarkbGuest681: yes, I'm accumulating things to review. Need to do a local reboot and breakfast then will dive into things15:19
Guest681Sounds good. Thanks for adding it to your queue!15:19
Guest681Enjoy breakfast. Tell the girls I say hi :D15:19
clarkbwill do, thanks!15:20
*** marios is now known as marios|out15:30
*** jpena is now known as jpena|off15:34
opendevreviewClark Boylan proposed openstack/project-config master: Bump Zuul tenant default ansible version to 6
clarkbcorvus: ^ fyi that was on my list15:59
*** dviroel|lunch is now known as dviroel16:07
fungii'm approving 847213 since there don't seem to be any meetings in progress or coming up soon16:16
corvusclarkb: thx, i think we can approve that whenever16:16
fungiand done16:17
opendevreviewMerged openstack/project-config master: Bump Zuul tenant default ansible version to 6
opendevreviewMerged opendev/system-config master: Setup #openinfra-envirosig IRC Channel
*** Guest681 is now known as diablo_rojo17:19
*** dviroel is now known as dviroel|afk17:50
*** rlandy|ruck is now known as rlandy|ruck|mtg17:54
clarkbfungi: did yo ustill want to try and do for meetpad today?18:15
clarkbalso re mm3 testing, maybe pipermail next now that it seems we've largely got hte migration down?18:15
fungioh, yep! let's do meetpad18:19
fungiand agreed, now that we have a clean migration (sans a few overly large text fields in old openstack lists we should decide what to do with), testing mysqldump and the pipermail redirects are next on the agenda18:20
fungifor the three templates which errored about being too wide for their columns, i'd probably preemptively fix them up in mm2 before migrating the openstack site18:21
fungibut i'm open to other suggestions18:21
clarkbthat seems reasonable. Updating upstream mm3 to have wider fields is probably a lot more effort than necessary for something like that18:22
clarkbone thing to keep in mind with the meetpad change is what while a revert should work for returning the state to working should something fo wrong. It won'y remove the new files we are managing. They will only avoid affecting the system due to lack of bind mounts I think18:23
clarkbbasically don't forget to clean those up by hand if we do revert18:23
clarkbBut also starting with code review would be good, I'm not entirely sure I got everything correct there18:24
clarkbfungi: also we need to add the keystore passwd to prod group vars before we approve it18:28
*** rlandy|ruck|mtg is now known as rlandy|ruck18:30
*** dviroel|afk is now known as dviroel18:33
fungioh, right. as in generate a keystore passwd and add it on bridge18:38
fungireviewing now18:38
clarkbright, since a keystore without a password didn't seem to be an option18:39
fungiclarkb: looks like i already left a comment there. can you update the jvb.conf with a comment indicating where it came from, similar to what i did in the recent update to other files like meet.conf?19:03
fungilooks like meetpad_jvb_keystore_password is the thing we need to generate and add on bridge19:03
clarkbfungi: I can, but I don't see that comment on the change19:04
clarkbI think the reason I did not add a comment is I hvae no idea what the comment format is for that file19:04
fungioh, sorry, looks like i forgot to save that review, the comment was still in draft19:05
clarkbrandom googleing seems to indicate C style comments19:06
fungifor whatever reason, meet.conf had existing comments prefixed by # so i stuck with that19:07
clarkbthey are different formats fo rthe different config files19:07
clarkband they don't really explain them anywhere :/19:07
fungijvb.conf seems to use the same syntax, just doesn't have any comments19:07
fungimaybe they're in a different language, but it doesn't seem like it19:08
opendevreviewClark Boylan proposed opendev/system-config master: Update colibri for all the JVBs
clarkbfungi: it is. meet.conf is an nginx config file. JVB is a java jetty config I think19:10
clarkb shows C style comments at least19:11
fungieep, so different configs in different languages in the same directory with the same file extension?19:12
clarkbthey are in different directories in the end result on the server19:12
clarkbbut otherwise yes19:12
clarkbwe can double check the service logs on that update to ensure I didn't break anything doing that19:13
fungithanks! mainly i just don't want anyone to have to spend as much time as i did when refreshing all the configs recently19:23
fungibecause information about when and where the prior versions came from was not clearly recorded19:24
clarkb and dont show any obvious signs of distress20:03
clarkbfungi:  if that looks good did you want to single core approve it?20:03
clarkband if so who wants to edit the group vars?20:03
fungii just added a meetpad_jvb_keystore_password entry with a long random string to group_vars/jvb.yaml and group_vars/meetpad.yaml on bridge20:06
fungiand approved the change20:06
fungiclarkb: should i go ahead and take the jvb servers out of the emergency disable list so they'll get deployed to?20:07
clarkbfungi: sure, though I think this is testable with the all in onealone too20:07
fungioh, good point20:07
fungithough we'll want to also test with a standalone jvb and the jvb container on the aio server stopped to make sure communication is set up correctly, right?20:08
clarkbbasically if it doesn't work on meetpad all in one the others won't work either. But if the all in one works then we still want to test the extra jvbs20:08
clarkbyes exactly20:08
fungiso probably would be good to let it deploy to them initially to save time, or avoid deploying to them for now and save ourselves some cleanup in case we revert?20:09
fungii'm on the fence20:09
clarkbI think that is the tradeoff. And ya me too. Maybe leave them out for now. If all in one works then we can be reasonably confident the jvbs will too and pick this up next week20:10
*** dviroel is now known as dviroel|pto20:37
opendevreviewMerged opendev/system-config master: Update colibri for all the JVBs
fungiwatching for the deploy to complete21:03
*** tosky is now known as Guest76521:33
*** tosky_ is now known as tosky21:33
clarkblooks like it restarted the jvb but not web. I think we need web to restart too in order to update the https config on that side21:33
clarkbonce the job finishes we can manually restart it and then test /me needs to relocate back to desk with headset21:34
fungidid the deploy job for it finish? i haven't seen the buildset report yet, but maybe there's more jobs to go21:35
fungioh, job's still running, okay21:35
clarkbit hasn't finished yet and there is one more job in the buildest21:35
fungigot it21:35
clarkbok it has finished now. Let me relocate and we can manually down then up the whole cluste rto pick up that nginx config update in web too21:35
fungibut it's past the point where it would have restarted the web container i guess? so we probably need to update the file list21:35
fungiso want me to down and up all the containers on meetpad.o.o?21:38
clarkbI just did it21:38
clarkbbut yes that is what I did docker compose down then up -d21:38
fungicool, thanks!21:38
clarkb when ready21:39
fungii can hear you21:41
fungii don't have my camera plugged in21:41
clarkbyou're muted21:41
clarkbbut ya my laptop camera isn't working21:42
fungiit shows me unmuted21:42
fungii seem to have gotten my window manager into an odd state too, just a sec21:42
clarkbfungi: huh it shows you as muted on my side21:42
fungii'm still trying to diagnose the sudden problem with my window manager21:45
clarkbfungi: its back up and workign now22:28
*** dasm is now known as dasm|of22:43
*** dasm|of is now known as dasm|off22:43
*** dasm|off is now known as Guest77222:44
opendevreviewClark Boylan proposed opendev/system-config master: Fix jitsi meet jvb connection info and cert CN
clarkbI've manually hacked ^ into the server in prod22:48
clarkbthat seems to get us working video with tls between nginx and the jvb websocket22:48
clarkbI expect that tonight zuul and ansible will undo a bunch of that which should only break video at worst on the meetpad server and we can restore that monday by landing ^ if it looks good22:49
clarkbthere were two issues we ran into. The first is that dns lookups in the nginx process in the web container don't seem to work so we couldn't proxy to We addressed that by switching to the ip address and then ran into SNI problems. The SNI problems went away by changing the CN in the keystore cert that is generated22:51
clarkbBut once those two things were addressed video started working22:52
fungithird (minor) problem was updating the configs triggered a restart of the jvb container but not the web container22:53
fungithough i'm not immediately seeing where in our ansible we restart the jvb containers either22:59
clarkbfungi: I think it may be automagic because the env vars updated23:00
clarkbdocker compose knows how to dela with that23:00
clarkbbasically docker compose sees that a variable has chagned that only jvb uses and so it restarts the jvb23:00
clarkbmaybe that means we should refactor the meet.conf change I made to be env var toggled23:01
clarkbThough once it is working that bit of the config is unlikely to ever change again23:03
clarkbfungi: supposedly nginx won't look at /etc/hosts only a real resolver. But we have one of those too that should resolve the name so I don't know what is going on. I'm going to stop looking though as ip addresses here are fine23:22
clarkb and look good for putting the ip address where we want it to go23:23
clarkbI think we can go ahead and move the old keystore aside and land that on monday and haev it deploy to the jvbs as well23:24
fungiyep, sounds great. thanks for the work on that!23:42

Generated by 2.17.3 by Marius Gedminas - find it at!