| opendevreview | Merged zuul/zuul-jobs master: linters: lint that library files don't start with #! https://review.opendev.org/c/zuul/zuul-jobs/+/857981 | 00:13 |
|---|---|---|
| corvus | the jaeger tracing server change is passing its test now and is ready for review: https://review.opendev.org/855983 | 00:18 |
| corvus | it'd be cool if we can go ahead and merge that, since i think the first few changes in zuul are about ready to merge too. we can get that set up and evaluate things as we go | 00:19 |
| corvus | https://zuul.opendev.org/t/openstack/build/851349d3c22a495db95bc41f97be1099 is the successful build | 00:19 |
| fungi | lgtm | 00:28 |
| fungi | i don't know much about what jaeger itself needs configured, but the surrounding boilerplate for deployment and cert management looks like what i expect | 00:29 |
| fungi | and at least there's a test that it came up | 00:29 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make each compontent in 9-stream configurable https://review.opendev.org/c/zuul/zuul-jobs/+/857988 | 00:40 |
| ianw | fungi: ^ i've also dropped a comment in https://review.opendev.org/c/zuul/zuul-jobs/+/857730 for context on that | 00:52 |
| ianw | corvus: it might be nice to do an actual wget, and maybe even a screenshot ... with the cert setup it's as easy as cmd = host.run('curl https://tracing99.opendev.org') | 00:56 |
| ianw | test_paste.py would have examples; it's just a few lines but really does validate end-to-end working from apache->service | 00:57 |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 01:24 | |
| *** ysandeep|out is now known as ysandeep | 01:35 | |
| corvus | ianw: i guess we could perform a get on / and assume a 200 is okay. i don't think anything past that is worthwhile at this point -- i don't know what we would do to check it. to be honest, i think the current evaluation is sufficient for a brand new server... | 01:46 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add Jaeger tracing server https://review.opendev.org/c/opendev/system-config/+/855983 | 01:53 |
| corvus | ianw: fungi ^ that adds the curl | 01:53 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add Jaeger tracing server https://review.opendev.org/c/opendev/system-config/+/855983 | 02:21 |
| corvus | i guess we need to assert cmd.succeeded for that | 02:22 |
| corvus | that passes now | 02:52 |
| opendevreview | Ian Wienand proposed opendev/system-config master: run-selenium: save container logs https://review.opendev.org/c/opendev/system-config/+/857994 | 03:03 |
| *** ysandeep is now known as ysandeep|afk | 04:05 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run jobs with a focal bridge.openstack.org https://review.opendev.org/c/opendev/system-config/+/857799 | 04:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: testinfra: Update selenium calls https://review.opendev.org/c/opendev/system-config/+/858003 | 04:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: testinfra: Update selenium calls https://review.opendev.org/c/opendev/system-config/+/858003 | 05:19 |
| opendevreview | Ian Wienand proposed opendev/system-config master: afs-release: better info when can not get lockfile https://review.opendev.org/c/opendev/system-config/+/858009 | 05:27 |
| opendevreview | Ian Wienand proposed opendev/system-config master: afs-release: better info when can not get lockfile https://review.opendev.org/c/opendev/system-config/+/858009 | 05:37 |
| *** pojadhav|out is now known as pojadhav | 06:18 | |
| ianw | don't worry about reviewing https://review.opendev.org/q/topic:bridge-ansible-venv just yet. i think i have a handle on everything from installing ansible in a venv -> upgrading bridge.o.o to focal but i just need to bash a few more bits into shape. probably ready for review end of my monday | 06:32 |
| ianw | sorry -> upgrading bridge to jammy | 06:33 |
| ianw | the selenium issues weren't too bad -- ultimately it was just that it had tacitly pinned itself to a lower version of selenium as they must have dropped python3.6 support. so updating the python to 3.10 brought in a much later version that had of course removed apis etc. | 06:34 |
| opendevreview | Bin Yang proposed openstack/project-config master: Add oran-o2 app to StarlingX https://review.opendev.org/c/openstack/project-config/+/858011 | 06:34 |
| *** ysandeep|afk is now known as ysandeep | 06:37 | |
| *** jpena|off is now known as jpena | 06:38 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: afs-release: better info when can not get lockfile https://review.opendev.org/c/opendev/system-config/+/858009 | 06:44 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run jobs with a jammy bridge.openstack.org https://review.opendev.org/c/opendev/system-config/+/857799 | 06:44 |
| opendevreview | Ian Wienand proposed opendev/system-config master: testinfra: Update selenium calls https://review.opendev.org/c/opendev/system-config/+/858003 | 06:44 |
| opendevreview | Merged openstack/diskimage-builder master: added elrepo element https://review.opendev.org/c/openstack/diskimage-builder/+/853817 | 09:50 |
| *** frenzyfriday is now known as frenzyfriday|doc | 11:05 | |
| *** dviroel|out is now known as dviroel | 11:23 | |
| *** frenzyfriday|doc is now known as frenzyfriday | 11:56 | |
| *** dasm|off is now known as dasm | 11:57 | |
| opendevreview | Merged openstack/diskimage-builder master: changed release check logic in lvm element https://review.opendev.org/c/openstack/diskimage-builder/+/854566 | 12:30 |
| opendevreview | Merged zuul/zuul-jobs master: configure-mirrors: make each compontent in 9-stream configurable https://review.opendev.org/c/zuul/zuul-jobs/+/857988 | 14:39 |
| *** ysandeep is now known as ysandeep|away | 15:02 | |
| *** dviroel is now known as dviroel|lunch | 15:07 | |
| clarkb | fungi: I just responded to an old thread on openstack-discuss and it made me wonder if mm3 would handle that properly. It must since the mailing lists use header tag data to track that stuff? Do you think it is worth testing explicitly? | 15:12 |
| clarkb | infra-root today is the day we said we'd try to test the meetpad jvb colibri update, but the change doesn't have any reviews yet https://review.opendev.org/c/opendev/system-config/+/856553 | 15:14 |
| clarkb | Would be great if I could get some feedback even if it means I need to work on new patchsets today :) | 15:14 |
| fungi | clarkb: we can test, but it should handle it just fine. i don't see why it shouldn't at least | 15:15 |
| clarkb | fungi: ya seems like a use case they would've considered very important right? | 15:15 |
| fungi | yes. the archive import, from what i understand, makes the imported messages essentially the same as if they'd been handled by mm3 directly when originally sent | 15:17 |
| Guest681 | fungi: clarkb can I get some eyes on this when you have a moment? https://review.opendev.org/c/opendev/system-config/+/847213 | 15:18 |
| clarkb | Guest681: yes, I'm accumulating things to review. Need to do a local reboot and breakfast then will dive into things | 15:19 |
| Guest681 | Sounds good. Thanks for adding it to your queue! | 15:19 |
| Guest681 | Enjoy breakfast. Tell the girls I say hi :D | 15:19 |
| clarkb | will do, thanks! | 15:20 |
| *** marios is now known as marios|out | 15:30 | |
| *** jpena is now known as jpena|off | 15:34 | |
| opendevreview | Clark Boylan proposed openstack/project-config master: Bump Zuul tenant default ansible version to 6 https://review.opendev.org/c/openstack/project-config/+/858200 | 15:59 |
| clarkb | corvus: ^ fyi that was on my list | 15:59 |
| *** dviroel|lunch is now known as dviroel | 16:07 | |
| fungi | i'm approving 847213 since there don't seem to be any meetings in progress or coming up soon | 16:16 |
| corvus | clarkb: thx, i think we can approve that whenever | 16:16 |
| fungi | and done | 16:17 |
| opendevreview | Merged openstack/project-config master: Bump Zuul tenant default ansible version to 6 https://review.opendev.org/c/openstack/project-config/+/858200 | 16:24 |
| opendevreview | Merged opendev/system-config master: Setup #openinfra-envirosig IRC Channel https://review.opendev.org/c/opendev/system-config/+/847213 | 16:42 |
| *** Guest681 is now known as diablo_rojo | 17:19 | |
| *** dviroel is now known as dviroel|afk | 17:50 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 17:54 | |
| clarkb | fungi: did yo ustill want to try and do https://review.opendev.org/c/opendev/system-config/+/856553 for meetpad today? | 18:15 |
| clarkb | also re mm3 testing, maybe pipermail next now that it seems we've largely got hte migration down? | 18:15 |
| fungi | oh, yep! let's do meetpad | 18:19 |
| fungi | and agreed, now that we have a clean migration (sans a few overly large text fields in old openstack lists we should decide what to do with), testing mysqldump and the pipermail redirects are next on the agenda | 18:20 |
| fungi | for the three templates which errored about being too wide for their columns, i'd probably preemptively fix them up in mm2 before migrating the openstack site | 18:21 |
| fungi | but i'm open to other suggestions | 18:21 |
| clarkb | that seems reasonable. Updating upstream mm3 to have wider fields is probably a lot more effort than necessary for something like that | 18:22 |
| clarkb | one thing to keep in mind with the meetpad change is what while a revert should work for returning the state to working should something fo wrong. It won'y remove the new files we are managing. They will only avoid affecting the system due to lack of bind mounts I think | 18:23 |
| clarkb | basically don't forget to clean those up by hand if we do revert | 18:23 |
| clarkb | But also starting with code review would be good, I'm not entirely sure I got everything correct there | 18:24 |
| clarkb | fungi: also we need to add the keystore passwd to prod group vars before we approve it | 18:28 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 18:30 | |
| *** dviroel|afk is now known as dviroel | 18:33 | |
| fungi | oh, right. as in generate a keystore passwd and add it on bridge | 18:38 |
| fungi | reviewing now | 18:38 |
| clarkb | right, since a keystore without a password didn't seem to be an option | 18:39 |
| fungi | clarkb: looks like i already left a comment there. can you update the jvb.conf with a comment indicating where it came from, similar to what i did in the recent update to other files like meet.conf? | 19:03 |
| fungi | looks like meetpad_jvb_keystore_password is the thing we need to generate and add on bridge | 19:03 |
| clarkb | fungi: I can, but I don't see that comment on the change | 19:04 |
| clarkb | I think the reason I did not add a comment is I hvae no idea what the comment format is for that file | 19:04 |
| fungi | oh, sorry, looks like i forgot to save that review, the comment was still in draft | 19:05 |
| clarkb | random googleing seems to indicate C style comments | 19:06 |
| fungi | for whatever reason, meet.conf had existing comments prefixed by # so i stuck with that | 19:07 |
| clarkb | they are different formats fo rthe different config files | 19:07 |
| clarkb | and they don't really explain them anywhere :/ | 19:07 |
| fungi | jvb.conf seems to use the same syntax, just doesn't have any comments | 19:07 |
| fungi | maybe they're in a different language, but it doesn't seem like it | 19:08 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update colibri for all the JVBs https://review.opendev.org/c/opendev/system-config/+/856553 | 19:10 |
| clarkb | fungi: it is. meet.conf is an nginx config file. JVB is a java jetty config I think | 19:10 |
| clarkb | https://github.com/jitsi/jitsi-videobridge/blob/master/CONFIG.md shows C style comments at least | 19:11 |
| fungi | eep, so different configs in different languages in the same directory with the same file extension? | 19:12 |
| clarkb | they are in different directories in the end result on the server | 19:12 |
| clarkb | but otherwise yes | 19:12 |
| clarkb | we can double check the service logs on that update to ensure I didn't break anything doing that | 19:13 |
| fungi | thanks! mainly i just don't want anyone to have to spend as much time as i did when refreshing all the configs recently | 19:23 |
| fungi | because information about when and where the prior versions came from was not clearly recorded | 19:24 |
| clarkb | https://zuul.opendev.org/t/openstack/build/91516c3bf46d4a61a0c9331d0417a027/log/meetpad01.opendev.org/docker/jitsi-meet-docker_jvb_1.txt and https://zuul.opendev.org/t/openstack/build/91516c3bf46d4a61a0c9331d0417a027/log/jvb01.opendev.org/docker/jitsi-meet-docker_jvb_1.txt dont show any obvious signs of distress | 20:03 |
| clarkb | fungi: if that looks good did you want to single core approve it? | 20:03 |
| clarkb | and if so who wants to edit the group vars? | 20:03 |
| fungi | i just added a meetpad_jvb_keystore_password entry with a long random string to group_vars/jvb.yaml and group_vars/meetpad.yaml on bridge | 20:06 |
| fungi | and approved the change | 20:06 |
| fungi | clarkb: should i go ahead and take the jvb servers out of the emergency disable list so they'll get deployed to? | 20:07 |
| clarkb | fungi: sure, though I think this is testable with the all in onealone too | 20:07 |
| fungi | oh, good point | 20:07 |
| fungi | though we'll want to also test with a standalone jvb and the jvb container on the aio server stopped to make sure communication is set up correctly, right? | 20:08 |
| clarkb | basically if it doesn't work on meetpad all in one the others won't work either. But if the all in one works then we still want to test the extra jvbs | 20:08 |
| clarkb | yes exactly | 20:08 |
| fungi | so probably would be good to let it deploy to them initially to save time, or avoid deploying to them for now and save ourselves some cleanup in case we revert? | 20:09 |
| fungi | i'm on the fence | 20:09 |
| clarkb | I think that is the tradeoff. And ya me too. Maybe leave them out for now. If all in one works then we can be reasonably confident the jvbs will too and pick this up next week | 20:10 |
| fungi | wfm | 20:10 |
| *** dviroel is now known as dviroel|pto | 20:37 | |
| opendevreview | Merged opendev/system-config master: Update colibri for all the JVBs https://review.opendev.org/c/opendev/system-config/+/856553 | 20:54 |
| fungi | watching for the deploy to complete | 21:03 |
| *** tosky is now known as Guest765 | 21:33 | |
| *** tosky_ is now known as tosky | 21:33 | |
| clarkb | looks like it restarted the jvb but not web. I think we need web to restart too in order to update the https config on that side | 21:33 |
| clarkb | once the job finishes we can manually restart it and then test /me needs to relocate back to desk with headset | 21:34 |
| fungi | did the deploy job for it finish? i haven't seen the buildset report yet, but maybe there's more jobs to go | 21:35 |
| fungi | oh, job's still running, okay | 21:35 |
| clarkb | it hasn't finished yet and there is one more job in the buildest | 21:35 |
| fungi | got it | 21:35 |
| clarkb | ok it has finished now. Let me relocate and we can manually down then up the whole cluste rto pick up that nginx config update in web too | 21:35 |
| fungi | but it's past the point where it would have restarted the web container i guess? so we probably need to update the file list | 21:35 |
| clarkb | ya | 21:37 |
| fungi | so want me to down and up all the containers on meetpad.o.o? | 21:38 |
| clarkb | I just did it | 21:38 |
| clarkb | but yes that is what I did docker compose down then up -d | 21:38 |
| fungi | cool, thanks! | 21:38 |
| clarkb | https://meetpad.opendev.org/isitbroken when ready | 21:39 |
| fungi | i can hear you | 21:41 |
| fungi | i don't have my camera plugged in | 21:41 |
| clarkb | you're muted | 21:41 |
| clarkb | but ya my laptop camera isn't working | 21:42 |
| fungi | it shows me unmuted | 21:42 |
| fungi | i seem to have gotten my window manager into an odd state too, just a sec | 21:42 |
| clarkb | fungi: huh it shows you as muted on my side | 21:42 |
| fungi | i'm still trying to diagnose the sudden problem with my window manager | 21:45 |
| clarkb | fungi: its back up and workign now | 22:28 |
| *** dasm is now known as dasm|of | 22:43 | |
| *** dasm|of is now known as dasm|off | 22:43 | |
| *** dasm|off is now known as Guest772 | 22:44 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix jitsi meet jvb connection info and cert CN https://review.opendev.org/c/opendev/system-config/+/858224 | 22:44 |
| clarkb | I've manually hacked ^ into the server in prod | 22:48 |
| clarkb | that seems to get us working video with tls between nginx and the jvb websocket | 22:48 |
| clarkb | I expect that tonight zuul and ansible will undo a bunch of that which should only break video at worst on the meetpad server and we can restore that monday by landing ^ if it looks good | 22:49 |
| clarkb | there were two issues we ran into. The first is that dns lookups in the nginx process in the web container don't seem to work so we couldn't proxy to meetpad01.opendev.org. We addressed that by switching to the ip address and then ran into SNI problems. The SNI problems went away by changing the CN in the keystore cert that is generated | 22:51 |
| clarkb | But once those two things were addressed video started working | 22:52 |
| fungi | third (minor) problem was updating the configs triggered a restart of the jvb container but not the web container | 22:53 |
| fungi | though i'm not immediately seeing where in our ansible we restart the jvb containers either | 22:59 |
| clarkb | fungi: I think it may be automagic because the env vars updated | 23:00 |
| clarkb | docker compose knows how to dela with that | 23:00 |
| clarkb | basically docker compose sees that a variable has chagned that only jvb uses and so it restarts the jvb | 23:00 |
| clarkb | maybe that means we should refactor the meet.conf change I made to be env var toggled | 23:01 |
| clarkb | Though once it is working that bit of the config is unlikely to ever change again | 23:03 |
| clarkb | fungi: supposedly nginx won't look at /etc/hosts only a real resolver. But we have one of those too that should resolve the name so I don't know what is going on. I'm going to stop looking though as ip addresses here are fine | 23:22 |
| clarkb | https://zuul.opendev.org/t/openstack/build/817122144e244d088c05265e3abe0f52/log/meetpad01.opendev.org/jitsi-meet/jvb/jvb.conf#46 and https://86b0b24a7c436641b992-d94eb819944fd5bc8105d713aef77d0b.ssl.cf5.rackcdn.com/858224/1/check/system-config-run-meetpad/8171221/bridge.openstack.org/ara-report/results/453.html look good for putting the ip address where we want it to go | 23:23 |
| clarkb | I think we can go ahead and move the old keystore aside and land that on monday and haev it deploy to the jvbs as well | 23:24 |
| fungi | yep, sounds great. thanks for the work on that! | 23:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!