| opendevreview | Clark Boylan proposed opendev/system-config master: Update Etherpad to 2.3.0 https://review.opendev.org/c/opendev/system-config/+/946842 | 00:19 |
|---|---|---|
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Etherpad to 2.3.0 https://review.opendev.org/c/opendev/system-config/+/946842 | 00:40 |
| *** ykarel_ is now known as ykarel | 06:00 | |
| opendevreview | JaromÃr Wysoglad proposed openstack/project-config master: Add jobs for new Aetos repository https://review.opendev.org/c/openstack/project-config/+/946742 | 07:25 |
| opendevreview | Merged openstack/project-config master: Add jobs for new Aetos repository https://review.opendev.org/c/openstack/project-config/+/946742 | 12:34 |
| fungi | /opt/backups-202010 on backup02.ca-ymq-1.vexxhost has reached 91% utilization, so i've initiated a prune in a root screen session just now | 13:33 |
| *** noonedeadpunk_ is now known as noonedeadpunk | 13:35 | |
| *** dxld_ is now known as dxld | 13:36 | |
| *** frickler_ is now known as frickler | 13:48 | |
| *** sean-k-mooney1 is now known as sean-k-mooney | 14:17 | |
| fungi | #status log Pruned backups on backup02.ca-ymq-1.vexxhost reducing volume usage from 91% to 63% | 14:26 |
| opendevstatus | fungi: finished logging | 14:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Etherpad to 2.3.0 https://review.opendev.org/c/opendev/system-config/+/946842 | 14:50 |
| clarkb | screenshots do seem to show adding the = suffix on vars unsets our initial default value | 14:50 |
| clarkb | hopefully screenshots show that latest patchset corrects the problem | 14:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add new Noble review03 to the inventory https://review.opendev.org/c/opendev/system-config/+/946637 | 14:55 |
| clarkb | I added review03 to the backup group in ^ but otherwise the change is the same. I think today is a decent day to land that and monitor nothing unexpected happens then we have tomorrow and/or early next week to start testing the move. | 14:56 |
| clarkb | Probably worth sending an announcement tomorrow with a plan to swap servers a week later and if we don't make that time frame oh well we can update teh announcement | 14:56 |
| clarkb | fungi: re the release notes -37 quirk zuul addressed that by labelling those as "In Development" set via the :unreleased-version-title: value to the reno release-notes sphinx extension | 15:06 |
| clarkb | https://codesearch.opendev.org/?q=unreleased-version-title&i=nope&literal=nope&files=&excludeFiles=&repos= maybe openstack should consider adopting that | 15:07 |
| * JayF sciencing this with ironic release notes | 15:10 | |
| JayF | Hm, I either did it wrong or misunderstood twhat that means | 15:14 |
| clarkb | JayF: https://zuul-ci.org/docs/zuul/latest/releasenotes.html#in-development it should do that instead of 11.3.0-37 or whatever | 15:15 |
| JayF | I assume this isn't enough, is what I mean https://www.irccloud.com/pastebin/abdeczED/ | 15:16 |
| JayF | nothing changed in the built output | 15:16 |
| clarkb | you set it when you call the extension. My codesearch link above has examples | 15:16 |
| JayF | OH | 15:17 |
| clarkb | there may be a way to use conf.py too but that isn't how anyone else has done it | 15:17 |
| JayF | I'm sorry, I probably don't misdo this if not PTG brained :| | 15:17 |
| JayF | I ran out of brain sometime midday Wednesday ;) | 15:17 |
| gouthamr | woah meetpad recordings can be longer than 60 mins? :) pleasant change! | 15:18 |
| clarkb | gouthamr: I think its limited by your browser's max file size for writing | 15:19 |
| clarkb | the estimate is 60 minutes to be on the safe side | 15:19 |
| gouthamr | oh, i never knew that.. | 15:19 |
| fungi | clarkb: JayF: yeah, we do the same thing in bindep too: https://opendev.org/opendev/bindep/raw/branch/master/doc/source/releasenotes.rst | 15:19 |
| clarkb | fungi: ya codesearch found a number of examples. reno does it for itself too | 15:19 |
| fungi | gouthamr: right, starting a new recording hourly 1. helps you exclude the dead air for hourly breaks, and 2. avoids having the recording end unexpectedly when it reaches your browser's limit | 15:20 |
| gouthamr | fungi: yes totally, i forgot to end a recording, and it kept running so was pleasantly surprised | 15:26 |
| clarkb | gouthamr: I want to say the limit is 1gb but I haevn't found any documentation of that from chrome directly. That means you could theoretically watch the file size and wait for it to near that limit before stopping | 15:30 |
| clarkb | but again I am not 100% certain of that limit | 15:30 |
| clarkb | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aee/openstack/aee2dc3b790c4c53b88d434786c23bb3/bridge99.opendev.org/screenshots/etherpad-pad.png success | 15:34 |
| clarkb | we should probably still hold a node and sanity check things work as expected with chat and editor colors and so on | 15:35 |
| clarkb | and all of tha should wait for the ptg to complete | 15:35 |
| JayF | thank you! it looks much better, will propose it to Ironic https://usercontent.irccloud-cdn.com/file/IMxIU3Ia/image.png | 15:41 |
| clarkb | https://docs.openstack.org/releasenotes/ironic/unreleased.html for comparison | 15:44 |
| clarkb | https://docs.openstack.org/releasenotes/ironic/2024.2.html or maybe that is better comparison since it shows a more confusing case of unreleased code having a version | 15:45 |
| clarkb | your old anchors may break too but I think that is fine as every new commit breaks the anchors anyway so this is an improvement that makes them stable | 15:45 |
| clarkb | ok https://review.opendev.org/c/opendev/system-config/+/946637 passes testing after adding review03 to the backup servers list | 15:50 |
| opendevreview | Tim Burke proposed opendev/irc-meetings master: swift: Fix meeting cadence https://review.opendev.org/c/opendev/irc-meetings/+/946938 | 15:57 |
| opendevreview | Merged opendev/irc-meetings master: swift: Fix meeting cadence https://review.opendev.org/c/opendev/irc-meetings/+/946938 | 16:23 |
| clarkb | fungi: do you think you might have time to look at the review03 inventory change? ianw weighed in on the older patchset and said it looked fine and the new patchset was a minimal change | 17:46 |
| clarkb | if you think it is mergeable I expect ot be around all day today and can help monitor | 17:48 |
| fungi | checking | 17:50 |
| clarkb | the discussion on https://etherpad.opendev.org/p/i_vt63v18c3RKX2VyCs3 is related too | 17:51 |
| clarkb | essentially the idea is we add a new gerrit server that should get configured but not turn on any services | 17:53 |
| clarkb | then we can synchronize things between servers and turn things on for testing. Then we can schedule a downtmie to do an actual switch synchronization and migrate servers | 17:54 |
| fungi | is there a pending change for post-cutover things like reenabling reeplication? i don't see it as a todo step in the pad either | 17:54 |
| clarkb | no not yet | 17:55 |
| fungi | i assume most other stuff will be handled by moving it from review-staging to review hostgroup | 17:55 |
| clarkb | all review-staging changes is the manage-projects playbook | 17:56 |
| clarkb | that playbook excludes the review-staging group | 17:56 |
| clarkb | but yes one of the things we will need to do as part of the actual move is configure replication on the new server | 17:56 |
| clarkb | we don't configure it now because we don't want the new server accidentally overwriting git content on giteas | 17:57 |
| fungi | so where are things like gerrit init/startup and git cg cron disabled for it? | 17:57 |
| fungi | s/cg/gc/ | 17:57 |
| clarkb | the gerrit role disables gerrit init and startup by default | 17:57 |
| clarkb | the gc cron won't be disabled I don't think | 17:57 |
| fungi | oh, okay so not something explicit here | 17:57 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gerrit/defaults/main.yaml#L5-L8 | 17:58 |
| fungi | oh, never mind, the commit message mentioned gerrit reindexing and my mind substituted git gc | 17:58 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/946637/3/inventory/service/host_vars/review03.opendev.org.yaml is a copy from review02 with the cert names updated and replication removed | 17:59 |
| clarkb | then we're relying on defaults otherwise | 17:59 |
| clarkb | though it just occurred to me before we land that we need to make sure secrets on bridge are set up for it | 17:59 |
| fungi | so the things the commit message says it doesn't do are things the gerrit role already doesn't do | 17:59 |
| clarkb | which I even have as a pre step in the ehteprad. I'm going to go look at that now | 17:59 |
| clarkb | fungi: yes, unless explicitly enabled | 18:00 |
| clarkb | we do use those features for the test job and maybe when we rename repos with the playbook. I'm not sure about renaming repos | 18:00 |
| fungi | that explains why they didn't seem to be explicitly disabled (other than the empty replication config) | 18:00 |
| clarkb | I put it in the commti message to call out that is the expected behavior. But reviewers should double check that expectation is carried through by the ansible | 18:03 |
| clarkb | looks like we use host specific vars for secrets rather than group vars | 18:04 |
| clarkb | looking at the existing vars for the old server I think we want to just make a straight port over | 18:05 |
| clarkb | so I'll do that now | 18:05 |
| clarkb | we have host keys lsited for the old gitea servers. I will clear those out. There is also some old info in there for github replication that we shouldn't need and openstack watch but I think cleaning that up should happen separately to this effrot. Enough moving parts already | 18:06 |
| clarkb | ok secret vars should be in place now. I also checked that review02 doesn't show up as a string in any of those vars that would need updating (it doesn't) | 18:10 |
| clarkb | fungi: looks like you +2'd I guess if you'r comfortable with those assertions about how this is safe and the secret vars check out to you we can probably proceed? | 18:12 |
| fungi | yes | 18:13 |
| fungi | i didn't know whether you were waiting for more feedback from anyone else | 18:13 |
| clarkb | I mean it is the sort of thing that more feedback on is always nice to have | 18:14 |
| clarkb | but I'm also concerned that if I wait for more I'll be waiting for a long time | 18:14 |
| fungi | i'm fine with moving ahead so we can make progress sooner | 18:14 |
| clarkb | and ianw did review the change earlier | 18:14 |
| clarkb | called out https://review.opendev.org/c/opendev/system-config/+/783183 as a similar one the last time we did this | 18:15 |
| clarkb | I do notice the host vars in ^ had a lot more content than I added for review03 | 18:15 |
| clarkb | let me track down where those are now set | 18:15 |
| clarkb | I think the bulk of them migrated into https://opendev.org/opendev/system-config/src/branch/master/inventory/service/group_vars/review.yaml and gerrit_database_type is not h2 we use mariadb | 18:16 |
| clarkb | so that explains that on | 18:17 |
| fungi | good improvement, we've centralized the stuff that can be safelty | 18:17 |
| fungi | safely | 18:17 |
| clarkb | ya | 18:17 |
| clarkb | lunch is fast approaching. We can sit on it until then in case any other brain light bulbs go off or anyone else reviews it then approve after | 18:17 |
| fungi | sgtm | 18:18 |
| clarkb | after it applies I think the main things to check are that the service is actually down, that it didn't magically conjure a replication config and that if/when manage-projects runs it ignores this server as expected. If it doesn't ignore the server then we'll have manage-projects running from both review02 and review03 against review02 | 18:18 |
| clarkb | I suspect 90% of the time that would be fine (updating acls or nooping daily). The main issues would be creating new projects | 18:19 |
| clarkb | so maybe we add it to inventory and avoid creating new projects until afte we've confirmed review03 is ignored as expected | 18:19 |
| clarkb | and I'm more than happy for people to poke holes in this approach/plan | 18:20 |
| clarkb | I'd rather get it right and be prepared then rush in and deal with unexpected issues | 18:20 |
| clarkb | side note the servier id uuid value must match between the wto servers for gerrit to treat the notedb content from the old server as valid | 18:20 |
| clarkb | I guess this way you can move a repo into a new gerrit and leave its old notedb content behind if you don't want to port the changes in? | 18:21 |
| clarkb | fungi: I see now the comment in the groups file about review-staging says it disables replication, but I can't find any evidence of it doing that | 18:30 |
| clarkb | the only thing I can see it doing is excludingservers in review-staging from running manage-projects | 18:30 |
| fungi | maybe it used to | 18:37 |
| fungi | like in the pre-plugin days or something | 18:38 |
| clarkb | possible | 18:38 |
| clarkb | I think the way to do that would be to check if we are in the review-staging group and use that to write out or not the replication config? | 18:38 |
| clarkb | anyway writing an empty list of replication targets should be sufficient | 18:39 |
| fungi | that would be the most sensible approach | 18:39 |
| fungi | but this also works | 18:39 |
| fungi | and yeah, i confirm that the only reference to the review-staging group is its exclusion for tasts in the manage-projects playbook | 18:40 |
| fungi | so i don't see any way that part of the comment can be correct | 18:41 |
| clarkb | or maybe that was the intention then was never implemented as the empty list approach should be sufficient | 18:41 |
| fungi | appears that comment was added with https://review.opendev.org/c/opendev/system-config/+/780698 in 2021 which only implemented the bits we have now | 18:42 |
| clarkb | ok I'll pop out for lunch in a few minutes. Then we can approve at 19:45 UTC probably if there are no other new concerns | 18:45 |
| clarkb | infra-root ^ fyi that is the plan for adding review03 to the inventory | 18:46 |
| fungi | perfect | 18:46 |
| clarkb | raise concerns now if you have them :) | 18:46 |
| clarkb | as an extra sanity check i quickly checked if gerrit_replication is defined in secret vars (it is not). Also it looks like we will write out the replication config file but it should only contain the top level options and no replication targets | 19:28 |
| clarkb | so that all continues to look good to me | 19:28 |
| clarkb | fungi: I'm done eating a little early. Happy for that to be approved now if we're ready. We have a bit of time until it merges too if anything else comes up. | 19:30 |
| fungi | approved | 19:31 |
| clarkb | shoudl finish up right around 1900 UTC | 19:38 |
| fungi | i've got a summit session to lead starting at 2100 utc, but that's plenty of time to help check stuff | 19:39 |
| clarkb | especially with our much quicker runtimes | 19:39 |
| fungi | so very, very nice... | 19:40 |
| fungi | oh, actually when you said "should finish up right around 1900" it was already 19:39 utc | 19:55 |
| fungi | guessing you meant 20:00 | 19:55 |
| clarkb | oh yes I did | 19:56 |
| fungi | but yeah, it's on track to merge in about another 5 minutes per zuul's estimate. that'll put it behind the hourly jobs, so deploy is unlikely to happen until closer to 20:20 i'm gussing | 19:57 |
| fungi | guessing | 19:57 |
| clarkb | hourlies sjust queued up so this will go after them | 20:01 |
| opendevreview | Merged opendev/system-config master: Add new Noble review03 to the inventory https://review.opendev.org/c/opendev/system-config/+/946637 | 20:08 |
| clarkb | hourlies are just finishing up too so that should start almost immediately | 20:08 |
| clarkb | yup there it goes | 20:09 |
| clarkb | service-review and manage-projects are near the end | 20:09 |
| clarkb | so far backups have been configured and there is an LE cert | 20:35 |
| clarkb | the job to actually configure it as a gerrit just started | 20:35 |
| fungi | yeah, and infra-prod-manage-projects is going now | 20:39 |
| fungi | which should skip the server | 20:39 |
| clarkb | ya we need to look at logs on bridge to confirm | 20:39 |
| clarkb | fwiw review03 is running apache but not the backend services according to docker ps -a | 20:39 |
| clarkb | so I think review03 did as expected | 20:40 |
| fungi | and if manage-projects tries to run on it, we should see the job fail anyway | 20:40 |
| fungi | as the api isn't up | 20:40 |
| clarkb | fungi: no, I noted this in the etherpad but manage-projects is configured to talk to review.o.o not localhost | 20:41 |
| fungi | ah, right, so it would connect from 03 to 03 | 20:41 |
| fungi | er, to 02 | 20:41 |
| clarkb | so it would talk to review02 which is potentially problematic if we have review02 and review03 both running manage-projcst at the same time. I think it is ok in the noop case and maybe even in the update acls case | 20:41 |
| clarkb | the create a new project case is the one that worries me, but this run should noop and we can confirm it is skipped entirely in the safe noop case | 20:42 |
| clarkb | manage projects log on bridge lgtm | 20:42 |
| clarkb | there are tasks for review02 but not for review03 and the play summary doesn't even include review03 | 20:43 |
| clarkb | everything I've seen so far looks good to me | 20:43 |
| clarkb | the next step will be syncing data and potentially starting up gerrit to test (I need to think about how safe that is if we resync later. I think it is fine if we overwrite on the target which is scary but something we probably have to do anyway) | 20:44 |
| clarkb | maybe tomorrow we can start piecing together the sync | 20:44 |
| clarkb | fungi: before we call this done it would be good if you can double check the manage project log just to confirm | 20:44 |
| fungi | yeah, i have it open already | 20:45 |
| fungi | the play recap shows it ran against the 6 gitea servers and review02, but no mention of review03 | 20:46 |
| fungi | so bridge didn't contact the new server for that | 20:47 |
| clarkb | that was my read too. searching for review03 doesn't show anything | 20:47 |
| clarkb | and if you check the timestamp on the file its from 20:42 which is new enough to have review03 in the inventory (just calling that out as old logs would not have review03 in them as the server didn't exist then) | 20:47 |
| fungi | agreed | 20:48 |
| fungi | well, i mean, i was also tailing it as the job ran | 20:48 |
| clarkb | another good way to check that :) | 20:48 |
| clarkb | so ya I think we're in good shape to finish modling this new server into the new review | 20:49 |
| clarkb | but you've got a ptg session in 10 minutes so I'm good with pausing here and picking it up again tomorrow | 20:49 |
| clarkb | in the meantime I'm going to work on holding a test etherpad server for that etherpad upgrade | 20:49 |
| fungi | cool, i'll be running a ptg session starting in a few minutes | 20:51 |
| fungi | but can probably take a look at etherpad upgrade test results in a little over an hour | 20:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 20:51 |
| clarkb | and maybe we upgrade etherpad tomorrow after the ptg. Removing meetpad servers from the emergency file will alos upgrade meetpad when the ptg is over | 20:52 |
| fungi | yeah, that'd be great | 20:54 |
| clarkb | the held etherpad test server will be at 213.32.76.191 | 21:24 |
| clarkb | timburke: was the swift epoxy release the first one without python2.7 support? | 21:39 |
| fungi | i half expected there to already be content in the clarkb-test pad on the held node | 22:12 |
| fungi | https://etherpad.opendev.org/p/testing has surprising content | 22:12 |
| fungi | oh, i bet that's something we inject via testinfra | 22:13 |
| fungi | based on the timestamps in the timeslider | 22:14 |
| fungi | bingo: https://opendev.org/opendev/system-config/src/branch/master/testinfra/test_etherpad.py#L58 | 22:14 |
| clarkb | yes it is | 22:15 |
| clarkb | sorry I'm doing a small errand but then will test the held node too | 22:15 |
| fungi | no worries, take all the time you need | 22:15 |
| fungi | i just wanted to poke at it before my evening gets in full swing | 22:15 |
| clarkb | are you using the testing pad then? | 22:18 |
| fungi | https://etherpad.opendev.org/p/fungi | 22:18 |
| fungi | ordered lists aren't incrementing | 22:19 |
| fungi | can you try an ordered list test too, make sure it's not something up with my browser? | 22:20 |
| clarkb | sure | 22:21 |
| clarkb | I think it works but I used incognito tabs to avoid caching problems since we're hijacking /etc/hosts | 22:22 |
| clarkb | I shutdown the crhome tab. I think this is working | 22:24 |
| fungi | okay, that's weird, when i followed an unordered list with an ordered list it resumed an earlier ordered list test | 22:24 |
| clarkb | i wonder if the existing server does that | 22:25 |
| clarkb | I can confirm I get the same behavior | 22:25 |
| clarkb | let me jump off the test server and check if this happens on the existing server | 22:26 |
| clarkb | https://etherpad.opendev.org/p/opendev-server-replacement-sprint bottom of that pad reproduces the behavior so this isn't a new bug | 22:27 |
| fungi | yep, good enough by me then | 22:28 |
| clarkb | I'll take it if thats the worst issue we can find :) | 22:28 |
| fungi | so i think we're still on track to upgrade after the ptg | 22:28 |
| fungi | no obvious blockers | 22:29 |
| clarkb | ++ | 22:29 |
| clarkb | jitsi meet made a new release not too long ago too. We would've updated twice | 22:29 |
| clarkb | I'm glad we put the hosts in the don't touch for now list | 22:29 |
| fungi | yep | 22:30 |
| fungi | clarkb: i reproduced the other ordered list behavior i saw on the version we're currently running too: https://etherpad.opendev.org/p/testing | 22:36 |
| fungi | i think it wants to continue ordered lists which are interrupted by an unordered list, but when there is no prior ordered list present it gets... confused? maybe? | 22:37 |
| clarkb | you think its the same issue in both cases? | 22:45 |
| clarkb | fungi: https://github.com/ether/etherpad-lite/issues/5160 | 22:50 |
| clarkb | looks like john has had good weather :) | 22:51 |
| clarkb | that is the first issue. Not claer to me if it is the same issue as numbers continuing on though. Probably are related at least | 22:51 |
| fungi | yeah, my guess is the two are related | 23:02 |
| fungi | looks like he implemented a testcase, but no fix | 23:04 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
