| ianw | as a bonus, it seems that comments in tickets don't work in firefox | 00:15 |
|---|---|---|
| opendevreview | Merged opendev/system-config master: Fix docker wait requires at least one argument https://review.opendev.org/c/opendev/system-config/+/856341 | 00:20 |
| fungi | ianw: in rackspace tickets? it's worked for me | 00:21 |
| ianw | fungi: https://imgur.com/a/0eiYNf1 is what i see :/ | 00:22 |
| ianw | anyway, i have filed a ticket on the translate db upgrade because it seems stuck | 00:23 |
| fungi | i have had things like that get stuck in the past. seems like they end up in scheduling deadlocks | 00:24 |
| ianw | it says it has "1 replica" but I can't see the replica | 00:24 |
| ianw | so i can't detach that replica, and so the main db is stuck waiting for a replica to detach that i can't access, afaict | 00:25 |
| clarkb | I've restarted the manual zuul_reboot run and it seems happy | 00:26 |
| clarkb | it seems to be doing what I expect now | 00:27 |
| clarkb | ze01 did its reboot and started back up again and now ze02 stop ran and we are waiting as expected | 00:28 |
| clarkb | and that is running in screen with typical log location | 00:28 |
| ianw | the translate db is still responding; despite it's half-upgraded state in the UI. so i guess i just need to wait for a response on the ticket | 00:41 |
| opendevreview | Merged zuul/zuul-jobs master: upload-npm : support authToken argument https://review.opendev.org/c/zuul/zuul-jobs/+/856225 | 01:13 |
| *** ysandeep|out is now known as ysandeep | 02:27 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: upload-npm: use token auth https://review.opendev.org/c/openstack/project-config/+/856365 | 02:58 |
| ianw | ok; rax issue turns out to be what they think is a UI bug in the website; and navigating directly i've managed to detach the replica. i've updated the connection strings in hiera, run puppet and rebooted both servers. so that basically completes that | 03:15 |
| ianw | #status log translate and translate-dev RAX hosted mysql instances upgraded to mysql 5.7 | 03:16 |
| opendevstatus | ianw: finished logging | 03:16 |
| ianw | i'll remove the old instances -- the upgrade notice says to make sure to do this to avoid their script doing it for us | 03:17 |
| Clark[m] | ianw do we need to update hiera config somewhere to keep puppet doing the right thing? | 03:29 |
| ianw | Clark[m]: yep, i've commited those changes on bridge and did a manual run to deploy the updated connect strings on the servers | 03:30 |
| Clark[m] | Excellent thanks | 03:30 |
| opendevreview | Merged openstack/project-config master: upload-npm: use token auth https://review.opendev.org/c/openstack/project-config/+/856365 | 03:31 |
| *** ysandeep is now known as ysandeep|afk | 03:41 | |
| *** ysandeep|afk is now known as ysandeep | 06:18 | |
| *** jpena|off is now known as jpena | 07:42 | |
| *** pojadhav is now known as pojadhav|afk | 08:08 | |
| *** frenzy_friday is now known as frenzyfriday|afk | 09:09 | |
| *** pojadhav|afk is now known as pojadhav | 09:29 | |
| *** frenzyfriday|afk is now known as frenzyfriday | 09:54 | |
| *** rlandy|out is now known as rlandy | 10:33 | |
| *** dasm|off is now known as dasm | 12:56 | |
| slittle1_ | sorry, internet dropped just after I reported my problem | 13:23 |
| slittle1_ | Can't push a signed tag to ssh://review.opendev.org:29418/starlingx/rook-ceph.git ssh://review.opendev.org:29418/starlingx/test.git | 13:24 |
| slittle1_ | the tag was created with git tag -s, and verified with git tag -v | 13:24 |
| slittle1_ | acl ... | 13:25 |
| slittle1_ | [access "refs/tags/*"] | 13:25 |
| slittle1_ | createSignedTag = group starlingx-release | 13:25 |
| slittle1_ | i'm a member of starlingx-release | 13:25 |
| fungi | slittle1_: what error message do you get when pushing that? | 13:59 |
| fungi | is it possible you're not pushing with your username? i don't see any username in the remotes you quoted, so openssh is likely assuming your local username is valid on the gerrit server (and i don't know whether they match in your case) | 14:00 |
| slittle1_ | git push gerrit 7.0.0 | 14:08 |
| slittle1_ | Enumerating objects: 1, done. | 14:08 |
| slittle1_ | Counting objects: 100% (1/1), done. | 14:08 |
| slittle1_ | Writing objects: 100% (1/1), 814 bytes | 814.00 KiB/s, done. | 14:08 |
| slittle1_ | Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 | 14:08 |
| slittle1_ | remote: error: branch refs/tags/7.0.0: | 14:08 |
| slittle1_ | remote: use a SHA1 visible to you, or get update permission on the ref | 14:08 |
| slittle1_ | remote: User: slittle1 | 14:08 |
| slittle1_ | remote: Contact an administrator to fix the permissions | 14:08 |
| slittle1_ | remote: Processing changes: refs: 1, done | 14:08 |
| slittle1_ | To ssh://review.opendev.org:29418/starlingx/rook-ceph.git | 14:08 |
| slittle1_ | ! [remote rejected] 7.0.0 -> 7.0.0 (prohibited by Gerrit: update for creating new commit object not permitted) | 14:08 |
| slittle1_ | error: failed to push some refs to 'ssh://review.opendev.org:29418/starlingx/rook-ceph.git' | 14:08 |
| fungi | slittle1_: are you tagging a commit which doesn't exist on gerrit, or has a commit in its history which isn't already in the public repository? that's suggesting you have | 14:09 |
| fungi | maybe you ended up with some dirty merges in the local repository state? | 14:10 |
| fungi | what's the commit id (sha-1 hash) of the commit you tagged as 7.0.0? | 14:12 |
| slittle1_ | 99ecd76b8b78d3720398227741a93e8356f9e9a9 | 14:12 |
| slittle1_ | looks loike review 851132 is outstanding | 14:12 |
| fungi | assuming your intent was to tag the current branch tip of the r/stx.7.0 branch, it looks like that would be 16466c2d84d97a3f285a9a7ccd3e8e9144176a02 instead: https://opendev.org/starlingx/rook-ceph/commits/branch/r/stx.7.0 | 14:13 |
| slittle1_ | zuul barfed on https://review.opendev.org/c/starlingx/rook-ceph/+/851132, which I would like to include un the tagging | 14:15 |
| slittle1_ | Zuul: This change is unable to merge due to a missing merge requirement. | 14:16 |
| slittle1_ | Oh: Removed Workflow+1 by chen haochuan | 14:17 |
| fungi | yes, that would count as a merge requirement | 14:18 |
| fungi | the change no longer had sufficient votes to merge when zuul finished testing it | 14:18 |
| *** bhagyashris is now known as bhagyashris|ruck | 14:37 | |
| opendevreview | Steven Webster proposed openstack/project-config master: Add STS-Silicom app to StarlingX https://review.opendev.org/c/openstack/project-config/+/856533 | 14:50 |
| *** ysandeep is now known as ysandeep|out | 14:54 | |
| *** dviroel is now known as dviroel|lunch | 14:57 | |
| *** dviroel_ is now known as dviroel | 14:57 | |
| *** pojadhav is now known as pojadhav|out | 15:07 | |
| clarkb | the manual run of zuul_reboot is on to ze10 now. Seems to have run happily overnight. Once it gets to the mergers we can confirm that all the changes made are good. | 15:15 |
| clarkb | slittle1_: fungi: it is important that you always update your local copy of the remote branch and tag off of that to ensure you don't end up pushing things you shouldn't. It appears new Gerrit is more selective about that now than in the past which is great, but this has caused problems in the past with pushing code that was never intended for Gerrit and otherwise pushing bad | 15:24 |
| clarkb | releases | 15:24 |
| clarkb | For example even if that change was merged, if it merged with a merge commit then tagging the change commit itself wuold represent an improper state | 15:25 |
| clarkb | you'd need to tag the resulting merge commit to include other changes that are mixed into the HEAD of that branch | 15:25 |
| fungi | agreed, that's why the documentation i linked yesterday includes a git pull --ff-only as its first step | 15:27 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update python builder and base image https://review.opendev.org/c/opendev/system-config/+/856537 | 15:34 |
| *** dviroel is now known as dviroel_ | 15:46 | |
| *** dviroel|lunch is now known as dviroel | 15:46 | |
| *** gibi is now known as gibi_off | 16:16 | |
| *** jpena is now known as jpena|off | 16:33 | |
| *** ykarel is now known as ykarel|away | 16:37 | |
| clarkb | fungi: I started looking at the jvb thing more closely and found https://community.jitsi.org/t/jvb-websockets-and-ssl/87463 I ran up against the need to figure out what the key-store-path should point to but tldr there seems to be no one knows ow to set this up :/ | 16:55 |
| fungi | er, um... wow! | 16:55 |
| clarkb | I'm beginning to wonder if that actually means do ssl client cert auth and use the keystore to check it? | 16:57 |
| clarkb | rather than do normal tls connection verification for privacy purposes | 16:57 |
| clarkb | https://community.jitsi.org/t/activating-public-interface-for-colibri-stats/95993/6 this shows someone generating a keystore | 16:58 |
| clarkb | so I'm now looking at openssl docs | 16:59 |
| clarkb | oh wait keytool is a java thing | 17:01 |
| clarkb | I half wonder if this is just a simple way to do unverified encyption via tls. Nothing I can find in docs or forums indicates you need a shared CA across these things so maybe that is all it is doing | 17:04 |
| clarkb | However, it is nginx thati s proxying these connections I would've expected nginx to verify the connection | 17:04 |
| fungi | not necessarily. like in haproxy we have the option to tell it to care about the ssl cert validity or not | 17:05 |
| fungi | nginx may similarly not care about validity of backend certs when proxying unless told explicitly that it should | 17:06 |
| clarkb | good point | 17:06 |
| clarkb | https://docs.oracle.com/cd/E35976_01/server.740/es_admin/src/tadm_ssl_convert_pem_to_jks.html apperas to be how you convert our LE provisioned pem to a java keystore file | 17:06 |
| clarkb | of course it isn't a single command or something that can be run by default without user interaction :/ | 17:07 |
| fungi | so i guess that's basically just to get the jetty serving your le cert? | 17:07 |
| clarkb | ya. Alternatively we could just generate a random cert | 17:08 |
| clarkb | and load that in. That would be easier to automate. Then see if that functions | 17:08 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Update colibri for all the JVBs https://review.opendev.org/c/opendev/system-config/+/856553 | 17:20 |
| clarkb | fungi: ^ I did everything but the java keystore stuff in that change. Firewall rules, setting WS_SERVER_ID, copying in the new jvb.conf and mounting it into the container and so on. | 17:20 |
| fungi | awesome, curious to see how it tests out | 17:23 |
| fungi | once i'm done making lunch, assuming no new fires, i can work on mm3 imports | 17:24 |
| clarkb | I suspect it won't actually fail because we don't check if jvb functionality is working? Unless the container startup itself breaks due to my config edits | 17:24 |
| clarkb | fungi: sounds good. | 17:24 |
| clarkb | I'm going to work on getting a bike ride in now before it gets hot | 17:24 |
| fungi | have fun! | 17:25 |
| fungi | okay, 6 of 7 production mailing lists (including kata's) are now copied onto the held 173.231.255.72 node via rsync; the openstack site is the last and currently underway, but will take a while due to its sheer size | 19:21 |
| fungi | once that's done i'll fiddle with more test imports, but in the meantime i'm going to pop out to get some vaccines. shouldn't hopefully take too long | 19:22 |
| *** rlandy is now known as rlandy|mtg | 19:30 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update python builder and base image https://review.opendev.org/c/opendev/system-config/+/856537 | 20:01 |
| *** rlandy|mtg is now known as rlandy | 20:31 | |
| fungi | managed to get vaccines and come back before the rsync finished | 20:41 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update colibri for all the JVBs https://review.opendev.org/c/opendev/system-config/+/856553 | 20:49 |
| clarkb | Thats a bit of a "I wonder if this works" | 20:50 |
| clarkb | I'm using the java keytool command to generate a random keystore | 20:50 |
| clarkb | the zuul reboot playbook just started on the mergers | 20:50 |
| clarkb | and now zuul01 is restarting | 20:58 |
| clarkb | and now zuul02 | 21:04 |
| clarkb | I expect that this playbook will run happily over the weekend based on this run being happy | 21:04 |
| clarkb | the zuul reboot playbook has completed successfully. I've closed the screen session. Logs are in the normal log file | 21:09 |
| *** dasm is now known as dasm|off | 21:13 | |
| fungi | excellent | 21:15 |
| clarkb | infra-root any idea what might be happening here: https://zuul.opendev.org/t/openstack/build/5996ddc0bb8e4b1eaaba92df124490f2/log/job-output.txt#1189-1191 it seems like docker is hanging up? All of those image builds are timing out | 21:46 |
| fungi | for the record, `du -sh /var/lib/mailman/import` reports 27G after copying all the current mm2 sites into it | 21:47 |
| clarkb | It looks like the RUN commands that happen after disabling recommends is an `apt-get update && apt-get install -y dumb-init && apt-get clean` | 21:49 |
| clarkb | I wonder if the apt-get install -y dumb-init is blocking on some expected input from the user | 21:49 |
| clarkb | running docker build locally against that dockerfile does not have any issues | 21:51 |
| clarkb | I've rechecked and if things stall out again I'll ssh into the test node to look | 21:52 |
| fungi | the manpage suggests -y should entirely sidestep interaction | 21:53 |
| *** dviroel is now known as dviroel|out | 21:57 | |
| ianw | clarkb: the linux/arm64 would be the bit i guess ... that's binary emulation | 21:58 |
| clarkb | ianw: oh! | 21:59 |
| fungi | ooh | 21:59 |
| clarkb | in the past that has been quick enough with file copies, but maybe we're doing some extra work here for some reason | 22:00 |
| ianw | probably worth a re-run just incase it was a particularly slow node, but yeah something could have changed that we're suddenly building something etc. | 22:00 |
| clarkb | ya I already rechecked and have ssh'd into one of the nodes to take a look | 22:00 |
| clarkb | if it is the emulation I should see a cpu sitting at like 100% all the time doing that I expect | 22:00 |
| clarkb | I think I see some of the issue. For amd64 we don't need to upgrade pip it is already up to date but on arm we are a few releases behind? | 22:03 |
| clarkb | oh maybe not I built python3.10 locally and maybe they update pip to match some python release version stuff | 22:03 |
| *** rlandy is now known as rlandy|bbl | 22:04 | |
| clarkb | ya both amd and arm updated pip on python3.8 based images | 22:04 |
| fungi | stripping out the "mailman" lists which we won't migrate, we have 48 total currently we could import across all 7 sites | 22:10 |
| clarkb | this recheck is looking happy so ya maybe just a slow node with the arm emulation | 22:12 |
| clarkb | still no libc update for bullseye :/ | 22:13 |
| clarkb | fungi: and this is on the newer held node right? | 22:22 |
| fungi | yep | 22:22 |
| clarkb | I've just found it and I see your session. I need to clean up my old gitea gravatar testing hold. And I guess once we've got working imports on this newer node we should delete the old hold for mm3 | 22:23 |
| clarkb | ianw: can I delete your dib rocky 9 hold? I think that got sorted | 22:24 |
| ianw | oh yes, please do | 22:25 |
| clarkb | done. | 22:25 |
| clarkb | frickler: you have a couple of holds too. The older of which I'm pretty sure can be deleted too | 22:26 |
| clarkb | https://zuul.opendev.org/t/openstack/build/d83a759b5a0a4923ba760bf329fc6f07/log/job-output.txt#16402-16403 heres another unrelated change that had a similar timout. I think in the same cloud region | 22:29 |
| clarkb | I guess keep an eye out to see if this is a persistent issue that we need to debug | 22:29 |
| clarkb | half worried it could be a zuul/ansible thing | 22:30 |
| clarkb | currently insufficient evidence to say that though | 22:31 |
| fungi | i started a scripted loop import of the opendev.org site a couple of minutes ago in a screen session on the held node, logging a full transcript to file, with timings for each command | 22:42 |
| fungi | the source of the trivial shell script is in https://etherpad.opendev.org/p/mm3migration | 22:43 |
| fungi | and it's already done | 22:46 |
| fungi | so around 5 minutes to import all the opendev site lists | 22:47 |
| clarkb | fungi: for the pipermail redirects I don't thnk I could use the exact path you had in the old comment due to needing to address things properly by site. If you look at the apache config the path should be clera though | 22:47 |
| clarkb | and I guess if you have a suggestion for being more like what you had in mind I'm open to it I just couldn't figure it out with mod rewrite | 22:48 |
| fungi | https://paste.opendev.org/show/816657 is the log of the import | 22:48 |
| fungi | ick, the percent counter is rather annoying that way | 22:49 |
| fungi | strange that for the computing-force-network ml archive import it warned thus: not importing messages older than Aug. 25, 2022. Use "--since=<date>" to import older messages. | 23:00 |
| clarkb | should we add a --since for 1900 just to be extra sure they'll all go on all lists? | 23:02 |
| fungi | though it does say go on to say "Indexing 3 emails" which is the total number in the imported mbox file, with dates of 2022-08-09, 2022-08-19 and 2022-08-25 respectively | 23:02 |
| fungi | also, none of the other archive imports spewed such a warning | 23:03 |
| fungi | just the first one | 23:03 |
| fungi | anyway, i'm starting a trial import of all of lists.openstack.org now | 23:10 |
| fungi | oh, interesting, i think some of these are getting skipped for not being in our initial config so we don't have lists created for them to migrate into | 23:11 |
| fungi | Error: No such list: openstack-announce@lists.openstack.org | 23:12 |
| fungi | et cetera | 23:13 |
| clarkb | hrm I dropped them all out of the config because I didn't want to have valid working lists for things on the server until we are ready ot migrate the | 23:13 |
| clarkb | which meant I dropped all but opendev out | 23:13 |
| fungi | oh, got it | 23:13 |
| clarkb | fungi: should I undo that? I think we can create them all but mark them as private | 23:14 |
| clarkb | and then when we want to migrate mark them public as appropriate and migrate into them | 23:14 |
| fungi | well, doesn't matter for now. i think this will still import the archives just not the configs | 23:15 |
| clarkb | got it | 23:15 |
| fungi | so should still be a good test of the db fix | 23:15 |
| clarkb | I'll look into creating them as a locked down list tomorrow then. I'm pretty sur eit is possible and sound slike that will make migrating easier | 23:15 |
| fungi | it's currently working on importing the openstack-discuss archive | 23:16 |
| clarkb | on the colibri ws change I seem to be hanging on the keytool generation. Going to run it on the test node in the foreground to see if it is asking for some info | 23:18 |
| clarkb | bah it wants name and org and location info | 23:19 |
| fungi | gah | 23:19 |
| fungi | it's 2020. why can tools not be run unattended/headless? | 23:19 |
| fungi | er, 2022 even | 23:19 |
| fungi | but still feels like 2020 isn't over yet | 23:20 |
| fungi | openstack-discuss archive is 1/3 imported and haven't seen that db error yet | 23:20 |
| fungi | can't recall how far in it errored last time | 23:20 |
| clarkb | ya its crazy to me they have a bunch of command line flags but none for these values | 23:23 |
| clarkb | I've got input sorted out for the commnd now .Just need to figure out how to plumb it through ansible | 23:25 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update colibri for all the JVBs https://review.opendev.org/c/opendev/system-config/+/856553 | 23:27 |
| clarkb | maybe something like that | 23:27 |
| ianw | clarkb: https://review.opendev.org/c/zuul/zuul/+/855309 is one i posted last week in the ongoing saga of console files on the static hosts. it is a host/playbook var to disable the console stream -- i came around to this thinking it might be a better generic solution than us just running a cron purge. i think we can use it like : https://review.opendev.org/c/opendev/system-config/+/855472 | 23:30 |
| fungi | clarkb: yep, no db packet size error this time | 23:31 |
| clarkb | fungi: excellent, we need to check the mysqldumps too | 23:32 |
| clarkb | but that is 50% of the problem solved :) | 23:33 |
| clarkb | ya I like that approach and will attempt a review tomorrow morning | 23:37 |
| fungi | shortly i should have a total archive import time estimate for the openstack.org site, as a rough upper-bound for outage windows | 23:43 |
| fungi | oof, we assumed importing openstack-discuss was a good benchmark, we should have been testing imports of openstack-stable-maint | 23:53 |
| fungi | in recent years that's been averaging over 100 messages a day because of all the broken bitrot jobs | 23:54 |
| clarkb | fungi: wow | 23:54 |
| clarkb | maybe we should remove the smtp reporter from zuul | 23:55 |
| fungi | at the moment it's around 120 messages a day (and more on weekends) | 23:55 |
| clarkb | if people aren't looking at it and all it does is fill our mailing list disks | 23:55 |
| fungi | i've brought it up with elodilles before but it seemed like he wanted to keep getting them | 23:55 |
| fungi | those messages don't end up with random attachments though, so while it probably uses more index it doesn't have the raw message sizes openstack-discuss does | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!