| tonyb | FWIW I have a "Ubuntu Noble OpenDev 20240724" on all clouds+regions now. I'm going to run launch-node (again for all clouds+regions) to verify it works. Once that's done I'll add some docs for next time. | 00:25 |
|---|---|---|
| tonyb | I ended up using the cloud ansible module to do the uploads as I figured it'd know the ins-and-outs of glance etc to d the right thing | 00:26 |
| fungi | thanks tonyb!!! | 00:41 |
| fungi | fwiw, there's not that many options you need to worry about with `openstack image create ...` either, but whatever's easiest | 00:42 |
| tonyb | fungi: I/clarkb were concerned that `openstack image create` didn't know how to handle the swift upload and import task needed by some clouds. So I looked for other options (hence the ansible) | 00:48 |
| fungi | i've used it for uploading into rackspace, ovh and vexxhost (i have personal accounts on all 3 and prefer to upload my own debian/testing images) | 00:57 |
| tonyb | Oh cool. Good to know. | 00:57 |
| Clark[m] | fungi: oh neat so it does know how to do the multi step process through swift? | 01:14 |
| fungi | seems to just do it magically | 01:19 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [base/unbound] Setup the initial trust anchor https://review.opendev.org/c/opendev/system-config/+/925311 | 02:54 |
| gtema | I was yesterday caught by one unpleasant situation where my job in codegenerator started to fail in gate. Analysis pointed me to the glance and here is the issue: according to https://opendev.org/openstack/glance/commits/branch/master latest change in glance is 5 days ago, but according to gerrit change that affected me was merged exactly between my check-gate (https://review.opendev.org/c/openstack/glance/+/881940). Is this desired that we do | 09:32 |
| gtema | not see when the commit landed in the master and instead when it was created? | 09:32 |
| frickler | I'm not sure git even supports tracking that data unless we would change the merge-strategy to always create a merge commit? but iiuc it was decided not to do the latter in order to keep the git history simple if possible | 09:52 |
| gtema | it's interesting that sometimes I see explicit merge commit, but sometimes it is not | 10:10 |
| gtema | if you look at the glance commits history you will see merge commits (i.e. Merge "Revert "Fix import job to provide valid disk-formats list to tempest"") | 10:12 |
| tonyb | gtema: Yeah. I think it's if the parent of a chnage is the curret HEAD you don't need a merge commit | 10:12 |
| gtema | but those "add location api" commits are not having that | 10:12 |
| tonyb | Correct that's why they have the date of the original commit | 10:12 |
| gtema | that's all pretty confusing - you see the commit but you do cannot reliably say when it landed in the master | 10:13 |
| tonyb | gtema: I expect it's in the blob somewhere | 10:13 |
| gtema | which blob? | 10:14 |
| tonyb | Try: `git show --format=fuller HEAD` | 10:16 |
| tonyb | (in openstack/glance) | 10:16 |
| gtema | well, according to it commit date is Fri Jul 26 06:34:52 2024 +0000 while according to gerrit it was merged today at 9:29 | 10:18 |
| tonyb | Sure, that's the best git can do, as frickler said, without a merge commit | 10:20 |
| gtema | you also will unlikely want to checkout multiple repositories locally to figure out where things got changed while investigating gate failures | 10:20 |
| gtema | that's sad | 10:21 |
| tonyb | Sure, but gitea shows the commit date as 5 days ago which matches the date from the fuller output | 10:21 |
| gtema | right - there is no difference between gitea and git locally | 10:22 |
| gtema | actually this way we are not even sure about order of commits | 10:23 |
| tonyb | Of course we are. If it isn't linear there has to be a merge commit | 10:27 |
| tonyb | You can get the merge date from gerrit: tony@thor:~/src/opendev.org/openstack/glance$ curl -skL 'https://review.opendev.org/changes/I6e7bdc2d092c32c060d2f3c72fc83cecc73ee7f6' | cut -zb6- | jq .submitted | 10:30 |
| tonyb | infra-root: I spent this afternoon testing the noble images uploaded to all clouds/regions. I can't test OSUOSL as we seem to be out of quota, for RAX (all regions) the image booted but wasn't functional (no ssh). I'll do more debugging tomorrow :/ | 10:41 |
| gtema | tonyb: thanks but that is not usable. As I described the use case I had is that all of a sudden gate fails and you want to quickly check whether in a certain project there are changes merged recently | 10:41 |
| gtema | But yes, gerrit is the only source of info | 10:43 |
| tonyb | gtema: Sure, if you don't know what (repo) broke it's hard to know where to look. | 10:44 |
| gtema | Exactly | 10:44 |
| tonyb | gtema: I'd expect the check logs vs the gate logs to show different SHAs for any require-repos | 10:45 |
| gtema | Oh, that's hardcore | 10:46 |
| tonyb | As a place to look | 10:48 |
| tonyb | gtema: https://review.opendev.org/q/mergedafter:%222024-07-30+00:00:00%22+branch:master | 10:51 |
| tonyb | shows all commits on master branches for all repos that have merged after 2024/07/30 00:00:00 (UTC | 10:52 |
| gtema | Uff, but yes, something I was thinking about | 10:52 |
| tonyb | so for your case you could use the timestamp from where zuul voted V+1 (converted to UTC) to give you a list | 10:53 |
| gtema | Thanks. I guess I would need to bookmark that for future | 10:54 |
| tonyb | gtema: there is also mergedbefore | 10:54 |
| tonyb | I was hoping they'd support relative timestamps 'mergedafter:now-24h' but they don't seem to | 10:55 |
| gtema | Thanks | 10:56 |
| fungi | gtema: gerrit stores its data in git. if you add a notes source for your gerrit remote you can fetch the git notes containing that additional metadata whenever you fetch normal refs, and git subcommands like log can be configured to display related notes. that's what i do | 12:14 |
| fungi | keep in mind that even without gerrit, you can't say when a was added to a branch of a git repository, that's simply not part of git's data model | 12:15 |
| fungi | er, can't say when a commit was added to a branch | 12:16 |
| gtema | thanks fungi | 12:16 |
| fungi | actually, we replicate the notes into gitea as well, now that i look closer at my ~/.gitconfig that's where i fetch them from | 12:18 |
| fungi | these are the relevant options i set: https://paste.opendev.org/show/bJui7pZJWy5rgak2hS7c/ | 12:19 |
| fungi | we filed a wishlist bug for gitea to be able to automatically display any related notes refs, but they weren't interested in that being a feature | 12:20 |
| fungi | here's what a current glance master branch log looks like with that enabled: https://paste.opendev.org/show/b6y58YRVs7mFTCJpIxRD/ | 12:24 |
| frickler | oh, nice, that includes "Submitted-at: Wed, 31 Jul 2024 07:29:40 +0000" then | 12:24 |
| fungi | yeah, i make extensive use of that as well as the Reviewed-on change urls | 12:25 |
| fungi | it's not especially useful for querying, but good when you have a commit locally and you want to know when it merged | 12:26 |
| fungi | without having to load it up in gertty or pull up a browser | 12:26 |
| frickler | I used to just search for the commit-id on gerrit, but having the direct URL is one less indirection | 12:26 |
| fungi | right, that's why i use it | 12:27 |
| frickler | this should really be part of our developer 101 somewhere | 12:27 |
| fungi | it does cause git to do an extra fetch any time you're updating a repository, so it's not without some performance impact | 12:29 |
| fungi | but yeah, this has been in gerrit for many years, at least since the introduction of the notesdb backend, possibly longer | 12:30 |
| Thibault | Hello! Thanks for the group! I have some questions regarding the utilisation of Mistral locally | 13:49 |
| corvus | gtema (Artem Goncharov): tonyb https://opendev.org/openstack/openstack is a lock-step chronological history of all openstack project merges, and the full repo state of all projects at each merge. | 13:57 |
| gtema | corvus: ok, it's not so easy in interpretation in the UI, but it does the job, thanks | 13:59 |
| corvus | yeah, a little easier to work with locally where you can see the full commit messages (notice they have the originating commit msgs, project names, and change ids too) | 14:00 |
| gtema | cool, thks | 14:03 |
| opendevreview | Dan Smith proposed openstack/project-config master: Add promote-test-images job for os-test-images https://review.opendev.org/c/openstack/project-config/+/925401 | 14:15 |
| fungi | looks like Thibault disconnected before i could ask for clarification | 14:28 |
| clarkb | corvus: gtema note that the openstack/openstack repo hasn't been properly maintained over time so it may lack projects that are relevant. but for the long lived projects that have been around a long time that shouldn't be a problem | 15:53 |
| gtema | thks for the hint | 15:53 |
| fungi | in the past there were changes like https://review.opendev.org/c/openstack/openstack/+/736012 to update the repository list, but it may make more sense to have an automated proposal job for that | 15:59 |
| fungi | looks like the release management team are the core reviewers for it at the moment, but if they don't want that responsibility maybe the tc could take it over or something: https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/openstack.config | 16:01 |
| clarkb | I don't see any feedback from frickler or tonyb overnight on the service coordinator election plan. I'm going to take that as a good sign and send out email today. We can always provide feedback on the mailing list too | 16:43 |
| clarkb | corvus: did you want to weigh in on https://review.opendev.org/c/openstack/project-config/+/925303 ? that sets th ezuul tenant to ansible 9 by default. You probably understand the impliciations of that bette rthan anyone else | 17:11 |
| frickler | clarkb: +1 for the election | 17:19 |
| clarkb | cool I'm like 80% done with my draft email. Hope to have it out shortly | 17:20 |
| corvus | clarkb: lgtm; though i'm a bit scattered today so not going to +w it myself | 17:26 |
| clarkb | corvus: sound sgood just wanted to make sure there werren't any specific concerns with doing that | 17:27 |
| corvus | nope sounds great to me | 17:27 |
| clarkb | email should be in your inboxes momentarily | 17:32 |
| clarkb | we've got a bunch of +2s on the tenant config update to change the default ansible version. I guess I'll go ahead and approve that now | 21:18 |
| clarkb | tonyb: re images not booting in rax rackspace has very specific networking requirements like static configuration of interfaces. Seems like there was something else that cloud init couldn't do for the longest time but can now | 21:20 |
| tonyb | clarkb: thanks. I'll see what I can find. it definitely looks like networking is failing to come up | 21:25 |
| opendevreview | Merged openstack/project-config master: Set OpenDev and Zuul Zuul tenants to use ansible 9 by default https://review.opendev.org/c/openstack/project-config/+/925303 | 21:30 |
| fungi | clarkb: yeah, i didn't want to just approve in case you had an announcement in mind for that, but impact should be fairly contained | 21:38 |
| clarkb | ya I don't think we need it for opendev and zuul. | 21:39 |
| clarkb | onc eit deploys we can recheck a bindep change or something to see if things generally work | 21:39 |
| fungi | tonyb: you should be able to use `openstack console url show ...` to get the oob console (novnc?) url for an instance in rackspace | 21:40 |
| fungi | may or may not be helpful in this case | 21:41 |
| tonyb | yeah I tried that. it's less helpful than I'd like. I get a few boot messages, and then a long pause and the getty | 21:42 |
| tonyb | no accounts work and without ssh/networking .... | 21:43 |
| fungi | harumph | 21:45 |
| clarkb | you might need to edit an image to add a user and password and then boot that and debug from there | 21:45 |
| clarkb | wouldn't be the first time we've had to do similra to debug glean or dib things | 21:45 |
| fungi | yeah, that would be my next step. or supply userdata to add a login or set the root password | 21:46 |
| tonyb | good idea. I forgot about adding a user data fragment | 21:48 |
| fungi | though that assumes cloud-init probably | 21:58 |
| tonyb | yeah true the same base image is working as expected on other clouds so I expect it is generally fine | 22:02 |
| fungi | i think in rackspace we rely on metadata (service or configdrive) | 22:37 |
| clarkb | I approved https://review.opendev.org/c/opendev/bindep/+/907366 to see if ansible 9 is happy ness | 22:49 |
| clarkb | Ansible Version: 2.16.9 I think that looks right | 22:51 |
| clarkb | https://zuul.opendev.org/t/opendev/build/a7915c91a5c54744b2617497f945a2f2/log/job-output.txt#35 this job passed so it isn't broken broken | 22:52 |
| fungi | yay! | 22:52 |
| opendevreview | Merged opendev/bindep master: Add classifiers for py310, py311 https://review.opendev.org/c/opendev/bindep/+/907366 | 22:59 |
| clarkb | all the jobs were successful | 22:59 |
| clarkb | honestly if we can flip the switch on that for openstack in the next week - two weeks we could sneak it in early enough to avoid release impacts? It seems like something that would probably work 95% of the time and not be a huge issue and we could revert if necessary | 23:04 |
| clarkb | I'll have to think about that a bit more | 23:04 |
| clarkb | Otherwise we're lookingat aerly october to avoid all the release stuff | 23:04 |
| fungi | yeah, i think so | 23:09 |
| tonyb | Can I get some reviews on: [base/unbound] Setup the initial trust anchor | https://review.opendev.org/c/opendev/system-config/+/925311 That's preventing Noble from being usable on non-rax clouds. | 23:36 |
| fungi | tonyb: for our test nodes i also added the dns-root-data package: https://review.opendev.org/920208 | 23:44 |
| fungi | do we need that as well? | 23:44 |
| fungi | er, actually i think it might have been addressing the same problem, just in a different way? | 23:46 |
| fungi | so maybe we need to figure out whether we want to be consistent or see long-lived servers as a different situation where unbound's trust anchor data is concerned | 23:48 |
| clarkb | seems like the dns root key package is simpler if the package installation does what we need it to reather than needing a separate step? But I don't know what the difference is between the two | 23:50 |
| fungi | i think unbound-anchor is a separate job/service that periodically updates the root data and trust anchor? | 23:50 |
| fungi | vs dns-root-data which is just a packaged copy of the files | 23:51 |
| clarkb | theoretically the ubuntu package would update periodically as needed too | 23:51 |
| fungi | right | 23:51 |
| clarkb | the way tonyb's change is written it wouldn't update I think? | 23:52 |
| clarkb | it could be modified to update but probably just installing the dns-root-data package is simpler and solves the problem | 23:52 |
| clarkb | and ya looking at the package changelog it does appear to be synced occasionally (once or twice a year) | 23:53 |
| johnsom | Correct, the unbound-anchor package is just the binary | 23:53 |
| tonyb | The docs indicated that the update/sync is still done by unbound via the config setting but it refuses to generate the initial file. and that unbound-anchor will basically just do that part | 23:53 |
| clarkb | huh that is an interesting behavior choice | 23:53 |
| tonyb | because if the inital setup is "bogus" unbound can trust the wrong DNS? | 23:54 |
| clarkb | fwiw it sounds like unbound also respects what dns-root-data installs (and that aligns with fungi's change fixing tyhings) | 23:55 |
| tonyb | I can verify that dns-root-data does the right thing | 23:56 |
| clarkb | I think my preference is to go that route since we're already doing it and it appears to be simpler | 23:56 |
| tonyb | assuming it does I'll use that to be consistent | 23:57 |
| fungi | right, to be clear, before noble we were installing dns-root-data implicitly but they moved it to a recommends in noble and so it fell out of our default package list. all my change did was explicitly add it back | 23:58 |
| clarkb | ya just discovered it is a recommends now | 23:59 |
| fungi | it was a quick change to write, a rather lengthy one to research and figure out what exactly had happened which led to the (marginally misleading) error that also caused me to initially think we wanted unbound-anchor installed | 23:59 |
| tonyb | Oh okay. | 23:59 |
| clarkb | and ya this appears to be what ubuntu packaging expects you to do. not sure why it is just a recommends though | 23:59 |
| fungi | normally recommends are installed by default | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!