Thursday, 2024-10-03

mnasiadkaclarkb: regarding Ansible and locale - see https://review.opendev.org/c/openstack/kolla-ansible/+/930975 (I don't know if this is any helpful information)09:23
mnasiadkafungi: I'm mainly seeing those logger issues on instances in rax-ord and rax-dfw09:42
kevko+110:00
kevkofungi: this is log for ansible -m setup before change https://901249311ab7ecbd3f16-b4ffb63fd72a873cfc8fbd2b6e893a02.ssl.cf2.rackcdn.com/930975/4/check/kolla-ansible-rocky9-kvm/432ae64/primary/logs/ansible/initial-setup  and this is after my change mnasiadka mentioned10:05
kevkohttps://de62b6c8c004e1afcffe-25456b362a22ad76281304acb8641637.ssl.cf2.rackcdn.com/930975/25/check/kolla-ansible-rocky9/2b38a38/primary/logs/ansible/initial-setup    ..... check 'ansible_env' key in ansible  dictionary  ..in failed there is more language variables set ...    AND this is normally working ubuntu/debian10:05
kevkohttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a5b/909912/50/check/kolla-ansible-ubuntu-kvm/a5b72ec/primary/logs/ansible/initial-setup ...where is only LANG also .....10:05
kevkoso I hope it will help 10:06
clarkbmnasiadka: I don't think that change properly fixes the issue because the images are configured to not install C.utf8 content14:49
clarkbmnasiadka: it probably "works" beacuse glibc will afllback to C or similar if it doesn't have the lang that is set14:49
clarkbmnasiadka: I think https://review.opendev.org/c/openstack/diskimage-builder/+/930932 is the proper fix for the missing locale but I am hoping someone with a centos background or interest can help review that and at least confirm it is unlikely to make anything worse14:50
clarkbI don't think it can be broken more than it is now so I suspect that change is safe and if it helps great and if not no big deal. But I'm not certain of that14:50
* clarkb is going to pop into the monthly gerrit community meeting in ~5 minutes14:55
clarkbit has been a while since I attended due to summer activity and summit travel. I can't think of anything to bring up from our end but yu have a couple of minutes to let me know if there is something14:56
clarkbgerrit user summit is next week. I knew that but then forgot. There will be live streaming of sessions via microsoft teams with a link on the docs for the event15:07
JayF:-O how long have  I been able to expand files inline in gerrit, without opening them up one by one15:55
JayFI 'misclicked' and learned a cool thing15:55
mnasiadkaclarkb: thanks for referencing that patch, looks sane enough15:58
clarkbJayF: since we upgraded to 3.215:58
JayFI think that may be the first time in my life I've thought about a gerrit version number :D ... since it's 3.9.6 now that means ... a long time? years?15:59
clarkbyes about 3 years looks like15:59
fungii need to take a friend to the mainland to pick up their car from the mechanic, so will be disappearing for a few hours, but will check back in once i'm home16:06
JayFo/16:07
clarkbthe openstack release is done so now it is time to think about landing https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 I'll also get a change up to update the default ansible version in the remaining opendev tenants16:20
clarkbwe can land both after JayF successfully releases their bugfixes for ironic though16:20
opendevreviewClark Boylan proposed openstack/project-config master: Switch the remaining opendev zuul tenants to ansible 9 by default  https://review.opendev.org/c/openstack/project-config/+/93132016:26
corvusclarkb fungi i'm looking into rax-flex object storage.  i believe we set up the old rax containers using this method: https://docs.rackspace.com/docs/set-up-cloud-files-and-acls16:56
corvusi don't know if (a) usernames in the old control panel would exist in the flex world in order to be added to swift.16:57
corvusi also don't know (b) how to get the token that would be needed to do the manual acl step16:57
corvusi feel like i could spend a lot of time trying to solve both of those, and eventually finding out that it just doesn't actually work with that swift16:59
corvusso if we don't have any actual docs that suggest that there's a way to do swift acls with flex, maybe we should assume no for now?17:00
clarkbcorvus: thats a good point. I suspect those acls are a rax add on and not in normal swift and since rax flex is more normal openstack wouldn't have support for that?17:01
clarkbhttps://docs.openstack.org/swift/latest/overview_acl.html but maybe I'm wrong17:01
corvusyeah, i think they are sort of standard17:02
corvusbut that still requires a new user in keystone i think17:02
clarkbya seems likely unless they will automatically lookup that user from keystone to the other old auth system17:02
corvusthere is the ability to create an application credential; i don't know if that shows up as a user in keystone?17:02
clarkbjamesdenton: do you know the answer to those question above ^?17:03
clarkbjamespage: basically how automatic is the integration between swift auth in rax flex and the normal rax auth systems. For nova et al we just used the api key as a keystone password and things worked17:04
clarkbjamespage: sorry that message was intended for jamesdenton 17:04
clarkbjamesdenton: does the swift integration work in a similar way if we create a new swift specific user in the old backend and then set up acls on containers in swift within rax-flex?17:05
corvusit's getting confusing with all these jameses ;)17:05
clarkbcorvus: fwiw I have no objection to setting up a container for nodepool in zuul in an existing working swift setup while we figure out these details for rax flex17:05
corvusclarkb: ack; i'll continue poking at flex for a bit to see what i can learn17:07
corvusokay i uploaded it with ansible and just the top level credentials.  that half-worked.  the file was uploaded, but the openstack ansible module threw an error: https://paste.opendev.org/show/bz0kgPz2QMXsTZSzuDV0/17:16
clarkbcorvus: I think ansible's openstack moduels have very specific openstacksdk requirements. Might be worth doubl checking what you have installed aligns?17:18
clarkblike maybe that explains why there is a mismatch in type expectations. Otherwise I don't have any great ideas17:18
corvuspage says openstacksdk >= 1.0.0 and i have 4.0.017:18
corvushttps://docs.ansible.com/ansible/latest/collections/openstack/cloud/object_module.html#ansible-collections-openstack-cloud-object-module-requirements17:19
clarkbhttps://github.com/openstack/ansible-collections-openstack?tab=readme-ov-file#branches-and-non-backward-compatibility-%EF%B8%8F also seems to support that you are good17:20
clarkboh thats hosted on opendev.17:20
clarkb#openstack-ansible-sig may have more ideas?17:21
corvushttps://opendev.org/openstack/openstacksdk/src/branch/master/openstack/object_store/v1/_proxy.py#L47217:26
corvusthe code path where filename is provided and data is not (which is what i'm doing) will return None17:26
corvushttps://docs.openstack.org/openstacksdk/latest/user/connection.html17:26
corvusthat appears to differ from the docs ^ (search for create_object; i don't see a deep link option sorry)17:27
corvusthe docs say it will always return an Object17:27
clarkbI was as far as the object_store create_object method and trying to find the _create def17:27
clarkbbut in this case you are dealing with images which go through the other path17:28
clarkbI agree this seems liek a bug in the docs at least and then subsequently in the ansible modules17:28
corvusyeah, i think we want the filename path for this one due to size17:28
clarkbcorvus: I think maybe you can return the response of the large object upload17:29
clarkbit ultimately calls https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/object_store/v1/_proxy.py#L720-L726 which may return an object? I'm trying to figure that out17:30
clarkbbut now i have to jump over to keystoneauth for that17:30
corvusAttributeError: 'Response' object has no attribute 'metadata17:32
corvusso it's at least more complicated than that17:33
clarkbya looks like it is a python requests response object17:33
clarkbnot an openstack api object17:33
corvusin zuul-jobs swift upload, we just ignore the result17:34
corvusexception=bad, no exception=success17:34
clarkbin this case we are getting an exception though right?17:35
corvusno; the exception is from ansible17:35
corvussorry -- to be clear, zuul-jobs swift upload does not use the openstack ansible modules17:36
corvusit uses the sdk directly17:36
clarkboh right the ansible module expects an object not None17:36
clarkbbut we could do like zuul jobs logs uploads and upload directly then only care if the sdk generates an exception. That seems reasonable17:36
corvusyeah, so we might need a tiny ansible module for this17:36
corvusor... does flex have an s3 api?  :)17:37
clarkbI'm not sure. I believe the swift s3 api support is middleawre you have to add and not built in (so can't be assumed to be present)17:37
corvusand and it's not on the old rax stuff17:38
corvushttps://bugs.launchpad.net/ansible-collections-openstack/+bug/206160417:41
corvushttps://review.opendev.org/c/openstack/ansible-collections-openstack/+/92650617:41
corvuswhy are all the jobs dependent on pep8?17:42
corvusclearly that patch has a linter error.  i would like to know, as a developer and contributor, whether it actually works regardless of the linter error.17:43
clarkbI've largely given up on trying to convince people that reducing round trips is a good thing17:45
clarkbthere is a sense that we're optimizing by running fewer jobs and returning less info to contributors17:45
corvuswelp, it worked backwards here.  instead of me fixing up this patch for free, i'm moving on to something else.17:46
corvusto anyone else reading this: your time is valuable.  it is definitely more valuable than the amount of money it costs to run a virtual machine long enough to run a test job.  please let the ci system run as many jobs as possible so that you and other developers don't have to waste time on round trips or running jobs themselves.17:51
clarkbfwiw that code does look correct after reading _find()s implementation. It is a bit excessive in terms of extra api requests but maybe thats the only way to get a complete picture after a multi part upload17:51
JayFpre-commit is useful for these kinda things, ensuring basic quick checks are done before code lands17:52
clarkbJayF: I really dislike precommit...17:52
JayFbut even if you have it enabled for a repo, you cannot force someone to have the hook installed17:52
corvusi agree in principle that running some quick checks before upload can save everyone time, and typically do so myself17:53
clarkbthere is nothing special that precommit does especially if you are setup with tox or nox already. But the way it manages dependencies breaks a bunch of assumptions in the python world17:53
clarkbits crazy to me that people have created a bunch of meta repos just to make precommit happy isntead of just fixing pre commit17:55
JayFHonestly, that's not entirely true; the first part of that clause17:55
clarkbbut the path of least resistance is to just make a git repo with some metadata I guess17:55
JayFpre-commit only runs configured checkers on /changed files/17:55
JayFwhich is a significant speed boost vs running pep8, codespell, docs job, and similar jobs that are just a  'sanity check'. Having it happen in <30 seconds on commit is *way* faster than having tox spin up all those tests17:56
JayFI don't know anything about how it manages dependencies tbh, I mainly patterned Ironic's use after using it in Nova and seeing how much it improved my workflow17:57
clarkbJayF: I think most of those tools also support that functionality17:57
clarkbbut maybe generating the list of changed files is the issue17:57
clarkblike flake8 can work on the diff17:57
clarkbJayF: precommit only installs things from git repos17:57
clarkbor maybe it does other repos too but only from source not packages. So tools like ruff have a meta repo that says I'm a package that depends on the package in pypi because building ruff from source requires a rust toolchain and is slow17:58
clarkbwhich breaks constraints and lockfiles and pacakge caching17:59
JayFI don't fully follow tbh18:00
JayFI get 'it only installs from git repos'18:00
JayFI don't understand the meta repo + ruff thing?18:00
clarkbright so if you naively list ruff in precommit as a source repo to install and execute you need to be able to compile ruff from source with a rust toolchain where you are running precommit. This is slow and also requires tools that may not be easy for everyone to get (because rust is bleeding edge and all that)18:01
clarkbinstead you tell precommit to use the ruff-precommit source repo and install that. That then has a simple setup.py (or similar) that says I depend on ruff which causes the from source package install to pull down ruff's wheel from pypi and install that18:02
clarkbprecommit could be fixed to install pypi packages directly then you don't need meta packages for each tool with slow compiles18:02
clarkbmeanwhile you're ignoring any constraints and lockfiles you may have because this is a separate installation (you may still end up using cached packages at least since I think it goes through pip for that redirect)18:03
clarkbhttps://github.com/astral-sh/ruff-pre-commit/blob/main/pyproject.toml#L518:03
clarkbits just a lot of boilerplate which ends up in a less than ideal situation compared to doing the right thing upfront imo18:04
JayFahh, I see18:06
clarkbit also introduces another potential failure mode for CI systems (because now you're fetching random git repos that the CI system is likely unaware of)18:06
JayFI reallly wonder if some of this is patched outta the gentoo version18:06
JayFand that may be why I didn't have similar experiences18:06
clarkbas an alternative you could just have a pre commit hook that runs ruff with tox18:07
JayFThis might be good to bring up in cross-project stuff18:08
clarkbI brought it up in the thread that was started about this stuff on the mailing list recently18:09
JayFbecause I am extremely unlikely to introduce usage of that tool that's flipped from how other groups, like nova, are using it18:09
JayFand nova has tox run pre-commit aiui18:09
clarkbyes to keep the results in sync18:09
clarkb(otherwise tox and precommit are very likely to use different versions of tools and produce different results)18:10
clarkbside note: still not AAAA record for github.com18:12
corvusi have used an application credential to upload a file18:14
corvusthat may not help us much with access control, since i think it's likely to have all the same access18:14
corvusbut it at least makes creds easier to rotate18:15
clarkbcorvus: so the open question for raxflex is still how to apply acls?18:15
corvusyep18:15
clarkbcorvus: I am going to push up a pep8 fixed update to that change since it sounds like you weren't planning to18:15
corvuswfm18:16
corvusyou might consider fixing their ppc while you're at it :)18:16
clarkboh oops just pushed18:16
* clarkb looks at that next18:17
corvusi mean it's probably a good second change :)18:17
timburkeha! we raced, clarkb! i happened to see swift mentioned here, then happened upon ben's patch and though, "hey, i know that guy! i can help him out a little"18:19
clarkbremote:   https://review.opendev.org/c/openstack/ansible-collections-openstack/+/931335 Run functional testing regardless of pep8/linter results18:20
clarkbtimburke: oh oops18:20
timburkeno worries! it all came out the same :-)18:21
corvusi have +2d both of those.18:22
corvusapplications credentials have two parts: an id (looks like a uuid) and a secret (long random string)18:50
corvusdo we consider the id to be sensitive?  (should i encrypt it?)18:50
Clark[m]I want to say id is not secret. But we I think in system config we manage them as secret because it's never been super clear how sensitive that info is?18:55
fungioof, a lot of scrollback to catch up on18:57
corvusfungi: oh i wouldn't bother :)18:57
corvusjust a bunch of humans feeding the llms nonsense18:58
corvusClark: yeah, my guess is probably fine to have plaintext but why not go ahead and encrypt it since it's not meaningful anyway18:59
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Add artifact upload support  https://review.opendev.org/c/opendev/zuul-jobs/+/93134019:15
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Add artifact upload support  https://review.opendev.org/c/opendev/zuul-jobs/+/93134019:17
corvusClark: fungi ^ can you take a look at that and let me know if that looks okay.  if it does, i'd like to merge it, and then get the okay to just merge a bunch of followup changes until it actually works (since the secrets involved make it post-review)?19:18
corvusalso... should we consider making a trusted-check pipeline for the opendev tenant?19:18
clarkbcorvus: looking now. Will haev a couple of comments19:24
fungii have finally caught up with re-training my meat-based llm on the scrollback and am taking a look at those changes19:25
clarkbcorvus: posted19:26
fungiis comparing roles/image-upload-swift/library/image_upload_swift.py to roles/upload-logs-base/library/zuul_swift_upload.py worthwhile?19:29
jamesdentonclarkb sorry for the delay. Are you suggesting creating a new user in the portal and using that user for Flex-Swift?19:30
clarkbjamesdenton: I think we're wondering how well https://docs.rackspace.com/docs/set-up-cloud-files-and-acls maps onto rax flex. One of the steps in there is to create a dedicated user that can have acls applied to it19:31
jamesdentonGood question - let me ask and i will get right back to you19:33
corvusfungi: some of the code comes from there; it's much simpler though19:37
fungik, thanks. that helps19:38
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Add artifact upload support  https://review.opendev.org/c/opendev/zuul-jobs/+/93134019:43
corvusclarkb: ^ replied and addressed19:43
clarkbcorvus: +2 from me thanks19:44
clarkbfor a little while now I've been noodling some way to communicate how to take advantage of more advanced opendev/zuul features. I'm somewhat committed to making that a reality over the next couple of months so I've started a brainstorm for the topics that I think would be helpful for people basedon misconceptions and misunderstandings I've seen in the past19:48
clarkbhttps://etherpad.opendev.org/p/advanced-opendev-brainstorm19:48
clarkbhappy for people to add notes or questions on tsuff they would like more info for19:48
jamesdentonclarkb i think the answer is... it ought to work but would be helpful to know if it doesn't? :D19:49
jamesdentonbut you'd want to use the Flex swift endpoint instead19:49
clarkbright19:49
clarkbjamesdenton: is there a good contact to followup with if we hit problems or want to report an all good? maybe that is you?19:49
jamesdentonThat email thread we already have would be perfect. Kevin can nail someone down19:50
clarkbjamesdenton: great thanks you19:51
opendevreviewMerged opendev/zuul-jobs master: Add artifact upload support  https://review.opendev.org/c/opendev/zuul-jobs/+/93134020:00
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Fix build_diskimage_image_name variable name  https://review.opendev.org/c/opendev/zuul-jobs/+/93134620:15
opendevreviewMerged opendev/zuul-jobs master: Fix build_diskimage_image_name variable name  https://review.opendev.org/c/opendev/zuul-jobs/+/93134620:18
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: testing  https://review.opendev.org/c/opendev/zuul-jobs/+/93134720:32
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: testing  https://review.opendev.org/c/opendev/zuul-jobs/+/93134720:33
clarkbI need to run an errand in like an hour ish. Things seem quiet though so don't expect any problems21:09
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: Reduce LVM extent usage  https://review.opendev.org/c/openstack/diskimage-builder/+/93095021:14
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: testing  https://review.opendev.org/c/opendev/zuul-jobs/+/93134721:27
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: testing  https://review.opendev.org/c/opendev/zuul-jobs/+/93134722:06
corvushttps://zuul.opendev.org/t/opendev/build/8cf4e0abcd2f40d8a969072d9ec97929/console#4/0/3/ubuntu-noble22:15
corvusclarkb: fungi ^ i could use another set of eyes on that... i'm not seeing what's wrong...22:15
corvusi'm wondering if it's something about how that complex variable is being created...22:16
corvushrm yeah...22:18
corvusmaybe that's late evaluation happening after the secret freezing stuff22:18
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: testing  https://review.opendev.org/c/opendev/zuul-jobs/+/93134722:30
corvushttps://zuul.opendev.org/t/opendev/build/0533ca1c11dc4b3e85f7523f5de4967d22:34
corvusapparently "apt-get install python3-openstacksdk" is not sufficient to get a working openstacksdk :(22:35
fungiwhat is it missing?22:38
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: testing  https://review.opendev.org/c/opendev/zuul-jobs/+/93134722:38
corvuspython3-oslo.utils apparently22:38
fungiinteresting. if i pip install openstacksdk into a venv it also doesn't install oslo.utils22:40
fungiit seems to soft integrate passing in oslo_config objects22:44
fungiwithout explicitly declaring any dependency on that22:44
fungibut that's the only oslo reference i spot in openstacksdk22:44
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: testing  https://review.opendev.org/c/opendev/zuul-jobs/+/93134722:46
corvuswoo that worked; now to clean it up22:51
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: testing  https://review.opendev.org/c/opendev/zuul-jobs/+/93134722:55
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Finish upload job  https://review.opendev.org/c/opendev/zuul-jobs/+/93135522:58
fungithat's an exciting commit title22:59
clarkberrand complete reviewing now22:59
corvusthose 2 changes are basically identical; the first one is running now but will intentionally fail in gate; the second one is the "real" change which we can review now and approve once we're happy with the behavior of the first one23:00
clarkbcorvus: why do we set pass to parent now?23:00
clarkbwe shouldn't need it since we're doing the upload in this job right?23:01
corvusclarkb: it's because we're adding the secret in the project-pipeline variant.  this is not something we've done before (normally we have separate build and upload jobs) but on a whim i thought it might be interesting to explore this idea.23:01
clarkboh I see there is a child job variant23:02
corvus(though, if we had separate build/upload jobs, we still might need the same construct)23:02
fungiyeah, i like the "pass a secret if you want to upload" idea, personally23:02
clarkb+2 from me23:03
fungibut i can see how it might seem slightly magic too23:03
corvusi think to avoid pass-to-parent, we would need to make a lower-level job in a config project that does the uploading23:03
clarkbshould that job be protected? I think that is the term23:04
clarkb(to keep it from being reused elsewhere?)23:04
corvus(something like "base-upload-image" in opendev/base-jobs that did the upload and had the secret; then have the "image-build-base" job in opendev/zuul-jobs inherit from it)23:04
corvusclarkb: the current thing?  nah, it's fine.  no one can inherit the project-pipeline variant with the secret23:05
corvusif we did the base-upload-image in opendev/base-jobs we would need to think about that though23:05
clarkback also worst case they would just upload stuff23:05
clarkbI don't think they could influce the actual fetch and reupload to the clouds portion23:05
fungii think worst case they would just build and not upload stuff?23:05
fungisince they can't inherit the creds23:06
clarkbya I meant if they could inherit the creds they could upload to the container23:06
fungioh, that, right23:06
clarkbthough maybe they could inject an overlapping image or something but ya seems like a non issue23:06
corvusyeah, that's not possible with this construct, but it is with the base job construct.  so if we did that, we would probably want to protect it.23:06
fungi"if they could inherit the creds" i think we'd consider that a severe security vulnerability in zuul and fix it right away23:07
fungioh, with the separate job. yes23:07
fungibut that doesn't exist (currently anyway)23:07
fungiso still not a concern, just worth remembering23:08
corvushttps://zuul.opendev.org/t/opendev/build/2d3e3929d688419bb6d62923e3034bbf is exactly the intentional failure we want, and the artifact looks good -- however -- i think we want to change the artifact file name to include the build uuid :)23:08
clarkb++ to having a uuid there23:11
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: test new upload name  https://review.opendev.org/c/opendev/zuul-jobs/+/93135623:11
corvusi did it like "uuid-imagename.qcow2".  since these are short-lived, i'm not too worried about needing a directory hierarchy23:12
corvusbut happy to change if others have differing preferences23:13
clarkba timestamp might also be helpful but I think swift can provide that to us if necessary23:13
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Finish upload job  https://review.opendev.org/c/opendev/zuul-jobs/+/93135523:14
corvusyeah; at least, the timestamp shows up in the web ui23:14
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: WIP: testing  https://review.opendev.org/c/opendev/zuul-jobs/+/93134723:15
opendevreviewJames E. Blair proposed opendev/zuul-jobs master: Finish upload job  https://review.opendev.org/c/opendev/zuul-jobs/+/93135523:18
corvusokay that looks like it worked; clarkb fungi if you want to re-review https://review.opendev.org/931355 with the name change, i think we're done23:22
clarkblgtm23:28
corvuswell, that did not work for some unknown reason (no_log)23:56
corvusthe test change took 2 seconds to upload the 10 byte file, but the real change took 30s to fail23:58
corvusthat makes me think that there's something about uploading the actual image file that it didn't like.23:59
corvusprobably the way to proceed is to get an autohold, then try running that ansible task manually from the node.23:59
Clark[m]++ sorry I'm switching to dinner mode so not much help now23:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!