Tuesday, 2022-12-13

« Monday, 2022-12-12 Index Wednesday, 2022-12-14 »
-@gerrit:opendev.org- Ian Wienand proposed:00:16
- [zuul/zuul-jobs] 866953: ensure-kubernetes: add microk8s support https://review.opendev.org/c/zuul/zuul-jobs/+/866953
- [zuul/zuul-jobs] 867063: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/867063
-@gerrit:opendev.org- Ian Wienand proposed:00:24
- [zuul/zuul-jobs] 866953: ensure-kubernetes: add microk8s support https://review.opendev.org/c/zuul/zuul-jobs/+/866953
- [zuul/zuul-jobs] 867063: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/867063
-@gerrit:opendev.org- Ian Wienand proposed:00:35
- [zuul/zuul-jobs] 866953: ensure-kubernetes: add microk8s support https://review.opendev.org/c/zuul/zuul-jobs/+/866953
- [zuul/zuul-jobs] 867063: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/867063
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 867063: use-buildset-registry: support microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/86706301:02
-@gerrit:opendev.org- Tim Beermann proposed: [zuul/zuul-jobs] 866679: Add yamllint job. https://review.opendev.org/c/zuul/zuul-jobs/+/86667906:37
-@gerrit:opendev.org- Tim Beermann proposed: [zuul/zuul-jobs] 866679: Add yamllint job. https://review.opendev.org/c/zuul/zuul-jobs/+/86667906:47
-@gerrit:opendev.org- Tim Beermann proposed: [zuul/zuul-jobs] 866679: Add yamllint job. https://review.opendev.org/c/zuul/zuul-jobs/+/86667906:59
-@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 866863: Store pause and resume events on the build and report them https://review.opendev.org/c/zuul/zuul/+/86686308:02
-@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 866863: Store pause and resume events on the build and report them https://review.opendev.org/c/zuul/zuul/+/86686308:08
-@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 866863: Store pause and resume events on the build and report them https://review.opendev.org/c/zuul/zuul/+/86686308:11
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 865687: Consider queue settings for topic dependencies https://review.opendev.org/c/zuul/zuul/+/86568715:07
@clarkb:matrix.orgHello Zuulians. I have been asked to remind you all that the OpenInfra Summit CFP closes in four weeks. The summit runs June 13-15 in Vancouver, BC Canada. If you would like to talk about your use of Zuul or other relevant technologies feel free to submit a proposal here: https://cfp.openinfra.dev/app/vancouver-2023/19. You can learn more about the event here: https://openinfra.dev/summit/vancouver-202316:42
-@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 859939: Implement Gitea driver https://review.opendev.org/c/zuul/zuul/+/85993917:11
-@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 859940: Implement tests of the gitea driver https://review.opendev.org/c/zuul/zuul/+/85994017:11
-@gerrit:opendev.org- Ade Lee proposed: [zuul/zuul-jobs] 866881: Add ubuntu to enable-fips role https://review.opendev.org/c/zuul/zuul-jobs/+/86688117:12
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 865923: Avoid acquiring pipeline locks in manager postConfig https://review.opendev.org/c/zuul/zuul/+/86592318:07
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 867175: Reuse queue items after reconfiguration https://review.opendev.org/c/zuul/zuul/+/86717521:20
@iwienand:matrix.org```23:10
2022-12-13 23:09:51,449 ERROR nodepool.builder.CleanupWorker.0: Unable to delete image ubuntu-xenial-1670176248 from rax-iad:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/nodepool/builder.py", line 326, in _deleteUpload
manager.deleteImage(upload.external_name,
File "/usr/local/lib/python3.11/site-packages/nodepool/driver/openstack/provider.py", line 223, in deleteImage
return self._client.delete_image(dict(id=id))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/openstack/cloud/_image.py", line 223, in delete_image
self.image._IMAGE_OBJECT_KEY in image.properties
^^^^^^^^^^^^^^^^
@iwienand:matrix.orgthis is on nb02, and looks like something openstackclient-y23:11
@iwienand:matrix.orgit's in a pretty tight loop on that23:11
@clarkb:matrix.orghrm that could be related to the openstacksdk update.23:12
@clarkb:matrix.orgWhen I spot checked it it looked like images were deleting and uploading properly though23:12
@clarkb:matrix.orgianw:  is that traceback complete?23:13
@iwienand:matrix.orgthat traceback is all we're getting, yep23:14
@clarkb:matrix.org`AttributeError: 'dict' object has no attribute 'properties'` is the bit that is missing23:15
@iwienand:matrix.orgohhh, sorry, copy-paste fail23:15
@clarkb:matrix.organd an image list shows that we are leaking them in the cloud.23:16
@iwienand:matrix.orghttps://review.opendev.org/c/openstack/openstacksdk/+/823092 seems to be where it changed to looking at .properties 23:17
@clarkb:matrix.orglooking at that code this just seems to be broken23:17
@clarkb:matrix.orgget_image is returning a dict and this is exploding because dicts don't have a .properties23:18
@iwienand:matrix.orgwell it might be something like the other bug with rax where ".properties" isn't set due to -EVERYOLD or something.  it only seems to be rax23:18
@clarkb:matrix.orgyes, though the error says image is a dict and dicts don't have a .properties23:19
@clarkb:matrix.orgif they are modifying dict objects with a .properties...23:19
@jim:acmegating.comremember that shade used to have munch objects which were dicts that behaved like objects with accessors for fields23:20
@jim:acmegating.comnot sure what the current status is on that with the latest sdk update23:20
@clarkb:matrix.orgwe should be able to run a little script that does a cloud.get_image() and then see what the results are23:21
@jim:acmegating.comi'll do that.  i should be able to re-purpose my test environment23:22
@iwienand:matrix.orgi also have some test bits left over from the last openstacksdk rax breakage :)23:23
@clarkb:matrix.org```import openstack23:24
@clarkb:matrix.org * ```import openstack23:24
```
@clarkb:matrix.orgI don't know how to do rich text23:24
@jim:acmegating.comClark: to block quote something, 3 backticks on one line to start, and the same to end23:25
@clarkb:matrix.org * ```import openstack23:25
conn = openstack.connect(cloud='rax')
```
@clarkb:matrix.org * ```23:26
import openstack
#TODO how to set region?
conn = openstack.connect(cloud='rax')
image = conn.get_image('imageuuidorname')
print(image)
```
@clarkb:matrix.orgI think this image management path is fairly unique to rax too beacuse they do the swift backed images whereas everyone else is direct upload to glance23:28
@iwienand:matrix.org```23:29
r = cloud.image.get_image('6604871b-3d19-4727-8b05-2501a4a8afad')
print(r.properties)
```
@iwienand:matrix.orgworks23:29
@iwienand:matrix.orghave to quickly pick up kid from shop, back in 523:29
@jim:acmegating.comthat's not the equivalent call; it's cloud.image.get_image({'id': '660...'})23:35
@jim:acmegating.comget_image({'id': '6604871b-3d19-4727-8b05-2501a4a8afad'})23:35
@jim:acmegating.comthat reproduces the error23:35
@clarkb:matrix.orglook at the definition of _get_entity. I think it short circuits and returns what was passed if you pass it a dict like that23:36
@clarkb:matrix.orgianw's case goes throughactual lookup23:37
@clarkb:matrix.orgso maybe we need to call it like ianw in nodepool?23:37
@jim:acmegating.comyes, this looks like it may be an api change from sdk23:37
@clarkb:matrix.orgyeah. But the fix should be easy I guess?23:38
@clarkb:matrix.orgwe could even hand patch nodepool on nb01/nb02 and restart the containers to see if it works23:38
@clarkb:matrix.orgthe new thing should at least be backward compatible though I think23:39
@clarkb:matrix.orgbut we may want to set openstacksdk>=0.103.0 anyway23:39
@iwienand:matrix.orgget_image just says 23:39
@iwienand:matrix.orgimage – The value can be the ID of a image or a Image instance. 23:39
@iwienand:matrix.orgi wonder if it ever said a dict23:40
@jim:acmegating.comClark: agree the fix is simple; unsure about backwards compat, the version min may be safest23:40
@iwienand:matrix.orglet me hand-patch nb02 and see23:41
@clarkb:matrix.orgianw you will need to docker-compose stop then start it after you edit it23:42
@iwienand:matrix.orgi just hup'd it23:43
@jim:acmegating.comi'm not sure it's worth digging in deeper, but i think that these methods probably haven't really changed much -- except that previously they were simple enough that they would work with {'id': '...'} as input, but now they expect a more fully-formed object as input23:43
@iwienand:matrix.orgit's deleting a bunch now with no warnings23:43
@jim:acmegating.com * i'm not sure it's worth digging in deeper, but i think that these methods probably haven't really changed much -- except that previously they were simple enough that they would work with {'id': '...'} as input, but now they expect a more fully-formed object as input (by accessing the '.properties' attr)23:43
@iwienand:matrix.orghttps://paste.opendev.org/show/bXkJefwBIGI94wgIdEDq/23:43
@jim:acmegating.comso basically, there previously was an old informal api where you could pass in a "good enough" object, and now that isn't supported anymore so we  should fix nodepool.23:44
@clarkb:matrix.orgcorvus: there is a change from may that converts a number of calls that similarly called get_image to just pass in the id bare23:44
@clarkb:matrix.orgin openstacksdk I mean23:44
@clarkb:matrix.orgianw: did you want to propose the fix since you wrote the change on nb02?23:44
@clarkb:matrix.orgianw: if so can you update the requirements to require openstacksdk>=0.103.0?23:44
@jim:acmegating.comClark: ianw may i propose the fix?23:44
@clarkb:matrix.orgcorvus: oh sure. I'm happy for anyone to do it :)23:45
@iwienand:matrix.orgsure23:45
@jim:acmegating.comthx, i want to stack the adapter on it at the same time23:45
@iwienand:matrix.orgok, well happily it was just the deletion path, i was a bit worried at first with all the flooding messages23:45
@iwienand:matrix.orgnow back to why tox-py38 is failing on dib...23:46
@clarkb:matrix.orgspot checking the cleanup it is doing now I see images it says are deleting are no longer present and they were in my early image-list'ing23:47
@clarkb:matrix.orgBut there are still a number of deleting images present so this may take some time I guess23:47
@clarkb:matrix.orgdoing a somewhat naive grep for similar patterns in the openstack driver produces no other results which hopefully means this is the only place we're broken like this23:48
@clarkb:matrix.orgI have now confirmed via `wc -l` that the numer of images is steadily dropping so ya just needs time23:49
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed:23:50
- [zuul/nodepool] 862522: Convert openstack driver to statemachine https://review.opendev.org/c/zuul/nodepool/+/862522
- [zuul/nodepool] 867578: Fix openstack image deletion with newer sdk https://review.opendev.org/c/zuul/nodepool/+/867578
@jim:acmegating.comianw: Clark ^23:51
@jim:acmegating.comClark: would you mind re-adding your +2 on the statemachine change while it's fresh in your mind?  (should be the only diff)23:51
@clarkb:matrix.orgcorvus: ya23:51
@iwienand:matrix.org... i should spend some time with that too, sorry it's mostly passed me by23:53
@clarkb:matrix.orggerrit did confirm that was the only diff23:53
« Monday, 2022-12-12 Index Wednesday, 2022-12-14 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!