Tuesday, 2023-01-24

opendevreviewMichael Kelly proposed zuul/zuul-jobs master: roles: Add git-submodule-init role  https://review.opendev.org/c/zuul/zuul-jobs/+/87153900:07
clarkbinfra-root I've just updated the meeting agenda page. I'll give it another 10 minutes or so before sending it out if anyone has anything else to add00:09
clarkbfwiw I'm punting on the rescue image stuff beacuse I just don't have bw for it now00:20
clarkb(so removed it from the agenda)00:20
clarkbI've dug around in the gitea source and I think https://github.com/go-gitea/gitea/blob/v1.18.3/modules/indexer/code/bleve.go#L339-L348 is where we get a Match query or a "Fuzzy" query. Those map onto prefix and phrase match here: http://blevesearch.com/docs/Query/#match-phrase:8f767fbc41af8ff1ddcf4c60ed8c0fe900:40
clarkbbleve is capable of doing regex queries: http://blevesearch.com/docs/Query-String-Query/ but I don't think gitea is exposing that00:40
clarkbianw: ^ fyi00:41
fungiso that might be an easy improvement to propose00:42
clarkbpossibly via another match type. However, another considering to make here is that they also support elasticsearch as the indexer and I'm pretty sure it odesn't do regexes. THat may make them wary of supporting regexes at all?00:43
clarkbheh nevermind elasticsearch does regex queries now00:43
clarkband now to send that meeting agenda I said I'd send before getting nerd sniped on code searches00:46
clarkbhttps://github.com/go-gitea/gitea/blob/v1.18.3/modules/indexer/code/bleve.go#L142 I think this also confirms that fuzzy in this case largely means ignore case00:52
clarkbfungi: ^ I think you pointed out that appeared to be the behavior00:52
fungiyeah, that seems to match my observations01:01
funginothing else i could think of which would count as "fuzzy" seemed to work anyway01:01
opendevreviewMichael Kelly proposed zuul/zuul-jobs master: roles: Add git-submodule-init role  https://review.opendev.org/c/zuul/zuul-jobs/+/87153901:11
opendevreviewMichael Kelly proposed zuul/zuul-jobs master: roles: Add git-submodule-init role  https://review.opendev.org/c/zuul/zuul-jobs/+/87153904:53
*** ysandeep|out is now known as ysandeep05:05
opendevreviewMichael Kelly proposed zuul/zuul-jobs master: roles: Add git-submodule-init role  https://review.opendev.org/c/zuul/zuul-jobs/+/87153906:02
ianwkevinz / others : (kevinz hope you are enjoying new year and do not read this :)06:28
ianwi'll send an email, but in the new linaro cloud i basically did what i mentioned in correspondence about resizing things06:29
ianwi deleted all the cinder volumes and started again.  i made a 150g partition and then made that the cinder vg -- it worked and i recreated the mirror volume on that, reattached it to the mirror, and it seems happy06:29
ianwthe rest of it (the remaining space after that 150g partition, and the second nvme drive) i put into another vg called "openstack"06:30
ianwthat's mounted on /opt/openstack.  i stopped nova+glance, made a nova + glance directory, copied the old volume data into it, set the relevant datadir_volume in the config, redeployed kolla and ... to my surprise it seems to have "just worked"06:32
ianwi've removed the nova_compute and glance volumes from the root partition, everything is still working06:32
ianwi'm out of time for today but glance seems to be receiving images from nb0406:36
ianwthis is the biggest machine i've played with kolla on.  very impressed all round :)06:36
*** ysandeep is now known as ysandeep|lunch07:39
*** ysandeep|lunch is now known as ysandeep08:10
*** jpena|off is now known as jpena08:35
*** ysandeep is now known as ysandeep|afk08:55
*** ysandeep|afk is now known as ysandeep09:07
*** cloudnull2 is now known as cloudnull09:09
*** rlandy|out is now known as rlandy11:09
*** dviroel|out is now known as dviroel11:18
*** ysandeep is now known as ysandeep|afk11:22
*** ysandeep|afk is now known as ysandeep12:11
*** ysandeep is now known as ysandeep|afk13:43
*** dasm|off is now known as dasm13:58
pojadhavfolks, please update agenda if any @ https://hackmd.io/iraYQWGBT4qPCKH0VNG31A#2023-01-24-Community-Call for today's community call14:41
fungipojadhav: wrong channel?15:00
pojadhavfungi, yeah.. sorry15:01
*** dviroel is now known as dviroel|lunch15:19
*** ysandeep|afk is now known as ysandeep|out15:24
clarkbmy tuesday mornings of meetings have somehow become tuesday mornings with a hole I can do other things in16:17
fungimine became openstack security advisory time16:24
*** dviroel|lunch is now known as dviroel16:30
clarkbianw: fyi I WIP'd https://review.opendev.org/c/opendev/jeepyb/+/869873 due to your discovery of the RO projects resulting in errors on push. We need to handle those seprately so they don't block everything else16:35
clarkbMostly bookkeeping at this point as I'm not sure what the best way to handle that is. Maybe we just need to remove RO projects from pojects.yaml?16:36
clarkbI'll have to think on this one a bit16:36
fungii wonder if we can do a bit of parsing to detect ahead of time that the ack is read-only and then skip it if so16:42
clarkbya we could search the acl file contents for the read only = true content (or whatever that actually is)16:42
clarkbthat might be the best way to handle it since its explicit about the behavior we want16:43
*** dviroel is now known as dviroel|doc_appt16:43
clarkbrather than trying to parse errors or something which can get confusing16:43
fungioh, though there's a catch-22 with that approach. we'll never actually apply the read-only acl for projects retired in the future because we'll skip them before we do16:46
clarkbah yup. So may need a hybrid. Push and if error check if read only is set16:47
fungiyeah, that could be the ticket16:47
*** jpena is now known as jpena|off17:21
clarkbGerrit is adding pop up alerts for changes on your alert list. People seem really excited about this and it makes me wonder why I've got such a strong reaction in the other direction17:51
clarkbyou can disable it thankfully17:51
clarkbas a user, I don't think it can be turned off globally17:52
opendevreviewClark Boylan proposed opendev/git-review master: Switch from tox to nox  https://review.opendev.org/c/opendev/git-review/+/87165218:17
clarkbI think git-review's tox.ini is nonfunctional too fwiw18:17
funginot surprising18:20
opendevreviewClark Boylan proposed opendev/jeepyb master: Switch from tox to nox  https://review.opendev.org/c/opendev/jeepyb/+/87165318:33
opendevreviewClark Boylan proposed opendev/git-review master: Switch from tox to nox  https://review.opendev.org/c/opendev/git-review/+/87165218:33
fungiclarkb: do you happen to know whether the docker's copy command recurses specified directories?18:49
fungitrying to figure out if the 404 errors from screenshotting 869091 is because the files aren't in the assets image or because of some other reason18:50
clarkbfungi: `docker cp` or the COPY directivein a Dockerfile?18:51
fungidockerfile copy command18:51
fungihttps://docs.docker.com/engine/reference/builder/#copy seems to imply i need a glob match on the files18:52
clarkbI think it is recursive since we sometimes copy entire git trees in18:52
fungiah, no, it should work18:53
clarkbCOPY . /tmp/src from zuul's dockerfile for example18:53
fungi"f <src> is a directory, the entire contents of the directory are copied, including filesystem metadata."18:53
fungiokay, so maybe the files are ending up in the wrong place somehow18:53
fungiwhat's the easiest way to check the resulting file tree for one of our built container images in a check job?18:53
clarkbfungi: they should be listed in the artifacts list and you can docker pull then run that image18:54
clarkbyou can also docker build locally yourself18:54
fungiyeah, https://zuul.opendev.org/t/openstack/build/b093f7a18d1c430ca827737e6107c58c/artifacts lists it but i'm not sure how to just download the file and unpack it18:55
fungii'm assuming it's some sort of simple archive i can just inspect with standard tools, but maybe i'm hoping for too much18:56
clarkbin this case `docker run insecure-ci-registry.opendev.org:5000/opendevorg/assets:b093f7a18d1c430ca827737e6107c58c_latest bash` is what I would probably do18:56
clarkboh but this is the image that may not have bash in it?18:56
fungiright, it's just some files i think. i simply want to know what files are inside it18:57
clarkbits a docker image which has a json manifest of layers then a bunch of layers. Its not simple18:57
fungitempted to add a task to our jobs to just do a find inside each container we build and dump that to a text file18:58
clarkbwhere is the dockerfile for this image?18:58
fungii can see from https://zuul.opendev.org/t/openstack/build/b093f7a18d1c430ca827737e6107c58c/console#3/0/12/ubuntu-jammy that it seems to think it copied the donors directory to the same place it put the other files18:59
fungibut maybe those aren't exposed to apache correctly18:59
fungithough the vhost config is pretty straightforward19:01
clarkbfungi: maybe look at the gitea image build to see how it is copying the images out of the assets container19:03
clarkbit might be referring to specific files? I don't recall19:03
fungioh! i didn't realize it didn't simply add the assets as a layer or mount the container somewhere19:04
fungiyeah, that may be the problem19:07
fungiRUN --mount=type=bind,from=opendevorg/assets,target=/tmp/assets cp /tmp/assets/* /custom/public/img/19:07
fungii think we're not doing a recursive copy that way19:11
*** dviroel|doc_appt is now known as dviroel19:13
corvusianw:  re https://paste.opendev.org/show/bRhxa0ix8C982EI5jypb/ what host?19:15
ianwcorvus: nl03, talking to linaro-regionone19:16
opendevreviewJeremy Stanley proposed opendev/system-config master: Feature our cloud donors on opendev.org  https://review.opendev.org/c/opendev/system-config/+/86909119:18
corvus>>> print(client.get_image('1afe1a96-a572-4c10-b277-7f4505ffd050'))19:28
corvusianw: ^ i get that in an interactive session; there may be an issue with that image..19:28
clarkbcorvus: ianw: could this be caused by the mismatched provider names?19:29
ianwopenstack --os-cloud=opendevzuul-linaro image list | grep 1afe1a96-a572-4c10-b277-7f4505ffd05019:30
ianwthat is interesting, because that image doesn't exist ...19:30
ianw... but ... this is a corner case because this cloud was stuck uploading images as it was out of disk19:31
corvusi got the image id from nodepool image-list19:31
ianwyesterday i reworked all the storage, which allowed images to start uploading again19:31
corvusso nodepool thinks it's a ready image19:31
corvusi may not understand the cloud/provider names here19:32
corvusso i'm not 100% sure i'm talking to the right cloud19:32
corvusin the nodepool image list, i see nodepool provider names 'linaro' 'linaro-regionone' and 'linaro-us-regionone'19:33
corvus(the failing server was being launched in linaro-regionone19:33
ianwyeah i have forgotten the "-regionone" on the builder config19:33
corvusokay, provider 'linaro-regionone' uses cloud 'linaro' region 'regionone'19:34
corvusso my interactive session was against the correct cloud (the same cloud+region used by the nodepool 'linaro-regionone' provider) so i think my method of verifying that the image does not exist in that provider is correct, and that agrees with ianw's listing.19:35
corvussorry for the detour, just wanted to make sure i got that straight19:35
corvusso i think the state is: the launcher is failing to boot the nodes because the image doesn't exist (so no bug there except for maybe we should catch that and return a specific error).19:36
ianwyeah, i have dropped the -regionone in one of the configs.  we will fix that up once we've removed the old cloud19:37
ianw(we being i :)19:37
corvusianw: so is the actual error that the builder has a mismatched provider name and uploaded an image to one cloud but set the provider to a different name?19:37
ianwyep, thanks, that makes sense.  i was surprised to see a traceback19:37
ianwcorvus: i'm not sure what state that image was in, because the cloud ran out of disk for glance to upload images.  so some were stuck at one point19:38
corvusokay cool.  yeah, i think we can definitely catch that error explicitly.19:38
ianwin this case i think we can delete it; it's definitely not from general normal operation it went missing19:39
corvusoh interesting.  i'd be surprised if nodepool would have marked it as ready without the cloud telling it so, but also, if there was a problem, it would not be the first time a cloud lied to us.19:40
opendevreviewMerged openstack/project-config master: nodepool: empty linaro-us cloud  https://review.opendev.org/c/openstack/project-config/+/87122019:44
ianwi've "nodepool image-delete --provider linaro-regionone --build-id 0000058743 --upload-id 0000000001 --image ubuntu-focal-arm64" 19:45
ianwthat is now deleting19:48
ianw| 0000058743 | 0000000001 | linaro-regionone    | ubuntu-focal-arm64        | ubuntu-focal-arm64-1673833826        | 1afe1a96-a572-4c10-b277-7f4505ffd050 | deleting | 00:00:03:15  |19:48
ianwbut i wonder if the missing image will hang that?19:48
opendevreviewJoshua Watt proposed zuul/zuul-jobs master: doc: docker-image: Add recommended dependency  https://review.opendev.org/c/zuul/zuul-jobs/+/87165719:57
fungii'll go ahead with the mm3 containers restart now19:58
ianw1afe1a96-a572-4c10-b277-7f4505ffd050 | deleting  | 00:00:32:01  |20:16
ianwi'm guessing this isn't going to go away normally20:17
fungigrr, still 404... https://zuul.opendev.org/t/openstack/build/d3ce29c9ee784a4a9af92e06085bb9f6/log/gitea99.opendev.org/apache2/gitea-ssl-access.log#63020:37
ianwmay donors needs a trailing / or something to copy the whole directory?20:42
opendevreviewJeremy Stanley proposed opendev/system-config master: Feature our cloud donors on opendev.org  https://review.opendev.org/c/opendev/system-config/+/86909120:42
fungiianw: not based on my local attempts, but that ^ sets the cp to verbose so i can see if it's missing at the source side or not20:42
ianwfungi: they are copied into the root -> https://paste.opendev.org/show/b1vr4yK8eh3OMGHqvVjJ/20:47
ianwfrom that log it looks like "GET /assets/img/donors/rackspace.jpg"20:47
fungioh, is that what docker means by copy being recursive but only copying files?20:48
ianwso maybe drop the donors/ ?20:48
ianwmaybe? :)  it does interesting things with ADD depending on what you specify too, sometimes extracting things that end with .gz and sometimes not, etc.20:49
fungi"Note: The directory itself is not copied, just its contents." https://docs.docker.com/engine/reference/builder/#copy20:49
fungii guess we'd need a separate step to create the directory and then copy into it20:50
ianwyeah, i just pulled insecure-ci-registry.opendev.org:5000/opendevorg/assets:d7e756c0eaa643a58cf53eef79baf9da_latest ; ran create and exported the result to see 20:51
opendevreviewJeremy Stanley proposed opendev/system-config master: Feature our cloud donors on opendev.org  https://review.opendev.org/c/opendev/system-config/+/86909120:52
funginot sure if copy will create a target directory if it doesn't exist20:52
fungi"If <dest> doesn’t exist, it is created along with all missing directories in its path."20:53
fungiokay, hopefully this will do the trick20:53
ianw" In this case, if <dest> ends with a trailing slash /, it will be considered a directory and the contents of <src> will be written at <dest>/base(<src>)."20:55
fungiyeah, not sure how to parse that part ;)20:55
ianwme either, and you did have a trailing / ... so20:57
ianwPredicted remaining provider quota: {'co21:07
ianwmpute': {'cores': 104, 'instances': 33, 'ram': -6144}}21:07
fungii wonder how well the nodes will run on negative ram21:08
ianwyes it's an interesting one21:09
ianw# openstack --os-cloud=opendevzuul-linaro quota show | grep ram21:10
ianw| ram                   | 51200    21:10
ianwwe can put a zero on that21:20
fungithat would be very nice21:24
clarkbI think if you do a limits show it will show you what it thinks it is using21:35
opendevreviewJoshua Watt proposed zuul/zuul-jobs master: use-buildset-registry: Prepend buildset registry to mirrors  https://review.opendev.org/c/zuul/zuul-jobs/+/86976021:40
opendevreviewClark Boylan proposed opendev/jeepyb master: Raise an error if acl pushes fail  https://review.opendev.org/c/opendev/jeepyb/+/86987321:41
ianwshould i restart the executors with the images with updated skopeo?21:54
opendevreviewJoshua Watt proposed zuul/zuul-jobs master: upload-container-image: Add option to stage in separate repository  https://review.opendev.org/c/zuul/zuul-jobs/+/87166421:54
opendevreviewJoshua Watt proposed zuul/zuul-jobs master: promote-artifactory-image: Add role  https://review.opendev.org/c/zuul/zuul-jobs/+/87166521:54
fungiianw: what did we need the updated skopeo for?22:02
ianwfungi: to upload nodepool images that are made with buildx22:02
ianw(and any other images using buildx, but nodepool's the one i'm aware of)22:03
ianwi need new nodepool images to fix rocky linux builds in the gate22:03
fungigot it22:03
fungiyeah, sounds like a good reason for a rolling restart sooner rather than waiting for the weekend22:03
ianwi'll take a look in a bit; just trying to sort out linaro launching nodes22:04
opendevreviewMerged openstack/project-config master: nodepool: drop linaro-us  https://review.opendev.org/c/openstack/project-config/+/87119622:06
opendevreviewIan Wienand proposed openstack/project-config master: nodepool: fix new linaro provider name in nb04  https://review.opendev.org/c/openstack/project-config/+/87166622:07
ianwhttps://zuul.opendev.org/t/zuul/build/942b6fdabbcd4989a44717b0599f3d14 is actually another job that fails, but same reason22:29
clarkbI'm reviewing the copycondition change now. Wondering if openstack should try and standardize this and put it in their central acl inherited by everything22:29
clarkbanyway doing that wouldn't be for us to solve just thinking out loud as I go22:29
ianwyeah the usage is inconsistent, but i think sometimes not always intended that way22:31
clarkbianw: I +2'd it but didn't approve because i think it may be a good idea to announce the change before hadn (not necessarily with a ton of lead time) just so that users can call out behavior changes if we misinterpreted gerrit docs22:35
clarkb(I left the same comment on the change)22:35
ianwsure i can send a mail22:35
ianwi'm going to let 871196 apply, then manually fix up that bad provider name, and we can merge 87166622:36
ianwif the new cloud still isn't picking up nodes after that, then i'm starting to really be at a loss for what's wrong.  afaict nodepool thinks it has enough quota to run vm's, it's just not accepting any node requests22:37
clarkbthe nodepool logs are pretty good if it is a quota thing (it logs what it thinks the quota is and how much room it has)22:39
clarkbthat should be able to help rule things out too22:40
corvusis there a specific request it should be handling but isn't?22:42
corvus(cause looking into "mostly idle provider is idle" is tricky to find interesting log entries :)22:43
*** dviroel is now known as dviroel|out22:44
fungiyay! http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f88/869091/6/check/system-config-run-gitea/f88ff85/bridge99.opendev.org/screenshots/gitea-main.png22:48
funginow i can clean up my useless debugging and un-wip finally22:48
opendevreviewJeremy Stanley proposed opendev/system-config master: Feature our cloud donors on opendev.org  https://review.opendev.org/c/opendev/system-config/+/86909122:50
*** rlandy is now known as rlandy|out23:01
*** dasm is now known as dasm|off23:09
ianwok, there's a bunch of images like23:29
ianw| 0000042108 | 0000000001 | linaro-regionone    | centos-8-stream-arm64     | centos-8-stream-arm64-1673834289     | 18cf4707-7156-40eb-b9da-d59cc593eea5 | deleting  | 00:00:05:27  |23:29
ianwi think that may have come from nb03 when i started this a long time ago23:29
ianwoh interesting, they got reaped with the name change23:31
ianwnb04 is now uploading with the correct name23:31
opendevreviewMerged openstack/project-config master: nodepool: fix new linaro provider name in nb04  https://review.opendev.org/c/openstack/project-config/+/87166623:38
ianwso i'm thinking of running zuul_rolling_restart in a root screen on bridge?  is that the best way to get the executors restarted?23:39
ianwafter a zuul_pull23:40
fungithat sounds right to me... clarkb ^ ?23:54
clarkbianw: that playbook will do 6 executors at a time23:54
clarkbso it is quicker than the reboot playbook.23:54
clarkbthings are not super busy so that should be fine. Demand on zuul is the biggest consideration there. Also it won't do the scheduler (I don't think you need the scheduler) so a should be good23:55
ianwnope just executors23:55
ianwok, it's running in a root screen23:56

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!