| *** bauzas_ is now known as bauzas | 02:33 | |
| frickler | clarkb: iiuc this was the solution https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434/10..14/test-prepare-keys.yml | 05:16 |
|---|---|---|
| noonedeadpunk | clarkb: yeah, I did. So we relied on variable ansible_user_id to detect the user. But somehow while running as `zuul` user `ansible_user_id` was still detected as `root`. So ansible_user_id: root while lookup('env', 'USER') was returning zuul | 06:58 |
| noonedeadpunk | kind of weird and smells like some facts cache when they were gathered with become or smth like that... | 06:59 |
| noonedeadpunk | or I did smth wrong in the patch... but at least I've figured out what's going on | 07:00 |
| opendevreview | Merged openstack/project-config master: Drop repos with config errors from x/ namespace https://review.opendev.org/c/openstack/project-config/+/923509 | 09:04 |
| *** bauzas_ is now known as bauzas | 09:14 | |
| *** elodilles_ooo is now known as elodilles | 12:32 | |
| fungi | noonedeadpunk: zuul does maintain a fact cache for the build so that facts set in one playbook in the build can be referenced by later playbooks in the same build, so maybe yes | 12:37 |
| *** bauzas_ is now known as bauzas | 12:40 | |
| *** Guest650 is now known as beand | 12:58 | |
| *** bauzas_ is now known as bauzas | 13:17 | |
| *** bauzas_ is now known as bauzas | 13:42 | |
| *** bauzas_ is now known as bauzas | 14:15 | |
| clarkb | noonedeadpunk: that sounds like a fun bug, glad you figured it out | 15:35 |
| dansmith | clarkb: how much work is it to have some opendev-hosted git repo for a thing? I've got a set of good and bad (pretty small) disk images and I want to put them somewhere so we can reference them for testing | 15:38 |
| clarkb | dansmith: its not bad you basically push a change to openstack/project-config to create the project in gerrit. That can seed the repo with existing content and then you push new things to it via typical gerrit review processes | 15:39 |
| clarkb | a lot of the extra steps that are needed for things in openstack aren't required in the general case | 15:39 |
| clarkb | https://docs.opendev.org/opendev/infra-manual/latest/creators.html this document is quite verbose because it tries to cover all the things but the end result shouldn't be complicated for something like what you describe | 15:40 |
| dansmith | do you think that's a reasonable thing for a repo of images? the largest one I have is 10MiB because it's a raw, but probably could be trimmed down.. the rest are teensy | 15:40 |
| clarkb | dansmith: assuming these are media images and not disk images then yes. zuul has a media repo with some small stuff in it like that | 15:42 |
| clarkb | in opendev's case we just put them in opendev/system-config along with everything else. but most are svgs so they aren't really binary data | 15:42 |
| clarkb | dansmith: https://opendev.org/zuul/zuul-website-media is the example I'm thinking of | 15:43 |
| dansmith | clarkb: oh sorry I'm talking about sample disk images | 15:43 |
| dansmith | they're all basically images of blank disks in various formats and variations so no real content | 15:43 |
| clarkb | oh I see. Not anything anyone would really be booting, but useful in a testing context. The main concerns I would have with disk imagse are that typically they are binary blobs that people want ot update regularly which is a bad fit for git | 15:44 |
| clarkb | there may also be concern about shipping what is executable code without source? fungi may have thoughts on that angle specifically. | 15:45 |
| dansmith | well, in this case I kinda want them versioned even though you can't diff them | 15:45 |
| clarkb | dansmith: ya the problem is more on the git hosting side | 15:45 |
| clarkb | repos like that balloon | 15:45 |
| dansmith | ah okay | 15:45 |
| dansmith | I can just throw them in github, I just figured maybe better to have them in opendev somewhere | 15:45 |
| clarkb | because git can't be smart about storage, it basically has to have an entire copy of everythign each time you make a change. Its probably fine if updates are infrequent but if say its a cirros or ubuntu or whatever image that gets updated regularly that can be a problem | 15:46 |
| fungi | yeah, basically git is not a great storage medium for large binary blobs | 15:47 |
| dansmith | no, these have no content, so no real reason to ever change an image unless it's like determined to not be in the format we thought or something like that | 15:47 |
| fungi | though if this is small "broken" images used as test vectors, it's probably not bad | 15:47 |
| fungi | this is pretty common in projects that do input validation checks | 15:47 |
| clarkb | if they are small and only update when there is a bug rather than to keep up with linux development its probably fine | 15:47 |
| dansmith | well, it is from my perspective of making sure they don't change without us knowing.. a file of hashes could also change, and I could put the file of hashes in git and have them reference files elsewhere, that's just getting more complicated than I want | 15:48 |
| dansmith | and github will work | 15:48 |
| dansmith | can we just delete the repo without much fanfare if it gets out of hand? | 15:48 |
| fungi | e.g. security scanners embedding samples of malware-like files that exercise pattern matches, cryptographic libraries including pre-generated keys, etc | 15:48 |
| dansmith | fungi: right it's kinda like that | 15:49 |
| dansmith | I could also just make the repo a set of scripts to generate the files, it just means that anyone testing them have to have the tooling required to do that (i.e. not great for a tempest run that just wants to point to them via url" | 15:49 |
| fungi | i wouldn't have any problem with it. assuming they're custom made and not taken from somewhere else, just put them under an ultra-permissive license like pd or cc0 | 15:49 |
| dansmith | but doable | 15:50 |
| dansmith | most of them are just from qemu-img create with various options | 15:50 |
| fungi | content licenses for files like that should be fine since they're not really source code | 15:50 |
| fungi | and no need to host them on github in my opinion, could just include them in oslo or in a separate repo under the oslo team if preferable | 15:51 |
| frickler | do you need to hand craft the images or could git contain just the code to generate them? then we could publish the results as build artefacts | 15:51 |
| dansmith | meh, I think putting them into oslo is less good... especially if it adds heft to people cloning that repo or the repo itself | 15:51 |
| fungi | that's a good point too, a publication step could just push them to tarballs.o.o or something too if that's preferable | 15:52 |
| clarkb | ya I was under the impression they were real images, but I agree for these more static less executable as a bootable unit things this seems far less problematic | 15:52 |
| dansmith | frickler: that's what I mentioned above.. some of them have to be hand-crafted, although I can script anything my fingers can do of course, so I could go that way, it's just less helpful for test scenarios that just want a url they can point to | 15:52 |
| dansmith | clarkb: no there's not any content in any of them really | 15:52 |
| clarkb | the upside to using a separate repo is that you can rewrite the git history more easily | 15:52 |
| dansmith | they're images of a blank disk in multiple formats | 15:52 |
| clarkb | which can be useful when dealing with binary artifacts | 15:52 |
| clarkb | dansmith: ack then ya I think that should probably be fine under both of the concerns I listed previously | 15:53 |
| dansmith | let me try to just script all their creation and see how that goes | 15:53 |
| dansmith | is there some place we could just manually push image binaries to that get created from this script so they're curl'able? | 15:53 |
| fungi | scripting does make it a little easier for others to reproduce and confirm they're what they claim to be, but i wouldn't consider lack of scripted routine for them as a show-stopper either | 15:54 |
| dansmith | fungi: yeah, I was going to document the creation but code is obviously nice | 15:54 |
| dansmith | I'll just have to see how annoying it gets. the only way I've found a vmdk with a stream footer is with an actual image from vmware, but I did hack it up in a unit test already | 15:55 |
| fungi | we could stick them in afs, so they're served from static.o.o or any of the sites it hosts, but really that's just what a publication job in zuul would do too | 15:55 |
| dansmith | I dunno if vmware would actually eat that image though | 15:55 |
| dansmith | okay well, let me work on the script and we'll see from there | 15:55 |
| fungi | also publishing them that way would be possible regardless of whether they're checked directly into the repo or generated by the publication job on the fly, you could even have a mix of both if you needed | 15:56 |
| dansmith | yeah, if that's doable then generating on publish would be fine | 15:56 |
| fungi | so maybe for some images that are easy to reproduce programmatically you do that, and check the others in that need hand-massaging | 15:56 |
| dansmith | ack | 15:56 |
| clarkb | oh ya hosting on afs like tarballs and docs etc work work if you can script the generation of the artifacts | 15:57 |
| fungi | and even if you can't script some of them, the publish job can just upload those blobs directly from the checked-out worktree too | 15:57 |
| *** dasm is now known as Guest1321 | 16:07 | |
| opendevreview | Clark Boylan proposed openstack/project-config master: Set linaro cloud's max servers to 0 https://review.opendev.org/c/openstack/project-config/+/925029 | 16:38 |
| opendevreview | Carlos Eduardo proposed openstack/project-config master: Implement manila-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/924430 | 18:05 |
| dansmith | clarkb: fungi: so is x/os-test-images appropriate in gerrit/projects? | 18:34 |
| fungi | dansmith: any way it could be governed by an openstack project team, like oslo or glance? then it could go in openstack/ instead. the x/ namespace was really just a convention we used for the repos formerly in openstack/ that weren't claimed by openstack project governance and whose maintainers didn't get back to us when asked what new namespace(s) they wanted created, but if this is | 18:44 |
| fungi | going to be something that's not just used by openstack we could pick a new namespace to group it and any similar efforts under. but the name implies it's for openstack, so would be better to just make it part of openstack officially | 18:44 |
| dansmith | it would be fine with me to call it a glance thing, and I think that'd be appropriate | 18:45 |
| fungi | we've been trying to avoid new additions to x/ because just having a random dumping ground for unclaimed projects isn't great | 18:45 |
| dansmith | however I guess that needs some buy-in from the ptl yeah? | 18:45 |
| fungi | yeah, but you could propose the addition starting there. a +1 from the glance ptl or glance infra liaison would suffice | 18:46 |
| dansmith | okay sounds good | 18:46 |
| fungi | project creation and the corresponding governance change can happen in parallel | 18:46 |
| dansmith | and then I can re-use glance-core I suppose | 18:46 |
| dansmith | how much governance change is required? | 18:46 |
| dansmith | I just don't want to generate paperwork for people for what likely will see very little change over the years... | 18:46 |
| dansmith | (undue amounts of paperwork I mean) | 18:47 |
| dansmith | and presumably the release cadence for this can be whenever-we-damn-well-please yeah? | 18:47 |
| fungi | just a one-liner adding it to the glance section of the reference/projects.yaml in openstack/governance i think. repo additions fall under house rules for fast approval with no majority required | 18:47 |
| dansmith | okay | 18:47 |
| fungi | i'm pretty sure if it's independently (or never) released you don't need to add anything to openstack/releases | 18:48 |
| dansmith | well, whatever we need to do to run the generate script and push the files out to afs or whatever you guys said above :) | 18:49 |
| dansmith | so skimming that extensive doc, it seems like if I'm adding under glance and re-using glance-core, I just need the few lines of the project in there, with name,desc and the glance-core acl is that right? | 18:51 |
| dansmith | and then I can let it create an empty repo and then clone and populate it myself? There's no useful history so far to import since I just wrote the generate script in the last few hours | 18:52 |
| fungi | running the publish job can just be done each time a change merges to the repo, no need for git tags unless you want them for explicit repo versioning purposes | 18:57 |
| dansmith | excellent | 18:57 |
| fungi | technically the typical pattern for something like this would involve two jobs, one that builds the files and/or tests them in check and gate pipelines, then one that runs in the post pipeline to copy the resulting files generated from the gate pipeline build into afs for publication. that gives you the ability to do things like acceptance testing, exercising them with image-inspector | 19:04 |
| fungi | or even devstack/tempest to avoid having a change in the test data break common jobs | 19:04 |
| fungi | er, runs in the promote pipeline i mean, not post pipeline | 19:04 |
| fungi | more or less the same, just promote works based on changes merged rather than branch state updates | 19:04 |
| fungi | but promote pipeline builds give us the ability to easily tie back to the corresponding gate pipeline builds which preceded them | 19:05 |
| fungi | so that you don't wind up building things twice | 19:05 |
| fungi | this is, for example, how our docs publishing jobs work. actual docs built in gate, then archived as temporary artifacts which promote pipeline jobs copy into afs for publication if the change successfully merges | 19:06 |
| dansmith | yeah making them be re-generated on check is a good idea for sure | 19:08 |
| dansmith | generated and thrown away I mean, and published on gate or whatever | 19:08 |
| opendevreview | Dan Smith proposed openstack/project-config master: Add openstack/os-test-images project under glance https://review.opendev.org/c/openstack/project-config/+/925043 | 19:09 |
| opendevreview | Dan Smith proposed openstack/project-config master: Add openstack/os-test-images project under glance https://review.opendev.org/c/openstack/project-config/+/925043 | 19:12 |
| dansmith | fungi: I didn't add tags: per the instructions since I didn't see those anywhere else in the file.. perhaps that has changed and the doc needs updating? | 19:13 |
| fungi | dansmith: probably yes, governance tags haven't been a thing for a while now i think | 19:21 |
| dansmith | The tl;dr from the earlier convo is that I've got a small generator script that reads a yaml manifest of instructions to create all the images | 19:25 |
| dansmith | aside from a little bit of python for one very complex one it's mostly qemu-img and dd commands | 19:25 |
| dansmith | and it can generate a json file or something that can be published next to the actual images to make it easy for tests to see what all images are available | 19:26 |
| fungi | sounds awesome | 19:41 |
| dansmith | fungi: the -1 I'm getting on governance.. is that expected until the project-config patch merges? | 20:08 |
| fungi | dansmith: yes, i believe there is a governance check job that tests whether the repository exists, so basically the sequence will be we merge the project-config change and then once the deploy job for that completes recheck the governance change | 20:09 |
| dansmith | yeah I wasn't sure if the depends-on should be accounting for that or not, but that's what's failing, so I assume not | 20:09 |
| fungi | right, because it doesn't just depend on the change merging, but actually being deployed to production | 20:10 |
| dansmith | oh, I messed up the depends-on anyway! | 20:10 |
| opendevreview | Dan Smith proposed openstack/project-config master: Add openstack/os-test-images project under glance https://review.opendev.org/c/openstack/project-config/+/925043 | 21:05 |
| *** bauzas_ is now known as bauzas | 21:11 | |
| opendevreview | Clark Boylan proposed openstack/project-config master: Use the osuosl mirror for deb packages in image builds https://review.opendev.org/c/openstack/project-config/+/925048 | 22:00 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Remove labels and diskimages from the linaro cloud https://review.opendev.org/c/openstack/project-config/+/925049 | 22:00 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Remove linaro cloud from Nodepool https://review.opendev.org/c/openstack/project-config/+/925050 | 22:00 |
| opendevreview | Merged openstack/project-config master: Implement manila-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/924430 | 22:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!