pabelanger | mordred: zuul_site_traceroute_host? | 00:00 |
---|---|---|
pabelanger | I see that in site-variables.yaml | 00:00 |
jeblair | mordred: fwiw, that worked for me now. so let's just keep an eye out. | 00:00 |
mordred | pabelanger: yup. that would be it | 00:01 |
mordred | pabelanger: roles/validate-host/defaults/main.yaml - the defaults are backwards | 00:02 |
pabelanger | mordred: okay, I'll let you push up the patch :) | 00:04 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Use zuul_site_traceroute_host properly https://review.openstack.org/502197 | 00:04 |
mordred | pabelanger: I tried to stage it as a three step rename ... but I clearly failed | 00:05 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove reference to unused variable https://review.openstack.org/502199 | 00:05 |
mordred | pabelanger: we'll have to force-merge the first one | 00:06 |
pabelanger | kk | 00:06 |
mordred | and sorry about that | 00:06 |
mordred | pabelanger: that line was SUPPOSED to be 'zuul_traceroute_host: "{{ zuul_site_tracereoute_host|default(omit) }}"' | 00:06 |
mordred | so that we could seamlessly roll out the change | 00:06 |
jlk | hrm, does zanata-cli have it's own secret it needs? | 00:06 |
pabelanger | mordred: ah, ya | 00:07 |
mordred | turns out 'zuul_site_traceroute_host: "{{ zuul_tracereoute_host|default(omit) }}"' is backwards | 00:07 |
mordred | jlk: yah - and the script probably assumes that secret exists already on the proposal slave as managed by puppet | 00:07 |
jlk | yup. | 00:07 |
jlk | so... I might have to punt on this since I don't think we have that secret prepared. | 00:07 |
mordred | kk. I can pick it up | 00:08 |
jlk | I'll leave some hints | 00:08 |
mordred | woot | 00:08 |
jlk | or maybe I can do a partial WIP that you could pick up and run with | 00:09 |
pabelanger | ya, I can help with secret bits too | 00:09 |
jlk | can you list multiple parents? | 00:11 |
mordred | jlk: nope. single inheritance only | 00:12 |
jlk | hrm, okay. So this needs both the parts that copy slave scripts AND the parts that set up the tarball secrets | 00:12 |
jlk | hrm. | 00:12 |
jlk | I can just call that as a pre I suppose | 00:12 |
mordred | jlk: finding a happy medium between jobs reusing/sharing roles and and reusing jobs is an interesting balance ... | 00:12 |
jlk | the script copy | 00:12 |
mordred | jlk: yah | 00:13 |
jlk | ahaha crap | 00:14 |
pabelanger | mordred: ya, force merge on 502197 needed | 00:14 |
jlk | the pre playbook from proposal has a task to write out the ssh key, which we wouldn't have unless we made that a parent. | 00:14 |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Use zuul_site_traceroute_host properly https://review.openstack.org/502197 | 00:15 |
pabelanger | jlk: is there an etherpad of job structure? | 00:15 |
jlk | pabelanger: not yet. THe job I'm looking at requires slave-scripts prepared (which exists in the proposal pre playbook), and it needs a secret to upload tarballs. That said, I _think_ I can just make the translation upload job it's own thing that has the parent on the tarball secret | 00:16 |
mordred | jlk: yah - I think whatyou said there sounds right | 00:16 |
jlk | how exactly do we do 'post-run' jobs? | 00:17 |
jlk | something that takes the place of a 'publisher' ? | 00:17 |
mordred | jlk: post-run playbooks | 00:18 |
jlk | ah, so we can't link jobs? | 00:18 |
jlk | it's the post-run that needs the tarball secret | 00:18 |
mordred | jlk: jobs can totally depend on other jobs (it's a full DAG even) | 00:18 |
mordred | jlk: but currently there is no pass-artifact-between-jobs | 00:18 |
jlk | ah right | 00:19 |
jlk | that's the catch | 00:19 |
mordred | jlk: in general we've been tackling this via just making independent jobs that re-use roles and secrets | 00:19 |
jlk | so what I said doesn't exactly work | 00:19 |
jlk | this job needs elements from two distinct parents. Unless I copy-pasta | 00:19 |
jlk | oh, I suppose I can just copy the secret. Derp. | 00:19 |
mordred | well - and the secret is a thing you can refer to by name | 00:19 |
jlk | so long as the secret doesn't need any preparation | 00:19 |
mordred | so you can just have two jobs that request the same secret | 00:20 |
jlk | unless that secret needs some prep, like writing it to the filesystem in some way | 00:20 |
mordred | well - if it does, that can go into a role and you can add that to a pre-playbook for the job or something perhaps? | 00:20 |
jlk | oh yeah. | 00:20 |
mordred | jlk: (it seems like you got one of the fun ones! it took pabelanger and I a couple of weeks to figure out python tarball publishing originally :) ) | 00:21 |
jlk | hah, that's what I'm cribbing a bit from | 00:21 |
mordred | ++ | 00:21 |
mordred | it's the right thing to crib from | 00:21 |
jlk | the publish-openstack-artifact job has no run, just a secret. Is that... | 00:22 |
jlk | hrm. | 00:22 |
jlk | it has a post-run | 00:22 |
mordred | yah. is uses the secret in the post-run | 00:22 |
jlk | the hell? where is the add-fileserver role located? | 00:23 |
pabelanger | jlk: mordred: ya, i really found it helpful when we all got into etherpad, and whiteboard'd what the job looked like | 00:24 |
jlk | it's referenced in playbooks/publish/openstack-artifacts.yaml | 00:24 |
pabelanger | we when through a few iterations, but eventually got it | 00:24 |
pabelanger | jlk: zuul-jobs | 00:24 |
jlk | oh | 00:25 |
clarkb | jlk: ya I've been struggling with that too. Makes me wonder if we shouldn't allow for scoped roles | 00:26 |
clarkb | zuul-jobs/add-fileserver or something | 00:26 |
pabelanger | clarkb: why not? | 00:26 |
clarkb | pabelanger: I think theo nly thing that might prevent that from working reliably is ansible role lookups | 00:27 |
clarkb | I have no idea if that would just work | 00:27 |
clarkb | (also it potentially breaks inheritance? | 00:27 |
pabelanger | we could do role: zuul-jobs.add-fileserver in playbook, to better indicate where it came from. But we'd need to make some changes to support that | 00:27 |
mordred | it could be done - but I think there are some fuller things to discuss in that area | 00:27 |
pabelanger | ya, I agree it is confusing in some cases | 00:28 |
pabelanger | especially when first looking at stuff | 00:28 |
clarkb | my biggest problem is git grep doesn't work | 00:29 |
clarkb | so then you basically go now what | 00:29 |
clarkb | (and usually results in find/proper grep) | 00:29 |
pabelanger | ya, the reason to have the role in zuul-jobs, is to make it available to others. Moving it into project-config, would be a problem for downstream things | 00:29 |
pabelanger | I like the idea of namespacing roles with project they come from, since that is just a directory name local to playbook | 00:30 |
pabelanger | something to dive into at PTG for sure | 00:31 |
jlk | I think it just means that in SCM it would be like git.openstack.org/openstack-infra/zuul-jobs/zuul-jobs/roles/foo | 00:32 |
jlk | the zuul-jobs repo would have a dir of "zuul-jobs" | 00:32 |
jlk | or we could alter how they get put onto the filesystem/search path | 00:32 |
pabelanger | ya, IIRC, we are doing symlinking for roles into jobdir | 00:32 |
clarkb | if you set your search path to include ../ it would allow you to avoid needing redundant zuul-jobs dirs right? | 00:33 |
pabelanger | so, that is possible | 00:33 |
clarkb | (but maybe that gives access to things you don't want to expose?) | 00:33 |
jlk | argh | 00:34 |
jlk | hahah | 00:34 |
jlk | this publish-artifacts-to-fileserver role is _almost_ what I need | 00:34 |
jlk | except it only publishes what's in the artifacts/ directory | 00:34 |
jlk | I need it to publish ... hrm. | 00:35 |
*** jkilpatr has quit IRC | 00:35 | |
jlk | I wonder if I can prep this so that it takes in the translation-source/**/*.pot files and puts them in the artifacts/ dir? | 00:35 |
pabelanger | jlk: yup, I do something the same for afs-docs job | 00:37 |
pabelanger | so, likely we just need to rework existing jobs to use artifacts folder now for pubishing to tarballs.o.o | 00:38 |
pabelanger | mordred: jeblair: great, release jobs working again | 00:41 |
pabelanger | thank you | 00:41 |
jlk | Are we dealing at all with publishing to static.openstack.org, the logs/$LOG_PATH stuff? | 00:44 |
pabelanger | ya, see upload-logs role in zuul-jobs | 00:45 |
jlk | but is it basically automatic by using base? | 00:46 |
pabelanger | Yup, we have a post-run playbook from base jobs | 00:46 |
pabelanger | that is in project-config | 00:46 |
jlk | argh this is odd. zuul_fileserver_project_path is used by publish-artifacts-to-fileserver, but it's set by add_host in roles/add-fileserver/tasks/main.yaml | 00:47 |
jlk | so if you want to overload where the content gets sent.... | 00:47 |
jeblair | jlk: where do you need to send content (and what content)? | 00:51 |
jlk | "source: 'translation-source/**/*.pot'" "target: 'tarballs/'" | 00:51 |
jlk | to the tarballs.openstack.org host | 00:51 |
jeblair | why do we put that on tarballs...? | 00:51 |
jlk | ¯\_(ツ)_/¯ | 00:51 |
jlk | translations are dark art. | 00:52 |
clarkb | I think we stage them for push to zanata | 00:52 |
pabelanger | http://tarballs.openstack.org/translation-source/ | 00:52 |
clarkb | as pushing to zanata requires adminy creds | 00:52 |
jlk | maybe not | 00:52 |
jeblair | maybe we should rethink this as push to zanata from executor? | 00:53 |
clarkb | and generating .pots is potentially arbitrary python | 00:53 |
jlk | because the playbook already has a zanata push | 00:53 |
jeblair | jlk: oh, hrm | 00:53 |
clarkb | jlk: in that case I am stumpted | 00:53 |
pabelanger | so, you should be able to sync pots file to artifacts/translation-source folder on executor, and our existing artifact publisher will push up changes | 00:53 |
jlk | the playbook does a zanata push and then this upload is done as a post job | 00:53 |
jeblair | okay, i'm going to need an etherpad so i'll start one | 00:53 |
jeblair | https://etherpad.openstack.org/p/XIWAMDDcXB | 00:53 |
pabelanger | ty | 00:53 |
jlk | I was trying to figure out if anything else uses the 'tarballs/' path, but the existing artifact upload things we've done already just seem to put them into a subdir of the project name | 00:54 |
pabelanger | jlk: ya, possible we need to expose that to user | 00:55 |
pabelanger | which we did recently with afs_publisher_target | 00:55 |
jlk | pabelanger: right, I was thinking about that, which is how I found that the destination path is currently "exposed" as a host var to the added fileserver host | 00:56 |
jlk | so we could make that a variable with a default in the play that does the add_host | 00:57 |
jlk | or something else. | 00:57 |
jeblair | we need to be careful not to let cloudkitty publish something into nova's dir | 00:58 |
jlk | ah right. | 00:58 |
jlk | so letting it be host overridden is dangerous. | 00:58 |
jlk | maybe we need a copy-pasta that publishes to tarballs/ | 00:58 |
jlk | or maybe yell at translations and stop dumping them all in tarballs/ ? | 00:59 |
pabelanger | jeblair: ya | 01:00 |
jeblair | jlk, pabelanger, clarkb: https://review.openstack.org/309560 is the most relevant commit i can find about this | 01:01 |
jlk | I'm trying to find where any of these translation files might wind up on the tarballs host, and I'm failing | 01:02 |
jlk | oh | 01:02 |
*** harlowja has quit IRC | 01:02 | |
jeblair | jlk: that makes me think that proceeding with your plan of working out a way to use the tarball publishing stuff is necessary. at least it seems that publishing of pot files is an end-goal; it's not an intermediary step or anything (and explains why nothing consumes it) | 01:03 |
jlk | but... | 01:03 |
pabelanger | jlk: jeblair: with 309560 in mind, if we set path: /src/static/tarballs/translate-source for http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.yaml#n609 it should be the same publisher using add-fileserver role | 01:03 |
pabelanger | because we append project.short_name | 01:03 |
jlk | I'm really confused then by line 14 of the etherpad | 01:04 |
pabelanger | that is /src/static/tarballs on our server | 01:05 |
jlk | pabelanger: you mean duplicating the secret and making it a new secret name? | 01:05 |
jlk | oooh damn I think I get it now | 01:05 |
pabelanger | jlk: you can use the same secret, but overload path in playbook | 01:05 |
jeblair | yeah site:tarballs.openstack.org gets you as far as 'static.openstack.org:/src/static'. target:tarballs/ gets you 'static.openstack.org:/src/static/tarballs'. then the rest of the path comes from 'source' because that's copied over. | 01:05 |
jlk | because of how things work, it's going to use SOME of the source path as the destination path | 01:05 |
jeblair | jlk: yep | 01:06 |
pabelanger | ya | 01:06 |
jlk | gross | 01:06 |
*** fbo has quit IRC | 01:06 | |
jeblair | jenkins compatability :) | 01:06 |
pabelanger | hehe | 01:06 |
jlk | will a playbook var set in zuul.yaml override the secret var? | 01:07 |
jlk | I thought secrets win, so that they couldn't be altered? | 01:07 |
pabelanger | Hmm | 01:08 |
jeblair | https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html#variables | 01:08 |
jlk | (so that you couldn't trick zuul into copying the secret somewhere else by changing a host var) | 01:08 |
jeblair | jlk: secrets beat job vars | 01:08 |
jlk | yeah, so changing the host name at the job level isn''t going to work | 01:09 |
pabelanger | Ya, so because a secret path will be hard to overright | 01:09 |
pabelanger | override* | 01:09 |
pabelanger | this is one case where mordred change to break out non-secret data is better | 01:09 |
jeblair | (intentionally -- that's the cloudkitty/nova thing in action) | 01:09 |
pabelanger | yup | 01:09 |
jeblair | pabelanger: well, it's not strictly better, it's different | 01:09 |
pabelanger | jeblair: ya, different use case for sure | 01:09 |
jlk | so there is a way to do this | 01:10 |
jlk | in the role, it's kind of ugly, let me etherpad a moment | 01:10 |
pabelanger | afs_publish_target for afs-docs would be a way to do this i think. tarballs_publish_target variable, but jeblair cloudkitty / nova comment is still issue | 01:11 |
pabelanger | maybe we'd still append project.short_name | 01:11 |
jeblair | zuul_fileserver_project_path is set by one role and used by another; it's possible to change that at any point in between, so really the security here is just based on the fact that these are only used inside of a trusted playbook | 01:14 |
jeblair | so i think it would be okay for add-fileserver to take an extra parameter in building it, or for us to pass something different to the publish-artifacts-to-fileserver role. | 01:15 |
jlk | See line 53 | 01:15 |
jeblair | jlk: ya | 01:15 |
jlk | you could allow for an override, and it'll be used if set, otherwise fall back to the secret provided "path" variable. | 01:15 |
jeblair | i think my question at the moment is -- why was zuul_fileserver_project_path put in the add_host task | 01:15 |
pabelanger | jeblair: ya, I think that needs to be moved | 01:16 |
jlk | because it's used as a variable for that host? In the other role | 01:16 |
pabelanger | I believe mordred added that logic | 01:16 |
jlk | commit fa7507fd98b284adaf80791739742cbc73791c9d | 01:17 |
jlk | maybe it was done so that downstream roles don't need to access the secret directly? | 01:18 |
jeblair | what about instead of overriding the whole path, have an optional section between the secret.path and the project.short_name? | 01:18 |
jlk | I'm not entirely sure why ti's a secret anyway | 01:18 |
jeblair | that feels a little closer (for better or worse) to the idea that the root path is part of the secret | 01:18 |
jlk | could do that yeah | 01:18 |
jlk | alter the secret to have multiple vars | 01:19 |
jeblair | it helps me to think of 'secret' as 'bundle of related information' | 01:19 |
jlk | base_path: srv/static | 01:19 |
jeblair | jlk: i wasn't thinking of adding the inserted path to the secret, but rather to the add-fileserver role | 01:19 |
jlk | target_path: tarballs/ | 01:19 |
jlk | ah | 01:19 |
jlk | reduce the secret to just path: /srv/static/ ? | 01:20 |
jlk | and then let the publish-artifacts role have a default path mid-path of tarballs/ ? | 01:20 |
jeblair | jlk: that's an option, but we don't have to (our translation-source is under tarballs). so the secret can stay as is (it's the "tarball site root"), then artifact publishing has no mid-path, and translation publishing has 'translation-source/' as a mid-path. | 01:21 |
jlk | oooh right | 01:21 |
pabelanger | ya, I like the idea of keeping secret.path with tarballs then mid-path option | 01:22 |
jlk | okay, I somehow was lost on a tangent, thinking we needed to change the tarballs/ part of that | 01:22 |
jeblair | i put lines 45-49 in etherpad | 01:23 |
jeblair | i think the hardest part will be naming the mid-path variable. | 01:23 |
jlk | will synchronize be okay with /srv/static/tarballs///<project> ? | 01:23 |
jeblair | jlk: we can default it to '/' and drop a / from the path (see what i did in line 49) | 01:24 |
jlk | that would require that the user always add a / to the right part on a mid path | 01:24 |
jlk | seems dangerous. | 01:24 |
jeblair | hrm. maybe there's some jinja magic we could use. | 01:25 |
jeblair | if mid path doesn't end with / or something | 01:25 |
jlk | there is. | 01:25 |
jlk | I'll cook something up | 01:25 |
jeblair | cool | 01:25 |
jeblair | speaking of cook... :) | 01:25 |
pabelanger | jeblair: off-topic, on git push of tag, will zuul post pipeline be used for a project? | 01:27 |
jlk | jeblair: see line 58. | 01:30 |
jlk | zuul_fileserver_project_path: "{{ fileserver.path) }}{{ '/' + fileserver_something_path if fileserver_something_path is defined else ''}}/{{ zuul.project.short_name }}" | 01:30 |
jeblair | jlk: nice | 01:31 |
jeblair | pabelanger: i believe we have configured our post pipelines to only run on branch updates, not tag pushes | 01:31 |
jeblair | - event: ref-updated | 01:31 |
jeblair | ref: ^(?!refs/).*$ | 01:31 |
jeblair | that's gerrit-event-speak for branches only. | 01:32 |
jeblair | (gerrit strips refs/ from the ref name which something is pushed to a branch) | 01:32 |
jeblair | /which/when/ | 01:32 |
pabelanger | jeblair: okay, maybe I'm just tired, but how are we building docs for tags ? http://files.openstack.org/docs/nova/ | 01:32 |
pabelanger | I can't see anything in our current release pipelines | 01:33 |
jeblair | pabelanger: we have a tag pipeline | 01:33 |
jlk | what do we think of "fileserver_leading_path" as the name for the part between the base and the end ? | 01:33 |
jeblair | jlk: better than anything i've come up with so far :) | 01:33 |
jeblair | pabelanger: look at the contents of the .root-marker file to find out what job published a directory | 01:34 |
pabelanger | jeblair: Oh, so that is a different job | 01:34 |
pabelanger | {name}-releasenotes' | 01:34 |
pabelanger | we don't use unified-docs for tags it seems | 01:34 |
pabelanger | that only runs in post today | 01:35 |
pabelanger | which, is odd, because we have tag logic | 01:35 |
pabelanger | so, in that case https://docs.openstack.org/sandbox/ is good | 01:37 |
pabelanger | I can quickly create a new patch and see if we publish | 01:37 |
pabelanger | maybe for the morning | 01:37 |
jlk | hrm, do we have examples of translating "{template-name}" into zuul variables? | 01:46 |
jeblair | jlk: you can skip that one i think; it should just be jjb debug info | 01:47 |
jlk | it seems to be passed as an argument to the script | 01:48 |
jlk | JOBNAME | 01:48 |
jlk | oh but not used, wtf. | 01:48 |
jlk | interesting. | 01:49 |
pabelanger | k, I need to get some food | 01:52 |
openstackgerrit | Jesse Keating proposed openstack-infra/zuul-jobs master: Add content to support translation jobs [WIP] https://review.openstack.org/502207 | 02:12 |
jlk | okay two WIPs uploaded | 02:14 |
dmsimard | did mordred ever send that email ? | 03:33 |
*** jkilpatr has joined #zuul | 10:51 | |
*** jkilpatr has quit IRC | 11:13 | |
*** olaph1 is now known as olaph | 12:28 | |
Shrews | mordred: is vexxhost the only provider where we get ipv6 nodes? having a devil of a time getting 502137 to get one | 12:48 |
Shrews | not only do we have this ipv6 issue, but seems there is also a race in the streaming tests where the build uuid is not found (separate issue) | 12:49 |
Shrews | so this has been fun | 12:49 |
Shrews | one last recheck, then finding better things to do with my Sat morning | 12:57 |
*** jkilpatr has joined #zuul | 13:31 | |
clarkb | Shrews: rackspace too | 13:52 |
*** jkilpatr has quit IRC | 13:58 | |
*** pleia2 has quit IRC | 14:11 | |
*** pleia2 has joined #zuul | 14:18 | |
dmsimard | Shrews: you could do a fake review that creates 10 jobs that all run unit tests and then do rechecks | 14:19 |
dmsimard | Shrews: you're bound to run into one on each recheck :P | 14:19 |
dmsimard | If any cores are around, need reviews on the /etc/ci/mirror_info.sh stuff -- they will fix the v3 fedora devstack jobs: https://review.openstack.org/#/c/501578/ and https://review.openstack.org/#/c/501954/ | 14:33 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add node label to inventory https://review.openstack.org/502248 | 14:55 |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add node label to inventory https://review.openstack.org/502248 | 15:31 |
fungi | hrm... i'm probably pretty behind on what's going on, but shouldn't 500201 have enqueued into the gate when mordred approved it? | 16:40 |
fungi | its depends-on was already merged some several days prior | 16:40 |
mordred | fungi: v2 is doing a check job first - stale check I believe | 16:43 |
fungi | yeah, but once the verify +1 got added, it should have enqueued right? | 16:44 |
fungi | at least if we're going for behavior parity w/ 2.5.x | 16:45 |
mordred | fungi: oh - shade has no gate jobs in v3 | 16:51 |
mordred | fungi: it's still gating in v2 | 16:51 |
mordred | because shade has devstack gate jobs, which means we can't move all of its jobs over yet | 16:51 |
fungi | aha, right-o | 16:52 |
*** olaph1 has joined #zuul | 17:25 | |
*** olaph has quit IRC | 17:26 | |
*** olaph has joined #zuul | 17:28 | |
*** olaph1 has quit IRC | 17:29 | |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Clean up obsolete TODO for _setGitSsh() https://review.openstack.org/500925 | 19:07 |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove reference to unused variable https://review.openstack.org/502199 | 19:41 |
*** olaph has quit IRC | 19:52 | |
*** olaph has joined #zuul | 19:53 | |
*** olaph1 has joined #zuul | 19:59 | |
*** olaph has quit IRC | 19:59 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add zuul_log_url to emit-job-header https://review.openstack.org/502270 | 22:37 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Rename log_path and generate-log-url https://review.openstack.org/502271 | 22:50 |
*** olaph1 is now known as olaph | 23:00 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 23:02 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!