| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove comments questioning API capabilities https://review.openstack.org/499358 | 00:03 |
|---|---|---|
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Allow description fields in projects and project templates https://review.openstack.org/499365 | 00:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle logging ansible python errors properly https://review.openstack.org/499397 | 00:21 |
| mordred | jeblair: ^^ that fixes your logging issue from earlier - and adds a little thing to the integration test that duplicates it - so you should be able to go to the logs of that test and see that it properly shows tracebacks for the last two tasks | 00:22 |
| mordred | dmsimard: ^^ | 00:22 |
| mordred | jeblair: interesting! http://logs.openstack.org/99/497699/26/check/devstack-legacy-tempest-dsvm-neutron-full/5c2c8b7/job-output.txt.gz#_2017-08-30_23_31_48_094132 | 00:35 |
| mordred | jeblair: that implies to me that the "checkout appropriate branch" might be able to be collapsed to just "git reset --hard" | 00:36 |
| mordred | jeblair: remote: https://review.openstack.org/499400 Set the flag that allows pushing to checked out branch | 00:47 |
| mordred | jeblair: maybe that | 00:47 |
| mordred | jeblair: I'm less convinced that git reset --hard will work though - cause it's only an issue if we've checked out the branch that we're pushing, which won't be the case on clones of non-master | 00:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove no_log from fetch-sphinx-output https://review.openstack.org/499401 | 00:52 |
| pabelanger | mordred: I think we talked about removing no_log: true and filtering some other way? | 00:52 |
| pabelanger | I don't think that will be a lot of data, but likey helpful when debugging stuff | 00:52 |
| mordred | pabelanger: yah - the main idea was to do it via html version of the .json file in the web - since there we can do nice expand/contract sections and whatnot | 00:54 |
| pabelanger | mordred: ya, that is what I remember | 00:55 |
| *** xinliang has quit IRC | 02:31 | |
| *** xinliang has joined #zuul | 02:44 | |
| *** xinliang has joined #zuul | 02:44 | |
| *** bhavik1 has joined #zuul | 05:20 | |
| *** isaacb has joined #zuul | 06:24 | |
| *** isaacb has quit IRC | 06:31 | |
| *** hashar has joined #zuul | 07:29 | |
| *** bhavik1 has quit IRC | 07:39 | |
| *** openstackgerrit has quit IRC | 08:17 | |
| *** electrofelix has joined #zuul | 09:06 | |
| *** hashar is now known as hasharLunch | 10:14 | |
| *** jkilpatr has quit IRC | 10:41 | |
| *** jkilpatr has joined #zuul | 11:01 | |
| *** hasharLunch is now known as hashar | 11:24 | |
| *** jkilpatr has quit IRC | 12:03 | |
| *** jkilpatr has joined #zuul | 12:09 | |
| *** jkilpatr has quit IRC | 12:12 | |
| *** jkilpatr has joined #zuul | 12:12 | |
| *** jkilpatr has quit IRC | 12:17 | |
| *** jkilpatr has joined #zuul | 12:30 | |
| Shrews | mordred: the bindep_command vars are not needed in https://review.openstack.org/498209 now. want to clean those up now before i +3, or do it later? | 12:57 |
| Shrews | hrm, it doesn't really validate that the expected outout actually appears, does it? | 13:05 |
| mordred | Shrews: it does not validate the expected output - it's purely a visual eyeball thing atm | 13:10 |
| Shrews | mordred: *nod* | 13:11 |
| *** openstackgerrit has joined #zuul | 13:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove bindep_command lines https://review.openstack.org/499589 | 13:11 |
| mordred | Shrews: there's a followup removing the bindep command lines | 13:12 |
| mordred | Shrews: of course, now there's a weird test failure ... | 13:16 |
| Shrews | sweet | 13:17 |
| Shrews | mordred: i wonder if we can use lineinfile module somehow to do output verification... | 13:17 |
| * Shrews experiments | 13:20 | |
| *** rcarrillocruz has quit IRC | 13:21 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add integration test for zuul_stream https://review.openstack.org/498209 | 13:22 |
| *** rcarrillocruz has joined #zuul | 13:24 | |
| Shrews | mordred: also, where did the debug var output go? | 13:24 |
| Shrews | mordred: http://logs.openstack.org/09/498209/14/check/zuul-stream-functional/c79fe46/stream-files/stream-job-output.txt | 13:25 |
| Shrews | should be a whole bunch of stuff, yeah? | 13:25 |
| *** dkranz has joined #zuul | 13:38 | |
| mordred | Shrews: yah - lemme doublecheck that I didn't get that line incorrect | 13:42 |
| Shrews | mordred: i ran it locally and it seemed ok | 13:43 |
| mordred | Shrews: with the callback plugin enabled? I just ran it and also don't have data :( | 13:44 |
| Shrews | mordred: ah, no | 13:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for debug statements https://review.openstack.org/499608 | 14:06 |
| mordred | Shrews: that should have the debug output in the log file | 14:06 |
| mordred | Shrews: btw - at some point literally EVERYTHING about that file needs a refactor | 14:06 |
| mordred | Shrews: I should mabye make a little helper script to set up an env for local testing of the callback plugins like I do here | 14:07 |
| mordred | Shrews: it's not super hard to do - but it does involve a couple of pieces of setup | 14:07 |
| Shrews | mordred: yeah, i'm trying to get those pieces in place myself. like, where can i get logging.json? | 14:12 |
| * Shrews looks forward to mordred automating all the things | 14:12 | |
| mordred | Shrews: so - logging.json is easy - "python3 zuul/ansible/logconfig.py" | 14:16 |
| Shrews | woot http://logs.openstack.org/08/499608/1/check/zuul-stream-functional/222ec5e/stream-files/stream-job-output.txt | 14:16 |
| mordred | that will spit out a logging.json into your current working dir that sets logging to go to a file called log-output.txt in your current directory | 14:16 |
| mordred | woot! | 14:16 |
| mordred | Shrews: the second thing you need is an ansible.cfg - this is the one I have in my zuul source dir: http://paste.openstack.org/show/620080/ | 14:17 |
| mordred | Shrews: (I comment and uncomment the action plugins line depending on whether I want trusted or untrusted behavior) | 14:18 |
| mordred | Shrews: then finally I have http://paste.openstack.org/show/620081/ as my hosts.yaml that is referenced in that ansible.cfg | 14:19 |
| Shrews | mordred: hostfile is so 1.9 | 14:21 |
| Shrews | i've never seen the yaml format for the inventory | 14:22 |
| mordred | Shrews: it's basically the same thing we have zuul write out | 14:23 |
| mordred | Shrews: http://logs.openstack.org/08/499608/1/check/zuul-stream-functional/222ec5e/zuul-info/inventory.yaml | 14:23 |
| mordred | for instance | 14:23 |
| mordred | Shrews: but yah- hostfile is meh | 14:24 |
| mordred | Shrews: I like the yaml form much better | 14:24 |
| mordred | Shrews: oh - btw- the openstack inventory plugin landed in 2.4 | 14:24 |
| Shrews | cool | 14:24 |
| Shrews | oh interesting. the yaml format was deprecated long ago, but re-introduced in 2.1 | 14:26 |
| jeblair | mordred: yeah, we need to set the push setting, then push, then reset --hard, then checkout (because we may be checking out a different branch). | 14:26 |
| mordred | jeblair: yah- I pushed up a patch for the push setting- but I'm now thinking it should be different - one sec | 14:30 |
| pabelanger | morning | 14:32 |
| jeblair | mordred: re 499397 -- i don't see the errors in the stream test: http://logs.openstack.org/97/499397/1/check/zuul-stream-functional/df3b66e/stream-files/stream-job-output.txt | 14:35 |
| mordred | jeblair: hrm. so you don't - gimme a sec | 14:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Handle pushing to non-bare remote repos https://review.openstack.org/499620 | 14:38 |
| mordred | jeblair: I think that does all the things (my original project-config patch was the wrong place for it - we only need to set the flag when we're going to push, so it should accompany the push logic) | 14:39 |
| jeblair | mordred: lgtm | 14:40 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Only add changes to status page with jobs https://review.openstack.org/499191 | 14:45 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove bindep_command and bindep_fallback references https://review.openstack.org/498913 | 14:46 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add roles for adding and removing launchpadlib credentials https://review.openstack.org/498633 | 14:49 |
| mordred | jeblair: also - it's a scary world when I write javascript - and when it's merged into something | 14:49 |
| dmsimard | mordred: left a comment on that logging patch, thanks for pointing it out | 14:56 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Handle pushing to non-bare remote repos https://review.openstack.org/499620 | 14:57 |
| mordred | dmsimard: cool - thanks | 15:07 |
| mordred | dmsimard: also - I have a patch coming for you in just a little bit | 15:07 |
| dmsimard | mordred: I need to work on something for ara, those callback exceptions like the ones I pointed out for zuul_stream are really bothersome | 15:07 |
| dmsimard | They're hard to catch (and bubble up to get the actual exception) | 15:08 |
| dmsimard | Thinking maybe a decorator to wrap the entire v2_on_* hook in a try/except or something like that | 15:09 |
| pabelanger | jeblair: mordred: I believe we might need to bindmount some kerberos things into bwrap, or write an new role to handle that: http://logs.openstack.org/4b/79c6453cd269d56d2d3dd255445340afc030084b/post/publish-openstack-python-docs/d014be7/job-output.txt.gz#_2017-08-31_15_19_54_882218 | 15:26 |
| mordred | pabelanger: maybe update to use service/zuul-launcher@OPENSTACK.ORG instead of just service/zuul-launcher | 15:27 |
| mordred | pabelanger: remote: https://review.openstack.org/499681 Fully qualify AFS principal | 15:28 |
| pabelanger | mordred: ya, I was first going to try that too. But, I don't think any of our cell info will be inside bubblewrap for afs also | 15:28 |
| pabelanger | +2 | 15:28 |
| mordred | pabelanger: it doesn't need to be- it should be published in DNS properly | 15:28 |
| pabelanger | mordred: okay, cool. That is also what I was hoping for | 15:29 |
| mordred | pabelanger: that said - we could ALSO bind-mount some config perhaps as an optimization - but it should work with no specific config | 15:29 |
| mordred | (which is how I can get a kerberos ticket and aklog to do things in /afs/openstack.org on my laptop) | 15:30 |
| pabelanger | mordred: okay, lets do that first. not a fan of bindmounts :) | 15:30 |
| pabelanger | I mean, I know we needed it for stuff | 15:30 |
| mordred | pabelanger: ++ | 15:31 |
| *** bhavik1 has joined #zuul | 16:26 | |
| pabelanger | 2017-08-31 16:36:09.679725 | localhost | aklog: can't get afs configuration (afsconf_Open(/etc/openafs)) | 16:36 |
| pabelanger | :( | 16:36 |
| pabelanger | mordred: so, we'll likely need to bindmount and work on role to populate that post PTG? | 16:37 |
| jeblair | pabelanger: yeah, there doesn't appear to be a way to avoid that. i say ro-bindmount it in in our configuration | 16:49 |
| pabelanger | remote: https://review.openstack.org/499717 Add /etc/openafs/ to trusted_ro_paths for zuul-executor | 16:50 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: DNM: see if we can validate job output https://review.openstack.org/499732 | 17:04 |
| *** bhavik1 has quit IRC | 17:04 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: DNM: see if we can validate job output https://review.openstack.org/499732 | 17:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add /etc/alternatives to bwrap https://review.openstack.org/499736 | 17:11 |
| jeblair | dmsimard, pabelanger, mordred, SpamapS: ^ | 17:11 |
| dmsimard | lgtm, did you test that it works ? | 17:12 |
| jlk | o/ | 17:12 |
| jeblair | dmsimard: ya, this works: zuul-bwrap --ro-paths=/etc/alternatives /tmp -- ansible localhost -m shell -a 'awk' | 17:14 |
| jeblair | jlk: maybe you could take a look at https://review.openstack.org/499736 ? | 17:14 |
| jlk | sure! | 17:15 |
| *** jkilpatr has quit IRC | 17:16 | |
| jlk | jeblair: do we allow packages to be installed inside bwrap? | 17:16 |
| jlk | if so, what happens if a package wants to write to /etc/alternatives? | 17:17 |
| jeblair | jlk: i don't think mucking with system packages is in scope for even a trusted job -- it's already prohibited by /usr and most other system paths being mounted ro | 17:18 |
| jlk | okay | 17:18 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: DNM: see if we can validate job output https://review.openstack.org/499732 | 17:19 |
| *** jkilpatr has joined #zuul | 17:29 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add /etc/alternatives to bwrap https://review.openstack.org/499736 | 17:33 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Validate zuul_stream func test output https://review.openstack.org/499732 | 17:40 |
| Shrews | mordred: ^^^ fwiw, there is no output from the last two tasks of test-stream.yaml, so can't really do much with those | 17:41 |
| dmsimard | Shrews: fwiw I'm planning on setting up unit tests for the stream callback sometime when I have a chance | 17:46 |
| dmsimard | All of the hard work on how to test the callback is in ARA already so mostly copypasta | 17:47 |
| Shrews | oh, cool. maybe my change is irrelevant then | 17:47 |
| dmsimard | Nop, definitely not a bad thing | 17:47 |
| dmsimard | And it's inexpensive anyway so might as well | 17:48 |
| dmsimard | A good sanity check :) | 17:48 |
| *** hashar has quit IRC | 17:52 | |
| *** electrofelix has quit IRC | 17:59 | |
| mordred | dmsimard, jeblair, Shrews: sigh. I found what's wrong the the logging patch and the test output - fix coming | 17:59 |
| SpamapS | jlk jeblair: isn't installing packages also prohibited by the fact that the zuul user should never never never have sudo? | 18:01 |
| jlk | probably. I just couldn't remember all our layers :D | 18:01 |
| jeblair | i cleaned out /var/lib/zuul/builds on ze01 and restarted it | 18:09 |
| SpamapS | You'd have to try pretty hard to enable sudo I think. | 18:10 |
| *** hashar has joined #zuul | 18:11 | |
| Shrews | hrm, test_periodic_override test failed on a non-related patch: http://logs.openstack.org/32/499732/4/check/tox-cover/404ac59/testr_results.html.gz | 18:13 |
| *** hashar has quit IRC | 18:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle logging ansible python errors properly https://review.openstack.org/499397 | 18:17 |
| mordred | Shrews, dmsimard, jeblair, jlk, SpamapS: ^^ that should properly catch all the cases and should output the right things in the test | 18:18 |
| jlk | kk | 18:18 |
| mordred | the test playbook before was not showing output because it marks those tasks as failed_when: false - so they went through the _on_ok hooks rather than the _on_failure hooks that the patch had fixed | 18:18 |
| mordred | but that shows that becaues of failed_when, the "we received a MODULE FAILURE exception" case can happen on success too :) | 18:19 |
| mordred | so it's now all fixed | 18:19 |
| mordred | Shrews: and you should hopefully have lines to be able to grep for now | 18:20 |
| *** hashar has joined #zuul | 18:23 | |
| dmsimard | mordred: tangentially related, the output is kinda hard to read because of the -vvv | 18:24 |
| dmsimard | I wonder if we can split the two streams somehow | 18:24 |
| mordred | we can actually probably get rid of the -vvv - I'm not sure it provides much value atm | 18:24 |
| dmsimard | mordred: worst case | 18:25 |
| dmsimard | you can probably do a block | 18:25 |
| dmsimard | and then rescue, you re-run with -vvv | 18:25 |
| dmsimard | so that you run verbose only if there's a failure to troubleshoot | 18:25 |
| mordred | yah - but actually, we get: | 18:25 |
| mordred | [WARNING]: Failure using method (v2_runner_on_failed) in callback plugin | 18:25 |
| mordred | (<ansible.plugins.callback.zuul_stream.CallbackModule object at | 18:25 |
| mordred | 0x7f8002f25390>): exceptions must derive from BaseException | 18:25 |
| mordred | without -vvv | 18:25 |
| mordred | and -vvv doesn't actually give us the exception anyway | 18:25 |
| dmsimard | bcoca told me exception bubbling might be better in 2.4 | 18:26 |
| mordred | here's hoping | 18:26 |
| dmsimard | haven't seen it myself yet because yeah, troubleshooting exceptions really sucks sometimes, I end up using pdb more than I would like to | 18:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove -vvv from playbook invocation for log streaming test https://review.openstack.org/499753 | 18:27 |
| mordred | dmsimard: I put in tons of print statements with various profanity | 18:27 |
| jeblair | mordred, dmsimard: in this role: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/mirror-workspace-git-repos/tasks/main.yaml | 18:27 |
| *** hashar has quit IRC | 18:27 | |
| jeblair | mordred, dmsimard: the 'git checkout {{ item.1 }}' in the last task is running with the argument 'changed' | 18:28 |
| dmsimard | you probably want statuses.something | 18:28 |
| jeblair | is the register on the previous task wrong? | 18:28 |
| jeblair | http://logs.openstack.org/99/497699/26/check/devstack-legacy-tempest-dsvm-neutron-full/2fe451d/ara/ | 18:28 |
| dmsimard | a bit like how the find task from the other day you had to use item.path | 18:28 |
| jeblair | there's output you can explore | 18:28 |
| jeblair | dmsimard: yeah, but when i look at the ara output it seems that the item doesn't have anything other than "changed" | 18:29 |
| dmsimard | jeblair: you want item.1.stdout | 18:31 |
| dmsimard | jeblair: do a ctrl+f "stdout" here: http://logs.openstack.org/99/497699/26/check/devstack-legacy-tempest-dsvm-neutron-full/2fe451d/ara/result/3477afb3-a49a-49ca-8f10-08e296a790e8/ | 18:31 |
| mordred | jeblair: http://paste.openstack.org/show/620125/ | 18:32 |
| *** hashar has joined #zuul | 18:32 | |
| mordred | jeblair: that's putting the var into debug once the 'fix-debug-in-logging' patch is applied | 18:32 |
| jeblair | dmsimard: how did you get that deep link? | 18:33 |
| dmsimard | jeblair: when clicking on a task result, there's a permalink icon (blue chain link) at the top right | 18:33 |
| dmsimard | however clarkb told me it didn't work on firefox the other day, haven't had a chance to look yet | 18:33 |
| jeblair | dmsimard: it doesn't do anything | 18:33 |
| jeblair | yeah that | 18:33 |
| jeblair | okay bear with me here | 18:34 |
| dmsimard | well, I mean, I reproduced the issue locally but I haven't found the root cause | 18:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for debug statements https://review.openstack.org/499608 | 18:34 |
| jeblair | dmsimard: http://paste.openstack.org/show/620126/ | 18:34 |
| jeblair | that's from the run that failed... | 18:34 |
| jeblair | dmsimard: if item.1 is "changed", i would not expect item.1.stdout to work -- it does not look like a hash | 18:35 |
| mordred | oh - interesting - let me try a different local reproduction | 18:35 |
| dmsimard | jeblair: that's from zuul projects, no ? | 18:35 |
| jeblair | dmsimard: first element is, yes. second is from 'statuses'; the registered variable | 18:35 |
| mordred | jeblair: you want statuses.results | 18:37 |
| mordred | jeblair: then in the content itself you want item.1.stdout | 18:38 |
| jeblair | i guess the question i'm asking is -- how can i *know* what this complex data structure is without asking the two of you? :) | 18:38 |
| mordred | jeblair: let me show you! | 18:38 |
| jeblair | because it does not appear in the debug product. :) | 18:38 |
| jeblair | that thing is a string. it doesn't have a stdout, and it doesn't have a results. | 18:39 |
| mordred | jeblair: http://paste.openstack.org/show/620127/ | 18:39 |
| mordred | jeblair: that's what I did locally | 18:39 |
| jeblair | mordred: sure -- how did you know to type ".results" ? | 18:39 |
| mordred | jeblair: ah - because FIRST I just put in a debug: statuses | 18:39 |
| mordred | or debug: var=statuses rather | 18:40 |
| mordred | then I realized the statuses is a dict that has changed and msg - and then a results list | 18:40 |
| dmsimard | ah I was about to recommend using .results yes | 18:40 |
| mordred | so the statuses.results was me verifying that supposition | 18:40 |
| dmsimard | because the register is registering a task with with_items | 18:40 |
| dmsimard | and thus the list of results is inside var.results which is what you want to iterate over | 18:41 |
| dmsimard | there is no results key when it's a regular non-looped task | 18:41 |
| dmsimard | mordred: that reminds me I still haven't "fixed" the out-of-order issue about the task status returning before the last item status in ara :/ | 18:42 |
| dmsimard | you said you addressed that in zuul right ? | 18:42 |
| dmsimard | and yes, using a debug task is how you typically reverse engineer things, unfortunately | 18:43 |
| jeblair | dmsimard, mordred: why does that appear as "changed" in ara instead of the dictionary we see with debug:var ? | 18:43 |
| dmsimard | because ansible is probably iterating through the dict (values) that has the "results" key | 18:43 |
| dmsimard | and it so happens that one of the values is "changed", I'd guess | 18:44 |
| mordred | it's actually the keys - and one of the keys on the top level statuses dict is changed | 18:44 |
| dmsimard | mordred: oh, right -- it's "changed": True | 18:44 |
| mordred | yup | 18:44 |
| jeblair | ah i see that now | 18:46 |
| dmsimard | jeblair: tbh I literally have a "ansible-sandbox" directory tree on my laptop which is a skeleton of a playbook file with one role in it and I use that to test/debug things | 18:46 |
| dmsimard | jeblair: ansible adhoc commands don't give you a lot of room to experiment | 18:46 |
| jeblair | so when you register the return value of a task, the value of that register var is a dictionary | 18:47 |
| jeblair | if you debug it, you'll see that | 18:47 |
| jeblair | if you iterate over it, you get a list of keys in the normal python manner | 18:47 |
| jeblair | in this case, those keys included 'changed', 'msg', 'results', etc... | 18:48 |
| jlk | json, everything is a dict. | 18:48 |
| jlk | just one big bag of... | 18:48 |
| *** hashar has quit IRC | 18:48 | |
| jeblair | okay, now i think i have this in my head and can navigate it :) | 18:48 |
| dmsimard | dict or a list | 18:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Don't output complex items in the summary line https://review.openstack.org/499755 | 18:49 |
| *** hashar has joined #zuul | 18:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix error in mirr-workspace-git-repos https://review.openstack.org/499757 | 18:50 |
| jeblair | jlk, mordred, dmsimard: ^ | 18:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix error in mirror-workspace-git-repos https://review.openstack.org/499757 | 18:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Don't output complex items in the summary line https://review.openstack.org/499755 | 18:51 |
| mordred | jeblair: lgtm | 18:52 |
| mordred | jeblair, jlk, dmsimard, Shrews: ^^ that last change there is from looking at the logging from investigating the results question | 18:52 |
| *** jkilpatr has quit IRC | 18:57 | |
| *** jkilpatr has joined #zuul | 18:57 | |
| Shrews | mordred: since tests are failing, what did the original complex item output look like? | 18:57 |
| jeblair | Shrews: http://logs.openstack.org/99/497699/26/check/devstack-legacy-tempest-dsvm-neutron-full/e8fbbff/job-output.txt.gz | 18:58 |
| Shrews | oh my | 18:58 |
| jlk | lol | 18:59 |
| jlk | SO VERBOSE | 18:59 |
| dmsimard | jeblair: executor wasn't reloaded to take the awk fix ? | 19:00 |
| jeblair | dmsimard: yes it was | 19:01 |
| dmsimard | jeblair: seeing missing awks all over that link | 19:01 |
| jeblair | dmsimard: probably an old one | 19:01 |
| jeblair | http://logs.openstack.org/57/499757/2/check/tox-linters/0d52fe9/job-output.txt.gz#_2017-08-31_18_54_06_630581 | 19:01 |
| jeblair | that error ^ seems to be happening everywhere | 19:02 |
| jeblair | mordred: ^ | 19:03 |
| jeblair | mordred: seems to be a problem with I7499e6b1091d745984ca36179de2793827c9f98f | 19:05 |
| jeblair | okay it looks like the _fail_if_local_module function is fatally flawed -- it expects a moudle name but passes it into a function that expects a module object | 19:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Pass the correct object to local module check https://review.openstack.org/499761 | 19:08 |
| jeblair | normally i would revert that, however, it fixes a security issue, and i'm not willing to open that back up | 19:08 |
| mordred | jeblair: ^^ | 19:08 |
| mordred | jeblair: sorry - I should have mentionde that I was working on a fix | 19:08 |
| jeblair | woulda saved some time :) | 19:09 |
| mordred | jeblair: that's the fix - I've verified breakage without it locally and that the breakage is fixed with it - I will now work on figuring out why nothing triggered that in testing | 19:09 |
| jeblair | mordred: cool thx | 19:09 |
| * clarkb reviews | 19:10 | |
| clarkb | jlk beat me to it | 19:12 |
| fungi | mordred: (not urgent, but when you get a free sec...) with the current work on afs docs publishing in v3, is 488878 obsolete now? | 19:13 |
| fungi | and 488879 i guess? | 19:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add tests of accessing paths with file is ok https://review.openstack.org/499762 | 19:18 |
| mordred | jeblair, clarkb, jlk: ^^ that should fail - but should succeed once we land the other patch | 19:19 |
| mordred | I pushed it up not based on the other in an attempt to show that it woudl fail properly - but I'm not sur ethat's possile | 19:19 |
| jeblair | mordred: i will downgrade ze01 to merge the fix | 19:19 |
| jeblair | or we can force-merge it | 19:19 |
| jeblair | i'll downgrade | 19:20 |
| mordred | jeblair: cool - that way we can also witness the test failing | 19:21 |
| mordred | so that we can be sure we actually do have a test that covers this | 19:21 |
| jeblair | okay, ze01 restarted | 19:21 |
| mordred | fungi: yes - those are both obsolete | 19:21 |
| jeblair | rechecked fix | 19:21 |
| fungi | thanks mordred! | 19:22 |
| mordred | jeblair: yay! http://logs.openstack.org/62/499762/1/check/tox-py35/7b669d1/testr_results.html.gz fails with the traceback in the logs that existed for us | 19:36 |
| *** jkilpatr has quit IRC | 19:40 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Pass the correct object to local module check https://review.openstack.org/499761 | 19:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add tests of accessing paths with file is ok https://review.openstack.org/499762 | 19:49 |
| mordred | rebased on top of fix | 19:49 |
| *** olaph1 has joined #zuul | 19:57 | |
| *** olaph has quit IRC | 19:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove bindep_command lines https://review.openstack.org/499589 | 19:57 |
| jeblair | restarting ze01 | 20:01 |
| jeblair | done | 20:02 |
| *** jkilpatr has joined #zuul | 20:03 | |
| *** olaph has joined #zuul | 20:05 | |
| *** olaph1 has quit IRC | 20:05 | |
| mordred | jeblair: woot | 20:07 |
| pabelanger | jeblair: mordred: okay, afs client was able to obtain tokens, but I have to work on run-docs.sh first. | 20:16 |
| pabelanger | remote: https://review.openstack.org/499778 WIP: Move run-docs.sh intree | 20:16 |
| pabelanger | we've going to have to do that dance for publishing job and I figure we could have project-config use the role from openstack-zuul-jobs over itterating on it in project-config | 20:17 |
| pabelanger | jeblair: mordred: that said, before I spend time on that effort, I'd still like to publish something to afs. So, thinking I could quickly shift to another job that needs to be written for afs or maybe just a simple publish job to touch a file | 20:18 |
| pabelanger | thoughts? | 20:18 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix error in mirror-workspace-git-repos https://review.openstack.org/499757 | 20:22 |
| mordred | pabelanger: the infra docs jobs should just do the openstack-doc-build stuff with afs publishing | 20:22 |
| mordred | pabelanger: so maybe just make one of those, since we have the openstack-doc-build job / roles, and we can validate it as we land zuul patches, yeah? | 20:23 |
| pabelanger | mordred: okay, let me hack out upper-constraints from sandbox and quickly test | 20:24 |
| pabelanger | once afs is working I'll focus on ansiblefy run-docs.sh | 20:24 |
| jlk | Are there any more things to take off a checklist of stuff to work on? | 20:25 |
| pabelanger | jlk: mordred: mind a +3 on https://review.openstack.org/499401/ need more information about what we're syncing back to executor for run-docs | 20:33 |
| mordred | jlk: I'm working off of this etherpad: https://etherpad.openstack.org/p/zuulv3-special-jobs - unfortunately I think many of the jobs there require adding secrets... | 20:36 |
| pabelanger | did inventory.yaml file move locations? | 20:37 |
| pabelanger | I see it under zuul-info now | 20:37 |
| mordred | jlk: that said - I think the three from 76-85 are just jobs that will likely use propose-updates as a base job so could be worked on by someone without access to the secrets | 20:38 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove no_log from fetch-sphinx-output https://review.openstack.org/499401 | 20:46 |
| mordred | jeblair: didn't you add support for descriptions in project-templates already? | 20:50 |
| jeblair | mordred: yes, but we may not have restarted scheduler | 20:51 |
| mordred | Shrews: feel like nudging this across theline: https://review.openstack.org/#/c/499762/ ? | 20:51 |
| mordred | jeblair: ah. right. scheduler | 20:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for debug statements https://review.openstack.org/499608 | 20:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Don't output complex items in the summary line https://review.openstack.org/499755 | 20:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove -vvv from playbook invocation for log streaming test https://review.openstack.org/499753 | 20:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle logging ansible python errors properly https://review.openstack.org/499397 | 20:52 |
| Shrews | mordred: k k | 20:52 |
| mordred | pep8 whitespace error in the base patch of that stack | 20:52 |
| mordred | which is also ready to do now | 20:52 |
| jeblair | mordred: if you want to restart it now, that's fine, just recheck 497699 pls. otherwise, i can restart it after that is finished; i'm going to afk for a short while. | 20:53 |
| mordred | jeblair: no rush- I've got plenty of other things to do | 20:53 |
| *** dkranz has quit IRC | 20:54 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Ensure afs_target directory exists https://review.openstack.org/499800 | 20:55 |
| pabelanger | mordred: jeblair: ^thoughts? I know zuul_afs library will handle some of the directory creating, but figured this might be easier to see if we actually have issues with AFS before entering zuul_afs. | 20:57 |
| pabelanger | I still need to debug http://logs.openstack.org/88/9e2c0ba429f68c13b1e9163889ff11a96b432488/post/publish-openstack-python-docs/3f3e569/job-output.txt.gz#_2017-08-31_20_48_28_499194 | 20:57 |
| pabelanger | which is some weirdness | 20:57 |
| pabelanger | possible that jinja2 isn't being done | 20:57 |
| pabelanger | or typo some place | 20:57 |
| jeblair | pabelanger: it seems like the second issue is more pressing? | 21:03 |
| jeblair | pabelanger: the error is originating here: http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/publish/openstack-afs.yaml | 21:05 |
| jeblair | pabelanger: if you fix that, then zuul_afs runs and then you'll have your answer about whether afs works | 21:05 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add tests of accessing paths with file is ok https://review.openstack.org/499762 | 21:07 |
| jlk | mordred: where would I find the source scripts to these jobs? | 21:09 |
| jeblair | jlk: probably project-config/jenkins/scripts (or project-config/jenkins/jobs) | 21:10 |
| jlk | thanks | 21:10 |
| pabelanger | jeblair: agree, I'm trying to reproduce using zuul-bwrap locally | 21:11 |
| jeblair | pabelanger: you probably don't need zuul-bwrap; you can probably reproduce that with a simple role and playbook test | 21:12 |
| *** olaph1 has joined #zuul | 21:14 | |
| *** olaph has quit IRC | 21:16 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Log cachecontrol info by default https://review.openstack.org/499257 | 21:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Change the parameters to add-fileserver https://review.openstack.org/499811 | 21:30 |
| mordred | jlk: also - in the propose-updates base job, there is a role that runs that copies the scripts directory to the remote host | 21:32 |
| mordred | jlk: so for now rather than rewriting the scripts thmselves, I just made the proposal jobs consume the copy of the script that the propose-updates job copies over from project-config | 21:33 |
| mordred | jlk: (so that we can get a working copy of the job set up to be ready for migration without having to fully translate it just yet) | 21:34 |
| jlk | yeah I'm digging that part. I'm still familiarizing myself with the existing jobs too | 21:34 |
| mordred | ++ | 21:35 |
| *** hashar has quit IRC | 21:35 | |
| mordred | jlk: yup - that's important - migh tneed to provide the invocation with different parameters or set up some variables it expects or whatnot | 21:35 |
| mordred | pabelanger: I was going to suggest the same approach for run-docs.sh - but then I looked at your patch copying it over and realized that there is legitimately stuff in that script that needs to be based on the new zuul variables, so I think you're on th eright track | 21:36 |
| jlk | I can't seem to find what's meant by "propose-project-config-update" | 21:41 |
| jlk | oh maybe this is it | 21:42 |
| mordred | jlk: jenkins/jobs/infra.yaml ? | 21:46 |
| jlk | yup | 21:46 |
| jlk | Do we have a standard way of telling zuul what branch of a thing it's supposed to work on? | 21:53 |
| jlk | this job specifically is supposed to work on the master branch | 21:53 |
| jeblair | stand by for docs link! | 22:04 |
| jeblair | jlk: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.override-branch | 22:04 |
| jlk | YASS | 22:04 |
| jeblair | jlk: that's *probably* the thing you need, but also be aware that if you need to override the branch of a project *other* than the project associated with the change, you can do that with this: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.required-projects.override-branch | 22:05 |
| jlk | nod | 22:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Fix syntax issue with NamedTemporaryFile https://review.openstack.org/499800 | 22:36 |
| pabelanger | jlk: jeblair: mordred: tested locally this time :)^ | 22:36 |
| jeblair | i just sent an email to openstack-infra about zuul-cloner: http://lists.openstack.org/pipermail/openstack-infra/2017-August/005570.html | 22:46 |
| jeblair | if you're looking for something to do, that's a fairly indepedent python programming task. | 22:46 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix syntax issue with NamedTemporaryFile https://review.openstack.org/499800 | 22:56 |
| *** olaph has joined #zuul | 23:05 | |
| *** olaph1 has quit IRC | 23:05 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Allow user to select source directory in fetch-sphinx-output https://review.openstack.org/499852 | 23:22 |
| pabelanger | jeblair: mordred: ^ would be helpful for afs-docs | 23:24 |
| mordred | pabelanger: already +2 :) | 23:24 |
| jeblair | looking | 23:33 |
| pabelanger | let me push up new commit message | 23:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Allow user to select source directory in fetch-sphinx-output https://review.openstack.org/499852 | 23:35 |
| jeblair | pabelanger: wfm +3. where does that occur? | 23:38 |
| jeblair | (also, that's an old role; we should add variable docs to it) | 23:38 |
| pabelanger | jeblair: Agree for docs. | 23:40 |
| jeblair | pabelanger: base-test has gather_facts: false, but base does not. | 23:41 |
| jeblair | pabelanger: were you in the middle of doing something there? | 23:41 |
| pabelanger | right, I think I have a patch up to test it | 23:41 |
| pabelanger | let me check | 23:41 |
| pabelanger | Oh, right, I needed to rebase 490945 | 23:42 |
| pabelanger | let me do that quickly and we can see about promoting | 23:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - testing base-test playbook https://review.openstack.org/490945 | 23:43 |
| pabelanger | jeblair: as for +3 where does it occur, I actually need to add a trailing / to the path, because I don't want to copy the html directory into afs, just the contents of html/ | 23:46 |
| jeblair | i'm turning on keep and holding the node from the next devstack-legacy failure | 23:47 |
| pabelanger | okay, base-test worked, logs updated. I'll propose patch to promote it up | 23:47 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Allow user to select source directory in fetch-sphinx-output https://review.openstack.org/499852 | 23:54 |
| mordred | pabelanger, jeblair: I'm +2 on https://review.openstack.org/#/c/499854 and think it's good to go into base | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!