| pabelanger | http://grafana.openstack.org/dashboard/db/zuul-status | 00:00 |
|---|---|---|
| pabelanger | ze03 leading the pack with running most jobs | 00:00 |
| pabelanger | all other pretty much the same | 00:00 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add reprepo documentation https://review.openstack.org/511678 | 00:01 |
| *** esberglu has quit IRC | 00:01 | |
| *** ijw has quit IRC | 00:03 | |
| *** pahuang has quit IRC | 00:05 | |
| pabelanger | did we just do something to zuulv3.o.o? | 00:08 |
| SamYaple | pabelanger: can confirm its not returning for me with cli tool | 00:09 |
| pabelanger | infra-root: load on zuulv3.o.o is starting to climb and we just started swapping | 00:09 |
| pabelanger | Um | 00:09 |
| pabelanger | in fact, we have zero swap on zuulv3.o.o right now | 00:09 |
| pabelanger | we are out of memory | 00:09 |
| SamYaple | you changed the result by measuring it! | 00:09 |
| pabelanger | I'm having issues keep ssh connection going | 00:11 |
| pabelanger | and zuul-scheduler just died | 00:12 |
| pabelanger | I'm going to fix swap now | 00:12 |
| SamYaple | how much swap got used? | 00:12 |
| pabelanger | 0 | 00:12 |
| pabelanger | as we don't have a swap | 00:13 |
| SamYaple | OH | 00:13 |
| SamYaple | well thats different | 00:13 |
| *** bobh has joined #openstack-infra | 00:13 | |
| ianw | Killed process 20159 (zuul-scheduler) :( | 00:13 |
| pabelanger | yah | 00:14 |
| *** vhosakot has quit IRC | 00:14 | |
| mnaser | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all odd | 00:14 |
| mnaser | looks like it was destined to go out from when it first got restarted | 00:15 |
| pabelanger | sudo swapon /dev/xvde1 | 00:16 |
| pabelanger | we have swap again | 00:16 |
| pabelanger | I am going to start scheduler again | 00:16 |
| pabelanger | since we are down | 00:16 |
| ianw | pabelanger: did you just add it to fstab, or was it there? | 00:16 |
| pabelanger | infra-root: objections?^ | 00:16 |
| pabelanger | ianw: if was in fstab | 00:16 |
| ianw | not from me ... it was only zuul-scheduler that got killed | 00:17 |
| pabelanger | but guess we never did swapon after we redid repart | 00:17 |
| SamYaple | anything in the logs about failing to activate swap? | 00:17 |
| SamYaple | ah so it hasnt been restarted since then | 00:17 |
| * SamYaple blames mordred | 00:17 | |
| fungi | pabelanger: no objection, but i suspect this means we restarted earlier today with a new memory leak merged in the past few days since the prior restart (friday, was it?) | 00:17 |
| pabelanger | zuul-scheduler restarted | 00:18 |
| ianw | what i mean is, the only thing killed was zuul-scheduler and then memory usage dropped to zero, so we can infer that it was zuul-scheduler at fault from the cacti graphs | 00:18 |
| pabelanger | fungi: yah, looking at cacti, something has changed since our last restart | 00:18 |
| fungi | looks like maybe the previous restart was saturday, digging in logs for a timestamp now | 00:19 |
| pabelanger | fungi: so, we did update main.yaml | 00:20 |
| pabelanger | I wonder if that was the time when we started increasing memory | 00:20 |
| pabelanger | puppet would have done a reload | 00:20 |
| *** smatzek has joined #openstack-infra | 00:20 | |
| fungi | what time was that? | 00:20 |
| pabelanger | k, zuul-scheduler running again | 00:21 |
| pabelanger | looking | 00:21 |
| pabelanger | we should also send out a notice about unscheduled restart | 00:21 |
| pabelanger | and to recheck jobs | 00:21 |
| fungi | last modified timestamp on the file was 16:40 utc | 00:21 |
| pabelanger | https://review.openstack.org/512678/ | 00:22 |
| fungi | if this is a relatively steady leak starting from the last restart, then i'm worried it'll be back into swap before i wake up tomorrow | 00:22 |
| *** pahuang has joined #openstack-infra | 00:23 | |
| pabelanger | Oct 17 16:41:01 zuulv3 puppet-user[32440]: (/Stage[main]/Zuul::Scheduler/Exec[zuul-reload]) Triggered 'refresh' from 1 events | 00:23 |
| pabelanger | that was zuul-reload | 00:23 |
| *** liusheng has quit IRC | 00:23 | |
| pabelanger | fungi: when did we restart zuul-executors? | 00:24 |
| *** smatzek has quit IRC | 00:25 | |
| fungi | i don't know if they got restarted yet today? i know clarkb restarted them all yesterday | 00:25 |
| * fungi checks status log | 00:25 | |
| clarkb | not really here, they got restarted today too by jeblair | 00:26 |
| pabelanger | looks to be around 21:20UTC | 00:26 |
| clarkb | should be in sb | 00:26 |
| pabelanger | sorry | 00:27 |
| fungi | yeah, didn't make it into #status log so irc i guess | 00:27 |
| pabelanger | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-10-17.log.html#t2017-10-17T21:53:54 | 00:27 |
| ianw | will there be nodepool cleanup required after this? | 00:27 |
| pabelanger | ianw: I don't think so, zuul should have release locks, and nodepool-launcher would have seen that | 00:28 |
| *** andreas_s has joined #openstack-infra | 00:28 | |
| ianw | ok, quite a few "delete" stuff sitting around ... may be pre-existing | 00:29 |
| pabelanger | status notice due to unscheduled restart of zuulv3.o.o you will need to 'recheck' your jobs that were last running. Sorry for the inconvenience. | 00:30 |
| pabelanger | now does that look? | 00:30 |
| ianw | ++ | 00:30 |
| pabelanger | #status notice due to unscheduled restart of zuulv3.o.o you will need to 'recheck' your jobs that were last running. Sorry for the inconvenience. | 00:31 |
| openstackstatus | pabelanger: sending notice | 00:31 |
| pabelanger | ianw: http://grafana.openstack.org/dashboard/db/nodepool show nodepool-launcher deleting all our nodes when zuul-scheduler died | 00:31 |
| pabelanger | ianw: the existing delete likely need to be cleaned up with provider help | 00:32 |
| -openstackstatus- NOTICE: due to unscheduled restart of zuulv3.o.o you will need to 'recheck' your jobs that were last running. Sorry for the inconvenience. | 00:32 | |
| *** psachin has joined #openstack-infra | 00:32 | |
| *** rmcallis has joined #openstack-infra | 00:33 | |
| *** bobh has quit IRC | 00:33 | |
| openstackstatus | pabelanger: finished sending notice | 00:33 |
| *** liusheng has joined #openstack-infra | 00:34 | |
| *** Apoorva has quit IRC | 00:35 | |
| *** andreas_s has quit IRC | 00:37 | |
| *** xinliang has quit IRC | 00:37 | |
| ianw | looking at say, http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=63979&rra_id=5&view_type=&graph_start=1507000000&graph_end=1508287021 ... what's "steady state"? | 00:39 |
| ianw | seems ~5gb? | 00:39 |
| pabelanger | so, for some reason, zuul-scheduler didn't do anything for 10mins | 00:39 |
| pabelanger | http://paste.openstack.org/show/623897/ | 00:40 |
| pabelanger | I'm not sure why that is | 00:40 |
| fungi | 2017-10-14 15:49:38,058 INFO zuul.Scheduler: Starting scheduler | 00:40 |
| pabelanger | yah | 00:40 |
| ianw | i thought it was expected to take around 6-10 minutes to reload? | 00:40 |
| fungi | 2017-10-17 14:29:51,750 INFO zuul.Scheduler: Starting scheduler | 00:40 |
| pabelanger | I also pasted the wrong thing | 00:40 |
| pabelanger | sorry | 00:40 |
| fungi | so if we merged a leak, it was ~between those (modulo puppet deployment delays) | 00:41 |
| jeblair | well, i expect the weakref thing to be the biggest candidate | 00:43 |
| *** isq has quit IRC | 00:45 | |
| *** isq has joined #openstack-infra | 00:45 | |
| *** ijw has joined #openstack-infra | 00:45 | |
| *** cuongnv has joined #openstack-infra | 00:46 | |
| *** salv-orlando has joined #openstack-infra | 00:48 | |
| ianw | dmsimard: oh that change ... i'm wondering if that's now wrong because mordred opened up the permissions | 00:49 |
| ianw | https://review.openstack.org/#/c/512194/ | 00:50 |
| *** xinliang has joined #openstack-infra | 00:50 | |
| *** xinliang has quit IRC | 00:50 | |
| *** xinliang has joined #openstack-infra | 00:50 | |
| dmsimard | ianw: removed -W pending verification | 00:51 |
| dmsimard | ianw: where did mordred do that ? | 00:51 |
| ianw | ahh, just going back a bit ... | 00:51 |
| jeblair | fungi, ianw, pabelanger, clarkb: i will restart the scheduler with a revert of 511355 manually applied. it's the leading candidate for the leak. | 00:51 |
| fungi | jeblair: thanks | 00:52 |
| ianw | dmsimard: https://review.openstack.org/#/c/512285/ (from the issues page) | 00:52 |
| fungi | that seems like a prudent test, at least | 00:52 |
| *** salv-orlando has quit IRC | 00:52 | |
| dmsimard | ianw: can we reliably reproduce the issue that your fix was meant to resolve ? | 00:52 |
| *** markvoelker_ has quit IRC | 00:53 | |
| *** markvoelker has joined #openstack-infra | 00:54 | |
| pabelanger | jeblair: ack | 00:54 |
| jeblair | here is a list of changes merged between the time periods fungi mentioned: http://paste.openstack.org/show/623899/ | 00:54 |
| ianw | dmsimard: if you were to zuul-clone as an alternative user from a repo you don't own, yes. it was during tox runs as stack user, where requirements changes use zuul-cloner | 00:54 |
| *** andreas_s has joined #openstack-infra | 00:55 | |
| ianw | requirements s/changes/installation/ | 00:55 |
| *** LindaWang has joined #openstack-infra | 00:56 | |
| dmsimard | ianw: ok we can try to break the thing | 00:56 |
| *** thorst has joined #openstack-infra | 00:57 | |
| jeblair | scheduler restarted | 00:58 |
| *** markvoelker has quit IRC | 00:58 | |
| *** smatzek has joined #openstack-infra | 01:02 | |
| *** thorst has quit IRC | 01:02 | |
| *** thorst has joined #openstack-infra | 01:03 | |
| mnaser | did gerrit decide it wants to have a field day right now too? | 01:03 |
| *** andreas_s has quit IRC | 01:04 | |
| *** andreas_s has joined #openstack-infra | 01:04 | |
| *** jamesmcarthur has joined #openstack-infra | 01:05 | |
| *** huanxie has joined #openstack-infra | 01:05 | |
| jeblair | mnaser: gc graph looks fine | 01:06 |
| mnaser | jeblair it was much more sluggish clicking/opening a review, seems okay now. i'll report if i see anything else :X | 01:07 |
| ianw | dmsimard: few comments on 512194 ... i don't know what the answer is :/ | 01:07 |
| *** thorst has quit IRC | 01:08 | |
| dmsimard | ianw: I'm writing some additional tests and we can probably write a test patch on top of those tests to verify the behavior | 01:08 |
| mnaser | is there a generic job that does ansible lint-ing ? | 01:08 |
| *** iyamahat has quit IRC | 01:09 | |
| ianw | dmsimard: ++ on tests for this. i think though the behaviour is whatever we choose it to be | 01:09 |
| *** iyamahat has joined #openstack-infra | 01:09 | |
| *** jamesmcarthur has quit IRC | 01:09 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 01:12 |
| dmsimard | ianw: ^ | 01:12 |
| otherwiseguy | doh, the recheck that was finally going to pass after several weeks of broken gate got killed. is now an acceptable time for me to do another recheck, or are jobs going to go away again? | 01:12 |
| *** andreas_s has quit IRC | 01:13 | |
| jeblair | otherwiseguy: go for it | 01:14 |
| *** tobiash has quit IRC | 01:15 | |
| dmsimard | ianw: feel free to write a patch on top of that which could test the behavior we're looking for or adds a depends-on/rebase on top of your patch, need to afk for a while | 01:15 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Add irrelevant-files to puppet jobs https://review.openstack.org/512906 | 01:18 |
| mnaser | i'd appreciate an extra pair of eyes on that ^ we are releasing puppet modules soon and that means 20*5~6 more jobs will run .. for no reason | 01:20 |
| *** mriedem has quit IRC | 01:20 | |
| *** tobiash has joined #openstack-infra | 01:24 | |
| *** smatzek has quit IRC | 01:25 | |
| *** smatzek has joined #openstack-infra | 01:26 | |
| *** yamahata has quit IRC | 01:27 | |
| *** iyamahat has quit IRC | 01:27 | |
| *** andreas_s has joined #openstack-infra | 01:27 | |
| *** ijw has quit IRC | 01:30 | |
| *** bobh has joined #openstack-infra | 01:36 | |
| *** bobh has quit IRC | 01:37 | |
| *** gildub has quit IRC | 01:39 | |
| *** salv-orlando has joined #openstack-infra | 01:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 01:49 |
| ianw | dmsimard: ^ just interested if that works | 01:49 |
| *** andreas_s has quit IRC | 01:50 | |
| *** dingyichen has joined #openstack-infra | 01:50 | |
| *** smatzek has quit IRC | 01:50 | |
| *** salv-orlando has quit IRC | 01:53 | |
| *** hongbin has joined #openstack-infra | 01:54 | |
| *** yamamoto has joined #openstack-infra | 01:56 | |
| openstackgerrit | Tom Barron proposed openstack-infra/zuul-jobs master: Collect output from coverage job https://review.openstack.org/512910 | 01:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove legacy template jobs from ansible-role-nodepool https://review.openstack.org/512911 | 01:59 |
| *** andreas_s has joined #openstack-infra | 01:59 | |
| dmsimard | ianw: not going to work, those are trusted roles | 02:01 |
| dmsimard | Part of the reason why we can't test them | 02:01 |
| dmsimard | (yet) | 02:02 |
| ianw | dmsimard: yeah, i see that now with all the red :) | 02:02 |
| *** iyamahat has joined #openstack-infra | 02:02 | |
| *** thorst has joined #openstack-infra | 02:03 | |
| *** dhinesh has quit IRC | 02:07 | |
| *** andreas_s has quit IRC | 02:07 | |
| *** andreas_s has joined #openstack-infra | 02:08 | |
| *** thorst has quit IRC | 02:08 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add reprepo documentation https://review.openstack.org/511678 | 02:08 |
| *** hemna_ has quit IRC | 02:09 | |
| *** huanxie has quit IRC | 02:09 | |
| dmsimard | I need to work on a framework for testing trusted roles | 02:09 |
| dmsimard | mordred had started patterning something off | 02:10 |
| dmsimard | looks like ram is spiking again on zuul.o.o, not sure if legit http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all | 02:10 |
| *** huanxie has joined #openstack-infra | 02:10 | |
| jeblair | it went up to 10g before the restart | 02:11 |
| jeblair | with all the depends-on changes that were just submitted, current usage is not surprising. | 02:11 |
| *** HenryG has quit IRC | 02:13 | |
| pabelanger | odd, just noticed we are installing fetch-zuul-cloner even for non-legacy jobs | 02:15 |
| mnaser | i've been noticing that too pabelanger | 02:15 |
| mnaser | and um, i guess free benchmarking? sorry about that :< | 02:16 |
| pabelanger | I'll look in the morning, see if we can find a way to opt out of fetch-zuul-cloner role | 02:16 |
| *** andreas_s has quit IRC | 02:17 | |
| *** HenryG has joined #openstack-infra | 02:17 | |
| *** rlandy has quit IRC | 02:17 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add irrelevant-files to puppet jobs https://review.openstack.org/512906 | 02:18 |
| mnaser | before i add this to zuulv3-issues, does anyone have any recommended troubleshooting jobs for "ERROR No valid playbook found" | 02:19 |
| mnaser | https://review.openstack.org/#/c/512839/ | 02:20 |
| mnaser | the job works fine otherwise, interestingly, its the only one with a "branches" config option set | 02:20 |
| mnaser | https://review.openstack.org/#/c/512759/ -- most recently seen working here with no issues, just a few hours ago | 02:21 |
| *** andreas_s has joined #openstack-infra | 02:22 | |
| pabelanger | looking | 02:25 |
| *** pramodrj07 has quit IRC | 02:29 | |
| *** MasterOfBugs has quit IRC | 02:29 | |
| *** andreas_s has quit IRC | 02:30 | |
| *** gouthamr has joined #openstack-infra | 02:32 | |
| *** dhinesh has joined #openstack-infra | 02:33 | |
| *** gildub has joined #openstack-infra | 02:37 | |
| pabelanger | mnaser: I think the issue is with https://review.openstack.org/#/c/512866/11/playbooks/prepare-node-common.yaml | 02:38 |
| pabelanger | possible syntax error | 02:38 |
| pabelanger | trying to find more | 02:38 |
| mnaser | pabelanger that change is in gate (you can check status) and had passed that same job | 02:39 |
| *** huanxie has quit IRC | 02:42 | |
| *** gouthamr has quit IRC | 02:42 | |
| *** gouthamr has joined #openstack-infra | 02:42 | |
| *** dave-mccowan has quit IRC | 02:44 | |
| pabelanger | oh wait | 02:45 |
| *** namnh has joined #openstack-infra | 02:46 | |
| *** chlong has quit IRC | 02:47 | |
| *** cuongnv has quit IRC | 02:47 | |
| *** jcoufal has quit IRC | 02:47 | |
| *** yamahata has joined #openstack-infra | 02:47 | |
| *** cuongnv has joined #openstack-infra | 02:47 | |
| *** dave-mccowan has joined #openstack-infra | 02:48 | |
| pabelanger | yah, not sure | 02:48 |
| pabelanger | mnaser: if you remove depends-on for 512839, what happens? | 02:48 |
| *** salv-orlando has joined #openstack-infra | 02:50 | |
| mnaser | https://review.openstack.org/#/c/512839/ .. hmm, the integration job never ran | 02:50 |
| mnaser | and i dont think it should | 02:50 |
| pabelanger | mnaser: where is the pipline configuration for puppet-openstacklib? | 02:51 |
| mnaser | pabelanger not migrated so should be project-config | 02:52 |
| mnaser | pabelanger also by "the integration job never ran" i meant that by looking at zuul previous comments (before patches that added irrelevant-files) | 02:52 |
| pabelanger | mnaser: I'm struggling to see out puppet-openstacklib even run puppet-openstack-integration-4-scenario004-tempest-centos-7 | 02:53 |
| pabelanger | but my eyes are starting to blur | 02:53 |
| *** nicolasbock has quit IRC | 02:53 | |
| mnaser | its getting late, we can resume this later :D | 02:54 |
| *** salv-orlando has quit IRC | 02:54 | |
| mnaser | oh i think i know whats going on | 02:54 |
| mnaser | puppet-openstack-integration-jobs-all project-template runs for all puppet-openstacklib | 02:55 |
| pabelanger | do share | 02:55 |
| mnaser | that includes all 4 scenario jobs | 02:55 |
| *** dhinesh_ has joined #openstack-infra | 02:55 | |
| *** dhinesh has quit IRC | 02:56 | |
| mnaser | the issue we ran into a few days ago where backporting zuul.yaml was failing was fixed by jeblair but he said that all the definitions have to absolutely match for jobs | 02:56 |
| mnaser | so i think what i did is scenario4 is now different from master to stable branch, because one has irrelevant-files and the other doesnt | 02:56 |
| *** namnh has quit IRC | 02:56 | |
| *** cuongnv has quit IRC | 02:56 | |
| mnaser | so its getting treated like a variant | 02:56 |
| *** namnh has joined #openstack-infra | 02:57 | |
| *** cuongnv has joined #openstack-infra | 02:57 | |
| mnaser | im not sure actaully, i had it in my head and lost it in translation. there's a change pending adding the jobs to stable/newton which hasnt merged yet, ill see if that affects it heh | 02:57 |
| pabelanger | okay, I'll have to look inthe morning | 02:58 |
| dmsimard | ianw: I have a solution for fetch-zuul-cloner, I'll have a patch soon | 02:58 |
| dmsimard | Is there a reason why 'use-cached-repos' is in project-config ? It doesn't seem like it holds anything trusted or running against localhost, while on the other hand, mirror-workspace-git-repos is in zuul-jobs with executor delegated tasks | 03:01 |
| dmsimard | mirror-workspace-git-repos: https://github.com/openstack-infra/zuul-jobs/blob/master/roles/mirror-workspace-git-repos/tasks/main.yaml | 03:01 |
| dmsimard | use-cached-repos: https://github.com/openstack-infra/project-config/blob/master/roles/use-cached-repos/tasks/main.yaml | 03:01 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: Add pike branch to zun channel notification https://review.openstack.org/512923 | 03:01 |
| *** andreas_s has joined #openstack-infra | 03:02 | |
| *** pahuang has quit IRC | 03:04 | |
| openstackgerrit | RUIJIE YUAN proposed openstack-infra/project-config master: add merge-check job template for senlin-tempest-plugin https://review.openstack.org/512925 | 03:04 |
| *** thorst has joined #openstack-infra | 03:04 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: remove legacy panko jobs https://review.openstack.org/512430 | 03:08 |
| *** thorst has quit IRC | 03:09 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: drop legacy panko jobs https://review.openstack.org/512429 | 03:10 |
| *** iyamahat has quit IRC | 03:11 | |
| *** andreas_s has quit IRC | 03:11 | |
| *** andreas_s has joined #openstack-infra | 03:12 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 03:15 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for use-cached-repos https://review.openstack.org/512927 | 03:15 |
| dmsimard | ianw: ^ I *think* that should work. 'use-cached-repos' is what copies the repos from /opt/git to /home/zuul/<src_dir>. 'mirror-workspace-git-repos' is the trusted role that pushes an updated copy of the required-project (i.e, depends-on) to the node but it's not required to test that fetch-zuul-cloner works. | 03:17 |
| *** ihrachys has quit IRC | 03:17 | |
| *** andreas_s has quit IRC | 03:20 | |
| *** openstackgerrit has quit IRC | 03:22 | |
| pabelanger | seem to be having issues with control stream | 03:24 |
| *** rosmaita has quit IRC | 03:24 | |
| pabelanger | web-debug.log on zuulv3.o.o has some exceptions | 03:24 |
| pabelanger | but brain is fried to debug | 03:24 |
| *** hwoarang has quit IRC | 03:26 | |
| *** huanxie has joined #openstack-infra | 03:27 | |
| *** liusheng has quit IRC | 03:28 | |
| jungleboyj | Anybody know why review.o.o is down? | 03:29 |
| *** liusheng has joined #openstack-infra | 03:29 | |
| mnaser | jungleboyj noticing the same, it's been a lil sluggish, hopefully it gets recovers | 03:29 |
| jungleboyj | Ok, maybe it is a sign I need to take a break from work? | 03:30 |
| *** Guest94812 has quit IRC | 03:31 | |
| *** shu-mutou has quit IRC | 03:31 | |
| *** shu-mutou has joined #openstack-infra | 03:31 | |
| *** shu-mutou has joined #openstack-infra | 03:31 | |
| *** ijw has joined #openstack-infra | 03:31 | |
| *** hwoarang has joined #openstack-infra | 03:31 | |
| *** pahuang has joined #openstack-infra | 03:32 | |
| pabelanger | I am not able to ssh to review.o.o | 03:32 |
| pabelanger | ianw: ^ | 03:32 |
| *** aeng has joined #openstack-infra | 03:33 | |
| pabelanger | ianw: I have to EOD here, hopefully you can take a look into review.o.o | 03:34 |
| pabelanger | also explains the merger-failure status on zuulv3.o.o | 03:34 |
| pabelanger | good luck | 03:34 |
| *** rmcallis has quit IRC | 03:37 | |
| *** yamamoto_ has joined #openstack-infra | 03:39 | |
| *** hongbin has quit IRC | 03:40 | |
| ianw | ok ... | 03:40 |
| clarkb | ianw: I've pulled out the laptop to help | 03:41 |
| *** dhill_ has quit IRC | 03:41 | |
| clarkb | host pings | 03:41 |
| ianw | trying to pull up a console now | 03:41 |
| clarkb | but cacti shows its gone away | 03:41 |
| *** andreas_s has joined #openstack-infra | 03:41 | |
| *** yamamoto has quit IRC | 03:42 | |
| ianw | xen errors on the console | 03:42 |
| ianw | i think this is something underneath the vm | 03:42 |
| ianw | have we rebooted? | 03:43 |
| ianw | i'm thinking a --force reboot at this point | 03:43 |
| clarkb | I don't think we rebooted, this looks similar to what we saw with elasticsearch07 | 03:43 |
| clarkb | with that host rebooting didn't seem to fix it immediately either | 03:43 |
| ianw | ok, well i don't think a reboot will make it *worse*? should we go for it/ | 03:44 |
| ianw | ? | 03:44 |
| clarkb | not sure | 03:45 |
| *** andreas_s has quit IRC | 03:45 | |
| clarkb | one concern would be potentially leaving git repos in an unhappy state if we do a forced reboot | 03:46 |
| *** links has joined #openstack-infra | 03:46 | |
| clarkb | whereas in theory if we let it run and it recovers the process will just keep executing and be fine? | 03:46 |
| ianw | :/ ... i'd not be confident it will come back | 03:47 |
| SamYaple | review.oo inaccessible? | 03:47 |
| SamYaple | ah yea i see the scrollback | 03:47 |
| *** dhajare has joined #openstack-infra | 03:47 | |
| SamYaple | win 12 | 03:47 |
| clarkb | rackspace status page doesn't show us anything | 03:47 |
| ianw | clarkb: do you get the messages they send that show up on the web console? | 03:47 |
| ianw | as emails i mean? let me log in and see if there's anything | 03:48 |
| clarkb | I just pulled up the public status page becaues that is easy and sometimes useful info | 03:48 |
| ianw | all green on the web ui | 03:50 |
| clarkb | as far as coming back without a reboot I rebooted es07 and it took like half an hour before it became useable and it had these same messages on it | 03:50 |
| ianw | Rackspace is migrating the cloud servers listed below from their current location in our DFW datacenter to another location in the DFW datacenter. -> elasticsearch07.openstack.org (b9952fa5-6e57-4c79-84ac-af0cb8d75363) | 03:50 |
| *** salv-orlando has joined #openstack-infra | 03:50 | |
| clarkb | so unsure if it would've come back on its own after 30 minutes or if the reboot triggered it | 03:50 |
| ianw | interesting! | 03:51 |
| ianw | Oct 15, 2017 - 4:27 PM AEDT | 03:51 |
| clarkb | oh that is interesting | 03:51 |
| *** smatzek has joined #openstack-infra | 03:51 | |
| clarkb | I bet we don't notice with other instances because they are small | 03:51 |
| clarkb | but gerrit and elasticsearch are the opposite of small | 03:51 |
| ianw | there is a pending trove maintence, but not until 30th | 03:52 |
| clarkb | I wonder if review.o.o is being migrated too | 03:52 |
| ianw | these support tickets seem to go to mordred and jeblair | 03:52 |
| dmsimard | ianw: I'm going to bed, but I pinged you about two reviews which would probably work to test our zuul-cloner things. | 03:52 |
| ianw | dmsimard: heh, i'd like to be able to see them :) | 03:53 |
| dmsimard | ianw: yeah, me too, I was waiting on the check jobs before going to bed :P | 03:53 |
| ianw | well, so it doesn't seem like planned maintenance, but similar to the elasticsearch problems that were ... urgh | 03:54 |
| clarkb | considering how similar this is to es07 I'm inclined to think a migration is the cause currently | 03:54 |
| clarkb | and if that is the case a reboot likely won't affect it right? | 03:54 |
| dmsimard | live migrations ? in openstack ? :D | 03:54 |
| dmsimard | Never tried live migrations on openstack w/ xen fwiw | 03:54 |
| clarkb | dmsimard: it appears to be of the non live variety | 03:54 |
| dmsimard | lol | 03:55 |
| clarkb | (assuming that is what it is) | 03:55 |
| *** salv-orlando has quit IRC | 03:55 | |
| ianw | clarkb: i have no idea. i can put in a ticket, but is anyone going to see it? | 03:55 |
| *** smatzek has quit IRC | 03:55 | |
| clarkb | on our end or theirs? if we know we've got a ticket in we'll look but ya unsure if anyone is checking on their end | 03:56 |
| ianw | i mean file a support ticket with rax | 03:57 |
| *** kiennt26 has quit IRC | 03:57 | |
| clarkb | ya | 03:57 |
| ianw | i really don't think it's anything we've done | 03:57 |
| clarkb | ianw: I agree | 03:57 |
| *** kiennt26 has joined #openstack-infra | 03:57 | |
| *** iyamahat has joined #openstack-infra | 03:57 | |
| *** huanxie has quit IRC | 03:58 | |
| clarkb | filing a ticket can't hurt, why don't we start there | 03:58 |
| ianw | Ticket ID | 03:59 |
| ianw | 171018-ord-0000074 | 03:59 |
| ianw | maybe someone in australia read thems? | 03:59 |
| *** dhill_ has joined #openstack-infra | 03:59 | |
| ianw | if there is a phone number, it's not immediately obvious | 03:59 |
| mnaser | i dont think they would be doing non-live-migration while keeping an instance running? so just gonna guess that it could be unlikely | 03:59 |
| SamYaple | i used to have access to all the hypervisors. seems like that would be helpful right about now | 03:59 |
| SamYaple | let me ping someone | 03:59 |
| clarkb | cloudnull: ^ may know? | 04:00 |
| dmsimard | I suspect RAX has follow-the-sun support, it would be silly for a managed hosting provider to be 9 to 5 lol | 04:00 |
| SamYaple | there is also a #rackspace irc channel we all used to half monitor | 04:01 |
| SamYaple | dmsimard: there are teams all shifts for sure | 04:01 |
| SamYaple | but if you have inside knowledge.... use it is my motto | 04:01 |
| clarkb | ianw: does the ticket number imply its for ORD ofr some reason? (server is in DFW) | 04:01 |
| ianw | ? it knew "review.openstack.org" as the server name in the ticket entry form | 04:01 |
| clarkb | I'm probably overthinking it :) | 04:02 |
| *** gongysh has joined #openstack-infra | 04:02 | |
| SamYaple | the phone banks are manned by engineers 24/7 if someone wants to call | 04:02 |
| *** Jeffrey4l has quit IRC | 04:02 | |
| clarkb | 1-800-961-4454 looks like the number for openstack cloud support | 04:03 |
| SamYaple | i dont know what type of account infras got or anything, but that is sorta rackspaces deal "call and talk to someone that can fix it" | 04:03 |
| *** Jeffrey4l has joined #openstack-infra | 04:03 | |
| ianw | ok, i have to install chrome to make a call via hangouts! | 04:04 |
| SamYaple | oh boy! | 04:05 |
| clarkb | ianw: fun, and thank you (I'm in bed so really want to avoid doing it myself :/) | 04:05 |
| SamYaple | you can also do it from an android phone | 04:05 |
| dmsimard | "Hi, I manage the OpenStack infrastructure, our server is down"... Confusion ensues | 04:05 |
| SamYaple | dmsimard: all you gotta do is verify you are teh account owner/user which is a secret question kinda deal | 04:06 |
| SamYaple | thats why i was saying i dont know how infras account is setup | 04:06 |
| dmsimard | We need to know mordred's favorite dish ? | 04:06 |
| SamYaple | you need to know the question *and* the answer | 04:07 |
| SamYaple | they arent supposed to ask the question | 04:07 |
| SamYaple | just ask for the answer | 04:07 |
| *** pfallenop has quit IRC | 04:07 | |
| SamYaple | i worked private cloud, but i had to do pubcloud stuff a bit during the heavy time periods of chaos | 04:08 |
| *** udesale has joined #openstack-infra | 04:09 | |
| ianw | ok, talking to someone ... | 04:09 |
| SamYaple | is it bob? | 04:09 |
| SamYaple | screw that guy | 04:09 |
| *** ykarel|away has joined #openstack-infra | 04:10 | |
| *** rmcallis has joined #openstack-infra | 04:11 | |
| *** claudiub has joined #openstack-infra | 04:11 | |
| ianw | yeah, i'm not authenticated in the right way | 04:12 |
| *** iyamahat has quit IRC | 04:12 | |
| *** iyamahat has joined #openstack-infra | 04:12 | |
| clarkb | can you at least pass the ticket number along since that was submitted via an authenticated user? | 04:13 |
| SamYaple | ianw: best to just ask if there is an incident in progress in DFW that could be affecting the server in the ticket | 04:13 |
| SamYaple | they should be able to give that out | 04:14 |
| ianw | haha ok i hacked in | 04:14 |
| *** udesale has quit IRC | 04:14 | |
| SamYaple | nice | 04:14 |
| ianw | mordred, i've changed your security question | 04:14 |
| SamYaple | did you use the admin password "p@ssw0rd"? | 04:14 |
| SamYaple | we add the number and symbol for stregth | 04:14 |
| *** udesale has joined #openstack-infra | 04:15 | |
| clarkb | ha ianw with the backdoor | 04:15 |
| dmsimard | mordred's favorite dish is now: mac'n'cheese | 04:15 |
| SamYaple | dmsimard: no ianw said he changed it | 04:16 |
| *** dhajare has quit IRC | 04:17 | |
| clarkb | now its ianw's favorite dish | 04:17 |
| SamYaple | blood pudding! | 04:17 |
| *** CrayZee has quit IRC | 04:18 | |
| ianw | i never had mac'n'cheese until i lived in the USA ... it's not an .au thing (or wasn't ... my kids eat it now) | 04:18 |
| clarkb | I had it for lunch today | 04:18 |
| ianw | don't worry, the on hold pan pipe music is keeping me very calm | 04:18 |
| clarkb | I think mordred is in europe now so may be awake in a couple hours | 04:19 |
| ianw | ok, he's suggesting a hard reboot | 04:19 |
| SamYaple | have we not done that yet? | 04:19 |
| ianw | so it was live migrating apparently | 04:19 |
| SamYaple | i could have told you frontline was going to ask "have you tried turning it off an on again" | 04:19 |
| clarkb | SamYaple: we didn't because es07 did this recently too and a reboot didn't fix it | 04:20 |
| dmsimard | In Canada we don't have Mac'n'cheese, we have: Kraft Dinner | 04:20 |
| clarkb | and it turns out es07 was migrated | 04:20 |
| clarkb | so wasn't sure if that would have an affect | 04:20 |
| ianw | so we ok if i hard reboot it? | 04:20 |
| *** dbecker has quit IRC | 04:20 | |
| clarkb | ianw: if that is what they are saying to do I guess we do it | 04:20 |
| SamYaple | hard reboot is different than a reboot though | 04:20 |
| clarkb | ianw: did the migration fail or something? | 04:20 |
| ianw | ok, rebooting | 04:21 |
| clarkb | (would be nice if that was exposed in the server status better) | 04:21 |
| dmsimard | It's usually better to do a reboot. The reboot eventually times out to a hard reboot if the normal reboot doesn't work. | 04:21 |
| clarkb | dmsimard: that a nova behavior? | 04:21 |
| dmsimard | Yup, unless it's changed since I've last poked around those bits of code | 04:22 |
| clarkb | neat, didn't know that | 04:22 |
| SamYaple | dmsimard: what defines a timeout? | 04:22 |
| ianw | we need to turn of this silly graphical boot console | 04:23 |
| dmsimard | I last looked at this at $oldjob so > 2 years ago, you're asking a lot from my memory :) | 04:23 |
| SamYaple | dmsimard: and rax is running xen, dont forget that | 04:23 |
| clarkb | a live migration would explain the slowness that mnaser complained about | 04:23 |
| clarkb | then I guess it went belly up at some point? | 04:23 |
| clarkb | (or devolved into a non live migration?) | 04:24 |
| SamYaple | dirtied ram to quick to migrate, migrate failed and fallback failed? | 04:24 |
| SamYaple | then they rebooted the box anyway? | 04:24 |
| *** stakeda has quit IRC | 04:24 | |
| SamYaple | i like being able to speculate now that i dont work there | 04:25 |
| ianw | ok, so should be back | 04:25 |
| ianw | hmm | 04:25 |
| ianw | it was confirmed to me that it was live migrated. so i don't think there's much we can do to avoid that, it is what it is. details will be in that ticket apparently | 04:26 |
| *** huanxie has joined #openstack-infra | 04:26 | |
| clarkb | server is there and gerrit process is running but apache claims it not there | 04:26 |
| * clarkb digs in | 04:26 | |
| mnaser | ianw: if they did a live migration using the api, "nova instance-action-log <server-uuid>" could confirm/sho wthat | 04:26 |
| clarkb | ianw: good to know the ticket will apparently have details | 04:26 |
| SamYaple | is aaction-log exposed to uses? | 04:26 |
| mnaser | im pretty sure it is exposed to users | 04:27 |
| ianw | why is it still 503'ing | 04:27 |
| SamYaple | mnaser: on rax? | 04:27 |
| mnaser | SamYaple oh, that i dont know.. i know it is in openstack.. if thats what they run :p | 04:27 |
| dmsimard | SamYaple: instnace-action-log is not typically priviledged | 04:27 |
| SamYaple | mnaser: i wouldnt call public cloud opentack.... | 04:27 |
| dmsimard | SamYaple: scoped per tenant | 04:28 |
| SamYaple | rax public lcoud anyway | 04:28 |
| clarkb | ianw: nothing is listening on port 8081 yet | 04:28 |
| ianw | clarkb: should we restart gerrit? | 04:29 |
| clarkb | ianw: yes I think so, I don't see anything logged from this current process | 04:29 |
| ianw | maybe it doesn't like coming up during boot | 04:29 |
| ianw | doing that now | 04:29 |
| dmsimard | hmm, now I was looking for the code about the soft reboot timing out to a hard reboot and I can't find it anywhere :( | 04:30 |
| SamYaple | dmsimard: i was not familiar with that code myself | 04:31 |
| SamYaple | haha! my irc VM in dfw just live-migrated :) | 04:31 |
| SamYaple | must be rolling reboots | 04:31 |
| ianw | Starting Gerrit Code Review: FAILED | 04:31 |
| ianw | i do not like this | 04:31 |
| dmsimard | starting to wonder if it was a client-side implementation in novaclient or openstackclient or something.. I really want to find that now | 04:31 |
| clarkb | ianw: though the process is running | 04:32 |
| dmsimard | ianw: is the database up ? | 04:32 |
| clarkb | ianw: I think the init script has a timeout and our gerrit is big enough that it runs into that before fully started? | 04:32 |
| SamYaple | dmsimard: hmm nova-client doesnt poll though, so that should be possible | 04:32 |
| SamYaple | dmsimard: i mean not for reboot operation (i dont think) | 04:32 |
| clarkb | /home/gerrit2/review_site/logs/error_log is the log file | 04:32 |
| mnaser | SamYaple dmsimard i think that soft reboot then hard reboot was pulled .. like way back. | 04:32 |
| clarkb | [2017-10-18 04:32:32,436] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.13.9-4-g2a605d5 ready | 04:33 |
| SamYaple | mnaser: prior to grizzly? because i revieweed the grizzly flow in great detail. so its either new, or way old | 04:33 |
| dmsimard | mnaser: so I'm not crazy right, that existed ? | 04:33 |
| dmsimard | SamYaple: was probably in icehouse era | 04:33 |
| clarkb | ianw: its listening on 8081 now | 04:33 |
| dmsimard | at least at $oldjob | 04:34 |
| ianw | ok, maybe just patience required | 04:34 |
| ianw | i've probably never restarted it cold | 04:34 |
| SamYaple | dmsimard: so working theory is was added and then removed. got it :) | 04:34 |
| clarkb | ianw: so I think gerrit is just very slow to start, I do not know why it may not have started on boot though as it didn't log anything that time. Possibly it raced the mount for /home/gerrit2? | 04:34 |
| * EmilienM waves on infra friends and wish good luck on whatever you're debugging | 04:34 | |
| clarkb | ianw: that would explin it actually as we'd mount over wher ethe logs go | 04:34 |
| *** dbecker has joined #openstack-infra | 04:34 | |
| clarkb | ianw: though there shouldn't be enough content at the unmounted /home/gerrit2 location to even run a process | 04:35 |
| clarkb | so maybe that doesn't explain it | 04:35 |
| clarkb | in any case it looks happy now? | 04:35 |
| ianw | clarkb: network race, not quite able to get to trove? | 04:35 |
| ianw | i'm not going to reboot it again to try debugging it :) | 04:35 |
| clarkb | ianw: ya that could be it too | 04:35 |
| clarkb | and I agree on not rebooting to debug :) | 04:36 |
| *** janki has joined #openstack-infra | 04:36 | |
| *** openstackgerrit has joined #openstack-infra | 04:36 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config master: Zuul-v3: add required projects for neutron-lib periodic jobs https://review.openstack.org/512811 | 04:36 |
| clarkb | looks like people can push code ^ | 04:36 |
| clarkb | web ui seems to work for me | 04:37 |
| ianw | yep, i think let's monitor for a bit | 04:37 |
| clarkb | error_log also looks ok. There are ome exceptions about diff took too long but those are normal | 04:37 |
| clarkb | and ssh server received EOFs (also normal) | 04:37 |
| clarkb | ianw: we should watch the replication as that always takes a while after a server restart | 04:37 |
| clarkb | 14662 tasks as of a few seconds ago | 04:38 |
| mnaser | dmsimard SamYaple libvirt/wait_soft_reboot_seconds | 04:38 |
| ianw | [2017-10-18 04:38:20,365] [d477c0d0] Push to git@github.com:openstack-infra/gearman-plugin.git references: [RemoteRefUpdate[remoteName=refs/pull/1/head, NOT_ATTEMPTED, (null)...0000000000000000000000000000000000000000, srcRef=null, forceUpdate, message=null], .. blah blah | 04:39 |
| ianw | maybe another corrupt repo? | 04:39 |
| mnaser | dmsimard SamYaple https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2448-L2471 .. looks like libvirt specfiic behaviour | 04:39 |
| clarkb | now 14348 tasks so trending the right direction | 04:39 |
| clarkb | ianw: or it could be a branch delete? | 04:40 |
| dmsimard | mnaser: wow I was looking at the API/compute/client, never thought it was at a driver level | 04:40 |
| dmsimard | mnaser: glad to see it's still there then | 04:40 |
| clarkb | and now 13.7 ish k tasks | 04:41 |
| mnaser | dmsimard yup, looks like it does a soft reboot request and (by default) waits 120s | 04:42 |
| ianw | alright, firedrill over? i'll keep an eye but i think we're ok | 04:43 |
| clarkb | something is querying all closed changes for the month of decenmber 2016 | 04:43 |
| clarkb | ianw: ya I'm not seeing anything terribly out of the ordinary other than that query | 04:44 |
| *** rmcallis has quit IRC | 04:44 | |
| clarkb | but thats a user thing not gerrit got rebooted thing | 04:44 |
| clarkb | https://review.openstack.org/monitoring looks alright and the task queue is headed in the right direction | 04:44 |
| ianw | #status log review.o.o hard rebooted due to failure during live migration (rax ticket: 171018-ord-0000074). manually restarted gerrit after boot, things seem ok now | 04:45 |
| openstackstatus | ianw: finished logging | 04:45 |
| mnaser | if a job passed gate in that little outage, do we need to reverify? | 04:45 |
| dmsimard | Make sure to send mordred a mail or something for the favorite dish | 04:45 |
| clarkb | mnaser: yes | 04:45 |
| mnaser | clarkb ok, thank you | 04:46 |
| clarkb | mordred: fungi ^ you'll be next to wake I think. Please see ianw's status log. Everything seems to be functioning normally but if you can keep an eye on it that would be great | 04:46 |
| clarkb | ianw: you'll have to take us someplace where you like the beer :P | 04:47 |
| *** kiennt26 has quit IRC | 04:47 | |
| *** dave-mccowan has quit IRC | 04:47 | |
| clarkb | there was a git upload pack timeout error from someone pushing a change to nova I assume? | 04:48 |
| *** armaan has joined #openstack-infra | 04:48 | |
| *** ijw has quit IRC | 04:48 | |
| clarkb | might be worth pushing up a noop change to nova to make sure that was not a consistent problem | 04:48 |
| * clarkb looks for local nova repo | 04:49 | |
| *** armaan has quit IRC | 04:49 | |
| *** signed8bit has joined #openstack-infra | 04:50 | |
| clarkb | 10k tasks now | 04:51 |
| *** salv-orlando has joined #openstack-infra | 04:51 | |
| *** fguillot has joined #openstack-infra | 04:51 | |
| ianw | just syncing my nova tree | 04:52 |
| *** fguillot has left #openstack-infra | 04:52 | |
| *** dhill_ has quit IRC | 04:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix ZUUL_REFNAME for translation jobs https://review.openstack.org/512675 | 04:52 |
| clarkb | ianw: I've got a fresh clone I can push from if you want | 04:53 |
| ianw | it's thinking ... | 04:53 |
| ianw | https://review.openstack.org/#/c/512929/ seems ok | 04:54 |
| clarkb | cool, thanks for checking and with that I think I will call it a day | 04:55 |
| clarkb | thanks again | 04:55 |
| ianw | thank you! go team | 04:55 |
| *** salv-orlando has quit IRC | 04:57 | |
| *** dhill_ has joined #openstack-infra | 04:58 | |
| AJaeger | ianw: could you +2A https://review.openstack.org/#/c/512770/ , please? | 05:01 |
| jlvillal | Any idea why this patch didn't go into the Zuul gate phase? https://review.openstack.org/#/c/509669/ | 05:04 |
| jlvillal | It has a +1 from Zuul, Code-Review +2, and Workflow +1 | 05:04 |
| *** kjackal_ has joined #openstack-infra | 05:04 | |
| *** dhill_ has quit IRC | 05:04 | |
| *** huanxie has quit IRC | 05:05 | |
| *** thorst has joined #openstack-infra | 05:05 | |
| *** hyakuhei has quit IRC | 05:05 | |
| *** coolsvap has joined #openstack-infra | 05:07 | |
| mnaser | jlvillal zuul had a crash at around that time | 05:07 |
| *** huanxie has joined #openstack-infra | 05:07 | |
| jlvillal | mnaser, Ah. Is there anything I can do to make it realize it should do something? | 05:08 |
| mnaser | jlvillal: i think you could cheat and +A the change if you have core and it'll go into gate again | 05:08 |
| mnaser | if you aren't core, you'll have to recheck it | 05:08 |
| jlvillal | mnaser, I don't have that kind of power :( | 05:08 |
| mnaser | jlvillal: i'd recheck in that case | 05:09 |
| jlvillal | mnaser, Will that work? Since it already has a +1 from Zuul. | 05:09 |
| *** dhill_ has joined #openstack-infra | 05:09 | |
| mnaser | jlvillal yeah, it'll just do check+gate again though so one extra round of checks | 05:09 |
| jlvillal | mnaser, Thanks | 05:10 |
| *** thorst has quit IRC | 05:10 | |
| *** armaan has joined #openstack-infra | 05:13 | |
| clarkb | new gerrit fixed issue with rechecks in this case | 05:13 |
| *** salv-orlando has joined #openstack-infra | 05:15 | |
| *** e0ne has joined #openstack-infra | 05:15 | |
| *** yolanda has quit IRC | 05:15 | |
| jlvillal | clarkb, thanks | 05:18 |
| *** cuongnv has quit IRC | 05:19 | |
| *** e0ne has quit IRC | 05:19 | |
| *** cuongnv has joined #openstack-infra | 05:23 | |
| *** shardy has quit IRC | 05:23 | |
| *** shardy has joined #openstack-infra | 05:25 | |
| *** janki has quit IRC | 05:25 | |
| *** gongysh has quit IRC | 05:31 | |
| ianw | jlvillal: seems to be moving ok? let me know if issues | 05:32 |
| *** iyamahat has quit IRC | 05:32 | |
| jlvillal | ianw, No issues. Just going through the check now. By the time I wake up it should be merged :) | 05:33 |
| *** gouthamr has quit IRC | 05:34 | |
| *** markvoelker has joined #openstack-infra | 05:35 | |
| *** markvoelker_ has joined #openstack-infra | 05:37 | |
| *** markvoelker has quit IRC | 05:39 | |
| *** kjackal_ has quit IRC | 05:45 | |
| *** markvoelker_ has quit IRC | 05:46 | |
| *** markvoelker has joined #openstack-infra | 05:47 | |
| *** markvoelker has quit IRC | 05:50 | |
| *** armaan has quit IRC | 05:50 | |
| *** markvoelker has joined #openstack-infra | 05:50 | |
| *** markvoelker has quit IRC | 05:51 | |
| *** huanxie has quit IRC | 05:53 | |
| *** aeng has quit IRC | 05:56 | |
| *** gildub has quit IRC | 05:58 | |
| *** aeng has joined #openstack-infra | 06:13 | |
| *** tetsuro has joined #openstack-infra | 06:13 | |
| *** andreas_s has joined #openstack-infra | 06:14 | |
| SamYaple | who manages openstack/requiremetns? (which team owns it) | 06:17 |
| SamYaple | is it infra? | 06:17 |
| *** kiennt26 has joined #openstack-infra | 06:17 | |
| *** xinliang has quit IRC | 06:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix REFNAME in propose_update.sh https://review.openstack.org/512770 | 06:19 |
| *** kjackal_ has joined #openstack-infra | 06:21 | |
| *** aeng has quit IRC | 06:22 | |
| AJaeger | SamYaple: requirements team ;) | 06:22 |
| AJaeger | SamYaple: check governance repo for such questions | 06:23 |
| *** janki has joined #openstack-infra | 06:24 | |
| SamYaple | of course. what afool i am! i knew that. its late, i should just sleep and come back later | 06:24 |
| SamYaple | thanks AJaeger as always | 06:24 |
| *** xinliang has joined #openstack-infra | 06:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix zanata cert import https://review.openstack.org/512940 | 06:27 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish requirements loci images to DockerHub https://review.openstack.org/512941 | 06:28 |
| tetsuro | AJaejer: Could you review https://review.openstack.org/#/c/512223/ ? I've amended the patch, and it's ready for review. | 06:28 |
| *** liusheng has quit IRC | 06:30 | |
| AJaeger | tetsuro: asked a question on it... | 06:30 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish keystone loci images to DockerHub https://review.openstack.org/512793 | 06:31 |
| *** liusheng has joined #openstack-infra | 06:31 | |
| AJaeger | ianw, frickler,jlk, mordred : Next step in getting translations working is https://review.openstack.org/512940 - could you review, please? | 06:33 |
| *** salv-orlando has quit IRC | 06:33 | |
| *** tetsuro has quit IRC | 06:33 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in refstack https://review.openstack.org/512943 | 06:33 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in refstack https://review.openstack.org/512945 | 06:34 |
| *** salv-orlando has joined #openstack-infra | 06:35 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: required-projects for periodic jobs https://review.openstack.org/512946 | 06:37 |
| AJaeger | SamYaple: what's the logic behind https://review.openstack.org/#/c/512697 and friends? | 06:39 |
| *** ykarel|away is now known as ykarel | 06:40 | |
| *** jtomasek has joined #openstack-infra | 06:41 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/openstack-zuul-jobs master: Zuul-v3: add neutron to legacy-periodic-py35-with-neutron-lib-master https://review.openstack.org/512947 | 06:41 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config master: Zuul-v3: add required projects for neutron-lib periodic jobs https://review.openstack.org/512811 | 06:43 |
| *** jtomasek has quit IRC | 06:44 | |
| *** jtomasek has joined #openstack-infra | 06:45 | |
| *** pcaruana has joined #openstack-infra | 06:45 | |
| *** markvoelker has joined #openstack-infra | 06:45 | |
| *** armax has quit IRC | 06:45 | |
| *** kjackal_ has quit IRC | 06:47 | |
| *** hashar has joined #openstack-infra | 06:48 | |
| *** signed8b_ has joined #openstack-infra | 06:50 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: required-projects for periodic jobs https://review.openstack.org/512946 | 06:52 |
| *** signed8bit has quit IRC | 06:53 | |
| frickler | AJaeger: I have an idea for simplification on https://review.openstack.org/512893, if you agree I can do a new PS for that | 06:54 |
| *** signed8b_ has quit IRC | 06:56 | |
| *** ykarel_ has joined #openstack-infra | 06:57 | |
| SamYaple | AJaeger: had a few people still trying to use those repos with those codebases being broken during the zuulv3 stuff for a few weeks | 06:58 |
| *** thorst has joined #openstack-infra | 06:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix zanata cert import https://review.openstack.org/512940 | 06:58 |
| SamYaple | AJaeger: team didnt know the correct decom process and that was a quick fix for that issue | 06:59 |
| SamYaple | AJaeger: weve got a meeting to discuss it | 06:59 |
| *** ykarel has quit IRC | 06:59 | |
| AJaeger | decom? Decommission? | 07:00 |
| *** eumel8 has joined #openstack-infra | 07:00 | |
| SamYaple | well we couldnt use those repos during teh zuulv3 stuff, so they were out of commison for a few weeks and we played with a single repo in the meantime which looks like the direction it is going to head | 07:01 |
| AJaeger | frickler: that's jeblair's change ;) What about doing a followup for these? But it's not worth it IMHO, those job will be soon rewritten as native v3 ones... | 07:01 |
| AJaeger | SamYaple: ah, I see. | 07:02 |
| eumel8 | good morning | 07:02 |
| *** thorst has quit IRC | 07:03 | |
| *** jpich has joined #openstack-infra | 07:05 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/openstack-zuul-jobs master: Ceilometer: remove legacy-ceilometer-dsvm-tempest-plugin-es job https://review.openstack.org/512546 | 07:08 |
| frickler | AJaeger: yes, sure, that is the other option to view it, so I'm fine with just approving it, then | 07:11 |
| AJaeger | frickler: if this would be a native v3 job, I would hvae said: -1 or update :) | 07:12 |
| AJaeger | frickler: and welcome to the core team! | 07:12 |
| *** pgadiya has joined #openstack-infra | 07:16 | |
| *** tesseract has joined #openstack-infra | 07:17 | |
| frickler | AJaeger: thx for your coaching, that has certainly contributed to me getting here | 07:17 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix requirements cross-test jobs https://review.openstack.org/512893 | 07:21 |
| AJaeger | frickler: thanks. If you have any questions, feel free to ask. Happy to help if I'm around... | 07:21 |
| *** shardy is now known as shardy_afk | 07:26 | |
| *** martinkopec has joined #openstack-infra | 07:26 | |
| *** gcb has quit IRC | 07:34 | |
| *** ccamacho has joined #openstack-infra | 07:34 | |
| *** mikal has quit IRC | 07:40 | |
| *** mikal has joined #openstack-infra | 07:41 | |
| *** martinkopec has quit IRC | 07:41 | |
| *** jpena|off is now known as jpena | 07:42 | |
| openstackgerrit | RUIJIE YUAN proposed openstack-infra/project-config master: add system-required and check-requirements job for senlin-tempest-plugin https://review.openstack.org/512925 | 07:43 |
| *** martinkopec has joined #openstack-infra | 07:43 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Zuul-v3: add neutron to legacy-periodic-py35-with-neutron-lib-master https://review.openstack.org/512947 | 07:46 |
| *** egonzalez has joined #openstack-infra | 07:46 | |
| *** ykarel_ is now known as ykarel | 07:46 | |
| *** martinkopec has quit IRC | 07:47 | |
| *** huanxie has joined #openstack-infra | 07:48 | |
| *** gildub has joined #openstack-infra | 07:51 | |
| *** amoralej|off is now known as amoralej | 07:53 | |
| *** lin_yang has quit IRC | 07:54 | |
| *** udesale has quit IRC | 07:55 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in storlets https://review.openstack.org/512958 | 07:58 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in storlets https://review.openstack.org/512959 | 07:59 |
| *** sshnaidm|off is now known as sshnaidm | 07:59 | |
| *** shardy_afk is now known as shardy | 08:00 | |
| *** gildub has quit IRC | 08:00 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove bashate legacy jobs https://review.openstack.org/512117 | 08:01 |
| *** _ari_ has quit IRC | 08:06 | |
| *** lucas-afk has quit IRC | 08:07 | |
| *** ari[m] has quit IRC | 08:07 | |
| *** arxcruz has quit IRC | 08:07 | |
| *** rook has quit IRC | 08:07 | |
| *** nhicher has quit IRC | 08:07 | |
| *** wznoinsk has quit IRC | 08:07 | |
| *** weshay|ruck has quit IRC | 08:07 | |
| *** jpena has quit IRC | 08:07 | |
| *** nhicher has joined #openstack-infra | 08:07 | |
| *** mdbooth has quit IRC | 08:08 | |
| *** wznoinsk has joined #openstack-infra | 08:08 | |
| *** rook has joined #openstack-infra | 08:08 | |
| *** jpena has joined #openstack-infra | 08:08 | |
| *** rook is now known as Guest83828 | 08:09 | |
| *** _ari_ has joined #openstack-infra | 08:09 | |
| *** weshay has joined #openstack-infra | 08:09 | |
| *** lucasagomes has joined #openstack-infra | 08:09 | |
| *** arxcruz has joined #openstack-infra | 08:10 | |
| *** ari[m] has joined #openstack-infra | 08:12 | |
| *** mdbooth has joined #openstack-infra | 08:12 | |
| *** martinkopec has joined #openstack-infra | 08:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add project-templates to documentation https://review.openstack.org/512961 | 08:13 |
| *** HenryG has quit IRC | 08:15 | |
| *** HenryG has joined #openstack-infra | 08:16 | |
| *** e0ne has joined #openstack-infra | 08:18 | |
| *** martinkopec has quit IRC | 08:18 | |
| *** martinkopec has joined #openstack-infra | 08:19 | |
| *** e0ne_ has joined #openstack-infra | 08:21 | |
| *** ykarel_ has joined #openstack-infra | 08:22 | |
| openstackgerrit | Michele Baldessari proposed openstack-infra/tripleo-ci master: Switch info links to /home/zuul https://review.openstack.org/512963 | 08:22 |
| *** slaweq has joined #openstack-infra | 08:23 | |
| *** ralonsoh has joined #openstack-infra | 08:23 | |
| *** ethfci_ has joined #openstack-infra | 08:24 | |
| *** ykarel__ has joined #openstack-infra | 08:25 | |
| *** manfred has joined #openstack-infra | 08:25 | |
| *** Qiming_ has joined #openstack-infra | 08:25 | |
| *** pblaho1 has joined #openstack-infra | 08:25 | |
| *** ethfci_ has quit IRC | 08:25 | |
| *** ethfci_ has joined #openstack-infra | 08:26 | |
| *** ralonsoh_ has joined #openstack-infra | 08:26 | |
| *** ethfci_ has quit IRC | 08:26 | |
| *** ethfci_ has joined #openstack-infra | 08:27 | |
| *** ethfci_ has quit IRC | 08:27 | |
| *** clayton_ has joined #openstack-infra | 08:27 | |
| *** ykarel_ has quit IRC | 08:27 | |
| *** e0ne has quit IRC | 08:28 | |
| *** ykarel has quit IRC | 08:28 | |
| *** zz_ja has quit IRC | 08:28 | |
| *** gtmanfred has quit IRC | 08:28 | |
| *** ethfci has quit IRC | 08:28 | |
| *** clayton has quit IRC | 08:28 | |
| *** Qiming has quit IRC | 08:28 | |
| *** pblaho has quit IRC | 08:28 | |
| *** zz_ja has joined #openstack-infra | 08:28 | |
| *** zz_ja has quit IRC | 08:28 | |
| *** zz_ja has joined #openstack-infra | 08:28 | |
| *** manfred is now known as gtmanfred | 08:28 | |
| *** clayton_ is now known as clayton | 08:28 | |
| *** ralonsoh has quit IRC | 08:29 | |
| *** jamesmcarthur has joined #openstack-infra | 08:29 | |
| *** jrist has quit IRC | 08:30 | |
| *** mnaser has quit IRC | 08:30 | |
| *** ethfci has joined #openstack-infra | 08:31 | |
| *** jamesmcarthur has quit IRC | 08:33 | |
| *** panda|rover|off has quit IRC | 08:34 | |
| *** kjackal_ has joined #openstack-infra | 08:36 | |
| *** gildub has joined #openstack-infra | 08:37 | |
| *** iyamahat has joined #openstack-infra | 08:37 | |
| *** yamamoto_ has quit IRC | 08:37 | |
| *** leyal has quit IRC | 08:38 | |
| *** yamamoto has joined #openstack-infra | 08:40 | |
| *** yamamoto has quit IRC | 08:40 | |
| *** dtantsur|afk is now known as dtantsur | 08:40 | |
| *** kjackal_ has quit IRC | 08:40 | |
| *** florianf has quit IRC | 08:40 | |
| *** mnaser has joined #openstack-infra | 08:40 | |
| *** iyamahat has quit IRC | 08:43 | |
| *** electrofelix has joined #openstack-infra | 08:45 | |
| *** derekh has joined #openstack-infra | 08:45 | |
| *** leyal has joined #openstack-infra | 08:45 | |
| *** ralonsoh_ is now known as ralonsoh | 08:48 | |
| *** jtomasek has quit IRC | 08:48 | |
| *** jtomasek has joined #openstack-infra | 08:49 | |
| *** kjackal_ has joined #openstack-infra | 08:49 | |
| *** sileht has quit IRC | 08:50 | |
| *** sileht has joined #openstack-infra | 08:50 | |
| *** dingyichen has quit IRC | 08:51 | |
| *** david-lyle has quit IRC | 08:53 | |
| *** dhinesh_ has quit IRC | 08:53 | |
| *** panda has joined #openstack-infra | 08:53 | |
| *** yamahata has quit IRC | 08:55 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test containers update https://review.openstack.org/511175 | 08:55 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Ceilometer: remove legacy-ceilometer-dsvm-tempest-plugin-es job https://review.openstack.org/512546 | 08:57 |
| *** ykarel__ is now known as ykarel | 08:59 | |
| *** thorst has joined #openstack-infra | 08:59 | |
| *** ociuhandu has quit IRC | 08:59 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for use-cached-repos https://review.openstack.org/512927 | 09:02 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 09:02 |
| rgerganov | How can I recheck Zuul on this patch: https://review.openstack.org/#/c/509417/ | 09:03 |
| electrofelix | Is there a guide to how tenants will work in zuul v3? Can they be used by separate teams while still sharing the same pipelines, or do you 'have' to define different pipelines? | 09:03 |
| *** thorst has quit IRC | 09:04 | |
| *** kjackal_ has quit IRC | 09:06 | |
| *** yamamoto has joined #openstack-infra | 09:07 | |
| *** tosky has joined #openstack-infra | 09:10 | |
| *** kjackal_ has joined #openstack-infra | 09:10 | |
| *** yamamoto has quit IRC | 09:11 | |
| *** tosky has quit IRC | 09:12 | |
| *** tosky has joined #openstack-infra | 09:13 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Use 'build_reno' setuptools extension if available https://review.openstack.org/475034 | 09:16 |
| *** yamamoto has joined #openstack-infra | 09:17 | |
| *** yamamoto has quit IRC | 09:21 | |
| *** gcb has joined #openstack-infra | 09:25 | |
| *** pfallenop has joined #openstack-infra | 09:25 | |
| *** pgadiya has quit IRC | 09:26 | |
| *** yamamoto has joined #openstack-infra | 09:28 | |
| *** panda is now known as panda|rover | 09:29 | |
| *** egonzalez has quit IRC | 09:31 | |
| *** yamamoto_ has joined #openstack-infra | 09:34 | |
| *** yamamoto has quit IRC | 09:37 | |
| *** pfallenop has quit IRC | 09:38 | |
| *** pfalleno1 has joined #openstack-infra | 09:39 | |
| frickler | rgerganov: simply make a comment on that review saying "recheck" | 09:44 |
| rgerganov | frickler, I already made this and it didn't help | 09:45 |
| rgerganov | frickler, "recheck" retriggers only Jenkins | 09:46 |
| *** ociuhandu has joined #openstack-infra | 09:47 | |
| frickler | rgerganov: no, we have switched to running only zuul on Sunday, please try once more and you should see a new response from zuul in due time | 09:47 |
| rgerganov | frickler, ok thanks | 09:47 |
| frickler | rgerganov: you can also watch progress for your patch being checked at http://zuulv3.openstack.org/ now | 09:50 |
| *** kjackal__ has joined #openstack-infra | 09:50 | |
| *** egonzalez has joined #openstack-infra | 09:50 | |
| *** kjackal___ has joined #openstack-infra | 09:52 | |
| *** kjackal_ has quit IRC | 09:53 | |
| *** hashar is now known as hasharAway | 09:53 | |
| *** dhinesh has joined #openstack-infra | 09:54 | |
| *** rhallisey has joined #openstack-infra | 09:54 | |
| *** yamamoto_ has quit IRC | 09:56 | |
| *** kjackal__ has quit IRC | 09:56 | |
| *** andreas_s has quit IRC | 09:57 | |
| *** andreas_s has joined #openstack-infra | 09:58 | |
| *** dhinesh has quit IRC | 09:58 | |
| *** gildub has quit IRC | 09:59 | |
| *** LindaWang has quit IRC | 10:02 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 10:04 | |
| *** kiennt26 has quit IRC | 10:05 | |
| *** gmann is now known as gmann_afk | 10:07 | |
| *** andreas_s has quit IRC | 10:07 | |
| *** efoley has joined #openstack-infra | 10:10 | |
| *** cuongnv has quit IRC | 10:12 | |
| *** andreas_s has joined #openstack-infra | 10:12 | |
| *** baoli has joined #openstack-infra | 10:14 | |
| *** kjackal___ has quit IRC | 10:17 | |
| *** Guest83828 is now known as rook | 10:18 | |
| *** baoli has quit IRC | 10:19 | |
| *** andreas_s has quit IRC | 10:21 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Senlin https://review.openstack.org/512995 | 10:26 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Senlin https://review.openstack.org/512996 | 10:26 |
| *** yamamoto has joined #openstack-infra | 10:26 | |
| *** andreas_s has joined #openstack-infra | 10:27 | |
| *** pbourke has quit IRC | 10:27 | |
| *** sdague has joined #openstack-infra | 10:28 | |
| *** pbourke has joined #openstack-infra | 10:28 | |
| *** ldnunes has joined #openstack-infra | 10:29 | |
| *** mat128 has joined #openstack-infra | 10:33 | |
| *** andreas_s has quit IRC | 10:36 | |
| *** boden has joined #openstack-infra | 10:38 | |
| *** egonzalez has quit IRC | 10:39 | |
| *** andreas_s has joined #openstack-infra | 10:41 | |
| *** kjackal_ has joined #openstack-infra | 10:42 | |
| *** andreas_s has quit IRC | 10:42 | |
| *** andreas_s has joined #openstack-infra | 10:43 | |
| *** kjackal__ has joined #openstack-infra | 10:45 | |
| *** mrunge has quit IRC | 10:45 | |
| *** mrunge has joined #openstack-infra | 10:48 | |
| *** dbecker has quit IRC | 10:49 | |
| evrardjp | hello | 10:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Setup nodes for upstream-translation job https://review.openstack.org/513000 | 10:50 |
| evrardjp | is there reason why previous proposal bot doesn't work anymore? How can I debug this? | 10:50 |
| evrardjp | (in other words: Is there something to do with zuul v3 to make proposal bot work?) | 10:51 |
| AJaeger | evrardjp: which job exactly are you talking about? Yes, there are known failures... | 10:55 |
| evrardjp | AJaeger: it was just to know if there is something to do or not | 10:55 |
| AJaeger | evrardjp: I know some jobs are broken, so you need to look at log files and check whether those work. | 10:56 |
| evrardjp | ok simply that. | 10:56 |
| evrardjp | there is no permissions changes or anything | 10:56 |
| evrardjp | cool | 10:56 |
| evrardjp | thanks | 10:56 |
| AJaeger | So, either check zuulv3.openstack.org to see what fails - or go to the log files and see why they do not run | 10:56 |
| * AJaeger is working on translation jobs... | 10:57 | |
| *** LindaWang has joined #openstack-infra | 10:58 | |
| *** kjackal__ has quit IRC | 10:59 | |
| *** kjackal_ has quit IRC | 10:59 | |
| *** thorst has joined #openstack-infra | 11:00 | |
| *** chlong has joined #openstack-infra | 11:01 | |
| *** efoley has quit IRC | 11:02 | |
| *** efoley_ has joined #openstack-infra | 11:02 | |
| *** tpsilva has joined #openstack-infra | 11:03 | |
| *** sileht has quit IRC | 11:04 | |
| *** namnh has quit IRC | 11:04 | |
| *** thorst has quit IRC | 11:05 | |
| *** sileht has joined #openstack-infra | 11:07 | |
| *** jrist has joined #openstack-infra | 11:09 | |
| *** zzzeek has quit IRC | 11:10 | |
| *** fanzhang has quit IRC | 11:10 | |
| evrardjp | AJaeger: understood the problem. Fixed in https://github.com/openstack-infra/project-config/commit/bf61f85cf83487bb9a65babd38f7fb44a7b68328#diff-c1da69bf931ed082b3674c61a6696dd1 :p | 11:10 |
| evrardjp | thanks for the patch! | 11:10 |
| AJaeger | evrardjp: does it finally work? | 11:13 |
| * AJaeger didn't see logs yet... | 11:13 | |
| *** huanxie has quit IRC | 11:13 | |
| evrardjp | well I will see, I will do a patch and follow its course | 11:13 |
| AJaeger | thanks | 11:14 |
| *** zzzeek has joined #openstack-infra | 11:14 | |
| *** smatzek has joined #openstack-infra | 11:19 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: WIP: Add script for deterministic key generation https://review.openstack.org/513003 | 11:25 |
| *** ramishra has quit IRC | 11:31 | |
| *** dave-mccowan has joined #openstack-infra | 11:31 | |
| *** dave-mcc_ has joined #openstack-infra | 11:35 | |
| *** dave-mccowan has quit IRC | 11:37 | |
| *** lucasagomes is now known as lucas-hungry | 11:38 | |
| *** ramishra has joined #openstack-infra | 11:39 | |
| *** nicolasbock has joined #openstack-infra | 11:40 | |
| *** gcb has quit IRC | 11:45 | |
| *** huanxie has joined #openstack-infra | 11:46 | |
| *** gcb has joined #openstack-infra | 11:47 | |
| *** thorst has joined #openstack-infra | 11:52 | |
| *** rosmaita has joined #openstack-infra | 11:54 | |
| *** efoley has joined #openstack-infra | 11:56 | |
| *** efoley_ has quit IRC | 11:56 | |
| *** eumel8 has quit IRC | 11:58 | |
| *** yamamoto has quit IRC | 11:59 | |
| *** amoralej is now known as amoralej|lunch | 12:00 | |
| *** dtantsur is now known as dtantsur|brb | 12:01 | |
| *** eharney has joined #openstack-infra | 12:05 | |
| *** jcoufal has joined #openstack-infra | 12:05 | |
| *** trown|outtypewww is now known as trown | 12:05 | |
| *** jpena is now known as jpena|lunch | 12:06 | |
| *** kjackal_ has joined #openstack-infra | 12:06 | |
| *** Dinesh_Bhor has quit IRC | 12:09 | |
| pabelanger | morning | 12:12 |
| *** dprince has joined #openstack-infra | 12:12 | |
| *** kjackal_ has quit IRC | 12:16 | |
| *** huanxie has quit IRC | 12:16 | |
| *** edmondsw has joined #openstack-infra | 12:21 | |
| AJaeger | morning, pabelanger ! | 12:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 12:23 |
| AJaeger | pabelanger: the service-data-type you fixed yesterday still fails - it cannot find the temp directory ;( I added to etherpad in case you have some idea on how to fix these (around line 200). Help is welcome. | 12:24 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Update make_rest_client to work with version discovery https://review.openstack.org/513013 | 12:25 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: fix branch handling in release clone_repo function https://review.openstack.org/513014 | 12:25 |
| dhellmann | pabelanger, mordred : back on the tag-releases fix party this morning ^^ | 12:26 |
| *** salv-orlando has quit IRC | 12:26 | |
| mordred | dhellmann: woot! | 12:27 |
| AJaeger | mordred: could you review https://review.openstack.org/512802 and https://review.openstack.org/512765 , please? Those are needed for deploy guide. | 12:27 |
| dhellmann | https://www.youtube.com/watch?v=gTZ91YtKLN4 | 12:27 |
| mordred | dhellmann: ++ | 12:27 |
| *** salv-orlando has joined #openstack-infra | 12:28 | |
| *** pblaho1 has quit IRC | 12:29 | |
| *** dave-mcc_ is now known as dave-mccowan | 12:29 | |
| *** pblaho1 has joined #openstack-infra | 12:30 | |
| *** huanxie has joined #openstack-infra | 12:30 | |
| smcginnis | dhellmann: :) | 12:30 |
| odyssey4me | we'd appreciate reviews for https://review.openstack.org/512816 please - we're missing lint checks on our jobs :/ | 12:30 |
| *** lucas-hungry is now known as lucasagomes | 12:30 | |
| mordred | AJaeger: both look good to me | 12:31 |
| *** pblaho1 has quit IRC | 12:31 | |
| *** adrian has joined #openstack-infra | 12:31 | |
| mordred | odyssey4me: lgtm. fwiw, you could also define those in each of those repos (nothing about those that I think needs to be central - but doing them central is fine) | 12:32 |
| *** rlandy has joined #openstack-infra | 12:33 | |
| AJaeger | mordred: thanks! | 12:33 |
| *** pblaho has joined #openstack-infra | 12:33 | |
| mordred | odyssey4me: I mention because we're still learning best practices with central vs. distributed so talking about it more rather than less seems good | 12:34 |
| odyssey4me | mordred yeah, doing in one repo vs 40-odd though... thanks! | 12:34 |
| mordred | odyssey4me: ++ | 12:34 |
| odyssey4me | right now it's a band-aid to get us to a good state - we may revise again later to improve how things are done | 12:34 |
| mordred | ++ | 12:34 |
| mordred | odyssey4me: btw - I'm lurking in a session florian is giving on o-a so I can heckle him - but so far I haven't had any good opportunities. maybe make worse software next time k? | 12:35 |
| odyssey4me | I expect that we might want to implement some sort of job template, but there's a fair chance we'll ditch the extra layer of using tox rather than continue to use tox for these. | 12:35 |
| odyssey4me | hahaha, thanks :) | 12:35 |
| *** ykarel is now known as ykarel|afk | 12:36 | |
| mordred | odyssey4me: yah - with in-repo job config I think there are a whole new set of things that used tox before as a poor-man's-job-config that can be reworked ... | 12:36 |
| AJaeger | dhellmann: we still need a new reno release to publish releasenotes, correct? | 12:36 |
| mordred | so that tox can be used for when tox is wanted | 12:36 |
| pabelanger | AJaeger: is https://review.openstack.org/512911/ done correctly? removes legacy jobs from a project | 12:37 |
| AJaeger | pabelanger: Looks good to me... | 12:38 |
| AJaeger | if there'S a followup to remove the legacy jbos from openstack-zuul-jobs... | 12:38 |
| *** jhesketh_ has joined #openstack-infra | 12:38 | |
| pabelanger | AJaeger: yah, not just yet, but will be. They are still used by a few more roles | 12:38 |
| AJaeger | pabelanger: normally order is different: First change for in-repo, then project-config, then openstack-zuul-jobs removal | 12:39 |
| *** armaan has joined #openstack-infra | 12:39 | |
| AJaeger | pabelanger: ah, I see | 12:39 |
| AJaeger | pabelanger, mordred, frickler: Translation updates on the new dynamicly allocated nodes does not have lxml installed by default, so we need to change the setup - please review https://review.openstack.org/513000 | 12:40 |
| AJaeger | pabelanger: regarding your linters change: I'm fine taking it in and iterate over it. I'm confused that while the job runs on openstack-zuul-jobs, it reports only about zuul-jobs and nothing about openstack-zuul-jobs. That's my only concern right now | 12:42 |
| pabelanger | AJaeger: yah, I should be able to do that order once I get another job running. and happy to update for that | 12:42 |
| *** wolverineav has joined #openstack-infra | 12:42 | |
| AJaeger | if you add a todo or whatever "Run it on all three repos", I'm fine ;) | 12:42 |
| AJaeger | as said, happy to take it in and iterate - as long as we know the limits | 12:43 |
| pabelanger | AJaeger: well, it only reports about zuul-jobs today, because we have lint issues. Once we fix them, ozj will stop reporting them and hopefully just catch ozj things | 12:43 |
| *** jhesketh has quit IRC | 12:43 | |
| pabelanger | AJaeger: basically, once our merged code is good and job running on each repo, they should only catch issues in their own repo, because of gating | 12:43 |
| AJaeger | pabelanger: ah, so it stops there... Ok, then let'S get it in and fix those ;) | 12:44 |
| pabelanger | and if we do see issues in another, we can properly depends-on to fix them | 12:44 |
| *** links has quit IRC | 12:44 | |
| mordred | AJaeger: https://review.openstack.org/513000 is fine by me. I'm assuming we'll circle back around to those scripts once the jobs are working again yeah? | 12:44 |
| pabelanger | mordred: https://review.openstack.org/512715/ is likely of interested, that is what AJaeger is asking about | 12:45 |
| openstackgerrit | Merged openstack-infra/project-config master: fix branch handling in release clone_repo function https://review.openstack.org/513014 | 12:45 |
| *** mat128 has quit IRC | 12:45 | |
| mordred | pabelanger: looking | 12:45 |
| pabelanger | shared linters jobs between the 3 repos in infra | 12:45 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for use-cached-repos https://review.openstack.org/512927 | 12:45 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 12:45 |
| *** clayton has quit IRC | 12:45 | |
| AJaeger | mordred: yes, we need to improve those once we have them working. But since these used before the proposal node, more work is needed - so, let's fix first and then iterate on that... | 12:47 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-(keystone|mariadb) https://review.openstack.org/513022 | 12:47 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-(keystone|mariadb) to zuul.d https://review.openstack.org/513023 | 12:47 |
| dmsimard | Still looking for a last +2+A on https://review.openstack.org/#/c/504789/ for multinode integration tests | 12:47 |
| mordred | pabelanger, AJaeger: yah - looking at that I kind of think we should consider ditching tox and just writing a custom playbook forthis - either running ansible lint on the playbooks in all three repos - or running the tox role three times | 12:48 |
| mordred | AJaeger: ++ | 12:48 |
| mordred | pabelanger, AJaeger: because just running lint on the ozj playbooks on ozj changes would miss changes to zuul-jobs roles that ozj playbooks might catch | 12:48 |
| *** clayton has joined #openstack-infra | 12:49 | |
| mordred | (since the entrypoint for the linter is a playbook) | 12:49 |
| AJaeger | pabelanger: yeah, custom playbook works as well - we do not need tox. Your call... | 12:49 |
| mordred | well - I think we need a custom playbook in either case - to potentially run the tox role three times ... lemme do a followup patch real quick | 12:50 |
| *** kjackal_ has joined #openstack-infra | 12:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add linters job to all OSA role jobs https://review.openstack.org/512816 | 12:51 |
| *** rhallisey has quit IRC | 12:52 | |
| *** gmann_afk is now known as gmann | 12:52 | |
| *** rhallisey has joined #openstack-infra | 12:53 | |
| pabelanger | k, let me try run tox role 3 times first | 12:55 |
| pabelanger | right | 12:55 |
| dhellmann | AJaeger : reno 2.5.1 includes everything in master | 12:55 |
| dhellmann | AJaeger : including the zuulv3 fix | 12:55 |
| pabelanger | mordred: dmsimard: mind a +3 on https://review.openstack.org/509491/ | 12:56 |
| *** camunoz has joined #openstack-infra | 12:56 | |
| pabelanger | mordred: dmsimard: sorry, https://review.openstack.org/509491/ | 12:57 |
| pabelanger | ugh | 12:57 |
| dmsimard | pabelanger: I don't have +3 on that :) | 12:57 |
| pabelanger | https://review.openstack.org/512911/ | 12:57 |
| mordred | pabelanger: neat - I don't have +3 on that either- but I +1'd it | 12:57 |
| pabelanger | thats what I get for posting links with out coffee | 12:57 |
| pabelanger | mordred: dmsimard: yah, my bad on the first (duplicate) urls | 12:58 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header https://review.openstack.org/509436 | 12:58 |
| smcginnis | post_failures again? | 12:59 |
| *** mriedem has joined #openstack-infra | 12:59 | |
| *** yamamoto has joined #openstack-infra | 12:59 | |
| *** shardy has quit IRC | 13:00 | |
| AJaeger | dhellmann: http://logs.openstack.org/09/09bdd552d13b6dbec8eca361de4f1bef8cf2c786/post/publish-openstack-releasenotes/b73553b/ failed, let me get exact lines... | 13:00 |
| *** shardy has joined #openstack-infra | 13:00 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for emit-job-header and set-zuul-log-path-fact https://review.openstack.org/511948 | 13:00 |
| AJaeger | dhellmann: http://logs.openstack.org/09/09bdd552d13b6dbec8eca361de4f1bef8cf2c786/post/publish-openstack-releasenotes/b73553b/job-output.txt.gz#_2017-10-18_09_51_27_110837 "Unknown reference origin/stable/newton" | 13:01 |
| *** signed8bit has joined #openstack-infra | 13:01 | |
| dhellmann | AJaeger : reno is pinned to 2.5.0 in that job | 13:01 |
| *** huanxie has quit IRC | 13:01 | |
| AJaeger | dhellmann: argh ;( Can we unpin? | 13:01 |
| dhellmann | maybe we need to land a constraint update? | 13:01 |
| *** sileht has quit IRC | 13:01 | |
| smcginnis | Oh, that was before handling of "origin/", right? | 13:02 |
| dhellmann | yeah, that fix is in 2.5.1 | 13:02 |
| *** sileht has joined #openstack-infra | 13:02 | |
| dhellmann | AJaeger : what project is that job running for? | 13:02 |
| *** vdrok has quit IRC | 13:02 | |
| *** tommylikehu has quit IRC | 13:02 | |
| smcginnis | Seems like it would be safe (and necessary) to update the constraints then. | 13:02 |
| dhellmann | nova | 13:02 |
| AJaeger | dhellmann: nova | 13:02 |
| AJaeger | smcginnis: necessary! ;) | 13:02 |
| *** vdrok has joined #openstack-infra | 13:02 | |
| AJaeger | shall I propose and you +2 ? | 13:03 |
| smcginnis | AJaeger: Works for me. | 13:03 |
| dhellmann | global requirements has 2.5.1 as the constrained version | 13:03 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for emit-job-header and set-zuul-log-path-fact https://review.openstack.org/511948 | 13:03 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for emit-job-header and set-zuul-log-path-fact https://review.openstack.org/511948 | 13:03 |
| *** tommylikehu has joined #openstack-infra | 13:03 | |
| *** armaan has quit IRC | 13:03 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header https://review.openstack.org/509436 | 13:03 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header https://review.openstack.org/509436 | 13:03 |
| *** jpena|lunch is now known as jpena | 13:03 | |
| AJaeger | mmh, that has 2.5.1 - so why is it using 2.5.0 instead? | 13:04 |
| dhellmann | which patch is that? | 13:04 |
| dhellmann | 09bdd552d13b6dbec8eca361de4f1bef8cf2c786 doesn't appear in gerrit | 13:04 |
| *** fdegir has quit IRC | 13:04 | |
| dhellmann | it would be cool if post jobs dumped the full commit message of the thing they're running on | 13:04 |
| smcginnis | I know nova was wrapping up some final newton patches. I wonder if that's it. | 13:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy template jobs from ansible-role-nodepool https://review.openstack.org/512911 | 13:05 |
| AJaeger | change is "Merge "Target context for build notification in conductor" into stable/pike" | 13:06 |
| dhellmann | ah, the pike constraints may need to be updated then | 13:06 |
| dhellmann | although no, that job should run on master | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 13:07 |
| *** yamamoto has quit IRC | 13:07 | |
| dhellmann | is zuul checking out the pike version of constraints because the patch merged into that branch? | 13:07 |
| AJaeger | might be... | 13:07 |
| *** fdegir has joined #openstack-infra | 13:07 | |
| dhellmann | hmm | 13:08 |
| *** kgiusti has joined #openstack-infra | 13:09 | |
| dhellmann | I don't know if that counts as a bug or not. Or if there's a way to specify a different behavior. | 13:09 |
| dhellmann | is there some way to look at the logs for what was checked out? | 13:09 |
| AJaeger | mordred, pabelanger , any idea to check this? Looks like we use the wrong branch for requirements check out on the releasenotes job | 13:09 |
| AJaeger | dhellmann: http://logs.openstack.org/09/09bdd552d13b6dbec8eca361de4f1bef8cf2c786/post/publish-openstack-releasenotes/b73553b/zuul-info/inventory.yaml is one way | 13:09 |
| dhellmann | the pike branch of requirements does have reno=2.5.0 | 13:10 |
| dmsimard | mordred: I'm confused by https://review.openstack.org/#/c/512653/ and https://review.openstack.org/#/c/512408/ | 13:10 |
| *** efoley has quit IRC | 13:10 | |
| dhellmann | AJaeger : that shows the repo, but doesn't say which branch was actually used or what commit was actually checked out | 13:10 |
| dmsimard | mordred: re: tox with sudo | 13:10 |
| dmsimard | mordred: revoke-sudo doesn't seem included by default in base jobs or in tox jobs, what are we after ? It just seems like the revoke-sudo is included somewhere where it shouldn't be | 13:11 |
| *** shardy has quit IRC | 13:11 | |
| *** jaypipes has joined #openstack-infra | 13:12 | |
| AJaeger | dhellmann: hope pabelanger and mordred can help further - I need to leave for a bit now... | 13:12 |
| openstackgerrit | Merged openstack-infra/shade master: Handle glance image pagination links better https://review.openstack.org/511963 | 13:12 |
| dmsimard | mordred: I'm finding a reference that is probably in legacy jobs, ex: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/ansible-role-jobs.yaml#n10 but that's it | 13:12 |
| *** shardy has joined #openstack-infra | 13:13 | |
| *** efoley has joined #openstack-infra | 13:13 | |
| AJaeger | dhellmann: oh, found something quickly... | 13:13 |
| *** rmcallis has joined #openstack-infra | 13:15 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use master for publishing releasenotes https://review.openstack.org/513029 | 13:15 |
| AJaeger | dhellmann, mordred, pabelanger ^ | 13:15 |
| *** felipemonteiro_ has joined #openstack-infra | 13:15 | |
| smcginnis | Nice! | 13:16 |
| *** lbragstad has joined #openstack-infra | 13:16 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: prepare-docs-for-afs: Allow reuse https://review.openstack.org/512802 | 13:16 |
| dhellmann | AJaeger : ++ | 13:16 |
| *** felipemonteiro__ has joined #openstack-infra | 13:17 | |
| *** kjackal_ has quit IRC | 13:18 | |
| *** rmcallis has quit IRC | 13:19 | |
| *** felipemonteiro_ has quit IRC | 13:20 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server https://review.openstack.org/510362 | 13:21 |
| *** efoley has quit IRC | 13:21 | |
| *** gouthamr has joined #openstack-infra | 13:24 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: have release functions script source its own requirements https://review.openstack.org/513032 | 13:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix deploy-guide publish https://review.openstack.org/512765 | 13:26 |
| dhellmann | AJaeger, pabelanger,mordred : I'm not sure how that worked before ^^ | 13:26 |
| *** efoley has joined #openstack-infra | 13:27 | |
| *** AL34N1X has joined #openstack-infra | 13:28 | |
| *** AL34N1X has quit IRC | 13:28 | |
| *** shiyas has joined #openstack-infra | 13:31 | |
| *** dbecker has joined #openstack-infra | 13:32 | |
| *** smatzek has quit IRC | 13:33 | |
| *** smatzek has joined #openstack-infra | 13:33 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Floor Serializer bug https://review.openstack.org/513034 | 13:36 |
| *** esberglu has joined #openstack-infra | 13:36 | |
| *** smatzek has quit IRC | 13:38 | |
| *** gcb has quit IRC | 13:38 | |
| *** gcb has joined #openstack-infra | 13:38 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Floor Serializer bug https://review.openstack.org/513034 | 13:39 |
| *** signed8bit has quit IRC | 13:42 | |
| *** signed8b_ has joined #openstack-infra | 13:44 | |
| *** kjackal_ has joined #openstack-infra | 13:46 | |
| *** mat128 has joined #openstack-infra | 13:47 | |
| *** efoley has quit IRC | 13:48 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Update make_rest_client to work with version discovery https://review.openstack.org/513013 | 13:49 |
| *** armax has joined #openstack-infra | 13:50 | |
| *** dbecker has quit IRC | 13:52 | |
| *** efoley has joined #openstack-infra | 13:54 | |
| *** yamamoto has joined #openstack-infra | 13:54 | |
| *** yamamoto has quit IRC | 13:55 | |
| jamespage | q: does review.openstack.org have a anon addressable REST API? | 13:56 |
| *** jamesmcarthur has joined #openstack-infra | 13:57 | |
| *** markus_z has joined #openstack-infra | 13:59 | |
| smcginnis | jamespage: This works for me: curl https://review.openstack.org/projects/?d | 13:59 |
| fungi | oh boy, it was another 600 scrollback line night i see. reading now | 13:59 |
| markus_z | Devstack code removal (trivial patch): https://review.openstack.org/#/c/510780/ please review | 14:00 |
| smcginnis | fungi: Make sure you've grabbed your coffee first. | 14:00 |
| *** iyamahat has joined #openstack-infra | 14:01 | |
| fungi | and on the zuulv3 cacti graphs i see a traffic spike (inbound and outbound?) leading up to complete packet loss around 06:00... periodic jobs kicking off? | 14:01 |
| *** markus_z has quit IRC | 14:02 | |
| fungi | memory usage looks reasonably steady for now since the ~00:00 utc restart at least | 14:03 |
| pabelanger | yah | 14:03 |
| *** eharney has quit IRC | 14:04 | |
| *** psachin has quit IRC | 14:05 | |
| *** amoralej|lunch is now known as amoralej | 14:06 | |
| SpamapS | memory is ticking up steadily since 13:00 UTC actually | 14:09 |
| fungi | guessnig all the live migrations in rax are to deal with recent xen security advisories | 14:09 |
| smcginnis | fungi: We could use a second on here if you have the time: https://review.openstack.org/#/c/513032/ | 14:10 |
| pabelanger | SpamapS: maybe expected, we are running more jobs now then this morniing | 14:10 |
| pabelanger | up to 115 in check pipeline | 14:10 |
| *** salv-orlando has quit IRC | 14:11 | |
| *** salv-orlando has joined #openstack-infra | 14:11 | |
| pabelanger | I'd love to know the different between used memory and cached memory as it relates to zuulv3, is anybody is able to explain | 14:12 |
| *** huanxie has joined #openstack-infra | 14:12 | |
| SpamapS | cached is OS cache | 14:12 |
| SpamapS | as in, stuff on disk sitting in the cache | 14:12 |
| SpamapS | instantly reclaimable at the cost of I/O pain | 14:12 |
| *** sambetts|afk is now known as sambetts | 14:14 | |
| *** yolanda has joined #openstack-infra | 14:16 | |
| *** salv-orlando has quit IRC | 14:16 | |
| pabelanger | it is interesting we basically have same values for both | 14:17 |
| pabelanger | maybe .5GB difference | 14:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 14:18 |
| *** dbecker has joined #openstack-infra | 14:18 | |
| *** baoli has joined #openstack-infra | 14:19 | |
| *** baoli has quit IRC | 14:21 | |
| *** rbrndt has joined #openstack-infra | 14:22 | |
| *** janki has quit IRC | 14:25 | |
| *** camunoz has quit IRC | 14:25 | |
| AJaeger | regarding periodic jobs - these two changes limit the number of wrong jobs running: https://review.openstack.org/512490 and https://review.openstack.org/512524 , reviews welcome | 14:26 |
| chandankumar | clarkb: hello, did you get a chance to look at my username gerrit issue? | 14:26 |
| *** baoli has joined #openstack-infra | 14:28 | |
| *** smatzek has joined #openstack-infra | 14:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Setup nodes for upstream-translation job https://review.openstack.org/513000 | 14:30 |
| boden | hi, do we have any ETA when the “Issues needing triage” will be looked at? For “stable branch ignoring constraints” we have patches in our queue that can’t merge because of this | 14:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Use master for publishing releasenotes https://review.openstack.org/513029 | 14:32 |
| *** erlon has joined #openstack-infra | 14:32 | |
| *** efoley has quit IRC | 14:33 | |
| *** efoley has joined #openstack-infra | 14:33 | |
| *** shiyaz has quit IRC | 14:33 | |
| *** shiyas has quit IRC | 14:33 | |
| *** gmann is now known as gmann_afk | 14:33 | |
| *** shiyaz has joined #openstack-infra | 14:33 | |
| *** shiyas has joined #openstack-infra | 14:34 | |
| jeblair | frickler, AJaeger: i started work on the v3 replacement for the reqs cross test jobs: https://review.openstack.org/512901 | 14:34 |
| fungi | boden: i think people are picking those up as they get freed up from whatever they were fixing previously, so hard to provide clear timelines for remediation. however indicating the severity of the symptoms in here does help us prioritize which ones we grab next | 14:34 |
| openstackgerrit | Merged openstack-infra/project-config master: have release functions script source its own requirements https://review.openstack.org/513032 | 14:34 |
| *** smatzek has quit IRC | 14:34 | |
| mnaser | i just added a new issue for to triage, did my best to try and find the source but i believe it is a zuul bug at this point | 14:34 |
| *** smatzek has joined #openstack-infra | 14:36 | |
| mnaser | i have to afk for most of the day but if anyone could anyone have a quick peek just to check that the problem is well described (aka, makes sense to whoever will pick it up later .. to avoid any delays?) | 14:36 |
| fungi | i'm going to move the fixed issues to the fixed-issues pad, as they're comprising about half of the issues pad again | 14:37 |
| AJaeger | jeblair: I'm eager to learn how this will work - dirk is not happy yet... I'll subscribe to that, thanks! | 14:38 |
| *** david-lyle has joined #openstack-infra | 14:38 | |
| mnaser | fungi i was always gonna suggest to also move it at the end of the etherpad | 14:38 |
| AJaeger | fungi, thanks for moving over! | 14:38 |
| fungi | well, the issues pad itself is getting so long it's starting to get buggy in my browser | 14:38 |
| *** smatzek has quit IRC | 14:40 | |
| *** camunoz has joined #openstack-infra | 14:41 | |
| *** smatzek has joined #openstack-infra | 14:42 | |
| *** jcoufal_ has joined #openstack-infra | 14:42 | |
| *** xarses has joined #openstack-infra | 14:42 | |
| *** huanxie has quit IRC | 14:42 | |
| *** jcoufal_ has quit IRC | 14:42 | |
| jeblair | mordred: can you take a look at https://review.openstack.org/512901 and tell me what's wrong with my use of zuul_work_dir ? | 14:44 |
| *** jcoufal has quit IRC | 14:45 | |
| *** jcoufal has joined #openstack-infra | 14:46 | |
| *** smatzek has quit IRC | 14:46 | |
| *** ociuhandu has quit IRC | 14:46 | |
| clarkb | chandankumar: I haven't but I think I should be able to get to it this morning | 14:47 |
| chandankumar | clarkb: thanks :-) | 14:47 |
| *** smatzek has joined #openstack-infra | 14:48 | |
| *** dbecker has quit IRC | 14:49 | |
| fungi | do we have sufficient debugging details in job logs to indicate which branches/commits were checked out for which required-projects? if so, i'm not finding it | 14:50 |
| fungi | trying to determine exactly which openstack/requirements commit this ran with: http://logs.openstack.org/90/512390/3/check/openstack-tox-pep8/169ad0e/ | 14:51 |
| fungi | because evidence is suggesting it did not run with the same branch of requirements as the proposed change was targeting on its own repo | 14:51 |
| *** smatzek has quit IRC | 14:52 | |
| fungi | zuul-cloner used to be pretty verbose about what was getting checked out | 14:52 |
| *** chlong has quit IRC | 14:52 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Collect output from coverage job https://review.openstack.org/512910 | 14:52 |
| *** shiyas has quit IRC | 14:53 | |
| clarkb | chandankumar: can you confirm that 12393 with ssh username chkumar246 is the account you have been using and intend to use going forward? | 14:54 |
| clarkb | chandankumar: if so I think I can just disable the old account | 14:54 |
| clarkb | fungi: ^ fyi that ssh username and account id match up and the older account (8944) can probably just be disabled I think | 14:54 |
| chandankumar | Account ID 12393 and Username | 14:54 |
| fungi | i suppose the "Determine local HEAD" loop does tell us, as long as we count the same number of log entries to match the order with one of the prior loops which mentions the repository name | 14:54 |
| chandankumar | chkumar246 | 14:54 |
| chandankumar | clarkb: ^^ | 14:55 |
| clarkb | chandankumar: ok cool, I think we will start by just disabling 8944 | 14:55 |
| chandankumar | clarkb: thanks :-) | 14:55 |
| fungi | clarkb: yeah, i'd just mark the older account inactive in that case | 14:55 |
| *** dhinesh has joined #openstack-infra | 14:55 | |
| *** yamamoto has joined #openstack-infra | 14:55 | |
| clarkb | #status log Gerrit account 8944 set to inactive to handle a duplicate account issue | 14:56 |
| openstackstatus | clarkb: finished logging | 14:56 |
| clarkb | chandankumar: should be done now | 14:56 |
| jeblair | fungi: we could probably add that to one of the git repo setup roles | 14:56 |
| fungi | jeblair: yeah, pouring over these, i think all the loops should echo the name of the required-project they're processing at each iteration | 14:58 |
| fungi | because they all tell you what they did, but not what the did it to | 14:58 |
| jeblair | fungi: to be clear, you're saying you think that job did not use stable/ocata for openstack-requirements? | 14:58 |
| fungi | jeblair: not sure yet. it looks like it checked out the right branch for all those, so now i'm starting to wonder if this is a side effect of tox-siblings installing newer source than the constrained version for some dependency | 14:59 |
| openstackgerrit | Sam Betts proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs https://review.openstack.org/511264 | 14:59 |
| *** baoli has quit IRC | 14:59 | |
| *** dhinesh has quit IRC | 14:59 | |
| jeblair | fungi: k. i'll paste some log entries from the executor in the etherpad | 14:59 |
| fungi | thanks | 14:59 |
| *** baoli has joined #openstack-infra | 15:00 | |
| clarkb | fungi: that should be fine as long as the sibling install happens after the constraint install and both sides of the software are happy talking to each other | 15:01 |
| jeblair | fungi: okay one line apparently. and all it says was that it checked out stable/ocata. | 15:01 |
| fungi | jeblair: yep, thanks | 15:01 |
| *** baoli has quit IRC | 15:02 | |
| *** efoley has quit IRC | 15:02 | |
| *** efoley has joined #openstack-infra | 15:02 | |
| *** baoli has joined #openstack-infra | 15:02 | |
| *** yamamoto has quit IRC | 15:05 | |
| *** coolsvap has quit IRC | 15:06 | |
| *** tpsilva has quit IRC | 15:07 | |
| *** ramishra has quit IRC | 15:07 | |
| jeblair | fungi: based on the line you just pasted, it looks like the upgrade neutron-lib requirement came from vmware-nsxlib, which is a sibling package that was upgraded | 15:09 |
| jeblair | this is the upgrade line: http://logs.openstack.org/90/512390/3/check/openstack-tox-pep8/169ad0e/job-output.txt.gz#_2017-10-17_10_43_53_207580 | 15:09 |
| *** ykarel|afk is now known as ykarel | 15:10 | |
| fungi | yeah, i think it's not applying the constraints file when doing that. seems like it's happening from within their tox_install.sh script but it doesn't appear to set -x so hard to see what pip command it's issuing | 15:10 |
| *** trown is now known as trown|brb | 15:11 | |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Create a playbook for publishing contributor-guide https://review.openstack.org/512865 | 15:12 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Set contributor-guide location to /contributors https://review.openstack.org/512871 | 15:12 |
| thingee | AJaeger: thanks for the help ^ | 15:13 |
| pabelanger | jeblair: mordred: Shrews does it make sense to move zuulv3-shim (role: fetch-zuul-cloner) into legacy-base job? I've notice we are installing it for non-legacy jobs currently | 15:13 |
| pabelanger | that would also discourage new jobs that parent to base from using zuul-cloner | 15:14 |
| pabelanger | which, is something I think we want | 15:14 |
| *** trown|brb is now known as trown | 15:15 | |
| *** jamesmcarthur has quit IRC | 15:16 | |
| jeblair | pabelanger: wfm. | 15:16 |
| pabelanger | okay, will work up some jobs | 15:17 |
| *** LindaWang has quit IRC | 15:18 | |
| boden | fungi: I can rerun the patch with -x in our tox_install script if needed | 15:19 |
| boden | tho it seems zuul jobs are lagging a good bit now, so results likely take awhile | 15:19 |
| *** pcaruana has quit IRC | 15:19 | |
| EmilienM | good morning folks, can someone review https://review.openstack.org/#/c/512795/ please? it's a project-config to add coverage on one of our projects. Thanks | 15:19 |
| fungi | boden: i just pushed one (513064) | 15:19 |
| sambetts | AJaeger: is Depends-On: project-config-change supposed to work with zuulv3 ? | 15:20 |
| boden | fungi: ok.. i thought you were debugging openstack/vmware-nsx | 15:20 |
| *** tesseract has quit IRC | 15:20 | |
| boden | not fwaas | 15:20 |
| fungi | boden: oh, yep you're right | 15:20 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Move fetch-zuul-cloner into legacy-base jobs https://review.openstack.org/513067 | 15:20 |
| AJaeger | sambetts: with project-config: no | 15:21 |
| mwhahaha | hey v3 question, can anyone tell me why it ran a playbook from stable/ocata on a master change? http://logs.openstack.org/22/512822/2/check/puppet-openstack-module-build/bdd627e/job-output.txt.gz#_2017-10-18_13_28_13_386929 | 15:21 |
| sambetts | ah thats a shame :( still can't test changes to jobs etc with a depends-on | 15:21 |
| fungi | boden: i pushed 513068 for that now | 15:22 |
| AJaeger | sambetts: you can if those jobs are defined elsewhere... Or add the job as part of your change for testing and move over later | 15:23 |
| *** shiyaz has quit IRC | 15:23 | |
| boden | fungi: ok… BTW a side question… should we (vmware-nsx) define our own bindeps file? I think we want the same as what neutron has, but not sure if need a separate copy to maintain and sync w/neutron | 15:23 |
| pabelanger | mwhahaha: if you look in the inventory file, you can see zuul._inheritance_path variable for the path. it doesn't give the why, but some extra info. I'm looking for myself too | 15:24 |
| fungi | boden: yeah, it's always possible you could require additional system packages which neutron itself does not | 15:25 |
| pabelanger | mwhahaha: i am guess it is because your depends-on links to change that for other branches (ocata) | 15:25 |
| mwhahaha | pabelanger: they didn't exist at the time | 15:25 |
| mwhahaha | pabelanger: i just backported them | 15:25 |
| *** eharney has joined #openstack-infra | 15:25 | |
| fungi | boden: plus, it's intended as a convenience to your developers for them to be able to see what system packages you expect them to install to have a working development environment for that repo | 15:25 |
| boden | fungi: but on the flip side, we now need to sync w/neutron’s all the time | 15:26 |
| mwhahaha | pabelanger: where does that _inheritance_path get generated? deep in the bowels of zuul? | 15:26 |
| fungi | boden: that's a good question... is vmware-nsx a neutron plug-in or something? | 15:26 |
| boden | fungi: yes, a neutron plugin | 15:26 |
| pabelanger | mwhahaha: yah, zuul generates that | 15:26 |
| *** rmcallis has joined #openstack-infra | 15:27 | |
| boden | fungi: would be nice if we could “extend” neutrons bindeps, just if had any to add/change atop what’s done in neutron | 15:27 |
| *** rmcallis has quit IRC | 15:27 | |
| fungi | boden: it's possible we want to think about ways to extend the bindep language to support merging system dependencies from other lists in other projects, though it's a really simple tool at the moment and that seems like a lot of added complication | 15:27 |
| *** hongbin has joined #openstack-infra | 15:27 | |
| *** slaweq_ has joined #openstack-infra | 15:28 | |
| pabelanger | mwhahaha: I'll have to defer to jeblair or mordred on that. I am not sure, I haven't done much cross branch testing yet | 15:28 |
| fungi | boden: basically the goal is, when a developer checks out vmware-nsx, they should be able to read the bindep.txt or run bindep aganist it to see what packages they're missing | 15:28 |
| *** andreas_s has quit IRC | 15:28 | |
| boden | fungi: understood… I may opt to just not define the bindep right now… if we need some other pacakges then I guess we could duplicate neutrons… | 15:28 |
| *** martinkopec has quit IRC | 15:29 | |
| mwhahaha | pabelanger: k i'm backporting those fixes as they are needed anyway but we didn't have anything in the chain that was for ocata unless it came from the zuul queue or something | 15:29 |
| boden | fungi: I totally get the point, it just becomes a real pain to stay in sync with neutrons | 15:29 |
| fungi | boden: agreed, it's probable the idea behind bindep doesn't translate well to repos which are plugins or extensions of some other software | 15:29 |
| pabelanger | mwhahaha: since playbooks is merged now, if you remove depends-on, I'd be curious if that works | 15:30 |
| mwhahaha | pabelanger: yea i'm trying that as well | 15:30 |
| pabelanger | but still would be good to understand why | 15:30 |
| mwhahaha | it's on the zuul issues page | 15:30 |
| fungi | boden: since you often really need that "other" software and its system dependencies to be able to work on and test your plugin | 15:30 |
| mwhahaha | but since this is blocking the rlease i'm trying to figure out some workarounds | 15:30 |
| pabelanger | mwhahaha: this is one area I didn't get a change to test much before rollout, have limited amount of things with branches :) | 15:30 |
| boden | fungi: ack. lets not belabor.. I’m sure you all have plenty of other things to worry about right now :) | 15:31 |
| jeblair | mwhahaha: where in the issues etherpad? | 15:31 |
| EmilienM | quick question, when I look at https://review.openstack.org/#/c/512784/ - why don't we see the all zuul jobs bellow the votes? | 15:31 |
| EmilienM | is this because ovb jobs overrided the previous results | 15:32 |
| EmilienM | so zuul thinks it's a recheck? | 15:32 |
| sambetts | AJaeger: it was more that I wanted to do something like: https://review.openstack.org/#/c/512589/ to test that the changes I'd made in my project-config patch, but I can see why it wouldn't work for project-config | 15:32 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Create a playbook for publishing contributor-guide https://review.openstack.org/512865 | 15:33 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Set contributor-guide location to /contributors https://review.openstack.org/512871 | 15:33 |
| pabelanger | EmilienM: I think we have an open issue on etherpad about that | 15:33 |
| mwhahaha | jeblair: Puppet integration jobs mysteriosuly appearing when they shouldn't (reported by mnaser, no idea what the issue is) | 15:33 |
| jeblair | mwhahaha: thx | 15:33 |
| mwhahaha | jeblair: see green bits under there (that's me) | 15:33 |
| pabelanger | EmilienM: possible related to javascript in gerrit | 15:34 |
| mwhahaha | pabelanger, jeblair: fyi removing depends-on did not fix the inheritance | 15:35 |
| *** dtantsur|brb is now known as dtantsur | 15:36 | |
| *** iyamahat has quit IRC | 15:37 | |
| *** iyamahat has joined #openstack-infra | 15:38 | |
| *** jamesmcarthur has joined #openstack-infra | 15:38 | |
| jeblair | mwhahaha: i'm finding this a little difficult to follow since the current state of the repos seems to have moved on since the problem description. | 15:39 |
| mwhahaha | jeblair: but they haven't | 15:39 |
| mwhahaha | jeblair: wel the original issue they have | 15:40 |
| mwhahaha | jeblair: the current issue is that the build job is failing because it's picking up the wrong common playbook | 15:40 |
| mwhahaha | jeblair: because it's pulling stable/ocata for a master change | 15:40 |
| mwhahaha | let me split that one out into a seperate issue | 15:40 |
| jeblair | mwhahaha: and that ran with all the relevant changes to .zuul.yaml landed? | 15:40 |
| mwhahaha | jeblair: yes | 15:40 |
| jeblair | ok, that helps | 15:40 |
| mwhahaha | stable/ocata playbook is newer in terms of git (not sure if that factors into how the inheritence works | 15:41 |
| jeblair | it shouldn't, but good to know | 15:42 |
| openstackgerrit | Raissa Sarmento proposed openstack-infra/openstack-zuul-jobs master: Remove manila legacy jobs https://review.openstack.org/513075 | 15:43 |
| openstackgerrit | Raissa Sarmento proposed openstack-infra/project-config master: Remove manila legacy jobs https://review.openstack.org/513076 | 15:43 |
| *** CrayZee has joined #openstack-infra | 15:44 | |
| odyssey4me | is there a place I can see which cached images (eg: cirros) are implemented in the nodepool image and where they're located? | 15:46 |
| odyssey4me | assuming that's still a things | 15:46 |
| pabelanger | odyssey4me: you can look at our build logs: http://nb04.openstack.org/dib.ubuntu-xenial.log | 15:46 |
| pabelanger | odyssey4me: but /opt/cache/files is where we store them | 15:47 |
| odyssey4me | oh awesome, thanks | 15:48 |
| *** felipemonteiro__ has quit IRC | 15:48 | |
| pabelanger | np! | 15:49 |
| CrayZee | Hi infra, can anyone please review https://review.openstack.org/#/c/512785 ? | 15:49 |
| *** wolverineav has quit IRC | 15:49 | |
| *** smatzek has joined #openstack-infra | 15:50 | |
| *** Sukhdev has joined #openstack-infra | 15:51 | |
| jeblair | mwhahaha: okay, this is tricky. i'm going to dig into it. it will take a while. | 15:51 |
| mwhahaha | jeblair: can i land the patches to fix it so we can release or should i hold off? | 15:51 |
| mwhahaha | jeblair: because we can work around it by just backporting the fix to stable/ocata | 15:51 |
| jeblair | mwhahaha: ok | 15:52 |
| mwhahaha | i figured you can still see it because the inhieriance shows stable/ocata instead of master | 15:52 |
| mwhahaha | that probably won't change | 15:53 |
| jeblair | mwhahaha: yeah, we sholud be able to look at a later run and verify the fix by looking at that | 15:53 |
| *** links has joined #openstack-infra | 15:55 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove fetch-zuul-cloner from base(-test) job https://review.openstack.org/513079 | 15:55 |
| *** vhosakot has joined #openstack-infra | 15:55 | |
| *** dhinesh has joined #openstack-infra | 15:56 | |
| *** jamesmcarthur has quit IRC | 15:57 | |
| *** jpich has quit IRC | 15:59 | |
| *** slaweq_ has quit IRC | 16:00 | |
| *** dizquierdo has joined #openstack-infra | 16:00 | |
| *** dhinesh has quit IRC | 16:00 | |
| *** e0ne_ is now known as e0ne | 16:01 | |
| *** andreas_s has joined #openstack-infra | 16:01 | |
| *** jamesmcarthur has joined #openstack-infra | 16:02 | |
| pabelanger | infra-root: AJaeger: dmsimard: mnaser: https://review.openstack.org/513067 and https://review.openstack.org/513079 move our zuul-cloner shim into the legacy-base job (out from base). Please take a moment to look, and would like us to land that when we are ready. There is a potential for zuulv3 that parent to base and still us zuul-clone to fail, but the fix would be to reparent to legacy-base. Which, | 16:02 |
| pabelanger | I think is the correct long-term solution | 16:02 |
| *** kjackal_ has quit IRC | 16:02 | |
| boden | fungi: fyi looks like you have some results for https://review.openstack.org/#/c/513068 from zuul | 16:04 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Remove ironic legacy jobs https://review.openstack.org/513084 | 16:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix ZUUL_REFNAME for translation jobs https://review.openstack.org/512675 | 16:08 |
| *** camunoz has quit IRC | 16:09 | |
| *** Apoorva has joined #openstack-infra | 16:09 | |
| *** camunoz has joined #openstack-infra | 16:09 | |
| *** andreas_s has quit IRC | 16:09 | |
| dmsimard | pabelanger, AJaeger: I'd like to iron out https://review.openstack.org/#/c/512194/ (with https://review.openstack.org/#/c/512904/ ) first | 16:12 |
| *** salv-orlando has joined #openstack-infra | 16:12 | |
| fungi | boden: heisenbug? it installed neutron-lib==1.1.0 this time: http://logs.openstack.org/68/513068/1/check/openstack-tox-pep8/f9fea6e/job-output.txt.gz | 16:13 |
| *** ijw has joined #openstack-infra | 16:14 | |
| *** Rockyg has joined #openstack-infra | 16:14 | |
| *** andreas_s has joined #openstack-infra | 16:14 | |
| pabelanger | dmsimard: well, once we move out of base jobs, it makes testing easier too | 16:14 |
| pabelanger | oh | 16:15 |
| pabelanger | no, that is right | 16:15 |
| *** lucasagomes is now known as lucas-afk | 16:15 | |
| boden | fungi: hmm… interesting.. let me retry my patch | 16:15 |
| pabelanger | it doesn't use localhost | 16:15 |
| *** baoli has quit IRC | 16:15 | |
| clarkb | catching up on gerrit status after last night, task queue is mostly empty, melody looks good. Have we seen any behavior that we are worried about in gerrit since last night? | 16:15 |
| pabelanger | clarkb: nothing that I've had to look into yet | 16:16 |
| fungi | boden: oh! the change where you noted the incorrect constraint application for the pep8 job was one where you were modifying the tools/tox_install_project.sh script for that repo | 16:16 |
| *** martinkopec has joined #openstack-infra | 16:16 | |
| boden | fungi: correct | 16:16 |
| fungi | boden: so i think that means something you're doing in 512390 is breaking constraint application | 16:16 |
| boden | fungi: who me? never! ;) | 16:17 |
| *** salv-orlando has quit IRC | 16:17 | |
| fungi | boden: for example, line 37 there where you're not applying constraints but using -U and not --no-deps | 16:17 |
| smcginnis | Can anyone tell me what failed with this? http://logs.openstack.org/d7/d716f5e96f3562e09bfc54d85de0bcc1091fb1e1/release-post/tag-releases/3bedc9e/job-output.txt.gz | 16:17 |
| fungi | boden: so it's basically blindly upgrading to the newest versions which match the dependency spec | 16:18 |
| smcginnis | POST_FAILURE again. Something external? | 16:18 |
| boden | fungi: ack.. I’m revisiting… sorry I’m taking over this gate logic for someone else so I’m still newbie in this area | 16:18 |
| pabelanger | smcginnis: job passed, will need to look at executor | 16:18 |
| pabelanger | looking now | 16:19 |
| smcginnis | pabelanger: Thank you. | 16:19 |
| fungi | boden: i _think_ that logic should already be handled by the tox-siblings role we're applying anyway, but mordred might have some ideas about what to do in 512390 (if he's around) | 16:19 |
| pabelanger | Oh | 16:19 |
| pabelanger | how neat | 16:19 |
| pabelanger | smcginnis: there is a syntax error in project-config/playbooks/release/post.yaml | 16:20 |
| pabelanger | but we didn't expose it properly to end user | 16:20 |
| clarkb | pabelanger: I checked inode and disk usage really quickly just to be sure it wasn't that and we look good | 16:20 |
| boden | fungi: hmm, I had thought projects were still required to install dependant projects… thats what I was told the other day, but maybe I misunderstood | 16:20 |
| pabelanger | smcginnis: can you add this to etherpad so we can debug properly | 16:20 |
| clarkb | 08 is currently most utilized | 16:20 |
| pabelanger | we should be bubbling up that error into gerrit | 16:21 |
| *** trown is now known as trown|lunch | 16:21 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix typo in remove-launchpad-credentials role https://review.openstack.org/513090 | 16:23 |
| pabelanger | smcginnis: clarkb: ^ | 16:23 |
| *** signed8b_ has quit IRC | 16:23 | |
| pabelanger | also, our global linter job would have caught that too | 16:23 |
| smcginnis | pabelanger: That was the cause for the post_failure? | 16:23 |
| pabelanger | smcginnis: Yah, this really should have been failure | 16:24 |
| pabelanger | but, we seem to have an issue in zuulv3 | 16:24 |
| *** alex_xu has quit IRC | 16:24 | |
| smcginnis | pabelanger: Cool, thanks! | 16:24 |
| clarkb | pabelanger: I don't see that role in project-config or ozj | 16:25 |
| pabelanger | smcginnis: adding this to issue to https://etherpad.openstack.org/p/zuulv3-issues will help us properly debug it, if you don't mind added it | 16:25 |
| pabelanger | clarkb: zuul-jobs | 16:25 |
| clarkb | huh thatseems like a weird one to have in zuul-jobs | 16:25 |
| smcginnis | Sure | 16:25 |
| *** salv-orlando has joined #openstack-infra | 16:25 | |
| pabelanger | ty | 16:26 |
| *** andreas_s has quit IRC | 16:28 | |
| *** links has quit IRC | 16:28 | |
| *** baoli has joined #openstack-infra | 16:30 | |
| *** baoli has quit IRC | 16:30 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Remove py35 based template from nodepool https://review.openstack.org/513092 | 16:30 |
| *** alex_xu has joined #openstack-infra | 16:31 | |
| *** baoli has joined #openstack-infra | 16:32 | |
| *** shiyaz has joined #openstack-infra | 16:32 | |
| *** markvoelker has quit IRC | 16:32 | |
| *** markvoelker has joined #openstack-infra | 16:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix branch ordering when loading in-repo config https://review.openstack.org/513094 | 16:34 |
| *** vhosakot has quit IRC | 16:34 | |
| *** vhosakot has joined #openstack-infra | 16:35 | |
| jeblair | mwhahaha, mnaser: ^ i think that's the fix | 16:37 |
| openstackgerrit | JiaJunsu proposed openstack-infra/bindep master: Add dependency in bindep installation doc https://review.openstack.org/513096 | 16:37 |
| *** mat128 has quit IRC | 16:38 | |
| *** markvoelker has quit IRC | 16:38 | |
| clarkb | jeblair: question on ^ but otherwise makes sense | 16:39 |
| *** wolverineav has joined #openstack-infra | 16:39 | |
| *** vhosakot has quit IRC | 16:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Revert "Use weakref for change cache" https://review.openstack.org/513097 | 16:40 |
| *** vhosakot has joined #openstack-infra | 16:40 | |
| *** andreas_s has joined #openstack-infra | 16:41 | |
| *** derekh has quit IRC | 16:42 | |
| jeblair | clarkb: responded | 16:42 |
| jeblair | clarkb: oh i think i misunderstood | 16:43 |
| *** gyee has joined #openstack-infra | 16:43 | |
| jeblair | clarkb: responded again | 16:43 |
| clarkb | jeblair: thanks | 16:43 |
| *** dizquierdo has quit IRC | 16:45 | |
| jeblair | the etherpad is turning into a really weird mix of code and normal text | 16:45 |
| *** vhosakot has quit IRC | 16:46 | |
| *** vhosakot has joined #openstack-infra | 16:46 | |
| SamYaple | code is normal text to me | 16:47 |
| clarkb | I see the empty tuple now, the lack of self. prefix setting those attributes broke my grepping | 16:47 |
| jeblair | i'm using the etherpad term of art. it uses the word "Normal". | 16:48 |
| jeblair | clarkb: ah yeah, initializers are special in that class | 16:48 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Revert "Add linters job to all OSA role jobs" https://review.openstack.org/513125 | 16:50 |
| odyssey4me | mordred ^ you were right, sorry | 16:50 |
| mwhahaha | jeblair: i also had a question about the sort ordering | 16:50 |
| *** andreas_s has quit IRC | 16:50 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Switch puppet-nova to in-tree testing https://review.openstack.org/513131 | 16:50 |
| *** shiyas has joined #openstack-infra | 16:50 | |
| *** Sukhdev has quit IRC | 16:50 | |
| odyssey4me | I'd appreciate it if we could get that patch reviewed quite quickly as we've found ourselves blocked on some repositories. | 16:51 |
| *** andreas_s has joined #openstack-infra | 16:51 | |
| *** jamesmcarthur has quit IRC | 16:53 | |
| *** shiyaz has quit IRC | 16:54 | |
| jeblair | mwhahaha: replied, let me know if that makes sense. | 16:55 |
| *** andreas_s has quit IRC | 16:55 | |
| jeblair | odyssey4me: +2; let's ask pabelanger to review that too | 16:56 |
| odyssey4me | thanks jeblair - hopefully soon you'll be fielding less of these and more interesting things :) | 16:56 |
| pabelanger | jeblair: odyssey4me" +3 | 16:56 |
| odyssey4me | ta | 16:56 |
| EmilienM | AJaeger: hey, I might use some help. I'm trying to move puppet jobs in-tree for puppet-nova (starting small), I would love your review on https://review.openstack.org/#/q/topic:zuulv3/puppet-nova+(status:open+OR+status:merged) when you can | 16:58 |
| mwhahaha | jeblair: i think you just are moving the problem with that patch | 16:59 |
| *** e0ne has quit IRC | 16:59 | |
| jeblair | mwhahaha: we may be talking past each other | 17:01 |
| *** ralonsoh has quit IRC | 17:01 | |
| mwhahaha | fair enough, i'll defer to you guys on this | 17:01 |
| jeblair | mwhahaha: if you look at the inheritance path debug info for that build, you'll note that all the things on ocata lacked branch matchers | 17:02 |
| jeblair | mwhahaha: that's why the ocata jobs ended up running on master changes | 17:02 |
| fungi | boden: the tox-siblings role will, for regular tox-based jobs, install the checkouts of your required-projects into the target tox env without you needing to explicitly do the same. regardless, the bug seems to be that in tools/tox_install_project.sh for that change you're adding a `pip install -U -e ${PROJ_DIR}` which is unconditionally installing and if possible upgrading (because of -U) all | 17:02 |
| fungi | dependencies (because of not using --no-deps) without respecting constraints (because of not using -c$CONSTRAINTS_FILE) | 17:02 |
| mwhahaha | jeblair: ok so sounds like we have a bug in our definitions then | 17:03 |
| pabelanger | ah, that now make sense after jeblair explained it | 17:03 |
| *** leyal has quit IRC | 17:04 | |
| jeblair | mwhahaha: with this fix in place, those jobs will end up with the ocata branch matchers they should have had. | 17:04 |
| *** sdague has quit IRC | 17:04 | |
| jlvillal | Any issue with: http://zuulv3.openstack.org/ ? I saw "proxy" error. And when I reloaded the page I don't see any jobs. | 17:04 |
| *** leyal has joined #openstack-infra | 17:05 | |
| jeblair | jlvillal: looks like a memory usage spike | 17:05 |
| boden | fungi: ok so we dont need to install the required-projects anymore; thats news to me | 17:06 |
| odyssey4me | argh, that might be my fault - I just pushed ~40 patches in quick succession | 17:06 |
| jeblair | odyssey4me: did they all have .zuul.yaml changes? | 17:06 |
| odyssey4me | yup | 17:07 |
| jeblair | odyssey4me: that would match the symptoms then. :) | 17:07 |
| odyssey4me | sorry | 17:07 |
| SamYaple | lol | 17:07 |
| odyssey4me | :D | 17:07 |
| boden | fungi: so what should ‘testenv’ in our tox.ini be doing then? admittedly I’m confused now | 17:08 |
| jeblair | each one of those uses something like 100mb of ram (i don't know the exact number) | 17:08 |
| *** ykarel is now known as ykarel|away | 17:09 | |
| jlvillal | Is there some check we can add to our pep8 jobs to check zuul.d/ changes we make in our repository? Or is that already done by something else? | 17:09 |
| jlvillal | Basically I was thinking sanity check. Properly formed YAML and things along those lines. | 17:10 |
| pabelanger | jlvillal: great question! | 17:10 |
| jlvillal | And something that we could run locally in our tox job | 17:10 |
| *** ociuhandu has joined #openstack-infra | 17:10 | |
| jeblair | jlvillal: zuul itself will check the syntax and validity | 17:10 |
| *** ykarel|off has joined #openstack-infra | 17:10 | |
| *** Rockyg has quit IRC | 17:10 | |
| fungi | boden: i'm not super clear still on how to leverage tox-siblings in such situations, but as far as your 512390 change i think you probably want to add the --no-deps option to line 37 of tox_install_project.sh | 17:10 |
| jlvillal | pabelanger: :) | 17:10 |
| jlvillal | jeblair: Okay. Thanks. That should work. | 17:11 |
| *** Rockyg has joined #openstack-infra | 17:11 | |
| *** jpena is now known as jpena|off | 17:11 | |
| pabelanger | ++ | 17:11 |
| *** Swami has joined #openstack-infra | 17:12 | |
| jeblair | infra-root: we may need a bigger zuulv3 server -- not because of memory leaks, but in order to handle large numbers of simultaneous config changes | 17:13 |
| jeblair | also, i did manage to grab queues | 17:13 |
| jeblair | maybe we should restart the scheduler now? | 17:14 |
| fungi | jeblair: oh, wow, it did spike up crazily there huh? | 17:14 |
| clarkb | jeblair: tis restart would get the branch fix you just pushed? | 17:14 |
| fungi | and yes, i think a restart is required at this point | 17:14 |
| jeblair | clarkb: i'll have to manually pull it in | 17:14 |
| jeblair | but i think that's reasonable to do | 17:14 |
| fungi | and i guess we can put the weakref patch back in (it'll already be checked out i suppose?) | 17:15 |
| pabelanger | wfm | 17:15 |
| jeblair | fungi: can you clarify weakref? | 17:15 |
| jeblair | fungi: like, do you want the patch reverted or...? | 17:15 |
| *** Swami has quit IRC | 17:15 | |
| fungi | jeblair: last night you restarted without the weakref patch you said | 17:16 |
| fungi | on a hunch that it was leaking memory, leading to the oom condition we saw | 17:16 |
| *** Swami has joined #openstack-infra | 17:16 | |
| fungi | which now seems less likely | 17:17 |
| jeblair | fungi: ah, i see. no, i think weakref is still bad | 17:17 |
| fungi | okay | 17:17 |
| jeblair | fungi: the current condition is because odyssey4me pushed up 40 .zuul.yaml changes (not his fault! he should totally be able to do that!) | 17:17 |
| jeblair | but each .zuul.yaml change eats something like 100mb of ram | 17:18 |
| fungi | got it. so we temporarily balloon (but then never reclaim because python, we can only reuse thereafter) additional memory allocation | 17:18 |
| jeblair | fungi: exactly | 17:18 |
| *** bhavik1 has joined #openstack-infra | 17:18 | |
| *** andreas_s has joined #openstack-infra | 17:18 | |
| tosky | dear -infra, there is a patch that should have been merged, but it did not happen: https://review.openstack.org/#/c/507080/ | 17:18 |
| tosky | despite the final +2 from zuul | 17:19 |
| tosky | now, maybe the patch got caught in the transition and recheck, but what should we do? | 17:19 |
| dmsimard | jeblair: when you say "zuulv3 server", are there different components that live on that node that could be split out or scaled horizontally ? | 17:19 |
| fungi | i suppose however large we make the available memory on the scheduler, there's a risk that enough zuul configurations enqueued in parallel will exhaust available memory... is there any easy way to throttle how many we're try in parallel? | 17:19 |
| fungi | s/we're/we'll/ | 17:20 |
| jeblair | starting scheduler | 17:20 |
| *** felipemonteiro_ has joined #openstack-infra | 17:20 | |
| *** jamesmcarthur has joined #openstack-infra | 17:20 | |
| jeblair | dmsimard: this is the non-scalable scheduler component (scaling this is zuul v4) | 17:20 |
| odyssey4me | yeah, I was just thinking the same - is there some sort of rate limiter? and can the scheduling somehow be scaled horizontally | 17:21 |
| dmsimard | jeblair: ok, so when we say "the zuulv3 server", it's really the scheduler ? | 17:21 |
| *** felipemonteiro__ has joined #openstack-infra | 17:21 | |
| *** shiyas has quit IRC | 17:21 | |
| dmsimard | I'm not intimately familiar with the v3 server architecture (yet), sorry if that sounds obvious | 17:21 |
| jeblair | dmsimard: well, when i said we need a "bigger zuulv3 server" i meant, literally, the server that we use for zuulv3.openstack.org needs to be bigger | 17:22 |
| clarkb | tosky: that can happen if you have unmet depends on or if the parent commit isfor and old patchset on a change (neither case apperas to be the state here) | 17:22 |
| *** huanxie has joined #openstack-infra | 17:22 | |
| frickler | tosky: clarkb: seems that patch needs a rebase | 17:22 |
| dmsimard | jeblair: yup, I was looking at https://docs.openstack.org/infra/zuul/admin/components.html so that scheduler box is the one thing we're talking about. Ok. | 17:22 |
| *** Rockyg has quit IRC | 17:22 | |
| tosky | clarkb: but in both cases shouldn't it not do the final merge, or complain that something failed to merge or so? | 17:22 |
| jeblair | dmsimard: normally "zuulv3 server" doesn't really mean anything from a zuul architecture pov. the scheduler is the zuul component under discussion. so, to tie it off -- the server we run the scheduler on may need to be bigger. | 17:22 |
| dmsimard | jeblair: for me zuulv3.openstack.org could've been the webui :/ | 17:23 |
| frickler | it is based on PS3 of https://review.openstack.org/505741 but PS4 got merged there | 17:23 |
| tosky | frickler: I rebased, but still it was a bit puzzling | 17:23 |
| clarkb | oh it is that situation | 17:23 |
| frickler | and indeed if that is the issue, zuul should be more verbose about it | 17:23 |
| *** Apoorva_ has joined #openstack-infra | 17:23 | |
| jeblair | odyssey4me, fungi: yeah, i'll think about whether we can limit. | 17:23 |
| *** rbrndt has quit IRC | 17:24 | |
| *** jamesmcarthur has quit IRC | 17:24 | |
| odyssey4me | the limit could perhaps be implemented either on the send side (gearman), or the receive side (scheduler)... | 17:24 |
| *** dtantsur is now known as dtantsur|afk | 17:24 | |
| odyssey4me | assuming I'm looking at the right diagram, which dmsimard just linked | 17:24 |
| jeblair | odyssey4me: for the moment, do you think you could just submit a few of those at a time? maybe start with 5 and see what happens? | 17:24 |
| odyssey4me | jeblair they're all pushed up now - so I can recheck bunches at a time if you like | 17:25 |
| odyssey4me | ? | 17:25 |
| jeblair | also, i really want to have the scheduler emit a graphite metric of "number of layouts in memory", but i haven't gotten back to that | 17:25 |
| *** felipemonteiro_ has quit IRC | 17:25 | |
| jeblair | odyssey4me: sounds good | 17:25 |
| jeblair | i'm about to re-enqueue the changes that were in check, but will omit these openstack-ansible related changes | 17:25 |
| *** Apoorva has quit IRC | 17:26 | |
| odyssey4me | ok, so I should hold on for a bit? | 17:26 |
| *** amoralej is now known as amoralej|off | 17:27 | |
| jeblair | odyssey4me: it should be okay for you to go ahead and recheck the first few (though it'll take zuul a few minutes to catch up to that point) | 17:27 |
| *** jamesmcarthur has joined #openstack-infra | 17:27 | |
| odyssey4me | should I recheck this one, or are you reenqueing it? https://review.openstack.org/#/c/513125/ | 17:28 |
| jeblair | odyssey4me: that one is on my list to re-enqueue | 17:28 |
| AJaeger | pabelanger: your zuul-cloner move looks fine - didn't approve in case others want to review. | 17:28 |
| pabelanger | AJaeger: maybe clarkb or fungi can review and approve :D | 17:29 |
| *** mat128 has joined #openstack-infra | 17:29 | |
| pabelanger | https://review.openstack.org/513079/ and depends-on | 17:29 |
| dmsimard | We need to figure out the issue with the linking/permission thing though | 17:29 |
| odyssey4me | jeblair if it matters - most of those patches included a depends-on as well | 17:30 |
| *** efoley has quit IRC | 17:30 | |
| SamYaple | odyssey4me: you should be in QA | 17:30 |
| pabelanger | dmsimard: sure, but that is independant of this | 17:30 |
| *** andreas_s has quit IRC | 17:31 | |
| *** jamesmcarthur has quit IRC | 17:32 | |
| jeblair | odyssey4me: that shouldn't use any more memory, it's still just one copy of the layout per patch that changes it (even if it takes multiple changes to build that configuration). | 17:33 |
| *** vhosakot has quit IRC | 17:34 | |
| *** bobh has joined #openstack-infra | 17:35 | |
| *** andreas_s has joined #openstack-infra | 17:36 | |
| *** trown|lunch is now known as trown | 17:37 | |
| AJaeger | EmilienM: commented | 17:37 |
| EmilienM | AJaeger: thank yopu | 17:37 |
| EmilienM | you* | 17:37 |
| SpamapS | jeblair: have you thought about maybe a governor for the scheduler too? Like, queue up events somewhere not-memory if we have more than X layouts? | 17:38 |
| *** jamesmcarthur has joined #openstack-infra | 17:39 | |
| *** e0ne has joined #openstack-infra | 17:41 | |
| *** electrofelix has quit IRC | 17:41 | |
| *** tosky has quit IRC | 17:41 | |
| jeblair | SpamapS: if we stop processing events, we won't make effective use of resources. but it might be possible to go ahead and enqueue the items but not process their layouts until we have more space. | 17:41 |
| jeblair | SpamapS: that would let non-layout-modifying changes continue to get tested | 17:41 |
| dhellmann | https://review.openstack.org/#/c/513090/ was approved ~40 minutes ago and doesn't appear on http://zuulv3.openstack.org and isn't merged. Was there a restart? Should I recheck it? | 17:44 |
| *** martinkopec has quit IRC | 17:44 | |
| *** Apoorva_ has quit IRC | 17:44 | |
| *** Apoorva has joined #openstack-infra | 17:44 | |
| jeblair | dhellmann: yes, that must have slipped through, sorry | 17:44 |
| dhellmann | jeblair ; np, I can recheck it | 17:45 |
| *** sambetts is now known as sambetts|afk | 17:46 | |
| SpamapS | jeblair: isn't the reason we're behind and queueing (and thus holding on to old layouts) because we're already starved for resources? | 17:47 |
| odyssey4me | happy for me to enqueue some more? | 17:47 |
| clarkb | pabelanger: fungi: when you get a moment, care to look at https://etherpad.openstack.org/p/infra-sydney-summit-project-update ? | 17:47 |
| melwitt | AJaeger: thanks for commenting on the patch I was trying to test. so it only needs a rebase? no other changes for zuul v3? | 17:47 |
| jeblair | SpamapS: we're not behind and queueing -- we ran out of memory | 17:47 |
| *** bhavik1 has quit IRC | 17:47 | |
| SpamapS | jeblair: I thought we ran out of memory because we were holding onto layouts because we were behind and queueing. | 17:48 |
| jeblair | SpamapS: no it's because too many layout changes were uploaded at once. | 17:49 |
| SpamapS | ew.... that's a nasty one | 17:49 |
| SpamapS | kk | 17:49 |
| *** ykarel|away has quit IRC | 17:50 | |
| jeblair | odyssey4me: yes, zuul seems to be in a steady state at 5G now, feel free to recheck a few more. | 17:50 |
| AJaeger | melwitt: no, rebase - and touch the right files. Which change was it? | 17:50 |
| *** ykarel|off has quit IRC | 17:50 | |
| pabelanger | clarkb: fungi: yah, looks good in general, that is already a lot to talk about in 20mins | 17:50 |
| melwitt | AJaeger: this one, basically what I want to do is get a tempest novnc test run on it to see if the TLS encryption stack under it works https://review.openstack.org/#/c/396289 | 17:50 |
| clarkb | pabelanger: ya I think we mostly want to trim it down from here (there are a few things where I wonder if we want to go into detail about them, curious about thoughts on those) | 17:50 |
| *** dhinesh has joined #openstack-infra | 17:51 | |
| pabelanger | clarkb: I have to run to airport now, but will have some internet before flights | 17:52 |
| AJaeger | melwitt: that will not work as you did. If you want to run it, you can copy the job definition from openstack-zuul-jobs and the playbook over to nova, add a .zuul.yaml with the job in it and run it. | 17:52 |
| melwitt | AJaeger: and I was thinking maybe I could do that by running the experimental queue. but I'm seeing now there's no such thing as experimental job for project-config | 17:52 |
| AJaeger | melwitt: the job is now called legacy-oldname and defined in openstack-zuul-jobs. That change can be abandoned | 17:52 |
| AJaeger | melwitt: and even if there was an experimental queue, it would not run thta change ;) | 17:53 |
| melwitt | AJaeger: haha, okay. sorry I don't know what I'm doing. okay, I'll try to do what you said. thanks :) | 17:53 |
| *** andreas_s has quit IRC | 17:54 | |
| AJaeger | melwitt: see some of the move zuul v3 changes - https://review.openstack.org/#/q/file:.zuul.yaml | 17:54 |
| AJaeger | melwitt: and just move over this single job. Definition and playbooks are in openstack-zuul-jobs. | 17:54 |
| melwitt | AJaeger: thank you | 17:55 |
| AJaeger | melwitt: and add it to your .zuul.yaml file - and then push and have it test it | 17:55 |
| melwitt | k | 17:55 |
| AJaeger | melwitt: give it a try and if you need help, feel free to ask here again | 17:55 |
| *** lbragstad has quit IRC | 17:56 | |
| AJaeger | SamYaple: do you want to retire these loci repos completely? This needs a change to project-config to take them out... | 17:59 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Move fetch-zuul-cloner into legacy-base jobs https://review.openstack.org/513067 | 17:59 |
| *** andreas_s has joined #openstack-infra | 17:59 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 17:59 |
| odyssey4me | jeblair ready for another batch? | 18:00 |
| SamYaple | AJaeger: that looks like what will probably happen yes, but not right at this second. we want ot be absolutely sure first | 18:00 |
| AJaeger | SamYaple: ok - you could still revert... | 18:00 |
| *** ociuhandu has quit IRC | 18:00 | |
| SamYaple | oh you mean retire and then revert the retirement? | 18:00 |
| *** rloo has joined #openstack-infra | 18:01 | |
| *** baoli has quit IRC | 18:02 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Switch puppet-nova to in-tree testing https://review.openstack.org/513131 | 18:02 |
| EmilienM | AJaeger: replied, updated. | 18:03 |
| *** huanxie has quit IRC | 18:03 | |
| *** baoli has joined #openstack-infra | 18:04 | |
| jeblair | odyssey4me: give it a few more minutes -- you can tell that it hasn't finished creating configuration for the current batch because the later entries in check don't have a list of jobs yet | 18:04 |
| jeblair | odyssey4me: though looks like it's on the last one now | 18:05 |
| odyssey4me | jeblair ok, I'll monitor the page to verify that it's showing them all before adding more | 18:05 |
| odyssey4me | thanks a bunch | 18:06 |
| jeblair | odyssey4me: thank you! | 18:06 |
| jeblair | we're at 6G of ram now, i'd feel comfortable pushing this up to 8G, but when we get there, i'd wait for the earlier ones to finish and report back before pushing up more. | 18:07 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all | 18:07 |
| clarkb | that has a 5 minute refresh interval | 18:08 |
| *** tosky has joined #openstack-infra | 18:10 | |
| odyssey4me | alright, let me wait for ~30 mins to give it time to catch up a bit | 18:10 |
| AJaeger | EmilienM: yes, that file is v2 JJB config | 18:10 |
| EmilienM | AJaeger: perfect. Thx for confirming | 18:11 |
| *** camunoz has quit IRC | 18:12 | |
| *** baoli has quit IRC | 18:13 | |
| *** baoli has joined #openstack-infra | 18:14 | |
| *** andreas_s has quit IRC | 18:16 | |
| *** lbragstad has joined #openstack-infra | 18:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo in remove-launchpad-credentials role https://review.openstack.org/513090 | 18:17 |
| AJaeger | jeblair: https://pypi.python.org/pypi/zuul-sphinx does not have zuul-sphinx 0.2.0 which you tagged yesterday | 18:19 |
| AJaeger | dmsimard, clarkb, fungi, frickler, could you go over the list of changes on top of https://etherpad.openstack.org/p/zuulv3-issues and +2A as you see fit, please? | 18:20 |
| dmsimard | AJaeger: yeah I'm going through a batch of reviews right now | 18:20 |
| AJaeger | dmsimard: great, thanks! | 18:20 |
| Shrews | clarkb: AJaeger: if either of you have a moment, very easy review to fix our nodepool jobs: https://review.openstack.org/513092 | 18:21 |
| *** Sukhdev has joined #openstack-infra | 18:21 | |
| jeblair | AJaeger: yep, jobs were broken, haven't had a chance to re-enqueue | 18:21 |
| AJaeger | Shrews: +3 | 18:22 |
| Shrews | AJaeger: danke! | 18:22 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for emit-job-header and set-zuul-log-path-fact https://review.openstack.org/511948 | 18:22 |
| *** rbrndt has joined #openstack-infra | 18:23 | |
| *** ykarel|away has joined #openstack-infra | 18:23 | |
| *** ykarel|off has joined #openstack-infra | 18:23 | |
| fungi | dhellmann: wow, i was just reading through the first message in the entrypoints thread on distutils-sig and thinking, "hrm... i should probably point this one out to dhellmann" ;) | 18:24 |
| dhellmann | fungi :-) | 18:24 |
| *** rwsu has quit IRC | 18:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Zun-client legacy jobs https://review.openstack.org/510422 | 18:25 |
| *** mat128 has quit IRC | 18:25 | |
| clarkb | jeblair: I'm not sure I understand the -2 on https://review.openstack.org/#/c/509436/ "2 changes followed by" ? In any case that change now depends on the change that adds the tests. Is that sufficient to remove your -2? | 18:25 |
| *** camunoz has joined #openstack-infra | 18:26 | |
| *** andreas_s has joined #openstack-infra | 18:26 | |
| *** ijw has quit IRC | 18:27 | |
| jeblair | clarkb: at the time, there was a 2 patch series, and then a test pointed at the second of those. | 18:29 |
| AJaeger | EmilienM: btw. you just migrate the templates, not much sense in that IMHO. I would move jobs over, rewrite them so that they are native v3 - and then update/migrate templates. | 18:30 |
| jeblair | clarkb: that meant that the first change was effectively untested | 18:30 |
| jeblair | (alone) | 18:30 |
| *** weshay is now known as weshay|ruck|brb | 18:30 | |
| EmilienM | AJaeger: we want to keep playbooks in puppet-openstack-integration I guess, to avoid duplication | 18:30 |
| jeblair | clarkb: yes, the current situation is fine | 18:31 |
| AJaeger | EmilienM: having them in a central place is fine. | 18:31 |
| markmcclain | I'm looking for a pointer on how to fix the set of tests Zuul runs for stable/newton in networking-arista. After the switch to v3, openstack-tox-py35 is now enabled and we didn't support py35 until Ocata. | 18:31 |
| EmilienM | AJaeger: we'll refactor them for sure | 18:31 |
| EmilienM | AJaeger: it's a separated effort though, I guess | 18:31 |
| EmilienM | AJaeger: my first effort is to cleanup project-config and run the intree layout | 18:32 |
| AJaeger | EmilienM: wait a sec - I see one problem. | 18:32 |
| clarkb | dmsimard: what does failed_when: false mean? | 18:33 |
| AJaeger | EmilienM: you want to move the project-template in the end in-tree as well, correct? | 18:33 |
| *** slaweq_ has joined #openstack-infra | 18:33 | |
| dmsimard | clarkb: it means that task will never fail | 18:33 |
| *** sdague has joined #openstack-infra | 18:33 | |
| dmsimard | clarkb: as opposed to 'ignore_errors: yes' which properly fails on failure but is non-fatal to the playbook execution | 18:33 |
| *** andreas_s has quit IRC | 18:33 | |
| AJaeger | EmilienM: I mean those three templates you currently have in puppet-nova | 18:33 |
| clarkb | dmsimard: is there a reason we do that in tests https://review.openstack.org/#/c/504789/20/tests/multi-node-bridge.yaml then check all the results after the fact? | 18:33 |
| EmilienM | AJaeger: yeah | 18:33 |
| EmilienM | AJaeger: no the templates are already in p-o-i | 18:34 |
| EmilienM | AJaeger: in https://github.com/openstack/puppet-openstack-integration/blob/master/.zuul.yaml | 18:34 |
| clarkb | oh I see there is a question about hat from ianw too | 18:34 |
| dmsimard | clarkb: that's because we want the assert to decide on the failure | 18:34 |
| AJaeger | EmilienM: checking | 18:34 |
| *** ykarel|away has quit IRC | 18:34 | |
| AJaeger | EmilienM: puppet-module-unit-jobs not | 18:34 |
| dmsimard | clarkb: otherwise if the task fails, it's fatal and the asserts don't get to run so we lose out on failures that occur later in the execution | 18:34 |
| *** ykarel|off has quit IRC | 18:35 | |
| clarkb | dmsimard: ok I see the comment explaining it now. I'd worry about accidentally letting tests pass because we forget to check something rather than just letting the tsks fail but we can worry about that if we run into it | 18:35 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Run gerrit periodic jobs on proper branch https://review.openstack.org/512490 | 18:35 |
| dmsimard | clarkb: the failed_when defines the task status, if the thing doesn't ping, it'll return rc == 1 and we assert that rc == 0 | 18:35 |
| EmilienM | AJaeger: ok, please comment (/me afk 15m for lunch) | 18:36 |
| AJaeger | EmilienM: so, move puppet-module-unit-jobs first in-repo - otherwise you need to update lots of repos when you move it around | 18:36 |
| AJaeger | your call - will comment... | 18:36 |
| clarkb | dmsimard: right the minor concern si that we'll add a check or modify checks and forget to do the rc == 0 test (or equivalent) | 18:37 |
| dhellmann | the latest release job failure is because the job doesn't have gerrit credentials. Do we have a role or something to add those to a job already? | 18:37 |
| dmsimard | clarkb: we can let the task fail, I don't have a strong opinion | 18:37 |
| *** slaweq_ has quit IRC | 18:37 | |
| EmilienM | AJaeger: thx for your patience, still learning here | 18:38 |
| clarkb | dmsimard: its fine I've approved the change, just wanted to understand what was going on there | 18:38 |
| clarkb | dhellmann: this is gerrit credentials to push tag(s)? | 18:39 |
| dhellmann | clarkb : yes | 18:39 |
| dhellmann | the job is failing on "git review -s" right now | 18:39 |
| AJaeger | EmilienM: enjoy lunch | 18:39 |
| dhellmann | I have a release_ssh_key set up, but I guess that's not enough? | 18:39 |
| dmsimard | AJaeger: for https://review.openstack.org/#/c/512323/, zuul/layout.yaml is v2 exclusive right ? | 18:39 |
| AJaeger | dhellmann: we need to install git-review then | 18:39 |
| dhellmann | AJaeger : git review is now being installed, we updated the bindep | 18:39 |
| dhellmann | AJaeger, clarkb : http://logs.openstack.org/9e/9e810dbbd2f786836460ae79153c022d1b57358f/release-post/tag-releases/093107d/job-output.txt.gz#_2017-10-18_18_32_06_187252 | 18:40 |
| dmsimard | AJaeger: sounds like flaper87 is adding a v2 job which will not be very effective | 18:40 |
| clarkb | dhellmann: AJaeger you may need to configure git(-review) to use the correct username | 18:40 |
| AJaeger | dmsimard: yes, it is - but I left that through in case we would go back to v2 that we have minimal setup. | 18:40 |
| clarkb | *use the correct gerrit username | 18:40 |
| dhellmann | clarkb : I have no idea to do that under the new system. | 18:40 |
| dmsimard | AJaeger: ok, I'll +A with a comment | 18:40 |
| AJaeger | dmsimard: unlikely now - so if you want to -1, we can remove that as well | 18:40 |
| clarkb | dhellmann: I don't think its any different than with the old syste, git config --global gitreview.username 'openstack-proposal-bot' or whatever the name is | 18:41 |
| AJaeger | dmsimard: https://review.openstack.org/#/c/512330 is the one that ads the jobs, we need two steps | 18:41 |
| clarkb | dhellmann: do you have a link to a logfile? | 18:41 |
| dmsimard | AJaeger: oh, I wasn't there yet, okay | 18:41 |
| dhellmann | http://logs.openstack.org/9e/9e810dbbd2f786836460ae79153c022d1b57358f/release-post/tag-releases/093107d/job-output.txt.gz#_2017-10-18_18_32_06_187252 | 18:41 |
| dhellmann | I see those steps being run. maybe something is happening in the wrong order | 18:41 |
| fungi | clarkb: problem with the old system is that it was a persistent node, so as long as the job had at one time correctly configured global things like that, it was entirely possible for it to silently regress later | 18:42 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove py35 based template from nodepool https://review.openstack.org/513092 | 18:42 |
| clarkb | ya that traceback definitely seems to point at "your username + auth" didn't work. So if we assume auth/key isgood then should be the username | 18:42 |
| fungi | i can dump the ~jenkins/.gitconfig from signing01.ci.o.o... just a sec | 18:42 |
| clarkb | dhellmann: the first traceback happens before git config runs in that job | 18:43 |
| clarkb | dhellmann: the second one fails too though after git config happens | 18:43 |
| dhellmann | clarkb : http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/release-tools/functions#n128 | 18:43 |
| dhellmann | that's where we try to use git review, see it fails, do the configuration, then try again | 18:44 |
| clarkb | gotcha | 18:44 |
| dhellmann | it appears to be failing the second time, too | 18:44 |
| clarkb | so it is intentionally ordered that way | 18:44 |
| fungi | .gitconfig: http://paste.openstack.org/show/624012 | 18:44 |
| *** markvoelker has joined #openstack-infra | 18:44 | |
| AJaeger | regarding proposal jobs, change the project-config proposal job fails with "RSA host key for IP address '104.130.246.91' not in list of known hosts", see http://logs.openstack.org/periodic/git.openstack.org/openstack-infra/project-config/master/propose-project-config-update/8e03407/ara/ | 18:44 |
| dhellmann | looks like we're missing the username? | 18:45 |
| AJaeger | dhellmann might run into that as well ^ | 18:45 |
| dmsimard | Can we think of a less awkward version than 'openstack-python-jobs-neutron', 'openstack-python-jobs-horizon' and so on ? :/ | 18:45 |
| fungi | dmsimard: those are openstack-python-jobs project-templates which preinstall neutron or horizon for use by their respective plugin/extension projects, so nothing better comes to mind | 18:46 |
| clarkb | dhellmann: fungi the paste from fungi says username is release but failed job used proposal-bot. Possible we've crossed the streams on what key goes to what user? | 18:46 |
| dhellmann | maybe | 18:46 |
| fungi | dmsimard: maybe adding -plugin to the end would make that fact more obvious? | 18:46 |
| dhellmann | I'll stop trying to use that common setup code and just reproduce it | 18:47 |
| fungi | clarkb: dhellmann: yeah, the proposal-bot account doesn't have permission to push tags | 18:47 |
| fungi | the release account does | 18:47 |
| clarkb | http://logs.openstack.org/9e/9e810dbbd2f786836460ae79153c022d1b57358f/release-post/tag-releases/093107d/job-output.txt.gz#_2017-10-18_18_31_53_723299 seems to indicate we've written a key for the job | 18:47 |
| dhellmann | I haven't even gotten as far as pushing tags. It's failing to connect *AT ALL* | 18:47 |
| dmsimard | fungi: I don't know, it just feels.. clunky to me ? What if a project needs horizon *and* neutron ? Are we going to create openstack-python-jobs-horizon-and-neutron ? | 18:47 |
| AJaeger | dmsimard: currently we just add required-repos lines... | 18:48 |
| dhellmann | sorry. I've spent 2+ days on this now. It's getting a bit old. | 18:48 |
| openstackgerrit | Merged openstack/os-client-config master: Protect against p_opt not having prompt attribute https://review.openstack.org/506194 | 18:48 |
| AJaeger | dmsimard: yes, this required-repos needs a better approach - ideas welcome ;) | 18:48 |
| evrardjp | has zuul an API to tell which job comes from which repo? | 18:49 |
| fungi | dmsimard: if _a_ project needs that, then no. if a dozen projects need that, then maybe? | 18:49 |
| dhellmann | fungi, clarkb : is there some way to verify that the release_ssh_key in secrets.yaml matches the one for the release user? | 18:49 |
| dmsimard | I don't have any bright ideas right now, my brain is exhausted :( | 18:49 |
| dmsimard | evrardjp: a bit like print-template macro from zuul v2 ? | 18:49 |
| evrardjp | or a way to simply find all the repo matching a certain job name | 18:49 |
| clarkb | dhellmann: we'll have to decrypt it and compare to what is on the old release node. I'm not sure what the process for that is but I'm sure it can be done because the jobs essentially do the same) | 18:50 |
| evrardjp | dmsimard: I don't know | 18:50 |
| fungi | clarkb: alternatively, if the encryption process is deterministic (i believe it may be?) then we can just reencrypt and compare | 18:50 |
| dhellmann | clarkb : could we take the one on the old release node and re-encrypt it? | 18:50 |
| clarkb | fungi: oh ya | 18:50 |
| dmsimard | evrardjp: tbh even I get a bit confused as to what is where between project-config, zuul-jobs and openstack-zuul-jobs so I'd be a consumer of such tool which does not yet exist. What I've been doing so far is to use codesearch.openstack.org | 18:50 |
| fungi | i can't recall of that uses randomized padding | 18:51 |
| dhellmann | even if the encrypted results don't match, we could replace the value in secrets.yaml with the newly encrypted value | 18:51 |
| fungi | dhellmann: i'm on board with that | 18:51 |
| fungi | it's clearly not being successfully used now, so can't make matters worse | 18:51 |
| SamYaple | famous last words | 18:51 |
| dhellmann | fungi : if you send me the results, I can include them in my patch to fix the other settings. maybe we can cut down the number of retries on this by 1. | 18:51 |
| fungi | dhellmann: yup, just rereading the howto for that now | 18:52 |
| *** ccamacho has quit IRC | 18:52 | |
| dhellmann | fungi : ty | 18:52 |
| fungi | ftr: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/encryption.html#encryption | 18:54 |
| *** jamesmcarthur has quit IRC | 18:54 | |
| dhellmann | that's a lot of italics | 18:54 |
| *** jamesmcarthur has joined #openstack-infra | 18:55 | |
| SamYaple | its pretty straightforward. i didnt have an issue with it other than some ssl bits | 18:56 |
| *** efoley has joined #openstack-infra | 18:57 | |
| *** efoley has quit IRC | 18:57 | |
| *** efoley has joined #openstack-infra | 18:58 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-grenade-dsvm-ceilometer https://review.openstack.org/513162 | 18:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Add ansible-role-k8s-cookiecutter https://review.openstack.org/512323 | 19:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix Octavia jobs to be voting again https://review.openstack.org/512444 | 19:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix octavia-dashboard py27/35 jobs https://review.openstack.org/512439 | 19:00 |
| openstackgerrit | Dan Prince proposed openstack-infra/openstack-zuul-jobs master: Make tripleo undercloud-containers voting https://review.openstack.org/513163 | 19:00 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Create a playbook for publishing contributor-guide https://review.openstack.org/512865 | 19:00 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Set contributor-guide location to /contributors https://review.openstack.org/512871 | 19:00 |
| jeblair | markmcclain: did anyone answer your question? | 19:01 |
| fungi | tools/encrypt_secret.py https://zuulv3.openstack.org/openstack gerrit openstack-infra/project-config --infile ~/.ssh/id_rsa | 19:01 |
| fungi | urllib2.HTTPError: HTTP Error 404: Not Found | 19:01 |
| fungi | what am i missing there? | 19:01 |
| fungi | i tried it as git.openstack.org/openstack-infra/project-config as well | 19:02 |
| clarkb | fungi: might need to be a full url? | 19:03 |
| fungi | clarkb: what might, the project field? | 19:03 |
| clarkb | fungi: ya | 19:04 |
| fungi | nope | 19:04 |
| clarkb | https://git.openstack.org/openstack-infra/project-config ? | 19:04 |
| fungi | i still get the same error | 19:04 |
| fungi | i'll plug in some print() calls to see what url it's turning all this into | 19:04 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 19:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Don't run networking-arista changes on stable/newton https://review.openstack.org/513165 | 19:05 |
| jeblair | markmcclain: ^ | 19:05 |
| jeblair | that commit message is gibberish | 19:06 |
| markmcclain | jeblair: awesome thanks! | 19:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Don't run py35 jobs on newton networking-arista changes https://review.openstack.org/513165 | 19:06 |
| openstackgerrit | Dan Prince proposed openstack-infra/openstack-zuul-jobs master: Make tripleo undercloud-containers voting https://review.openstack.org/513163 | 19:06 |
| *** ociuhandu has joined #openstack-infra | 19:06 | |
| fungi | looks like my first invocation should have been correct since the key i want is https://zuulv3.openstack.org/keys/gerrit/openstack-infra/project-config.pub | 19:07 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update jobs for features/zuulv3 branch https://review.openstack.org/512642 | 19:07 |
| fungi | oh | 19:07 |
| fungi | where is the tenant? | 19:07 |
| clarkb | fungi: in the url, oh but not in the one you just pasted | 19:07 |
| jeblair | fungi: no tenant yet | 19:07 |
| *** rbrndt has quit IRC | 19:07 | |
| fungi | hah | 19:07 |
| fungi | so i don't want /openstack | 19:08 |
| jeblair | fungi: this gets fixed in https://review.openstack.org/504807 | 19:08 |
| fungi | thanks! i'll review that when i finish this | 19:08 |
| *** dprince has quit IRC | 19:08 | |
| *** slaweq_ has joined #openstack-infra | 19:08 | |
| fungi | yup, works great if i leave off the tenant | 19:08 |
| jeblair | fungi: no rush -- it's pep8 broken and behind some other web reorgs; we won't get to it for a few weeks | 19:08 |
| dmsimard | AJaeger: should we not encourage projects to define their jobs in-tree ? i.e, https://review.openstack.org/#/c/512536/ | 19:09 |
| SamYaple | fungi: python encrypt_secret.py --infile t --outfile p http://zuulv3.openstack.org gerrit openstack/loci | 19:09 |
| SamYaple | fungi: thats what i use | 19:09 |
| fungi | SamYaple: yep, confirmed, the docs say to have the tenant on the end of the zuul url, but apparently we don't support that yet | 19:10 |
| SamYaple | got it | 19:10 |
| jeblair | fungi: i think it's because we rewrite that on the openstack serevr | 19:10 |
| * fungi sometimes follows directions *too* well | 19:10 | |
| fungi | yeah, makes sense | 19:10 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 19:10 |
| openstackgerrit | Merged openstack/os-client-config master: Treat clouds.yaml with one cloud like envvars https://review.openstack.org/505388 | 19:11 |
| AJaeger | dmsimard: https://docs.openstack.org/infra/manual/zuulv3.html#what-not-to-convert - we want the standard jobs in project-config, only the specific ones in-repo | 19:11 |
| AJaeger | dmsimard: so, all those should be - by policy - in project-config | 19:11 |
| clarkb | fungi: did it produce a different value than what is already in the secret? | 19:11 |
| dmsimard | AJaeger: ack | 19:11 |
| fungi | clarkb: not sure, it produced http://paste.openstack.org/show/624017 | 19:11 |
| jeblair | clarkb, fungi: you should get a different value every time | 19:11 |
| fungi | yeah, random salt/padding, which i guess i should have remembered | 19:12 |
| jeblair | fungi: before i resume my pypi tasks -- this isn't for pypi is it? | 19:12 |
| fungi | jeblair: no, this is the ssh privkey for the release bot account in gerrit | 19:13 |
| *** rbrndt has joined #openstack-infra | 19:13 | |
| fungi | for pushing tags from automation | 19:13 |
| jeblair | ok. i'll go back to re-enqueing my pypi upload job. | 19:13 |
| fungi | yeah, you pushed your own tag, so didn't run into this | 19:13 |
| jeblair | i'm solving the problems 2 steps ahead. it's zuul-style parallelism. | 19:14 |
| dmsimard | AJaeger: I'll re-iterate that my understanding of that (we were discussing that earlier this week?) is that the *jobs* definition must remain in project-config, but that the layout can be entirely in-tree. | 19:14 |
| dmsimard | Can infra-root deny or confirm ^ ? https://docs.openstack.org/infra/manual/zuulv3.html#what-not-to-convert | 19:14 |
| fungi | jeblair: yeah, you're jumping the queue | 19:14 |
| AJaeger | dmsimard: https://review.openstack.org/#/c/511948/ needs rebase | 19:14 |
| dmsimard | AJaeger: thanks | 19:14 |
| AJaeger | dmsimard: last night discussion? yes, we need to followup and improve docs... | 19:15 |
| fungi | dmsimard: except the project list and the system-required project-template need to happen in project-config (along with any project-templates or individual jobs mandated by the consistent testing interface in governance, but only for official deliverables bound by the cti) | 19:15 |
| AJaeger | dmsimard: my udnerstanding from 5th October was that we want the layout in project-config as well | 19:15 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for emit-job-header and set-zuul-log-path-fact https://review.openstack.org/511948 | 19:15 |
| AJaeger | dmsimard: fungi said it much better than I did ;) | 19:16 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: replace the common git review configuration for release jobs https://review.openstack.org/513166 | 19:16 |
| *** jcoufal has quit IRC | 19:16 | |
| dmsimard | fungi: ok, so every project (under governance? those that aren't are exempt?) must have a "project" definition in project-config/zuul.d with the "system" jobs ? | 19:17 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: replace the common git review configuration for release jobs https://review.openstack.org/513166 | 19:17 |
| AJaeger | dmsimard: *every* project must have one definition with system-required | 19:17 |
| AJaeger | dmsimard: every project in *governance* must have the PTI jobs in project-config | 19:18 |
| *** weshay|ruck|brb is now known as weshay|ruck | 19:18 | |
| AJaeger | fungi, care to review the three open infra-manual changes, please? | 19:19 |
| dmsimard | If it's something that is applied to *every* project, it should be an implicit job that is there for every project on every branch, not an explicit one, IMHO. Otherwise you end up with a lot of code to maintain. | 19:19 |
| dmsimard | The ability to declare a "system job" or a "global job" that applies to every project likely doesn't exist today but maybe it should, somehow | 19:19 |
| *** huanxie has joined #openstack-infra | 19:19 | |
| fungi | dmsimard: the system-required project-template is where we're putting things we want run on every project in the system, for example merge-check. i agree having those centrally configurable as implicitly inherited would be nice... what existing feature of zuul configuration did you have in mind leveraging there? | 19:19 |
| fungi | otherwise, let's live with this for now, and look at ways to improve zuul to support that use case after we get this working | 19:20 |
| dmsimard | fungi: I don't think the feature we're talking about exists yet. If it exists one day, I'd put it at the tenant layer of configuration maybe ? ex: https://github.com/openstack-infra/project-config/blob/master/zuul/main.yaml | 19:21 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Create a playbook for publishing contributor-guide https://review.openstack.org/512865 | 19:22 |
| dmsimard | fungi: sure, we have to live with this for now and that's fine, we should slightly improve the wording on the docs. There is https://review.openstack.org/#/c/512607/ that addresses part of that | 19:22 |
| jeblair | as you think about it, i'd ask that you consider whether what you find objectionable is the system-required template, or the others. | 19:22 |
| fungi | the main objection i've heard is that right now infra remains a bottleneck reviewing project additions to the layout, but ultimately we're still having to review project additions to other things in project-config so it doesn't seem like a huge deal to me | 19:24 |
| dmsimard | jeblair: I have nothing against the system-required template. What bothers me is to maintain a list of >1000 projects with the 'system-required' template defined in project-config (for projects that have nothing else in project-config) | 19:24 |
| AJaeger | team, our proposal jobs fails when doing an ssh with " "RSA host key for IP address '104.130.246.91' not in list of known hosts." | 19:25 |
| fungi | dmsimard: ooh, except that do have plenty more in project-config, just not for _zuul_ | 19:25 |
| dmsimard | jeblair: if system-required is /really/ required by the system, it should be implicitly added to every project | 19:25 |
| jeblair | i hate that name | 19:25 |
| AJaeger | how shall we avoid that? | 19:25 |
| jeblair | it's not system-required | 19:25 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: Dont install python-virtualenv for py3k in deb https://review.openstack.org/513168 | 19:25 |
| jeblair | it's policy-required | 19:25 |
| dmsimard | bikeshedding aside | 19:25 |
| fungi | i'm up for a rename | 19:25 |
| dmsimard | we want a set of jobs to apply to every project | 19:25 |
| AJaeger | jeblair, fungi : Sent a patch ;) | 19:26 |
| jlk | AJaeger: sounds like a secret use is not going right, and it isn't interacting with the right fileserver | 19:26 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: Dont install python-virtualenv for py3k in deb https://review.openstack.org/513168 | 19:26 |
| jeblair | anyway, let's discuss this later. | 19:26 |
| AJaeger | jlk: do you wnat to dig into it? http://logs.openstack.org/periodic/git.openstack.org/openstack-infra/project-config/master/propose-project-config-update/8e03407/ | 19:26 |
| AJaeger | jeblair: since it's all in one repo, we could rename ;) | 19:26 |
| fungi | dmsimard: remember that these same projects already have to add themselves to project-config in gerrit/projects.yaml and usually gerritbot/channels.yaml and potentially other places too | 19:27 |
| *** iyamahat has quit IRC | 19:28 | |
| jeblair | i think zuul is currently in a slow reconfig loop -- where we've merged a bunch of config changes which force a tenant reconfig, which in turn is causing all the dynamic changes to regenerate their configs | 19:28 |
| jeblair | it'll get through it, but things are unusually slow right now | 19:28 |
| clarkb | fungi: and the main.yaml for zuul | 19:28 |
| dmsimard | jeblair: I've indeed been reviewing a huge backlog of reviews in project-config | 19:28 |
| dmsimard | I'll stop for now | 19:28 |
| jlk | So, ssh -p 29418 proposal-bot@review.openstack.org gerrit query --current-patch-set status:open project:openstack-infra/project-config topic:project-yaml-normalization owner:proposal-bot is the command that failed | 19:29 |
| jlk | this is from the script, the task "Propose constraints update" | 19:29 |
| jeblair | i think we need to break out some more stats here, so we know the management queue length too (that's where tenant reconfigure events go; also, we may need to collapse dupes in that if we aren't already) | 19:29 |
| jlk | presumably, the host "review.openstack.org" was the host added at the task "Add host key information to known hosts" | 19:29 |
| johnsom | Is zuul not accepting patches? I committed https://review.openstack.org/#/c/512870/ but it's not on the dashboard. | 19:29 |
| clarkb | jlk: above we've discovered that we may have crossed the streams on user:key mappings | 19:30 |
| dmsimard | johnsom: zuul is a bit slow right now, it'll show up. | 19:30 |
| jlk | oh? | 19:30 |
| clarkb | jlk: the proposal-bot and release user in particular | 19:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Clarify patchsets for new project configuration https://review.openstack.org/512607 | 19:30 |
| jeblair | johnsom: see the queue lengths at the top of the status page; when they hit 0, zuul is caught up | 19:30 |
| jlk | oh right, so it's not the host key that failed | 19:30 |
| jlk | it's permission denied on the user's public ssh key | 19:31 |
| *** salv-orlando has quit IRC | 19:31 | |
| johnsom | jeblair Ah, ok, yeah, probably at the back-o-the-line. Thanks, now I know what those are... | 19:31 |
| clarkb | jlk: so we need to double check that we have appropriate secrets for both users with the correct key in each case | 19:31 |
| clarkb | jlk: fungi and dhellmann are working on that for the release user | 19:31 |
| AJaeger | clarkb, fungi : ah - need to cover this for all the other cases as well. Thanks, jlk! | 19:32 |
| jlk | alright. I don't think I have access to those details, so can they look at propose-bot too? :D | 19:32 |
| clarkb | jlk: AJaeger can you point me at the secret being used by the job? | 19:33 |
| clarkb | is it proposal_ssh_key ? | 19:33 |
| fungi | jlk: clarkb: well, the username was definitely wrong in the tag-releases job, right? we just regenerated the encrypted ssh key for that account too while we were at it in case it was also wrong | 19:33 |
| fungi | i should say, reencrypted the private ssh key | 19:33 |
| fungi | to make sure we're using the one for the right account (release instead of proposal-bot) | 19:34 |
| clarkb | ya | 19:34 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Don't cleanup TripleO CI Tempest resources https://review.openstack.org/513169 | 19:34 |
| AJaeger | clarkb: log is http://logs.openstack.org/periodic/git.openstack.org/openstack-infra/project-config/master/propose-project-config-update/8e03407/ | 19:35 |
| *** salv-orlando has joined #openstack-infra | 19:35 | |
| jlk | looking | 19:36 |
| *** yolanda has quit IRC | 19:36 | |
| dmsimard | dhellmann: FYI I had +W https://review.openstack.org/#/c/512788/, I took it out as per your comment in https://review.openstack.org/#/c/512788/. Please let us know when you are ready. | 19:36 |
| dhellmann | dmsimard : thanks | 19:37 |
| * AJaeger calls it a day - bye for now | 19:37 | |
| dmsimard | dhellmann: wrong paste but I think you understood | 19:37 |
| dhellmann | dmsimard : yeah | 19:37 |
| *** rwsu has joined #openstack-infra | 19:37 | |
| jlk | secret is "proposal_ssh_key" | 19:37 |
| jlk | clarkb: | 19:37 |
| clarkb | jlk: thanks | 19:37 |
| jlk | the job is 'propose-update-constraints' and others, but the parent is just 'propose-updates' which is where the secret is attached | 19:37 |
| clarkb | looking at the secrets the host key stuff is all wrong too | 19:37 |
| clarkb | review.openstack.org is at 104.130.246.91 not 23.253.232.87 | 19:38 |
| clarkb | but I don't think that is fatal because the name is listed too | 19:39 |
| clarkb | (so we trust dns in this case) | 19:39 |
| clarkb | I'll work on a patch to fix both the host key things and to reencrypt the right ssh key | 19:39 |
| fungi | ahh, yeah, ssh will auto-add new ip addresses if the host key matches the dns name you used | 19:39 |
| *** panda|rover is now known as panda|rover|off | 19:43 | |
| clarkb | fungi: dhellmann I'm fixing the release ssh key known hosts too, have a change I can rbase mine onto to avoid merge conflicts? | 19:45 |
| dhellmann | sure, hang on | 19:45 |
| *** camunoz has quit IRC | 19:45 | |
| dhellmann | https://review.openstack.org/#/c/513166/ | 19:46 |
| *** camunoz has joined #openstack-infra | 19:46 | |
| clarkb | thanks | 19:46 |
| * clarkb needs to find the public key in gerrit for the proposal-bot user to make sure this is the correct private key | 19:46 | |
| *** rbrndt has quit IRC | 19:47 | |
| inc0 | hey, quick question, to get rid of tarball hack, do you mind if I add basic publishing pipeline to dockerhub/quay.io? | 19:47 |
| inc0 | straight from periodic build job | 19:47 |
| *** iyamahat has joined #openstack-infra | 19:47 | |
| inc0 | and nightly/weekly | 19:48 |
| inc0 | then we can move to dockerhub images with gates and use caching proxy to lower bandwidth | 19:48 |
| fungi | clarkb: if it helps, i did the release one by ssh'ing into the old persistent slave and just using ~jenkins/.ssh/id_rsa as the input file | 19:49 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Set contributor-guide location to /contributors https://review.openstack.org/512871 | 19:49 |
| clarkb | fungi: ya I'm on proposal.slave.openstack.org doing that, juts wanted to double check. Appears to be the key | 19:49 |
| fungi | ahh, yeah not a terrible idea | 19:49 |
| clarkb | fungi: where did you run the zuul command? on zuul itslef? I guess anyhwere you have a zuul installed? | 19:50 |
| fungi | the one on signing01.ci is bound to be the correct key for release | 19:50 |
| fungi | clarkb: i cloned zuul on the old slave | 19:50 |
| clarkb | aha /me does this | 19:50 |
| *** huanxie has quit IRC | 19:50 | |
| fungi | then checked out the zuulv3 feature branch and ran it there | 19:50 |
| fungi | less copying of plaintext secrets over the network thataway | 19:50 |
| fungi | always encrypt closest to the source when you can | 19:51 |
| clarkb | ya this is nice and easy | 19:51 |
| clarkb | fungi: tools/encrypt_secret.py https://zuulv3.openstack.org/ gerrit openstack-infra/project-config --infile ~/.ssh/id_rsa | 19:51 |
| clarkb | is that what I want to run? | 19:51 |
| fungi | yup, or you can omit the trailing / on the url (i did) | 19:52 |
| fungi | mainly because i saw the script was concatenating and would have made a // | 19:52 |
| clarkb | thanks | 19:52 |
| fungi | which apache will still reduce to / but whatever | 19:52 |
| *** slaweq_ has quit IRC | 19:57 | |
| *** pcrews has quit IRC | 19:57 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Redo the gerrit proposal bot secret https://review.openstack.org/513176 | 19:57 |
| clarkb | AJaeger: jlk fungi dhellmann ^ | 19:57 |
| clarkb | inc0: yes I think that would be fine | 19:58 |
| inc0 | can I schedule job for particular time? cron style | 19:59 |
| inc0 | and which time would be best then? start with weekly over weekend? | 19:59 |
| *** armax has quit IRC | 19:59 | |
| clarkb | inc0: its a pipeline config iirc so you can use the preexisting daily pipeline or propose a new pipeline to run less frequently | 19:59 |
| clarkb | inc0: also we don't cache quay.io | 19:59 |
| inc0 | dockerhub works too | 20:00 |
| inc0 | we don't have daily pipeline currently | 20:00 |
| odyssey4me | Has there been any though given to making use of containers in some shape or form in order to do tests like lint tests, rather than use single-use-slaves? I realise that nodepool makes single-use-slaves cheap in terms of their use, but I figure that using containers on such slaves could introduce quite a bit more speed in processing some jobs. | 20:01 |
| openstackgerrit | Merged openstack-infra/project-config master: sahara IRC notifications: add a repo and future branches https://review.openstack.org/503936 | 20:01 |
| dmsimard | odyssey4me: there is a container nodepool driver in the works | 20:01 |
| *** salv-orlando has quit IRC | 20:02 | |
| clarkb | odyssey4me: re speed I don't think it makes much of a difference, but ya there is work in progress to add different drivers to nodepool and be able to schedule to different flavors more effectively | 20:02 |
| *** salv-orlando has joined #openstack-infra | 20:02 | |
| clarkb | odyssey4me: the containers would be running on the same VMs so performnce should be roughly the same | 20:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove fetch-zuul-cloner from base-test job https://review.openstack.org/513079 | 20:02 |
| *** rbrndt has joined #openstack-infra | 20:03 | |
| pabelanger | fungi: ^yes! I've updated the patch | 20:03 |
| dmsimard | odyssey4me: you have to keep in mind that nodepool pre-provisions nodes so that they are already available ahead of time (min-ready config) | 20:03 |
| dhellmann | clarkb : I guess the known hosts value for the release key is wrong, too | 20:04 |
| dmsimard | odyssey4me: https://review.openstack.org/#/c/468753/ | 20:04 |
| clarkb | dhellmann: it had the wrong IPs in it which mostly worked because the name is in there too, but I fixed it in both places | 20:05 |
| dhellmann | oh, the other change was below the scroll window so I didn't notice that | 20:05 |
| dhellmann | thanks | 20:05 |
| *** yamahata has joined #openstack-infra | 20:06 | |
| odyssey4me | dmsimard clarkb ja, makes sense - I'm thinking more about a situation like we're in where using nodepool would be useful, but resources are more scarce so we might want to use containers to kinda slice up each node more to get more out of it | 20:06 |
| *** pcrews has joined #openstack-infra | 20:06 | |
| odyssey4me | right now we're just using smaller flavors, which I suppose also makes sense in a non openstack-ci world | 20:06 |
| clarkb | odyssey4me: tristanC is working on the container driver(s) | 20:07 |
| *** gyee has quit IRC | 20:07 | |
| *** salv-orlando has quit IRC | 20:07 | |
| clarkb | right now though we are mostly focused on making the zuulv3 transition go smoothly, but expect that work to pick up steam as things settle in | 20:07 |
| dmsimard | odyssey4me: that boils down more to flavor usage than containers -- for example, you likely don't need a full blown 8vcpu/8gb ram VM for running tox -e pep8 | 20:07 |
| odyssey4me | thanks for the review link *adds to his starred list* | 20:08 |
| dmsimard | odyssey4me: tobiash was working on some great patches (not sure if they landed?) to be able to maximize quota usage regardless of flavor usage, this would potentially allow us to do things like give smaller VMs to jobs that don't need high amounts of resources. | 20:08 |
| pabelanger | odyssey4me: dmsimard: clarkb: public IPs might be the next bottleneck for container things | 20:08 |
| clarkb | pabelanger: ya thats a more general flavor problem too | 20:08 |
| clarkb | pabelanger: most of our clouds have a hard instance cap, we can't boot anymore 2GB instances than 8GB instances | 20:09 |
| clarkb | IPs are a similar problem and so on | 20:09 |
| pabelanger | yah | 20:09 |
| odyssey4me | pabelanger fair enough, although I'd be thinking that container hosts are more like medium-lived slaves... they live for a day or so, and act as executors | 20:09 |
| dmsimard | odyssey4me: https://review.openstack.org/#/q/topic:quota-second-try | 20:09 |
| odyssey4me | ie each container doesn't get an ip - only the instance | 20:09 |
| clarkb | odyssey4me: ya thats a possibility though further limits the utility of containers as general job runners | 20:10 |
| clarkb | (and maybe that is ok) | 20:10 |
| odyssey4me | the instance then accepts jobs of a known type which can execute a dockerfile or something | 20:10 |
| *** jkilpatr has quit IRC | 20:10 | |
| pabelanger | odyssey4me: or external ks8, but both require more ops to keep them alive over dynamic nodes from nodepool | 20:10 |
| pabelanger | so some trade-offs | 20:10 |
| odyssey4me | oh sure, definitely not general purpose - they'd have to be used for docs builds and such | 20:10 |
| dmsimard | IMO container driver is useful if you don't happen to have an openstack cloud | 20:10 |
| dmsimard | if you have an openstack cloud, the quota patches I've mentioned will make things much easier to manage | 20:11 |
| clarkb | dmsimard: ya thats sort of how I feel about it too. Or if you don't have to do systems software testing (like testing openstack) | 20:11 |
| pabelanger | but agree with clarkb, once things stablize on rollout, there will be much discussions on containers I think | 20:11 |
| clarkb | because openstack forces you to open up the test containers to the point where you can't trust isolation so may as well just VM | 20:11 |
| smcginnis | odyssey4me: pep8 runs might be another good use. | 20:11 |
| dmsimard | jeblair: is zuul out of the slow reconfigure loop ? looks like the queue is caught up but wanted to make sure before going for another round of job/project-config reviews | 20:12 |
| clarkb | dmsimard: I think the queues being zerod is likely the best indiication of that | 20:12 |
| odyssey4me | pabelanger sure, but in my thinking these would run on nodes that run for a while, then get replaced - at least daily if not more often | 20:12 |
| odyssey4me | they would still be ephemeral - just live a little longer | 20:12 |
| dmsimard | clarkb: ok, I was wondering because RAM is starting to peak into 10GB territory | 20:12 |
| openstackgerrit | Merged openstack-infra/project-config master: remove legacy-grenade-dsvm-ceilometer https://review.openstack.org/512801 | 20:13 |
| odyssey4me | anyway, it was just a thought running through my mind while watching zuul chug along | 20:13 |
| odyssey4me | and yeah, it's crawling back up to ~13GB now | 20:14 |
| mriedem | anyone here know why new gerrit moves the cursor around when you're reviewing something? like i've expanded a comment somewhere toward the top, and when i want to comment below, it moves the cursor back to the top | 20:14 |
| mriedem | drives me crazy | 20:14 |
| clarkb | mriedem: the cursor starts on line 1 | 20:14 |
| smcginnis | mriedem: Try going into settings and set rendering to Slow. | 20:14 |
| clarkb | and expanding a comment doen't move the cursor | 20:14 |
| *** ldnunes has quit IRC | 20:14 | |
| *** esberglu has quit IRC | 20:14 | |
| * odyssey4me misses the 'e' hotkey for 'edit' in the new gerrit. | 20:14 | |
| clarkb | odyssey4me: its still there but its ^e or something | 20:14 |
| pabelanger | odyssey4me: yah, that's not been the first time somebody has suggested it. Until now, it was use nova-docker, with zuulv2, but who know in the future :) | 20:14 |
| mriedem | clarkb: even if i move the cursor down | 20:14 |
| * smcginnis missed the 'f' hotkey | 20:14 | |
| mriedem | looking for this rendering setting | 20:15 |
| *** esberglu has joined #openstack-infra | 20:15 | |
| mriedem | ah there it is | 20:15 |
| odyssey4me | pabelanger heh with the nova-lxc driver you could do a bunch of things you normally do with instances, but you need a cloud that has it | 20:15 |
| pabelanger | odyssey4me: right | 20:15 |
| odyssey4me | nova-lxd I mean | 20:16 |
| clarkb | odyssey4me: ctrl + alt + e says the help layover | 20:16 |
| odyssey4me | but nodepool really is awesome if you think of just how general purpose it is | 20:16 |
| odyssey4me | clarkb ORLY? I wonder how that translates for a mac | 20:16 |
| pabelanger | odyssey4me: containers from nodepool, is a brave new world, but we have some design issue to first solve. | 20:17 |
| clarkb | odyssey4me: they might give you a different help screen on a mac? but unsure | 20:17 |
| clarkb | odyssey4me: hit ? on the diff screen | 20:17 |
| pabelanger | but I'm excited to see what that looks like, because it means more projects / ppl using zuul :D | 20:17 |
| clarkb | mriedem: I think you might have to show me what you mean, but ya the cursor itslef seems to want ot start at the top and expanding comments and stuff doens't move it | 20:17 |
| *** openstackgerrit has quit IRC | 20:17 | |
| clarkb | making a new comment does appear to move it | 20:17 |
| clarkb | but not a reply | 20:18 |
| *** hasharAway has quit IRC | 20:18 | |
| mriedem | rendering=slow seems to help | 20:18 |
| *** salv-orlando has joined #openstack-infra | 20:18 | |
| mriedem | i was about to just give up on software as a profession | 20:18 |
| mriedem | well, open source software at least, where code reviews are required | 20:18 |
| *** eharney has quit IRC | 20:19 | |
| *** esberglu has quit IRC | 20:19 | |
| mriedem | i could enterprise software all day where there are no tests or code reviews | 20:19 |
| mriedem | hence, enterprise | 20:19 |
| *** openstackgerrit has joined #openstack-infra | 20:19 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix branch ordering when loading in-repo config https://review.openstack.org/513094 | 20:19 |
| fungi | sorta like the starship | 20:20 |
| clarkb | mriedem: I'm not sure you'd have software either in that case >_> | 20:20 |
| clarkb | does it count as software if it doesn't compile/run? | 20:20 |
| clarkb | or just bytes on a disk | 20:20 |
| fungi | very soft ware | 20:20 |
| mriedem | umm, guys, | 20:20 |
| mriedem | compiling is the test | 20:20 |
| mriedem | if it compiles, it ships | 20:20 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: make a gentoo nodepool image https://review.openstack.org/504530 | 20:20 |
| *** Sukhdev has quit IRC | 20:21 | |
| inc0 | clarkb: do you have any example of periodic job in specific time? | 20:22 |
| clarkb | inc0: no thats what I'm saying the jobs don't run at a specific time. Instead the pipeline they run in enqueues them at a time | 20:22 |
| clarkb | I don't think zuulv3 changed this | 20:22 |
| inc0 | hmm ok, well only reason for it to be in specific time is to optimize bandwidth consumption | 20:23 |
| pabelanger | priority is also low for periodic | 20:23 |
| inc0 | ok that's fine then | 20:23 |
| inc0 | and periodic is once a day by default or can I specify freq? | 20:24 |
| pabelanger | every 24 hours | 20:24 |
| fungi | inc0: basically, if you want to run a job at a different time than is set in the existing periodic pipelines, you'd need a new pipeline with a periodic trigger for a different time/frequency | 20:24 |
| *** smatzek has quit IRC | 20:24 | |
| inc0 | fungi: I don't care so much about time as about not consuming resources other people might need | 20:25 |
| *** smatzek has joined #openstack-infra | 20:25 | |
| inc0 | but low priority is fine, also 24hrs is ok I guess | 20:25 |
| fungi | inc0: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml#n205 | 20:25 |
| inc0 | anyway, I'll start experimenting with that so giving you heads up | 20:25 |
| dmsimard | infra-root: ram on zuulv3 is getting dangerously high | 20:25 |
| fungi | inc0: so that generic "periodic" pipeline we have runs once daily at 06:00 utc | 20:26 |
| *** jkilpatr has joined #openstack-infra | 20:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: switch whereto to publish docs on docs.o.o https://review.openstack.org/507243 | 20:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron to required projects for net-cisco custom jobs https://review.openstack.org/512593 | 20:26 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Revert "Use weakref for change cache" https://review.openstack.org/513097 | 20:26 |
| *** smatzek_ has joined #openstack-infra | 20:27 | |
| inc0 | that's fine, I think 6utc is pretty close to lower traffic anyway | 20:27 |
| fungi | inc0: which is... why we have it set to that | 20:27 |
| jeblair | odyssey4me: probably should hold off on any more rechecks for now | 20:27 |
| odyssey4me | jeblair any rechecks, or just job changing rechecks? | 20:28 |
| jeblair | odyssey4me: the latter | 20:28 |
| inc0 | yup, you're way ahead of me;) another question - openstack irc bot announcing health of periodic jobs - is it possible? | 20:28 |
| odyssey4me | ok, will do - thanks for monitoring | 20:28 |
| *** smatzek has quit IRC | 20:29 | |
| dmsimard | inc0: there is a new zuul dashboard coming soon in order to provide historical results (a bit like jenkins, if you will) | 20:30 |
| *** smatzek_ has quit IRC | 20:31 | |
| jeblair | but completely different | 20:31 |
| fungi | inc0: also the openstack health dashboard has an rss feed you could subscribe to for specific jobs | 20:31 |
| inc0 | ok, we'll need to figure out effective way for community to monitor health of periodic jobs, but I guess that's for later | 20:32 |
| inc0 | gonna be nice vacation-style hacking;) | 20:32 |
| dmsimard | inc0: I know that evrardjp told me about how they monitor their periodic jobs in a fairly clever way by exporting ara results as subunit and getting them picked up by openstack-health | 20:33 |
| fungi | or at least i thought the health dashboard had rss feeds for jobs, but maybe it's just per project | 20:33 |
| inc0 | ok, I'll ping him later on | 20:33 |
| dmsimard | We're not quite ready to do this at any kind of large scale, however, we need to re-think some fundamental things in openstack-health, like the database schema :) | 20:34 |
| clarkb | fungi: its both | 20:34 |
| fungi | http://health.openstack.org/runs/key/project/openstack/kolla/recent/rss | 20:34 |
| odyssey4me | inc0 yeah, I can point you at the bits if you like? | 20:34 |
| *** esberglu has joined #openstack-infra | 20:34 | |
| SamYaple | fungi: nice! thanks | 20:34 |
| fungi | clarkb: ahh, i just don't get the fancy rss icon on jobs i guess | 20:34 |
| jeblair | have we agreed on a project-config-core tag? | 20:34 |
| inc0 | odyssey4me: thanks, I'll forget it anyway, still need to write periodic jobs | 20:34 |
| jeblair | for irc highlighting? | 20:34 |
| inc0 | but I'll ping you guys when I'm done if that's ok | 20:34 |
| inc0 | thank you! | 20:34 |
| fungi | jeblair: that one looks as good as any | 20:34 |
| odyssey4me | inc0 https://github.com/openstack/openstack-ansible/blob/master/scripts/scripts-library.sh#L158-L160 | 20:35 |
| weshay|ruck | pabelanger, are there some gate jobs not reporting success/fail after they trigger atm? | 20:35 |
| odyssey4me | oh, sure thing | 20:35 |
| *** armax has joined #openstack-infra | 20:35 | |
| dmsimard | jeblair: I have a ping set up for infra-root, project-config-core is fine too for cases where infra-root is not necessarily involved | 20:35 |
| weshay|ruck | dmsimard, have you heard anything along those lines? | 20:35 |
| jeblair | we're merging a number of changes to the zuul v2 config in project-config, which is causing puppet to tell zuul to perform full reconfigurations | 20:35 |
| inc0 | thakns odyssey4me | 20:35 |
| jeblair | those take 3+ minutes | 20:35 |
| weshay|ruck | for example https://review.openstack.org/#/c/509605/ | 20:36 |
| weshay|ruck | I see zuul started gate jobs, but nothing was reported | 20:36 |
| smcginnis | I believe we will need this before we can do any releases: https://review.openstack.org/#/c/513176 | 20:36 |
| dmsimard | weshay|ruck: that's from yesterday, I believe there was a full zuul restart yesterday | 20:36 |
| jeblair | we could either stop merging changes to the v2 config, or change the puppet configuration to be more specific (but that's only necessary for the next ~3 days) | 20:36 |
| weshay|ruck | so a recheck is in order? | 20:36 |
| dmsimard | weshay|ruck: there was a notice sent out to channels and twitter https://twitter.com/openstackinfra/status/920447826001317888 | 20:37 |
| dmsimard | weshay|ruck: yeah, recheck | 20:37 |
| weshay|ruck | heh.. will follow that from now on :) | 20:37 |
| weshay|ruck | thanks | 20:37 |
| dmsimard | weshay|ruck: you also have https://wiki.openstack.org/wiki/Infrastructure_Status | 20:38 |
| dmsimard | weshay|ruck: which is... not on twitter :) | 20:38 |
| weshay|ruck | thanks as always | 20:39 |
| fungi | and also openstackstatus echoes those messages into all irc channels it is set to join | 20:39 |
| jeblair | fungi, dmsimard: thoughts about freezing zuul v2 layout? | 20:39 |
| *** esberglu has quit IRC | 20:39 | |
| fungi | jeblair: i'm fine with freezing the v2 layout, this is understandably painful | 20:39 |
| dmsimard | jeblair: if we are past the point of no return, +1 | 20:40 |
| jeblair | i should send that email | 20:40 |
| *** esberglu has joined #openstack-infra | 20:40 | |
| *** esberglu has quit IRC | 20:41 | |
| *** esberglu has joined #openstack-infra | 20:41 | |
| *** jklare has quit IRC | 20:42 | |
| dmsimard | fungi, jeblair: so we have a job "project-config-frozen-files", http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n632 .. We can make it voting if we want. | 20:43 |
| *** gouthamr has quit IRC | 20:43 | |
| *** jklare has joined #openstack-infra | 20:43 | |
| jeblair | dmsimard: i'm inclined to like that idea. it will mean that any further changes to jobs that use the jenkins scripts will have to start with copying those scripts somewhere. | 20:45 |
| jeblair | or we could drop it from checking that directory. it's mostly the layout.yaml changes that are the operational problem. | 20:45 |
| inc0 | hey, soo...dockerhub publisher account | 20:45 |
| *** camunoz has quit IRC | 20:45 | |
| inc0 | I don't want to use my personal account or mail address for rather obvious resons. Kolla as project doesn't have mailbox;) | 20:46 |
| inc0 | how do we want to do it? | 20:46 |
| inc0 | SamYaple: did you create dedicated loci account? | 20:46 |
| *** notmyname has quit IRC | 20:48 | |
| SamYaple | inc0: we publish under openstackloci right now | 20:48 |
| inc0 | namespace, but which account? | 20:48 |
| inc0 | aka docker id | 20:48 |
| SamYaple | openstackloci is a group not a user | 20:48 |
| inc0 | so which user are you using in publisher jobs | 20:48 |
| SamYaple | we add whoever we want to the group for managers | 20:48 |
| jeblair | fungi, dmsimard, clarkb: https://etherpad.openstack.org/p/4RnXwkwbNI look good? | 20:49 |
| SamYaple | ah i created a one off account for that | 20:49 |
| inc0 | ok, so nothing official and used temp email thingy? | 20:49 |
| SamYaple | i used an email that goes back to an account i own, but i dont know the password to the account (autogenerated) | 20:50 |
| SamYaple | i *could* reset it if i needed, but i dont plan on ever touching it again | 20:50 |
| SamYaple | worst case we can just create another account and regen the secret | 20:50 |
| inc0 | yeah I guess I can do that too | 20:50 |
| dmsimard | jeblair: (on phone right now) I'd make it a bit more explicit what files/directories are involved | 20:51 |
| fungi | jeblair: lgtm, thanks for drafting! | 20:51 |
| *** sbezverk has quit IRC | 20:54 | |
| jeblair | dmsimard: ack, will so revise and send | 20:55 |
| *** notmyname has joined #openstack-infra | 20:55 | |
| *** andreas_s has joined #openstack-infra | 20:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add management event queue length to status json https://review.openstack.org/513182 | 21:00 |
| clarkb | jeblair: ya that looks good to me | 21:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add project-templates to docs https://review.openstack.org/513185 | 21:03 |
| clarkb | inc0: does that work? I think one of the goals from infras perspective with thes changes is that we don't have to sit in the middle of all this stuff anymore (because that doesn't scale well) | 21:03 |
| jeblair | dhellmann: it looks like the pypi uploads work now (re the twine url fix from yesterday) | 21:03 |
| dhellmann | jeblair : great, thanks! | 21:04 |
| dhellmann | now if we can get the job to run that far... | 21:04 |
| inc0 | clarkb: I just made fake account and we'll roll with it for now | 21:04 |
| *** thorst has quit IRC | 21:04 | |
| clarkb | jeblair: care to review https://review.openstack.org/#/c/513176/1 and its parent? those are related to getting release tags working | 21:04 |
| jeblair | on it | 21:04 |
| inc0 | organization is what matters and we have it set up already | 21:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Change manila tempest plugin jobs https://review.openstack.org/512583 | 21:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Restore cover test to oslo.messaging https://review.openstack.org/512813 | 21:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Add pike branch to zun channel notification https://review.openstack.org/512923 | 21:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs from python-openstackclient https://review.openstack.org/512888 | 21:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Add new project networking-spp https://review.openstack.org/512223 | 21:05 |
| boden | fungi: I’m still struggling with https://review.openstack.org/#/c/512390/ as per the latest results http://logs.openstack.org/90/512390/6/check/openstack-tox-py27/04dbe69/job-output.txt.gz#_2017-10-18_20_14_00_873082 it’s pulling in some other version of requirements now.. almost like it’s using the stable/pike (not ocata) of the required-projects | 21:05 |
| *** andreas_s has quit IRC | 21:06 | |
| *** andreas_s has joined #openstack-infra | 21:06 | |
| clarkb | jeblair: could ^ be related to that bug tripleo ran into? | 21:08 |
| jeblair | clarkb: you mean puppet-openstack? | 21:08 |
| clarkb | oh was it puppet? but ya the base branch ordering thing | 21:09 |
| jeblair | that ran since the restart, so that fix should be in effect. i'll look. | 21:09 |
| fungi | boden: it probably makes sense for me to start by troubleshooting the problem you're trying to solve with 512390 rather than troubleshooting the results of that change | 21:09 |
| boden | fungi: absoultely… I’m digging with my change in the meantime… | 21:10 |
| *** gyee has joined #openstack-infra | 21:10 | |
| *** gyee has quit IRC | 21:10 | |
| *** trown is now known as trown|outtypewww | 21:10 | |
| boden | fungi: I do see ubuntu-xenial | Switched to branch 'stable/ocata' in the log, but only for 1 required project (seems like it should be there for all required projects) | 21:10 |
| *** andreas_s has quit IRC | 21:11 | |
| *** gyee has joined #openstack-infra | 21:11 | |
| *** gyee has quit IRC | 21:11 | |
| fungi | boden: it's unclear to me from the commit messages on 512390 what your issue was with the original jobs. is it the failed unit tests i can see on my 513068 debugging change? | 21:13 |
| jeblair | clarkb: oh that's the problem fungi was looking into earlier. the repos are all checked out at stable/ocata. something happens afterwords. | 21:14 |
| jeblair | i'm going to let fungi continue to drive this | 21:14 |
| boden | fungi: the original issue is in the https://etherpad.openstack.org/p/zuulv3-issues | 21:14 |
| fungi | jeblair: yeah, i discovered after some digging that the errors mentioned in the etherpad were being introduced by the change they were testing. that wasn't obvious from the original problem description | 21:14 |
| boden | fungi: just trying to get our stable branches working with zuul v3 is all | 21:15 |
| fungi | boden: is it different than the entry i looked at earlier? the bugs you pointed out were the result of your proposed change to the repo | 21:15 |
| openstackgerrit | Merged openstack-infra/project-config master: add system-required and check-requirements job for senlin-tempest-plugin https://review.openstack.org/512925 | 21:15 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Add linters job to all OSA role jobs" https://review.openstack.org/513125 | 21:15 |
| openstackgerrit | Merged openstack-infra/project-config master: networking-midonet: required-projects for periodic jobs https://review.openstack.org/512946 | 21:15 |
| fungi | boden: i'd rather figure out what was originally broken that your change aims to fix before trying to figure out why your change isn't fixing what you wanted it to | 21:15 |
| dmsimard | pabelanger: we did not have swap enabled on zuulv3 earlier today, did we resolve that ? Because we're about to be maxed out | 21:16 |
| *** salv-orlando has quit IRC | 21:16 | |
| boden | fungi: understood.. if you look at PS1 https://review.openstack.org/#/c/512390/1 you’ll see it’s just a dummy change to trigger v3 gate; those results are whats translated onto the wiki and what’s made me try to change the install script | 21:17 |
| inc0 | where can I find kolla's pubkey for secret encryption? | 21:17 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Logrotate track upstream logs https://review.openstack.org/513189 | 21:17 |
| fungi | inc0: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/encryption.html | 21:18 |
| *** salv-orlando has joined #openstack-infra | 21:18 | |
| clarkb | infra-root ^ after last nights review.o.o reboot stuff I noticed that / is getting near full there. Tis one file is a big offender so reviews to get it rotated would be much appreciated | 21:18 |
| inc0 | thanks fungi | 21:18 |
| *** ijw has joined #openstack-infra | 21:18 | |
| fungi | inc0: the url is "https://zuulv3.openstack.org" and the source is "gerrit" with project names in the form of "openstack/kolla" | 21:18 |
| boden | fungi: let me find the original error for you | 21:18 |
| *** gouthamr has joined #openstack-infra | 21:20 | |
| *** dizquierdo has joined #openstack-infra | 21:20 | |
| *** yolanda has joined #openstack-infra | 21:20 | |
| fungi | boden: i'm guessing it's the version conflicts my dummy change saw in the unit tests: http://logs.openstack.org/90/512390/6/check/openstack-tox-py27/04dbe69/job-output.txt.gz#_2017-10-18_20_13_59_495835 | 21:21 |
| fungi | pkg_resources.VersionConflict: (oslo.log 3.20.1 (/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py27/lib/python2.7/site-packages), Requirement.parse('oslo.log>=3.22.0')) | 21:21 |
| *** salv-orlando has quit IRC | 21:21 | |
| boden | fungi: yes | 21:21 |
| fungi | https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/ocata says you should be constrained to oslo.log===3.20.1 which is what seems to have been installed | 21:23 |
| *** salv-orlando has joined #openstack-infra | 21:23 | |
| fungi | but something wants oslo.log>=3.22.0 | 21:23 |
| *** smatzek has joined #openstack-infra | 21:23 | |
| *** ociuhandu has quit IRC | 21:24 | |
| *** jamesmcarthur has quit IRC | 21:24 | |
| *** kgiusti has quit IRC | 21:24 | |
| *** andreas_s has joined #openstack-infra | 21:25 | |
| fungi | and i agree that looks like your requirements.txt entry for stable/pike: http://git.openstack.org/cgit/openstack/vmware-nsx/tree/requirements.txt?h=stable%2Fpike#n21 | 21:25 |
| *** jamesmcarthur has joined #openstack-infra | 21:25 | |
| *** e0ne has quit IRC | 21:25 | |
| *** e0ne has joined #openstack-infra | 21:26 | |
| *** eharney has joined #openstack-infra | 21:26 | |
| *** e0ne has quit IRC | 21:27 | |
| *** e0ne has joined #openstack-infra | 21:28 | |
| *** e0ne has quit IRC | 21:28 | |
| *** jamesmcarthur has quit IRC | 21:29 | |
| dhellmann | boden , fungi : what cache plugin is being loaded at the point where those tests fail? that's the library that has the oslo.log dependency that doesn't match the constraints | 21:29 |
| *** Sukhdev has joined #openstack-infra | 21:29 | |
| *** andreas_s has quit IRC | 21:29 | |
| *** e0ne has joined #openstack-infra | 21:29 | |
| *** e0ne has quit IRC | 21:30 | |
| fungi | good question... i don't know what a cache plugin is | 21:31 |
| dhellmann | oslo.cache is telling dogpile to load some plugin | 21:31 |
| *** e0ne has joined #openstack-infra | 21:31 | |
| boden | dogpile/cache/region.py", line 415, in configure ?? | 21:31 |
| *** e0ne has quit IRC | 21:31 | |
| dhellmann | I'm not sure how it was installed without using the constraints, but if we figure out what the library is that's one more piece of info | 21:32 |
| inc0 | fungi: https://zuulv3.openstack.org cries that cert is invalid - planned or bug? | 21:32 |
| *** rbrndt has quit IRC | 21:32 | |
| fungi | inc0: known, it will eventually in a week or two perhaps) be renamed to zuul.o.o and we have a valid cert for that | 21:32 |
| dhellmann | boden : I'm looking at this section: http://logs.openstack.org/90/512390/6/check/openstack-tox-py27/04dbe69/job-output.txt.gz#_2017-10-18_20_13_59_497957 | 21:32 |
| inc0 | ok | 21:32 |
| dhellmann | boden: from there to the end of the traceback | 21:32 |
| *** e0ne has joined #openstack-infra | 21:33 | |
| *** purp_too is now known as purp | 21:33 | |
| *** e0ne has quit IRC | 21:33 | |
| *** rbrndt has joined #openstack-infra | 21:34 | |
| *** baoli has quit IRC | 21:35 | |
| *** smatzek has quit IRC | 21:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: replace the common git review configuration for release jobs https://review.openstack.org/513166 | 21:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Redo the gerrit proposal bot secret https://review.openstack.org/513176 | 21:37 |
| odyssey4me | are there any suse repositories in infra, similar to the centos/epel/ubuntu repositories? or are they cached at all? | 21:38 |
| *** claudiub has quit IRC | 21:38 | |
| fungi | dhellmann: boden: i'm starting to wonder if this is behavior getting introduced by tox-siblings, since http://logs.openstack.org/90/512390/6/check/openstack-tox-py27/04dbe69/tox/py27-5.log shows about half a dozen other deps installed from source | 21:39 |
| dhellmann | I don't know what "tox-siblings" is. Is that something new? | 21:39 |
| fungi | wondering if there's an unreleased state in one of their stable/ocata branches with a late backport from master after the requirements freeze | 21:39 |
| *** dizquierdo has quit IRC | 21:40 | |
| *** jtomasek has quit IRC | 21:40 | |
| fungi | dhellmann: it's... i'm still not 100% clear as i wasn't following the design discussions but i believe it tries to install required-projects from source on matching branches instead of released packages when you're using tox | 21:40 |
| dhellmann | hrm | 21:41 |
| fungi | it's still a knowledge gap on my part. i'm sure someone knows why this isn't opening us up to test holes for unreleased features, but i haven't been able to suss that out yet | 21:41 |
| dhellmann | there are cases where we want to gate changes against unreleased things, but usually those are gates on the library not the consumer | 21:42 |
| fungi | yeah, in the case of neutron plugins like vmware-nsx i suppose that's the case for some parts of the neutron ecosystem they're integrating with? | 21:42 |
| *** andreas_s has joined #openstack-infra | 21:42 | |
| dhellmann | vmware-nsx does have a lot of cross-repo dependencies listed | 21:43 |
| fungi | like... they can't pip install neutron itself without doing so from source at a minimum | 21:43 |
| dhellmann | epi shows 3 plugins for dogpile.cache when I run "tox -e py27 --notest" locally and check the contents: http://paste.openstack.org/show/624032/ | 21:43 |
| *** edmondsw has quit IRC | 21:44 | |
| dhellmann | my guess is it's using the dict one, since the unit tests would want to use ram | 21:44 |
| dhellmann | but that seems to point to an incompatible verison of oslo.cache being installed | 21:44 |
| dhellmann | I don't see oslo.cache listed in the dependencies for vmware-nsx in projects.yaml but it is likely to be a second-order dependency of something like neutron, which is listed | 21:46 |
| dhellmann | do we have logs showing tox-siblings installing those other things? can we get version numbers or branch names out of it? | 21:46 |
| fungi | dhellmann: http://logs.openstack.org/90/512390/6/check/openstack-tox-py27/04dbe69/tox/py27-siblings.txt | 21:47 |
| *** openstackgerrit has quit IRC | 21:48 | |
| dhellmann | well, I don't see it upgrading oslo.log there | 21:49 |
| clarkb | fungi: when you have a moment, there is email to infra list about ssh connection limits to gerrit. What are the chances that that is something like the great firewall killing the connection without gerrit noticing so they reconnect over and over? | 21:49 |
| clarkb | oh thats taiwan though so likely not a problem | 21:50 |
| *** edmondsw has joined #openstack-infra | 21:50 | |
| fungi | it's not. it ends up with oslo.log===3.20.1 according to pip freeze, but then we have something getting imported which is complaining it actually declared a requirement on oslo.log>=3.22.0 which is unmet | 21:50 |
| boden | dhellmann fungi: you guys are digging deeper than my knowledge level and I don’t want to derail, but I saw http://logs.openstack.org/90/512390/6/check/openstack-tox-py27/04dbe69/job-output.txt.gz#_2017-10-18_20_02_00_362309 in the log and would’ve expected to see that for all the required projects.. my initial impression here is that the other required projects didn’t get set to stable/ocata… I’m probably way off tho | 21:50 |
| fungi | clarkb: it's likely the great corporate firewall of whatever company they're working for, or the terrible overloaded internet service provider they're connecting through | 21:51 |
| *** andreas_s has quit IRC | 21:51 | |
| dhellmann | fungi : yeah, if tox-siblings installed something that wanted a newer version of a dependency I would have expected it to upgrade that thing at the same time | 21:52 |
| dhellmann | the pike dependency for oslo.log is 3.22.0 so maybe a pike version of a library is being pulled in | 21:53 |
| fungi | fun... networking-l2gw imports oslo.log without declaring it as a dependency. that's pretty unsafe | 21:53 |
| *** edmondsw has quit IRC | 21:54 | |
| dhellmann | I'll bet that's not the only place that happens. | 21:55 |
| *** edmondsw has joined #openstack-infra | 21:56 | |
| *** efoley has quit IRC | 21:56 | |
| fungi | yeah, it used to happen in nova and then one day one of their dependencies released a new version which no longer used something they'd been importing without declaring a dependency on in nova | 21:56 |
| *** markvoelker_ has joined #openstack-infra | 21:57 | |
| fungi | then r1chardj0n3s wrote a tool to check for that, but never got buy-in to start using it in check jobs | 21:57 |
| dhellmann | do you remember the name of the tool? | 21:57 |
| *** edmondsw_ has joined #openstack-infra | 21:58 | |
| *** slaweq_ has joined #openstack-infra | 21:58 | |
| fungi | it changed names at one point, but is currently https://pypi.python.org/pypi/pip_check_reqs | 21:58 |
| fungi | fwiw, tap-as-a-service also fails to declare a dependency on oslo.log but still imports from it | 22:00 |
| *** edmondsw has quit IRC | 22:00 | |
| *** markvoelker has quit IRC | 22:00 | |
| *** bobh has quit IRC | 22:00 | |
| *** edmondsw_ has quit IRC | 22:02 | |
| *** slaweq_ has quit IRC | 22:02 | |
| fungi | anyway, i've checked through all the required-projects for that job and all of the ones which declare a dep on oslo.log in their requirements lists on stable/ocata match the stable/ocata global requirement (compatible with the stable/ocata upper constraint) | 22:02 |
| *** lbragstad has quit IRC | 22:03 | |
| *** rwsu has quit IRC | 22:05 | |
| dhellmann | fungi : oslo.cache 1.26.0 wants oslo.log > 3.22.0 | 22:07 |
| dhellmann | why is that version of oslo.cache being used/allowed? | 22:07 |
| fungi | great question | 22:07 |
| dhellmann | boden : new clue ^^ | 22:07 |
| fungi | constraints would have chosen 1.17.0 | 22:08 |
| fungi | http://logs.openstack.org/90/512390/6/check/openstack-tox-py27/04dbe69/job-output.txt.gz#_2017-10-18_20_07_11_291460 | 22:08 |
| fungi | neutron wants it | 22:09 |
| *** rloo has left #openstack-infra | 22:09 | |
| fungi | though neutron just declares oslo.cache>=1.5.0 so we're not constraining when installing there i suspect | 22:09 |
| dhellmann | not in the ocata branch, though | 22:09 |
| dhellmann | ah, yeah | 22:09 |
| dhellmann | so, does tox-siblings honor constraints? | 22:10 |
| fungi | it's supposed to | 22:10 |
| fungi | but yes, that's in the "TASK [tox-siblings : Install any sibling python packages]" section | 22:11 |
| *** dave-mccowan has quit IRC | 22:11 | |
| dhellmann | what do those messages about not being able to find the remote mean in http://logs.openstack.org/90/512390/6/check/openstack-tox-py27/04dbe69/tox/py27-5.log | 22:11 |
| fungi | mordred: jeblair: ^ who is our tox-siblings expert? | 22:11 |
| *** rhallisey has quit IRC | 22:11 | |
| fungi | dhellmann: that's a benign error because pip is being asked to install from a git tree on disk with no origin remote | 22:12 |
| dhellmann | ah | 22:12 |
| fungi | it would include the origin url if it could find it, but instead it throws that error and includes a comment in the pip freeze output to that effect | 22:12 |
| dhellmann | I think I'm going to call it a day. | 22:12 |
| *** bstinson has quit IRC | 22:12 | |
| *** jklare has quit IRC | 22:13 | |
| *** openstackgerrit has joined #openstack-infra | 22:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Merge tenant reconfiguration events https://review.openstack.org/513195 | 22:13 |
| jeblair | fungi: oh hi | 22:14 |
| fungi | jeblair: how do we tell tox-siblings to apply a constraints file when it's installing dependencies of required-projects into the tox env? | 22:14 |
| fungi | that seems to be the underlying issue... the tox-siblings task has installed oslo.cache==1.26.0 as a transitive dependency of neutron stable/ocata (declared as a required project for the job) | 22:15 |
| fungi | but stable/ocata constraints should have limited that to 1.17.0 | 22:15 |
| jeblair | fungi: welp, i'm guessing this is where it should be done: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/tox-siblings/library/tox_install_sibling_packages.py#n162 | 22:15 |
| *** gyee has joined #openstack-infra | 22:17 | |
| *** jklare has joined #openstack-infra | 22:18 | |
| fungi | thanks, so i guess we're missing constraints support here actually: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/tox-siblings/library/tox_install_sibling_packages.py#n167 | 22:18 |
| jeblair | fungi: so we need to find the right constraints file and add "-c that file" to line 167? | 22:19 |
| fungi | ahh, note the TODO comment just above :/ | 22:19 |
| *** notmyname has quit IRC | 22:19 | |
| fungi | # TODO(mordred) Account for upper-constraints during this install | 22:19 |
| jeblair | fungi: yeah, sorry, i tried to link to that, but cgit highlighting :( | 22:19 |
| *** bstinson has joined #openstack-infra | 22:19 | |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Create a playbook for publishing contributor-guide https://review.openstack.org/512865 | 22:19 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Set contributor-guide location to /contributors https://review.openstack.org/512871 | 22:19 |
| fungi | jeblair: oh, duh, i should have looked at the line number in the url. i have a tall browser and didn't notice it had landed me near the end of the file so thought you had linked me to the get_sibling_python_packages() def further up | 22:20 |
| *** notmyname has joined #openstack-infra | 22:21 | |
| jeblair | fungi: so do you reckon adding "-c <requirements file>" is the solution? or is it more complex than that? | 22:21 |
| fungi | okay, so the long and short of it is we should not be using tox-siblings on jobs run for projects which need constraints | 22:21 |
| jeblair | erm i'm not sure | 22:21 |
| openstackgerrit | Michael Johnson proposed openstack-infra/infra-manual master: Clarify which jobs belong in the project block https://review.openstack.org/513197 | 22:21 |
| fungi | jeblair: probably, and then we need to have a way to pass in a constraints file | 22:21 |
| jeblair | fungi: i'm not sure we shouldn't run tox-siblings, because, tbh, i don't know what it means to clone these repos on to disk and not use them :) | 22:22 |
| fungi | and i guess a sane fallback of not using -c when there is no constraints file provided | 22:22 |
| fungi | jeblair: i meant tox-siblings is not safe in its current form for projects using constraints | 22:22 |
| *** lbragstad has joined #openstack-infra | 22:22 | |
| jeblair | fungi: yes. though i'm not sure the alternative is either. :) | 22:22 |
| *** lukebrowning has quit IRC | 22:22 | |
| *** med_ has quit IRC | 22:23 | |
| *** felipemonteiro__ has quit IRC | 22:23 | |
| *** med_ has joined #openstack-infra | 22:23 | |
| *** jklare has quit IRC | 22:23 | |
| *** jklare has joined #openstack-infra | 22:23 | |
| *** med_ is now known as Guest99060 | 22:23 | |
| fungi | in v2 we pip installed everything except the project in question and they all had bespoke tooling to zuul-cloner their source dependencies | 22:23 |
| fungi | which should in theory still have worked for legacy jobs without the tox-siblings magic | 22:24 |
| jeblair | fungi: so i guess if we turn off tox-siblings, the bespoke tooling will still do the installing? | 22:24 |
| jeblair | fungi: but these aren't legacy jobs, right? | 22:24 |
| jeblair | these are the new v3 tox-pyxx jobs? | 22:24 |
| fungi | yeah, that seems to be the behavior change here. we're preemptively installing their source dependencies and doing so differently than what they were doing under v2 | 22:24 |
| fungi | jeblair: ahh, right, so i guess we made v3 native jobs which didn't have complete support for constraints | 22:25 |
| fungi | anyway, we can probably fix this, i just worry that we have a lot of (particularly plugin/extension) projects with constraints-related breakage, particularly on their stable branches | 22:26 |
| jeblair | fungi: i think we have 3 options: 1) add constraints support to tox-siblings; 2) temporarily turn off tox-siblings for all openstack-tox jobs; 3) turn off tox-siblings just for the neutron jobs (maybe in the project-template)? | 22:26 |
| fungi | s/probably// | 22:26 |
| jeblair | fungi: i *think* option 1 might not be hard | 22:26 |
| jeblair | why don't i poke at it for a few mins | 22:26 |
| fungi | right, i'm just trynig to wrap my head around the plumbing to get from the job into that function call wit the constraints details | 22:26 |
| jeblair | me too | 22:27 |
| jeblair | i think all we need to do is add it as a role variable and pass it in to the module. | 22:27 |
| fungi | cool, that means at least one of us will hopefully have a good enough understanding to review what the other writes ;) | 22:27 |
| jeblair | it should already be set by the job | 22:28 |
| fungi | ahh, right, since this "script" is actually an ansible module, we should already have context into that without needing to plumb it through the main task for the role | 22:29 |
| fungi | so no need for command-line parameter nonsense | 22:30 |
| fungi | or do modules not get a fancy context object with all the ansible variables? looks like we define an argument list in this one and then have to pass stuff in from the task anyway | 22:31 |
| jeblair | fungi: yeah, we have to pass it explicitly | 22:32 |
| *** edmondsw has joined #openstack-infra | 22:32 | |
| fungi | i guess that makes for a more stable interface anyway | 22:32 |
| *** esberglu has quit IRC | 22:33 | |
| *** salv-orlando has quit IRC | 22:33 | |
| *** esberglu has joined #openstack-infra | 22:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Support upper-constraints in tox-siblings https://review.openstack.org/513199 | 22:35 |
| jeblair | fungi: maybe this ^ | 22:35 |
| fungi | okay, so we already have constraints support plumbed all the way to the main task of the tox role in zuul-jobs | 22:35 |
| fungi | aha, i'll review | 22:35 |
| *** edmondsw has quit IRC | 22:36 | |
| clarkb | jeblair: event merging lgtm but left a couple minor comments on it | 22:36 |
| jeblair | clarkb: cool thx | 22:37 |
| jeblair | dirk, frickler, AJaeger: the new v3-native requirements cross-check job i wrote is ready for review here: https://review.openstack.org/512901 | 22:37 |
| fungi | jeblair: that matches what we seem to be getting from the tox role, so seems reasonable | 22:38 |
| fungi | (on 513199 i mean) | 22:39 |
| fungi | and default(omit) seems to be used as a fallback in configure-mirrors and validate-host roles too, though i was unfamiliar with that syntax | 22:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Merge tenant reconfiguration events https://review.openstack.org/513195 | 22:41 |
| jeblair | clarkb: i have implemented your suggestions! | 22:41 |
| *** gouthamr has quit IRC | 22:41 | |
| jeblair | fungi: yeah, i'm pretty sure it means "don't pass this argument to the module". so that lets the module do whatever its default handling is. | 22:41 |
| jeblair | which i'm hoping, in this case, is just "don't have an entry in the dict" | 22:42 |
| *** andreas_s has joined #openstack-infra | 22:42 | |
| *** nicolasbock has quit IRC | 22:43 | |
| *** esberglu has quit IRC | 22:43 | |
| fungi | it's amusing... if i attempt to fix the mishmash of normal and code style fonts in the issues pad by highlighting a section and choosing normal style, i get immediately disconnected from the etherpad and prompted to force-reconnect | 22:44 |
| jeblair | me too! | 22:45 |
| fungi | so, i guess they're going to remain that way | 22:45 |
| jeblair | i've had some success deleting small sections and retyping them, but the virus has taken hold and i don't think the patient can be saved | 22:45 |
| jeblair | thank goodness it's just courier and not papyrus | 22:46 |
| fungi | comic sans ftw | 22:46 |
| *** andreas_s has quit IRC | 22:46 | |
| fungi | wow, even trying to change the style on just one line crashes me out | 22:47 |
| *** rbrndt has quit IRC | 22:49 | |
| *** eharney has quit IRC | 22:52 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Fix error in job parser https://review.openstack.org/513201 | 22:53 |
| jeblair | fungi, clarkb: can you +3 that? i have tested it with openstack-zuul-jobs and verified it works now. i'll tag a new release after it merges. | 22:54 |
| jeblair | (it's only the new functionality that's broken) | 22:55 |
| fungi | jeblair: sure | 22:55 |
| clarkb | fungi: can I also get a review on https://review.openstack.org/#/c/513189/ so that doesn't get lost | 22:55 |
| jeblair | oh, let me update the examples so it's tested in repo | 22:55 |
| *** andreas_s has joined #openstack-infra | 22:56 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Fix error in job parser https://review.openstack.org/513201 | 22:56 |
| jeblair | there we go | 22:56 |
| fungi | jeblair: i just realized that tox-siblings might need more... pip install with a constraints set won't work _if_ any of the packages being installed from source are also in the constraints | 22:57 |
| clarkb | fungi: ugh ya that problem is annoying | 22:58 |
| fungi | we work around it elsewhere by filtering the constraints list, or doing two passes | 22:58 |
| fungi | er, well actually both | 22:58 |
| jeblair | fungi: does tox do this automatically or something? | 22:58 |
| fungi | jeblair: no, projects again have their own bespoke tooling to filter their packages out of the constraints lists or to preinstall their requirements list with constraints before installing themselves without | 22:59 |
| fungi | and then using --no-deps on the latter to keep from upgrading any constrained requirements | 22:59 |
| *** andreas_s has quit IRC | 23:00 | |
| fungi | in this case we don't necessarily have context to just install the sibling projects requirements without installing them (requirements.txt and the like are openstackisms) | 23:00 |
| fungi | so we probably need to remove the siblings from the constraints list | 23:00 |
| *** Sukhdev has quit IRC | 23:00 | |
| fungi | in which case i guess then we can get by with only one pip install invocation | 23:01 |
| jeblair | fungi: okay, so the fix to tox-siblings should make a new requirements file for each "pip install" it does, omitting the thing being pip installed? | 23:01 |
| jeblair | oh | 23:01 |
| jeblair | or the thing you said | 23:01 |
| *** rwsu has joined #openstack-infra | 23:01 | |
| jeblair | either thing is easy -- we're in python :) | 23:01 |
| fungi | yeah, thinking through the options, that's probably the simplest and requires the least extra state | 23:01 |
| fungi | lemme get a good example | 23:02 |
| jeblair | fungi: one install or multiple? | 23:02 |
| ianw | jeblair: http://paste.openstack.org/show/624036/ : AttributeError: 'MergeJob' object has no attribute 'updated' <-- not important, just don't remember seeing that running unit tests before. not exactly sure where it came from | 23:02 |
| jeblair | ianw: which test? | 23:03 |
| fungi | jeblair: oh, reading through that module, it's installing the sibling packages in a loop, so we probably have to stop that and switch to a single install command which does them all at once | 23:03 |
| *** tosky has quit IRC | 23:03 | |
| fungi | since pip won't preserve any state between runs | 23:04 |
| ianw | jeblair: that's the thing, it just spewed out on the console, so not sure | 23:04 |
| fungi | and we'll end up reconstraining and installing a package for something we installed from source | 23:04 |
| *** xarses has quit IRC | 23:04 | |
| jeblair | fungi: want to -1 my change with that info, and wait for mordred to chime in? (in case there's an important reason it's a loop like that) | 23:05 |
| fungi | jeblair: yep, will do. also this is the edit-constraint implementation from the requirements package: http://git.openstack.org/cgit/openstack/requirements/tree/openstack_requirements/cmds/edit_constraint.py | 23:05 |
| *** dave-mccowan has joined #openstack-infra | 23:06 | |
| *** sdague has quit IRC | 23:07 | |
| *** lbragstad has quit IRC | 23:09 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Disable tox-siblings https://review.openstack.org/513205 | 23:10 |
| jeblair | fungi: ^ maybe do this for now? | 23:11 |
| fungi | i'm equally concerned that we now have jobs depending on tox-siblings behavior | 23:11 |
| jeblair | it's pretty new :) | 23:11 |
| jeblair | i was worried until you reminded me they all have custom install thingies | 23:12 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder master: Dont install python-virtualenv for py3k in deb https://review.openstack.org/513168 | 23:12 |
| jeblair | do you think any have been removed? | 23:12 |
| fungi | good point, that's unlikely at this early stage | 23:12 |
| SpamapS | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all ... ~200MB to OOM? | 23:12 |
| fungi | most of the early adopter v3 jobs have not been tox-based | 23:13 |
| *** ijw has quit IRC | 23:13 | |
| fungi | SpamapS: we have swap now | 23:13 |
| jeblair | swapping kills us | 23:13 |
| fungi | and we're ~2.5gb into it at this point | 23:13 |
| jeblair | oh so we're probably dead | 23:14 |
| fungi | yeah, it's probably performing terribly at this point but not actually resulting in an oom | 23:14 |
| jeblair | huh. it actually looks ok right now | 23:14 |
| SpamapS | heh, might even be one of the times where it's ok to swap.. if these old layouts just sit there unused until the very end when they get GC'd | 23:14 |
| jeblair | i guess it swapped out idle pages. | 23:14 |
| SpamapS | Right, unlike on the executor, where you have constant usage of most of the memory, in this case, most of it is just there for something in the future. | 23:16 |
| jeblair | the layouts themselves are pretty continually accessed, but they are big, and we only look at a small part once we've launched jobs, so it's possible that entire pages worth of them can be swapped out | 23:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Add _projects to convert project list to dictionary https://review.openstack.org/512868 | 23:17 |
| *** andreas_s has joined #openstack-infra | 23:18 | |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Fix error in job parser https://review.openstack.org/513201 | 23:19 |
| *** hongbin has quit IRC | 23:21 | |
| SpamapS | jeblair: the ratio of data swapped to the actual swapping activity would suggest that's what is happening. | 23:22 |
| SpamapS | probably not a long term state you want to be in, but perhaps we'll find that a normal level of zuul layout churn won't produce such a wide array of layouts and this is just the price of a mass migration. | 23:23 |
| *** andreas_s has quit IRC | 23:23 | |
| *** jamesmcarthur has joined #openstack-infra | 23:30 | |
| *** aeng has joined #openstack-infra | 23:32 | |
| *** thorst has joined #openstack-infra | 23:32 | |
| *** salv-orlando has joined #openstack-infra | 23:34 | |
| *** ijw has joined #openstack-infra | 23:34 | |
| *** jamesmcarthur has quit IRC | 23:35 | |
| *** thorst has quit IRC | 23:36 | |
| *** smatzek has joined #openstack-infra | 23:37 | |
| jeblair | i'm starting to think there may be another leak | 23:38 |
| *** ijw has quit IRC | 23:38 | |
| smcginnis | Hmm, this doesn't look good: http://logs.openstack.org/92/513192/1/check/build-openstack-sphinx-docs/0bfe0d9/job-output.txt.gz#_2017-10-18_23_17_38_644262 | 23:39 |
| *** salv-orlando has quit IRC | 23:39 | |
| smcginnis | Do we need fresh images or something? | 23:39 |
| jeblair | smcginnis: it's probably transient, but can you add it to the etherpad? | 23:40 |
| *** Apoorva_ has joined #openstack-infra | 23:40 | |
| smcginnis | jeblair: Sure, I'll recheck and add it. | 23:40 |
| *** smatzek has quit IRC | 23:41 | |
| jeblair | i think i'm going to have to restart the scheduler; i'll try to get as much info as i can first | 23:41 |
| *** Apoorva has quit IRC | 23:43 | |
| *** ijw has joined #openstack-infra | 23:44 | |
| *** andreas_s has joined #openstack-infra | 23:46 | |
| *** smatzek has joined #openstack-infra | 23:46 | |
| *** Apoorva_ has quit IRC | 23:47 | |
| *** Apoorva has joined #openstack-infra | 23:47 | |
| EmilienM | is it just me or http://zuulv3.openstack.org/ is really slow? | 23:49 |
| EmilienM | (even not working for me) | 23:50 |
| *** andreas_s has quit IRC | 23:50 | |
| *** ijw has quit IRC | 23:51 | |
| EmilienM | status.json: Proxy Error | 23:51 |
| jeblair | EmilienM: yeah, i'm going to restart it in a few minutes, after i collect some debug info | 23:52 |
| EmilienM | ok | 23:52 |
| *** rlandy is now known as rlandy|bbl | 23:56 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Give layout objects a unique ID https://review.openstack.org/513207 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!