| *** xarses_ has quit IRC | 00:05 | |
| dhinesh | when i run the command "echo 'status' | nc 127.0.0.1 4730 -w 1" gearman does not show the jobs configured under zuul/layout/layout.yaml | 00:06 |
|---|---|---|
| dhinesh | is there the way to verify if it is getting settings from layout.yml? | 00:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add ability to work in other auth contexts https://review.openstack.org/519394 | 00:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add helper property to get the current user id https://review.openstack.org/519511 | 00:13 |
| clarkb | dhinesh: I think that zuul logs when it registers jobs and it logs what gearman server it connects to | 00:13 |
| clarkb | dhinesh: you should be able to see if that is happening and if so against what server | 00:14 |
| *** Sukhdev has joined #openstack-infra | 00:14 | |
| pabelanger | dhellmann: yah, build-openstack-puppet-tarball should provide that test to build puppet tarballs. | 00:16 |
| dhinesh | clarkb: looks like there are errors in the log , "ERROR gerrit.GerritWatcher: Exception on ssh event stream" | 00:18 |
| clarkb | mordred: if you are wanting to review things https://review.openstack.org/#/c/517831/ is relatively important now that git 2.15 is hitting places | 00:18 |
| pabelanger | ianw: yah, I want to say we can drop rdo-release now, and have devstack job just install that package if needed | 00:18 |
| clarkb | dhinesh: that indicates a problem connecting to gerrit for the stream events | 00:18 |
| mordred | clarkb: the comment in there indicates that set-upstream-to is not supported in 1.7 ... do we need to support both forms? | 00:19 |
| clarkb | mordred: I did a scan of allt he distros and oldest I could find is centos 7 with 1.8 | 00:20 |
| mordred | oh - I see the comment about reqiuring 1.8 now | 00:20 |
| mordred | cool | 00:20 |
| clarkb | where all the distros ~= ubuntu, suse, centos | 00:20 |
| mordred | clarkb: wfm | 00:20 |
| clarkb | worst case they install older version | 00:20 |
| dhinesh | clarkb: yeah but that issue seems orthogonal, tying to fix that one anyways | 00:21 |
| ianw | dhinesh: also, gear is ssl these days, so "echo status | openssl s_client -quiet -connect 127.0.0.1:4730 -cert /etc/zuul/ssl/client.pem -key /etc/zuul/ssl/client.key" will work | 00:21 |
| ianw | (depending on your key locations, etc) | 00:21 |
| *** e0ne has quit IRC | 00:23 | |
| *** hongbin has quit IRC | 00:23 | |
| dhinesh | ianw: looks like i don't have client.key or client.pem under /etc/zuul/ssl is there a way to create them? | 00:24 |
| *** stakeda has joined #openstack-infra | 00:25 | |
| ianw | dhinesh: this is going to be part of the ssl setup in your environment ... just if you're talking to port 4730 and nothing is coming back, that's probably something to look at | 00:27 |
| *** edmondsw has quit IRC | 00:27 | |
| *** edmondsw has joined #openstack-infra | 00:28 | |
| ianw | but, you're not going to see jobs in that status for zuulv3 | 00:28 |
| ianw | pabelanger: yeah, the rdo-release is gone but the centos-release-openstack-* was a little overlooked by myself, at least | 00:29 |
| clarkb | noticing we don't seem to have job runtime broken down by node provider in graphite anymore | 00:29 |
| pabelanger | ianw: ah | 00:29 |
| clarkb | does anyone know if I'm juts overlooking where that is now? | 00:29 |
| *** rlandy|rover is now known as rlandy|rover|bbl | 00:30 | |
| pabelanger | I haven't looked recently | 00:30 |
| *** Goneri has quit IRC | 00:31 | |
| *** edmondsw has quit IRC | 00:32 | |
| *** salv-orlando has joined #openstack-infra | 00:38 | |
| *** andreas_s has joined #openstack-infra | 00:42 | |
| *** salv-orlando has quit IRC | 00:43 | |
| *** tpsilva has quit IRC | 00:44 | |
| ianw | ... ok, context switched back in my project-config building env and doing some test dib builds on that now | 00:45 |
| ianw | pabelanger: btw, that reminds me that i noticed there's a few hacks in windmill to get dib installed in a virtualenv into /usr/local/bin | 00:45 |
| ianw | nodepool should take a config path for disk-image-create rather than asssuming it's location. we added a bunch of stuff so that it activates the virtualenv within itself so it "juts works" (famous last words) | 00:46 |
| ianw | on my todo list :) | 00:46 |
| *** andreas_s has quit IRC | 00:47 | |
| ianw | i also want to update my daily log generator to have a todo list section. that's like the ultimate in procrastination | 00:47 |
| pabelanger | ianw: yah, for the most part, I did that for cross-repo testing. But do remember nodepool does activate things now | 00:47 |
| ianw | yeah, i think it's probably valid to manage nodepool & dib in different virtualenvs, though, if you want | 00:49 |
| *** edmondsw has joined #openstack-infra | 00:51 | |
| *** sdague has quit IRC | 00:55 | |
| *** sree has joined #openstack-infra | 00:55 | |
| *** edmondsw has quit IRC | 00:55 | |
| *** Nil_ has quit IRC | 00:56 | |
| *** sree has quit IRC | 00:59 | |
| *** dingyichen has joined #openstack-infra | 01:00 | |
| *** cuongnv has joined #openstack-infra | 01:02 | |
| *** aeng has joined #openstack-infra | 01:04 | |
| *** jkilpatr has quit IRC | 01:05 | |
| *** cshastri has joined #openstack-infra | 01:08 | |
| dhinesh | ianw: is the ssl setup required even if gearmen and zuul are running on the single system? | 01:25 |
| *** liujiong has joined #openstack-infra | 01:25 | |
| pabelanger | dhinesh: I recommend it, since we pass secrets via gearman | 01:26 |
| *** aeng has quit IRC | 01:31 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove legacy jobs from ansible-role-diskimage-builder https://review.openstack.org/519519 | 01:34 |
| *** gongysh has joined #openstack-infra | 01:35 | |
| *** kiennt26 has joined #openstack-infra | 01:37 | |
| *** rwsu has joined #openstack-infra | 01:39 | |
| *** salv-orlando has joined #openstack-infra | 01:39 | |
| *** sree has joined #openstack-infra | 01:42 | |
| *** gcb has joined #openstack-infra | 01:42 | |
| *** eharney has quit IRC | 01:42 | |
| pabelanger | mnaser: ianw: mind a +3 https://review.openstack.org/519519/ | 01:43 |
| pabelanger | hacking on native zuulv3 jobs :) | 01:43 |
| *** salv-orlando has quit IRC | 01:44 | |
| *** Apoorva_ has joined #openstack-infra | 01:44 | |
| *** sree has quit IRC | 01:46 | |
| *** Apoorva has quit IRC | 01:48 | |
| *** aeng has joined #openstack-infra | 01:48 | |
| *** Apoorva_ has quit IRC | 01:49 | |
| *** caphrim007 has quit IRC | 01:52 | |
| *** robynh has joined #openstack-infra | 01:52 | |
| *** mriedem has quit IRC | 01:54 | |
| *** kiennt26 has quit IRC | 01:54 | |
| *** Douhet has quit IRC | 01:57 | |
| *** aeng has quit IRC | 02:00 | |
| *** aeng has joined #openstack-infra | 02:00 | |
| *** Douhet has joined #openstack-infra | 02:02 | |
| *** sbezverk has quit IRC | 02:03 | |
| *** aeng has quit IRC | 02:04 | |
| dhinesh | pabelanger: i created self signed certificate with openssl and added it as part of /etc/zuul/ssl | 02:05 |
| dhinesh | pabelanger: looks like it did not help the setup | 02:05 |
| *** liujiong has quit IRC | 02:06 | |
| *** liujiong has joined #openstack-infra | 02:06 | |
| *** sbezverk has joined #openstack-infra | 02:09 | |
| pabelanger | dhinesh: this should help: http://git.openstack.org/cgit/openstack-infra/zuul/tree/tests/fixtures/gearman/README.rst?h=feature/zuulv3 | 02:12 |
| mnaser | pabelanger: don’t we want to remove those jobs once we have the new ones in? | 02:13 |
| pabelanger | mnaser: no, is fine in this case. I won't be approving anything until new jobs land | 02:13 |
| *** namnh has joined #openstack-infra | 02:13 | |
| pabelanger | dhinesh: https://review.openstack.org/473916/ is more info about SSL and gearman | 02:14 |
| *** liujiong has quit IRC | 02:14 | |
| *** liujiong has joined #openstack-infra | 02:15 | |
| *** liujiong has quit IRC | 02:15 | |
| dhinesh | pabelanger: thanks, i will look into it, and one question, does the layout.yaml file location /etc/zuul/layout.yaml or /etc/zuul/layout/layout.yaml | 02:15 |
| *** annp has joined #openstack-infra | 02:16 | |
| pabelanger | dhinesh: layout.yaml is zuulv2.5, you'll want main.yaml. The default should be /etc/zuul/main.yaml | 02:17 |
| pabelanger | but for reasons, we likely have it in /etc/zuul/layout/main.yaml | 02:17 |
| *** links has joined #openstack-infra | 02:17 | |
| mnaser | pabelanger: cool done | 02:18 |
| dhinesh | pabelanger: looks like i have zuul2.5 | 02:19 |
| *** rfolco has quit IRC | 02:19 | |
| *** camunoz has quit IRC | 02:19 | |
| pabelanger | dhinesh: okay, in that case, I don't think we have SSL support on gearman for 2.5 | 02:20 |
| pabelanger | dhinesh: I was assuming you were installing zuulv3 | 02:20 |
| *** kiennt26 has joined #openstack-infra | 02:20 | |
| pabelanger | mnaser: danke! | 02:20 |
| dhinesh | pabelanger: in that case is it recommended to move to zuulv3.0? | 02:21 |
| pabelanger | dhinesh: well, trickly question. If you move to zuulv3, it is more a developer preview right now. We haven't officially tag v3.0 just yet. | 02:23 |
| pabelanger | so, there is a chance something will still break or change | 02:23 |
| pabelanger | but, 2.5 isn't really meant to be run outside openstack-infra | 02:23 |
| clarkb | pabelanger: it is though | 02:24 |
| clarkb | just not the ansible zls | 02:24 |
| clarkb | 2.5 and newer is completely backward compat | 02:24 |
| pabelanger | yah, zuulv2.5 will work with jenkins | 02:24 |
| pabelanger | zuul-launcher is another story. | 02:25 |
| pabelanger | so, you might want to test out zuulv3, but would hold off running in production until we officially tag 3.0 | 02:25 |
| pabelanger | otherwise, zuulv2.5 and jenkins work as clarkb says | 02:25 |
| dhinesh | pabelanger: am looking to setup the CI, so i can merge cinder for queens release, so probably i want the stable version to make it possible. | 02:28 |
| dhinesh | pabelanger: what does "zuulv2.5 will work with jenkins" mean, i don't need zuul to communicate with gearmen then? | 02:30 |
| *** iyamahat_ has quit IRC | 02:30 | |
| *** yamahata has quit IRC | 02:30 | |
| pabelanger | dhinesh: no, you still need zuul to talk with gearman. Are you using our puppet-openstackci modules and third-party CI docs? | 02:31 |
| dhinesh | pabelanger: i am using most of this https://docs.openstack.org/infra/openstackci/third_party_ci.html to setup the CI | 02:32 |
| pabelanger | dhinesh: but back to your original question, you won't need SSL for gearman using zuulv2.5. I don't think we actually support it | 02:32 |
| pabelanger | dhinesh: okay, so you are on the right path. That is the docs I would recommend following | 02:32 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs from ansible-role-diskimage-builder https://review.openstack.org/519519 | 02:32 |
| dhinesh | pabelanger: if the ssl is not the cause, can you think of any other issues that would cause gearman not to show layout.yaml "jobs" on "echo 'status' | nc 127.0.0.1 4730 -w 1 " | 02:35 |
| dhinesh | pabelanger: i checked the logs | 02:36 |
| dhinesh | pabelanger: the zuul logs only show a bunch of warnings | 02:36 |
| ianw | dhinesh: the jobs endpoints registered as gearman functions aren't there with zuulv3 | 02:36 |
| pabelanger | dhinesh: which warnings? | 02:37 |
| dhinesh | pabelanger: http://paste.openstack.org/show/626215/ | 02:39 |
| pabelanger | dhinesh: ah, ya. That is okay, just a warning like you said. Shouldn't stop things from working | 02:40 |
| *** salv-orlando has joined #openstack-infra | 02:40 | |
| dhinesh | pabelanger: is there a way to bypass zuul and run tempest tests only through jenkins | 02:42 |
| pabelanger | dhinesh: what does your layout.yaml file look like? Can you explain what your issue is? | 02:44 |
| *** salv-orlando has quit IRC | 02:44 | |
| pabelanger | dhinesh: maybe, but zuul should be triggering the jobs for you | 02:44 |
| *** gyee_ has quit IRC | 02:47 | |
| dhinesh | pabelanger: my layout.yaml is almost the same as this https://git.openstack.org/cgit/openstack-infra/project-config-example/tree/zuul/layout.yaml | 02:50 |
| dhinesh | pabelanger: except for the email ids info | 02:51 |
| *** rlandy|rover|bbl is now known as rlandy|rover | 02:51 | |
| ianw | dhinesh: when you do the stats to port 4730 ... do you get *anything* back at all? | 02:52 |
| dhinesh | pabelanger: yes i get back the current jobs on my jenkins but not noop-check-communication or dsvm-tempest-full | 02:52 |
| *** hongbin has joined #openstack-infra | 02:53 | |
| dhinesh | pabelanger: i mean the current builds | 02:53 |
| dhinesh | pabelanger: only manually configured jenkins builds are visible, not the one from zuul | 02:54 |
| pabelanger | dhinesh: and have you posted a patch to opentack-dev/ci-sandbox? | 02:56 |
| dhinesh | yes i have been posting https://review.openstack.org/#/c/517731/ by configuring through jenkins UI to trigger on gerrit streams | 03:01 |
| dhinesh | pabelanger: but not to ci-sandbox but just sandbox | 03:01 |
| pabelanger | dhinesh: try ci-sandbox, default is to configure to it: https://git.openstack.org/cgit/openstack-infra/project-config-example/tree/zuul/layout.yaml#n61 | 03:02 |
| *** sree has joined #openstack-infra | 03:02 | |
| dhinesh | pabelanger: ok i can give it a shot, but i am not sure if it will be different since i just use jenkins to push results to gerrit, i doubt if i am using zuul to do it | 03:05 |
| *** sree has quit IRC | 03:06 | |
| pabelanger | dhinesh: well, if you get ci-sandbox working, zuul should trigger the jobs, then you can work on adding your zuul to 3rd party CI for cinder | 03:08 |
| *** robynh has quit IRC | 03:09 | |
| dhinesh | pabelanger: hmm, so the CI is able to post https://review.openstack.org/#/c/519528/, but the only problem is zuul did not add noop-check-communication or dsvm-tempest-full to my jenkins | 03:20 |
| pabelanger | dhinesh: so, that likely means a gearman issue. Did you get any error in zuul logs | 03:21 |
| dhinesh | pabelanger: no errors of failures in debug.log or zuul.log | 03:25 |
| pabelanger | dhinesh: and you have nodepool nodes online? | 03:26 |
| pabelanger | dhinesh: what is the output of echo status | nc 127.0.0.1 4730 again | 03:27 |
| dhinesh | yeah, i do nodepool is running fine, once the job is done it deletes and recreates a new slave node | 03:27 |
| pabelanger | dhinesh: and gearman in jenkins is setup properly? | 03:27 |
| *** [HeOS] has quit IRC | 03:28 | |
| pabelanger | https://docs.openstack.org/infra/openstackci/third_party_ci.html#setup-jenkins | 03:28 |
| pabelanger | you did: Enable Gearman, which is the Jenkins plugin zuul uses to queue jobs: | 03:29 |
| *** hemna_ has quit IRC | 03:29 | |
| dhinesh | pabelanger: http://paste.openstack.org/show/626216/ i just replaced the server name | 03:29 |
| pabelanger | dhinesh: yah, I'd check jenkins settings again | 03:30 |
| *** HeOS has joined #openstack-infra | 03:30 | |
| dhinesh | yeah "Gearman Plugin Config " has ip and port and says success when i do "Test Connection" | 03:31 |
| ianw | pabelanger: did you end up seeing that message about the pypi.slave node in rax from the other day? | 03:32 |
| *** rlandy|rover has quit IRC | 03:32 | |
| pabelanger | ianw: oh, I did. Yes, I think we can delete that. I think I asked about that last week too | 03:32 |
| pabelanger | dhinesh: and both jobs exist in jenkins? | 03:32 |
| ianw | pabelanger: ok, i'll pop it in for a quick topic tomorrow and pending no comments we can delete it | 03:33 |
| pabelanger | ianw: wfm | 03:33 |
| dhinesh | pabelanger: yes they are visible, let me try to restart zuul service and see if that makes a differnece | 03:33 |
| pabelanger | dhinesh: I was going to suggest cycling your nodepool nodes to see if jobs register again | 03:34 |
| pabelanger | possible the labels are not setup properly some how | 03:34 |
| dhinesh | pabelanger: do you mean to delete the nodepool nodes? | 03:35 |
| pabelanger | dhinesh: yah, that will cause new nodes to build and hopefully re-register jobs in gearman | 03:36 |
| pabelanger | dhinesh: you can use nodepool cli command do do that | 03:36 |
| pabelanger | $ nodepool delete <id> | 03:36 |
| dhinesh | pabelanger: ok will give it a shot | 03:37 |
| *** coolsvap has joined #openstack-infra | 03:38 | |
| *** salv-orlando has joined #openstack-infra | 03:41 | |
| *** ramishra has joined #openstack-infra | 03:43 | |
| *** salv-orlando has quit IRC | 03:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create ansible-role-gearman project https://review.openstack.org/519529 | 03:46 |
| dhinesh | pabelanger: ok all nodes are new, no new jobs registered, does gearman have specific log? | 03:46 |
| *** gongysh has quit IRC | 03:46 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create ansible-role-gearman project https://review.openstack.org/519529 | 03:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add jobs for ansible-role-gearman https://review.openstack.org/519530 | 03:49 |
| pabelanger | dhinesh: yes, it should. You'll have to check zuul logging settings | 03:49 |
| pabelanger | dhinesh: might be worth looking at jenkins logs at this point too | 03:50 |
| pabelanger | I'm running out of ideas, and it has been some time since I've last used jenkins | 03:50 |
| *** yamamoto has joined #openstack-infra | 03:52 | |
| *** Apoorva has joined #openstack-infra | 03:56 | |
| dhinesh | pabelanger: nothing interesting on the logs, is zuul and zuul-merger the only 2 services that should be running? | 03:57 |
| *** ramishra_ has joined #openstack-infra | 04:01 | |
| pabelanger | dhinesh: yah, that sounds correct | 04:01 |
| pabelanger | I wonder if this is the jenkins security issue | 04:02 |
| pabelanger | where you need to whitelist things | 04:02 |
| pabelanger | https://jenkins.io/security/advisory/2016-05-11/ | 04:03 |
| pabelanger | but, at this point, just a guess | 04:03 |
| *** ramishra has quit IRC | 04:03 | |
| *** udesale has joined #openstack-infra | 04:04 | |
| dhinesh | pabelanger: ok, will give it a try | 04:06 |
| *** dhinesh has quit IRC | 04:11 | |
| *** kiennt26 has quit IRC | 04:13 | |
| *** hichihara has joined #openstack-infra | 04:14 | |
| *** rosmaita has quit IRC | 04:15 | |
| *** psachin has joined #openstack-infra | 04:18 | |
| *** ykarel has joined #openstack-infra | 04:18 | |
| *** dave-mccowan has quit IRC | 04:19 | |
| *** dbecker has quit IRC | 04:23 | |
| *** andreas_s has joined #openstack-infra | 04:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Remove some non-standard things from bindep-fallback https://review.openstack.org/519533 | 04:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Remove ruby from bindep-fallback.txt https://review.openstack.org/519534 | 04:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Don't install centos-release-openstack-ocata https://review.openstack.org/519535 | 04:28 |
| *** andreas_s has quit IRC | 04:29 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy i18n-tox-doc-docs job https://review.openstack.org/518678 | 04:34 |
| *** dbecker has joined #openstack-infra | 04:38 | |
| *** psachin has quit IRC | 04:38 | |
| *** ramishra_ has quit IRC | 04:38 | |
| *** ramishra has joined #openstack-infra | 04:39 | |
| *** salv-orlando has joined #openstack-infra | 04:41 | |
| *** salv-orlando has quit IRC | 04:46 | |
| *** psachin has joined #openstack-infra | 04:46 | |
| *** ramishra has quit IRC | 04:51 | |
| *** ramishra has joined #openstack-infra | 04:53 | |
| *** hongbin has quit IRC | 05:01 | |
| *** sree has joined #openstack-infra | 05:13 | |
| *** sree_ has joined #openstack-infra | 05:14 | |
| *** sree_ is now known as Guest23423 | 05:14 | |
| *** rkukura_ has joined #openstack-infra | 05:16 | |
| *** sree has quit IRC | 05:17 | |
| *** rkukura has quit IRC | 05:20 | |
| *** rkukura_ is now known as rkukura | 05:20 | |
| *** sbezverk has quit IRC | 05:20 | |
| *** cuongnv has quit IRC | 05:20 | |
| *** cuongnv has joined #openstack-infra | 05:23 | |
| *** Apoorva has quit IRC | 05:27 | |
| *** gildub has joined #openstack-infra | 05:28 | |
| *** yamahata has joined #openstack-infra | 05:31 | |
| *** 3NAABTTL6 has quit IRC | 05:32 | |
| *** benj_ has joined #openstack-infra | 05:33 | |
| *** janki has joined #openstack-infra | 05:34 | |
| *** caphrim007 has joined #openstack-infra | 05:40 | |
| *** salv-orlando has joined #openstack-infra | 05:42 | |
| *** erlon has quit IRC | 05:45 | |
| *** salv-orlando has quit IRC | 05:47 | |
| *** dhinesh has joined #openstack-infra | 05:54 | |
| *** mat128 has quit IRC | 05:59 | |
| *** dhinesh has quit IRC | 06:00 | |
| *** armaan has joined #openstack-infra | 06:02 | |
| *** armaan has quit IRC | 06:04 | |
| *** armaan has joined #openstack-infra | 06:04 | |
| *** janki has quit IRC | 06:05 | |
| *** janki has joined #openstack-infra | 06:20 | |
| *** hichihara has quit IRC | 06:25 | |
| *** Guest23423 has quit IRC | 06:28 | |
| openstackgerrit | zhufl proposed openstack-infra/elastic-recheck master: Remove "Initializing SSH connection to" message from query https://review.openstack.org/519546 | 06:29 |
| *** salv-orlando has joined #openstack-infra | 06:30 | |
| *** liujiong has joined #openstack-infra | 06:30 | |
| openstackgerrit | zhufl proposed openstack-infra/elastic-recheck master: Remove "Initializing SSH connection to" message from query https://review.openstack.org/519546 | 06:32 |
| *** armax has quit IRC | 06:50 | |
| *** florianf has joined #openstack-infra | 06:50 | |
| *** armax has joined #openstack-infra | 06:51 | |
| *** armax has quit IRC | 06:51 | |
| *** armax has joined #openstack-infra | 06:52 | |
| *** armax has quit IRC | 06:52 | |
| *** mtreinish has quit IRC | 06:55 | |
| *** mtreinish has joined #openstack-infra | 06:55 | |
| *** rcernin has quit IRC | 07:02 | |
| *** mat128 has joined #openstack-infra | 07:03 | |
| *** liujiong has quit IRC | 07:05 | |
| *** pgadiya has joined #openstack-infra | 07:08 | |
| *** andreas_s has joined #openstack-infra | 07:11 | |
| *** liujiong has joined #openstack-infra | 07:13 | |
| *** liujiong has quit IRC | 07:15 | |
| *** liujiong has joined #openstack-infra | 07:15 | |
| *** threestrands has quit IRC | 07:15 | |
| *** namnh has quit IRC | 07:16 | |
| *** cuongnv has quit IRC | 07:16 | |
| *** annp has quit IRC | 07:16 | |
| *** annp has joined #openstack-infra | 07:16 | |
| *** namnh has joined #openstack-infra | 07:16 | |
| *** cuongnv has joined #openstack-infra | 07:16 | |
| *** xinliang has quit IRC | 07:21 | |
| *** makowals has joined #openstack-infra | 07:29 | |
| *** liusheng has left #openstack-infra | 07:30 | |
| *** gildub has quit IRC | 07:32 | |
| *** xinliang has joined #openstack-infra | 07:33 | |
| *** xinliang has joined #openstack-infra | 07:33 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Zaqar https://review.openstack.org/510731 | 07:34 |
| *** salv-orlando has quit IRC | 07:36 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add building of openSUSE Tumbleweed nodepool images https://review.openstack.org/504169 | 07:36 |
| *** salv-orlando has joined #openstack-infra | 07:36 | |
| *** ykarel is now known as ykarel|lunch | 07:37 | |
| *** mat128 has quit IRC | 07:38 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Add heat-tempest-plugin project https://review.openstack.org/519567 | 07:39 |
| *** salv-orlando has quit IRC | 07:40 | |
| *** ccamacho has quit IRC | 07:46 | |
| *** ccamacho has joined #openstack-infra | 07:46 | |
| *** salv-orlando has joined #openstack-infra | 07:47 | |
| *** ccamacho has left #openstack-infra | 07:53 | |
| *** ccamacho has quit IRC | 07:53 | |
| *** alexchadin has joined #openstack-infra | 07:54 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add role to build Puppet module https://review.openstack.org/519489 | 07:56 |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Add heat-tempest-plugin project https://review.openstack.org/519567 | 08:06 |
| *** d0ugal has quit IRC | 08:06 | |
| *** ralonsoh has joined #openstack-infra | 08:09 | |
| *** yamamoto has quit IRC | 08:09 | |
| *** pcaruana has joined #openstack-infra | 08:10 | |
| *** e0ne has joined #openstack-infra | 08:10 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Add heat-tempest-plugin project https://review.openstack.org/519567 | 08:11 |
| *** yamamoto has joined #openstack-infra | 08:12 | |
| *** yamamoto has quit IRC | 08:12 | |
| *** tesseract has joined #openstack-infra | 08:15 | |
| *** Sukhdev has quit IRC | 08:16 | |
| *** links has quit IRC | 08:17 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Add heat-tempest-plugin project https://review.openstack.org/519567 | 08:20 |
| *** Hal has joined #openstack-infra | 08:20 | |
| chandankumar | ramishra: please change the topic https://review.openstack.org/#/c/519567/ to goal-split-tempest-plugins | 08:23 |
| chandankumar | and governance one also | 08:23 |
| *** salv-orlando has quit IRC | 08:24 | |
| *** shardy has joined #openstack-infra | 08:31 | |
| *** links has joined #openstack-infra | 08:31 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/diskimage-builder master: elements: Respect devpts mount options https://review.openstack.org/518749 | 08:32 |
| *** inc0 has quit IRC | 08:33 | |
| AJaeger | chandankumar: new-project would be the proper topic for new project creation | 08:34 |
| *** mat128 has joined #openstack-infra | 08:36 | |
| AJaeger | config-core, please review https://review.openstack.org/517217, the dependency has finally merged. | 08:40 |
| *** iyamahat has joined #openstack-infra | 08:40 | |
| *** yamamoto has joined #openstack-infra | 08:41 | |
| *** salv-orlando has joined #openstack-infra | 08:42 | |
| *** bhavik has joined #openstack-infra | 08:42 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Switch Puppet tarball job to use Zuul role https://review.openstack.org/519491 | 08:42 |
| *** ykarel|lunch is now known as ykarel | 08:43 | |
| *** yamahata has quit IRC | 08:44 | |
| openstackgerrit | Rui Chen proposed openstack-infra/nodepool feature/zuulv3: Fix nodepool cmd TypeError when no arguemnts https://review.openstack.org/519582 | 08:47 |
| *** amoralej|off is now known as amoralej | 08:50 | |
| *** yamamoto has quit IRC | 08:50 | |
| *** dingyichen has quit IRC | 08:50 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config master: networking-hyperv: Adds neutron required project https://review.openstack.org/519435 | 08:51 |
| *** iyamahat has quit IRC | 08:51 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Avoid running Puppet module build job for no reason https://review.openstack.org/519494 | 08:51 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add Puppet build jobs to check and gate pipelines https://review.openstack.org/519495 | 08:51 |
| *** jpena|off is now known as jpena | 08:52 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config master: networking-hyperv: Adds neutron required project https://review.openstack.org/519435 | 08:52 |
| *** priteau has joined #openstack-infra | 08:54 | |
| openstackgerrit | Rui Chen proposed openstack-infra/nodepool feature/zuulv3: Fix nodepool cmd TypeError when no arguemnts https://review.openstack.org/519582 | 08:57 |
| *** jpich has joined #openstack-infra | 08:57 | |
| *** Wei_Liu has quit IRC | 09:03 | |
| *** Wei_Liu has joined #openstack-infra | 09:03 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config master: networking-hyperv: Uses release-notes-jobs-neutron https://review.openstack.org/519435 | 09:05 |
| *** links has quit IRC | 09:07 | |
| *** mat128 has quit IRC | 09:08 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Upgrade UC and OC using tripleo-upgrade role https://review.openstack.org/515643 | 09:08 |
| *** liujiong_lj has joined #openstack-infra | 09:19 | |
| *** ccamacho has joined #openstack-infra | 09:20 | |
| *** salv-orlando has quit IRC | 09:20 | |
| *** liujiong has quit IRC | 09:20 | |
| *** markvoelker has quit IRC | 09:21 | |
| *** links has joined #openstack-infra | 09:22 | |
| *** salv-orlando has joined #openstack-infra | 09:22 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Zaqar legacy jobs https://review.openstack.org/510732 | 09:23 |
| *** bhavik has quit IRC | 09:23 | |
| *** jianghuaw has quit IRC | 09:26 | |
| *** efoley has joined #openstack-infra | 09:26 | |
| *** salv-orlando has quit IRC | 09:26 | |
| *** tosky has joined #openstack-infra | 09:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove horizon legacy jobs (django variants, selenium-headless) https://review.openstack.org/519047 | 09:29 |
| *** slaweq has joined #openstack-infra | 09:31 | |
| *** rbergeron has quit IRC | 09:34 | |
| *** efoley has quit IRC | 09:34 | |
| *** efoley_ has joined #openstack-infra | 09:34 | |
| *** rbergeron has joined #openstack-infra | 09:34 | |
| *** derekh has joined #openstack-infra | 09:39 | |
| *** hrubi has quit IRC | 09:39 | |
| *** lucas-afk is now known as lucasagomes | 09:39 | |
| *** hrubi has joined #openstack-infra | 09:39 | |
| *** andreas__ has joined #openstack-infra | 09:41 | |
| *** hrubi has quit IRC | 09:41 | |
| *** hrubi has joined #openstack-infra | 09:41 | |
| *** hrubi has quit IRC | 09:42 | |
| *** hrubi has joined #openstack-infra | 09:43 | |
| *** andreas_s has quit IRC | 09:45 | |
| *** rfolco has joined #openstack-infra | 09:45 | |
| *** d0ugal has joined #openstack-infra | 09:45 | |
| *** d0ugal_ has joined #openstack-infra | 09:51 | |
| *** d0ugal has quit IRC | 09:52 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Allow run to be list of playbooks https://review.openstack.org/519596 | 09:53 |
| *** zaneb has quit IRC | 09:54 | |
| *** jianghuaw has joined #openstack-infra | 09:55 | |
| *** cshastri has quit IRC | 09:55 | |
| *** efoley_ has quit IRC | 09:57 | |
| *** jianghuaw has quit IRC | 09:57 | |
| *** jianghuaw has joined #openstack-infra | 09:58 | |
| *** yamamoto has joined #openstack-infra | 10:01 | |
| *** makowals has quit IRC | 10:01 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove barbican-tempest-plugins legacy jobs https://review.openstack.org/517217 | 10:01 |
| *** d0ugal_ is now known as dmatthew | 10:03 | |
| *** dmatthew has quit IRC | 10:03 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Upgrade UC and OC using tripleo-upgrade role https://review.openstack.org/515643 | 10:03 |
| *** makowals has joined #openstack-infra | 10:03 | |
| *** d0ugal has joined #openstack-infra | 10:03 | |
| *** namnh has quit IRC | 10:04 | |
| *** annp has quit IRC | 10:04 | |
| ianw | urgh ... http://logs.openstack.org/70/519570/1/check/legacy-tempest-dsvm-py35/83bcb0e/logs/devstacklog.txt.gz#_2017-11-14_08_13_56_850 | 10:05 |
| ianw | venv-tempest runtests: commands[0] | pip install -c ../requirements/upper-constraints.txt -r requirements.txt | 10:05 |
| ianw | ends up with | 10:06 |
| ianw | UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 78: ordinal not in range(128) | 10:06 |
| ianw | if nobody figures it out, i'll look more in my morning :/ | 10:06 |
| *** sshnaidm is now known as sshnaidm|afk | 10:06 | |
| *** mat128 has joined #openstack-infra | 10:07 | |
| *** gildub has joined #openstack-infra | 10:07 | |
| *** salv-orlando has joined #openstack-infra | 10:08 | |
| *** liujiong_lj has quit IRC | 10:10 | |
| *** salv-orlando has quit IRC | 10:11 | |
| *** salv-orlando has joined #openstack-infra | 10:11 | |
| *** felipemonteiro has joined #openstack-infra | 10:16 | |
| *** markvoelker has joined #openstack-infra | 10:21 | |
| *** HeOS has quit IRC | 10:26 | |
| *** HeOS has joined #openstack-infra | 10:27 | |
| *** hrubi has quit IRC | 10:30 | |
| *** hrubi has joined #openstack-infra | 10:30 | |
| *** ykarel is now known as ykarel|afk | 10:31 | |
| *** sdague has joined #openstack-infra | 10:32 | |
| fungi | infra-root: i'm away (appointments on the mainland) most of today and not around a computer. i _may_ be back for at least part of the infra meeting, but there's a good chance i'll end up missing it | 10:33 |
| *** efoley has joined #openstack-infra | 10:33 | |
| *** jascott1 has quit IRC | 10:35 | |
| *** jascott1 has joined #openstack-infra | 10:36 | |
| *** yamamoto has quit IRC | 10:36 | |
| *** yamamoto has joined #openstack-infra | 10:36 | |
| *** mat128 has quit IRC | 10:39 | |
| *** claudiub has joined #openstack-infra | 10:39 | |
| *** jascott1 has quit IRC | 10:40 | |
| openstackgerrit | Luka Peschke proposed openstack-infra/project-config master: Add pep8 jobs for cloudkitty-tempest-plugin https://review.openstack.org/519607 | 10:42 |
| *** alexchadin has quit IRC | 10:42 | |
| *** alexchadin has joined #openstack-infra | 10:42 | |
| *** pblaho has quit IRC | 10:44 | |
| *** efoley_ has joined #openstack-infra | 10:45 | |
| *** efoley has quit IRC | 10:45 | |
| *** d0ugal has quit IRC | 10:46 | |
| *** andreas__ has quit IRC | 10:53 | |
| *** jianghuaw has quit IRC | 10:56 | |
| *** stakeda has quit IRC | 10:58 | |
| *** andreas_s has joined #openstack-infra | 10:58 | |
| *** udesale has quit IRC | 11:00 | |
| *** caphrim007 has quit IRC | 11:01 | |
| *** andreas_s has quit IRC | 11:02 | |
| *** alexchadin has quit IRC | 11:02 | |
| *** wolverineav has joined #openstack-infra | 11:02 | |
| *** andreas_s has joined #openstack-infra | 11:03 | |
| *** pblaho has joined #openstack-infra | 11:04 | |
| *** cuongnv has quit IRC | 11:04 | |
| *** ldnunes has joined #openstack-infra | 11:04 | |
| *** andreas_s has quit IRC | 11:07 | |
| *** andreas_s has joined #openstack-infra | 11:08 | |
| *** gouthamr has joined #openstack-infra | 11:10 | |
| *** alexchadin has joined #openstack-infra | 11:10 | |
| *** andreas_s has quit IRC | 11:12 | |
| *** andreas_s has joined #openstack-infra | 11:13 | |
| *** d0ugal has joined #openstack-infra | 11:14 | |
| *** caphrim007 has joined #openstack-infra | 11:16 | |
| *** salv-orlando has quit IRC | 11:19 | |
| *** salv-orlando has joined #openstack-infra | 11:19 | |
| *** dtantsur|afk is now known as dtantsur | 11:20 | |
| *** andreas_s has quit IRC | 11:22 | |
| *** andreas_s has joined #openstack-infra | 11:27 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/zuul-jobs master: Fix npm-run-test https://review.openstack.org/518879 | 11:27 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/zuul-jobs master: Fix npm-run-test https://review.openstack.org/518879 | 11:31 |
| *** andreas_s has quit IRC | 11:32 | |
| *** efoley_ has quit IRC | 11:33 | |
| *** Wei_Liu has quit IRC | 11:33 | |
| *** pblaho has quit IRC | 11:33 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:34 | |
| *** andreas_s has joined #openstack-infra | 11:35 | |
| *** mat128 has joined #openstack-infra | 11:38 | |
| *** felipemonteiro has quit IRC | 11:38 | |
| *** andreas_s has quit IRC | 11:43 | |
| *** andreas_s has joined #openstack-infra | 11:44 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/openstack-zuul-jobs master: Remove horizon legacy jobs (django variants, selenium-headless) https://review.openstack.org/519047 | 11:45 |
| *** Wei_Liu has joined #openstack-infra | 11:51 | |
| *** d0ugal has quit IRC | 11:55 | |
| flaper87 | what's the best way to know what went wrong here? http://logs.openstack.org/02/519602/2/check/ansible-role-k8s-keystone-kubernetes-centos/db2a917/job-output.txt.gz#_2017-11-14_10_27_45_222368 | 11:57 |
| flaper87 | it seems to have failed loading the run.yml file | 11:57 |
| flaper87 | s/file/playbook/ | 11:57 |
| flaper87 | but I can't figure out where I should check to know what failed exactly | 11:57 |
| flaper87 | also, I was expecting it to run the run.yml file from the parent job first. Did I misunderstood something? | 11:58 |
| *** ykarel|afk is now known as ykarel | 11:58 | |
| flaper87 | jeblair: mordred dmsimard|off ^ | 11:58 |
| *** Wei_Liu has quit IRC | 11:58 | |
| *** alexchadin has quit IRC | 12:02 | |
| *** alexchadin has joined #openstack-infra | 12:03 | |
| *** tobiash has quit IRC | 12:04 | |
| *** pbourke has quit IRC | 12:05 | |
| *** askb has quit IRC | 12:05 | |
| *** tosky_ has joined #openstack-infra | 12:05 | |
| *** nsingh has quit IRC | 12:05 | |
| *** pahuang has quit IRC | 12:05 | |
| *** gouthamr has quit IRC | 12:05 | |
| *** gildub has quit IRC | 12:06 | |
| *** tobiash has joined #openstack-infra | 12:06 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added zuulv3 based inherited jobs for neutron tempest plugin https://review.openstack.org/519627 | 12:06 |
| *** pahuang has joined #openstack-infra | 12:06 | |
| *** lucasagomes is now known as lucas-hungry | 12:06 | |
| *** niraj_singh has joined #openstack-infra | 12:06 | |
| *** askb has joined #openstack-infra | 12:06 | |
| *** tobiash has quit IRC | 12:06 | |
| *** tosky has quit IRC | 12:06 | |
| chandankumar | AJaeger: is it correct in order to reuse the same job against neutron tempest plugin? | 12:06 |
| *** tobiash has joined #openstack-infra | 12:07 | |
| *** gildub has joined #openstack-infra | 12:07 | |
| *** pbourke has joined #openstack-infra | 12:07 | |
| *** udesale has joined #openstack-infra | 12:07 | |
| *** ramishra has quit IRC | 12:08 | |
| AJaeger | chandankumar: where are those jobs defined? | 12:09 |
| *** mat128 has quit IRC | 12:09 | |
| *** shardy is now known as shardy_lunch | 12:09 | |
| chandankumar | AJaeger: https://github.com/openstack/neutron/blob/master/.zuul.yaml | 12:09 |
| AJaeger | chandankumar: then I suggest to add a .zuul.yaml to neutron-tempest plugin repo | 12:10 |
| *** pblaho has joined #openstack-infra | 12:10 | |
| AJaeger | flaper87: I suggest to check the json file, it sometimes contains more information | 12:10 |
| *** smatzek has joined #openstack-infra | 12:10 | |
| chandankumar | AJaeger: with all playbooks ? is there a way to reuse it in another project? | 12:11 |
| *** d0ugal has joined #openstack-infra | 12:11 | |
| AJaeger | chandankumar: WE have a global name space, just use it. There | 12:13 |
| AJaeger | There's no difference adding them in-tree or in project-config | 12:13 |
| AJaeger | chandankumar: the whole idea with zuul v3 is that you define them in one place - conveninent for neutron - and then use them in all repos... | 12:14 |
| chandankumar | AJaeger: got it , thanks :-) | 12:14 |
| AJaeger | chandankumar: happy to review your change for neutron tempest plugin, just tell me here if you have questions | 12:14 |
| *** gouthamr has joined #openstack-infra | 12:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add pep8 jobs for cloudkitty-tempest-plugin https://review.openstack.org/519607 | 12:22 |
| *** tosky_ is now known as tosky | 12:23 | |
| mordred | flaper87: only one job's run playbook gets run- so if a child job defines a run, that overrides the parent rather than supplementing | 12:23 |
| frickler | flaper87: iiuc you need to have "run: tests/run.yml" now, i.e. use the full filename | 12:23 |
| *** sbezverk has joined #openstack-infra | 12:24 | |
| mordred | flaper87: yah - what frickler said | 12:24 |
| frickler | though I agree it would be nice if zuul did mention that the specified playbook doesn't exist | 12:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs from Zaqar https://review.openstack.org/510731 | 12:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-qinling-dsvm-tempest jobs https://review.openstack.org/519271 | 12:26 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove migrated legacy python-senlinclient jobs https://review.openstack.org/519211 | 12:28 |
| AJaeger | frickler: I think it complains about missing file if you use "run.yaml" - but not if extension is missing... | 12:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add ability to work in other auth contexts https://review.openstack.org/519394 | 12:31 |
| *** pgadiya has quit IRC | 12:35 | |
| frickler | AJaeger: hmm, the file "tests/run" doesn't exist, so if that (and only that) is what zuul is looking for, then it should complain just like if would for "tests/doesntexist.yaml", shouldn't it? | 12:40 |
| frickler | different issue: should we complain when projects install non-voting jobs into the gate queue in their local zuul config? see https://review.openstack.org/516934 | 12:40 |
| *** ricardoas has quit IRC | 12:41 | |
| *** jkilpatr has joined #openstack-infra | 12:42 | |
| *** rcernin has joined #openstack-infra | 12:44 | |
| *** niedbalski has quit IRC | 12:46 | |
| *** niedbalski has joined #openstack-infra | 12:46 | |
| *** zhurong has joined #openstack-infra | 12:46 | |
| flaper87 | frickler: mordred roger that, thanks | 12:47 |
| *** gildub has quit IRC | 12:52 | |
| *** gouthamr has quit IRC | 12:53 | |
| *** lucas-hungry is now known as lucasagomes | 12:55 | |
| *** links has quit IRC | 12:56 | |
| mordred | frickler: yah - at least I think a friendly reminder to people that it's a waste of resources | 12:56 |
| AJaeger | frickler: I give a -1 if I find those as reviews - and sometimes send direct patches. So, yes, complain loudly, please | 12:57 |
| AJaeger | mordred, frickler: We need to do some more education on how to do jobs - like non-voting only in check, voting in check and gate. | 12:58 |
| *** tosky has quit IRC | 12:58 | |
| frickler | infra-root: I've seen a couple of jobs timeouts today which all seem to be on inap-mtl01, latest is http://logs.openstack.org/85/519585/1/check/legacy-tempest-dsvm-neutron-full/325b9db/job-output.txt.gz#_2017-11-14_12_01_48_313575 . they all fail to collect the logs directory, making further analysis difficult | 12:58 |
| AJaeger | chandankumar: https://review.openstack.org/#/c/516934/5/.zuul.yaml your api job is non-voting, please remove it from gate pipeline, it's just wasting resources | 12:59 |
| *** tosky has joined #openstack-infra | 12:59 | |
| *** rosmaita has joined #openstack-infra | 13:01 | |
| *** gouthamr has joined #openstack-infra | 13:02 | |
| *** efoley_ has joined #openstack-infra | 13:05 | |
| *** efoley has joined #openstack-infra | 13:07 | |
| *** efoley_ has quit IRC | 13:07 | |
| *** dprince has joined #openstack-infra | 13:07 | |
| *** Douhet has quit IRC | 13:07 | |
| *** mat128 has joined #openstack-infra | 13:07 | |
| *** andreas_s has quit IRC | 13:08 | |
| *** Douhet has joined #openstack-infra | 13:10 | |
| *** zeih has joined #openstack-infra | 13:10 | |
| *** jpena is now known as jpena|lunch | 13:11 | |
| chandankumar | AJaeger: https://review.openstack.org/#/c/519646/ | 13:12 |
| *** trown|outtypewww is now known as trown | 13:12 | |
| *** jaypipes has joined #openstack-infra | 13:12 | |
| *** zeih has quit IRC | 13:13 | |
| *** andreas_s has joined #openstack-infra | 13:13 | |
| *** dave-mccowan has joined #openstack-infra | 13:15 | |
| *** amoralej is now known as amoralej|lunch | 13:16 | |
| *** LindaWang has joined #openstack-infra | 13:17 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/zuul-jobs master: Fix npm-run-test https://review.openstack.org/518879 | 13:19 |
| AJaeger | chandankumar: should work, one suggestion inline. Did you see my comment above regarding neutron? ^ | 13:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add helper property to get the current user id https://review.openstack.org/519511 | 13:19 |
| *** edmondsw has joined #openstack-infra | 13:22 | |
| *** erlon has joined #openstack-infra | 13:25 | |
| *** shardy_lunch is now known as shardy | 13:25 | |
| *** zhurong has quit IRC | 13:26 | |
| *** ykarel has quit IRC | 13:26 | |
| *** jbadiapa_ has joined #openstack-infra | 13:27 | |
| *** andreas_s has quit IRC | 13:27 | |
| *** jbadiapa has quit IRC | 13:27 | |
| *** rlandy has joined #openstack-infra | 13:28 | |
| *** andreas_s has joined #openstack-infra | 13:28 | |
| *** rlandy is now known as rlandy|rover | 13:29 | |
| *** ralonsoh has quit IRC | 13:32 | |
| openstackgerrit | Tytus Kurek proposed openstack-infra/project-config master: Add charm-interface-designate project https://review.openstack.org/517336 | 13:37 |
| *** gouthamr has quit IRC | 13:38 | |
| *** ramishra has joined #openstack-infra | 13:38 | |
| *** armaan has quit IRC | 13:38 | |
| *** links has joined #openstack-infra | 13:38 | |
| *** mat128 has quit IRC | 13:39 | |
| *** armaan has joined #openstack-infra | 13:39 | |
| *** e0ne_ has joined #openstack-infra | 13:40 | |
| *** salv-orlando has quit IRC | 13:40 | |
| *** salv-orlando has joined #openstack-infra | 13:41 | |
| *** e0ne has quit IRC | 13:43 | |
| *** andreas_s has quit IRC | 13:46 | |
| *** andreas_s has joined #openstack-infra | 13:48 | |
| *** Wei_Liu has joined #openstack-infra | 13:51 | |
| *** edmondsw_ has joined #openstack-infra | 13:53 | |
| *** esberglu has joined #openstack-infra | 13:54 | |
| *** coolsvap has quit IRC | 13:54 | |
| *** gouthamr has joined #openstack-infra | 13:54 | |
| *** edmondsw has quit IRC | 13:55 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Override HOME environment variable in bubblewrap https://review.openstack.org/519654 | 13:57 |
| *** szaher has quit IRC | 13:58 | |
| *** hashar has joined #openstack-infra | 13:58 | |
| *** bh526r has joined #openstack-infra | 13:59 | |
| *** edmondsw_ is now known as edmondsw | 13:59 | |
| *** edmondsw has quit IRC | 13:59 | |
| *** mriedem has joined #openstack-infra | 13:59 | |
| *** hashar has quit IRC | 14:00 | |
| *** [HeOS] has joined #openstack-infra | 14:00 | |
| *** andreas_s has quit IRC | 14:01 | |
| *** HeOS has quit IRC | 14:01 | |
| *** d0ugal_ has joined #openstack-infra | 14:01 | |
| *** szaher has joined #openstack-infra | 14:02 | |
| *** jianghuaw_ has joined #openstack-infra | 14:03 | |
| *** d0ugal has quit IRC | 14:03 | |
| *** d0ugal_ is now known as d0ugal | 14:04 | |
| *** d0ugal has quit IRC | 14:04 | |
| *** d0ugal has joined #openstack-infra | 14:04 | |
| *** d0ugal has quit IRC | 14:04 | |
| *** d0ugal has joined #openstack-infra | 14:04 | |
| *** andreas_s has joined #openstack-infra | 14:05 | |
| *** dmsimard|off is now known as dmsimard | 14:05 | |
| *** kgiusti has joined #openstack-infra | 14:07 | |
| *** Goneri has joined #openstack-infra | 14:07 | |
| *** jpena|lunch is now known as jpena | 14:08 | |
| *** alexchadin has quit IRC | 14:10 | |
| *** edmondsw has joined #openstack-infra | 14:12 | |
| *** efried has quit IRC | 14:14 | |
| *** mat128 has joined #openstack-infra | 14:16 | |
| *** bobh has joined #openstack-infra | 14:16 | |
| *** LindaWang has quit IRC | 14:20 | |
| *** LindaWang has joined #openstack-infra | 14:21 | |
| *** armaan has quit IRC | 14:23 | |
| *** armaan has joined #openstack-infra | 14:23 | |
| *** hamzy_ has quit IRC | 14:24 | |
| *** ralonsoh has joined #openstack-infra | 14:24 | |
| *** smatzek has quit IRC | 14:25 | |
| *** amoralej|lunch is now known as amoralej | 14:25 | |
| *** camunoz has joined #openstack-infra | 14:25 | |
| *** efried has joined #openstack-infra | 14:27 | |
| *** jascott1 has joined #openstack-infra | 14:37 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Save oooq generated files https://review.openstack.org/517136 | 14:39 |
| *** jascott1 has quit IRC | 14:42 | |
| *** markvoelker has quit IRC | 14:43 | |
| openstackgerrit | Merged openstack/os-client-config master: Add shade-functional-tips jobs https://review.openstack.org/514067 | 14:43 |
| *** markvoelker has joined #openstack-infra | 14:43 | |
| *** Nil_ has joined #openstack-infra | 14:44 | |
| *** dhill_ has quit IRC | 14:46 | |
| *** dhill_ has joined #openstack-infra | 14:51 | |
| *** smatzek has joined #openstack-infra | 14:53 | |
| *** smatzek has quit IRC | 14:53 | |
| *** yamahata has joined #openstack-infra | 14:53 | |
| *** smatzek has joined #openstack-infra | 14:53 | |
| *** jkilpatr has quit IRC | 14:53 | |
| *** ramishra has quit IRC | 14:54 | |
| *** iyamahat has joined #openstack-infra | 14:55 | |
| *** ramishra has joined #openstack-infra | 14:57 | |
| *** rcernin has quit IRC | 15:00 | |
| *** ykarel has joined #openstack-infra | 15:07 | |
| *** iyamahat has quit IRC | 15:08 | |
| *** eharney has joined #openstack-infra | 15:08 | |
| *** rbrndt has joined #openstack-infra | 15:12 | |
| *** dhill_ has quit IRC | 15:13 | |
| *** gouthamr has quit IRC | 15:13 | |
| *** dhill_ has joined #openstack-infra | 15:13 | |
| *** gouthamr has joined #openstack-infra | 15:16 | |
| *** psachin has quit IRC | 15:16 | |
| *** psachin has joined #openstack-infra | 15:16 | |
| *** armax has joined #openstack-infra | 15:19 | |
| *** dhill_ has quit IRC | 15:19 | |
| *** hongbin has joined #openstack-infra | 15:20 | |
| *** dhill_ has joined #openstack-infra | 15:20 | |
| *** armaan has quit IRC | 15:24 | |
| *** armaan has joined #openstack-infra | 15:24 | |
| *** hrubi has quit IRC | 15:25 | |
| *** hrubi_ has joined #openstack-infra | 15:25 | |
| *** [HeOS] is now known as HeOS | 15:28 | |
| *** ramishra has quit IRC | 15:29 | |
| *** ramishra has joined #openstack-infra | 15:29 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Upgrade UC and OC using tripleo-upgrade role https://review.openstack.org/515643 | 15:30 |
| *** dhill_ has quit IRC | 15:30 | |
| *** efoley has quit IRC | 15:32 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change the name of openstack-helm chair This PS changes the name of meeting chair to mattmceuen. mattmceuen is the current chair and PTL of openstack-helm. https://review.openstack.org/518831 | 15:35 |
| *** ramishra has quit IRC | 15:35 | |
| *** dhill_ has joined #openstack-infra | 15:35 | |
| *** chandankumar is now known as chkumar|upgrade | 15:37 | |
| *** janki has quit IRC | 15:39 | |
| *** xarses_ has joined #openstack-infra | 15:42 | |
| *** dtantsur is now known as dtantsur|brb | 15:46 | |
| *** dsariel has joined #openstack-infra | 15:47 | |
| *** caphrim007 has quit IRC | 15:48 | |
| *** caphrim007 has joined #openstack-infra | 15:49 | |
| *** e0ne_ has quit IRC | 15:51 | |
| *** e0ne has joined #openstack-infra | 15:52 | |
| *** caphrim007 has quit IRC | 15:53 | |
| *** jkilpatr has joined #openstack-infra | 15:58 | |
| *** hashar has joined #openstack-infra | 15:59 | |
| *** janki has joined #openstack-infra | 15:59 | |
| *** links has quit IRC | 16:00 | |
| *** ykarel has quit IRC | 16:02 | |
| *** salv-orlando has quit IRC | 16:09 | |
| clarkb | frickler: looks like ansible claims it cannot ssh to execute the rsync? | 16:09 |
| clarkb | I wonder if we are running out of fds? | 16:10 |
| *** chkumar|upgrade is now known as chandankumar | 16:10 | |
| clarkb | AJaeger: is the release notes job final in an attempt to avoid abuse of the afs publishing? | 16:14 |
| AJaeger | clarkb: it contains secrets, so that's why we made it final AFAIR. But better ask mordred (since jeblair is out). | 16:15 |
| AJaeger | clarkb: seeing the trove email now, we need to review all of these ;( | 16:16 |
| clarkb | we may want to add a comment with a big NOTE that says "this job is final, that means you cannot modify it in your projects. You cannot add required projects or branch restrictions" etc to make it clear what that means to everyone while we learn the new job configs | 16:17 |
| clarkb | I think the problem in detecting the failures early (pre merge) is that zuul won't apply those changes beyond a syntax check if they modify things on a post merge pipeline | 16:18 |
| clarkb | not sure if we can easily check final state during syntax checking (probably not) but that may be one way to catch this upfront | 16:19 |
| AJaeger | clarkb: and reviewer eductation - I didn't had that in mind when reviewing | 16:20 |
| clarkb | ya | 16:20 |
| *** LindaWang has quit IRC | 16:21 | |
| *** jkilpatr has quit IRC | 16:21 | |
| AJaeger | clarkb: question is how to handle the releasenotes jobs: Have specific jobs for releasenotes generation with all required repos, or ask them to fix their set up. We could remove all required-projects from releasenotes... | 16:21 |
| AJaeger | clarkb: sorry, no time to dig deeper into these this week | 16:21 |
| clarkb | AJaeger: considering dhellmann says you shouldn't need any other projects to generate release notes I'm inclined to push back on the projects and have them fix release note generation? | 16:22 |
| clarkb | seems liek that is the most correct way | 16:22 |
| AJaeger | clarkb: yes - that means they cannot use their custom install_command and need to hardwire dependencies separately. So, a change to tox.ini for them | 16:23 |
| AJaeger | but we should review whether other final jobs are in the same league | 16:24 |
| *** Hal has quit IRC | 16:25 | |
| dhellmann | perhaps the CTI for the release notes job needs to change to not use tox | 16:26 |
| AJaeger | clarkb: I wonder also about http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n54 (publish-openstack-sphinx-docs), http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n196 (release-openstack-python - this one seems unneeded since release-openstack-python already install horizon) | 16:26 |
| dhellmann | it could set up a virtualenv with pre-defined packages and run sphinx directly | 16:26 |
| frickler | clarkb: the devstack run before already takes 2.5h until it is killed by timeout, would need to look into devstacklog.txt to see more. I'm guessing it would be some networking issue on the node | 16:27 |
| clarkb | frickler: ya that wouldn't surprise me | 16:28 |
| clarkb | frickler: basically the instance was unhappy all along and its not until that point in the job where we notice in a fatal way | 16:28 |
| *** andreas_s has quit IRC | 16:29 | |
| *** jkilpatr has joined #openstack-infra | 16:29 | |
| *** salv-orlando has joined #openstack-infra | 16:31 | |
| clarkb | dhellmann: ya `virtualenv relenv && relenv/bin/pip install reno && reno generate notes` sort of thing | 16:32 |
| frickler | clarkb: yeah, the question would rather be what we do when we confirm that this seems to have an increased rate of occurrences at inap-mtl01, maybe ask them for known issues or reduce the quota we use there | 16:32 |
| clarkb | frickler: yup, mgagne may be able to help with that | 16:32 |
| *** e0ne has quit IRC | 16:33 | |
| *** andreas_s has joined #openstack-infra | 16:33 | |
| smcginnis | We got a POST_FAILURE on a trove release: http://lists.openstack.org/pipermail/release-job-failures/2017-November/000663.html | 16:36 |
| smcginnis | Logs look fine, so was it something after the job run? | 16:37 |
| smcginnis | And tagging the trove-dashboard release failed too: http://lists.openstack.org/pipermail/release-job-failures/2017-November/000662.html | 16:38 |
| *** andreas_s has quit IRC | 16:38 | |
| *** gyee_ has joined #openstack-infra | 16:38 | |
| frickler | infra-root: can someone take a look at static.o.o? getting very slow responses for logs (that is the same server, isn't it?) | 16:40 |
| *** hemna_ has joined #openstack-infra | 16:41 | |
| clarkb | frickler: does cacti show anything? | 16:41 |
| clarkb | and yes same server | 16:42 |
| *** psachin has quit IRC | 16:42 | |
| clarkb | smcginnis: trove-dashboard is the thing on the mailing list that I responded to dhellmann about right? | 16:43 |
| clarkb | ya | 16:43 |
| smcginnis | clarkb: Oh sorry, missed it. | 16:43 |
| smcginnis | Still catching up and having hit the ML yet. | 16:43 |
| dims | jeblair : clarkb : fungi : trying to crystallize the LTS discussion with specific proposals here - https://etherpad.openstack.org/p/LTS-proposal | 16:44 |
| clarkb | frickler: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all there is a spike in outbound traffic there | 16:44 |
| clarkb | frickler: the disk usage/iops doesn't look too bad though so wondering if it was just lots of things asking for data, logs.o.o seems to be responsive to me now | 16:45 |
| clarkb | load average and cpu look fine too | 16:45 |
| clarkb | as does memory/swap | 16:45 |
| clarkb | smcginnis: for your other thing http://logs.openstack.org/17/175bd53e7b18a9dc4d42e60fe9225a5748eded34/pre-release/release-openstack-python-without-pypi/7a9474c/job-output.txt.gz#_2017-11-14_16_22_57_628445 appears to be the logged error which is unfortunately not very verbose. Lets see if ara knows more | 16:46 |
| *** jkilpatr has quit IRC | 16:46 | |
| clarkb | frickler: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1871&rra_id=all seems suspicious though | 16:48 |
| *** pcaruana has quit IRC | 16:48 | |
| frickler | clarkb: there's a peak in tcp open conns + resets, still seeing very slow response here | 16:49 |
| clarkb | ya it went back to being slow for me trying to open ara for smcginnis' failure above | 16:49 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=140&rra_id=all and ya that looks unhappy. | 16:51 |
| * clarkb gets on host to see what is making all the connections | 16:51 | |
| clarkb | frickler: I don't see significant numbers of connections from a single source, I think that spike in connections is likely a side effect of whatever is slowing it down | 16:54 |
| *** mrunge has quit IRC | 16:54 | |
| clarkb | smcginnis: http://paste.openstack.org/show/626285/ from ara | 16:54 |
| clarkb | smcginnis: it is trying to set permissions on that file and failing because it is not allowed to if I am reading that correctly | 16:55 |
| smcginnis | clarkb: That's the impression I got, but really not clear. :) | 16:55 |
| smcginnis | clarkb: What would be trying to change the permissions? | 16:56 |
| clarkb | smcginnis: ansible is trying to chagne the permissions, I do not know why but it seems to think it should | 16:56 |
| clarkb | it is in the file module which is a module that does things like update perms | 16:56 |
| clarkb | oh I know | 16:57 |
| clarkb | I think we may do a +r to everything because otherwise the subsequent rsync will fail to read the files? | 16:57 |
| clarkb | it is possible that particular task wants to be become: yes so that it executes as root | 16:57 |
| frickler | clarkb: did you check for device errors? this looks like xvdo might actually be broken http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1869&rra_id=all | 16:58 |
| clarkb | frickler: xvdo doesn't show up in syslog and nothing is in dmesg. `pvs` output looks happy | 17:00 |
| clarkb | frickler: it is possible that lvm just isn't writing to that disk right now because it is logically full? | 17:01 |
| *** slaweq has quit IRC | 17:02 | |
| frickler | clarkb: ya, possible, just the coincidence of going from writing pretty much to next to nothing in the last two hours seemed suspicous to me | 17:03 |
| frickler | clarkb: but in fact it might have mostly moved to xvdp | 17:05 |
| clarkb | I'm not seeing anything immediately broken but do notice that a kworker is occasionally eating a bunch of cpu (so possible io contention?) | 17:05 |
| clarkb | I've got to go take care of the kids' breakfast now but will continue to look at it when I return | 17:05 |
| *** rhallisey has quit IRC | 17:05 | |
| *** caphrim007 has joined #openstack-infra | 17:05 | |
| *** armaan has quit IRC | 17:06 | |
| *** armaan has joined #openstack-infra | 17:06 | |
| *** armaan has quit IRC | 17:06 | |
| *** bh526r has quit IRC | 17:07 | |
| *** eharney has quit IRC | 17:10 | |
| *** janki has quit IRC | 17:10 | |
| *** eharney has joined #openstack-infra | 17:11 | |
| *** udesale has quit IRC | 17:11 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add username to build and upload information https://review.openstack.org/453968 | 17:12 |
| *** clarkb has quit IRC | 17:14 | |
| *** yamamoto has quit IRC | 17:15 | |
| *** yamahata has quit IRC | 17:15 | |
| *** hashar is now known as hasharAway | 17:17 | |
| *** efoley has joined #openstack-infra | 17:18 | |
| *** rlandy|rover is now known as rlandy|rover|brb | 17:19 | |
| *** lucasagomes is now known as lucas-afk | 17:19 | |
| *** salv-orl_ has joined #openstack-infra | 17:20 | |
| *** florianf has quit IRC | 17:20 | |
| openstackgerrit | Christian Groschupp proposed openstack-infra/shade master: Creating a server and specifying scheduler_hints fails https://review.openstack.org/519705 | 17:20 |
| *** yamamoto has joined #openstack-infra | 17:21 | |
| *** jpich has quit IRC | 17:21 | |
| *** clarkb has joined #openstack-infra | 17:22 | |
| *** salv-orlando has quit IRC | 17:22 | |
| *** Apoorva has joined #openstack-infra | 17:23 | |
| *** yamamoto has quit IRC | 17:26 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Be consistent with the ZK data model https://review.openstack.org/519706 | 17:27 |
| *** inc0 has joined #openstack-infra | 17:30 | |
| *** rlandy|rover|brb is now known as rlandy|rover | 17:33 | |
| mnaser | is logs.o.o responding for folks here? | 17:33 |
| mnaser | it seems to be responding slowly | 17:33 |
| *** setuid has left #openstack-infra | 17:33 | |
| *** yamamoto has joined #openstack-infra | 17:36 | |
| *** bauzas_ has joined #openstack-infra | 17:36 | |
| pabelanger | yah, I think clarkb is looking into ti | 17:37 |
| pabelanger | it* | 17:37 |
| panda | is there someone here with mathematical statistic background ? If the probability for a single job to fail is 4%, and we have an average of 4 jobs per gate, what's the probability that the entire set of jobs for a queue of 20 reviews will fail ? whats the probability for a queue of 5 instead ? | 17:39 |
| *** bauzas_ has quit IRC | 17:39 | |
| openstackgerrit | Christian Groschupp proposed openstack-infra/shade master: Fix creating a server with specifying scheduler_hints https://review.openstack.org/519705 | 17:40 |
| *** yamamoto_ has joined #openstack-infra | 17:40 | |
| *** gyee has quit IRC | 17:40 | |
| *** yamamoto has quit IRC | 17:41 | |
| *** gyee has joined #openstack-infra | 17:44 | |
| *** Apoorva has quit IRC | 17:44 | |
| *** salv-orl_ has quit IRC | 17:44 | |
| *** dsariel has quit IRC | 17:44 | |
| panda | dam, why all the statistics teacher made me hate it ? | 17:44 |
| *** Apoorva has joined #openstack-infra | 17:44 | |
| *** dtantsur|brb is now known as dtantsur | 17:45 | |
| *** bauzas_ has joined #openstack-infra | 17:45 | |
| mnaser | clarkb: id look into DNS, the behaviour seems related to DNS issues where it takes forever to load then pops right out | 17:45 |
| mnaser | so maybe double check if resolvers are working on that machine | 17:45 |
| *** yamamoto_ has quit IRC | 17:45 | |
| *** bauzas_ has quit IRC | 17:45 | |
| *** bauzas_ has joined #openstack-infra | 17:46 | |
| *** bauzas_ is now known as bauzas | 17:46 | |
| *** Swami has joined #openstack-infra | 17:48 | |
| *** jascott1 has joined #openstack-infra | 17:48 | |
| *** iyamahat has joined #openstack-infra | 17:49 | |
| mgagne | clarkb: do you have more details on the issue at inap? I saw the link but what's the actual issue? Zuul unabled to SSH to slave to gather logs? | 17:50 |
| *** salv-orlando has joined #openstack-infra | 17:52 | |
| *** yamamoto has joined #openstack-infra | 17:52 | |
| mgagne | also, I find it very hard to determine the actual instance in our cloud that was used for the job, there is no longer a hostname or UUID, only an IP. | 17:53 |
| mgagne | ok found it in on of the zuul-info files | 17:53 |
| *** derekh has quit IRC | 17:54 | |
| clarkb | mgagne: ya job took a long time and timed out. and then ssh failed to run the post tasks for cleanup | 17:54 |
| *** salv-orlando has quit IRC | 17:54 | |
| clarkb | mnaser: will do | 17:55 |
| *** salv-orlando has joined #openstack-infra | 17:55 | |
| *** salv-orl_ has joined #openstack-infra | 17:58 | |
| *** yamahata has joined #openstack-infra | 17:59 | |
| *** dsariel has joined #openstack-infra | 17:59 | |
| *** armaan has joined #openstack-infra | 18:00 | |
| *** salv-orlando has quit IRC | 18:01 | |
| *** yamamoto has quit IRC | 18:03 | |
| *** efoley has quit IRC | 18:07 | |
| *** claudiub|2 has joined #openstack-infra | 18:08 | |
| *** claudiub has quit IRC | 18:11 | |
| *** dsariel has quit IRC | 18:12 | |
| *** bobh has quit IRC | 18:19 | |
| clarkb | oh right infra meeting is an hour earlier now than before relative to my timezone | 18:20 |
| clarkb | infra meeting in ~39 minutes | 18:20 |
| *** salv-orl_ has quit IRC | 18:24 | |
| *** salv-orlando has joined #openstack-infra | 18:24 | |
| *** dsariel has joined #openstack-infra | 18:25 | |
| *** tosky has quit IRC | 18:26 | |
| *** rhallisey has joined #openstack-infra | 18:28 | |
| *** salv-orlando has quit IRC | 18:28 | |
| *** yamamoto has joined #openstack-infra | 18:29 | |
| *** dsariel has quit IRC | 18:30 | |
| clarkb | panda: you can probably treat failure as independent (though they won't be in all cases) then you want to calculate the chance of a single failure, 2 failures, 3 failures, 4 failures and sum them for total chance of a -1 | 18:31 |
| *** jkilpatr has joined #openstack-infra | 18:31 | |
| clarkb | panda: that covers it for a single buildset for one change. Then you do a similar thing with multiple changes | 18:31 |
| *** dprince has quit IRC | 18:33 | |
| clarkb | frickler: mnaser rereading cacti graphs I really thing the likeliest culprit is the large outbound data transfer that happened | 18:33 |
| clarkb | we seem to be recovering now based on the fall off of established tcp connections | 18:34 |
| mnaser | That can be an obvious cause as well | 18:34 |
| *** yamamoto has quit IRC | 18:35 | |
| *** yamamoto has joined #openstack-infra | 18:36 | |
| *** tesseract has quit IRC | 18:37 | |
| *** kjackal has quit IRC | 18:38 | |
| *** trown is now known as trown|lunch | 18:41 | |
| openstackgerrit | Merged openstack-infra/infra-specs master: Mark the ptgbot spec implemented https://review.openstack.org/516454 | 18:49 |
| *** jascott1 has quit IRC | 18:49 | |
| *** amoralej is now known as amoralej|off | 18:58 | |
| *** jpena is now known as jpena|off | 18:59 | |
| clarkb | infra meeting time by my UTC clock | 18:59 |
| clarkb | join us in #openstack-meeting | 18:59 |
| *** iyamahat_ has joined #openstack-infra | 19:03 | |
| *** iyamahat has quit IRC | 19:03 | |
| *** camunoz has quit IRC | 19:06 | |
| *** yamamoto has quit IRC | 19:11 | |
| *** tosky has joined #openstack-infra | 19:13 | |
| *** dave-mccowan has quit IRC | 19:21 | |
| *** mihalis68 has quit IRC | 19:23 | |
| *** Sukhdev has joined #openstack-infra | 19:26 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix final: true issue with build-wheel-mirror jobs https://review.openstack.org/519730 | 19:30 |
| *** dprince has joined #openstack-infra | 19:31 | |
| *** kjackal has joined #openstack-infra | 19:33 | |
| *** jascott1 has joined #openstack-infra | 19:33 | |
| pabelanger | AJaeger: ianw: clarkb: another final: true issue ^. I didn't see that when I proposed the original change | 19:34 |
| openstackgerrit | Ruby Loo proposed openstack-infra/openstack-zuul-jobs master: Add irrelevant-files to openstack-tox-functional https://review.openstack.org/519731 | 19:36 |
| *** trown|lunch is now known as trown | 19:39 | |
| *** bobh has joined #openstack-infra | 19:41 | |
| *** owalsh has quit IRC | 19:43 | |
| *** owalsh has joined #openstack-infra | 19:44 | |
| *** dtantsur is now known as dtantsur|afk | 19:45 | |
| *** ralonsoh has quit IRC | 19:46 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch puppet-nova to in-tree testing https://review.openstack.org/513131 | 19:50 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: be more prescriptive in log collection https://review.openstack.org/511526 | 19:52 |
| *** mat128 has quit IRC | 19:54 | |
| *** dhill_ has quit IRC | 19:57 | |
| openstackgerrit | Ruby Loo proposed openstack-infra/openstack-zuul-jobs master: Add irrelevant-files to openstack-tox-functional* https://review.openstack.org/519731 | 20:00 |
| *** Sukhdev has quit IRC | 20:01 | |
| *** dhill_ has joined #openstack-infra | 20:03 | |
| haleyb | clarkb: i just noticed a lot if ipv6 talk in the meeting channel, which reminded me of https://review.openstack.org/#/c/343041/ (support an ipv6 underlay network) - not sure it's exactly related to the topic discussed | 20:03 |
| clarkb | haleyb: not really, we are talking about ipv6 on top of any overlay in the test framework. But also we stillcan't vxlan + ovs +ipv6 on current linux distros as far as I know | 20:04 |
| clarkb | I want to say most recent ovs can ipv6 with vxlan though? so probably worth sorting out | 20:04 |
| haleyb | i think ovs 2.6+ supports it, been a while since i looked at what each distro is shipping | 20:06 |
| *** shardy has quit IRC | 20:07 | |
| *** mrunge has joined #openstack-infra | 20:08 | |
| *** yamamoto has joined #openstack-infra | 20:12 | |
| *** pcaruana has joined #openstack-infra | 20:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix final: true issue with build-wheel-mirror jobs https://review.openstack.org/519730 | 20:13 |
| openstackgerrit | Merged openstack-infra/os-loganalyze master: Support the log level "CRITICAL" https://review.openstack.org/509670 | 20:13 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Import TripleO legacy zuul jobs https://review.openstack.org/514793 | 20:14 |
| *** rfolco has quit IRC | 20:14 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Set ceph_use_fqdn for rdocloud https://review.openstack.org/519744 | 20:15 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Remove outdated legacy-castellan-dsvm-functional https://review.openstack.org/519745 | 20:15 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: WIP - Move openstack-infra/tripleo-ci jobs in-tree https://review.openstack.org/514778 | 20:15 |
| *** ldnunes has quit IRC | 20:16 | |
| *** armaan has quit IRC | 20:17 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/openstack-zuul-jobs master: Remove outdated legacy-castellan-dsvm-functional https://review.openstack.org/519746 | 20:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/openstack-zuul-jobs master: WIP - Move TripleO jobs/templates to tripleo-ci https://review.openstack.org/514792 | 20:22 |
| *** hasharAway has quit IRC | 20:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/openstack-zuul-jobs master: WIP - Move TripleO jobs/templates to tripleo-ci https://review.openstack.org/514792 | 20:23 |
| *** yamamoto has quit IRC | 20:23 | |
| prometheanfire | pabelanger: what did you mean with your comment here https://review.openstack.org/#/c/504530/ should I move the gentoo definition from nl02 to nodepool.yaml? | 20:24 |
| prometheanfire | I also didn't intend for that review to be the voting one, I'd rather have a smaller review that's easier to revert if it's broken :P | 20:25 |
| pabelanger | prometheanfire: image builds happen in nodepool.yaml right now | 20:25 |
| pabelanger | and nl01 and nl02 only launch images in the clouds | 20:26 |
| pabelanger | prometheanfire: so, right now, gentoo isn't actually added to nodepool-builder, because it is still missing in nodepool.yaml | 20:26 |
| prometheanfire | ah, so I need to add it to nl01 and nodepool.yaml both? | 20:27 |
| pabelanger | prometheanfire: well, all 3 right now, just to keep things consistant | 20:27 |
| pabelanger | I'm going to try and clean that up tomorrow | 20:28 |
| prometheanfire | k | 20:28 |
| AJaeger | pabelanger: could you review https://review.openstack.org/504169 - openSUSE tumbleweed nodepool addition, please? | 20:28 |
| *** eharney has quit IRC | 20:28 | |
| prometheanfire | it looks like it was already added to nl01 by jens | 20:29 |
| prometheanfire | to nodepool.yaml too | 20:29 |
| prometheanfire | 443b6417f2bc33d9f6a5c428012e811e7fe51e84 | 20:29 |
| pabelanger | AJaeger: yes | 20:29 |
| AJaeger | thanks, pabelanger | 20:30 |
| prometheanfire | so it's defined but not in use, so next step should be to have a single node available somewhere that could be gated on? | 20:30 |
| pabelanger | no, it isn't built yet | 20:30 |
| AJaeger | prometheanfire: thanks to zuul v3, you can just define your own job and test it on any project - once it's build... | 20:31 |
| pabelanger | it needs to be added to all providers to be uploaded | 20:31 |
| prometheanfire | ah | 20:31 |
| prometheanfire | does it need to be all providers or can there be a test provider (more or less)? | 20:32 |
| prometheanfire | and are you going to be doing this tomorrow pabelanger or should I added it to the providers now? | 20:32 |
| EmilienM | AJaeger: thanks for your help btw | 20:33 |
| pabelanger | AJaeger: clarkb: prometheanfire: actually, before we start bringing more images online, i really want to migrate nodepool-builders to feature/zuulv3 branch. right now we have the 1/2 and 1/2 setup between nodepool-builders and nodepool-launcher. Using a mixture of old and new syntax | 20:33 |
| pabelanger | or merge feature/zuulv3 back to master | 20:34 |
| prometheanfire | ok, I can wait | 20:34 |
| pabelanger | and rebuild nodepool-builders as python3 | 20:34 |
| prometheanfire | just ping me when I'm good to go :D | 20:34 |
| pabelanger | Shrews: mordred: ^ you might be interested too | 20:34 |
| *** rhallisey has quit IRC | 20:36 | |
| AJaeger | dirk: fyi ^ | 20:36 |
| *** rhallisey has joined #openstack-infra | 20:36 | |
| clarkb | pabelanger: what changed on the builder side? it was v3'd first and should have already haf yhe relevant code merged to master? | 20:37 |
| pabelanger | clarkb: nodepool config syntax is the big change, that is in feature/zuulv3 and not master | 20:37 |
| pabelanger | so we have old syntax still on nodepool-builders | 20:37 |
| clarkb | pabelanger: but none of it affects the builder right? | 20:38 |
| pabelanger | clarkb: not sure I follow | 20:38 |
| clarkb | the builder had all of its stuff updated to use zk and all that long ago and was merged to master | 20:38 |
| clarkb | we havent changed anything in thr builder on feature/zuulv3 since have we? | 20:39 |
| pabelanger | clarkb: yah, the syntax for nodepool.yaml changed | 20:39 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml is what nodepool-builder is running | 20:39 |
| pabelanger | and http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nl01.openstack.org.yaml is launchers | 20:40 |
| pabelanger | so, it is a little confusing to people when adding new images, because we have to switch between old and new syntax | 20:40 |
| *** ccamacho has quit IRC | 20:40 | |
| clarkb | I see, the diskimages section did not change though so should be able to use either? | 20:41 |
| clarkb | I guess the builders need cloud info for uploads | 20:41 |
| pabelanger | yah | 20:41 |
| pabelanger | so, I think we can remove or trim diskimages section from launchers, since they don't do builds | 20:42 |
| clarkb | maybe make a builder.yaml using old config to reduce confusion then update it to new syntax as next step? | 20:42 |
| clarkb | reduce confusion on what the file is used for | 20:42 |
| pabelanger | yah, that is what I was going to do tomorrow | 20:43 |
| pabelanger | then maybe see if we are ready to merge feature/zuulv3 to master | 20:43 |
| pabelanger | and redeploy nb04 / nb03 with new config syntax | 20:43 |
| clarkb | aiui merge to master on nodepool at this point requires a zuulv3, no more demand from gearman right? | 20:45 |
| clarkb | so will need to check openstackci puppet and maybe tag master first | 20:45 |
| pabelanger | yah | 20:46 |
| pabelanger | clarkb: mind a +3 on https://review.openstack.org/519730/ | 20:46 |
| pabelanger | going to head down to school to get kids, will return shortly | 20:47 |
| Shrews | pabelanger: i'd like to see the feature branches merged back to master, personally | 20:48 |
| Shrews | but maybe we were waiting for the static node changes to get merged first? | 20:48 |
| Shrews | i can't recall | 20:48 |
| clarkb | if we merge into master at this point on nodepool I would close feature/zuulv3 and add static nodes on master | 20:49 |
| Shrews | clarkb: well, they've already been proposed. i think all we need to do is review and approve | 20:50 |
| clarkb | I think its more about making sure v2 is ready to be evicted from master than anything else | 20:50 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Import TripleO legacy zuul jobs https://review.openstack.org/514793 | 20:51 |
| *** chasemp has joined #openstack-infra | 20:51 | |
| clarkb | pabelanger: approved, left comment for another way we could approach that situation but not mecessary to update before merging | 20:51 |
| *** gouthamr has quit IRC | 20:53 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix final: true issue with build-wheel-mirror jobs https://review.openstack.org/519730 | 20:55 |
| *** e0ne has joined #openstack-infra | 20:55 | |
| AJaeger | pabelanger, prometheanfire , this is the missing change for gentoo, isn't it? https://review.openstack.org/519197 | 20:56 |
| EmilienM | AJaeger: you're fast | 20:57 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Import TripleO legacy zuul jobs https://review.openstack.org/514793 | 20:57 |
| *** smatzek has quit IRC | 20:59 | |
| AJaeger | EmilienM: sometimes ;) | 21:00 |
| AJaeger | EmilienM: you cannot just move a template from one repo to another. If you use the same name, you have name collision in our single global namespace | 21:00 |
| EmilienM | AJaeger: because we still have it in legacy layout? | 21:01 |
| AJaeger | EmilienM: So, you have to use a different name. And then rename users, perhaps in multiple steps | 21:01 |
| AJaeger | EmilienM: exactly, still in use in openstack-zuul-jobs | 21:01 |
| EmilienM | ok I see | 21:01 |
| *** smatzek has joined #openstack-infra | 21:03 | |
| *** ramishra has joined #openstack-infra | 21:03 | |
| *** armaan has joined #openstack-infra | 21:05 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add MTU asjustment parameters to RDO Cloud settings https://review.openstack.org/519486 | 21:06 |
| *** e0ne_ has joined #openstack-infra | 21:07 | |
| *** smatzek has quit IRC | 21:07 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Import TripleO legacy zuul jobs https://review.openstack.org/514793 | 21:09 |
| *** e0ne has quit IRC | 21:11 | |
| *** kjackal has quit IRC | 21:11 | |
| *** dhinesh has joined #openstack-infra | 21:14 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Import TripleO legacy zuul jobs https://review.openstack.org/514793 | 21:15 |
| *** zoli has quit IRC | 21:16 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: WIP - Move openstack-infra/tripleo-ci jobs in-tree https://review.openstack.org/514778 | 21:17 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/openstack-zuul-jobs master: WIP - Move TripleO jobs/templates to openstack-infra/tripleo-ci https://review.openstack.org/514792 | 21:18 |
| *** salv-orlando has joined #openstack-infra | 21:18 | |
| *** zoli has joined #openstack-infra | 21:18 | |
| jlvillal | clarkb: Thanks for approving the CRITICAL log level change. When will that take effect? Does something have to be restarted? | 21:19 |
| clarkb | ianw: I'm dumping a restore tarball on zuulv3:/root/backup-restore-201711114 now | 21:20 |
| *** Goneri has quit IRC | 21:21 | |
| *** kgiusti has left #openstack-infra | 21:21 | |
| *** gouthamr has joined #openstack-infra | 21:23 | |
| *** e0ne_ has quit IRC | 21:24 | |
| *** slaweq has joined #openstack-infra | 21:25 | |
| mordred | clarkb: I just saw this: http://paste.openstack.org/show/626299/ | 21:25 |
| mordred | clarkb: in a job - note that it's using inap for some mirrors and rax for others | 21:25 |
| mordred | clarkb: is this a known issue? or should I investigate more? | 21:26 |
| pabelanger | wow, that is new for me | 21:26 |
| pabelanger | possible bug in UCA mirrors? | 21:26 |
| ianw | xenial-updates/pike/main ... | 21:26 |
| mordred | that job was also having issues installing python-docutils | 21:26 |
| clarkb | mordred: I think that means our compat thing for uca is broken | 21:27 |
| clarkb | mirror.dfw is the default uca location in the image builds iirc | 21:27 |
| clarkb | so the ansible tasks that replace the nodepool ready script are likely writing out the wrong data or no data at all | 21:28 |
| mordred | oh yah | 21:28 |
| pabelanger | do we use UCA for image builds? | 21:28 |
| mordred | sweet. well, at least it's a place to look | 21:28 |
| clarkb | pabelanger: we don't use it for builds but we write it in place as a disabled repo iirc | 21:28 |
| *** dprince has quit IRC | 21:28 | |
| clarkb | pabelanger: its in /etc/apt/sources.available or whatever and you ahve to symlink it over | 21:28 |
| pabelanger | ah, okay | 21:29 |
| *** rbrndt has quit IRC | 21:29 | |
| clarkb | pabelanger: in Sydney Citycloud was looking for feedback on their cloud so I'm trying to put together an email response with some info about current situation there. Are those leaked instances that won't delete still a problem? that was in kna1 right? | 21:29 |
| pabelanger | clarkb: yah, I think that was the region, there was 7 nodes IIRC | 21:31 |
| pabelanger | I also sent an email before the summit, pretty sure I CC'd you | 21:31 |
| clarkb | oh right, I'll followup from there | 21:32 |
| portdirect | in zuulv3 is there a way to specify the tags i want to apply to a playbook? | 21:32 |
| pabelanger | portdirect: no, we don't support --tags today | 21:33 |
| pabelanger | last I looked, you could only toggle that from the CLI | 21:34 |
| *** rbrndt has joined #openstack-infra | 21:34 | |
| *** chasemp has left #openstack-infra | 21:35 | |
| portdirect | hmm | 21:35 |
| *** felipemonteiro has joined #openstack-infra | 21:35 | |
| portdirect | so I suppose similar functionality could be achieved through import of roles, or putting conditionals on tasks from vars | 21:36 |
| clarkb | ianw: I've extracted /var/log/zuul/debug.log from the backup tar. It is 4.1GB large. Begins at 2017-11-13 06:25:03,188 and ends at 2017-11-14 05:38:38,211 according to the logged timestamps in the log file | 21:37 |
| clarkb | ianw: so that file both looks up to date and fairly complete | 21:37 |
| clarkb | ianw: without doing a full restore from backup I'm not sure there is much more I can do in order to check that the new backup server is working as expected | 21:37 |
| clarkb | ianw: I say move forward with the migration. I will review the change to flip other hosts shortly | 21:38 |
| pabelanger | portdirect: yes, that is how I do it. I usually define a task list variable, which gives me the ability to dynamically control tasks or roles | 21:38 |
| pabelanger | then I don't need to pass flags to the CLI, but toggle them via configs | 21:38 |
| portdirect | you happen to have an example handy? | 21:39 |
| ianw | clarkb: great, thanks. i'll keep an eye on things over the next few days to make sure data is coming in ok | 21:39 |
| ianw | although, i admit i didn't think about zuul's debug log ... at 4+ gb per day, presumably all fuzzed sufficiently to avoid any de-duplication efforts ... | 21:41 |
| ianw | maybe we should filter it | 21:41 |
| clarkb | pabelanger: can pabelanger-test1 be deleted from citycloud sto2? | 21:42 |
| pabelanger | clarkb: yes! | 21:42 |
| pabelanger | I must have missed it | 21:42 |
| clarkb | pabelanger: I think the presence of that server accounts for some of the quota miscalculation errors between nodepool and the cloud | 21:42 |
| *** e0ne has joined #openstack-infra | 21:42 | |
| pabelanger | clarkb: Oh | 21:42 |
| pabelanger | clarkb: i think that was the VM we keep to debug networking issues | 21:42 |
| clarkb | pabelanger: the ipv6? or something else? | 21:43 |
| pabelanger | maybe double ethernet interfaces? | 21:43 |
| pabelanger | but, yah, we can delete it now I think | 21:43 |
| clarkb | oh right that | 21:43 |
| clarkb | at this point it may be easier to just move forward and debug with new resources if necessary? | 21:44 |
| *** rcernin has joined #openstack-infra | 21:44 | |
| pabelanger | agree | 21:44 |
| clarkb | none of the current active instances in sto2 have multiple private interfaces | 21:44 |
| clarkb | pabelanger: ok I'm going to delete it then | 21:44 |
| pabelanger | ++ | 21:45 |
| *** e0ne has quit IRC | 21:45 | |
| *** armaan has quit IRC | 21:46 | |
| *** oomichi is now known as oomichi_afk | 21:46 | |
| clarkb | lon1 is the other region with quota funnyness but not seeing any immediate culprit for that. So that might be a "see anything on your end" question for citycloud | 21:47 |
| *** MasterOfBugs has joined #openstack-infra | 21:47 | |
| *** gouthamr has quit IRC | 21:49 | |
| clarkb | interesting we definitely seem to be leaking floating IPs | 21:50 |
| clarkb | 248 listed in sto2 | 21:50 |
| clarkb | 75 in lon1 | 21:50 |
| clarkb | instance quota is 50 in both cases so would expect fips to be <= 50 | 21:50 |
| clarkb | mordred: ^ fyi | 21:50 |
| clarkb | 18 in la1 which has no instances and 66 in kna1 | 21:51 |
| clarkb | I'm going to clean those up now but something we should watch | 21:52 |
| clarkb | created_at | None <- would be helpful if neutrno showed us that info | 21:53 |
| clarkb | mordred: before I manually start deleting things, we haven't add floating ip pool management to nodepool right? | 21:53 |
| clarkb | Shrews: ^ | 21:53 |
| mordred | clarkb: hrm. | 21:53 |
| mordred | clarkb: what do you mean when you say "floating ip pool management"? I thought we still ran the leaked floater cleanup though ... | 21:54 |
| EmilienM | have you already seen that one? http://logs.openstack.org/29/519229/3/gate/openstack-tox-validate/04a5d67/job-output.txt.gz#_2017-11-14_21_37_11_677355 | 21:54 |
| clarkb | mordred: fip reuse without delete create | 21:54 |
| mordred | nope. we never will | 21:54 |
| clarkb | mordred: and yes I would expect the leaker to address this | 21:54 |
| mordred | it's too racy to be actually usable | 21:54 |
| clarkb | mordred: so there are definitely things broken somewhere possibly in nodepool or in the cloud or both :) | 21:54 |
| mordred | awesome. lemme look | 21:55 |
| clarkb | mordred: ok I won't delete any yet | 21:55 |
| mordred | I say that - actually - when we're able to use the create-and-attach creation mechanism, we should be able to safely re-use | 21:55 |
| mordred | but no, we're not doing that in nodepool yet iirc | 21:55 |
| Shrews | doesn't ring a bell to me | 21:56 |
| mordred | clarkb: we do not have leaker cleanup in v3 that I can see | 21:56 |
| Shrews | we do not | 21:56 |
| *** pramodrj07 has joined #openstack-infra | 21:56 | |
| *** makowals_ has joined #openstack-infra | 21:56 | |
| *** trown is now known as trown|outtypewww | 21:56 | |
| mordred | oh - wait - yah we do ... | 21:57 |
| Shrews | leaked instances, yes | 21:57 |
| mordred | def cleanupLeakedFloaters(self): | 21:57 |
| mordred | with shade_inner_exceptions(): | 21:57 |
| mordred | self._client.delete_unattached_floating_ips() | 21:57 |
| clarkb | mordred: thats the leak cleanup but we don't reuse floating ips without deleting them first | 21:58 |
| mordred | right. | 21:58 |
| clarkb | mordred: so I think the leak checking must be broken | 21:58 |
| clarkb | do we need to explicityl turn it on? | 21:58 |
| Shrews | mordred: oh, heh. part of instance cleanup | 21:58 |
| *** smatzek has joined #openstack-infra | 21:58 | |
| mordred | yah - it's a config value ... | 21:58 |
| mordred | clean-floating-ips | 21:58 |
| clarkb | EmilienM: ya we've seen that happen occosionally but haven't tracked it down yet | 21:58 |
| *** adreznec_ has joined #openstack-infra | 21:58 | |
| EmilienM | ok no prob | 21:58 |
| clarkb | EmilienM: checking nodepool launch logs that IP has not been reused since that job ran so I'm fairly certain we didn't cross wires with different VMs | 21:59 |
| *** raissa_ has joined #openstack-infra | 21:59 | |
| *** adreznec has quit IRC | 21:59 | |
| *** raissa has quit IRC | 21:59 | |
| *** adreznec_ is now known as adreznec | 21:59 | |
| *** pahuang has quit IRC | 21:59 | |
| *** zigo has quit IRC | 21:59 | |
| clarkb | mordred: its not enabled for citycloud | 21:59 |
| *** pahuang has joined #openstack-infra | 22:00 | |
| clarkb | so probably the best thing to do here is enable it for citycloud, then check back in that we've cleaned them up later | 22:00 |
| *** makowals has quit IRC | 22:00 | |
| *** MasterOfBugs has quit IRC | 22:00 | |
| *** aeng has joined #openstack-infra | 22:00 | |
| * clarkb writes a patch | 22:01 | |
| *** zigo has joined #openstack-infra | 22:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Turn on floating ip cleanup on fip clouds https://review.openstack.org/519768 | 22:01 |
| mordred | clarkb: ^^ | 22:01 |
| clarkb | oh perfect | 22:01 |
| mordred | Shrews: ^^ you too :) | 22:01 |
| *** bandini has quit IRC | 22:01 | |
| *** smatzek has quit IRC | 22:02 | |
| *** pcaruana has quit IRC | 22:02 | |
| clarkb | I'll hold off on emailing citycloud until we run with that for a bit. Hoping to see our error rate fall off or drop to zero. And if not bring up remaining items with them | 22:03 |
| clarkb | does modifying that config item require a nl restart? | 22:04 |
| *** bandini has joined #openstack-infra | 22:04 | |
| *** pcaruana has joined #openstack-infra | 22:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove release.yaml from build-wheel-mirror jobs https://review.openstack.org/519769 | 22:07 |
| Shrews | clarkb: nope | 22:07 |
| *** pcaruana has quit IRC | 22:07 | |
| pabelanger | clarkb: mordred: ianw: more wheel-mirror fixes^. Getting close to the job being functional | 22:08 |
| clarkb | Shrews: cool, I'll check back on citycloud fip lists later today then | 22:08 |
| *** Krinkle has joined #openstack-infra | 22:11 | |
| Krinkle | Hey all, would be good to get a git-review update released soon that fixes https://storyboard.openstack.org/#!/story/2001266 - "set-upstream" option removed in git 2.x | 22:11 |
| clarkb | Krinkle: ya I've been running the fix patch locally without any problems. It just needs reviews at this point | 22:11 |
| clarkb | was hoping fungi could take a look but he was busy today | 22:12 |
| Krinkle | clarkb: Is it up on gerrit? | 22:12 |
| *** threestrands has joined #openstack-infra | 22:12 | |
| *** threestrands has quit IRC | 22:12 | |
| *** threestrands has joined #openstack-infra | 22:12 | |
| Krinkle | Didn't see a link | 22:12 |
| clarkb | Krinkle: ya let me get you a link | 22:12 |
| clarkb | Krinkle: https://review.openstack.org/#/c/517831/ | 22:12 |
| Krinkle | Im using the homebrew packaee for git-review btw | 22:12 |
| Krinkle | seems last release 1.25 was in 2015? | 22:12 |
| clarkb | ya its been fairly stable (intentionally so) | 22:12 |
| openstackgerrit | Merged openstack-infra/project-config master: Turn on floating ip cleanup on fip clouds https://review.openstack.org/519768 | 22:12 |
| Krinkle | clarkb: Would this justify a patch release? | 22:13 |
| Krinkle | Either from current master or following 1.25.x maybe | 22:13 |
| *** pramodrj07 has quit IRC | 22:13 | |
| clarkb | Krinkle: I think we'll just release a 1.26 as soon as someone else can review that patch above | 22:13 |
| clarkb | Krinkle: the timing of this around the openstack summit wasn't great | 22:14 |
| clarkb | ianw: couple comments on the backup server change. One I think should be fixed. The other I think we can live with and just clean out the old cron entries later | 22:16 |
| clarkb | ianw: let me know what you think | 22:16 |
| * mordred has +2d it already | 22:16 | |
| clarkb | Krinkle: if fungi isn't able to review it tomorrow I think we'll move ahead with other reviews but he volunteered to be primary git-review maintainer many moons ago so would like his feedback | 22:19 |
| Krinkle | sure | 22:20 |
| Krinkle | I'll downgrade to 2.14.x actually. Hadn't realised it was because of 2.15 | 22:20 |
| Krinkle | Git | 22:20 |
| *** threestrands has quit IRC | 22:21 | |
| *** threestrands has joined #openstack-infra | 22:22 | |
| *** threestrands has quit IRC | 22:22 | |
| *** threestrands has joined #openstack-infra | 22:22 | |
| *** Goneri has joined #openstack-infra | 22:22 | |
| clarkb | Krinkle: ya I'm impressed by how many people are getting latest and greatest git | 22:24 |
| *** aeng has quit IRC | 22:24 | |
| Krinkle | :) | 22:24 |
| clarkb | I guess homebrew may be partially to blame? I got it because I run tumbleweed | 22:24 |
| ianw | clarkb: i tried to enable a git copr to get latest git to test that on fedora and ... copr is apparently broken. this is why we can't have nice things :) | 22:25 |
| Krinkle | Yeah, I run 'brew outdated' from time to time and for most tools I use regularly myself I tend to stay on the edge and figure out issues early | 22:25 |
| *** gouthamr has joined #openstack-infra | 22:26 | |
| clarkb | ianw: did you see comments on https://review.openstack.org/#/c/516159/4 ? -1 is really only for the : | 22:27 |
| ianw | clarkb: yep, thanks just fixing now | 22:27 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add MTU adjustment parameters to RDO Cloud settings https://review.openstack.org/519486 | 22:27 |
| mordred | clarkb: I have git version 2.14.1 installed - which I have because PPA | 22:27 |
| clarkb | ianw: fwiw testing new git-review with old git is also valuable | 22:27 |
| ianw | yeah, that's the easy bit :) | 22:28 |
| clarkb | since this isn't theo nly change that has been made since last release | 22:28 |
| *** mat128 has joined #openstack-infra | 22:28 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove ci-backup-rs-ord.openstack.org https://review.openstack.org/516159 | 22:29 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add MTU adjustment parameters to RDO Cloud settings https://review.openstack.org/519486 | 22:32 |
| *** Swami has quit IRC | 22:38 | |
| ianw | well, fyi building git then setting PATH=~/path/to/gitdir:$PATH doesn't seem to work | 22:41 |
| clarkb | random ryi, but firefox 57 just released and is much much faster than older firefox. I've been running the 57 beta for a few weeks nwo and really like it. So if you firefox and don't rely on plugins broken by the new rendering engine stuff you should upgrade :) | 22:44 |
| clarkb | also there is support for u2f now | 22:45 |
| *** aeng has joined #openstack-infra | 22:46 | |
| ianw | Errors running git rebase -p -i remotes/gerrit/master | 22:46 |
| ianw | b'Cannot rebase: Your index contains uncommitted changes.\nPlease commit or stash them.' | 22:46 |
| ianw | ... resisting urge to unicode things ... | 22:46 |
| *** bobh has quit IRC | 22:46 | |
| clarkb | does git status show unmerged changes? | 22:46 |
| ianw | actually it looks totally borked -> http://paste.openstack.org/show/626301/ | 22:48 |
| *** priteau has quit IRC | 22:48 | |
| *** priteau has joined #openstack-infra | 22:49 | |
| *** eharney has joined #openstack-infra | 22:50 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Import TripleO legacy zuul jobs https://review.openstack.org/514793 | 22:52 |
| ianw | tonyb: so where does that leave us with https://review.openstack.org/#/c/510946/ ? i still don't like that change, but maybe it's mroe correct than i thought | 22:52 |
| *** priteau has quit IRC | 22:53 | |
| tonyb | ianw: Yeah I *think* we need something like that. there are 2 things happening at the same time | 22:54 |
| *** Douhet has quit IRC | 22:55 | |
| tonyb | 1) I messed up and processed the EOL for most of the repos which left nova stranded neededing to land stuff on newton (for a [now] public CVE) but not able to as neutron (for example) has EOLd | 22:55 |
| tonyb | and 2) the discussion around longer term stable branches. While that *shouldn't* be needed for newton we're goifn to have to do somethign for ocata+ .. Whiler we have a confused gate we can be a little opportunistic and lay some of the ground work | 22:56 |
| tonyb | ianw: I completely agree that the project-config/zuul cleanup, which jasn't happend for newton yet, makes things more complex and I don't have an answer there | 22:57 |
| clarkb | re 2) I don't think we'd kill the branch and eol anything so the feature matrix would just live on as is | 22:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-openafs master: Use regex to find afsd process https://review.openstack.org/519823 | 22:58 |
| clarkb | (though some of that is to be decided) | 22:58 |
| *** mat128 has quit IRC | 23:01 | |
| *** Apoorva_ has joined #openstack-infra | 23:01 | |
| pabelanger | clarkb: ianw: mind a +3 on https://review.openstack.org/519769/ | 23:02 |
| pabelanger | would like to enqueue another periodic job for wheel-mirror builds | 23:02 |
| *** wolverineav has quit IRC | 23:02 | |
| *** wolverineav has joined #openstack-infra | 23:03 | |
| pabelanger | I also think https://review.openstack.org/519823/ might explain why openafs-client isn't started by puppet | 23:03 |
| pabelanger | but, need to test that | 23:03 |
| *** edmondsw has quit IRC | 23:04 | |
| *** Apoorva has quit IRC | 23:04 | |
| *** edmondsw has joined #openstack-infra | 23:04 | |
| *** erlon has quit IRC | 23:04 | |
| *** Sukhdev has joined #openstack-infra | 23:05 | |
| ianw | pabelanger: so basically it waits for the periodic release job? | 23:06 |
| pabelanger | ianw: not sure I follow | 23:07 |
| ianw | oh, wait i see, this just happens in sequence http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n22644 | 23:07 |
| *** edmondsw_ has joined #openstack-infra | 23:07 | |
| pabelanger | yah | 23:07 |
| clarkb | mordred: Shrews floating ip cleanup seems to at least be partially working. La1 and Kna1 are much happier | 23:07 |
| pabelanger | ianw: release-wheel-mirror will have the right afs keytab | 23:07 |
| clarkb | pabelanger: I'm not sure I understand why / and [/] are different. Aren't they the same? | 23:08 |
| clarkb | or si the bare / not being treated literally? | 23:08 |
| *** edmondsw has quit IRC | 23:08 | |
| mordred | clarkb: woot! | 23:09 |
| clarkb | sto2 is down toe 192 | 23:09 |
| pabelanger | clarkb: I think ps -ef | grep /sbin/afsd is returning success, because it finds the grep command in ps. | 23:09 |
| *** rocky-shiyan has quit IRC | 23:09 | |
| clarkb | so must just be slowly processing it | 23:09 |
| pabelanger | where ps -ef | grep [/]sbin/afsd doesn't show grep | 23:09 |
| *** aeng has quit IRC | 23:10 | |
| clarkb | pabelanger: thats a timing behavior I don't think you should rely on | 23:10 |
| pabelanger | clarkb: why not? | 23:13 |
| clarkb | pabelanger: because if your cpu speeds up you could return to having grep in the results | 23:13 |
| dhinesh | when does gearman trigger logs?, right now i do not have any log in /var/log/zuul/gearman-server.log(configured in gearman-logging.conf) | 23:13 |
| pabelanger | I don't think our puppet Service[] task is firing, because ps returns process true (due to grep) | 23:13 |
| ianw | the old [p]rocess trick is ok ... but pgrep is probably better | 23:13 |
| *** edmondsw_ has quit IRC | 23:13 | |
| clarkb | (or slows down, not sure which direction it would need to go to do that) | 23:13 |
| pabelanger | clarkb: Hmm, I don't think we want grep at all | 23:13 |
| clarkb | ya pgrep might be better | 23:13 |
| pabelanger | can we control that in puppet? | 23:13 |
| clarkb | ianw: pabelanger oh is it working because [v] matches v and not [v] ? | 23:13 |
| clarkb | I'm a derp | 23:13 |
| clarkb | pabelanger: no probably not | 23:13 |
| clarkb | in that case I just need to get more caffeine | 23:13 |
| pabelanger | clarkb: I think we could add status => pgrep /sbin/afsd | 23:14 |
| pabelanger | if I am reading docs right | 23:14 |
| *** aeng has joined #openstack-infra | 23:14 | |
| *** erlon has joined #openstack-infra | 23:14 | |
| clarkb | the race with grep is whether or not ps is done before grep starts and shows up in the process table. [x] works because [x] won't match [x] | 23:15 |
| clarkb | so ya change as is it good enough for me | 23:15 |
| clarkb | I just had to wake up enough to understand what was going on there | 23:15 |
| pabelanger | ah, now I understand | 23:15 |
| *** wolverineav has quit IRC | 23:16 | |
| *** Douhet has joined #openstack-infra | 23:16 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Move tripleo-ui layout in-tree https://review.openstack.org/519828 | 23:16 |
| EmilienM | jrist: ^ | 23:17 |
| tonyb | clarkb: You're probably correct WRT my second point but I can imaging a case wheer we've EOL'd say swift but not nova so we'll need something in test-matrix wont we? | 23:17 |
| jrist | EmilienM: trying to understand all of what you're doing... | 23:18 |
| clarkb | tonyb: ya thats a good point, LTS doesn't necessarily imply LTS the same for all the things | 23:18 |
| clarkb | tonyb: hwoever, if glance depends on swift one would hope we don't do that | 23:18 |
| clarkb | (but there could be other examples that make more sense than swift) | 23:18 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/openstack-zuul-jobs master: Remove tripleo-ui legacy layout https://review.openstack.org/519829 | 23:18 |
| EmilienM | jrist: and ^ - so if that works, tripleo-ui moved to zuul v3 properly | 23:18 |
| EmilienM | jrist: no worries, it took me time to understand | 23:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove release.yaml from build-wheel-mirror jobs https://review.openstack.org/519769 | 23:19 |
| jrist | ok thanks | 23:19 |
| notmyname | oh no. did the LTS conversation move in here now? | 23:19 |
| EmilienM | jrist: note I removed legacy-tripleo-ci-centos-7-undercloud-oooq for now | 23:19 |
| clarkb | notmyname: not really. its more specific to a particular change that would enable other things but has potential to impact lts stuff too | 23:20 |
| EmilienM | jrist: I don't understand why we run this job in tripleo-ui | 23:20 |
| tonyb | clarkb: Sure that may have been a poor choice of projects but my point was longer stable branches doesn't mean longer stable branches for everything | 23:20 |
| jrist | I don't think it was voting either EmilienM | 23:20 |
| ianw | tonyb: so the answer to my question on 510946 "why didn't we need this before" is maybe just that we're getting more asynchronous with eol tags, so people are mix-and-matching older branches more? | 23:20 |
| clarkb | tonyb: ya I think that is probably valid end of the day | 23:20 |
| jrist | I think it got added when zuul change happened | 23:20 |
| tonyb | notmyname: not really, it's kinda a tangent | 23:20 |
| notmyname | tonyb: I'm just trying to keep up ;-) | 23:20 |
| EmilienM | jrist: it was here before | 23:20 |
| tonyb | notmyname: /me too | 23:21 |
| EmilienM | jrist: I think I know why, the undercloud job deploy UI. But really there is no tests... | 23:21 |
| EmilienM | jrist: it's imho a lost of time | 23:21 |
| *** rbrndt has quit IRC | 23:21 | |
| EmilienM | jrist: we'll probably need a new job in the future when the tempest plugin is ready | 23:21 |
| tonyb | ianw: basically asynchornous covers it. It was my mistake to EOL some but not all "defcore" projects | 23:21 |
| jrist | yes and we have a test from weshay | 23:21 |
| jrist | EmilienM: ^ | 23:21 |
| EmilienM | jrist: where? | 23:21 |
| jrist | EmilienM: that actually cehcks if UI is running | 23:21 |
| EmilienM | in undercloud-oooq? | 23:21 |
| EmilienM | let's move on #tripleo | 23:22 |
| jrist | EmilienM: https://review.openstack.org/#/c/514795/ | 23:22 |
| *** gyee_ has quit IRC | 23:23 | |
| *** pahuang has quit IRC | 23:23 | |
| *** pahuang has joined #openstack-infra | 23:25 | |
| *** rbrndt has joined #openstack-infra | 23:25 | |
| mnaser | if a project has jobs defined in its own repo, do we still want to leave noop-jobs in there? | 23:29 |
| *** rlandy|rover is now known as rlandy|rover|afk | 23:29 | |
| *** ramishra has quit IRC | 23:30 | |
| dhinesh | does zuul have a dependency on the version of jenkins? currently my jenkins is on 2.73 and zuul is 2.5.1 are they compatible? | 23:31 |
| *** jaypipes has quit IRC | 23:32 | |
| clarkb | dhinesh: they should be compatible. There are some known gotchas like with jenkins' env parameter sanitizing from plugins | 23:33 |
| *** felipemonteiro has quit IRC | 23:34 | |
| *** edmondsw has joined #openstack-infra | 23:36 | |
| *** eharney has quit IRC | 23:36 | |
| *** gyee_ has joined #openstack-infra | 23:36 | |
| clarkb | ok citycloud fips appear cleaned up | 23:36 |
| dhinesh | clarkb: do those env parameters affect the communication with zuul? since my builds defined in zuul/layout.yaml is not visible on jenkins UI | 23:37 |
| clarkb | dhinesh: it affects what environment parameter are available to running jobs. By default jenkins does not let jenkins plugins modify the environment. You can either whitelist entire plugins or specific vars that it will let pass through | 23:38 |
| *** xarses_ has quit IRC | 23:38 | |
| *** edmondsw has quit IRC | 23:40 | |
| *** slaweq has quit IRC | 23:41 | |
| *** dave-mccowan has joined #openstack-infra | 23:42 | |
| *** slaweq has joined #openstack-infra | 23:42 | |
| dhinesh | clarkb: do you know the specific setting which i can use to allow entire plugins to pass through? | 23:42 |
| clarkb | dhinesh: not off the top of my head but I want to say the third party ci documentation covers it? | 23:42 |
| dhinesh | clarkb: it has a part to securing it, but not much info on allowing a pass through, https://docs.openstack.org/infra/openstackci/third_party_ci.html#securing-jenkins-optional | 23:45 |
| clarkb | dhinesh: https://wiki.jenkins.io/display/JENKINS/Plugins+affected+by+fix+for+SECURITY-170 that has more info | 23:46 |
| *** slaweq has quit IRC | 23:46 | |
| clarkb | dhinesh: https://jenkins.io/security/advisory/2016-05-11/ too | 23:46 |
| dhinesh | clarkb: thanks, will take a look | 23:50 |
| *** stakeda has joined #openstack-infra | 23:52 | |
| *** gouthamr has quit IRC | 23:52 | |
| *** rbrndt has quit IRC | 23:52 | |
| *** rbrndt has joined #openstack-infra | 23:52 | |
| *** rbrndt has quit IRC | 23:52 | |
| *** claudiub|2 has quit IRC | 23:53 | |
| ianw | arrgghh ... why did i start looking at unicode stuff | 23:57 |
| openstackgerrit | Merged openstack-infra/shade master: Updated from global requirements https://review.openstack.org/519785 | 23:57 |
| *** Goneri has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!