| *** boden has quit IRC | 00:00 | |
| *** smatzek has quit IRC | 00:01 | |
| *** markvoelker_ has quit IRC | 00:03 | |
| smcginnis | There it is. | 00:04 |
|---|---|---|
| *** yamamoto has joined #openstack-infra | 00:05 | |
| jeblair | okay, i'm restarting the scheduler now | 00:05 |
| dmsimard | btw I'll be at OpenStack Canada day tomorrow so limited responsiveness | 00:06 |
| jeblair | i'll be able to restore the changes already in queues, but i've lost the unprocessed events | 00:06 |
| dmsimard | pabelanger will be giving a talk there too :D | 00:06 |
| *** bobh has joined #openstack-infra | 00:07 | |
| jeblair | dmsimard, pabelanger: yay! | 00:07 |
| dmsimard | "Zuulv3: Using Ansible to test, stage and deploy your OpenStack clouds" | 00:07 |
| dmsimard | I wonder what this Zuulv3 thing I keep hearing about | 00:07 |
| *** markvoelker has joined #openstack-infra | 00:09 | |
| *** hyakuhei has joined #openstack-infra | 00:12 | |
| EmilienM | jeblair: is a zuul able to recover previous jobs after a restart? or we still need to run recheck? | 00:12 |
| *** andreas_s has joined #openstack-infra | 00:13 | |
| jeblair | EmilienM: i'm re-equeuing the jobs that were already running; if they weren't, you'll need to recheck | 00:13 |
| EmilienM | ok | 00:13 |
| *** markvoelker has quit IRC | 00:13 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Update neutron-lbaas to use -neutron templates https://review.openstack.org/513208 | 00:14 |
| *** bobh has quit IRC | 00:15 | |
| ianw | jeblair: odd to get a check on https://review.openstack.org/#/c/509116/ (merged 3rd oct)? did you cat an old file? | 00:17 |
| *** markvoelker has joined #openstack-infra | 00:18 | |
| *** jkilpatr has quit IRC | 00:18 | |
| jeblair | yes i did. sigh | 00:19 |
| jeblair | re-restarting. | 00:19 |
| *** dingyichen has joined #openstack-infra | 00:19 | |
| *** gouthamr has joined #openstack-infra | 00:19 | |
| *** andreas_s has quit IRC | 00:21 | |
| *** bobh has joined #openstack-infra | 00:22 | |
| *** andreas_s has joined #openstack-infra | 00:22 | |
| *** markvoelker has quit IRC | 00:22 | |
| *** ijw has joined #openstack-infra | 00:23 | |
| jlvillal | Noticed one of our grenade jobs failed with POST_FAILURE. Error here: http://logs.openstack.org/21/508321/2/check/legacy-grenade-dsvm-ironic/ceb3b7e/job-output.txt.gz#_2017-10-18_23_48_16_712116 | 00:24 |
| jlvillal | The SSH key had changed... | 00:24 |
| jlvillal | First I saw that happen. | 00:24 |
| *** andreas_s has quit IRC | 00:26 | |
| jlvillal | And other patches with grenade are failing: http://logs.openstack.org/22/508322/2/check/legacy-grenade-dsvm-ironic/312ff69/job-output.txt.gz#_2017-10-18_23_58_18_038026 | 00:26 |
| jlvillal | After the tempest run finishes. Then it can't SSH to the node. Anyone else seeing this? | 00:26 |
| *** markvoelker has joined #openstack-infra | 00:27 | |
| ianw | jlvillal: that's from like 20 minutes ago right? | 00:27 |
| jlvillal | ianw, Could be. | 00:27 |
| jlvillal | Had about four different jobs fail with errors like that. | 00:28 |
| jlvillal | ianw, Was it a transient issue? | 00:28 |
| ianw | that's likely in the middle of a restart, i would wait a bit for a recheck | 00:28 |
| jlvillal | ianw, Okay. Thanks. | 00:28 |
| *** huanxie has joined #openstack-infra | 00:29 | |
| *** rwsu has quit IRC | 00:30 | |
| *** ijw has quit IRC | 00:31 | |
| *** thorst has joined #openstack-infra | 00:31 | |
| *** markvoelker has quit IRC | 00:32 | |
| *** andreas_s has joined #openstack-infra | 00:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove old buildsets https://review.openstack.org/513209 | 00:40 |
| *** kiennt26 has joined #openstack-infra | 00:40 | |
| *** wolverin_ has joined #openstack-infra | 00:43 | |
| *** wolverineav has quit IRC | 00:43 | |
| *** salv-orlando has joined #openstack-infra | 00:43 | |
| *** salv-orlando has quit IRC | 00:47 | |
| *** mriedem has quit IRC | 00:48 | |
| *** andreas_s has quit IRC | 00:49 | |
| *** cuongnv has joined #openstack-infra | 00:49 | |
| *** rwsu has joined #openstack-infra | 00:51 | |
| *** wolverin_ has quit IRC | 00:51 | |
| *** wolverineav has joined #openstack-infra | 00:52 | |
| *** bobh has quit IRC | 00:53 | |
| *** andreas_s has joined #openstack-infra | 00:54 | |
| *** markvoelker has joined #openstack-infra | 00:54 | |
| *** wolverineav has quit IRC | 00:56 | |
| *** thorst has quit IRC | 00:57 | |
| dhellmann | clarkb : I guess that host key update wasn't right? http://logs.openstack.org/92/513192/1/check/build-openstack-sphinx-docs/0bfe0d9/job-output.txt.gz#_2017-10-18_23_17_38_644262 | 00:58 |
| * dhellmann drops back offline | 00:58 | |
| *** LindaWang has joined #openstack-infra | 00:58 | |
| *** markvoelker has quit IRC | 00:59 | |
| *** mriedem has joined #openstack-infra | 00:59 | |
| *** gildub has joined #openstack-infra | 00:59 | |
| *** masuberu has joined #openstack-infra | 01:00 | |
| *** Apoorva_ has joined #openstack-infra | 01:01 | |
| *** Apoorva has quit IRC | 01:03 | |
| *** masber has quit IRC | 01:04 | |
| clarkb | dhellmann: that was from zuul to the test node I think. Fallout from zuul service restarts maybe | 01:05 |
| *** Apoorva_ has quit IRC | 01:05 | |
| *** andreas_s has quit IRC | 01:07 | |
| *** andreas_s has joined #openstack-infra | 01:12 | |
| *** markvoelker has joined #openstack-infra | 01:12 | |
| *** bobh has joined #openstack-infra | 01:16 | |
| *** andreas_s has quit IRC | 01:16 | |
| pabelanger | yah, executor to nodepool node. Is it possible our persistant SSH isn't working correctly? I wouldn't expect the connection to be closed | 01:21 |
| pabelanger | interesting enough, that is in the middle of an ansible-playbook | 01:22 |
| pabelanger | middle of a play | 01:22 |
| *** thorst has joined #openstack-infra | 01:23 | |
| *** namnh has joined #openstack-infra | 01:23 | |
| clarkb | pabelanger: its because zuul eas restarted I think | 01:24 |
| *** daidv_ has quit IRC | 01:24 | |
| *** daidv_ has joined #openstack-infra | 01:24 | |
| *** baoli has joined #openstack-infra | 01:25 | |
| *** ijw has joined #openstack-infra | 01:27 | |
| *** masayukig has quit IRC | 01:28 | |
| *** masayukig has joined #openstack-infra | 01:29 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Use -neutron template variants for neutron-fwaas job https://review.openstack.org/513212 | 01:30 |
| *** ijw has quit IRC | 01:32 | |
| *** thorst has quit IRC | 01:36 | |
| SamYaple | so i got some breakage in a post job i dont understand. http://logs.openstack.org/84/847cc0b4765262efd4935772cf85a7046223b0cd/post/publish-loci-cinder/84c2e23/job-output.txt.gz#_2017-10-18_23_19_56_997768 | 01:38 |
| SamYaple | any help would be appreciated | 01:38 |
| *** andreas_s has joined #openstack-infra | 01:39 | |
| *** salv-orlando has joined #openstack-infra | 01:44 | |
| *** masber has joined #openstack-infra | 01:44 | |
| *** masuberu has quit IRC | 01:47 | |
| *** markvoelker has quit IRC | 01:47 | |
| *** salv-orlando has quit IRC | 01:49 | |
| *** hongbin has joined #openstack-infra | 01:54 | |
| *** baoli has quit IRC | 01:55 | |
| *** ijw has joined #openstack-infra | 01:56 | |
| *** markvoelker has joined #openstack-infra | 02:00 | |
| *** ijw has quit IRC | 02:01 | |
| *** ijw has joined #openstack-infra | 02:02 | |
| *** andreas_s has quit IRC | 02:02 | |
| *** yamahata has quit IRC | 02:02 | |
| *** baoli has joined #openstack-infra | 02:03 | |
| *** iyamahat has quit IRC | 02:03 | |
| *** fanzhang has joined #openstack-infra | 02:04 | |
| *** markvoelker has quit IRC | 02:05 | |
| *** ijw has quit IRC | 02:06 | |
| *** dhinesh has quit IRC | 02:07 | |
| *** markvoelker has joined #openstack-infra | 02:10 | |
| *** andreas_s has joined #openstack-infra | 02:11 | |
| *** rlandy|bbl is now known as rlandy | 02:12 | |
| *** bobh has quit IRC | 02:14 | |
| *** markvoelker has quit IRC | 02:14 | |
| *** andreas_s has quit IRC | 02:16 | |
| *** masber has quit IRC | 02:19 | |
| *** markvoelker has joined #openstack-infra | 02:19 | |
| *** baoli has quit IRC | 02:20 | |
| *** stakeda has joined #openstack-infra | 02:20 | |
| *** gyee has quit IRC | 02:21 | |
| *** markvoelker has quit IRC | 02:23 | |
| *** rwsu has quit IRC | 02:24 | |
| *** andreas_s has joined #openstack-infra | 02:25 | |
| *** Swami has quit IRC | 02:25 | |
| *** jamesmcarthur has joined #openstack-infra | 02:25 | |
| *** markvoelker has joined #openstack-infra | 02:28 | |
| *** andreas_s has quit IRC | 02:29 | |
| *** jamesmcarthur has quit IRC | 02:30 | |
| jeblair | i think what we're seeing is zuul losing the connection to nodepool and having all the nodes killed from under it. i don't believe there's a direct path for zuul to realize that if that happens, it should disregard the results of those builds. it should be possible to fix. | 02:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add tripleo-heat-templates into tripleo shared queue for gate https://review.openstack.org/513222 | 02:31 |
| *** markvoelker has quit IRC | 02:33 | |
| *** ijw has joined #openstack-infra | 02:35 | |
| pabelanger | jeblair: is that via zookeeper? | 02:36 |
| jeblair | ya | 02:36 |
| pabelanger | so, that would make sense, as I understand out the locks work | 02:36 |
| pabelanger | how* | 02:37 |
| *** thorst has joined #openstack-infra | 02:37 | |
| *** ijw has quit IRC | 02:39 | |
| *** thorst has quit IRC | 02:42 | |
| *** andreas_s has joined #openstack-infra | 02:43 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Use -neutron template variants for neutron-fwaas job https://review.openstack.org/513212 | 02:44 |
| *** salv-orlando has joined #openstack-infra | 02:45 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in tap-as-a-service https://review.openstack.org/513227 | 02:45 |
| *** markvoelker has joined #openstack-infra | 02:46 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in tap-as-a-service https://review.openstack.org/513228 | 02:46 |
| *** dhinesh has joined #openstack-infra | 02:48 | |
| *** gmann_afk is now known as gmann | 02:49 | |
| *** salv-orlando has quit IRC | 02:49 | |
| *** gongysh has joined #openstack-infra | 02:50 | |
| namnh | AJaeger: hi Andreas, are you free? | 02:51 |
| tonyb | jhesketh_: Are you likely to be able to run the eol_branch script next week or should I ping someone else? | 02:52 |
| *** rosmaita has quit IRC | 02:55 | |
| *** andreas_s has quit IRC | 02:56 | |
| *** gouthamr has quit IRC | 02:57 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Move to dictionary list of projects zuul._projects https://review.openstack.org/513233 | 03:01 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs https://review.openstack.org/510720 | 03:03 |
| *** andreas_s has joined #openstack-infra | 03:05 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Move to dictionary list of projects zuul._projects https://review.openstack.org/513235 | 03:08 |
| ianw | tox-cover tox-cover : ERROR Unable to find playbook /var/lib/zuul/builds/6486cea579534fbe9bda622c555bed65/trusted/project_1/git.openstack.org/openstack-infra/zuul-jobs/playbooks/cover/tox/cover-post (non-voting) | 03:09 |
| ianw | interesting, we just added that no? | 03:09 |
| *** andreas_s has quit IRC | 03:10 | |
| pabelanger | ianw: https://review.openstack.org/512910 | 03:12 |
| pabelanger | fixing | 03:12 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Correct path to coverage job https://review.openstack.org/513236 | 03:12 |
| ianw | pabelanger: ^ | 03:12 |
| ianw | :) | 03:12 |
| *** toabctl has quit IRC | 03:14 | |
| pabelanger | ianw: lets find a job with cover and test with depends-on | 03:14 |
| pabelanger | we should have done that before merge the change | 03:14 |
| pabelanger | or added a test | 03:14 |
| ianw | pabelanger: yeah, i got a zuul one, hang on | 03:14 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Add _projects to convert project list to dictionary https://review.openstack.org/512868 | 03:15 |
| *** andreas_s has joined #openstack-infra | 03:15 | |
| ianw | pabelanger: ^ | 03:15 |
| *** ramishra has joined #openstack-infra | 03:16 | |
| ianw | the status page is currently at "slows my browser" stage :/ | 03:17 |
| ianw | one step better than "kills my browser", however | 03:18 |
| pabelanger | ianw: I +2'd, if passed I'd say merge to fix broken jobs. | 03:18 |
| pabelanger | going to work on getting cross repo job working in the moving for it, maybe run zuul tox-cover, if those files change | 03:19 |
| ianw | ++ | 03:19 |
| pabelanger | morning* | 03:19 |
| *** nikhil has quit IRC | 03:20 | |
| pabelanger | think I'm going to turn in, early day tomorrow. OpenStack days CANADA! | 03:20 |
| *** markvoelker has quit IRC | 03:20 | |
| ianw | have fun out and aboot | 03:20 |
| *** toabctl has joined #openstack-infra | 03:21 | |
| *** namnh has quit IRC | 03:25 | |
| *** markvoelker has joined #openstack-infra | 03:25 | |
| *** namnh has joined #openstack-infra | 03:26 | |
| *** mriedem has quit IRC | 03:28 | |
| *** markvoelker has quit IRC | 03:30 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 03:30 | |
| *** andreas_s has quit IRC | 03:33 | |
| *** markvoelker has joined #openstack-infra | 03:34 | |
| *** thorst has joined #openstack-infra | 03:38 | |
| *** markvoelker has quit IRC | 03:39 | |
| *** rlandy has quit IRC | 03:39 | |
| EmilienM | pabelanger: I'm jealous | 03:43 |
| EmilienM | pabelanger: I wish I could join :D | 03:43 |
| *** hongbin has quit IRC | 03:43 | |
| *** markvoelker has joined #openstack-infra | 03:43 | |
| *** jgriffith_ has joined #openstack-infra | 03:44 | |
| pabelanger | ianw: http://logs.openstack.org/68/512868/3/check/tox-cover/f934e78/ | 03:45 |
| pabelanger | ianw: so, we should fix that | 03:45 |
| *** thorst has quit IRC | 03:46 | |
| *** salv-orlando has joined #openstack-infra | 03:46 | |
| pabelanger | coverage_output_src: "src/{{ zuul.project.canonical_name }}/cover" | 03:46 |
| pabelanger | without trailing slatsh | 03:46 |
| pabelanger | slash* | 03:46 |
| pabelanger | that will sync back the directory, and not the contents of the directory | 03:46 |
| *** jgriffith_ is now known as jgriffith | 03:47 | |
| ianw | pabelanger: hmm, is there anything else in there but the report? | 03:47 |
| pabelanger | ianw: yah, a bunch of html files | 03:47 |
| *** markvoelker has quit IRC | 03:48 | |
| ianw | but they're still linked? | 03:48 |
| ianw | i guess the testr results, yeah | 03:49 |
| *** yamahata has joined #openstack-infra | 03:49 | |
| pabelanger | running cover locally to see | 03:49 |
| pabelanger | yah, bunch of html things | 03:50 |
| pabelanger | icon, js | 03:50 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Correct path to coverage job https://review.openstack.org/513236 | 03:50 |
| *** salv-orlando has quit IRC | 03:50 | |
| pabelanger | Yah, that should be better | 03:51 |
| pabelanger | is you want to recheck | 03:51 |
| pabelanger | ianw: also, something is wrong with ze06.o.o finger port | 03:51 |
| pabelanger | we are no longer running it | 03:51 |
| pabelanger | so, log streaming is failing | 03:51 |
| pabelanger | not sure if you want to poke at that, but we should make note on etherpad zuulv3-issues | 03:52 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Add _projects to convert project list to dictionary https://review.openstack.org/512868 | 03:52 |
| *** markvoelker has joined #openstack-infra | 03:52 | |
| ianw | i had a change to that anyway | 03:52 |
| ianw | looking at ze06 ... | 03:53 |
| ianw | ok, netstat says nothing on port 79 | 03:54 |
| pabelanger | yah, I couldn't connect to it using finger client | 03:54 |
| ianw | zuul 6047 6044 0 Oct17 ? 00:00:12 [zuul-executor] <defunct> wonder if that was it | 03:55 |
| pabelanger | I think that means LogStreamer some how died | 03:55 |
| pabelanger | oh, maybe | 03:55 |
| *** ramishra has quit IRC | 03:56 | |
| *** markvoelker has quit IRC | 03:57 | |
| *** dave-mccowan has quit IRC | 03:57 | |
| ianw | there's a few exceptions in the logs, but no smoking gun | 03:58 |
| pabelanger | ianw: btw, for openstack summit, do you have any recomendations for data SIM for cell phone? | 03:59 |
| openstackgerrit | Tovin Seven proposed openstack-infra/project-config master: Remove legacy pyghmi job https://review.openstack.org/513249 | 03:59 |
| openstackgerrit | Tovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy pyghmi job https://review.openstack.org/513250 | 04:00 |
| ianw | pabelanger: hmm, good question. i'm unsure for short term use, i'll ask around and get back to you | 04:00 |
| pabelanger | ianw: danke | 04:00 |
| ianw | i imagine like everything phones, it's a confuseopoly with no clear winner | 04:00 |
| *** markvoelker has joined #openstack-infra | 04:02 | |
| ianw | pabelanger: if there was anywhere to read-up it would be http://forums.whirlpool.net.au/forum/114 | 04:02 |
| pabelanger | kk | 04:02 |
| ianw | https://www.telstra.com.au/mobile-phones/prepaid-mobiles/10-prepaid-multi-fit-sim-starter-kit seems alright | 04:04 |
| pabelanger | cool | 04:05 |
| pabelanger | I'll see if avaiable at airport | 04:06 |
| *** markvoelker has quit IRC | 04:06 | |
| pabelanger | better! | 04:07 |
| pabelanger | http://logs.openstack.org/68/512868/4/check/tox-cover/40e47e3/ | 04:07 |
| ianw | pabelanger: yep, thanks for looking in on that, want to just push it to fix the jobs? | 04:07 |
| pabelanger | ianw: +3 | 04:09 |
| pabelanger | okay, not I EOD | 04:09 |
| pabelanger | later | 04:09 |
| *** markvoelker has joined #openstack-infra | 04:10 | |
| *** gongysh has quit IRC | 04:12 | |
| *** markvoelker has quit IRC | 04:15 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Give layout objects a unique ID https://review.openstack.org/513207 | 04:18 |
| *** huanxie has quit IRC | 04:19 | |
| *** markvoelker has joined #openstack-infra | 04:20 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Correct path to coverage job https://review.openstack.org/513236 | 04:20 |
| *** namnh has quit IRC | 04:21 | |
| *** namnh has joined #openstack-infra | 04:21 | |
| *** psachin has joined #openstack-infra | 04:22 | |
| *** ijw has joined #openstack-infra | 04:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Add _projects to convert project list to dictionary https://review.openstack.org/512868 | 04:26 |
| *** ijw has quit IRC | 04:28 | |
| *** claudiub has joined #openstack-infra | 04:31 | |
| *** huanxie has joined #openstack-infra | 04:36 | |
| *** jbadiapa_ has joined #openstack-infra | 04:43 | |
| *** jbadiapa has quit IRC | 04:46 | |
| *** kiennt26 has quit IRC | 04:46 | |
| *** salv-orlando has joined #openstack-infra | 04:47 | |
| *** salv-orlando has quit IRC | 04:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Move to dictionary list of projects zuul._projects https://review.openstack.org/513260 | 04:52 |
| *** jbadiapa_ has quit IRC | 04:53 | |
| *** markvoelker has quit IRC | 04:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Add _projects to convert project list to dictionary https://review.openstack.org/512868 | 04:55 |
| *** armax has quit IRC | 04:55 | |
| *** armax has joined #openstack-infra | 04:56 | |
| *** armax has quit IRC | 04:56 | |
| *** armax has joined #openstack-infra | 04:57 | |
| *** armax has quit IRC | 04:57 | |
| *** armax has joined #openstack-infra | 04:58 | |
| *** armax has quit IRC | 04:58 | |
| *** armax has joined #openstack-infra | 04:58 | |
| *** armax has quit IRC | 04:59 | |
| *** armax has joined #openstack-infra | 04:59 | |
| *** armax has quit IRC | 04:59 | |
| *** sree has joined #openstack-infra | 04:59 | |
| *** armax has joined #openstack-infra | 05:00 | |
| *** armax has quit IRC | 05:00 | |
| *** armax has joined #openstack-infra | 05:01 | |
| *** armax has quit IRC | 05:01 | |
| *** armax has joined #openstack-infra | 05:02 | |
| *** armax has quit IRC | 05:02 | |
| *** armax has joined #openstack-infra | 05:02 | |
| *** armax has quit IRC | 05:03 | |
| *** armax has joined #openstack-infra | 05:03 | |
| *** salv-orlando has joined #openstack-infra | 05:03 | |
| *** armax has quit IRC | 05:03 | |
| *** armax has joined #openstack-infra | 05:04 | |
| *** armax has quit IRC | 05:04 | |
| *** markvoelker has joined #openstack-infra | 05:08 | |
| *** markvoelker has quit IRC | 05:12 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add management queue length to status page https://review.openstack.org/513262 | 05:12 |
| *** Goneri has joined #openstack-infra | 05:15 | |
| *** masber has joined #openstack-infra | 05:15 | |
| *** markvoelker has joined #openstack-infra | 05:17 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add management queue length to new status page https://review.openstack.org/513263 | 05:17 |
| *** masuberu has joined #openstack-infra | 05:17 | |
| *** masber has quit IRC | 05:19 | |
| *** markvoelker has quit IRC | 05:22 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add management event queue length to status json https://review.openstack.org/513182 | 05:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Merge tenant reconfiguration events https://review.openstack.org/513195 | 05:26 |
| *** markvoelker has joined #openstack-infra | 05:26 | |
| dirk | mordred: AJaeger : jeblair : pabelanger : interesting zuul failure here https://review.openstack.org/387799 | 05:27 |
| *** gongysh has joined #openstack-infra | 05:27 | |
| *** markvoelker has quit IRC | 05:29 | |
| *** markvoelker has joined #openstack-infra | 05:29 | |
| *** gongysh has quit IRC | 05:32 | |
| *** thorst has joined #openstack-infra | 05:42 | |
| *** markvoelker has quit IRC | 05:42 | |
| *** markvoelker has joined #openstack-infra | 05:42 | |
| *** CrayZee has quit IRC | 05:42 | |
| *** markvoelker has quit IRC | 05:42 | |
| *** thorst has quit IRC | 05:47 | |
| *** spectr has joined #openstack-infra | 05:48 | |
| *** ijw has joined #openstack-infra | 05:50 | |
| *** alex_xu has quit IRC | 05:54 | |
| *** florianf has joined #openstack-infra | 05:54 | |
| *** bhavik1 has joined #openstack-infra | 06:00 | |
| *** alex_xu has joined #openstack-infra | 06:00 | |
| *** huanxie has quit IRC | 06:01 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Re-enable test_zuul_trigger_project_change_merged https://review.openstack.org/513267 | 06:02 |
| *** hashar has joined #openstack-infra | 06:03 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Remove unused function toList in scheduler https://review.openstack.org/513268 | 06:04 |
| *** aeng has quit IRC | 06:07 | |
| AJaeger | dirk: tox-cover was broken, should be fixed by now | 06:08 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Remove unused function toList from scheduler https://review.openstack.org/513268 | 06:10 |
| *** andreas_s has joined #openstack-infra | 06:11 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/openstack-zuul-jobs master: make openstack-tox-pypy run tox pypy https://review.openstack.org/513271 | 06:14 |
| *** dhinesh has quit IRC | 06:15 | |
| *** yolanda has quit IRC | 06:15 | |
| *** martinkopec has joined #openstack-infra | 06:16 | |
| namnh | AJaeger: could you take a look at this patch [1] and give me a comment about whether we should separate post.yaml or not. thanks in advance. | 06:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: v2 Config is frozen, make job gating https://review.openstack.org/513272 | 06:19 |
| namnh | AJaeger: [1] https://review.openstack.org/#/c/509348 | 06:20 |
| *** bhavik1 has quit IRC | 06:21 | |
| *** e0ne has joined #openstack-infra | 06:23 | |
| *** jamesmcarthur has joined #openstack-infra | 06:25 | |
| *** salv-orlando has quit IRC | 06:26 | |
| *** salv-orlando has joined #openstack-infra | 06:26 | |
| AJaeger | namnh: commented | 06:28 |
| *** jamesmcarthur has quit IRC | 06:30 | |
| AJaeger | project-config-core, openstack-zuul-jobs is currently broken - needs new zuul-sphinx release. If urgent fixes are needed, we can pin zuul-sphinx to version 0.1.6. I expect jeblair to tag 0.2.1 later today and therefore would just wait. | 06:30 |
| *** salv-orlando has quit IRC | 06:30 | |
| namnh | AJaeger: thanks, I am reading your comment. | 06:32 |
| *** huanxie has joined #openstack-infra | 06:33 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Remove requirements related jobs https://review.openstack.org/513275 | 06:38 |
| *** jtomasek has joined #openstack-infra | 06:42 | |
| *** jtomasek has quit IRC | 06:43 | |
| *** jtomasek has joined #openstack-infra | 06:43 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-cross-*-* jobs related to global requirements https://review.openstack.org/513278 | 06:46 |
| *** ijw has quit IRC | 06:48 | |
| *** markvoelker has joined #openstack-infra | 06:49 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Remove requirements related legacy jobs https://review.openstack.org/513275 | 06:50 |
| *** kiennt26 has joined #openstack-infra | 06:51 | |
| *** rossella_s has quit IRC | 06:52 | |
| *** rossella_s has joined #openstack-infra | 06:54 | |
| *** pcaruana has joined #openstack-infra | 06:55 | |
| *** jpich has joined #openstack-infra | 06:59 | |
| *** gildub has quit IRC | 07:01 | |
| *** dizquierdo has joined #openstack-infra | 07:07 | |
| *** gildub has joined #openstack-infra | 07:15 | |
| *** dhinesh has joined #openstack-infra | 07:16 | |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** dhinesh has quit IRC | 07:20 | |
| *** tesseract has joined #openstack-infra | 07:21 | |
| *** jbadiapa has joined #openstack-infra | 07:24 | |
| *** salv-orlando has joined #openstack-infra | 07:26 | |
| *** salv-orlando has quit IRC | 07:31 | |
| *** jpena|off is now known as jpena | 07:31 | |
| *** e0ne has quit IRC | 07:33 | |
| *** amoralej|off is now known as amoralej | 07:33 | |
| *** gildub has quit IRC | 07:34 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Tooz https://review.openstack.org/513296 | 07:38 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Tooz https://review.openstack.org/513297 | 07:38 |
| *** salv-orlando has joined #openstack-infra | 07:42 | |
| *** thorst has joined #openstack-infra | 07:43 | |
| *** pas-ha has joined #openstack-infra | 07:43 | |
| pas-ha | AJaeger: I'm fine with https://review.openstack.org/#/c/511265/, please merge to save infra resources from double jobs on ironic patches :) | 07:44 |
| *** andreas_s has quit IRC | 07:45 | |
| *** gridinv has joined #openstack-infra | 07:46 | |
| *** thorst has quit IRC | 07:47 | |
| *** kaisers has joined #openstack-infra | 07:49 | |
| AJaeger | pas-ha: thanks. And regarding the openstack-zuul-jobs one: recheck it once 511265 is in - it cannot merge if a project will use the defintions, so we cannot break ironic-inspector | 07:51 |
| *** eumel8 has joined #openstack-infra | 07:51 | |
| eumel8 | morning | 07:51 |
| *** andreas_s has joined #openstack-infra | 07:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove ironic legacy jobs https://review.openstack.org/511265 | 07:54 |
| *** gcb has quit IRC | 07:55 | |
| evrardjp | morning | 07:57 |
| *** ccamacho has joined #openstack-infra | 07:57 | |
| *** markvoelker has quit IRC | 08:01 | |
| *** ralonsoh has joined #openstack-infra | 08:03 | |
| evrardjp | AJaeger: the job we discussed yesterday (post-merge trigger to propose-updates job) doesn't seem to trigger, there is no logs in http://logs.openstack.org/periodic/propose-updates/ | 08:08 |
| evrardjp | but I guess I am checking in the wrong folder | 08:08 |
| evrardjp | because it's not a periodic trigger | 08:08 |
| evrardjp | should I check directly in my job details, to find if the trigger fires? | 08:08 |
| *** tmorin has joined #openstack-infra | 08:10 | |
| evrardjp | maybe is there another way to check at the logs? | 08:10 |
| tmorin | hi infraroot, could somebody have a look at https://review.openstack.org/#/c/512332/ ? .... (job blocking merges in a project, change to make it non-voting), already blessed by AJaeger and neutron liaison | 08:11 |
| *** lucas-afk is now known as lucasagomes | 08:16 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-(keystone|mariadb) https://review.openstack.org/513022 | 08:20 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-(keystone|mariadb) to zuul.d https://review.openstack.org/513023 | 08:20 |
| *** efoley has joined #openstack-infra | 08:27 | |
| *** rossella_s has quit IRC | 08:28 | |
| *** gcb has joined #openstack-infra | 08:29 | |
| pas-ha | AJaeger: I'm making a patch to inspector that re-uses the new job definition from ironic tree, seems to work already. I'll ask sambetts to add it as depends-on. We'd also need to walk thru all ironic-related projects (and nova) to update the jobs to use the new one defined in ironic tree, and only then merge that patch to openstack-zuul-jobs | 08:30 |
| *** rossella_s has joined #openstack-infra | 08:30 | |
| *** yamamoto has quit IRC | 08:32 | |
| *** dtantsur|afk is now known as dtantsur | 08:32 | |
| AJaeger | evrardjp: post-jobs are different, see https://docs.openstack.org/infra/manual/developers.html#post-processing how to get log files | 08:33 |
| evrardjp | thanks | 08:34 |
| evrardjp | will skim through that | 08:34 |
| *** yolanda has joined #openstack-infra | 08:34 | |
| AJaeger | pas-ha: thanks! Yes, that's the way forward - remove it from project-config then as well | 08:34 |
| *** rossella_s has quit IRC | 08:36 | |
| *** rossella_s has joined #openstack-infra | 08:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: n8g-bgpvpn: make triple-o job check/nv https://review.openstack.org/512332 | 08:41 |
| *** armaan has joined #openstack-infra | 08:45 | |
| *** derekh has joined #openstack-infra | 08:46 | |
| *** rossella_s has quit IRC | 08:46 | |
| *** spectr has quit IRC | 08:47 | |
| *** rossella_s has joined #openstack-infra | 08:47 | |
| *** spectr has joined #openstack-infra | 08:48 | |
| *** spectr has quit IRC | 08:48 | |
| *** spectr has joined #openstack-infra | 08:50 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in networking-calico https://review.openstack.org/513316 | 08:53 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in networking-calico https://review.openstack.org/513317 | 08:53 |
| *** dingyichen has quit IRC | 08:53 | |
| *** electrofelix has joined #openstack-infra | 08:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use -neutron template variants for networking-sfc jobs https://review.openstack.org/512244 | 08:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Don't run py35 jobs on newton networking-arista changes https://review.openstack.org/513165 | 09:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Update neutron-lbaas to use -neutron templates https://review.openstack.org/513208 | 09:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Use -neutron template variants for neutron-fwaas job https://review.openstack.org/513212 | 09:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Create a playbook for publishing contributor-guide https://review.openstack.org/512865 | 09:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Publish keystone loci images to DockerHub https://review.openstack.org/512793 | 09:09 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Clarify patchsets for new project configuration https://review.openstack.org/512607 | 09:13 |
| *** yamamoto has joined #openstack-infra | 09:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: networking-odl: Removing legacy jobs references https://review.openstack.org/512645 | 09:14 |
| *** yolanda has quit IRC | 09:16 | |
| *** namnh has quit IRC | 09:16 | |
| *** namnh has joined #openstack-infra | 09:17 | |
| *** tosky has joined #openstack-infra | 09:17 | |
| AJaeger | frickler: could you put https://review.openstack.org/#/c/512524/ on your low-prio review queue, please? | 09:17 |
| *** yolanda has joined #openstack-infra | 09:18 | |
| tosky | hi, zuul commented on (at least) two already merged reviews that "Merge Failed. etc etc" - https://review.openstack.org/508847 and https://review.openstack.org/508906 | 09:18 |
| tosky | no harm is done, but: is it a known issue? | 09:19 |
| frickler | tosky: if I read backlog correctly, that was a mistake during restarting zuul last night | 09:19 |
| tosky | ah, oki | 09:19 |
| frickler | tosky: should be safe to ignore these | 09:19 |
| tosky | sure, they were already in the "annoying only" bucket; now they are in the "ignore" bucket | 09:20 |
| *** e0ne has joined #openstack-infra | 09:20 | |
| frickler | AJaeger: there is no such thing ;) but seeing your earlier comment we cannot merge this currently anyway, is that correct? | 09:21 |
| *** salv-orlando has quit IRC | 09:21 | |
| *** salv-orlando has joined #openstack-infra | 09:22 | |
| *** armaan has quit IRC | 09:23 | |
| *** armaan has joined #openstack-infra | 09:23 | |
| *** sambetts|afk is now known as sambetts | 09:24 | |
| AJaeger | frickler: you're right - we cannot merge right now. Thanks for +2 - I'll +A later then... | 09:25 |
| *** jamesmcarthur has joined #openstack-infra | 09:25 | |
| *** salv-orlando has quit IRC | 09:27 | |
| *** pas-ha has left #openstack-infra | 09:28 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-(keystone|mariadb) https://review.openstack.org/513022 | 09:29 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-(keystone|mariadb) to zuul.d https://review.openstack.org/513023 | 09:29 |
| frickler | AJaeger: was the issue about tagging releases resolved? I may have missed that in backlog | 09:29 |
| *** jamesmcarthur has quit IRC | 09:30 | |
| *** panda|rover|off is now known as panda|rover | 09:31 | |
| *** e0ne has quit IRC | 09:33 | |
| *** rossella_s has quit IRC | 09:35 | |
| AJaeger | frickler: At least jeblair was able to push zuul-sphinx 0.2.0 out... But I don't know whether that's all... | 09:35 |
| *** sree_ has joined #openstack-infra | 09:36 | |
| *** sree_ is now known as Guest44289 | 09:36 | |
| *** rossella_s has joined #openstack-infra | 09:37 | |
| *** sree has quit IRC | 09:37 | |
| *** sdague has joined #openstack-infra | 09:38 | |
| * mordred is about to do a talk but should be back online to help with things soonish | 09:42 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Support upper-constraints in tox-siblings https://review.openstack.org/513199 | 09:42 |
| *** e0ne has joined #openstack-infra | 09:43 | |
| mordred | jeblair, fungi: ^^ updated jeblair's patch with an approach to filtering out the conflicting constraints when we do an install | 09:43 |
| *** thorst has joined #openstack-infra | 09:44 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix mixed canonical/non-canonical project merge https://review.openstack.org/513331 | 09:48 |
| *** thorst has quit IRC | 09:49 | |
| *** yamamoto has quit IRC | 09:49 | |
| *** yolanda has quit IRC | 09:51 | |
| *** armaan has quit IRC | 09:55 | |
| *** armaan has joined #openstack-infra | 09:55 | |
| *** Douhet has quit IRC | 09:56 | |
| *** Douhet has joined #openstack-infra | 09:56 | |
| *** andreas_s has quit IRC | 09:57 | |
| *** andreas_s has joined #openstack-infra | 09:58 | |
| *** boden has joined #openstack-infra | 10:01 | |
| *** andreas_s has quit IRC | 10:02 | |
| *** kiennt26 has quit IRC | 10:04 | |
| *** cuongnv has quit IRC | 10:07 | |
| *** rossella_s has quit IRC | 10:07 | |
| *** namnh has quit IRC | 10:08 | |
| *** andreas_s has joined #openstack-infra | 10:08 | |
| *** namnh has joined #openstack-infra | 10:09 | |
| *** rossella_s has joined #openstack-infra | 10:10 | |
| sambetts | AJaeger: can I get some eyes on https://review.openstack.org/#/c/512588 ? the networking-cisco CI is comepletely broken right now and its blocking a number of patches | 10:14 |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Use -neutron template variants for neutron-fwaas-dashboard job https://review.openstack.org/513336 | 10:17 |
| *** andreas_s has quit IRC | 10:18 | |
| *** lihi has quit IRC | 10:20 | |
| *** lihi has joined #openstack-infra | 10:20 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Use -neutron template variants for neutron-vpnaas-dashboard job https://review.openstack.org/513338 | 10:21 |
| *** rossella_s has quit IRC | 10:21 | |
| *** salv-orlando has joined #openstack-infra | 10:21 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix mixed canonical/non-canonical project merge https://review.openstack.org/513331 | 10:22 |
| *** andreas_s has joined #openstack-infra | 10:23 | |
| *** rossella_s has joined #openstack-infra | 10:23 | |
| *** iyamahat has joined #openstack-infra | 10:26 | |
| AJaeger | mordred: do you have time to look at set-service-type-data-fact role, please? It's now completely broken, see http://logs.openstack.org/cc/cccb086dce8638764beb35063dde9f7fb036dd3d/post/publish-api-ref/f7b0011/ | 10:28 |
| *** yamahata has quit IRC | 10:28 | |
| *** pbourke has quit IRC | 10:29 | |
| *** andreas_s has quit IRC | 10:30 | |
| *** andreas_s has joined #openstack-infra | 10:30 | |
| AJaeger | sambetts: commented | 10:31 |
| *** pbourke has joined #openstack-infra | 10:31 | |
| *** iyamahat has quit IRC | 10:32 | |
| * AJaeger will be offline for the next few hours... | 10:32 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Use -neutron template variants for neutron-vpnaas-dashboard job https://review.openstack.org/513338 | 10:32 |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Use -horizon template variants for neutron-vpnaas-dashboard job https://review.openstack.org/513338 | 10:33 |
| *** rossella_s has quit IRC | 10:33 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Use -horizon template variants for neutron-fwaas-dashboard job https://review.openstack.org/513336 | 10:36 |
| *** rossella_s has joined #openstack-infra | 10:36 | |
| *** Guest44289 has quit IRC | 10:37 | |
| *** ldnunes has joined #openstack-infra | 10:37 | |
| *** sree has joined #openstack-infra | 10:37 | |
| *** gildub has joined #openstack-infra | 10:38 | |
| openstackgerrit | Sam Betts proposed openstack-infra/openstack-zuul-jobs master: Add the correct branch overrides to the networking-cisco jobs https://review.openstack.org/513339 | 10:39 |
| *** namnh has quit IRC | 10:40 | |
| openstackgerrit | Merged openstack-dev/cookiecutter master: Fix docs.o.o URLs https://review.openstack.org/501755 | 10:45 |
| *** rgerganov has quit IRC | 10:45 | |
| *** huanxie has quit IRC | 10:49 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Remove networking-generic-jobs https://review.openstack.org/513347 | 10:49 |
| *** yamamoto has joined #openstack-infra | 10:50 | |
| *** yamamoto has quit IRC | 10:53 | |
| *** yamamoto has joined #openstack-infra | 10:53 | |
| *** yamamoto has quit IRC | 10:53 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/openstack-zuul-jobs master: Remove networking-generic-switch job https://review.openstack.org/513349 | 10:54 |
| mnaser | is it possible there are issues with zuul-merger? | 10:56 |
| mnaser | ..sigh nvm, the new filtering in zuulv3 doesnt show the entire tree | 10:58 |
| openstackgerrit | Merged openstack-dev/pbr master: Use 'build_reno' setuptools extension if available https://review.openstack.org/475034 | 10:58 |
| mnaser | so there were jobs ahead | 10:58 |
| *** jpena is now known as jpena|lunch | 10:59 | |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| *** sree has quit IRC | 11:05 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Remove networking-generic-switch jobs https://review.openstack.org/513347 | 11:06 |
| *** rhallisey has joined #openstack-infra | 11:10 | |
| *** smatzek has joined #openstack-infra | 11:12 | |
| *** tpsilva has joined #openstack-infra | 11:16 | |
| *** kjackal_ has joined #openstack-infra | 11:20 | |
| *** lucasagomes is now known as lucas-hungry | 11:22 | |
| *** markvoelker has joined #openstack-infra | 11:27 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 11:30 | |
| *** armaan has quit IRC | 11:31 | |
| *** armaan has joined #openstack-infra | 11:32 | |
| *** nicolasbock has joined #openstack-infra | 11:32 | |
| *** smatzek has quit IRC | 11:33 | |
| *** martinkopec has quit IRC | 11:34 | |
| openstackgerrit | Sam Betts proposed openstack-infra/openstack-zuul-jobs master: Add the correct branch overrides to the networking-cisco jobs https://review.openstack.org/513339 | 11:36 |
| *** nicolasbock has quit IRC | 11:38 | |
| *** boden has quit IRC | 11:41 | |
| mnaser | could we cut a bugfix release for zuul_sphinx which includes the fix jeblair added yesterday? https://review.openstack.org/#/c/513201/ | 11:43 |
| *** armaan has quit IRC | 11:43 | |
| mnaser | TIL "Show Change Number in Changes Table" in Gerrit's preferences.. why isn't this on by default! | 11:45 |
| sambetts | mnaser: :O thats awesome | 11:46 |
| mnaser | no idea why its not the default.. | 11:46 |
| *** nicolasbock has joined #openstack-infra | 11:50 | |
| *** edmondsw has joined #openstack-infra | 11:51 | |
| *** eharney has joined #openstack-infra | 11:52 | |
| *** salv-orlando has quit IRC | 11:54 | |
| *** sree has joined #openstack-infra | 11:56 | |
| openstackgerrit | Andreas Scheuring proposed openstack/diskimage-builder master: Add zipl element as s390x architecture bootloader https://review.openstack.org/443548 | 11:56 |
| *** rcernin has joined #openstack-infra | 11:58 | |
| *** sree has quit IRC | 12:01 | |
| *** jpena|lunch is now known as jpena | 12:02 | |
| *** dbecker has joined #openstack-infra | 12:09 | |
| *** stakeda has quit IRC | 12:09 | |
| *** ldnunes has quit IRC | 12:10 | |
| *** ldnunes has joined #openstack-infra | 12:10 | |
| mnaser | i'm not +W-ing anything for a little bit, but if I miss something in my code reviews, please yell at me :) | 12:11 |
| *** dave-mccowan has joined #openstack-infra | 12:12 | |
| *** thorst has joined #openstack-infra | 12:13 | |
| *** psachin has quit IRC | 12:13 | |
| *** dprince has joined #openstack-infra | 12:14 | |
| *** salv-orlando has joined #openstack-infra | 12:15 | |
| *** dizquierdo has quit IRC | 12:18 | |
| *** jtomasek has quit IRC | 12:22 | |
| *** jtomasek has joined #openstack-infra | 12:22 | |
| smcginnis | Anyone know the gerrit username our release jobs should use? | 12:25 |
| *** trown|outtypewww is now known as trown | 12:26 | |
| smcginnis | We're manually setting the git info to have gitreview.username of "review" but the logs end up with this: | 12:26 |
| smcginnis | http://logs.openstack.org/4b/4b221eecbe3a6350031381475635cbfd6e7ba1b2/release-post/tag-releases/a335cd7/job-output.txt.gz#_2017-10-19_03_13_04_303815 | 12:26 |
| *** wolverineav has joined #openstack-infra | 12:26 | |
| *** dtantsur is now known as dtantsur|brb | 12:26 | |
| smcginnis | From what I understood, everything was set up outside the jobs before, so I'm assuming "review" isn't the right username to set there. | 12:26 |
| mnaser | smcginnis dont release jobs run on a trusted server? | 12:28 |
| mnaser | and i think in zuulv3 this sort of thing is gone and replaced by jobs that define secrets in their config | 12:29 |
| mnaser | ok, lp_creds and ssh_key secrets are in the job config, my bad | 12:29 |
| *** rosmaita has joined #openstack-infra | 12:30 | |
| *** jamesdenton has joined #openstack-infra | 12:30 | |
| mnaser | hmm, i think nothing is creating those files however in the job | 12:31 |
| mnaser | yeah, nothing is consuming those secrets, they're just defined | 12:31 |
| smcginnis | mnaser: My understanding was this was all preconfigured before. But then Doug had to add this: https://review.openstack.org/#/c/512331/ | 12:33 |
| smcginnis | And other "setup" type steps. | 12:33 |
| *** e0ne_ has joined #openstack-infra | 12:34 | |
| * mnaser looking | 12:34 | |
| mnaser | yeah i'm not seeing anywhere that the secrets are consumed (aka: ssh key and credentials are created locally) | 12:34 |
| mnaser | so i think what is happening is the ssh private key is not there, so gerrit is unable to authenticate because the private key that it is using doesnt assosciate to a user | 12:35 |
| *** hashar has quit IRC | 12:35 | |
| *** hashar has joined #openstack-infra | 12:36 | |
| smcginnis | So is there a step we need to add to the jobs? Or the scripts we use within our release tools? | 12:36 |
| * smcginnis is still learning, Doug is the expert here. | 12:36 | |
| mnaser | smcginnis no problem. so before, the release jobs ran on a 'trusted' node that had a pre-installed ssh key and launchpad credentials. they did not run on a one-time server like all the other jobs do | 12:37 |
| *** e0ne has quit IRC | 12:37 | |
| mnaser | zuulv3 introduces the concept of secrets, which means that we can store encrypted data in the config of the job and be able to access it securely. this means that there is no point in having a "trusted" node, because any node can become trusted once it consumes the secrets and configures itself | 12:38 |
| mnaser | so what has to be done is the ssh private key from the secret should be installed, as well as the lp credentials, to get you in the same previous environment as that trusted node.. | 12:38 |
| mnaser | https://docs.openstack.org/infra/zuul/user/jobs.html#secrets | 12:38 |
| smcginnis | mnaser: OK, so this should be done in the job definition, if I'm following right. | 12:40 |
| mnaser | that seems to been already done in the job definition - https://github.com/openstack-infra/project-config/blob/57647cbd73b4bb871a7fdbb3423e31f4f53631f0/zuul.d/jobs.yaml#L623-L633 | 12:40 |
| mnaser | im looking at the job and seeing if it is doing what its supposed to do | 12:40 |
| mnaser | ok, it does call add-sshkey and add-launchpad-credentials (which i assume consume the secerts) ... let me see how that works | 12:41 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Remove networking-ovn legacy jobs https://review.openstack.org/513367 | 12:41 |
| mnaser | https://github.com/openstack-infra/project-config/blob/master/playbooks/release/pre.yaml (as you see here) | 12:41 |
| *** yolanda has joined #openstack-infra | 12:41 | |
| mnaser | ok it does seem to write things out properly .. as far as i can tell | 12:43 |
| *** LindaWang has quit IRC | 12:43 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add regex support to project stanzas https://review.openstack.org/513368 | 12:43 |
| mnaser | the only thing is i guess double checking that the correct private key is the one encrypted, other than that i am not sure :( | 12:43 |
| *** jaosorior has joined #openstack-infra | 12:43 | |
| *** lucas-hungry is now known as lucasagomes | 12:43 | |
| *** pblaho has quit IRC | 12:43 | |
| *** mriedem has joined #openstack-infra | 12:44 | |
| *** pblaho has joined #openstack-infra | 12:45 | |
| *** jcoufal has joined #openstack-infra | 12:46 | |
| *** rlandy has joined #openstack-infra | 12:47 | |
| smcginnis | mnaser: Could it be a mismatch between that key and the "release" gerrit account? | 12:50 |
| mnaser | smcginnis if its what you mean (to be more precise), i think its a mismatch between the private key encrypted in the repo (which is being installed on the vm), and the public key for the "release" gerrit account | 12:52 |
| mnaser | smcginnis https://github.com/openstack-infra/project-config/commit/056cfbb7930b91d4be5fbfeb8cb9d9ab07a4f9c9 looks like it was recently changed | 12:52 |
| smcginnis | mnaser: Yeah, basically that's what I was saying. Either we need to set it to something other than "release" as the gitreview.username, or there's a different key we should be using. At least that's my initial hunch. | 12:53 |
| smcginnis | Not sure how we check that with gerrit though. | 12:53 |
| mnaser | smcginnis do you know if that change above was submitted before or after your failure? | 12:53 |
| mnaser | not enough coffee to do timezones + math | 12:54 |
| smcginnis | ;) | 12:54 |
| smcginnis | mnaser: Yeah, it looks like settings and key were updated with this patch right around then: https://review.openstack.org/#/c/513166/2 | 12:55 |
| mnaser | smcginnis also, i dont think gitreview.username is needed | 12:55 |
| smcginnis | That one is a project-config change ^ | 12:55 |
| *** hemna_ has joined #openstack-infra | 12:55 | |
| smcginnis | There was another similar git config change in our release tools that did not have the gitreview.username line. | 12:55 |
| mnaser | afaik the logic is: when you run git review, it attempts to access gerrit with your private key installed in the system | 12:56 |
| mnaser | and using that, gerrit can identify who you are | 12:57 |
| *** gildub has quit IRC | 12:57 | |
| smcginnis | So maybe by actually specifying a name that is different than what matches on the key, it doesn't like it and prompts for a name? | 12:58 |
| *** amoralej is now known as amoralej|lunch | 12:58 | |
| *** dizquierdo has joined #openstack-infra | 12:59 | |
| *** StevenK has quit IRC | 12:59 | |
| *** pallen has joined #openstack-infra | 12:59 | |
| *** gildub has joined #openstack-infra | 13:00 | |
| *** ihrachys has joined #openstack-infra | 13:00 | |
| *** bh526r has joined #openstack-infra | 13:00 | |
| *** pallen has left #openstack-infra | 13:00 | |
| mnaser | smcginnis thats a valid theory actually | 13:05 |
| dhellmann | mnaser, smcginnis : the values we're using now were copied directly from the static node that used to run this job yesterday | 13:05 |
| dhellmann | I guess they could still be wrong somehow, but fungi dumped that info and re-encrypted the key to put into that patch | 13:06 |
| mnaser | is the release bot username not release-bot by any chance | 13:06 |
| * mnaser is looking at gerrit users | 13:06 | |
| smcginnis | Hmm. Any initial ideas dhellmann? | 13:06 |
| *** LindaWang has joined #openstack-infra | 13:06 | |
| dhellmann | the release user is the user with permission to push tags, so that's correct. maybe the key is wrong. | 13:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add regex support to project stanzas https://review.openstack.org/513368 | 13:06 |
| *** tmorin has quit IRC | 13:06 | |
| dhellmann | mnaser : oh, maybe? I could have fat-fingered that part | 13:06 |
| mnaser | https://review.openstack.org/#/admin/groups/11,members | 13:07 |
| mnaser | unfortunately it doesnt say the exact user | 13:07 |
| mnaser | but we have proposal-bot for proposal bot (afaik) | 13:07 |
| dhellmann | this is what I was given yesterday: http://paste.openstack.org/show/624012/ | 13:08 |
| dhellmann | proposal-bot does not have permission to push tags | 13:08 |
| mnaser | dhellmann yeah but i was speculating on the possible username but yeah i see it now its release | 13:08 |
| dhellmann | so maybe the key is bad? Is there some way to have git-review give more debugging info? | 13:09 |
| mnaser | the reason why i suspect they key is bad is - "Could not connect to gerrit." then it asks for a username | 13:09 |
| *** tmorin has joined #openstack-infra | 13:09 | |
| mnaser | pretty much sounds like "i cant ssh to gerrit at the port, please give me auth" | 13:09 |
| dhellmann | makes sense to me | 13:10 |
| * mnaser goes through git-review | 13:10 | |
| mnaser | https://github.com/openstack-infra/git-review/blob/master/git_review/cmd.py#L401-L402 | 13:10 |
| mnaser | https://github.com/openstack-infra/git-review/blob/master/git_review/cmd.py#L366-L367 | 13:10 |
| mnaser | looks like it fails to do a dry run push, but also it has verbose output | 13:10 |
| dhellmann | that key was also copied off of the signing node yesterday | 13:10 |
| dhellmann | how do we turn on verbose mode? | 13:11 |
| *** camunoz has joined #openstack-infra | 13:11 | |
| *** baoli has joined #openstack-infra | 13:11 | |
| mnaser | dhellmann https://github.com/openstack-infra/git-review/blob/master/git_review/cmd.py#L1467-L1468 -v | 13:11 |
| smcginnis | Does that key need to be added to the release gerrit account? | 13:11 |
| mnaser | i'd add it, it'll give us the output of the dry run push and hopefully that is a bit more explicit | 13:11 |
| dhellmann | smcginnis : it *should* be the key that's already there | 13:11 |
| mnaser | ^ | 13:11 |
| dhellmann | mnaser : patch incoming | 13:11 |
| *** baoli has quit IRC | 13:12 | |
| *** yolanda has quit IRC | 13:13 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add verbose flag to git-review when setting up repos for tag-releases https://review.openstack.org/513377 | 13:13 |
| *** sbezverk has joined #openstack-infra | 13:13 | |
| dhellmann | I wish there was some way to test this without having to wait to run it in CI | 13:15 |
| dhellmann | I mean, I can run the scripts as myself, but that doesn't help with this problem and I don't have the key to run it as the release user | 13:15 |
| *** StevenK has joined #openstack-infra | 13:16 | |
| smcginnis | Yeah, we need some kind of "pipecleaner" job. | 13:16 |
| mnaser | dhellmann yeah.. the cycle for testing these jobs is rough :( | 13:16 |
| *** StevenK is now known as Guest25792 | 13:16 | |
| mnaser | i see the number of revert/revert that smcginnis is doing and ouch :p | 13:16 |
| dhellmann | yeah, we've been at this 3 days. It takes an hour to test a small change. | 13:17 |
| *** gildub has quit IRC | 13:17 | |
| mnaser | i think a contributing factor is that post jobs run at lower priority | 13:17 |
| smcginnis | At least an hour... | 13:17 |
| *** Guest25792 is now known as StevenK | 13:18 | |
| mnaser | looks like zuulv3 is swapping | 13:19 |
| dhellmann | I'm around for a little while to give it one more go, but then I have to make up for the fact that I've met none of my other obligations this week, so I hope someone else is able to step in and look at it. | 13:19 |
| *** baoli has joined #openstack-infra | 13:19 | |
| smcginnis | I'm going to try, but I definitely don't have anywhere near the depth yet. | 13:19 |
| mnaser | i'm hoping its just a bad ssh private key | 13:20 |
| smcginnis | Let's hope this is it. | 13:20 |
| mnaser | i'll try to follow up on this so feel free to ping / add me on any related reviews smcginnis dhellmann | 13:20 |
| *** rcernin has quit IRC | 13:21 | |
| smcginnis | mnaser: Thanks | 13:21 |
| mnaser | now we wait for some infra-root to check out zuul and it's swapping :( | 13:21 |
| fungi | ahh, so we pushed the scheduler up over ram again? | 13:24 |
| fungi | looks like we only just started swapping heavily in the last 15 minutes or so | 13:25 |
| fungi | mordred: thanks for picking up the tox-siblings work! | 13:26 |
| fungi | the ssh private key should be fine as should the username. both are garnered from the old persistent worker where the prior equivalents of these jobs ran | 13:29 |
| dhellmann | could that command be running from the wrong directory? so it can't find the .gitreview file or git repo? | 13:30 |
| fungi | just a sec and i'll see if there's a log linked in scrollback i can look at. i've just been skimming since it's about 400 comments | 13:31 |
| *** tmorin1 has joined #openstack-infra | 13:31 | |
| dhellmann | fungi : http://logs.openstack.org/4b/4b221eecbe3a6350031381475635cbfd6e7ba1b2/release-post/tag-releases/a335cd7/job-output.txt.gz#_2017-10-19_03_13_04_303815 | 13:31 |
| *** tmorin has quit IRC | 13:31 | |
| dhellmann | the cd into the repo directory doesn't fail, so I don't think that's the problem | 13:31 |
| fungi | i guess http://logs.openstack.org/4b/4b221eecbe3a6350031381475635cbfd6e7ba1b2/release-post/tag-releases/a335cd7/ is a good place to start | 13:31 |
| fungi | dhellmann: ahh, thanks, just found it in scrollback too | 13:32 |
| fungi | taking a look | 13:32 |
| *** LindaWang has quit IRC | 13:33 | |
| *** LindaWang has joined #openstack-infra | 13:33 | |
| dhellmann | if we land https://review.openstack.org/513377 can we re-queue the job that failed without going through the revert/revert dance to save some time? | 13:33 |
| fungi | yeah | 13:34 |
| *** rcernin has joined #openstack-infra | 13:35 | |
| *** amoralej|lunch is now known as amoralej | 13:36 | |
| fungi | though i'm not done tracing through the log and job definition, so no clue what 513377 is yet | 13:36 |
| fungi | still catching up | 13:37 |
| *** rosmaita has quit IRC | 13:39 | |
| smcginnis | dhellmann: Should we add "--global" to the git config calls just to be safe? | 13:39 |
| dhellmann | smcginnis : we could try that, I'm not sure whether there's a difference in local vs. global other than the obvious scope (like is something assuming the values will be set globally) | 13:40 |
| *** kgiusti has joined #openstack-infra | 13:40 | |
| smcginnis | dhellmann: Just thinking in case we are in a directory we don't expect to be, or we missed setting that for a specific repo. | 13:40 |
| *** rcernin has quit IRC | 13:40 | |
| dhellmann | hmm | 13:41 |
| *** andreas_s has quit IRC | 13:41 | |
| dhellmann | sure, belt-and-braces | 13:41 |
| *** andreas_s has joined #openstack-infra | 13:42 | |
| *** eumel8 has quit IRC | 13:42 | |
| *** rosmaita has joined #openstack-infra | 13:42 | |
| fungi | so... the add-sshkey task doesn't appear to set restrictive permissions. this could easily cause the ssh client to ignore/skip it, but i would expect a more catastrophic failure bubbling up from git-review under those circumstances... i'm wishing we had an easy way to add debugging info to these jobs without actually triggering releases | 13:42 |
| *** jrist_ has joined #openstack-infra | 13:43 | |
| dhellmann | we could use a release of the release-test repo | 13:43 |
| dhellmann | that's set up to push all the way through to pypi I think | 13:43 |
| *** jrist has quit IRC | 13:43 | |
| fungi | i'd like to inject some commands before the `git review -s | 13:43 |
| fungi | ` | 13:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add node list to webapp https://review.openstack.org/513387 | 13:43 |
| dhellmann | I'll set up the release-test deliverable data so we have a job to retry that won't affect a real repository | 13:43 |
| fungi | mainly to directly test connecting to gerrit with extra verbosity, directory listing of ~/.ssh, et cetera | 13:44 |
| *** annp has joined #openstack-infra | 13:44 | |
| *** felipemonteiro has joined #openstack-infra | 13:44 | |
| fungi | this is running the release-tools/release_from_yaml.sh script in project-config presumably. patch on the way | 13:45 |
| *** felipemonteiro__ has joined #openstack-infra | 13:45 | |
| *** andreas_s has quit IRC | 13:46 | |
| dhellmann | fungi, smcginnis : test release: https://review.openstack.org/513388 | 13:47 |
| *** mriedem1 has joined #openstack-infra | 13:47 | |
| dhellmann | fungi : you could update https://review.openstack.org/#/c/513377/ with more detail if you want | 13:48 |
| dhellmann | that was also meant to add more debugging detail | 13:48 |
| *** andreas_s has joined #openstack-infra | 13:48 | |
| *** mriedem has quit IRC | 13:48 | |
| fungi | still trying to find where the git-review -s call is | 13:48 |
| dhellmann | look at that patch, I modified that call | 13:48 |
| dhellmann | I added the -v option mnaser suggested | 13:48 |
| *** mriedem1 is now known as mriedem | 13:48 | |
| fungi | oh, you already have a patch for this. perfect | 13:48 |
| *** felipemonteiro has quit IRC | 13:49 | |
| fungi | want me to add further debugging to 513377 or as a child patch? | 13:50 |
| dhellmann | go ahead and modify the patch directly so we only have to land 1 | 13:50 |
| dhellmann | waiting for all of these test patches to land is driving me nuts | 13:50 |
| smcginnis | Power glitch here but I'm back. Looks like a good plan so far. | 13:51 |
| dhellmann | I guess after we move the job definition to the releases repo we'll be able to land a test and code change at the same time | 13:51 |
| *** andreas_s has quit IRC | 13:53 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Add verbose flag to git-review for tag-releases https://review.openstack.org/513377 | 13:56 |
| fungi | dhellmann: smcginnis: ^ | 13:56 |
| *** shiyaz has joined #openstack-infra | 13:56 | |
| fungi | hopefully that gets us more useful data points to rule out possible issues | 13:56 |
| fungi | mnaser: look good to you? | 13:57 |
| dhellmann | fungi : would it be useful for that ssh call to have -v? or would that spew too much info? | 13:57 |
| fungi | done | 13:57 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Add verbose flag to git-review for tag-releases https://review.openstack.org/513377 | 13:57 |
| *** andreas_s has joined #openstack-infra | 13:59 | |
| fungi | okay, that should be in momentarily if the swapping scheduler isn't actually slowing things down | 14:00 |
| frickler | fungi: commented on PS2 but should still be relevant | 14:00 |
| fungi | frickler: oh, thanks, looking | 14:00 |
| fungi | frickler: it's the configure git-review step that's failing | 14:00 |
| fungi | so we're trying to see what the initial state is before it's attempted | 14:01 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for use-cached-repos https://review.openstack.org/512927 | 14:01 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 14:01 |
| frickler | fungi: ah, right, that should be fine, then | 14:01 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 14:01 |
| fungi | the tests being added before we run git-review -s don't rely on git-review configuration | 14:01 |
| fungi | we're basically testing/checking the things git-review is going to rely on when it runs | 14:02 |
| dhellmann | we should probably add something to clone_repo to cause it to return an error code if the clone fails | 14:02 |
| dhellmann | it looks like it always returns 0 now? | 14:02 |
| *** hongbin has joined #openstack-infra | 14:03 | |
| dhellmann | oh, nm, it runs set -e | 14:03 |
| *** lbragstad has joined #openstack-infra | 14:04 | |
| fungi | as observed before yesterday's scheduler restart, swapping there seems not especially detrimental... you can see the spikes on the swapping activity graph subside quickly, implying the allocations paged out are infrequently accessed anyway | 14:06 |
| *** esberglu has joined #openstack-infra | 14:07 | |
| *** andreas_s has quit IRC | 14:08 | |
| *** baoli has quit IRC | 14:09 | |
| *** baoli has joined #openstack-infra | 14:09 | |
| *** baoli has quit IRC | 14:09 | |
| *** baoli has joined #openstack-infra | 14:11 | |
| *** andreas_s has joined #openstack-infra | 14:11 | |
| *** kjackal_ has quit IRC | 14:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: v2 Config is frozen, make job gating https://review.openstack.org/513272 | 14:14 |
| *** tmorin1 has quit IRC | 14:15 | |
| fungi | though the current spike in swapping activity is making the status.json take too long to serve i think, so apache is timing it out and displaying a proxy error | 14:18 |
| *** tmorin has joined #openstack-infra | 14:19 | |
| smcginnis | dhellmann, fungi: Once that change lands, are we doing the test release or were you going to just reenqueue the process fungi? | 14:20 |
| *** Swami has joined #openstack-infra | 14:20 | |
| *** gouthamr has joined #openstack-infra | 14:20 | |
| dhellmann | whichever | 14:20 |
| dhellmann | I guess it might be better to test with the release-test repo in case something fails half way | 14:21 |
| frickler | fungi: looks like that last project-config merge might have tripped zuul over with starting another round of reconfigure-all-the-things | 14:21 |
| *** LindaWang has quit IRC | 14:22 | |
| fungi | perhaps. the swapping spike is slacking off now though | 14:23 |
| *** LindaWang has joined #openstack-infra | 14:23 | |
| fungi | and i'm getting status.json responses agani | 14:24 |
| fungi | there is a nice event/result queue backlog now | 14:24 |
| fungi | so this is likely it catching back up | 14:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Add verbose flag to git-review for tag-releases https://review.openstack.org/513377 | 14:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Use -horizon template variants for neutron-vpnaas-dashboard job https://review.openstack.org/513338 | 14:25 |
| fungi | dhellmann: smcginnis: 513377 merged, so we should be able to test the release sandbox tagging now? | 14:26 |
| fungi | and hopefully get a little more information on why git-review is unhappy | 14:26 |
| *** shiyaz has left #openstack-infra | 14:26 | |
| *** gmann is now known as gmann_afk | 14:26 | |
| *** rbrndt has joined #openstack-infra | 14:27 | |
| dhellmann | ok, I'll approve that | 14:27 |
| dhellmann | waiting for https://review.openstack.org/#/c/513388/2 | 14:27 |
| jeblair | if we're swapping, the server isn't going to last for long | 14:28 |
| *** mat128 has joined #openstack-infra | 14:28 | |
| fungi | agreed, looks like the most recent swapping activity spike never quite subsided, so we likely need another restart | 14:30 |
| fungi | system load is finally up to ~1 with all the iowait | 14:30 |
| smcginnis | Any way we can bump the priorities on these release jobs? Today is supposed to be the q-1 release deadline and we keep hitting roadblock after roadblock. | 14:31 |
| fungi | smcginnis: release jobs already get the top priority | 14:31 |
| fungi | at least the pre-release, release, tag and release-post pipelines do | 14:31 |
| fungi | as well as the gate pipeline | 14:32 |
| smcginnis | fungi: Oh, OK. Didn't realize that. | 14:32 |
| fungi | (we only have three priorities to choose from though, owing to gearman protocol heritage) | 14:32 |
| fungi | check pipelines get medium priority, and regular post and periodic are low priority | 14:34 |
| jeblair | fungi: do you have a copy of the queues from when status.json was responsive? | 14:35 |
| fungi | jeblair: it looks responsive now | 14:35 |
| fungi | i'm still able to get responses from the status page anyway | 14:36 |
| jeblair | there we go, it just moved again | 14:36 |
| *** armax has joined #openstack-infra | 14:36 | |
| jeblair | okay, i have a copy of the queues | 14:36 |
| fungi | was hoping we'd gain ground on the events queue before restarting, but i guess it's only rising, not falling | 14:37 |
| *** trown is now known as trown|brb | 14:38 | |
| fungi | anyway, 513388 seems to have made it into check so it'll get reenqueued | 14:38 |
| *** iyamahat has joined #openstack-infra | 14:38 | |
| *** spectr has quit IRC | 14:38 | |
| fungi | oh! it worked its way through all the events/results now | 14:39 |
| smcginnis | zuul caught up?? | 14:39 |
| fungi | depends on your definition of caught up, but it managed to enqueue anything it had an event for at least | 14:40 |
| jeblair | i'd like to grab an object graph now, which will kill it for a few minutes, and then restart it. | 14:40 |
| fungi | should i get another status dump now that the events queue is empty? | 14:41 |
| jeblair | fungi: i got one | 14:41 |
| fungi | ahh, thanks | 14:41 |
| *** gyee has joined #openstack-infra | 14:42 | |
| jeblair | i'm pretty sure the most recent slowdown was enough to upset the zk connection, so it may already be reporting bogus errors | 14:42 |
| *** jamesmcarthur has joined #openstack-infra | 14:42 | |
| *** jbadiapa has quit IRC | 14:43 | |
| jeblair | my plan today is to address that, as well as a potential area for memory reduction i saw yesterday | 14:43 |
| fungi | ooh, neat! | 14:43 |
| *** dizquierdo has quit IRC | 14:43 | |
| *** iyamahat has quit IRC | 14:45 | |
| *** trown|brb is now known as trown | 14:46 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config master: Zuul-v3: add required projects for neutron-lib periodic jobs https://review.openstack.org/512811 | 14:47 |
| jeblair | hrm, maybe we should just restart it without getting the object graph | 14:48 |
| *** dizquierdo has joined #openstack-infra | 14:49 | |
| jeblair | we can get back to work, and hopefully i can grab one before we hit swap next time | 14:49 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config master: Zuul-v3: add required projects for neutron-lib periodic jobs https://review.openstack.org/512811 | 14:49 |
| jeblair | fungi, smcginnis: zuul is restarting now; you can approve/recheck any changes and jump the queues | 14:50 |
| *** nikhil has joined #openstack-infra | 14:50 | |
| fungi | thanks jeblair! | 14:50 |
| jeblair | oh wait, sorry i have to restart again | 14:50 |
| fungi | okay | 14:51 |
| jeblair | (it reinstalled over my local patches last night) | 14:51 |
| jeblair | fungi, smcginnis: okay, starting again. you can approve/recheck | 14:51 |
| smcginnis | jeblair: It's ready for a recheck? | 14:52 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Rename ironic job for nova https://review.openstack.org/513410 | 14:52 |
| jeblair | smcginnis: yep | 14:52 |
| smcginnis | Thanks | 14:52 |
| dhellmann | fungi , smcginnis : I rechecked https://review.openstack.org/#/c/513388/2 | 14:52 |
| jeblair | (the status page won't be responsive yet, but the event listeners are running) | 14:52 |
| fungi | i'm sticking the releases test change into the gate anyway | 14:52 |
| dhellmann | fungi : ++ | 14:53 |
| smcginnis | A triple whammy. | 14:53 |
| * dhellmann imagines zuul pounding the patch into the repo with a hammer | 14:53 | |
| fungi | one of those oversized wooden mochi hammers | 14:53 |
| *** yamamoto has quit IRC | 14:54 | |
| jeblair | re-enqueueing saved queues | 14:54 |
| dhellmann | https://www.youtube.com/watch?v=Nxls1KnKCA4 | 14:55 |
| *** e0ne_ has quit IRC | 14:56 | |
| fungi | 513388,2 has nodes assigned in the gate now, so should merge in a few minutes | 14:56 |
| dhellmann | it's weird to see it in the check and gate queue at the same time :-) | 14:57 |
| dmsimard | dhellmann: that happens either when we manually bypass the check queue and enqueue directly to the gate -- or if you see a change in a pipeline with no jobs, that's just because it hasn't been determined yet if the change should be enqueued in that pipeline | 14:59 |
| *** iyamahat has joined #openstack-infra | 14:59 | |
| dhellmann | dmsimard : the latter case I've seen before. I guess the former happens infrequently enough that I haven't noticed. | 14:59 |
| dmsimard | it's quite exceptional that the check queue is bypassed, yes :) | 15:00 |
| *** e0ne has joined #openstack-infra | 15:00 | |
| AJaeger | dmsimard: do you have time to look at set-service-type-data-fact role, please? It's now completely broken, see http://logs.openstack.org/cc/cccb086dce8638764beb35063dde9f7fb036dd3d/post/publish-api-ref/f7b0011/ - the get_url is the problem | 15:02 |
| fungi | okay, release-post commit enqueued now | 15:02 |
| dmsimard | fungi, jeblair: I've successfully tested the sqlite middleware patch for ara on logs.rdoproject.org. Were we interested in testing it out on logs-dev ? I can work on the puppet-openstackci bits. | 15:02 |
| *** xarses has joined #openstack-infra | 15:02 | |
| *** jaosorior has quit IRC | 15:02 | |
| dmsimard | Also, an implementation detail but I suppose it's better to consider installing ara on a virtualenv ? | 15:02 |
| fungi | dmsimard: sure! | 15:02 |
| fungi | to logs-dev i mean... less sure about virtualenv vs system wide | 15:03 |
| *** jrist_ is now known as jrist | 15:03 | |
| *** jrist has quit IRC | 15:03 | |
| *** jrist has joined #openstack-infra | 15:03 | |
| dmsimard | fungi: ara just brings (unfortunately) a lot of dependencies which is why I bring up the option, I've also tested the wsgi middleware with a virtualenv. | 15:04 |
| *** iyamahat has quit IRC | 15:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use -horizon template variants for neutron-fwaas-dashboard job https://review.openstack.org/513336 | 15:05 |
| AJaeger | jeblair: could you tag zuul-sphinx 0.2.1, please? We cannot merge anything to openstack-zuul-jobs currently | 15:06 |
| dmsimard | AJaeger: looking | 15:07 |
| *** iyamahat has joined #openstack-infra | 15:07 | |
| *** baoli has quit IRC | 15:07 | |
| *** baoli has joined #openstack-infra | 15:08 | |
| *** baoli has quit IRC | 15:08 | |
| jeblair | AJaeger: oh, sorry, i thought i just broke new functionality. tagging now. | 15:08 |
| *** dtantsur|brb is now known as dtantsur | 15:08 | |
| jeblair | AJaeger: tag pushed | 15:09 |
| AJaeger | jeblair: thanks! | 15:10 |
| AJaeger | dmsimard: thanks | 15:11 |
| *** baoli has joined #openstack-infra | 15:11 | |
| dmsimard | AJaeger: we need to use result.dest instead of result.src, I'll send a patch | 15:11 |
| AJaeger | dmsimard: ah! thanks | 15:12 |
| *** ricky_laptop has joined #openstack-infra | 15:12 | |
| *** markvoelker has quit IRC | 15:12 | |
| *** markvoelker has joined #openstack-infra | 15:13 | |
| AJaeger | wow, 73 changes in gate queue - most from tripleo | 15:13 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Use dest instead of src for get_url downloaded resource https://review.openstack.org/513414 | 15:14 |
| dmsimard | AJaeger: ^ | 15:14 |
| dmsimard | need to close laptop for a bit, out of battery and no power outlet nearby :) | 15:14 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: ansible-role-k8s-cookiecutter to zuul.d/projects https://review.openstack.org/512330 | 15:15 |
| flaper87 | AJaeger: dmsimard ^ :D | 15:15 |
| smcginnis | dhellmann, fungi: Did the post job work that time? I don't see errors. | 15:15 |
| dhellmann | looking | 15:16 |
| AJaeger | dmsimard: thanks! | 15:16 |
| fungi | smcginnis: dhellmann: it broke after trying to cat ~/.gitconfig which apparently did not exist | 15:16 |
| fungi | and we have set -e | 15:16 |
| fungi | i suppose i should have unset that before the diagnostics | 15:17 |
| fungi | and reset it after | 15:17 |
| smcginnis | Bah | 15:17 |
| *** dhinesh has joined #openstack-infra | 15:17 | |
| *** nicolasbock has quit IRC | 15:17 | |
| *** markvoelker has quit IRC | 15:17 | |
| * smcginnis must be looking at the wrong log. | 15:17 | |
| dhellmann | why didn't that cause the job to report a failure | 15:17 |
| *** nicolasbock has joined #openstack-infra | 15:18 | |
| fungi | good question... are we running a script set -e from another script which is not set -e and performs additional tasks afterward? | 15:19 |
| * dhellmann looks | 15:20 | |
| *** dhinesh has quit IRC | 15:21 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-(keystone|mariadb) https://review.openstack.org/513022 | 15:21 |
| dhellmann | fungi : release.sh seems to set -e | 15:22 |
| *** nicolasbock has quit IRC | 15:22 | |
| dhellmann | oh, the master job script ignores errors | 15:23 |
| fungi | well, unless the final command run returns nonzero | 15:24 |
| *** jamesmcarthur has quit IRC | 15:26 | |
| *** jamesmcarthur has joined #openstack-infra | 15:26 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: accumulate errors and fail tag-releases if we have any https://review.openstack.org/513417 | 15:27 |
| dhellmann | fungi : ^^ | 15:27 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add _projects to convert project list to dictionary https://review.openstack.org/512868 | 15:28 |
| *** andreas_s has quit IRC | 15:28 | |
| smcginnis | Aren't we going to have the same issue yet though. Need to fix the cat of .gitconfig first. | 15:28 |
| dhellmann | oh, true | 15:30 |
| fungi | yeah, we also need to run those debug commands withouty set -e (which i didn't realize would propagate into the inlined subshell or i would have) | 15:30 |
| *** jamesmcarthur has quit IRC | 15:31 | |
| *** dhajare has joined #openstack-infra | 15:32 | |
| smcginnis | fungi: Do you have the one fungi? | 15:32 |
| *** andreas_s has joined #openstack-infra | 15:33 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: accumulate errors and fail tag-releases if we have any https://review.openstack.org/513417 | 15:33 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: avoid errors in debug commands in clone_repo https://review.openstack.org/513419 | 15:33 |
| dhellmann | let's try ^^ | 15:33 |
| dhellmann | fungi , smcginnis : ^^ | 15:33 |
| smcginnis | dhellmann: Nice! | 15:34 |
| fungi | smcginnis: sorry, getting sidetracked by tc office hour | 15:37 |
| *** andreas_s has quit IRC | 15:37 | |
| fungi | looking now | 15:37 |
| smcginnis | fungi: Yeah, way too many distractions and multitasking needed this week (or past few weeks). | 15:39 |
| smcginnis | Looking forward to being able to focus on one thing at a time some day. | 15:39 |
| fungi | i basically didn't engage in or even read any of the tc campaigning ml threads due to it being zuultime all the time | 15:40 |
| *** martinkopec has joined #openstack-infra | 15:41 | |
| smcginnis | Definitely lower on the priority list. | 15:41 |
| *** martinkopec has quit IRC | 15:41 | |
| fungi | was hoping to at least get around to reading everyone's platforms, but need to vote by tomorrow i guess | 15:42 |
| *** andreas_s has joined #openstack-infra | 15:42 | |
| smcginnis | fungi: Just do like most people and vote for the names you recognize. ;) | 15:43 |
| AJaeger | ;) | 15:43 |
| *** martinkopec has joined #openstack-infra | 15:44 | |
| *** andreas_s has quit IRC | 15:47 | |
| openstackgerrit | Merged openstack-infra/project-config master: avoid errors in debug commands in clone_repo https://review.openstack.org/513419 | 15:47 |
| fungi | smcginnis: luckily, i know most of the candidates personally and have worked closely with them on various things, so i feel like i already know what they stand for even if i don't get an opportunity to read their formal prose on the matter | 15:49 |
| *** dbecker has quit IRC | 15:49 | |
| clarkb | ugh slow start today... the weather has definitely transitioned from summer to fall now | 15:50 |
| *** eumel8 has joined #openstack-infra | 15:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: accumulate errors and fail tag-releases if we have any https://review.openstack.org/513417 | 15:50 |
| *** annp has quit IRC | 15:50 | |
| clarkb | jeblair: are there any zuul changes that need review? | 15:51 |
| jeblair | clarkb: https://review.openstack.org/513209 and https://review.openstack.org/513262 are the top | 15:52 |
| jeblair | i'm pretty close to pushing up another memory-reducing change | 15:52 |
| clarkb | AJaeger: do you know if that proposal job is happier since the key update? | 15:52 |
| dhellmann | fungi , smcginnis : it looks like we're ready to re-queue that release-test tag attempt now that those script changes have merged? | 15:53 |
| fungi | clarkb: jeblair: also mordred picked up the remaining constraints support work for tox-siblings (513199) | 15:54 |
| smcginnis | Was that it or did we have one more patch to go in first. I'm losing track. | 15:54 |
| smcginnis | Oh, nope. | 15:54 |
| smcginnis | Looks like we are good to try again. | 15:54 |
| fungi | smcginnis: dhellmann: they both merged, so yes we are clear to retry now | 15:54 |
| *** yamamoto has joined #openstack-infra | 15:55 | |
| smcginnis | fungi: Can you do the reenqueue magic? | 15:55 |
| *** tmorin has quit IRC | 15:55 | |
| fungi | yup. what was the test change this time? | 15:56 |
| fungi | oh, just reenqueue the post jobs | 15:56 |
| fungi | yeah, on it | 15:56 |
| smcginnis | fungi: Need the review #? | 15:57 |
| fungi | i need the merge commit sha actually, but i'm getting it now | 15:57 |
| clarkb | oh also I have replacement network gear arriving today, so at some point I'll likely drop off the internet while I replace gear and get my desktop connected again. such fun | 15:57 |
| smcginnis | clarkb: Ooh, infrastructure upgrades are always fun. :) | 15:58 |
| *** jamesmcarthur has joined #openstack-infra | 15:59 | |
| *** eumel8 has left #openstack-infra | 16:00 | |
| *** Guest17047 has quit IRC | 16:00 | |
| fungi | dhellmann: smcginnis: reenqueued that last releases merge into release-post | 16:00 |
| clarkb | jeblair: I've approved the buildsets change. It looks like we mostly weren't using that data anyways | 16:00 |
| *** trown is now known as trown|lunch | 16:00 | |
| clarkb | jeblair: eg the reach of that change was not very far | 16:01 |
| AJaeger | clarkb: haven't looked yet ;( | 16:01 |
| clarkb | smcginnis: in this case I had a switch/AP decide to lose significant numbers of packets so unplanned upgrade but excited to have things working again | 16:03 |
| *** jgriffith is now known as groot | 16:03 | |
| *** groot is now known as jgriffith | 16:04 | |
| clarkb | AJaeger: were those periodic jobs against eg nova? | 16:04 |
| * clarkb goes looking for logs | 16:04 | |
| dhellmann | fungi : thanks | 16:04 |
| AJaeger | clarkb: http://logs.openstack.org/periodic/git.openstack.org/openstack-infra/project-config/master/propose-project-config-update/665cd61/ara/ | 16:04 |
| *** jgriffith is now known as groot | 16:05 | |
| AJaeger | clarkb: that one fails again - now with "Permission denied (publickey)." - on the ssh run | 16:05 |
| * AJaeger will be back online later | 16:05 | |
| clarkb | AJaeger: ok so the warning about the host key thing did go away, so that is an improvment but not the one we want | 16:05 |
| clarkb | fungi: I think your idea that perms might be wrong on the private key is worth following up on | 16:06 |
| clarkb | ara says the mode on the file is 0600, could be the .ssh dir instead? | 16:06 |
| *** ianychoi has quit IRC | 16:06 | |
| fungi | clarkb: we already have debugging evidence to suggest that the perms are fine | 16:07 |
| clarkb | oh thats interesting | 16:07 |
| *** ianychoi has joined #openstack-infra | 16:07 | |
| clarkb | ara says the size of the file is 891 | 16:07 |
| clarkb | but looking back at the old slave node its 1679 | 16:07 |
| fungi | there's much more extensive debugging happening in a job running in release-post right now | 16:07 |
| smcginnis | Just finished | 16:08 |
| *** iyamahat has quit IRC | 16:08 | |
| clarkb | jeblair: when you have a moment, what sort of debugging do we expose around secrets? | 16:08 |
| fungi | new results at http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/cece9e2/job-output.txt.gz | 16:08 |
| clarkb | fungi: see http://logs.openstack.org/periodic/git.openstack.org/openstack-infra/project-config/master/propose-project-config-update/665cd61/ara/result/a8c6b15e-a335-4467-a297-1ebdddc1eba6/ that size is definitely much smaller than expected | 16:09 |
| dhellmann | it looks like the ssh key isn't working for connecting to gerrit? | 16:09 |
| fungi | gerrit ssh says "Permission denied (publickey)." | 16:09 |
| clarkb | dhellmann: ya Ithink we may be writing incomplete or wrong contents based on the ara link above | 16:09 |
| *** andreas_s has joined #openstack-infra | 16:09 | |
| *** martinkopec has quit IRC | 16:09 | |
| *** lucasagomes is now known as lucas-afk | 16:09 | |
| fungi | also ls -l says id_rsa is only 887 bytes long | 16:09 |
| dhellmann | this is one of the keys that's a list in the yaml file. is the writing code taking that into account? | 16:10 |
| fungi | which confirms clarkb's suspicion | 16:10 |
| jeblair | clarkb: it's, erm, *very* difficult to debug secrets. they get written to a tmpfs only accessible inside the jail. we can't even look at the decrypted form on disk. | 16:10 |
| thingee | hey infra, can I get this approved, it already has 2 +2's https://review.openstack.org/#/c/512871/6 | 16:10 |
| jeblair | clarkb: yay security | 16:10 |
| smcginnis | My local id_rsa is twice that size. | 16:11 |
| clarkb | smcginnis: ya it should be about twice that size | 16:11 |
| *** yamamoto has quit IRC | 16:11 | |
| *** salv-orl_ has joined #openstack-infra | 16:11 | |
| dhellmann | do we have another job that uses secrets we could compare to this one? | 16:11 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: periodic-neutron-dsvm-functional runs only on master https://review.openstack.org/512524 | 16:11 |
| dhellmann | (another that works :-) | 16:11 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix space character for bash command https://review.openstack.org/513435 | 16:11 |
| clarkb | thingee: done | 16:11 |
| clarkb | dhellmann: SamYaple's is working but I bet his secrets are shorter (which may lend credibility to your theory) | 16:12 |
| jeblair | clarkb: i do have the repl installed; i could try to convince the running zuul to decrypt it in memory and see what we get | 16:12 |
| *** Swami has quit IRC | 16:12 | |
| *** Apoorva has joined #openstack-infra | 16:12 | |
| EmilienM | dmsimard: when we migrated to zuulv3, did you change something in the shared-queues configuration? it seems like tripleo & puppet queues are shared - I'm not sure it was the case before (or maybe?) | 16:12 |
| EmilienM | weshay|ruck: ^ fyi asking here | 16:12 |
| clarkb | jeblair: or maybe we just need a test that decrypts a larger value to confirm that works generally? (assuming we don't have one) | 16:12 |
| *** pas-ha has joined #openstack-infra | 16:12 | |
| *** dizquierdo has quit IRC | 16:12 | |
| clarkb | EmilienM: I believe they were shared before | 16:13 |
| smcginnis | Can we manually go into a host an run some of these to see what's being written out? Or is that not possible with the way the secrets are handled? | 16:13 |
| clarkb | smcginnis: ya we can, we coul dput a sleep in the job to make that easier too | 16:13 |
| clarkb | fungi: ^ | 16:13 |
| fungi | http://paste.openstack.org/show/624017/ which is what we embedded for this job, does seem to indicate that the plaintext length should be 1675 bytes and we're only a little over half that according to ls -l | 16:13 |
| EmilienM | clarkb: ok | 16:13 |
| *** pas-ha has left #openstack-infra | 16:13 | |
| EmilienM | mwhahaha: ^ sounds like queues were shared before, so we just need to be patient, and don't recheck if something fails, instead create elastic recheck queries. | 16:13 |
| mwhahaha | yea they've always been shared | 16:14 |
| *** andreas_s has quit IRC | 16:14 | |
| fungi | clarkb: yep, though i don't think the streamed console log emits the ip address any longer? not sure how we go about tracking the ip address of the running node down in v3 | 16:14 |
| *** salv-orlando has quit IRC | 16:14 | |
| weshay|ruck | EmilienM, mwhahaha so my team can go on a elsastic-recheck hunt | 16:14 |
| EmilienM | weshay|ruck: that's what we should do all the time. | 16:15 |
| weshay|ruck | create new queries as we see them | 16:15 |
| jeblair | fungi: i'd start with the build uuid and work backwards | 16:15 |
| clarkb | fungi: we'd have to grep the build uuid out of the executor debug logs | 16:15 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove old buildsets https://review.openstack.org/513209 | 16:15 |
| EmilienM | hunt failures and add queries | 16:15 |
| weshay|ruck | EmilienM, yes.. noted | 16:15 |
| EmilienM | my teacher pabelanger told me that | 16:15 |
| clarkb | fungi: I can help with thta if we want to get a change in to sleep | 16:15 |
| fungi | jeblair: clarkb: thanks. so i guess we need a _really_ long sleep in the script, in that case | 16:15 |
| dhellmann | why are "long" keys stored as a list? | 16:15 |
| weshay|ruck | EmilienM, I thought he was professor PB | 16:16 |
| EmilienM | :) | 16:16 |
| fungi | dhellmann: because pkcs-1 does not support encrypting data of arbitrary length | 16:16 |
| clarkb | fungi: for X in `seq -w 1 10`; do echo $X && ssh ze$X.openstack.org grep $uuid /var/log/zuul/executor-debug.log ; done | 16:16 |
| dhellmann | ah | 16:16 |
| EmilienM | weshay|ruck, mwhahaha: another thing I just noticed: THT isn't in the tripleo queue | 16:16 |
| EmilienM | we need to fix that | 16:16 |
| EmilienM | if you look at http://zuulv3.openstack.org/ (in gate) | 16:16 |
| fungi | dhellmann: and the public key standards which do support arbitrary-length data are not available as well-supported python modules | 16:17 |
| weshay|ruck | it's in it's own queue | 16:17 |
| jeblair | fungi, clarkb: sudo ansible 'ze*' -m shell -a 'grep UUID /var/log/zuul/executor-debug.log' | 16:17 |
| * weshay|ruck pokes | 16:17 | |
| *** dhajare has quit IRC | 16:17 | |
| clarkb | jeblair: I find the ansible output harder to grep/read but maybe thats just me | 16:18 |
| jeblair | (on puppetmaster) | 16:18 |
| *** dhinesh has joined #openstack-infra | 16:18 | |
| *** pcaruana has quit IRC | 16:18 | |
| fungi | clarkb: jeblair: or should we just set hold-on-failure for the tag-releases job for now and retrigger? | 16:18 |
| jeblair | clarkb: try that sometime. it's the same as your ssh loop | 16:18 |
| smcginnis | clarkb: Not just you. | 16:18 |
| jeblair | fungi: ++ | 16:18 |
| fungi | that seems far easier | 16:18 |
| clarkb | fungi: ya that may be sompler | 16:19 |
| jeblair | fungi: does the playbook delete the secret? | 16:19 |
| *** e0ne has quit IRC | 16:19 | |
| jeblair | er, i mean, ssh key | 16:19 |
| fungi | jeblair: i don't believe it wipes ~zuul/.ssh/id_rsa | 16:19 |
| fungi | at least not that i saw | 16:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Set contributor-guide location to /contributors https://review.openstack.org/512871 | 16:19 |
| *** markvoelker has joined #openstack-infra | 16:20 | |
| clarkb | it appears to remove the authorized keys file entry for the test env and the fileserver private key but not the release/proposal keys | 16:20 |
| fungi | how is autohold done? i don't see it mentioned by nodepool --help | 16:21 |
| *** Apoorva has quit IRC | 16:21 | |
| fungi | and nodepool hold --help seems to imply that's still just for holding explicit node ids | 16:21 |
| clarkb | I wonder if this is a yaml string quoting issue | 16:22 |
| *** Apoorva has joined #openstack-infra | 16:22 | |
| clarkb | (I've just rereviewed the decryption of list of chunks and that seems fine) | 16:22 |
| *** dbecker has joined #openstack-infra | 16:22 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Stop storing dependent items on buildsets https://review.openstack.org/513441 | 16:22 |
| *** dhinesh has quit IRC | 16:22 | |
| jeblair | fungi: 'zuul autohold' | 16:22 |
| *** jpich has quit IRC | 16:23 | |
| jeblair | on zuulv3.o.o | 16:23 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Support deploying extra nodes in te-broker https://review.openstack.org/512899 | 16:23 |
| fungi | ahh | 16:24 |
| fungi | right, nodepool lacks the context to know | 16:24 |
| fungi | thanks | 16:24 |
| *** markvoelker has quit IRC | 16:24 | |
| dhellmann | clarkb : there are some punctuation characters in these encoded strings; you might be onto something | 16:25 |
| dhellmann | I mean, I would have expected the yaml writer to deal with that, but... | 16:25 |
| fungi | okay, autohold set and commit reenqueued | 16:26 |
| clarkb | dhellmann: ya I think = + and / are all yaml clean? | 16:26 |
| fungi | shortly i'll hopefully know what's actually written to ~zuul/.ssh/id_rsa in that job | 16:26 |
| dhellmann | my embedded yaml parser is a bit rusty | 16:26 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/system-config master: Add result_TIMED_OUT & result_RETRY_LIMIT as failure colors https://review.openstack.org/513444 | 16:26 |
| clarkb | I guess I can really quickly test that /me does this | 16:26 |
| jlvillal | ^^^ Small patch to color those two failures as failures (red) | 16:27 |
| fungi | does nodepool.o.o no longer have the ability to list nodes? | 16:27 |
| jeblair | fungi: run from nl01 or nl02 | 16:28 |
| jeblair | fungi: nodepool.o.o is only v2 | 16:28 |
| *** andreas_s has joined #openstack-infra | 16:28 | |
| fungi | except the zookeeper part of v3? | 16:28 |
| fungi | or is that on nl0x now too? | 16:28 |
| jeblair | fungi: right :) | 16:28 |
| *** LindaWang has quit IRC | 16:28 | |
| jeblair | fungi: use 'nodepool list --detail' to get ip addresses | 16:28 |
| fungi | ooh, that was about to be my next question. thanks! | 16:29 |
| fungi | wondered where all the ips had gone | 16:29 |
| jeblair | fungi: so has arin | 16:29 |
| fungi | touché | 16:29 |
| fungi | long time passing | 16:30 |
| *** iyamahat has joined #openstack-infra | 16:30 | |
| clarkb | I think I see what could be the bug | 16:30 |
| clarkb | the newlines get transformed as spaces | 16:31 |
| clarkb | rather than nulls | 16:31 |
| * clarkb goes to make sure zuul isn't doing something to address that | 16:31 | |
| jeblair | clarkb: which newlines? | 16:31 |
| clarkb | jeblair: in eg http://paste.openstack.org/show/624017/ the end of the first line of ciphertext has a newline and that becomes a space when I ready that data in locally | 16:32 |
| clarkb | and each newline seems to do that | 16:32 |
| *** andreas_s has quit IRC | 16:32 | |
| clarkb | so it becomes x1+fTO5A9nI+9ZzmPkcnhdGjUl831xromQ8r/j5/boiWmfQ2Cw3NGD6B+KosOmWXmF8f/ kzVz7+/D2cBwNlBci4mrLvJHqYMaHUVCq32ciokJaC3vgEuX6qyb4UDYaCNdHMxVK7wFB | 16:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add management queue length to status page https://review.openstack.org/513262 | 16:33 |
| clarkb | rather than x1+fTO5A9nI+9ZzmPkcnhdGjUl831xromQ8r/j5/boiWmfQ2Cw3NGD6B+KosOmWXmF8f/kzVz7+/D2cBwNlBci4mrLvJHqYMaHUVCq32ciokJaC3vgEuX6qyb4UDYaCNdHMxVK7wFB | 16:33 |
| *** ralonsoh has quit IRC | 16:33 | |
| clarkb | I think we may want the > quote method? /me tests | 16:34 |
| *** iyamahat has quit IRC | 16:34 | |
| smcginnis | Does that work with the "!encrypted" bit there? | 16:35 |
| clarkb | hrm > isn't what I want | 16:35 |
| *** kiennt26 has joined #openstack-infra | 16:35 | |
| fungi | yay, my autohold worked | 16:36 |
| weshay|ruck | clarkb, is the elastic search indexing an issue in a queue atm? Looking at it now it's 59 hours behind | 16:37 |
| jeblair | clarkb, smcginnis: yeah, the !encrypted causes zuul to load that itself, so, clarkb, if you're just using a straight yaml parser, you may not be getting the same results. | 16:37 |
| jeblair | what's the name of the secret? i'd like to poke in zuul's memory | 16:38 |
| dhellmann | jeblair : release_ssh_key | 16:38 |
| jeblair | thx | 16:38 |
| clarkb | right but all its doing is self.ciphertext = [base64.b64decode(x.value) for x in ciphertext] so unless x.value is doing something magic I think it will have whitespace of some form | 16:38 |
| fungi | clarkb: confirmed, ~zuul/.ssh/id_rsa is waaay too small, but seems to include a complete (just very short?) rsa key which in no way matches the original key we encoded | 16:38 |
| jeblair | fungi: does it have a comment at the end? | 16:39 |
| smcginnis | fungi: So decryption error? | 16:39 |
| fungi | jeblair: -----END RSA PRIVATE KEY----- | 16:39 |
| *** yamamoto has joined #openstack-infra | 16:39 | |
| smcginnis | Oh, hmm. | 16:39 |
| fungi | it still has the start and end markers and looks like base64 data with the right column widths and the last line of the data looks to be normally padded too | 16:39 |
| fungi | it really seems like a test vector or something | 16:40 |
| fungi | ssh root@162.242.144.195 'cat ~zuul/.ssh/id_rsa' | 16:40 |
| clarkb | fungi: does the prefix of the file match what is on the old slvae? | 16:41 |
| clarkb | and/or suffix I guess | 16:42 |
| fungi | nope, other than the begin/end markers and the first few bytes of the key (which is typical across keys) | 16:42 |
| *** andreww has joined #openstack-infra | 16:42 | |
| clarkb | fwiw tests/fixtures/test_id_rsa has roughly the same length as that key | 16:43 |
| * smcginnis needs to AFK for a bit, be back later | 16:44 | |
| *** xarses has quit IRC | 16:44 | |
| *** bnemec has quit IRC | 16:45 | |
| *** bnemec has joined #openstack-infra | 16:46 | |
| fungi | this is the only id_rsa file on the filesystem too, so it's not like we're accidentally writing it to the wrong location | 16:46 |
| *** iyamahat has joined #openstack-infra | 16:46 | |
| *** Apoorva_ has joined #openstack-infra | 16:46 | |
| fungi | two possibilities: we're writing the wrong key or we're overwriting the key later with a different/wrong/new key | 16:46 |
| *** iyamahat_ has joined #openstack-infra | 16:47 | |
| jeblair | what's the name of the job? | 16:47 |
| fungi | um | 16:47 |
| fungi | jeblair: tag-releases | 16:47 |
| fungi | so... /home/zuul/.ssh/id_rsa.pub contains a public key claiming to belong to zuul@ze09.openstack.org | 16:47 |
| *** ricky_laptop has quit IRC | 16:47 | |
| jeblair | fungi: that's probably the per-build autogenerated key | 16:48 |
| clarkb | reading zuul tests we test the encryption and decryption and serialization routines but below the yaml loader | 16:48 |
| fungi | we autogenerate ssh keys for the zuul user on test nodes? is that so they can ssh to themselves on localhost or something | 16:48 |
| fungi | ? | 16:48 |
| clarkb | I'm going to work on a test at the yaml loader level | 16:48 |
| jeblair | clarkb: we should have a test that goes through the whole system with a big secret | 16:49 |
| * clarkb greps more | 16:49 | |
| *** yamahata has joined #openstack-infra | 16:49 | |
| jeblair | clarkb: if not, i'm certain we at least do it with a small secret; if that's all, maybe extend that one | 16:49 |
| *** Apoorva has quit IRC | 16:50 | |
| fungi | i don't think the /home/zuul/.ssh/id_rsa.pub on the node is intended for ssh access to itself, since it's not included in /home/zuul/.ssh/authorized_keys | 16:51 |
| jeblair | clarkb: TestAnsible has a small secret at least, so if you don't find a big one, maybe add something to that test. | 16:51 |
| jeblair | clarkb: test_model has a big secret and should be exercising this | 16:51 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Be explicit about legacy var additions/omissions https://review.openstack.org/512769 | 16:51 |
| *** jpena is now known as jpena|off | 16:52 | |
| *** Apoorva_ has quit IRC | 16:52 | |
| clarkb | jeblair: ya but its two short passwords appended together so possibly may not exercise things as it gets longer? | 16:53 |
| jeblair | fungi: we autogenerate keys for every build so that we don't expose the nodepool ssh key (or even ssh-agent access to it) to jobs which might then find a way to ssh to other running nodes | 16:53 |
| clarkb | (if that makes sense) | 16:53 |
| *** slaweq_ has joined #openstack-infra | 16:53 | |
| clarkb | I'll work to modify that test to at least confirm or deny this | 16:53 |
| *** Apoorva has joined #openstack-infra | 16:53 | |
| jeblair | clarkb: yep | 16:53 |
| fungi | jeblair: but why would we install that onto the node itself? | 16:53 |
| *** yamamoto has quit IRC | 16:53 | |
| jeblair | fungi: there are some roles that install the keys on all the build's nodes for cross-communication. i don't know where the dividing line is. | 16:54 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make executor hostname configurable https://review.openstack.org/512629 | 16:54 |
| *** ijw has joined #openstack-infra | 16:54 | |
| fungi | ahh, okay, so maybe this is bootstrapping for potential node-to-node communication | 16:54 |
| fungi | so anyway, the ~zuul/.ssh/id_rsa matches this id_rsa.pub file according to ssh-keygen -y -f | 16:56 |
| fungi | which explains why it looks like a fully-formed key but is not the length we expect | 16:56 |
| jeblair | fungi: oh, wait, the job is writing to ~zuul/.ssh/id_rsa ? | 16:57 |
| fungi | jeblair: yes, the job expects to ssh to gerrit and previously used the default ~/.ssh/id_rsa to do so under v2 | 16:57 |
| fungi | and we puppeted that key into place on the persistent slave | 16:58 |
| jeblair | fungi: hrm. well, i would expect the job to do that after whatever base roles might set up the per-build key | 16:58 |
| jeblair | so we may not be writing the key as expected | 16:58 |
| fungi | agreed, so either 1. the key is getting replaced with this after we write the one we expected, or 2. we're not actually writing the key at all | 16:58 |
| clarkb | btw the single entry pkcs values seem to use | quoting but the doubles don't. | 17:00 |
| fungi | openstack-infra/zuul-jobs/roles/add-sshkey/tasks/main.yaml is the one i think should be doing it | 17:00 |
| *** ijw has quit IRC | 17:00 | |
| *** ccamacho has quit IRC | 17:00 | |
| *** derekh has quit IRC | 17:00 | |
| *** trown|lunch is now known as trown | 17:00 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul supplementary groups before setgid/setuid https://review.openstack.org/508444 | 17:01 |
| fungi | ugh, the task output for that indicates the file size is 997 bytes | 17:01 |
| fungi | maybe we're referencing the wrong key variable in the job? | 17:01 |
| fungi | er, 887 bytes i mean | 17:02 |
| dhellmann | the job has "add-sshkey" but I don't see where it specifies *which* key? | 17:02 |
| *** ijw has joined #openstack-infra | 17:02 | |
| fungi | so this key is being written by that role/task | 17:02 |
| *** dizquierdo has joined #openstack-infra | 17:02 | |
| *** slaweq_ has quit IRC | 17:02 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Change domain in test fixture to example.org https://review.openstack.org/513448 | 17:02 |
| dhellmann | oh, that's done in jobs.yaml | 17:02 |
| *** slaweq_ has joined #openstack-infra | 17:03 | |
| odyssey4me | if I define required-projects in a job, but in a child of that job define another list of required projects - are both sets merged, or does the child override the parent? | 17:03 |
| jeblair | dhellmann: yeah, the secret is 'renamed' to ssh_key, which is what the role expects | 17:03 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Change domain in test fixture to example.com https://review.openstack.org/513448 | 17:03 |
| *** dtantsur is now known as dtantsur|afk | 17:04 | |
| jeblair | odyssey4me: merged | 17:04 |
| fungi | wonder if this is some weird argument inheritance order/precedence | 17:04 |
| *** sambetts is now known as sambetts|afk | 17:04 | |
| * dhellmann nods | 17:04 | |
| jeblair | fungi: oooooh | 17:04 |
| odyssey4me | jeblair oh awesome - no need to duplicate the values in all the children... happy days :) | 17:04 |
| *** andreas_s has joined #openstack-infra | 17:04 | |
| *** tesseract has quit IRC | 17:04 | |
| jeblair | fungi: like, colliding with the key name we use for the per-build key | 17:04 |
| *** felipemonteiro__ has quit IRC | 17:05 | |
| jeblair | fungi: hrm, that uses zuul_temp_ssh_key | 17:06 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add ref_url column to the buildset reporter table https://review.openstack.org/503714 | 17:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Change domain in test fixture to example.com https://review.openstack.org/513448 | 17:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add oldrev/newrev column to the buildset reporter table https://review.openstack.org/507954 | 17:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Disable sql_reporter alembic downgrades https://review.openstack.org/512159 | 17:07 |
| jeblair | fungi: so now that i'm looking at it, i can confirm that the base job does generate a key, and installs pub/private in ~zuul/.ssh on all nodes. i believe it's the multinode base job that then adds it to authorized_keys. | 17:07 |
| fungi | jeblair: yeah, so guessing we're reusing the parameters inherited from the base job when we call add-ssh-key a second time? | 17:08 |
| pabelanger | EmilienM: see https://review.openstack.org/513222/ | 17:08 |
| pabelanger | EmilienM: noticed that last night | 17:08 |
| fungi | er, add-sshkey | 17:08 |
| EmilienM | pabelanger: good | 17:08 |
| EmilienM | pabelanger: thx | 17:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove networking-generic-switch jobs https://review.openstack.org/513347 | 17:08 |
| notmyname | are job runners going to be using new-hotness ubuntu 17.10, or are they sticking with LTS releases? | 17:09 |
| jeblair | fungi: the base job doesn't use add-sshkey; i don't see any overlap | 17:09 |
| fungi | huh... | 17:09 |
| *** iyamahat_ has quit IRC | 17:09 | |
| fungi | takes me forever to find where jobs are defined now, so haven't gotten there yet | 17:10 |
| jeblair | the autodoc thing is going to be awesome :) | 17:11 |
| fungi | got it, http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n623 | 17:11 |
| jeblair | but in the mean time, the role the base job uses is zuul-jobs/roles/add-build-sshkey | 17:12 |
| *** slaweq_ has quit IRC | 17:12 | |
| *** shardy has quit IRC | 17:12 | |
| *** slaweq_ has joined #openstack-infra | 17:13 | |
| *** hashar is now known as hasharDinner | 17:13 | |
| fungi | jeblair: so it's added in the pre playbook here: http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/release/pre.yaml#n16 | 17:13 |
| fungi | do we need to explicitly pass the secret parameters there? | 17:14 |
| fungi | or should we be using a different role than add-sshkey for this? | 17:15 |
| jeblair | fungi: don't think so; the fact that the secret was named 'ssh_key' means it should just be available | 17:15 |
| frickler | hmm, do we need "copy:" instead of "file:" in the role? | 17:15 |
| clarkb | ok testing seems toshow that encoding back to base64 removes the whitespace (still not sure how that happens but seems to work) | 17:15 |
| pabelanger | notmyname: we usually add LTS versions of ubuntu to nodepool, then work to port jobs to the latest lts | 17:15 |
| frickler | I don't see a "content" option here http://docs.ansible.com/ansible/latest/file_module.html | 17:16 |
| notmyname | pabelanger: ok :-) | 17:16 |
| jeblair | frickler: indeed, the base job uses copy to put its key in place | 17:16 |
| *** bh526r has quit IRC | 17:16 | |
| *** jamesmcarthur has quit IRC | 17:16 | |
| clarkb | notmyname: the reason for that is ubuntu only supports non LTS releases for 9 months now. Which is is too short to test more than one release on | 17:17 |
| frickler | in the log, "Write out ssh key" says "ok" instead of "changed", too | 17:17 |
| jeblair | fungi: have a link to a recent failed build? | 17:17 |
| pabelanger | notmyname: however, if diskimage-builder can build 17.10 and our infra elements work, we could add the image to nodepool. We'd just need to decide to do it or not | 17:17 |
| frickler | jeblair: http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/cece9e2/job-output.txt.gz | 17:17 |
| jeblair | frickler: thx | 17:17 |
| *** jamesmcarthur has joined #openstack-infra | 17:17 | |
| notmyname | clarkb: pabelanger: oh I think it's great. I don't want to have to mess with non-LTS versions :-) | 17:17 |
| fungi | yeah, that's the most recent one | 17:17 |
| *** slaweq_ has quit IRC | 17:17 | |
| pabelanger | clarkb: yah | 17:18 |
| *** florianf has quit IRC | 17:18 | |
| *** rbrndt has quit IRC | 17:19 | |
| *** smatzek has joined #openstack-infra | 17:19 | |
| *** dhinesh has joined #openstack-infra | 17:21 | |
| *** jamesmcarthur has quit IRC | 17:22 | |
| jeblair | frickler, fungi: i have no idea why nothing reported an error or warning, but i agree with frickler, there's no documentation for 'content' on the file module so we should change that to copy. | 17:22 |
| frickler | patch upcoming | 17:22 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/zuul-jobs master: Use copy instead of file for add-sshkey task https://review.openstack.org/513456 | 17:22 |
| jeblair | frickler, fungi: assuming it just ignores the parameter, the behavior matches the symptoms "yep, this is a file already and it's this big" | 17:22 |
| fungi | excellent. i wonder if the "size" output for that add-sshkey task is simply reporting the size of the file it found, not the size of a file that task created | 17:22 |
| jeblair | fungi: ya | 17:22 |
| fungi | yeah, that | 17:23 |
| jeblair | maybe ansible-lint would have caught this (or maybe whatever causes the file module not to reject 'content' would have fooled it too?) | 17:24 |
| *** dhinesh_ has joined #openstack-infra | 17:24 | |
| *** dprince has quit IRC | 17:25 | |
| *** jamesmcarthur has joined #openstack-infra | 17:25 | |
| fungi | unfortunately, searching for the keywords "ansible" "file" "content" is kinda a non-starter | 17:25 |
| *** salv-orlando has joined #openstack-infra | 17:25 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Clarify how the sql database is managed https://review.openstack.org/512160 | 17:26 |
| *** dhinesh has quit IRC | 17:26 | |
| fungi | ohh, "content" is a parameter to the "copy" task | 17:27 |
| fungi | :q | 17:27 |
| fungi | hah, you're not my editor | 17:28 |
| smcginnis | E37: No write since last change (add ! to override) | 17:28 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Change domain in test fixture to example.com https://review.openstack.org/513448 | 17:28 |
| *** salv-orl_ has quit IRC | 17:29 | |
| jeblair | fungi, clarkb: zuul's decrypted form of the ssh private key is 1675 bytes | 17:29 |
| clarkb | the test encrypt secret tool is not working for me :/ so this is going slowly | 17:29 |
| jeblair | that matches the paste from earlier | 17:29 |
| *** salv-orlando has quit IRC | 17:30 | |
| jeblair | so i have a lot of confidence when we fix the problem frickler found, we'll have the right data on disk | 17:30 |
| fungi | awesome, so this all seems to point to we were failing to overwrite the existing key on disk with it | 17:30 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move alembic_reporter back to alembic https://review.openstack.org/512161 | 17:30 |
| fungi | as soon as it merges, i'll retrigger our test commit | 17:31 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Allow the pool_recycle to be configured https://review.openstack.org/512162 | 17:31 |
| fungi | er, reenqueue our test commit i mean | 17:31 |
| *** andreas_s has quit IRC | 17:31 | |
| clarkb | oh thats why ara didnt say changed it just said ok? | 17:31 |
| jeblair | yep, that's the theory | 17:32 |
| *** smatzek has quit IRC | 17:32 | |
| fungi | i'm going to delete my autoheld node and set another autohold "just in case" | 17:32 |
| jeblair | fungi: ++ | 17:33 |
| clarkb | I'm going to sort out the test tool for encrypting things because now it is bugging me :) | 17:33 |
| clarkb | the error implies it may have worked with python2? | 17:34 |
| clarkb | its a bytes vs str problem deep in the C depths of $lib | 17:34 |
| *** lbragstad has quit IRC | 17:35 | |
| *** tosky has quit IRC | 17:36 | |
| fungi | ahh, yeah i ran it under python2.7 | 17:36 |
| fungi | just confirmed from shell history | 17:36 |
| *** amoralej is now known as amoralej|off | 17:37 | |
| *** kiennt26 has quit IRC | 17:38 | |
| *** groot is now known as jgriffith | 17:38 | |
| *** Swami has joined #openstack-infra | 17:39 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add noop job description https://review.openstack.org/508774 | 17:39 |
| *** lbragstad has joined #openstack-infra | 17:39 | |
| *** andreas_s has joined #openstack-infra | 17:41 | |
| *** mriedem has quit IRC | 17:41 | |
| *** pblaho has quit IRC | 17:41 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add noop job description https://review.openstack.org/508774 | 17:42 |
| *** mriedem has joined #openstack-infra | 17:43 | |
| *** masayukig[m] has quit IRC | 17:43 | |
| *** ilpianista_ has quit IRC | 17:43 | |
| Shrews | fungi: ooh, that gives me a chance to test autohold-list | 17:44 |
| fungi | went ahead and skipped 513456,1 (the add-sshkey fix) into the gate since there's still quite a backup in check | 17:44 |
| *** aspiers[m] has quit IRC | 17:44 | |
| jeblair | fungi: thx | 17:44 |
| jeblair | i'm afk for a bit | 17:44 |
| *** kzaitsev_pi has quit IRC | 17:44 | |
| Shrews | fungi: http://paste.openstack.org/show/624095/ \o/ | 17:45 |
| *** andreas_s has quit IRC | 17:45 | |
| fungi | Shrews: esselent | 17:45 |
| *** kzaitsev_pi has joined #openstack-infra | 17:45 | |
| Shrews | it's nice when something, ya know, works | 17:45 |
| AJaeger | could I get a +2A on https://review.openstack.org/#/c/513414/ - to fix the service-data-fact role and thus api-guide and api-ref publish jobs, please? | 17:46 |
| fungi | Shrews: feels like a rare commodity sometimes | 17:46 |
| *** jklare has quit IRC | 17:47 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Disable tox-siblings https://review.openstack.org/513205 | 17:48 |
| *** dprince has joined #openstack-infra | 17:49 | |
| *** jklare has joined #openstack-infra | 17:51 | |
| *** jamesmcarthur has quit IRC | 17:52 | |
| *** apetrich has quit IRC | 17:52 | |
| *** jamesmcarthur has joined #openstack-infra | 17:52 | |
| *** apetrich has joined #openstack-infra | 17:53 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:54 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy dragonflow jobs https://review.openstack.org/512785 | 17:54 |
| *** jamesmcarthur has quit IRC | 17:57 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use dest instead of src for get_url downloaded resource https://review.openstack.org/513414 | 17:58 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Use copy instead of file for add-sshkey task https://review.openstack.org/513456 | 17:58 |
| smcginnis | Any way to test if that made a difference. Or need to requeue the test job? | 17:58 |
| *** baoli has quit IRC | 17:58 | |
| fungi | smcginnis: i just reenqueued the test commit again | 17:59 |
| fungi | and i also have an autohold set so if we fail it i can ssh in and look at the new key on disk | 17:59 |
| smcginnis | ++ | 18:00 |
| *** bobh has joined #openstack-infra | 18:00 | |
| EmilienM | clarkb: when you have time to see my question: right now tripleo jobs don't index openstack logs in logstash (e.g. /var/log/nova/nova-api.log) because they aren't mentionned in https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/collect-logs/defaults/main.yml#n88 . So we have 2 options here: 1) patch | 18:01 |
| EmilienM | https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-log-client.yaml and add all files that we want to index. Or 2) Add files to https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/collect-logs/defaults/main.yml#n88 but I'm afraid logstash.txt will grow and becore big enough for us to have storage issues again one | 18:01 |
| EmilienM | day (and it also duplicate content that we already collect in logs). What option is the best for you? | 18:01 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add noop job description https://review.openstack.org/508774 | 18:01 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Rename ironic job for nova https://review.openstack.org/513410 | 18:03 |
| *** andreas_s has joined #openstack-infra | 18:03 | |
| *** efoley has quit IRC | 18:04 | |
| *** tosky has joined #openstack-infra | 18:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy pyghmi job https://review.openstack.org/513249 | 18:04 |
| *** dhinesh_ has quit IRC | 18:04 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Make tests' encrypt_secret.py work with python3 https://review.openstack.org/513462 | 18:05 |
| clarkb | I think ^ should do it | 18:05 |
| *** felipemonteiro has joined #openstack-infra | 18:06 | |
| clarkb | EmilienM: I'm not sure I understand how configuration in tripleo can affect the logstash indexing. The jenkins log client config (or whatever zuulv3 reads now) should drive all that? | 18:06 |
| *** baoli has joined #openstack-infra | 18:07 | |
| *** felipemonteiro__ has joined #openstack-infra | 18:07 | |
| EmilienM | clarkb: the jenkins log client config doesn't index http://logs.openstack.org/96/507796/6/check/legacy-tripleo-ci-centos-7-nonha-multinode-oooq/6697d00/logs/undercloud/var/log/nova/nova-api.log.txt.gz for example, that's our problem | 18:08 |
| *** _bmjen is now known as bmjen | 18:08 | |
| *** jamesmcarthur has joined #openstack-infra | 18:08 | |
| dhellmann | clarkb : i think you want to use "utf-8" not "utf8" because the former is more the standard, but I think python takes both | 18:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove fetch-zuul-cloner from base-test job https://review.openstack.org/513079 | 18:08 |
| clarkb | EmilienM: ok reading more, is logs/undercloud/var/log/extra/logstash.txt generated from all the logs listed in that tripleo file then indexed? | 18:08 |
| EmilienM | clarkb: logs/undercloud/var/log/extra/logstash.txt contains content of all files listed here: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/collect-logs/defaults/main.yml#n88 | 18:09 |
| clarkb | wow ok | 18:09 |
| EmilienM | yeah | 18:09 |
| EmilienM | so my question: how can we do, without exploding index & storage | 18:09 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Make tests' encrypt_secret.py work with python3 https://review.openstack.org/513462 | 18:10 |
| clarkb | dhellmann: ^ | 18:10 |
| *** electrofelix has quit IRC | 18:10 | |
| clarkb | EmilienM: well there is little we can do about exploding index size if we add significantly more content to the indexes | 18:10 |
| clarkb | EmilienM: but it would be good to get away from using duplicate files to do the work | 18:10 |
| EmilienM | clarkb: exactly, I'm all for it, that's why the only option I see is to add every log file we want to index in https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-log-client.yaml | 18:11 |
| *** felipemonteiro has quit IRC | 18:11 | |
| EmilienM | clarkb: but before doing so, I'm asking because it's a lot of work | 18:11 |
| *** bobh has quit IRC | 18:11 | |
| clarkb | EmilienM: project-config/roles/submit-logstash-jobs/defaults/main.yaml is the new zuulv3 thing | 18:11 |
| EmilienM | clarkb: unless we can do some regex but I'm not sure we can | 18:11 |
| EmilienM | ok let me look now | 18:11 |
| smcginnis | How'd the release post job go from running back to queued? | 18:11 |
| clarkb | smcginnis: it will do that if it failed in pre for some reason | 18:12 |
| clarkb | EmilienM: the new zuulv3 thing supports regexes | 18:12 |
| fungi | smcginnis: i'm wondering the same. i expect zuul had to try again or something | 18:12 |
| EmilienM | clarkb: ok that's *awesome* | 18:12 |
| smcginnis | Hopefully we didn't accidentally introduce a different error. | 18:12 |
| fungi | smcginnis: if it fails three times that way, it'll go ahead and report with a retry_limit result | 18:12 |
| EmilienM | weshay|ruck, clarkb: let me cook a small prototype then and I'll send it for review before I do the rest (if any) | 18:12 |
| EmilienM | clarkb: thank you | 18:13 |
| smcginnis | [fingers crossed] | 18:13 |
| clarkb | EmilienM: you can see that at the bottom of the file | 18:13 |
| clarkb | EmilienM: I think itis first match wins so the globs are towards the end | 18:13 |
| fungi | smcginnis: so hopefully we'll have ansible output from that we'll be able to work from if that's the case | 18:13 |
| EmilienM | clarkb: yeah... I didn't know about that file, indeed, it solves our problem more easily | 18:13 |
| frickler | fungi: error seen, coming up with a fix | 18:13 |
| frickler | http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/ded4b11 | 18:13 |
| fungi | frickler: oh! great, i'm looking forward to finding out | 18:13 |
| smcginnis | I'm looking forward to no errors. :) | 18:14 |
| fungi | ahh, yep i guess that first attempt did indeed upload some logs | 18:14 |
| smcginnis | Need to create the directory first? http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/ded4b11/job-output.txt.gz#_2017-10-19_18_10_22_668007 | 18:14 |
| fungi | src (or content) and dest are required | 18:14 |
| weshay|ruck | EmilienM, nice | 18:15 |
| *** SumitNaiksatam has quit IRC | 18:15 | |
| weshay|ruck | thanks clarkb | 18:15 |
| smcginnis | Oh yeah, see that earlier now. | 18:15 |
| fungi | looks like we don't specify a dest in 513456 | 18:15 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/zuul-jobs master: The copy module needs dest instead of path https://review.openstack.org/513465 | 18:16 |
| fungi | just a path | 18:16 |
| frickler | ^^ | 18:16 |
| clarkb | so the file module noops even if you give it different content? | 18:16 |
| frickler | subtle difference between file and copy module | 18:16 |
| frickler | clarkb: yes, content parameter seems to get silently ignored | 18:16 |
| smcginnis | Is there also an issue with the error two seconds later? | 18:16 |
| frickler | clarkb: at least when the file exists | 18:17 |
| fungi | approved and engated | 18:17 |
| clarkb | hrm file is for attributes and doens't have a content parameter documented | 18:17 |
| clarkb | weird | 18:17 |
| frickler | smcginnis: I think that is only because the pre terminated too early. might deserve a fix anyway | 18:18 |
| fungi | yeah, the job wasn't erroring about launchpadlib creds in previous runs, and the pre playbook is what would have installed that, so pretty sure it's just because we stopped it at the earlier task | 18:19 |
| frickler | smcginnis: yeah, looking at the older log, add-launchpad-credentials would have been the next task after add-sshkey | 18:19 |
| frickler | EOD for me, hope this will pass now, good luck ;) | 18:20 |
| fungi | thanks frickler! | 18:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Add tripleo-heat-templates into tripleo shared queue for gate https://review.openstack.org/513222 | 18:21 |
| *** andreas_s has quit IRC | 18:21 | |
| clarkb | EmilienM: as for index size we only index INFO and greater logs which helps | 18:23 |
| smcginnis | frickler: Thanks! | 18:23 |
| clarkb | but ya we are currently at a 10 day retention even with that in place | 18:23 |
| AJaeger | fungi, clarkb, the translation jobs fail git review -s asking for "Enter your gerrit username:" - is this part of the debugging you're doing? | 18:23 |
| EmilienM | clarkb: ok, good to know. | 18:23 |
| fungi | AJaeger: yes, i believe it's been running afoul of the same ssh-addkey role issue | 18:23 |
| EmilienM | clarkb: 10 days isn't too bad, I guess it takes lot of space after 10 days | 18:23 |
| fungi | AJaeger: so hopefully this fixes those failures too | 18:23 |
| clarkb | EmilienM: we currently have about 5TB max useable disk | 18:24 |
| * AJaeger crosses fingers | 18:24 | |
| clarkb | 6TB total but for recovery purposes can't go over 5TB | 18:24 |
| fungi | parity blocks! | 18:24 |
| fungi | it's like raid-5-over-network | 18:24 |
| *** caphrim007 has quit IRC | 18:25 | |
| SamYaple | dhellmann: clarkb my secrets are under the 4096 bytes. they are under 200 chars each | 18:25 |
| *** rbrndt has joined #openstack-infra | 18:25 | |
| *** dhinesh has joined #openstack-infra | 18:25 | |
| clarkb | currently using about 3.2TB but are behind on indexing so not sure if that is representative | 18:26 |
| openstackgerrit | Eric Kao proposed openstack-infra/irc-meetings master: Change Congress meeting time https://review.openstack.org/513467 | 18:26 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: The copy module needs dest instead of path https://review.openstack.org/513465 | 18:26 |
| clarkb | SamYaple: I think it got sorted out, and testing indicates that zuul does work properly even with longer keys | 18:26 |
| * AJaeger moves fixed issues from one etherpad to the other now... | 18:27 | |
| smcginnis | write out ssh key passed. | 18:28 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo: index /var/log/*.log.txt files https://review.openstack.org/513469 | 18:28 |
| EmilienM | clarkb, weshay|ruck ^ when you have time | 18:29 |
| smcginnis | And failure | 18:29 |
| * weshay|ruck looks | 18:29 | |
| smcginnis | fungi: Asking for gerrit user name still. You held that node, right? | 18:29 |
| weshay|ruck | EmilienM, you're a machine | 18:29 |
| EmilienM | not really | 18:30 |
| *** caphrim007 has joined #openstack-infra | 18:30 | |
| AJaeger | we still fail ssl cert validation with readthedocs ;( http://logs.openstack.org/fc/fc1fe410ef8b497553adfef76ffefc0a80890503/post/trigger-readthedocs/ffe9b79/ara/ | 18:30 |
| fungi | smcginnis: yeah, though first i'm double-checking that it ran the right playbook version since that was a retry from my previous reenqueue | 18:30 |
| * AJaeger adds to etherpad | 18:30 | |
| openstackgerrit | Eric Kao proposed openstack-infra/irc-meetings master: Add some aliases to congress meeting chairs https://review.openstack.org/513470 | 18:30 |
| jeblair | smcginnis: is it still running? | 18:31 |
| clarkb | EmilienM: are all of those files of the "console" format? | 18:31 |
| smcginnis | jeblair: No, says failure. | 18:31 |
| fungi | looks like the add-sshkey task reflected the right file size this time at least | 18:31 |
| EmilienM | clarkb: good question, I wasn't sure about the format - I guess we have 2 kind of format. oslo : http://logs.openstack.org/12/510212/3/gate/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq/1da4911/logs/undercloud/var/log/glance/api.log.txt.gz and console? | 18:32 |
| jeblair | fungi, smcginnis: i didn't see an autohold; not sure if it's because we caught that one or not. i have added another autohold, so we will get the next one, whatever it is. | 18:32 |
| EmilienM | http://logs.openstack.org/12/510212/3/gate/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq/1da4911/logs/undercloud/var/log/bootstrap-subnodes.log.txt.gz | 18:32 |
| fungi | though i think my autohold was from one of the previous runs so i may need to retrigger with a new autohold | 18:32 |
| jeblair | oh my autohold is gone, maybe we caught it? | 18:32 |
| fungi | jeblair: yeah, the previous playbook version was resutling in pre-stage failures so zuul was retrying those but probably autoheld the first of them | 18:32 |
| jeblair | | 0000325216 | rax-dfw | None | ubuntu-xenial | b22c2493-ea90-40b8-b572-7628c4b0d2f2 | hold | 00:00:01:23 | unlocked | ubuntu-xenial-rax-dfw-0000325216 | 104.130.141.213 | 10.209.1.232 | 2001:4800:7818:103:be76:4eff:fe04:ded9 | 22 | nl01.openstack.org-13058-PoolWorker.rax-dfw-main | 100-0000525221 | openstack git.openstack.org/openstack/releases tag-releases | jeblair ... | 18:33 |
| jeblair | ... debug key issues | | 18:33 |
| jeblair | fungi, smcginnis: ^ got it | 18:33 |
| smcginnis | ++ | 18:33 |
| fungi | correct length key | 18:34 |
| clarkb | EmilienM: that log is but logs like http://logs.openstack.org/12/510212/3/gate/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq/1da4911/logs/undercloud/var/log/deployed-server-os-collect-config.log.txt.gz are not sothey will all likely error during parsing (which falls back to best effort) | 18:36 |
| fungi | correct checksum, matching the ~jenkins/.ssh/id_rsa on signing01.ci.openstack.org | 18:36 |
| jeblair | fungi, smcginnis: git push --dry-run ssh://zuul@review.openstack.org:29418/openstack/release-test.git --all | 18:37 |
| jeblair | ssh://zuul@review.openstack.org:29418/openstack/release-test.git did not work. Description: Permission denied (publickey). | 18:37 |
| *** abishop has joined #openstack-infra | 18:37 | |
| *** baoli has quit IRC | 18:37 | |
| fungi | yeah, same ssh'ing to gerrit | 18:37 |
| clarkb | it shouldn't ssh as zuul | 18:37 |
| jeblair | fungi, smcginnis: is the username supposde to be zuul? | 18:37 |
| clarkb | it should ssh as 'release' iirc | 18:37 |
| EmilienM | clarkb: sorry, I didn't get the sentence, can you say it otherwise? | 18:37 |
| *** lin_yang has joined #openstack-infra | 18:38 | |
| fungi | jeblair: when i do `ssh -p 29418 release@review.openstack.org gerrit version` as zuul i get the same error though | 18:38 |
| clarkb | EmilienM: http://logs.openstack.org/12/510212/3/gate/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq/1da4911/logs/undercloud/var/log/deployed-server-os-collect-config.log.txt.gz will match your glob too I think but the format of that file is not of the "console" type so when logstash attempts to parse it it will fail (and do its best) | 18:38 |
| jeblair | fungi: maybe 2 problems? | 18:38 |
| EmilienM | clarkb: what format to you suggest? | 18:38 |
| fungi | jeblair: strangely, i can run the same command on signin01.ci as jenkins with no problem | 18:38 |
| clarkb | jeblair: re spaces and base64 I have confirmed the spaces are in the input but since whitespace is not valid in base64 data python seems to ignore it when decoding (testing with and without spaces seems to result in the same decoded data) | 18:39 |
| jeblair | clarkb: ack | 18:39 |
| clarkb | EmilienM: I don't think we have any parser rules that will match thta format | 18:39 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Rename ironic jobs for nova/neutron/devstack https://review.openstack.org/513410 | 18:40 |
| jeblair | fungi: oh, i see i was distracted by the git-review -s which is apparently not required to succeed... but still, seems like there's something worth fixing there. | 18:40 |
| clarkb | EmilienM: an important assumption we've made is that anything we index will have timestamps in a parseable format becuse we are indexing after the fact | 18:40 |
| jeblair | like, do the git config's before it or something | 18:40 |
| EmilienM | clarkb: the most important logs we need now are the openstack ones, I guess | 18:41 |
| EmilienM | clarkb: for the rest, best effort is fine probably for now | 18:41 |
| fungi | jeblair: yeah, we do still need to set a gitreview.user or something, but right now i'm trying to figure out why gerrit is refusing this key completely for the release user | 18:41 |
| fungi | checking gerrit's ssh logs now | 18:41 |
| fungi | [2017-10-19 18:39:33,210 +0000] 313acdaa release - AUTH FAILURE FROM 2001:4800:7818:103:be76:4eff:fe04:ded9 no-matching-key | 18:42 |
| jeblair | did someone said something about regenerating keys yesterday? | 18:43 |
| smcginnis | So wrong key on the job VM? Or wrong key configured in gerrit for the "release" account? | 18:43 |
| jeblair | fungi: sorry, you verified this works from signing.ci with the same content | 18:43 |
| jeblair | i'm still baffled | 18:43 |
| fungi | jeblair: the ~jenkins/.ssh/id_rsa on signing01.ci.openstack.org has the same checksum as the ~zuul/.ssh/id_rsa on the failing job node | 18:43 |
| clarkb | could the id_rsa.pub be confusing it? | 18:44 |
| clarkb | since it is for a different key | 18:44 |
| fungi | teh id_rsa.pub shouldn't even be touched | 18:44 |
| jeblair | i'll move it just to check | 18:44 |
| jeblair | ok wow that seemed to be it | 18:44 |
| fungi | -vvv says that /home/zuul/.ssh/id_rsa is offered and then rejected | 18:45 |
| EmilienM | clarkb: so I let console for now? or switch to something else? | 18:45 |
| clarkb | debug1: Offering RSA public key: /home/clark/.ssh/id_rsa seems to imply it shouldn'taffect it | 18:45 |
| fungi | jeblair: wha? that's nuts | 18:45 |
| clarkb | but it does apparently | 18:45 |
| jeblair | fungi: i am in full agreement | 18:45 |
| clarkb | EmilienM: console is our most generic one, we can try it that way and see how nasty the results end up being | 18:45 |
| * fungi throws a chair | 18:45 | |
| EmilienM | clarkb: ++ I'm fine | 18:46 |
| jeblair | fungi: i mved it to ~/generated-id-pub | 18:46 |
| clarkb | EmilienM: I think it will fall back to using the timestamp of indexing time (so could be hours later) and will just inject all the message as the message | 18:46 |
| EmilienM | clarkb: thanks for your help. | 18:46 |
| jeblair | fungi: if you want to cp it back and forth to double check (please) | 18:46 |
| EmilienM | clarkb: ack | 18:46 |
| fungi | jeblair: clarkb: okay, so should the ssh-addkey task delete id_rsa.pub? | 18:46 |
| jeblair | fungi: i will stand down and not do anything further on the host | 18:46 |
| clarkb | fungi: jeblair or we could consider writing to a different location | 18:46 |
| *** slaweq_ has joined #openstack-infra | 18:46 | |
| clarkb | for now though easiest thing is probably delete the .pub | 18:46 |
| fungi | jeblair: confirmed that does seem to be the problem | 18:47 |
| *** sshnaidm is now known as sshnaidm|off | 18:47 | |
| jeblair | clarkb: yeah, i like delete the pub, then use a different location (that's probably something to change between q-1 and q-2) | 18:47 |
| fungi | i can break and unbreak it at will just by puttnig the pubkey there | 18:47 |
| *** baoli has joined #openstack-infra | 18:48 | |
| jeblair | it's not every day i lean something so fundamental and new about openssh | 18:48 |
| jeblair | or learn to speel | 18:48 |
| fungi | it's plain nuts | 18:48 |
| *** jcoufal has quit IRC | 18:48 | |
| clarkb | I learned things about base64 and yaml, today we learn things! | 18:49 |
| *** slaweq_ has quit IRC | 18:49 | |
| jeblair | anyone writing the rm pub change, or should i jump on it? | 18:49 |
| clarkb | I am not | 18:49 |
| *** caphrim007 has quit IRC | 18:49 | |
| jeblair | clarkb: it's learnsday for sure! | 18:49 |
| jeblair | yeah, i'll just go write that now | 18:49 |
| smcginnis | Now we know, and knowing is half the battle. | 18:50 |
| fungi | you can have it. i was starting to try and figure out what it should look like but i've written roughly 0 ansible to date | 18:50 |
| clarkb | I think a second file taks that ensures absent | 18:50 |
| fungi | so was still hunting for a good file deletion example | 18:50 |
| clarkb | state: absent | 18:50 |
| fungi | is it that puppetey? wow | 18:50 |
| *** e0ne has joined #openstack-infra | 18:51 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: be more prescriptive in log collection https://review.openstack.org/511526 | 18:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add-sshkey: remove pubkey if present https://review.openstack.org/513476 | 18:52 |
| weshay|ruck | patch to reduce tripleo inodes https://review.openstack.org/#/c/511526/ | 18:53 |
| weshay|ruck | pabelanger, ^ | 18:53 |
| jeblair | fungi, clarkb, smcginnis: https://review.openstack.org/513476 | 18:53 |
| *** caphrim007 has joined #openstack-infra | 18:54 | |
| fungi | bnemec: awesome, i was drafting a version of that as an exercise and it's basically identical sans comments. lgtm | 18:54 |
| fungi | er, jeblair ^ | 18:54 |
| fungi | sorry bnemec | 18:54 |
| jeblair | fungi: then it's well reviewed! | 18:54 |
| clarkb | jlk beat me to it | 18:55 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/elastic-recheck master: Add query for 1724930 https://review.openstack.org/513477 | 18:55 |
| jeblair | fungi: want to enqueue it into gate? | 18:55 |
| fungi | done | 18:56 |
| fungi | smcginnis: as jeblair noted a bit ago. once we get past this it looks like we're still missing a gitreview.username in git configuration so git-review and friends are falling back to assuming the remote username is the same as the local one | 18:57 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove networking-generic-switch job https://review.openstack.org/513349 | 18:57 |
| clarkb | fungi: my reading of the code yeserday was that it will git review -s and fail and if it fails then it does git config and then re runs git review -s | 18:57 |
| fungi | or, rather, it happens too late for the git-review -s http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/7304174/job-output.txt.gz#_2017-10-19_18_28_44_804882 | 18:57 |
| clarkb | so I think it may work, but we may want to change that as we know it will always fail now | 18:58 |
| fungi | ahh, pergect | 18:58 |
| clarkb | that seems like an optimization for single use slaves | 18:58 |
| clarkb | er | 18:58 |
| clarkb | non single use | 18:58 |
| fungi | clarkb: or for local use | 18:58 |
| fungi | i.e., use the already working gerrit account you've set up, but if you haven't set one up then assume you're in the ci system and your gerrit user should be "release" | 18:59 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/elastic-recheck master: Add query for 1724930 https://review.openstack.org/513477 | 18:59 |
| AJaeger | fungi, what's missing in this case: http://paste.openstack.org/show/624101/ ? | 19:03 |
| jeblair | AJaeger: the proposal jobs use the same role we're fixing for the release job; 513476 is the next fix | 19:04 |
| fungi | AJaeger: probably 513476 | 19:04 |
| fungi | yeah | 19:04 |
| AJaeger | fungi: ah - so, since ssh fails, it asks for username? ok, got it. | 19:05 |
| *** salv-orlando has joined #openstack-infra | 19:05 | |
| * AJaeger was confused with username setting since that's here ;/ | 19:05 | |
| *** smatzek has joined #openstack-infra | 19:05 | |
| fungi | yep, specifically we just discovered that if you have an id_rsa.pub present which isn't for your corresponding id_rsa private key, openssh client will act as if the private key is rejected by the remote server. but if you remove the (unnecessary) public key file, things "just work" | 19:06 |
| fungi | baffling | 19:06 |
| AJaeger | fungi: yeah, got that, didn't connect the dots ;( | 19:07 |
| *** smatzek has quit IRC | 19:07 | |
| fungi | this may also be behavior particular to very specific versions of openssh (i certainly wouldn't be surprised to discover this is the case, anyway) | 19:07 |
| clarkb | jeblair: EmilienM's query above points out an interesting potential problem with the zuulv3 logstash indexing. The filename/tags are different because we dropped the .gz in the old system but include it now | 19:08 |
| *** smatzek has joined #openstack-infra | 19:08 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/elastic-recheck master: Add query for 1724930 https://review.openstack.org/513477 | 19:08 |
| clarkb | jeblair: EmilienM not the end of the world but we may need to go through and update queries | 19:08 |
| fungi | AJaeger: the fact that with -vvv it never even mentions the public key file yet still seems to care about it is... bizarre | 19:08 |
| EmilienM | clarkb: good to know | 19:08 |
| EmilienM | clarkb: 24 queries contain .txt but no .gz | 19:09 |
| EmilienM | clarkb: I'm preparing a separated patch to fix them | 19:10 |
| clarkb | EmilienM: ok | 19:10 |
| smcginnis | Is there a patch up to remove the .pub file? | 19:10 |
| AJaeger | fungi: indeed, bizarre... | 19:10 |
| AJaeger | smcginnis: 513476 | 19:10 |
| smcginnis | AJaeger: Thanks! | 19:10 |
| fungi | should just about be done in the gate by now | 19:11 |
| *** slaweq_ has joined #openstack-infra | 19:11 | |
| *** e0ne has quit IRC | 19:11 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add-sshkey: remove pubkey if present https://review.openstack.org/513476 | 19:12 |
| fungi | there we go | 19:12 |
| fungi | retriggering the test change now | 19:12 |
| *** smatzek has quit IRC | 19:12 | |
| smcginnis | fungi: Thanks for doing that, it was getting really painful doing the revert, revert-revert dance. | 19:12 |
| EmilienM | b qa | 19:13 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/elastic-recheck master: queries: match .txt.gz files instead of .txt https://review.openstack.org/513479 | 19:13 |
| EmilienM | oops | 19:13 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy pyghmi job https://review.openstack.org/513250 | 19:14 |
| fungi | smcginnis: yup, i totally get it. the release automation also isn't something we can easily do end-to-end testing on pre-production, so i understand it's painful to have to freeze release work to iterate on some of the most complicated ci automation we have | 19:14 |
| fungi | basically the case for just about everything we were running on persistent nodes in v2 | 19:15 |
| smcginnis | fungi: Yeah, doesn't seem like we have a good way to pre-test all of this easily. | 19:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Stop storing dependent items on buildsets https://review.openstack.org/513441 | 19:15 |
| jeblair | added an autohold | 19:16 |
| fungi | i have my fingers crossed this was the last missing piece, but also have to disappear shortly since i promised christine i'd take her out to eat | 19:16 |
| * clarkb lunches | 19:16 | |
| fungi | thanks jeblair, i didn't think to set another one | 19:16 |
| AJaeger | fungi, post and periodic proposals fall in the same category. Those are the biggest class of broken jobs. remember that neither post translatoin nor periodic translation work yet | 19:16 |
| AJaeger | fungi: Let's see what else we'll find - we're getting closer ;) | 19:16 |
| fungi | AJaeger: yep, those are definitely the other jobs to which i was referring | 19:17 |
| fungi | jeblair: are you done with the previous autoheld node (0000325216) now? i'll delete if so | 19:17 |
| *** nicolasbock has joined #openstack-infra | 19:18 | |
| jeblair | fungi: ++ | 19:19 |
| smcginnis | Dang it | 19:19 |
| jeblair | smcginnis: link? | 19:20 |
| fungi | gpg: skipped "OpenStack Release Bot <openstack-infra@lists.openstack.org>": secret key not available | 19:20 |
| smcginnis | Just watching the stream so far. | 19:20 |
| fungi | it's still running, but http://zuulv3.openstack.org/static/stream.html?uuid=ad494a2ef3994e6b8b2fe19b3d1eee9e&logfile=console.log | 19:20 |
| jeblair | i don't remember seeing the signing secret attached to the job | 19:20 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add integration tests for emit-job-header and set-zuul-log-path-fact https://review.openstack.org/511948 | 19:20 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove Zun-client legacy jobs https://review.openstack.org/510423 | 19:20 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: add jenkins_urlopen() method https://review.openstack.org/500103 | 19:21 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: build_job: return queue item identifier https://review.openstack.org/511937 | 19:21 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: add get_queue_item() method https://review.openstack.org/511938 | 19:21 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: better get_build_info() example https://review.openstack.org/511939 | 19:21 |
| fungi | jeblair: i agree, i expect that's missing entirely | 19:21 |
| fungi | i have to disappear now, but will be back as soon as i possibly can be | 19:21 |
| smcginnis | Thanks fungi, have fun. | 19:22 |
| *** jamesmcarthur has quit IRC | 19:22 | |
| * fungi has been accused of ignoring the real world for the past several weeks, must mitigate | 19:22 | |
| *** jamesmcarthur has joined #openstack-infra | 19:22 | |
| smcginnis | jeblair: You have an idea for what's next? | 19:23 |
| jeblair | smcginnis: i'm looking | 19:23 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: tripleo/paunch: run legacy-tripleo-ci-centos-7-nonha-multinode-oooq https://review.openstack.org/512795 | 19:25 |
| jeblair | smcginnis: while i look, can you verify that the job that just ran didn't do anything permanent (ie, push any tags or create any branches, etc)? | 19:25 |
| *** jamesmcarthur has quit IRC | 19:27 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make tests' encrypt_secret.py work with python3 https://review.openstack.org/513462 | 19:27 |
| *** ijw has quit IRC | 19:29 | |
| smcginnis | jeblair: Yep, rechecking now, but I think we're safe. | 19:29 |
| jeblair | do we use the same key we use to sign artifacts to sign the tags? | 19:31 |
| *** jamesmcarthur has joined #openstack-infra | 19:32 | |
| *** abishop has left #openstack-infra | 19:32 | |
| smcginnis | That I do not know. | 19:33 |
| jeblair | i think so; i'll proceed with that assumption | 19:34 |
| jeblair | (there's just the one key in ~jenkins on signing01) | 19:35 |
| smcginnis | I think you may be right. | 19:35 |
| smcginnis | Doesn't look from the logs like we did any permanent damage. Should be good to try it again when we're ready. | 19:35 |
| jeblair | okay, i think we need to make a new role to add the gpg key (the existing one i found does add/sign/remove all in one, but we need something more like the add-sshkey role). then update that job to use it and add the gpg key secret | 19:37 |
| *** ldnunes has quit IRC | 19:39 | |
| jeblair | smcginnis: i'm going to test this role locally as best i can; i expect to need around 30m until i'm ready to try running a change | 19:40 |
| dhellmann | smcginnis : I'm just coming back after some errands. How are things looking? | 19:40 |
| dhellmann | sounds like we're further along | 19:41 |
| jeblair | dhellmann: the tag-releases job is missing support for a gpg key, so i'm writing that now. | 19:41 |
| dhellmann | jeblair : oh, duh, yeah | 19:42 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: sahara-image-elements: remove the migrated jobs https://review.openstack.org/513484 | 19:42 |
| smcginnis | jeblair: That sounds like a prudent plan, thanks. | 19:42 |
| smcginnis | dhellmann: We're farther at least. | 19:42 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/openstack-zuul-jobs master: sahara-image-elements: remove the migrated jobs https://review.openstack.org/513485 | 19:42 |
| dhellmann | steps... | 19:43 |
| smcginnis | We're somewhere between crawling and stepping. ;) | 19:43 |
| dhellmann | heh | 19:44 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: sahara-image-elements: remove the migrated jobs https://review.openstack.org/513484 | 19:44 |
| *** salv-orl_ has joined #openstack-infra | 19:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove double python in translation script https://review.openstack.org/513486 | 19:47 |
| AJaeger | project-config-cores, could you review this quick fix, please? ^ | 19:47 |
| AJaeger | team, we're one step further with translations - we have a new bug ;) ^ | 19:47 |
| *** dprince has quit IRC | 19:49 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for 1724930 https://review.openstack.org/513477 | 19:49 |
| *** dave-mccowan has quit IRC | 19:49 | |
| *** salv-orlando has quit IRC | 19:49 | |
| *** nicolasbock has quit IRC | 19:49 | |
| *** dave-mccowan has joined #openstack-infra | 19:51 | |
| *** hasharDinner has quit IRC | 19:52 | |
| *** ijw has joined #openstack-infra | 19:52 | |
| *** ldnunes has joined #openstack-infra | 19:53 | |
| *** jamesmcarthur has quit IRC | 19:53 | |
| dmsimard | AJaeger: +2 | 19:53 |
| *** hashar has joined #openstack-infra | 19:54 | |
| *** jamesmcarthur has joined #openstack-infra | 19:54 | |
| inc0 | hey guys, gate vms have all ports open to each other? I'm trying to debug multinode mariadb timeouts - any chance it's security group issue in nodepool | 19:55 |
| *** nicolasbock has joined #openstack-infra | 19:55 | |
| inc0 | ? | 19:55 |
| AJaeger | dmsimard: thanks. Your change for the fact helped, but now the next problem http://logs.openstack.org/56/565651aa5c55572fa2fc88674a01b103ad451d0c/post/publish-api-ref/74de96e/ara/ ;( | 19:56 |
| *** hashar has quit IRC | 19:57 | |
| *** ijw has quit IRC | 19:57 | |
| dmsimard | AJaeger: ok, I'll look in a few mins | 19:58 |
| openstackgerrit | Merged openstack-infra/project-config master: tripleo/paunch: run legacy-tripleo-ci-centos-7-nonha-multinode-oooq https://review.openstack.org/512795 | 19:59 |
| AJaeger | dmsimard: I'm on it, have an idea... | 19:59 |
| *** nicolasbock has quit IRC | 20:00 | |
| clarkb | inc0: yes it should allow all traffic between the nodes if using the multinode overlay networking role | 20:01 |
| clarkb | inc0: if not then you'll need to punch holesyourself | 20:01 |
| *** kaisers has quit IRC | 20:01 | |
| *** iyamahat has quit IRC | 20:02 | |
| *** iyamahat_ has joined #openstack-infra | 20:02 | |
| *** dave-mccowan has quit IRC | 20:02 | |
| inc0 | overlay networking role? | 20:02 |
| clarkb | AJaeger: done | 20:03 |
| clarkb | inc0: zuulv3 has a role to setup an overlay network between the nodes so that they have a shared l2, if you use that role (or have the multinode job as parent) then you weill have an open firewall between the instances | 20:03 |
| AJaeger | clarkb: thanks | 20:03 |
| *** smatzek has joined #openstack-infra | 20:04 | |
| inc0 | any example how to add this role to my play? | 20:04 |
| inc0 | because that sounds right | 20:04 |
| AJaeger | dmsimard: the exit_json gets "{u'project': u'zun', u'service_type': u'application-container', u'api_reference': u'https://git.openstack.org/cgit/openstack/zun/tree/api-ref/source', u'aliases': [u'container']}" and then complains about "exit_json() takes 1 positional argument but 2 were given". Can we return a dict- or have to change this to return just the parameter? | 20:05 |
| jeblair | pabelanger: do you have the command you used to export the gpg key to make a secret out of it? | 20:05 |
| clarkb | inc0: I'm not sure if we have anything using the native zuulv3 multinode stuff yet but many jobs in openstack-zuul-jobs use legacy-dsvm-base-multinode as a parent which should do roughly the same thing though via devstack-gate maybe? | 20:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove double python in translation script https://review.openstack.org/513486 | 20:08 |
| inc0 | clarkb: well, I'm ok if we would be first ones to do zuul native | 20:09 |
| inc0 | I'd rather do it right:) | 20:09 |
| *** nicolasbock has joined #openstack-infra | 20:09 | |
| inc0 | if you have any docs or just want to help me, point me to your ansible coe, that'd be great | 20:09 |
| clarkb | inc0: right I just don't have an example for that | 20:09 |
| inc0 | well I guess you're about to get one;) | 20:10 |
| *** ihrachys has quit IRC | 20:10 | |
| dhinesh | am trying to use this https://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool to bring a CI setup, when i try to run "nodepool image-build dpc", | 20:10 |
| dhinesh | there is an error die 'Can not find Zuul public key!' | 20:11 |
| clarkb | inc0: zuul-jobs/playbooks/multinode/pre.yaml is the playbook for multinode base job | 20:11 |
| clarkb | inc0: and it runs bridge setup and firwall setup as two separate roles | 20:11 |
| AJaeger | dmsimard: document in etherpad around line 106 | 20:11 |
| dhinesh | is the public key for zuul auto generated? | 20:12 |
| clarkb | dhinesh: when building nodepool images you have to provide the ssh key that zuul will login as. by default this is the public key for the user running nodepool | 20:12 |
| clarkb | dhinesh: it is not autogenerated, but it will use the one for the current user if present | 20:12 |
| clarkb | if not present then you need to either put one there or explicitly set the env var in your nodepool imae config to point at one | 20:12 |
| inc0 | clarkb: switch and peers will come from zuul's inventory right? | 20:13 |
| inc0 | technically I should just be able to add this playbook to my own pre.yml | 20:13 |
| dhinesh | clarkb: ok thanks, i will try it out | 20:13 |
| clarkb | inc0: it is from the nodeset yes, you'll have to make sure you have those groups on your nodeset | 20:14 |
| inc0 | or translate it to nodenames I have | 20:14 |
| inc0 | ok, let's try | 20:14 |
| *** nicolasbock has quit IRC | 20:14 | |
| *** ldnunes has quit IRC | 20:15 | |
| *** nicolasbock has joined #openstack-infra | 20:15 | |
| mwhahaha | hey is there a place to watch a given zuul queue's history that isn't the dashboard? i'm trying to figure out what happened to the jobs as to why they got reset | 20:15 |
| dmsimard | inc0: let me give you an example | 20:17 |
| clarkb | mwhahaha: right now the closest thing to that is proably the openstack health dashboard. Nothing zuul specific yet (but that is planned) | 20:17 |
| dmsimard | inc0: https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/jobs.yaml#L76-L91 | 20:17 |
| mwhahaha | clarkb: yea that doesn't help because I didn't see what job failed out | 20:17 |
| jeblair | mwhahaha: did it not report back on the change? | 20:18 |
| mwhahaha | clarkb: the failed jobs don't seem to stick around as long as they used to in the old one | 20:18 |
| mwhahaha | i don't think i was on the change | 20:18 |
| mwhahaha | so i didn't get an email | 20:18 |
| dmsimard | AJaeger: ok looking now | 20:18 |
| jeblair | mwhahaha: oh, so you don't know which change, you just saw a red thing on the status page? | 20:18 |
| mwhahaha | jeblair: no i didn't see a red thing, i just noticed the jobs were rerunning | 20:18 |
| mwhahaha | jeblair: for example see the job runs for this one job http://logs.openstack.org/18/512818/3/gate/puppet-openstack-module-build/ | 20:19 |
| mwhahaha | i knew there were 3 changes above it but i don't remember which one is not there anymore | 20:19 |
| *** kgiusti has left #openstack-infra | 20:19 | |
| dmsimard | AJaeger: that module is weird ? We can do that in native ansible, no need for python | 20:19 |
| jeblair | mwhahaha: gotcha; you might be able to find them by looking at the inventory files for the change | 20:19 |
| dmsimard | maybe I'm missing something obvious | 20:19 |
| AJaeger | dmsimard: tell mordred ;) | 20:20 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add requirements as needed project for release-openstack-python job https://review.openstack.org/513492 | 20:20 |
| inc0 | thanks dmsimard | 20:20 |
| dhellmann | jeblair, smcginnis : ^^ | 20:20 |
| jeblair | mwhahaha: http://logs.openstack.org/18/512818/3/gate/puppet-openstack-module-build/ddec889/zuul-info/inventory.yaml has the items ahead of it in the queue when the build started | 20:20 |
| inc0 | dmsimard: https://review.openstack.org/#/c/512779/ <- this is change I'm playing around, feel free to drop feedback, would be greatly appreciated | 20:20 |
| mwhahaha | jeblair: yea i'll go throught the previous execution and see what was in the zuul info for tha tone | 20:20 |
| jeblair | mwhahaha: (under the key "items" under "zuul") | 20:21 |
| smcginnis | dhellmann: Were there issues with that too? | 20:21 |
| mwhahaha | jeblair: yea i found that part | 20:21 |
| dhellmann | smcginnis : there's a log linked from the commit message | 20:21 |
| dmsimard | inc0: for multinode you'll need the groups so that we know where to provision the switch and the peers (see the snippet I sent you) and the multinode base job will take care of setting up firewall rules and also /etc/hosts so that inventory hostnames resolve to the right place | 20:21 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: fix specs publishing https://review.openstack.org/513493 | 20:21 |
| AJaeger | another simple fix - for specs publishing ^. | 20:21 |
| smcginnis | dhellmann: Oh sure, now you expect me to read. :P | 20:22 |
| dhellmann | smcginnis : the networking-powervm folks pushed a tag on their own | 20:22 |
| dhellmann | smcginnis :-P | 20:22 |
| AJaeger | could an infra-root cleanup the specs.openstack.org site and remove the wrong toplevel directories, please? | 20:22 |
| smcginnis | dhellmann: Ah, I see. | 20:22 |
| smcginnis | And I also now see the failure emails. | 20:23 |
| *** ijw has joined #openstack-infra | 20:23 | |
| jeblair | dhellmann: are we going to need to install the neutron and horizon repos for some projects too? | 20:23 |
| *** nicolasbock has quit IRC | 20:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add roles to add/remove a GPG key https://review.openstack.org/513494 | 20:27 |
| *** ijw has quit IRC | 20:27 | |
| jeblair | dhellmann, smcginnis: i tested that with newly-created fake data, and it's very similar to the existing sign-artifacts role. | 20:28 |
| smcginnis | With READMEs even. | 20:29 |
| jeblair | smcginnis: they get turned into this: https://docs.openstack.org/infra/zuul-jobs/roles.html | 20:29 |
| fungi | okay, back and catching up | 20:29 |
| smcginnis | Nice | 20:29 |
| smcginnis | fungi: That was quick-ish. | 20:29 |
| jeblair | fungi: i'm glad you're back, but i worry that was insufficient time spent in the real world :) | 20:30 |
| *** andreas_s has joined #openstack-infra | 20:30 | |
| jeblair | fungi: if you could please review https://review.openstack.org/513494 -- i'm writing the change to use it now. | 20:30 |
| fungi | jeblair: smcginnis: seems you came to the correct conclusion, but yes the release tags are signed with the same openpgp key as we use for detached tarball signatures | 20:30 |
| fungi | (for simplicity) | 20:30 |
| jeblair | fungi: cool, that will hopefully save some confusion later :) | 20:31 |
| dhellmann | jeblair : install neutron and horizon? for tagging? | 20:32 |
| dhellmann | I certainly hope not, but now I'm not sure. | 20:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Run bindep for translation jobs https://review.openstack.org/513497 | 20:33 |
| jeblair | dhellmann: yes -- after a brief inspection of that error, i assumed the issue was that the tox installation of that project used openstack-requirements in its custom install commands, and i know that neutron projects do so for neutron, and likewise horizon. | 20:33 |
| AJaeger | next fix for translations, we were missing msgcat on the nodes ^ | 20:33 |
| jeblair | dhellmann: i did not spend long looking at it and am not at all confident in my assesment :) | 20:33 |
| dhellmann | jeblair : oh, I may have jumped the gun then. I haven't seen any of our tests make it that far, and I thought that was a missing declaration of using the requirements repo for constraints in a standard way. | 20:34 |
| AJaeger | dmsimard: would appreciate your review of 513497 as well, please | 20:34 |
| pabelanger | jeblair: I think I did: gpg --export-secret-keys --armor 83FA74EC from signing01.ci.o.o | 20:34 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Remove legacy-requirements-cross-* jobs https://review.openstack.org/513275 | 20:34 |
| *** andreas_s has quit IRC | 20:35 | |
| jeblair | pabelanger: cool, that's what i wound up with too | 20:35 |
| pabelanger | kk | 20:35 |
| fungi | reviewing 513494 as the next step in release automation fixing | 20:35 |
| pabelanger | also, zuulv3 talk was a hit :D People in Ottawa excited to start running it. | 20:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add GPG key to tag-releases job https://review.openstack.org/513498 | 20:36 |
| fungi | pabelanger: did you show them zuulv3 live? | 20:37 |
| *** nicolasbock has joined #openstack-infra | 20:37 | |
| pabelanger | fungi: I did! | 20:37 |
| jeblair | pabelanger: i hope they'll be happy to wait a few more months. :) | 20:37 |
| pabelanger | wifi was really good here | 20:37 |
| jeblair | fungi, smcginnis, dhellmann: https://review.openstack.org/513498 is next | 20:37 |
| fungi | thanks! | 20:38 |
| pabelanger | fungi: also had a chance to demo ARA per dmsimard request | 20:38 |
| dhellmann | jeblair : yeah, I think that's the "standard" script from tonyb that allows the current repo to be listed in the constraints file and still honor constraints in its dependencies | 20:38 |
| dhellmann | jeblair : it uses zuul-cloner to get the requirements repo and install it so it can use edit-constraints | 20:38 |
| dmsimard | yeah pabelanger totally plugged ara, thanks pabelanger :P | 20:38 |
| dirk | AJaeger: https://review.openstack.org/#/c/512487/ -- better? | 20:40 |
| * AJaeger calls it a day after reviewing dirk's change. would appreciate review of 513493 (and cleanup of specs.o.o) and 513497 (next step for translations) | 20:40 | |
| dhellmann | jeblair , fungi , smcginnis : I wonder if we are going to preemptively implement the tarball changes mordred describes in https://review.openstack.org/#/c/508693/ ? | 20:41 |
| dhellmann | to remove the use of tox | 20:41 |
| jeblair | good thing it has enough votes? :) | 20:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - testing base-test changes https://review.openstack.org/513502 | 20:43 |
| fungi | one more governance change i haven't gotten around to reviewing in the past several weeks | 20:43 |
| fungi | but, hey, it's short | 20:43 |
| jeblair | dhellmann: there's another version of that script that's more like this: http://git.openstack.org/cgit/openstack/networking-bagpipe/tree/tools/tox_install.sh | 20:44 |
| smcginnis | pabelanger: Was your talk recorded? | 20:45 |
| pabelanger | smcginnis: sadly no | 20:45 |
| dhellmann | jeblair : that one also uses zuul-cloner? but I guess it's cloning something else? I don't understand what that version is doing at first glance. | 20:45 |
| jeblair | dhellmann: sorry, look at the bottom -- install_project neutron | 20:46 |
| dhellmann | oh, that's installing *other* projects | 20:46 |
| dhellmann | sigh | 20:46 |
| pabelanger | smcginnis: I plan on uploading slides to interwebs | 20:46 |
| smcginnis | pabelanger: Oh well... might have been nice to watch in all of my free time. :) | 20:46 |
| smcginnis | pabelanger: Oh, that'd be cool. | 20:46 |
| fungi | jeblair: so looking at 513498 i guess secrets can just be added by name and the only reason the ssh_key secret was added to that job more verbosely was so that we could rename release_ssh_key to ssh_key? implying that the lp_creds secret addition could also be shortened to a one-line entry? | 20:46 |
| dhellmann | so networking-bagpipe doesn't appear in constraints but networking-powervm does? | 20:46 |
| dhellmann | no, networking-powervm does not appear in constraints | 20:47 |
| dhellmann | I think they probably don't need that script | 20:47 |
| jeblair | dhellmann: agreed | 20:47 |
| fungi | dhellmann: the cargo cult is powerfully compelling | 20:47 |
| dhellmann | so I'll abandon my patch | 20:47 |
| dhellmann | for the bagpipe thing, though, they will need extra repos available | 20:47 |
| jeblair | dhellmann: so my hazy understanding of this is your change will fix powervm, but i expect bagpipe to fail on a release. | 20:47 |
| dhellmann | yeah | 20:48 |
| dhellmann | is it possible for bagpipe to declare that their version of that job needs extra repos? | 20:48 |
| *** slaweq_ has quit IRC | 20:48 | |
| jeblair | dhellmann: we've worked through most of this (i hope) with the python jobs, and to my knowledge, it's just neutron and horizon that are like this | 20:48 |
| jeblair | see, eg: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n335 | 20:48 |
| *** mat128 has quit IRC | 20:48 | |
| fungi | which will be fixed once we fix tox-siblings and reenable it, until the pti change merges and we stop using tox to generate tarballs? | 20:48 |
| jeblair | dhellmann: absolutely. | 20:48 |
| jeblair | fungi: yep. :) | 20:49 |
| dhellmann | yeah, ok, good, as long as someone understands the steps for that | 20:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: DNM: testing base-test https://review.openstack.org/513505 | 20:49 |
| fungi | at least it makes plugin/extension projects more obviously "different" rather than just treating all projects that way | 20:49 |
| dhellmann | I still don't really know where tox-siblings fits into all of this | 20:49 |
| jeblair | dhellmann: so we could take the approach of just adding horizon and neutron to that job now for everything, or we could try to add them specifically to the repos that need it. | 20:49 |
| *** slaweq_ has joined #openstack-infra | 20:49 | |
| dhellmann | let's just add them for everything | 20:49 |
| dhellmann | I can write that | 20:50 |
| jeblair | dhellmann: ack | 20:50 |
| dhellmann | we can take them out when we remove the use of tox | 20:50 |
| jeblair | dhellmann: (tox-siblings is disabled now, until it gets constraints support added, so that's one variable removed) | 20:50 |
| jeblair | i believe that change merged this morning | 20:50 |
| jeblair | or what i call morning | 20:50 |
| tonyb | dhellmann: We have a plan to remove part of that but that needed to wait for the v3 switch to be flicked so I'll call for beta testers RSN and then roll it out at the same time we switch from getting upper-constratints.txt from cgit to the static files | 20:51 |
| dmsimard | AJaeger: got a patch for service type data | 20:52 |
| jeblair | fungi: you okay pushing things through now? if so, i'll take a break | 20:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove fetch-zuul-cloner from base job https://review.openstack.org/513506 | 20:53 |
| smcginnis | jeblair, dhellmann, fungi: Sorry, I've lost track again. Which/what if any patches are we waiting for now before we can try again? | 20:53 |
| fungi | jeblair: yep, i'm caught up and pushing hard now | 20:53 |
| fungi | please go enjoy a break | 20:53 |
| smcginnis | Ah, 513494 | 20:53 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add required-projects for the release-openstack-python jobs https://review.openstack.org/513507 | 20:53 |
| smcginnis | Thanks jeblair | 20:54 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: Use playbook from tripleo-quickstart-extras for OVB https://review.openstack.org/513508 | 20:54 |
| fungi | smcginnis: 513494 and 513498' | 20:54 |
| *** slaweq_ has quit IRC | 20:54 | |
| fungi | both of which are approved and should merge momentarily, then i'll retrigger | 20:54 |
| dhellmann | we'll need https://review.openstack.org/513507 for some of the real releases, but not for our test job | 20:54 |
| smcginnis | 494 looks close. | 20:54 |
| dhellmann | s/job/repo/ | 20:54 |
| *** felipemonteiro__ has quit IRC | 20:54 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add roles to add/remove a GPG key https://review.openstack.org/513494 | 20:55 |
| fungi | there's the first | 20:55 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Refactor fact configuration for service_type_data https://review.openstack.org/513509 | 20:55 |
| dmsimard | AJaeger, mordred ^ | 20:56 |
| *** ijw has joined #openstack-infra | 20:56 | |
| smcginnis | I'm not seeing 513498 in the zuul status. | 20:56 |
| smcginnis | Oh, there it is now. | 20:57 |
| fungi | it's in the gate now, according to comments on the review | 20:57 |
| fungi | and then i'll reenqueue the feb0fda test commit | 20:57 |
| fungi | once that project-config change merges in a sec | 20:58 |
| fungi | i'll also set another autohold for tag-releases just in case i need to inspect it | 20:58 |
| fungi | autohold set | 21:00 |
| *** Apoorva_ has joined #openstack-infra | 21:02 | |
| *** trown is now known as trown|outtypewww | 21:02 | |
| *** rvba has quit IRC | 21:02 | |
| *** rvba has joined #openstack-infra | 21:03 | |
| *** rvba has quit IRC | 21:03 | |
| *** rvba has joined #openstack-infra | 21:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add GPG key to tag-releases job https://review.openstack.org/513498 | 21:03 |
| fungi | and there it is. reenqueuing test commit now | 21:04 |
| dmsimard | pabelanger: https://review.openstack.org/#/c/513509/ | 21:04 |
| pabelanger | project-config-core: https://review.openstack.org/513506/ is ready to promote up out base-test to base, for fetch-zuul-cloner. I've attached logs showing working base and legacy-base jobs. I'd like to move forward with approval, to discourage the use of zuul-cloner in non-legacy jobs. Keep in mind, there is a chance we _could_ break a jobs that parents to base, but is still using zuul-cloner, but | 21:04 |
| pabelanger | the fix is to reparent to legacy-base instead. Hopefully no jobs break, but the longer we wait, the rate likely rises | 21:04 |
| *** felipemonteiro has joined #openstack-infra | 21:04 | |
| pabelanger | dmsimard: will look shortly, heading to next talk | 21:04 |
| smcginnis | pabelanger: Could we wait until post-Q-1 just to avoid the risk of breaking jobs for folks that are trying to wrap up for the milestone? | 21:05 |
| *** Apoorva has quit IRC | 21:05 | |
| smcginnis | Just a day or so. | 21:06 |
| *** dbecker has quit IRC | 21:06 | |
| fungi | yeah, the gate is full of people squeezing in final milestone-1 changes | 21:06 |
| smcginnis | grumble grumble grumble | 21:07 |
| smcginnis | secret key not available | 21:07 |
| fungi | well, we should have better data this time | 21:07 |
| openstackgerrit | Alan Pevec proposed openstack-infra/tripleo-ci master: Use infra proxy server for trunk.r.o in delorean-deps https://review.openstack.org/508884 | 21:08 |
| fungi | i don't actually see the new role in the pre phase. i may have triggered that one _too_ soon? | 21:09 |
| *** felipemonteiro has quit IRC | 21:09 | |
| fungi | yeah, i must have. triggering once more after resetting the autohold | 21:10 |
| *** iyamahat_ has quit IRC | 21:10 | |
| *** iyamahat__ has joined #openstack-infra | 21:10 | |
| fungi | new autohold in place, test commit reenqueued again | 21:10 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools master: fix the repo name detection when we don't have an origin set https://review.openstack.org/513513 | 21:11 |
| dhellmann | fungi , jeblair : the announce job is failing because it can't get the repo name ^^ | 21:11 |
| dhellmann | smcginnis : ^^ | 21:11 |
| fungi | dhellmann: thanks | 21:12 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for use-cached-repos https://review.openstack.org/512927 | 21:12 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Improve test coverage of the fetch-zuul-cloner role and the shim https://review.openstack.org/512904 | 21:12 |
| dmsimard | project-config-core: ^ would appreciate +3 on these two, I've had to rebase quite a few times now :( | 21:13 |
| *** markvoelker has joined #openstack-infra | 21:13 | |
| *** gouthamr has quit IRC | 21:15 | |
| smcginnis | fungi: Same thing. Is there a step missing somewhere? | 21:15 |
| *** thorst has quit IRC | 21:16 | |
| *** Goneri has quit IRC | 21:17 | |
| fungi | smcginnis: oh, i missed that ara paginates tasks over a certain number and the add-gpgkey pre tasks were on a second page | 21:18 |
| *** thorst has joined #openstack-infra | 21:18 | |
| fungi | the import task output does seem to indicate that it imported the key | 21:19 |
| *** rvba has quit IRC | 21:19 | |
| jeblair | fungi: link? | 21:20 |
| fungi | though, and i don't think this is the issue, it's using the pike cycle key. we need to amend our key rotation docs to cover replacing the key in zuulv3 job configuration (i only replaced the pike key with the queens key in hiera) | 21:20 |
| *** Apoorva_ has quit IRC | 21:20 | |
| fungi | jeblair: i can't get the permalinks in ara to work in firefox. i think that needs a newer ara release or something | 21:20 |
| jeblair | fungi: job link would be good enough :) | 21:20 |
| fungi | http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/75f2ebc | 21:20 |
| jeblair | (and yeah, that's fixed on the 1.0 branch) | 21:20 |
| *** Apoorva has joined #openstack-infra | 21:20 | |
| fungi | 77.81.7.176 is the new autoheld node | 21:21 |
| fungi | this much looks good aside from needing the queens key: http://paste.openstack.org/show/624107/ | 21:23 |
| *** dave-mccowan has joined #openstack-infra | 21:23 | |
| *** thorst has quit IRC | 21:23 | |
| *** Goneri has joined #openstack-infra | 21:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: add-sshkey: make the private key owner-writable https://review.openstack.org/513519 | 21:23 |
| fungi | yeah, saw that post failure too | 21:23 |
| jeblair | lower-priority, but we'll eventually need that :) | 21:23 |
| fungi | thanks | 21:23 |
| smcginnis | Is this normal: WARNING: options in `/home/zuul/.gnupg/gpg.conf' are not yet active during this run | 21:24 |
| *** jamesmcarthur has quit IRC | 21:24 | |
| fungi | i have a feeling this may be why i was embedding the keychain in hiera rather than importing the signing subkey | 21:25 |
| jeblair | fungi: oh, is it email addr not matching? | 21:25 |
| jeblair | so it can't find the 'correct' secret key? | 21:26 |
| fungi | oh! | 21:26 |
| fungi | good eye | 21:26 |
| fungi | yes | 21:26 |
| fungi | we use infra-root@openstack.org for our key address | 21:26 |
| *** smatzek has quit IRC | 21:26 | |
| smcginnis | Oh, good catch. No lists.o.o | 21:26 |
| fungi | right, that looks like the e-mail address of the release account in gerrit, not the address of our key | 21:27 |
| jeblair | what's different between this and signing01 though? | 21:27 |
| openstackgerrit | Merged openstack-infra/release-tools master: fix the repo name detection when we don't have an origin set https://review.openstack.org/513513 | 21:27 |
| fungi | we set user.signingkey = infra-root@openstack.org in .gitconfig | 21:28 |
| jeblair | aha, and is that in puppet or release scripts? | 21:28 |
| fungi | the .gitconfig on signing01.ci was puppeted, and likely not migrated to the release scripts | 21:28 |
| fungi | http://paste.openstack.org/show/624108/ | 21:29 |
| fungi | for the record | 21:29 |
| *** dave-mccowan has quit IRC | 21:29 | |
| smcginnis | So we need to update our setting of git config to match the cert address? | 21:30 |
| fungi | yeah, just another git config invocation in the script i guess? | 21:30 |
| jeblair | modules/openstack_project/manifests/signing_node.pp: $gitpgpkey = 'infra-root@openstack.org', | 21:30 |
| jeblair | confirmed in puppet ^ | 21:30 |
| jeblair | fungi: ++ | 21:30 |
| dtroyer | is there a way to iterate on a single job without having to run the entire set of jobs every time? I know I'm not going to get my first native devstack job right immediately and hate to 'waste' 8 nodes for each typo. Plus the cycle time | 21:32 |
| fungi | working on patch to jenkins/scripts/release-tools/functions now | 21:32 |
| openstackgerrit | wes hayutin proposed openstack-infra/elastic-recheck master: Track tripleo timeout or canceled deployments https://review.openstack.org/513524 | 21:32 |
| jeblair | dtroyer: you could probably modify the project definition to drop all the other jobs | 21:32 |
| jeblair | dtroyer: obviously don't merge it that way, but for "WIP" iteration i think it'd be fine | 21:33 |
| dtroyer | I thought about that, it'll work better after all of the projects jobs are moved in-repo, ie pep8/py27/py36 etc. Those are supposed to eventually move in-repo too, right? | 21:33 |
| dhellmann | not for the PTI jobs, I think | 21:34 |
| dhellmann | or CTI or whatever we call it | 21:34 |
| dhellmann | the common "you need to have these" jobs | 21:34 |
| jeblair | dtroyer, dhellmann: yeah. this is something we lose with that approach. something to consider. | 21:34 |
| *** iyamahat__ has quit IRC | 21:34 | |
| dtroyer | ah, ok. I can eliminate 4 of 8 then, that's an improvement. thanks | 21:34 |
| *** iyamahat__ has joined #openstack-infra | 21:34 | |
| jeblair | dtroyer: yeah, at least you won't be running extra devstacks | 21:35 |
| dtroyer | right, that's the cycle time killer | 21:35 |
| jeblair | fungi: i'll see about getting the queens key into yaml | 21:36 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Set the signingkey address for release jobs https://review.openstack.org/513525 | 21:36 |
| fungi | jeblair: smcginnis: dhellmann: ^ | 21:36 |
| *** tosky has quit IRC | 21:36 | |
| smcginnis | Cool, didn't know that was a separate config setting. | 21:36 |
| fungi | jeblair: thanks, i think we can iterate in parallel there since we have a good (albeit stale) key in place and are just attempting to postprocess a release for a test project, not a real one | 21:37 |
| *** dave-mccowan has joined #openstack-infra | 21:37 | |
| jeblair | fungi: ah good | 21:37 |
| smcginnis | This has all been a "learning experience". | 21:37 |
| fungi | smcginnis: no kidding | 21:37 |
| jeblair | fungi: +2 | 21:38 |
| jeblair | anyone else around to +3 it, or should i? | 21:38 |
| *** hongbin has quit IRC | 21:38 | |
| jeblair | done | 21:38 |
| fungi | thanks again | 21:39 |
| fungi | that should merge quickly since it's project-config | 21:39 |
| fungi | and then i'll fire another test | 21:39 |
| fungi | test volley | 21:39 |
| fungi | cleaning up and resetting autohold in the meantime | 21:39 |
| *** baoli has quit IRC | 21:40 | |
| *** tpsilva has quit IRC | 21:42 | |
| smcginnis | Almost there. | 21:43 |
| *** dave-mccowan has quit IRC | 21:44 | |
| fungi | jeblair: once you have the queens export done, can you make a paste of what you ran and i'll use that to update our key rotation howto | 21:45 |
| jeblair | fungi: i'm actually doing doc-driven development :) | 21:45 |
| jeblair | i'm updating the docs first | 21:45 |
| *** thorst has joined #openstack-infra | 21:47 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: add-sshkey: make the private key owner-writable https://review.openstack.org/513519 | 21:47 |
| fungi | jeblair: wow! too awesome | 21:47 |
| openstackgerrit | wes hayutin proposed openstack-infra/elastic-recheck master: track tripleo issues pulling containers https://review.openstack.org/513532 | 21:48 |
| *** markvoelker has quit IRC | 21:48 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set the signingkey address for release jobs https://review.openstack.org/513525 | 21:48 |
| fungi | and now both of those are in, triggering another | 21:48 |
| SamYaple | jeblair: now make the docs generate teh code! | 21:50 |
| *** dizquierdo has quit IRC | 21:50 | |
| jeblair | SamYaple: do i look like don knuth? | 21:50 |
| openstackgerrit | wes hayutin proposed openstack-infra/elastic-recheck master: Track tripleo timeout or canceled deployments https://review.openstack.org/513524 | 21:50 |
| SamYaple | jeblair: idk. youre still alive | 21:50 |
| *** yamamoto has joined #openstack-infra | 21:50 | |
| smcginnis | Or Don Knotts? | 21:51 |
| openstackgerrit | wes hayutin proposed openstack-infra/elastic-recheck master: track tripleo issues pulling containers https://review.openstack.org/513532 | 21:51 |
| jeblair | last i checked, he was very much alive | 21:51 |
| *** thorst has quit IRC | 21:51 | |
| SamYaple | oh i thought he died | 21:51 |
| smcginnis | Remote rejected, prohibited by Gerrit | 21:53 |
| fungi | new errors are progress, i guess | 21:53 |
| fungi | http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/ec03266/ | 21:53 |
| openstackgerrit | wes hayutin proposed openstack-infra/elastic-recheck master: track tripleo pulling containers from rdo registry https://review.openstack.org/513532 | 21:53 |
| EmilienM | clarkb: do you think we can get https://review.openstack.org/#/c/513469/ today? | 21:55 |
| fungi | looks like we split the git tag command line incorrectly? | 21:55 |
| fungi | the -s appears on a seconf line | 21:55 |
| fungi | oh, nevermind, that's split on quotes | 21:55 |
| *** masayukig[m] has joined #openstack-infra | 21:57 | |
| smcginnis | Wish it would give some indication why it was rejected. | 22:01 |
| *** markvoelker has joined #openstack-infra | 22:02 | |
| smcginnis | Actually... is this an ACL issue for the repo in gerrit? | 22:03 |
| *** jrist has quit IRC | 22:03 | |
| ianw | pabelanger: i'm starting on prep for the upcoming rename ... i guess it's now a multi-repo affair with the jobs. will document in https://etherpad.openstack.org/p/rename-2017-10-20 | 22:05 |
| *** markvoelker has quit IRC | 22:06 | |
| fungi | sorry, primary monitor on my workstation just went toes up so i had to switch computers for now | 22:08 |
| *** rwsu has joined #openstack-infra | 22:08 | |
| dhellmann | smcginnis : maybe, yes. I don't see the release bot in the acl list | 22:09 |
| dhellmann | I added the release bot | 22:09 |
| dhellmann | https://review.openstack.org/#/admin/groups/249,members | 22:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Update signing key rotation instructions for Zuul v3 https://review.openstack.org/513533 | 22:09 |
| smcginnis | I see Jenkins and Zuul, but not sure if that's enough. | 22:09 |
| fungi | i checked the acls... release-test inherits from all-projects which in turn allows the release managers group to push signed tags and the release bot is a member of the release managers group | 22:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Update signing key for Queens https://review.openstack.org/513534 | 22:10 |
| smcginnis | Is it? I didn't think I saw it in there. | 22:10 |
| dhellmann | fungi : https://review.openstack.org/#/admin/projects/openstack/release-test,access says release-tools-core can push. Is that additive? | 22:10 |
| fungi | smcginnis: in the release managers group? | 22:10 |
| jeblair | fungi: 533 and 534 are the key changes when you have a sec | 22:10 |
| smcginnis | Oh, nevermind, I'm blind. | 22:10 |
| fungi | dhellmann: it's additive unless the project acl sets the exclusive-group-permissions flag on it | 22:10 |
| dhellmann | ah | 22:10 |
| jeblair | oh, let me leave a comment on 533 about how i did that | 22:10 |
| dhellmann | I wish the gerrit UI showed that | 22:11 |
| smcginnis | Shows "Rights Inherited From", but definitely not obvious. | 22:11 |
| fungi | dhellmann: it has the "inherits from" at the top of the acl | 22:11 |
| dhellmann | fungi : the ssh command git-review is running uses zuul@ instead of release@ | 22:11 |
| *** markvoelker has joined #openstack-infra | 22:11 | |
| dhellmann | I wonder why that is | 22:11 |
| dhellmann | http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/ec03266/job-output.txt.gz#_2017-10-19_21_51_56_391812 | 22:11 |
| dhellmann | fungi : ah! | 22:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Update signing key rotation instructions for Zuul v3 https://review.openstack.org/513533 | 22:12 |
| fungi | dhellmann: yeah, that's likely it... i'll poke around the held node | 22:12 |
| smcginnis | Is this annotated tagging or signed tagging. | 22:12 |
| fungi | signed | 22:12 |
| dhellmann | git review dumps all of the config --get commands it runs, but not the results. that would be a useful enhancement at some point | 22:13 |
| dhellmann | fungi : does the job run as the "zuul" user? do we need to set another config value somewhere? | 22:14 |
| *** hemna_ has quit IRC | 22:14 | |
| smcginnis | Looks like that should be OK too. https://review.openstack.org/#/admin/groups/10,members | 22:15 |
| fungi | dhellmann: we have no .gitconfig, wonder if we're setting not --global and then moving between repos? | 22:15 |
| fungi | i need to trace back through the script, takes a minute | 22:16 |
| dhellmann | probably | 22:16 |
| *** markvoelker has quit IRC | 22:16 | |
| dhellmann | smcginnis mentioned setting that earlier | 22:16 |
| smcginnis | Shouldn't hurt at least. | 22:17 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: switch git configuration for release jobs to global https://review.openstack.org/513539 | 22:17 |
| openstackgerrit | Dean Troyer proposed openstack-infra/project-config master: Move non-voting legacy jobs to osc iand osc-lib repos https://review.openstack.org/513540 | 22:17 |
| *** signed8b_ has joined #openstack-infra | 22:18 | |
| smcginnis | I wonder if we should move it somewhere earlier too to get rid of http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/ec03266/job-output.txt.gz#_2017-10-19_21_51_56_392006 | 22:19 |
| fungi | i'm having trouble tracking down where on the filesystem of the held node that even got set | 22:19 |
| *** jrist has joined #openstack-infra | 22:19 | |
| smcginnis | fungi: It's not in release-test/.git/config? | 22:20 |
| fungi | it's not in ~zuul/src/git.openstack.org/openstack/releases/.git/config either that i can tell | 22:20 |
| fungi | oh, release-test | 22:20 |
| fungi | i don | 22:20 |
| fungi | 't even see where release-test was checked out | 22:20 |
| smcginnis | /opt/git/openstack/release-test ? | 22:20 |
| *** markvoelker has joined #openstack-infra | 22:20 | |
| *** ilpianista_ has joined #openstack-infra | 22:21 | |
| *** aspiers[m] has joined #openstack-infra | 22:21 | |
| fungi | nope, nor releases | 22:22 |
| dhellmann | release-test is checked out in a temporary directory by the script | 22:22 |
| dhellmann | it's probably been deleted | 22:22 |
| smcginnis | Wait, is this right? REPODIR=/home/zuul/src/git.openstack.org/openstack/releases/_tmp-release-tag-release-test-UVJ/openstack/release-test | 22:23 |
| dhellmann | yeah | 22:23 |
| *** thorst has joined #openstack-infra | 22:23 | |
| dhellmann | the script registers an exit handler to remove that when it's done, regardless of exit reason | 22:23 |
| fungi | yep, okay, so i agree that's likely the problem | 22:23 |
| dhellmann | we could probably eliminate that under the CI system, but it keeps local dev systems from getting cluttered up | 22:23 |
| dhellmann | is it easy/possible to tell that the job is running in CI? | 22:24 |
| smcginnis | I was just surprised to see it under openstack/releases. I thought we put it under /tmp | 22:24 |
| dhellmann | it uses mktemp but it probably calls it wrong | 22:24 |
| *** markvoelker has quit IRC | 22:25 | |
| *** aeng has joined #openstack-infra | 22:25 | |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Fix contributor guide post run job location https://review.openstack.org/513542 | 22:25 |
| smcginnis | So maybe some confusion between /home/zuul/src/git.openstack.org/openstack/releases/.git/config and /home/zuul/src/git.openstack.org/openstack/releases/_tmp-release-tag-release-test-UVJ/openstack/release-test/.git/config ? | 22:26 |
| smcginnis | Regardless, making it --global should take care of it. | 22:26 |
| dhellmann | the file we need to change is the latter one, but yeah --global is a good change here | 22:26 |
| *** thorst_ has joined #openstack-infra | 22:26 | |
| fungi | agreed | 22:26 |
| *** thorst has quit IRC | 22:27 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/git-review master: show the config value result after fetching it in verbose mode https://review.openstack.org/513543 | 22:28 |
| *** markvoelker has joined #openstack-infra | 22:29 | |
| *** thorst_ has quit IRC | 22:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: switch git configuration for release jobs to global https://review.openstack.org/513539 | 22:31 |
| smcginnis | I'll be back shortly. | 22:31 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Renaming collectd-ceilometer-plugin to collectd-openstack-plugins https://review.openstack.org/500768 | 22:33 |
| *** sdague has quit IRC | 22:34 | |
| ianw | jeblair: ^ just checking that's how you think this will work? i've dropped the changes to legcay bits, and we leave it up to the project to get changes merged to restore their jobs? (they have a local .zuul and imported legacy jobs) | 22:34 |
| *** markvoelker has quit IRC | 22:34 | |
| fungi | smcginnis: dhellmann: jeblair: cool, i've also approved the rotated queens key change (513534) so as soon as it merges i'll trigger yet another test | 22:34 |
| dhellmann | fungi : excellent. I'll hang out for a bit to watch. | 22:35 |
| SpamapS | I think I found a bug in Ansible 2.4's os_* modules | 22:35 |
| SpamapS | shade imports from 'os_client_config', but there's a new os_client_config module... | 22:35 |
| * SpamapS wonders where shade's home channel is these days | 22:36 | |
| dhellmann | SpamapS: #openstack-sdks I think? | 22:36 |
| ianw | jeblair: oh, interesting ... The project "openstack/collectd-openstack-plugins" was not found. All | 22:36 |
| ianw | projects referenced within a Zuul configuration must first be added to | 22:36 |
| ianw | the main configuration file by the Zuul administrator. | 22:36 |
| SpamapS | I'll poke around in #ansible too | 22:36 |
| ianw | do we need to have two changes ... merge into main first, let that update, then the project-config change? | 22:36 |
| SpamapS | it's not really a shade bug | 22:36 |
| fungi | ianw: i think the infra manual got updated with the corrected steps there. did it not? | 22:37 |
| ianw | fungi: read the instructions! crazy | 22:37 |
| jeblair | ianw: we may actually need: 1) drop project from zuul's main.yaml; 2) rename project in gerrit; 3) add project to main.yaml | 22:38 |
| fungi | ianw: jeblair: oh, is this for the rename tomorrow? | 22:38 |
| fungi | indeed, untrodden territory still | 22:39 |
| *** markvoelker has joined #openstack-infra | 22:39 | |
| ianw | fungi: yep ... i'm preparing a run sheet in https://etherpad.openstack.org/p/rename-2017-10-20 | 22:39 |
| *** jamesmcarthur has joined #openstack-infra | 22:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update signing key for Queens https://review.openstack.org/513534 | 22:40 |
| fungi | okay, another volley over the bow | 22:41 |
| *** markvoelker has quit IRC | 22:43 | |
| *** jamesmcarthur has quit IRC | 22:43 | |
| ianw | jeblair: do we still actually need to fully stop zuul, save the queues and restart? | 22:45 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Removes migrated legacy-neutron-lbaasv2-dsvm-* https://review.openstack.org/513548 | 22:46 |
| ianw | jeblair: (sorry for questions) and do we remove from main.yml AND the system jobs in projects.yaml in one change i presume? | 22:46 |
| *** andreas_s has joined #openstack-infra | 22:47 | |
| *** markvoelker has joined #openstack-infra | 22:47 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/openstack-zuul-jobs master: Removes migrated legacy-neutron-lbaasv2-dsvm-* https://review.openstack.org/513550 | 22:47 |
| jeblair | ianw: i think we need to stop/start zuul because we want to hard-stop it before we take gerrit offline | 22:48 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Removes migrated legacy-neutron-lbaasv2-dsvm-* https://review.openstack.org/513548 | 22:48 |
| jeblair | ianw: and yeah, i think main.yaml + projects.yaml in one change | 22:48 |
| dhellmann | "remote rejected (prohibited by gerrit)" http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/tag-releases/c76a505/job-output.txt.gz#_2017-10-19_22_48_39_500342 | 22:50 |
| dhellmann | so we're past the git review -s failure and creating the signed tag, but still can't push it | 22:51 |
| *** andreas_s has quit IRC | 22:51 | |
| dhellmann | fungi : ^^ | 22:51 |
| *** rbrndt has quit IRC | 22:51 | |
| dhellmann | fungi : if you want to run the script by hand on that node, you can do that via: /home/zuul/scripts/release-tools/release_from_yaml.sh /home/zuul/src/git.openstack.org/openstack/releases | 22:52 |
| dhellmann | that may be faster than requeueing the job | 22:52 |
| jeblair | though we have probably deleted the keys (yay security again) | 22:53 |
| dhellmann | oh, right | 22:53 |
| fungi | dhellmann: i think we need to fix the committer e-mail address too | 22:53 |
| dhellmann | what should that be? | 22:54 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Renaming collectd-ceilometer-plugin to collectd-openstack-plugins https://review.openstack.org/500768 | 22:54 |
| dhellmann | I'll propose the patch so you and jeblair can approve it | 22:54 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Drop collectd-ceilometer-plugin for rename https://review.openstack.org/513554 | 22:54 |
| fungi | looking at the .gitconfig on the old node we were setting user.email = infra-root@openstack.org too | 22:54 |
| dhellmann | ok | 22:54 |
| fungi | (refer to my earlier paste when i cat'ed that) | 22:54 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: set committer email to match the gerrit user for release jobs https://review.openstack.org/513555 | 22:55 |
| fungi | i think gerrit is refusing to allow a forged committer on that tag push | 22:55 |
| dhellmann | does the name need to match too? | 22:56 |
| dhellmann | the user.name that is | 22:56 |
| smcginnis | Forged? | 22:56 |
| fungi | no, it just gets picky about e-mail addresses | 22:56 |
| dhellmann | ok | 22:56 |
| *** dave-mccowan has joined #openstack-infra | 22:57 | |
| dhellmann | smcginnis : most users don't have permission to push patches committed by someone else | 22:57 |
| dhellmann | authored is ok, if you change it so you're the committer | 22:57 |
| fungi | i noticed the discrepancy when looking at the acls and group membership that gerrit showed the e-mail address for the release bot account as infra-root@openstack.org | 22:57 |
| *** bobh has joined #openstack-infra | 22:57 | |
| smcginnis | Ah | 22:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Stop storing dependent items on buildsets https://review.openstack.org/513441 | 22:57 |
| fungi | jeblair: on hand to approve 513555? | 22:58 |
| jeblair | yup | 22:58 |
| fungi | much obliged! | 22:58 |
| *** ijw has quit IRC | 22:58 | |
| jeblair | (also hoping to get that change ready for the upcoming zuul scheduler restart :) | 22:59 |
| jeblair | (we're at 80%) | 22:59 |
| fungi | jeblair: yep, looking. i guess that's a memory use reduction | 22:59 |
| *** andreww has quit IRC | 23:02 | |
| mwhahaha | you guys didn't change anything around $ZUUL_REF did you? | 23:03 |
| mwhahaha | just got a failure because it's set to None | 23:04 |
| mwhahaha | http://logs.openstack.org/10/509710/5/check/legacy-puppet-beaker-rspec-centos-7/97d576d/job-output.txt.gz#_2017-10-19_21_32_05_530237 | 23:04 |
| *** Apoorva has quit IRC | 23:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: set committer email to match the gerrit user for release jobs https://review.openstack.org/513555 | 23:04 |
| clarkb | mwhahaha: the ansible value is None but the compat role should rewrite it to 40 *'0' | 23:05 |
| *** Apoorva has joined #openstack-infra | 23:05 | |
| clarkb | mwhahaha: areyou seeing it None in ansible values or in shell values with the compat role? | 23:05 |
| mwhahaha | seemed ot be shell values | 23:06 |
| *** baoli has joined #openstack-infra | 23:06 | |
| mwhahaha | it ultimately failed because our zuul cloner didn't get it | 23:06 |
| * mwhahaha will keep looking | 23:06 | |
| *** thorst has joined #openstack-infra | 23:07 | |
| *** Apoorva has quit IRC | 23:07 | |
| *** wolverineav has quit IRC | 23:08 | |
| *** Apoorva has joined #openstack-infra | 23:08 | |
| mwhahaha | http://logs.openstack.org/10/509710/5/check/legacy-puppet-beaker-rspec-centos-7/97d576d/job-output.txt.gz#_2017-10-19_21_35_15_101909 is where the job failed because we do it again ourselves via shell | 23:08 |
| *** wolverineav has joined #openstack-infra | 23:08 | |
| mwhahaha | i think that relies on the shell env | 23:08 |
| clarkb | looking at the puppet jobs they default it to None if not set | 23:09 |
| clarkb | implying its not the ansible value but just unset | 23:10 |
| mwhahaha | these were working earlier so it's not something we changed our job config | 23:10 |
| mwhahaha | which is why i asked if there was a change around it recently | 23:10 |
| clarkb | I'm not finding where we set the compat vars | 23:12 |
| clarkb | is that in zuul itself? | 23:12 |
| fungi | clarkb: yes, filters | 23:12 |
| *** ijw has joined #openstack-infra | 23:12 | |
| dhinesh | when i run "nodepool image-upload all dpc" i get the following error " DiscoveryFailure: Could not determine a suitable URL for the plugin" , Does this mean that it is not able to reach the openstack cloud URL? | 23:12 |
| *** baoli has quit IRC | 23:12 | |
| clarkb | fungi: what determines if it is applied to the job? | 23:12 |
| clarkb | fungi: I think that is motsly what is missing | 23:12 |
| fungi | clarkb: if it inherits from the legacy job | 23:12 |
| *** wolverineav has quit IRC | 23:12 | |
| *** ijw has quit IRC | 23:13 | |
| fungi | playbook/something like that | 23:13 |
| *** ijw has joined #openstack-infra | 23:13 | |
| dhinesh | this is the url i use... auth-url: 'http://server:5000/v2.0/' | 23:13 |
| dhellmann | fungi : I think we're ready for one more attempt when you're up for it | 23:14 |
| dhellmann | oh, and now I see it's already queued :-) | 23:14 |
| *** andreas_s has joined #openstack-infra | 23:14 | |
| smcginnis | He's quick like that. :) | 23:15 |
| *** claudiub has quit IRC | 23:15 | |
| smcginnis | So, so glad we're not doing this through reverts. | 23:15 |
| clarkb | fungi: legacy-base doesn't seem to do anything special | 23:15 |
| clarkb | mwhahaha: http://logs.openstack.org/10/509710/5/check/legacy-puppet-beaker-rspec-centos-7/97d576d/job-output.txt.gz#_2017-10-19_21_31_57_652130 thats the env of the shell in that job | 23:16 |
| clarkb | sure enough there is no zuul ref | 23:16 |
| clarkb | oh! | 23:16 |
| clarkb | I bet it is because there really isn't such a thing anymore? | 23:16 |
| clarkb | jeblair: ^ are we just not setting that at all even in the compat thing? | 23:17 |
| clarkb | (because where would you fetch from) | 23:17 |
| jeblair | clarkb: yes http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/filter/zuul_filters.py?h=feature/zuulv3#n16 | 23:17 |
| jeblair | clarkb: that is applied individually to each legacy job | 23:17 |
| smcginnis | dhinesh: I saw something in the ML about the v2 keystone API being removed. Maybe you need to update to v3? | 23:18 |
| *** andreas_s has quit IRC | 23:18 | |
| clarkb | mwhahaha: ^ there is the answer | 23:18 |
| jeblair | clarkb: eg http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/ansible-func-centos-7/run.yaml#n100 | 23:19 |
| mwhahaha | that means nothing to me really | 23:19 |
| * mwhahaha sighs and will figure it out later i guess | 23:19 | |
| clarkb | mwhahaha: it is intentionally omitted | 23:19 |
| mwhahaha | when? | 23:20 |
| dhellmann | fungi, smcginnis, jeblair : \o/ tag-releases passed | 23:20 |
| mwhahaha | today? | 23:20 |
| clarkb | mwhahaha: its in the comment of the file jeblair linked | 23:20 |
| dhellmann | now to see if the actual release job works... | 23:20 |
| clarkb | mwhahaha: no I think it has been, but would have to read the git log of that file to know for sure | 23:20 |
| johnsom | FYI, there has been a openstack/puppet-octavia job in the "gate" pipeline tripleo queue for 8 hrs 22min. I'm guessing something is not right with it. | 23:20 |
| smcginnis | dhellmann: I was just going to say - did that actually just work?! | 23:20 |
| dhellmann | yup | 23:20 |
| mwhahaha | well we just had the zuul-cloners fail today | 23:20 |
| dhellmann | http://zuulv3.openstack.org/static/stream.html?uuid=9cfa99f087de4b2fbe954cd2e79e0f7d&logfile=console.log | 23:20 |
| mwhahaha | they were working | 23:20 |
| mwhahaha | which is why i ask | 23:20 |
| mwhahaha | anyway i need to go, i'll figure it out | 23:20 |
| fungi | mwhahaha: clarkb: all legacy jobs explicitly add zuul_legacy_vars filter to the environment parameter | 23:20 |
| jeblair | clarkb, mwhahaha: yeah, ZUUL_REF has never been in zuulv3 | 23:20 |
| smcginnis | dhellmann, fungi: Do we want to get rid of the verbosity? Or leave that for awhile until we are comfortable everything is all good? | 23:21 |
| dhellmann | smcginnis : I say we don't touch things that are working :-) | 23:21 |
| smcginnis | dhellmann: Good call. | 23:21 |
| fungi | smcginnis: i think we can clean it up, but no rush | 23:21 |
| mwhahaha | oh wait these are legacy jobs | 23:21 |
| smcginnis | dhellmann: Shall we try the vitrage release? | 23:21 |
| mwhahaha | so is that still working? | 23:21 |
| dhellmann | we can clean that up when we move the job over to the releases repo | 23:21 |
| fungi | wfm | 23:21 |
| dhellmann | smcginnis : let's wait and see if the packaging job actually works | 23:22 |
| smcginnis | Oh yeah, that would be a good time to do it. | 23:22 |
| dhellmann | we're on step 2 of a 4 step job pipeline | 23:22 |
| *** markvoelker has quit IRC | 23:22 | |
| smcginnis | Getting too excited. :) | 23:22 |
| jeblair | mwhahaha: i don't see a failure in that link | 23:22 |
| dhellmann | tag-releases, $repo-specific-job-to-build-a-release, sign it, upload it, announce it | 23:22 |
| dhellmann | I think sign and upload are 1 step? | 23:22 |
| dhellmann | maybe it's 5 steps | 23:23 |
| dhellmann | upload depends on the repo type, too | 23:23 |
| jeblair | mwhahaha: oh found it | 23:23 |
| fungi | sign and upload are parallel steps | 23:23 |
| dhellmann | so we'll have to test puppet modules, too | 23:23 |
| dhellmann | ok | 23:23 |
| clarkb | jeblair: mwhahaha its because they invoke it directly using v2 z-c I tink http://logs.openstack.org/10/509710/5/check/legacy-puppet-beaker-rspec-centos-7/97d576d/job-output.txt.gz#_2017-10-19_21_35_15_101909 | 23:23 |
| clarkb | I don't expectanything we have done has changed to make that work/not work | 23:23 |
| clarkb | it should just not work right? | 23:23 |
| *** signed8b_ is now known as signed8bit_Zzz | 23:24 | |
| dhellmann | fungi, smcginnis : upload to pypi worked https://pypi.python.org/pypi/openstack-release-test | 23:25 |
| jeblair | mwhahaha: drop the "--zuul-ref" argument? (it shouldn't have been needed even in zuulv3) | 23:25 |
| fungi | dhellmann: so far so good | 23:25 |
| jeblair | er zuulv2 | 23:25 |
| dhellmann | watching the announce job in http://zuulv3.openstack.org/static/stream.html?uuid=020c7f4c1cfa49538349f90f5c7f284c&logfile=console.log | 23:26 |
| smcginnis | Rube Goldberg would be proud. | 23:26 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Update CI links to use zuul user https://review.openstack.org/512358 | 23:26 |
| clarkb | oh this is ocata | 23:26 |
| clarkb | I bet this was maybe fixed onsome other branch but not ocata? | 23:26 |
| clarkb | could explain why it works in some cases but not others | 23:27 |
| * dhellmann wonders why the announce job installs postgresql-client | 23:27 | |
| jeblair | clarkb: it sounds familiar | 23:27 |
| dhellmann | and mongodb? wth | 23:27 |
| *** smatzek has joined #openstack-infra | 23:27 | |
| clarkb | dhellmann: because it uses the bindep fallback | 23:27 |
| jeblair | dhellmann: fallback? | 23:27 |
| dhellmann | ah | 23:27 |
| dhellmann | ok, we can clean that up | 23:27 |
| smcginnis | Gotta store copies of the email somewhere. :) | 23:27 |
| dhellmann | or maybe I'll rewrite that script to use a database because it's there | 23:27 |
| fungi | if the repo provides a specific bindep.txt then the job will use that instead | 23:27 |
| dhellmann | what if it's empty? | 23:28 |
| smcginnis | Then it falls back. | 23:28 |
| fungi | then it will install no extra packages | 23:28 |
| dhellmann | I guess we need to make sure we have python at least | 23:28 |
| fungi | empty (not missing) | 23:28 |
| smcginnis | Oh, if it's there but empty. | 23:28 |
| smcginnis | Failure | 23:28 |
| clarkb | you probably want python, git, gpg in the list? | 23:28 |
| jeblair | missing = fallback. empty = noop. | 23:28 |
| dhellmann | ok, the announce job failed on the URL thing | 23:28 |
| fungi | so close! | 23:28 |
| smcginnis | http://logs.openstack.org/dd/dd0bb2d44053e11d9d4a4775b3c71f2c0889dc2a/release/propose-update-constraints/953c933/job-output.txt.gz#_2017-10-19_23_27_21_053560 | 23:28 |
| dhellmann | was my fix for that not approved? I thought it was... | 23:29 |
| dhellmann | maybe it doesn't work | 23:29 |
| smcginnis | Only thing that failed though! | 23:29 |
| dhellmann | oh, it does work, it's bailing on the inability to get the value because set -e | 23:29 |
| dhellmann | stand by | 23:29 |
| SamYaple | i need to copy /home/zuul/.ansible_async directory to /logs, but im recieving this error http://logs.openstack.org/79/512479/2/check/loci-cinder/6a83216/job-output.txt.gz#_2017-10-17_05_03_58_816169 | 23:29 |
| SamYaple | i dont quite understand it and cant reproduce it outside teh gate | 23:30 |
| clarkb | SamYaple: that is zuul protectingitself from bad actors | 23:30 |
| clarkb | SamYaple: you are restricted to copying files out of only theworking dir | 23:30 |
| mwhahaha | jeblair: ok thanks I'll give that a shot | 23:30 |
| SamYaple | clarkb: even with become? | 23:30 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools master: do not fail on inability to find remote url https://review.openstack.org/513559 | 23:30 |
| SamYaple | become: true | 23:30 |
| jeblair | SamYaple: that's happening on the executor, which is probably not what you intended | 23:30 |
| dhellmann | smcginnis , fungi : ^^ | 23:31 |
| jeblair | SamYaple: like maybe src/dest reversed or something | 23:31 |
| jeblair | SamYaple: link to playbook? | 23:31 |
| *** hemna_ has joined #openstack-infra | 23:31 | |
| SamYaple | jeblair: https://review.openstack.org/#/c/512479/2/playbooks/post.yaml | 23:31 |
| fungi | SamYaple: also (and i haven't looked at the log) be sure you rename it to something not starting with a . or else you won't be able to retrieve it from logs.o.o because apache | 23:31 |
| SamYaple | specifically i need that directory because all the async tasks i run pump out logs there and no where else | 23:31 |
| *** smatzek has quit IRC | 23:31 | |
| jeblair | SamYaple: ah it's /logs/ that's the issue | 23:31 |
| SamYaple | fungi: ok thansk for the tip | 23:31 |
| SamYaple | jeblair: ok so create relative logs dir perhaps? | 23:32 |
| jeblair | SamYaple: you'll want {{ zuul.executor.log_root }}/ https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html#var-zuul.executor.log_root | 23:32 |
| *** hemna_ has quit IRC | 23:32 | |
| SamYaple | ok let me try with that | 23:32 |
| smcginnis | dhellmann: Do we need another test patch in release-test? | 23:33 |
| SamYaple | i cant remember where i copied the copy logs task from anyway | 23:33 |
| dhellmann | smcginnis : no, fungi should be able to reenqueue just the announce job for that one | 23:33 |
| smcginnis | dhellmann: Oh right, good. | 23:33 |
| dhellmann | did the other jobs pass? | 23:33 |
| smcginnis | Yeah, looked like this was it. | 23:33 |
| fungi | smcginnis: dhellmann: yeah, i should be able to work out the job retrigger for that, just a sec | 23:34 |
| dhellmann | http://logs.openstack.org/dd/dd0bb2d44053e11d9d4a4775b3c71f2c0889dc2a/release/ | 23:34 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Fix neutron-lbaas-dashboard to use -horizon jobs https://review.openstack.org/513561 | 23:34 |
| *** Swami has quit IRC | 23:34 | |
| dhellmann | smcginnis : propose-update-constraints failed, too http://logs.openstack.org/dd/dd0bb2d44053e11d9d4a4775b3c71f2c0889dc2a/release/propose-update-constraints/953c933/job-output.txt.gz | 23:34 |
| dhellmann | it's failing to figure out the right branch | 23:35 |
| dhellmann | let's leave that one for tomorrow | 23:35 |
| *** markvoelker has joined #openstack-infra | 23:36 | |
| jeblair | well... we don't really have a way to run just one job anymore.... we can only re-enqueue an item and run the whole set of jobs. (ie, we can re-trigger on the tag push) | 23:36 |
| *** gildub has joined #openstack-infra | 23:36 | |
| dhellmann | oh, well, that's fine | 23:36 |
| dhellmann | whatever the smallest amount of stuff we can redo is fine. the jobs should be idempotent, except for that announce job | 23:36 |
| dhellmann | and that announce job will eventually fail because it tries to email me using the lists.openstack.org server as the SMTPD host, but if we get that far I think we can count it as working | 23:37 |
| jeblair | ++ | 23:37 |
| fungi | jeblair: oh, the trigger-job script won't work now? | 23:37 |
| dhellmann | so for the python pipeline we're down to the announce job and the constraint update job | 23:37 |
| jeblair | fungi: correct | 23:37 |
| fungi | i'll stop bothering to reconstruct the command line for it | 23:37 |
| dhellmann | we also need to test our other languages (puppet and javascript are different, I think?) | 23:38 |
| openstackgerrit | Michael Johnson proposed openstack-infra/openstack-zuul-jobs master: neutron-lbaas-dashboard jobs require horizon https://review.openstack.org/513562 | 23:38 |
| *** gmann_afk is now known as gmann | 23:38 | |
| *** camunoz has quit IRC | 23:39 | |
| fungi | okay, i need to drop off for the evening. the real world is calling again ;) | 23:39 |
| dhellmann | yes, it's going to be dinner time here soon. shall we pick this up in the morning? | 23:40 |
| jeblair | clarkb: can you review https://review.openstack.org/513441 ? | 23:40 |
| smcginnis | Tomorrow morning should be good. | 23:40 |
| smcginnis | Thanks all. | 23:40 |
| *** markvoelker has quit IRC | 23:40 | |
| jeblair | dhellmann, smcginnis: if you're done, i'll restart the scheduler now | 23:40 |
| dhellmann | thanks fungi, jeblair, & smcginnis -- we made good progress today | 23:40 |
| clarkb | jeblair: do you want ^ reviewed before you restart? | 23:40 |
| clarkb | I've pulled the change up now | 23:40 |
| smcginnis | jeblair: Yeah, I think so. Unless fungi was still doing anything. | 23:41 |
| jeblair | clarkb: please; when you give it the okay, i'll go ahead and hand-merge it onto zuulv3.o.o | 23:41 |
| smcginnis | Yes, great progress today. | 23:41 |
| dhellmann | jeblair : if you can let https://review.openstack.org/513559 merge that would be good, otherwise we can recheck it to push it in tomorrow | 23:41 |
| dhellmann | it's pretty close now, I think, those jobs shouldn't take too long | 23:41 |
| smcginnis | It's just about done in gate. | 23:41 |
| jeblair | no prob | 23:42 |
| dhellmann | ok, dinner calls, thanks again, everyone! | 23:42 |
| smcginnis | Enjoy! | 23:42 |
| *** bobh has quit IRC | 23:44 | |
| smcginnis | fungi, dhellmann: For tomorrow, looks like the publish-static release job has issues too: http://logs.openstack.org/fe/feb0fdabca16208a18c443f41104f0568beaa3dc/release-post/publish-static/a2a4723/job-output.txt.gz#_2017-10-19_23_24_48_431426 | 23:44 |
| SamYaple | jeblair: that was not the fix http://logs.openstack.org/79/512479/3/check/loci-heat/5a6568e/job-output.txt.gz#_2017-10-19_23_42_19_762213 https://review.openstack.org/#/c/512479/3/playbooks/post.yaml | 23:44 |
| openstackgerrit | Merged openstack-infra/release-tools master: do not fail on inability to find remote url https://review.openstack.org/513559 | 23:44 |
| SamYaple | im all for bad-actor protection, but i really need those logs and dont know of another way to get them | 23:45 |
| *** markvoelker has joined #openstack-infra | 23:45 | |
| *** smatzek has joined #openstack-infra | 23:46 | |
| ianw | jeblair / clarkb / fungi : ok, https://etherpad.openstack.org/p/rename-2017-10-20 is i think complete. could you please read through. will you be around ... since this is a first with a new zuul and a new gerrit i think we should be a bit more careful than usual | 23:46 |
| jeblair | SamYaple: why are you setting rsync_opts? | 23:47 |
| clarkb | ianw: ya I can read after finishing with this zuul change | 23:48 |
| clarkb | jeblair: this change is dense | 23:48 |
| clarkb | (a lot going on in a little code) | 23:48 |
| *** markvoelker has quit IRC | 23:49 | |
| jeblair | SamYaple: please promise me you will never copy the auto-generated legacy playbooks again. :) | 23:49 |
| jeblair | SamYaple: something like this is all you need: http://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/fetch-devstack-log-dir/tasks/main.yaml | 23:49 |
| jeblair | SamYaple: src, dest, and mode. | 23:49 |
| dhinesh | smcginnis: the v3 does not seem to work either.. | 23:50 |
| SamYaple | jeblair: to be fair, that was implemented like day 1 of the first zuulv3 things... | 23:50 |
| SamYaple | jeblair: ok, but thats not the task thats failing | 23:51 |
| clarkb | jeblair: had one uestion about something that may have been overlooked, can you see the comments on the change? | 23:51 |
| *** andreas_s has joined #openstack-infra | 23:51 | |
| *** wolverineav has joined #openstack-infra | 23:51 | |
| SamYaple | jeblair: i never make it to the "Copy logs" tasks | 23:51 |
| clarkb | ianw: everything looks good. One thing to check on is whether or not we need to rename any gerrit groups (not sure if the playbook does tat) | 23:53 |
| jeblair | SamYaple: ah it's the first copy task | 23:53 |
| ianw | clarkb: playbook does rename-group | 23:54 |
| jeblair | SamYaple: can you use copy to pull a file from the remote? i thought the dest was always remote. | 23:54 |
| *** markvoelker has joined #openstack-infra | 23:54 | |
| clarkb | ianw: cool, though I had to patch jeepyb to create groups we may want to doulble check gerrit 2.13 works iwth what playbook is running | 23:54 |
| *** s-shiono has joined #openstack-infra | 23:54 | |
| *** andreas_s has quit IRC | 23:55 | |
| clarkb | ianw: looks like it should | 23:55 |
| ianw | clarkb: ok, i could just manually rename a group now and then name it back? | 23:55 |
| jeblair | SamYaple: you probably want to use synchronize for all of those. | 23:55 |
| SamYaple | jeblair: well therein lies the issue, whereever ansible is executing will have /home/zuul/.ansible_async, and thats the directory i need | 23:55 |
| clarkb | ianw: ya thats one way of testing but reading the docs real quick https://review.openstack.org/Documentation/cmd-rename-group.html I think we are good | 23:55 |
| jeblair | SamYaple: you want zuul's .ansible_async directory? | 23:56 |
| ianw | clarkb: ok, let's assume it's ok :) | 23:56 |
| SamYaple | jeblair: ive just realized why thats going to be a problem, yes | 23:56 |
| SamYaple | jeblair: but yes, thats what i need because async tasks write out the tasks logs to that directory | 23:57 |
| jeblair | SamYaple: they don't end up in the json or ara logs? | 23:57 |
| pabelanger | I would guess it is in zuul.executor.work_dir/.ansible_async | 23:57 |
| SamYaple | not from what i can tell | 23:57 |
| pabelanger | because that is what we set HOME var too | 23:57 |
| SamYaple | pabelanger: ill check that | 23:57 |
| pabelanger | /home/zuul, isn't actually the home dir inside bweap on executor | 23:58 |
| jeblair | well it is | 23:58 |
| *** Apoorva_ has joined #openstack-infra | 23:58 | |
| *** signed8bit has joined #openstack-infra | 23:59 | |
| SamYaple | to be clear, i need this file http://logs.openstack.org/79/512479/3/check/loci-heat/5a6568e/job-output.txt.gz#_2017-10-19_23_40_33_542123 | 23:59 |
| *** markvoelker has quit IRC | 23:59 | |
| SamYaple | where is that file stored? | 23:59 |
| jeblair | or at least ~zuul is correct | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!