| *** yamamoto has quit IRC | 00:00 | |
| *** liusheng has joined #openstack-infra | 00:01 | |
| *** yamamoto has joined #openstack-infra | 00:01 | |
| *** yamamoto has quit IRC | 00:01 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 00:01 |
|---|---|---|
| *** pcrews has quit IRC | 00:01 | |
| *** yamamoto has joined #openstack-infra | 00:01 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Use non-legacy job results in neutron-dynamic-routing grafana https://review.openstack.org/538073 | 00:01 |
| dmsimard | dhellmann: the log upload granularity toggles are in, I'll submit a patch to enable them for list-changes | 00:02 |
| dhellmann | dmsimard : already done | 00:02 |
| prometheanfire | timeouts still a thing? | 00:02 |
| dmsimard | dhellmann: oh, alright then | 00:03 |
| dhellmann | dmsimard : mordred showed me what to do and I went ahead and added the variables | 00:03 |
| dmsimard | prometheanfire: what timeouts ? | 00:03 |
| prometheanfire | every single recheck failed | 00:04 |
| prometheanfire | in requirements | 00:04 |
| prometheanfire | https://review.openstack.org/537645 is the first one I've seen that wasn't a post fail or timeout | 00:05 |
| *** yamamoto has quit IRC | 00:06 | |
| *** yamamoto has joined #openstack-infra | 00:07 | |
| *** bobh has joined #openstack-infra | 00:08 | |
| *** caphrim007_ has joined #openstack-infra | 00:13 | |
| *** caphrim007_ has quit IRC | 00:13 | |
| *** rcarrillocruz has quit IRC | 00:15 | |
| *** gongysh has quit IRC | 00:16 | |
| *** caphrim007 has quit IRC | 00:16 | |
| dmsimard | prometheanfire: taking a look | 00:18 |
| *** verdurin_ has quit IRC | 00:18 | |
| *** armax has quit IRC | 00:19 | |
| *** verdurin_ has joined #openstack-infra | 00:19 | |
| prometheanfire | dmsimard: looks like we are still waiting on https://review.openstack.org/#/c/537453 | 00:20 |
| dmsimard | prometheanfire: ah I've seen something like that mentioned but not particularly aware of what's the status | 00:21 |
| prometheanfire | it's in the integrated queue | 00:21 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Use non-legacy job results in neutron-dynamic-routing grafana https://review.openstack.org/538073 | 00:22 |
| *** markvoelker has quit IRC | 00:22 | |
| *** panda|bbl is now known as panda|off | 00:24 | |
| EmilienM | ianw: hello! I tried the new yum-puppetlabs mirror and it doesn't work yet: http://logs.openstack.org/72/537572/9/check/puppet-openstack-integration-4-scenario001-tempest-centos-7/8c9c98d/job-output.txt.gz#_2018-01-25_20_13_17_542024 | 00:25 |
| EmilienM | ianw: like the mirror isn't synced yet? http://mirror.mtl01.inap.openstack.org/yum-puppetlabs/el/7/dependencies/ | 00:25 |
| EmilienM | corvus: I'm still waiting for the debug on https://review.openstack.org/#/c/537981/ FYI | 00:26 |
| dmsimard | EmilienM: ianw is on holiday today (friday) I think | 00:26 |
| *** larainema has joined #openstack-infra | 00:29 | |
| EmilienM | oh, ok! | 00:29 |
| EmilienM | it can wait :-) | 00:29 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Use non-legacy periodic job results of neutron-dynamic-routing in neutron grafana https://review.openstack.org/538084 | 00:30 |
| *** bobh has quit IRC | 00:39 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Add release note job to missing TripleO projects https://review.openstack.org/538091 | 00:42 |
| EmilienM | dhellmann: ^ fyi so next time we catch the problem | 00:42 |
| *** ihrachys has quit IRC | 00:42 | |
| dhellmann | EmilienM : +1 | 00:47 |
| corvus | EmilienM: i don't see that change on the status page... | 00:47 |
| EmilienM | corvus: maybe I should recheck? | 00:47 |
| *** dhill_ has quit IRC | 00:48 | |
| corvus | EmilienM: i think we've run into a misfeature -- i think because there were *no* jobs, it decided it shouldn't report anything. | 00:48 |
| corvus | EmilienM: obviously, when the goal is to debug why no jobs are running, that's a problem. | 00:48 |
| corvus | EmilienM: i can extract the needed info from the debug log though. gimme a sec. | 00:48 |
| *** ihrachys has joined #openstack-infra | 00:49 | |
| corvus | EmilienM: http://paste.openstack.org/show/653527/ | 00:49 |
| *** rlandy|biab is now known as rlandy | 00:50 | |
| EmilienM | ah | 00:50 |
| EmilienM | I found out I guess | 00:50 |
| *** zhurong has joined #openstack-infra | 00:51 | |
| *** JasonCL has quit IRC | 00:52 | |
| *** zhurong_ has joined #openstack-infra | 00:52 | |
| EmilienM | corvus: thanks for the light | 00:52 |
| pabelanger | EmilienM: I'll look at repo tomorrow for puppetlabs | 00:53 |
| EmilienM | corvus: well in fact wait | 00:53 |
| EmilienM | pabelanger: no problem | 00:53 |
| EmilienM | corvus: which branch of puppet-openstack-integration is it supposed to be checkouted? | 00:53 |
| EmilienM | what was the thing we did for puppet-ceph again | 00:54 |
| corvus | EmilienM: so you could probably solve this by adding 0.6.x to the implied branch matcher pragma (or by explicitly adding it to the job branch matchers). however, 0.6.x seems a little more generic than ocata or jewel. so i wonder if this might be a situation where you want to address this mapping in the puppet-pacemaker project itself, using the new feature i'm proposing in | 00:54 |
| corvus | https://review.openstack.org/537655 | 00:54 |
| EmilienM | because we had the same thing | 00:54 |
| EmilienM | corvus: I can use the new feature, I don't min | 00:55 |
| EmilienM | mind* | 00:55 |
| *** apetrich has quit IRC | 00:55 | |
| EmilienM | corvus: but what was the solution we took for puppet-ceph again? | 00:55 |
| EmilienM | I don't find any patch | 00:55 |
| *** pcrews has joined #openstack-infra | 00:55 | |
| corvus | EmilienM: it should be the pragma thing, but i don't know which repo it ended up in | 00:56 |
| EmilienM | oh | 00:56 |
| EmilienM | it was https://review.openstack.org/#/c/527747/7/.zuul.yaml | 00:56 |
| EmilienM | corvus: yeah in puppet-ceph we did that thing ^ | 00:56 |
| EmilienM | corvus: I can do pragma | 00:56 |
| EmilienM | corvus: when is it in prod though? | 00:56 |
| EmilienM | is it already in prod? | 00:56 |
| corvus | EmilienM: oh you did that. well, the pragma feature is in prod now. it lets you do the patch you just linked, but without all the "branches:" lines | 00:57 |
| corvus | EmilienM: you can use it to solve this too, the only downside is that if there's ever another project with a 0.6.x branch, there's no way to tell the difference between them. 537655 resolves that by putting the logic in p-pacemaker rather than p-o-integration. | 00:58 |
| EmilienM | ok, I prefer 537655 | 00:58 |
| EmilienM | corvus: but https://review.openstack.org/#/c/537655/ isn't merged | 00:59 |
| corvus | EmilienM: ok. my guess it'll be in production next week. so if you can't wait for it, maybe either update the branch matchers like in 527747, or switch that to pragma for now | 00:59 |
| corvus | EmilienM: my *guess* is that https://etherpad.openstack.org/p/A5SChpNtYV is what 537981 will look like for you. | 01:02 |
| corvus | EmilienM: sorry, i mean : | 01:03 |
| corvus | EmilienM: my *guess* is that https://etherpad.openstack.org/p/A5SChpNtYV is what 537655 will look like for you. | 01:03 |
| EmilienM | corvus: I can wait next week | 01:04 |
| EmilienM | corvus: we don't have ongoing backports on this branch now afik | 01:04 |
| EmilienM | so no pressure | 01:04 |
| *** aviau has quit IRC | 01:05 | |
| *** aviau has joined #openstack-infra | 01:06 | |
| EmilienM | corvus: https://review.openstack.org/537612 | 01:07 |
| *** pcrews has quit IRC | 01:08 | |
| *** armax has joined #openstack-infra | 01:09 | |
| *** stakeda has joined #openstack-infra | 01:10 | |
| corvus | EmilienM: cool. you may want to start using the new depends-on syntax ;) | 01:10 |
| EmilienM | corvus: which one? (see last patchset) | 01:11 |
| corvus | EmilienM: http://lists.openstack.org/pipermail/openstack-dev/2018-January/126535.html | 01:12 |
| EmilienM | oh nice | 01:12 |
| EmilienM | I'm late in email | 01:13 |
| EmilienM | corvus: updated ;-) | 01:15 |
| *** cuongnv has joined #openstack-infra | 01:18 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 01:23 |
| EmilienM | if someone can review https://review.openstack.org/#/c/538091/ please | 01:23 |
| *** flaper87 has quit IRC | 01:24 | |
| *** bobh has joined #openstack-infra | 01:24 | |
| *** gongysh has joined #openstack-infra | 01:24 | |
| *** bobh has quit IRC | 01:26 | |
| *** kiennt26 has joined #openstack-infra | 01:29 | |
| *** yolanda has quit IRC | 01:30 | |
| johnsom | FYI, the projects page on docs.openstack.org is blank... https://docs.openstack.org/pike/projects.html | 01:30 |
| johnsom | Not sure if that is related to the earlier disk issues or not | 01:30 |
| *** xarses has quit IRC | 01:34 | |
| prometheanfire | is the current queue time of 14 hours right? | 01:35 |
| prometheanfire | actually, 22 hours for integrated... | 01:36 |
| *** flaper87 has joined #openstack-infra | 01:36 | |
| *** annp has joined #openstack-infra | 01:40 | |
| *** markvoelker has joined #openstack-infra | 01:41 | |
| pabelanger | yes | 01:45 |
| *** dave-mccowan has joined #openstack-infra | 01:45 | |
| pabelanger | looking at ML, there seems to be a regressing resetting gate for integrated | 01:46 |
| *** apetrich has joined #openstack-infra | 01:52 | |
| *** hongbin has quit IRC | 01:52 | |
| *** armax has quit IRC | 01:53 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add release note job to missing TripleO projects https://review.openstack.org/538091 | 01:54 |
| *** ihrachys has quit IRC | 02:02 | |
| *** ihrachys has joined #openstack-infra | 02:02 | |
| *** salv-orl_ has joined #openstack-infra | 02:02 | |
| *** caphrim007 has joined #openstack-infra | 02:04 | |
| *** salv-orlando has quit IRC | 02:05 | |
| *** dayou has quit IRC | 02:08 | |
| *** caphrim007 has quit IRC | 02:08 | |
| *** jamesmcarthur has joined #openstack-infra | 02:16 | |
| *** harlowja has quit IRC | 02:16 | |
| prometheanfire | woo | 02:17 |
| *** jamesmcarthur has quit IRC | 02:20 | |
| *** zhurong has quit IRC | 02:24 | |
| dmsimard | prometheanfire: things are passing ? | 02:24 |
| prometheanfire | dmsimard: not yet, integrated gate is 22 hours behind | 02:28 |
| dmsimard | prometheanfire: ah I thought it was a "woo" of joy | 02:29 |
| dmsimard | it was a "woo" of sadness :( | 02:29 |
| prometheanfire | lol, ya | 02:29 |
| dmsimard | or perhaps an ironic one. | 02:29 |
| dmsimard | s/ironic/sarcastic/ | 02:29 |
| prometheanfire | sarcastic | 02:29 |
| dmsimard | could've used some good news today | 02:29 |
| *** rkukura has quit IRC | 02:31 | |
| *** vaidy has joined #openstack-infra | 02:32 | |
| *** SergeyLukjanov has quit IRC | 02:33 | |
| *** isviridov_away has joined #openstack-infra | 02:33 | |
| *** SergeyLukjanov has joined #openstack-infra | 02:35 | |
| prometheanfire | ya, it's been rough | 02:41 |
| smcginnis | Anyone know what happened here: http://logs.openstack.org/fb/fb9d1708a56156cff94bad09a859f1bfa34cc5e2/pre-release/release-openstack-python-without-pypi/65e4cfb/job-output.txt.gz#_2018-01-26_01_35_45_102294 | 02:44 |
| smcginnis | Or rather, what needs to be done to make that pass? | 02:44 |
| dmsimard | smcginnis: this is the underlying error: 2018-01-26 01:35:45,092 DEBUG zuul.AnsibleJob: [build: 65e4cfb4da124bceab4e397ccf052b92] Ansible output: b"An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 1] Operation not permitted: '/srv/static/tarballs/trove/images/README'" | 02:47 |
| dmsimard | according to ara, things failed here: http://logs.openstack.org/fb/fb9d1708a56156cff94bad09a859f1bfa34cc5e2/pre-release/release-openstack-python-without-pypi/65e4cfb/ara/ | 02:48 |
| smcginnis | Hmm, so... what needs to be done for that? Something wrong on tarballs.o.o? | 02:48 |
| dmsimard | smcginnis: I have no idea but we'll find out, just being verbose as I jump through the hoops to figure it out :) | 02:48 |
| smcginnis | dmsimard: OK, excellent. Thanks. | 02:48 |
| dmsimard | smcginnis: hmm, so there is a README file at that location.. dating back to July 2017 with root/root permissions so the "jenkins" user can't overwrite it. The math checks out. | 02:51 |
| smcginnis | Odd. Can we just manually delete it and rerun the job? | 02:51 |
| dmsimard | This is the contents of that file: http://paste.openstack.org/raw/653530/ | 02:51 |
| *** dayou has joined #openstack-infra | 02:52 | |
| dmsimard | yeah, I mean the file is exactly this one in fact http://tarballs.openstack.org/trove/images/ | 02:52 |
| dmsimard | Did that file change recently perhaps ? | 02:52 |
| dmsimard | I'm trying to understand why it's failing just now | 02:52 |
| smcginnis | Looking... | 02:52 |
| pabelanger | I think we added that README manually, when we deleted trove images | 02:53 |
| smcginnis | I'm not seeing it in the repo. | 02:53 |
| smcginnis | So where would it be coming from now. I mean, it's trying to overwrite that from something, right? | 02:53 |
| dmsimard | smcginnis: oh I know why | 02:54 |
| *** caphrim007 has joined #openstack-infra | 02:54 | |
| dmsimard | it doesn't try to put anything in there -- but it tries to chmod it | 02:54 |
| dmsimard | http://logs.openstack.org/fb/fb9d1708a56156cff94bad09a859f1bfa34cc5e2/pre-release/release-openstack-python-without-pypi/65e4cfb/ara/ | 02:54 |
| dmsimard | 775 recurse yes | 02:54 |
| smcginnis | OK, I see. So we manually chown it? | 02:55 |
| dmsimard | hmm that's directory though, not file.. | 02:55 |
| dmsimard | that looks like an Ansible bug | 02:55 |
| smcginnis | Well, if that was from 2015, should we just remove it now? | 02:56 |
| *** xarses has joined #openstack-infra | 02:56 | |
| pabelanger | maybe change chown jenkins:jenkisn, run jobs again, see what happens | 02:57 |
| dmsimard | pabelanger: yeah I just did that | 02:58 |
| pabelanger | k, I can help more in the morning | 02:58 |
| pabelanger | good luck | 02:58 |
| dmsimard | np I'm going to babysit the fsck for a while | 02:58 |
| dmsimard | smcginnis: not really my call whether that should be delete but I just fixed the permissions | 02:58 |
| smcginnis | dmsimard: Either way is good with me. | 02:58 |
| dmsimard | smcginnis: do we need to rerun anything ? | 02:59 |
| smcginnis | I don't think that did anything permanent that we couldn't re-enqueue the job. | 03:00 |
| dmsimard | ok let me see if I can do that | 03:00 |
| smcginnis | Thanks! | 03:02 |
| smcginnis | That one is just release-openstack-python-without-pypi and announce-release, so as long as it's out on tarballs.o.o, we're fine. | 03:02 |
| smcginnis | Release email would be nice, but definitely not critical. | 03:02 |
| *** andreww has joined #openstack-infra | 03:03 | |
| *** xarses_ has joined #openstack-infra | 03:05 | |
| dmsimard | smcginnis: yeah it's getting the different parameters right that's a bit tricky | 03:06 |
| dmsimard | looks like this: zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release-post --project=openstack/releases --ref=refs/heads/master --newrev=323c387a2d1794e0679510657629470da8f7de92 | 03:07 |
| *** rosmaita has quit IRC | 03:08 | |
| *** andreww has quit IRC | 03:08 | |
| *** chenying has joined #openstack-infra | 03:08 | |
| smcginnis | Looks right, as far as I can tell. | 03:09 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add babel transpiling enabling use of ES6 features https://review.openstack.org/538125 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add StandardJS linting and analysis https://review.openstack.org/538126 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Fix source_url handling for jobs view https://review.openstack.org/538127 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Fix StandardJS warnings and turn them to errors https://review.openstack.org/538128 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Inject url endpoint information https://review.openstack.org/538130 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Make bundle of build web content https://review.openstack.org/538131 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Remove use strict https://review.openstack.org/538132 | 03:10 |
| dmsimard | smcginnis: yeah but that particular line was for fixing another thing, was just giving you an example -- anyway here's what our re-run of that job looks like: | 03:11 |
| dmsimard | zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=pre-release --project=openstack/trove --ref=refs/tags/9.0.0.0b3 --newrev=fb9d1708a56156cff94bad09a859f1bfa34cc5e2 | 03:11 |
| dmsimard | sending it now | 03:12 |
| smcginnis | fingers crossed | 03:13 |
| *** rkukura has joined #openstack-infra | 03:14 | |
| dmsimard | smcginnis: lgtm https://i.imgur.com/VvxLeHI.png | 03:18 |
| *** rkukura has quit IRC | 03:25 | |
| *** hongbin has joined #openstack-infra | 03:27 | |
| *** dougwig has quit IRC | 03:31 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Set the vhost_name for zuul01 to zuul.openstack.org https://review.openstack.org/538136 | 03:34 |
| dmsimard | infra-root: it would be pretty important to get that in quickly ^ we don't currently have zuul scheduler backups | 03:34 |
| *** olaph1 has joined #openstack-infra | 03:36 | |
| *** olaph has quit IRC | 03:38 | |
| *** xarses_ has quit IRC | 03:39 | |
| *** pramodrj07 has quit IRC | 03:43 | |
| *** dave-mccowan has quit IRC | 03:49 | |
| *** ihrachys has quit IRC | 04:05 | |
| *** gongysh has quit IRC | 04:07 | |
| *** ihrachys has joined #openstack-infra | 04:08 | |
| *** jbadiapa has quit IRC | 04:10 | |
| *** mriedem has quit IRC | 04:23 | |
| *** andreas_s has joined #openstack-infra | 04:25 | |
| *** andreas_s has quit IRC | 04:30 | |
| *** jbadiapa has joined #openstack-infra | 04:31 | |
| openstackgerrit | Merged openstack-infra/puppet-statusbot master: Add configuration for thanks/success page url https://review.openstack.org/531185 | 04:31 |
| openstackgerrit | Merged openstack-infra/system-config master: Set the success/thanks page url for status bot https://review.openstack.org/531187 | 04:31 |
| *** ganso has quit IRC | 04:33 | |
| *** zhurong_ has quit IRC | 04:39 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 04:41 |
| *** harlowja has joined #openstack-infra | 04:51 | |
| dmsimard | infra-root: fsck still in progress on logs.o.o, progress @ 75.6% on phase 2. Server vitals are still stable but I've reduced the file deletion timeout down to 600m since the server is going unsupervised for a bit. I've also added disk usage reporting in the loop. | 04:53 |
| dmsimard | Everything is in the root screen "fsck", there's 3 windows in the screen. | 04:53 |
| dmsimard | Going to sleep, talk to you tomorrow | 04:55 |
| *** openstackstatus has quit IRC | 04:56 | |
| *** openstackstatus has joined #openstack-infra | 04:57 | |
| *** ChanServ sets mode: +v openstackstatus | 04:57 | |
| *** gongysh has joined #openstack-infra | 05:11 | |
| *** jbadiapa has quit IRC | 05:11 | |
| *** hamzy has joined #openstack-infra | 05:12 | |
| *** gongysh has quit IRC | 05:16 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 05:17 |
| *** zhurong has joined #openstack-infra | 05:18 | |
| *** CrayZee has joined #openstack-infra | 05:18 | |
| *** CrayZee has quit IRC | 05:21 | |
| *** CrayZee has joined #openstack-infra | 05:22 | |
| *** CrayZee has quit IRC | 05:22 | |
| *** claudiub has joined #openstack-infra | 05:23 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: /{tenant}/projects.json routes https://review.openstack.org/537870 | 05:49 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 05:49 |
| *** links has joined #openstack-infra | 05:58 | |
| *** hongbin has quit IRC | 05:59 | |
| openstackgerrit | shangxdy proposed openstack-infra/gear master: Modify connection timeout process https://review.openstack.org/531059 | 06:12 |
| *** pcichy has joined #openstack-infra | 06:20 | |
| *** dbecker has quit IRC | 06:31 | |
| *** psachin has joined #openstack-infra | 06:38 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix cross source dependencies for github https://review.openstack.org/537428 | 06:44 |
| *** olaph has joined #openstack-infra | 06:45 | |
| *** dbecker has joined #openstack-infra | 06:46 | |
| *** olaph1 has quit IRC | 06:47 | |
| *** s-shiono has joined #openstack-infra | 06:47 | |
| *** gouthamr has joined #openstack-infra | 06:48 | |
| *** s-shiono_ has joined #openstack-infra | 06:48 | |
| *** s-shiono has quit IRC | 06:51 | |
| *** zhurong has quit IRC | 06:52 | |
| *** rlandy has quit IRC | 06:57 | |
| *** namnh has joined #openstack-infra | 07:03 | |
| *** pblaho has quit IRC | 07:05 | |
| *** andreas_s has joined #openstack-infra | 07:06 | |
| AJaeger | ian, frickler, could you put https://review.openstack.org/#/c/536100/ https://review.openstack.org/#/c/513228/ and https://review.openstack.org/537970 on your review queue , please? | 07:12 |
| *** harlowja has quit IRC | 07:23 | |
| *** slaweq has joined #openstack-infra | 07:27 | |
| openstackgerrit | shangxdy proposed openstack-infra/gear master: Modify connection timeout process https://review.openstack.org/531059 | 07:28 |
| *** slaweq has quit IRC | 07:31 | |
| *** slaweq has joined #openstack-infra | 07:33 | |
| *** slaweq has quit IRC | 07:33 | |
| *** jtomasek has joined #openstack-infra | 07:34 | |
| *** slaweq has joined #openstack-infra | 07:35 | |
| *** e0ne has joined #openstack-infra | 07:36 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in tap-as-a-service https://review.openstack.org/513227 | 07:40 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in tap-as-a-service https://review.openstack.org/513227 | 07:41 |
| *** psachin has quit IRC | 07:46 | |
| *** rcarrillocruz has joined #openstack-infra | 07:47 | |
| *** florianf has joined #openstack-infra | 07:53 | |
| *** olaph1 has joined #openstack-infra | 07:54 | |
| *** olaph has quit IRC | 07:55 | |
| *** d0ugal has quit IRC | 08:04 | |
| *** jpich has joined #openstack-infra | 08:14 | |
| *** armaan has quit IRC | 08:15 | |
| *** armaan has joined #openstack-infra | 08:16 | |
| *** d0ugal has joined #openstack-infra | 08:23 | |
| *** amoralej|off is now known as amoralej | 08:24 | |
| *** stakeda has quit IRC | 08:24 | |
| AJaeger | frickler: what do you think of promoting https://review.openstack.org/#/c/538027/ - see http://lists.openstack.org/pipermail/openstack-dev/2018-January/126586.html ? Should we wait for a gate reset? | 08:27 |
| *** cshastri has joined #openstack-infra | 08:28 | |
| *** cshastri has quit IRC | 08:28 | |
| *** cshastri has joined #openstack-infra | 08:30 | |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config master: Use horizon publish job for blazar-dashboard https://review.openstack.org/538182 | 08:31 |
| *** alexchadin has joined #openstack-infra | 08:33 | |
| *** gouthamr_ has joined #openstack-infra | 08:34 | |
| *** yamahata has joined #openstack-infra | 08:35 | |
| *** ralonsoh has joined #openstack-infra | 08:35 | |
| *** gouthamr has quit IRC | 08:36 | |
| *** gouthamr has joined #openstack-infra | 08:39 | |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config master: Add publish-to-pypi in nova-blazar repo https://review.openstack.org/538185 | 08:39 |
| *** s-shiono_ has quit IRC | 08:40 | |
| *** gouthamr_ has quit IRC | 08:41 | |
| *** alexchadin has quit IRC | 08:41 | |
| *** alexchadin has joined #openstack-infra | 08:42 | |
| *** jpena|off is now known as jpena | 08:43 | |
| *** kjackal has joined #openstack-infra | 09:03 | |
| *** salv-orl_ has quit IRC | 09:07 | |
| *** rpittau has joined #openstack-infra | 09:09 | |
| *** salv-orlando has joined #openstack-infra | 09:11 | |
| *** kzaitsev_pi has quit IRC | 09:15 | |
| *** liusheng has quit IRC | 09:19 | |
| *** liusheng has joined #openstack-infra | 09:19 | |
| *** kzaitsev_pi has joined #openstack-infra | 09:25 | |
| *** sambetts|afk is now known as sambetts | 09:27 | |
| *** olaph1 has quit IRC | 09:30 | |
| *** olaph has joined #openstack-infra | 09:31 | |
| *** derekh has joined #openstack-infra | 09:38 | |
| *** rossella_s has quit IRC | 09:43 | |
| *** rossella_s has joined #openstack-infra | 09:45 | |
| *** e0ne has quit IRC | 09:52 | |
| *** e0ne_ has joined #openstack-infra | 09:52 | |
| *** electrofelix has joined #openstack-infra | 09:53 | |
| *** alexchadin has quit IRC | 09:58 | |
| *** ganso has joined #openstack-infra | 09:58 | |
| *** arxcruz is now known as arxcruz|off | 10:04 | |
| *** lucas-afk is now known as lucasagomes | 10:07 | |
| *** cuongnv has quit IRC | 10:07 | |
| *** namnh has quit IRC | 10:08 | |
| Roamer` | yeah, promoting 538027 might indeed save quite a lot of rechecks today | 10:12 |
| *** kiennt26 has quit IRC | 10:14 | |
| stephenfin | Interesting issue here: It seems the GitHub mirror for nova-specs is not updated. See https://git.openstack.org/cgit/openstack/nova-specs/ vs. https://github.com/openstack/nova-specs | 10:22 |
| frickler | stephenfin: iirc this is a known issue, some weirdness with our git repo | 10:24 |
| stephenfin | frickler: Ah, OK. Once it's known | 10:24 |
| stephenfin | I was seeing merge conflicts that made no sense :) | 10:24 |
| *** gouthamr has quit IRC | 10:25 | |
| frickler | stephenfin: ah, if you base your local copy on github, that will be an issue indeed. infra-root, can someone give a better status update on this one? | 10:26 |
| *** yamamoto has quit IRC | 10:30 | |
| frickler | AJaeger: yeah, I'm going to try to promote that patch now, doesn't look like things can get much worse anyway | 10:30 |
| *** ldnunes has joined #openstack-infra | 10:33 | |
| *** pbourke has quit IRC | 10:38 | |
| *** pbourke has joined #openstack-infra | 10:40 | |
| *** rm_work has quit IRC | 10:42 | |
| *** yamahata has quit IRC | 10:43 | |
| *** rm_work has joined #openstack-infra | 10:46 | |
| openstackgerrit | Merged openstack-infra/project-config master: translation-jobs for neutron-vpnaas-dashboard https://review.openstack.org/537970 | 10:47 |
| *** yamamoto has joined #openstack-infra | 10:53 | |
| *** daidv has quit IRC | 10:58 | |
| *** yamamoto has quit IRC | 10:58 | |
| *** goutham has joined #openstack-infra | 10:59 | |
| *** tellesnobrega has quit IRC | 11:00 | |
| *** yamamoto has joined #openstack-infra | 11:01 | |
| *** yamamoto has quit IRC | 11:01 | |
| *** panda|off is now known as panda | 11:01 | |
| *** sshnaidm|off has quit IRC | 11:11 | |
| *** sshnaidm has joined #openstack-infra | 11:12 | |
| *** jkilpatr has quit IRC | 11:24 | |
| *** johnthetubaguy has quit IRC | 11:26 | |
| *** johnthetubaguy has joined #openstack-infra | 11:28 | |
| *** Adri2000 has quit IRC | 11:31 | |
| *** goutham has quit IRC | 11:38 | |
| *** Adri2000 has joined #openstack-infra | 11:39 | |
| *** Adri2000 has quit IRC | 11:39 | |
| *** Adri2000 has joined #openstack-infra | 11:39 | |
| ianw | frickler / stephenfin : yeah, i had it scheduled to fsck when we did a rename, but due to zuulv3 we haven't done any renames yet. see https://etherpad.openstack.org/p/rename-2017-10-20 (date gives it away :) | 11:43 |
| *** armaan has quit IRC | 11:47 | |
| *** armaan has joined #openstack-infra | 11:47 | |
| *** tellesnobrega has joined #openstack-infra | 11:54 | |
| *** rosmaita has joined #openstack-infra | 11:54 | |
| *** agopi|out has quit IRC | 11:58 | |
| *** andreas_s has quit IRC | 11:58 | |
| *** andreas_s has joined #openstack-infra | 11:58 | |
| *** jkilpatr has joined #openstack-infra | 11:59 | |
| *** b_bezak has joined #openstack-infra | 12:01 | |
| *** yamamoto has joined #openstack-infra | 12:02 | |
| *** armaan has quit IRC | 12:03 | |
| *** andreas_s has quit IRC | 12:03 | |
| *** armaan has joined #openstack-infra | 12:03 | |
| *** hjensas has quit IRC | 12:08 | |
| *** hjensas has joined #openstack-infra | 12:09 | |
| *** hjensas has quit IRC | 12:09 | |
| *** hjensas has joined #openstack-infra | 12:09 | |
| *** yamamoto has quit IRC | 12:13 | |
| *** dave-mccowan has joined #openstack-infra | 12:14 | |
| *** andreas_s has joined #openstack-infra | 12:14 | |
| *** kjackal has quit IRC | 12:20 | |
| *** annp has quit IRC | 12:21 | |
| *** salv-orlando has quit IRC | 12:26 | |
| *** salv-orlando has joined #openstack-infra | 12:26 | |
| *** salv-orlando has quit IRC | 12:31 | |
| *** salv-orlando has joined #openstack-infra | 12:38 | |
| *** jpena is now known as jpena|lunch | 12:39 | |
| *** hamzy has quit IRC | 12:43 | |
| *** hamzy has joined #openstack-infra | 12:46 | |
| *** janki has joined #openstack-infra | 12:51 | |
| *** kjackal has joined #openstack-infra | 12:52 | |
| *** ldnunes has quit IRC | 12:59 | |
| *** ldnunes has joined #openstack-infra | 13:00 | |
| *** gcb has quit IRC | 13:02 | |
| claudiub | hi. it seems that Jenkins always fails on https://review.openstack.org/#/c/536774/1 with the error message: error: can't copy 'etc/glance-image-import.conf': doesn't exist or not a regular file. any ideas? | 13:04 |
| *** dprince has joined #openstack-infra | 13:07 | |
| frickler | claudiub: there is no Jenkins, only Zuul. ;) also I looked at the error and didn't find anything obvious to me, best ask in either #openstack-requirements and/or #openstack-glance | 13:11 |
| *** trown|outtypewww is now known as trown|rover | 13:12 | |
| *** edmondsw has joined #openstack-infra | 13:13 | |
| *** jcoufal has joined #openstack-infra | 13:16 | |
| claudiub | ah, forgot that we switched from jenkins to zuul. | 13:17 |
| claudiub | anyways, thanks. :) | 13:17 |
| *** matbu has joined #openstack-infra | 13:21 | |
| *** rlandy has joined #openstack-infra | 13:34 | |
| *** liusheng has quit IRC | 13:40 | |
| *** jpena|lunch is now known as jpena | 13:45 | |
| *** armaan has quit IRC | 13:46 | |
| zxiiro | how do I get a patch that has hte necessary +2 and workflow +1 merged if the merge job never ran? https://review.openstack.org/479905 | 13:55 |
| *** panda is now known as panda|lunch | 13:55 | |
| *** kjackal has quit IRC | 13:55 | |
| *** jamesmcarthur has joined #openstack-infra | 13:55 | |
| *** dmellado has joined #openstack-infra | 13:58 | |
| dmellado | AJaeger: ping re clearlinux repo, seems to work now ;) | 14:00 |
| dmellado | thanks! | 14:00 |
| *** mriedem has joined #openstack-infra | 14:02 | |
| *** jamesmcarthur has quit IRC | 14:02 | |
| *** jamesmcarthur has joined #openstack-infra | 14:03 | |
| *** stephenfin is now known as finucannot | 14:05 | |
| AJaeger | dmellado: great to hear! | 14:06 |
| frickler | zxiiro: the patch needs a verified+1 from zuul now, just do a "recheck" | 14:06 |
| zxiiro | frickler: ah didn't notice. thanks for the tip! | 14:07 |
| gibi | AJaeger: FYI, I found more confusing behaviors regarding zuul job (re-)definitions and the irrelevant-files attribute http://lists.openstack.org/pipermail/openstack-dev/2018-January/126616.html | 14:08 |
| AJaeger | gibi: sorry, no time to help here - please discuss with the rest of the team... | 14:08 |
| *** kgiusti has joined #openstack-infra | 14:09 | |
| *** amoralej is now known as amoralej|lunch | 14:09 | |
| gibi | AJaeger: no problem | 14:09 |
| *** bobh has joined #openstack-infra | 14:17 | |
| *** esberglu has joined #openstack-infra | 14:18 | |
| *** Goneri has joined #openstack-infra | 14:20 | |
| *** jamesmcarthur has quit IRC | 14:20 | |
| *** Goneri has quit IRC | 14:21 | |
| *** armaan has joined #openstack-infra | 14:21 | |
| *** Goneri has joined #openstack-infra | 14:23 | |
| *** lucasagomes is now known as lucas-hungry | 14:26 | |
| *** agopi|out has joined #openstack-infra | 14:27 | |
| openstackgerrit | Javier Peña proposed openstack-infra/system-config master: Move AFS mirror code to puppet-openstackci https://review.openstack.org/529032 | 14:34 |
| *** david-lyle has quit IRC | 14:35 | |
| *** yamamoto has joined #openstack-infra | 14:38 | |
| *** agopi|out has quit IRC | 14:39 | |
| *** r-daneel has joined #openstack-infra | 14:39 | |
| *** yamamoto has quit IRC | 14:39 | |
| *** agopi|out has joined #openstack-infra | 14:39 | |
| *** efried is now known as fried_rice | 14:39 | |
| *** eharney has joined #openstack-infra | 14:41 | |
| *** dansmith is now known as superdan | 14:41 | |
| *** yamamoto has joined #openstack-infra | 14:41 | |
| *** claudiub has quit IRC | 14:47 | |
| *** claudiub|2 has joined #openstack-infra | 14:47 | |
| *** tesseract has joined #openstack-infra | 14:49 | |
| *** jamesmcarthur has joined #openstack-infra | 14:51 | |
| pabelanger | morning | 14:51 |
| *** yolanda has joined #openstack-infra | 14:54 | |
| *** amoralej|lunch is now known as amoralej | 14:56 | |
| *** jamesmcarthur has quit IRC | 14:56 | |
| *** d0ugal has quit IRC | 14:57 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Split out erasing from 'info' command into 'erase' https://review.openstack.org/537663 | 14:57 |
| *** jamesmcarthur has joined #openstack-infra | 14:57 | |
| *** CrayZee has joined #openstack-infra | 14:59 | |
| *** CrayZee is now known as snapiri- | 14:59 | |
| dhellmann | I'm just catching up with the release state and it looks like we had some more job failures a little while ago (within the last hour). Is there an issue being worked already? | 14:59 |
| *** tellesnobrega has quit IRC | 15:00 | |
| *** wolverineav has joined #openstack-infra | 15:00 | |
| dmsimard | mnaser: are we able to re-enable vexxhost in nodepool ? It's been disabled since december and every bit could help right now https://github.com/openstack-infra/project-config/commit/a1b3333b54637d7dfea634c5069977c496b4ef7f#diff-8b0ab6c0ec4c04693bc98cdbd59d975b | 15:00 |
| *** r-daneel has quit IRC | 15:01 | |
| *** salv-orlando has quit IRC | 15:03 | |
| pabelanger | before we do that, we'll need to look at mirror bandwidth, that was the bottle neck I believe last time | 15:03 |
| *** salv-orlando has joined #openstack-infra | 15:04 | |
| *** jaosorior has quit IRC | 15:04 | |
| dmsimard | dhellmann: things should be stable right now, what particular failure are you looking at ? | 15:04 |
| *** matbu has quit IRC | 15:05 | |
| *** janki has quit IRC | 15:05 | |
| dmsimard | dhellmann: I know that claudiub|2 mentioned a glance issue on the requirements repo but it didn't seem infrastructure related | 15:05 |
| *** panda|lunch is now known as panda | 15:05 | |
| dhellmann | dmsimard : this one just came in. http://lists.openstack.org/pipermail/release-job-failures/2018-January/000730.html | 15:05 |
| dhellmann | I'm running the release team meeting right now so I haven't looked at the logs yet | 15:05 |
| dmsimard | dhellmann: ok I'll take a quick look at that one | 15:06 |
| dmsimard | post failure with no logs, ew | 15:06 |
| dmsimard | hmm, that was actually like 9 hours ago now ? | 15:06 |
| dhellmann | oops, hang on | 15:07 |
| dmsimard | er, a bit longer than that Fri Jan 26 01:36:30 UTC 2018 | 15:07 |
| dhellmann | http://lists.openstack.org/pipermail/release-job-failures/2018-January/000733.html | 15:07 |
| dhellmann | try that one | 15:07 |
| dhellmann | we need the trove one rerun but that one is less than an hour old | 15:07 |
| dhellmann | and might be easier to debug | 15:07 |
| *** salv-orlando has quit IRC | 15:08 | |
| *** pcichy has quit IRC | 15:11 | |
| *** pcichy has joined #openstack-infra | 15:11 | |
| *** gibi is now known as giblet | 15:12 | |
| *** yamahata has joined #openstack-infra | 15:13 | |
| *** d0ugal has joined #openstack-infra | 15:13 | |
| *** jamesmcarthur has quit IRC | 15:14 | |
| *** rossella_s has quit IRC | 15:15 | |
| dmsimard | dhellmann: I know that last night I re-ran a trove pre-release with smcginnis | 15:16 |
| dmsimard | looking at that second one now | 15:16 |
| dmsimard | ok that second failure you sent me is definitely not normal | 15:17 |
| dhellmann | dmsimard : thanks, sorry for the confusion about the trove one | 15:17 |
| *** gcb has joined #openstack-infra | 15:17 | |
| *** matbu has joined #openstack-infra | 15:17 | |
| *** rossella_s has joined #openstack-infra | 15:19 | |
| frickler | infra-root: AJaeger: FYI I'll be mostly offline next week, I may step by occasionally, but don't count on it | 15:20 |
| dmsimard | infra-root: I'll also be mostly offline next week -- going through some certification training | 15:21 |
| *** gcb has quit IRC | 15:23 | |
| *** slaweq has quit IRC | 15:24 | |
| *** slaweq has joined #openstack-infra | 15:25 | |
| dmsimard | dhellmann: it failed during the log upload :( 2018-01-26 14:37:47,977 DEBUG zuul.AnsibleJob: [build: a313a980d05f4526aba5128189f45cd7] Ansible output: b'fatal: [logs.openstack.org]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \\"logs.openstack.org\\". Make sure this host can be reached over ssh", "unreachable": true}' | 15:25 |
| dmsimard | there was a failure prior to that as well, trying to understand what happened | 15:26 |
| dmsimard | mordred: looks like a bug with zuul_stream http://paste.openstack.org/raw/653638/ | 15:28 |
| dmsimard | I'll check out what's the warning for the configure mirrors role | 15:28 |
| *** kjackal has joined #openstack-infra | 15:28 | |
| *** slaweq has quit IRC | 15:29 | |
| *** armaan has quit IRC | 15:30 | |
| *** armaan has joined #openstack-infra | 15:30 | |
| dmsimard | er, that's two builds interleaved.. got me confused | 15:30 |
| *** dave-mccowan has quit IRC | 15:31 | |
| *** lucas-hungry is now known as lucasagomes | 15:31 | |
| *** camunoz has joined #openstack-infra | 15:32 | |
| dhellmann | dmsimard, smcginnis : I've updated https://etherpad.openstack.org/p/6GloEgik8b with the details from today's failures | 15:34 |
| smcginnis | Thanks | 15:34 |
| *** armaan_ has joined #openstack-infra | 15:36 | |
| *** armaan has quit IRC | 15:36 | |
| *** rossella_s has quit IRC | 15:37 | |
| *** david-lyle has joined #openstack-infra | 15:37 | |
| mordred | dmsimard: ugh | 15:39 |
| *** agopi|out has quit IRC | 15:39 | |
| *** rossella_s has joined #openstack-infra | 15:39 | |
| *** jamesmcarthur has joined #openstack-infra | 15:47 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move github webhook from webapp to zuul-web https://review.openstack.org/535711 | 15:48 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move status_url from webapp to web section https://review.openstack.org/536773 | 15:48 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove webapp https://review.openstack.org/536780 | 15:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add web server URL tests https://review.openstack.org/538291 | 15:50 |
| *** jamesmcarthur has quit IRC | 15:51 | |
| dhellmann | dmsimard : that error doesn't mean much to me | 15:51 |
| dmsimard | dhellmann: yeah, I added a full paste in the pad mostly to share with the rest of the infra team .. lots of things went wrong with that job :/ | 15:53 |
| dhellmann | should we investigate more before re-running it? I can put an experimental release for the release-test repo into the queue if we want a test job | 15:54 |
| dmsimard | dhellmann: we don't really need to investigate the log upload failure, but it's the failure that occured before that I'm trying to understand -- the logs don't provide a lot of info so I'm digging | 15:55 |
| dhellmann | ok | 15:55 |
| dhellmann | dmsimard : I'll look at some other things while you do that, so please ping me when you're comfortable with either re-running that one or setting up another as a test. | 15:55 |
| *** slaweq has joined #openstack-infra | 15:56 | |
| *** makowals has quit IRC | 15:56 | |
| dmsimard | thanks | 15:56 |
| dmsimard | will do. | 15:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add web server URL tests https://review.openstack.org/538291 | 15:57 |
| dmsimard | btw corvus, infra-root: I'd like to get https://review.openstack.org/#/c/538136/ in ASAP to get zuul scheduler backups working again | 15:58 |
| corvus | dmsimard: how does the backup script know what hostname to use? | 15:59 |
| *** slaweq_ has joined #openstack-infra | 16:00 | |
| *** slaweq has quit IRC | 16:00 | |
| *** armax has joined #openstack-infra | 16:00 | |
| dmsimard | corvus: it uses vhost_name, same thing as the vhost template -- http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/templates/zuul.vhost.erb && http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/manifests/web.pp#n93 | 16:01 |
| dmsimard | I figured that was the best parameter to use | 16:01 |
| corvus | dmsimard: gotcha | 16:01 |
| corvus | lgcm | 16:01 |
| *** rossella_s has quit IRC | 16:02 | |
| dmsimard | alternatively, I also need to set up *something* to let us know that the backups aren't being retrieved. I'll eventually have a chance to get to it. | 16:02 |
| *** agopi|out has joined #openstack-infra | 16:03 | |
| *** pcichy has quit IRC | 16:05 | |
| *** links has quit IRC | 16:05 | |
| AJaeger | config-core, could I get one review for https://review.openstack.org/536100 , please? This removes now obsolete legacy jobs - my quest to get rid of run-tox.sh continues... | 16:05 |
| *** gouthamr has joined #openstack-infra | 16:05 | |
| *** myoung|pto has quit IRC | 16:09 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Default max pool resources to math.inf https://review.openstack.org/537776 | 16:09 |
| *** e0ne_ has quit IRC | 16:09 | |
| *** slaweq_ has quit IRC | 16:10 | |
| *** myoung has joined #openstack-infra | 16:13 | |
| *** chenying has quit IRC | 16:13 | |
| *** chenying has joined #openstack-infra | 16:13 | |
| *** dougwig has joined #openstack-infra | 16:14 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Do not attempt to handle requests when disabled https://review.openstack.org/537954 | 16:17 |
| *** kjackal has quit IRC | 16:18 | |
| *** jamesmcarthur has joined #openstack-infra | 16:19 | |
| *** caphrim007 has quit IRC | 16:21 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Update docs to reflect newer Python requirement. https://review.openstack.org/538296 | 16:22 |
| *** jamesmcarthur has quit IRC | 16:23 | |
| *** melwitt is now known as jgwentworth | 16:23 | |
| *** matbu has quit IRC | 16:28 | |
| *** gouthamr has quit IRC | 16:29 | |
| pabelanger | infra-root: logs.o.o looks to have performance degradation | 16:29 |
| pabelanger | doesn't seem to be responding to an SSH | 16:29 |
| corvus | i agree | 16:29 |
| corvus | pabelanger: it's up, i have an open connection, but it's very slow | 16:29 |
| pabelanger | okay | 16:30 |
| corvus | optimistically, that could mean fsck has completed a phase and is swapping a lot. | 16:30 |
| dmsimard | I have a shell on it, taking a look | 16:30 |
| *** gyee has joined #openstack-infra | 16:31 | |
| dmsimard | yeah it's indeed not very fast | 16:31 |
| corvus | my shell is running "screen -x" which is still in the process of reconnecting, so i have no further info | 16:31 |
| corvus | snmp results on cacti show intermittent connectivity for the past hour | 16:32 |
| dmsimard | I actually exited the screen but the fsck was still running.. I didn't notice what % or phase it was at .. last official number I have is around 86% phase 2 (3 hours and a half ago) | 16:32 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul system-required template for new project Tatu. https://review.openstack.org/511339 | 16:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove heat-translator legacy job https://review.openstack.org/538300 | 16:33 |
| dmsimard | going to take a look at the console if there's anything going on | 16:33 |
| dmsimard | Have a console, nothing specific. corvus where was that password file ? | 16:34 |
| *** andreas_s has quit IRC | 16:34 | |
| *** andreas_s has joined #openstack-infra | 16:35 | |
| *** andreas_s has quit IRC | 16:35 | |
| corvus | dmsimard: join my screen session on puppetmaster | 16:35 |
| *** andreas_s has joined #openstack-infra | 16:35 | |
| *** matbu has joined #openstack-infra | 16:36 | |
| *** gouthamr has joined #openstack-infra | 16:36 | |
| AJaeger | mordred: looking at http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/pre.yaml#n65 - can't we just change run-tox once for good in project-config instead of replacing the lines every time? | 16:37 |
| dmsimard | logs.o.o just started responding | 16:38 |
| corvus | screen resumed -- 88.1% of phase 2 | 16:38 |
| dmsimard | swap amount hasn't changed so much but activity might have been high for a while | 16:39 |
| *** andreas_s has quit IRC | 16:40 | |
| mordred | AJaeger: yes. I believe that is there from when we were running v2 and v3 in parallel | 16:41 |
| *** matbu has quit IRC | 16:41 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul templates for new project python-tatuclient. https://review.openstack.org/538304 | 16:41 |
| dmsimard | corvus: neat. | 16:42 |
| AJaeger | mordred: ah, that's what I suspected... | 16:42 |
| dmsimard | load on logs.o.o is back to reasonable numbers | 16:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: remove service-type-authority tox job https://review.openstack.org/538306 | 16:43 |
| *** wolverineav has quit IRC | 16:44 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove obsolete legacy jobs https://review.openstack.org/536100 | 16:44 |
| *** armax has quit IRC | 16:44 | |
| *** wolverineav has joined #openstack-infra | 16:44 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul templates for new project tatu-dashboard. https://review.openstack.org/538307 | 16:46 |
| *** sjmc7 has left #openstack-infra | 16:46 | |
| *** armax has joined #openstack-infra | 16:46 | |
| *** wolverineav has quit IRC | 16:48 | |
| *** agopi|out is now known as agopi | 16:49 | |
| openstackgerrit | Merged openstack-infra/system-config master: Set the vhost_name for zuul01 to zuul.openstack.org https://review.openstack.org/538136 | 16:50 |
| *** gouthamr has quit IRC | 16:51 | |
| *** e0ne has joined #openstack-infra | 16:52 | |
| *** e0ne has quit IRC | 16:52 | |
| *** armaan_ has quit IRC | 16:54 | |
| *** armaan has joined #openstack-infra | 16:54 | |
| *** gouthamr has joined #openstack-infra | 16:55 | |
| *** wolverineav has joined #openstack-infra | 16:55 | |
| *** hjensas has quit IRC | 16:58 | |
| *** e0ne has joined #openstack-infra | 16:59 | |
| *** wolverineav has quit IRC | 17:00 | |
| *** chem has quit IRC | 17:02 | |
| *** kukacz has quit IRC | 17:02 | |
| *** chem has joined #openstack-infra | 17:02 | |
| *** chenying_ has joined #openstack-infra | 17:03 | |
| *** dave-mccowan has joined #openstack-infra | 17:05 | |
| *** chem has quit IRC | 17:05 | |
| *** chenying has quit IRC | 17:06 | |
| mordred | AJaeger: sorry, typo in https://review.openstack.org/#/c/538305 | 17:06 |
| *** chem has joined #openstack-infra | 17:06 | |
| pabelanger | dmsimard: look at logs for a313a980d05f4526aba5128189f45cd7, it was -2 on project-config/playbooks/publish/pypi.yaml. I wonder if it just failed to upload to pypi or already had a release | 17:06 |
| pabelanger | dmsimard: then logs.o.o failed, and due to load, to upload failed logs | 17:06 |
| dmsimard | pabelanger: yeah, there was a zuul_stream callback failure also | 17:06 |
| dmsimard | pabelanger: but we have no data on what failed | 17:07 |
| *** jamesmcarthur has joined #openstack-infra | 17:07 | |
| pabelanger | dmsimard: yah, we should fix that callback error | 17:07 |
| *** chem` has joined #openstack-infra | 17:09 | |
| dmsimard | pointed it out to mordred earlier -- it's hard to tell what the error actually is without keep and additional verbosity | 17:09 |
| *** harlowja has joined #openstack-infra | 17:10 | |
| dmsimard | that's why I said autokeep (and autohold triggering autokeep) would be useful to troubleshoot this kind of issue https://storyboard.openstack.org/#!/story/2001470 | 17:12 |
| *** yamahata has quit IRC | 17:12 | |
| *** chem has quit IRC | 17:12 | |
| dmsimard | logs.o.o is not very responsive again.. | 17:13 |
| dmsimard | looking | 17:13 |
| *** wolverineav has joined #openstack-infra | 17:15 | |
| *** fried_rice is now known as fried_rolls | 17:15 | |
| *** jbadiapa has joined #openstack-infra | 17:15 | |
| corvus | fyi, i'm responding to balazs gibizer's email on openstack-dev. it's going to take me a bit to research some things. | 17:18 |
| *** wolverineav has quit IRC | 17:18 | |
| *** wolverineav has joined #openstack-infra | 17:18 | |
| *** tellesnobrega has joined #openstack-infra | 17:20 | |
| dmsimard | I have no visibility on what is going on with logs.o.o, I was able to interrupt the loop that deletes old logs but commands are otherwise unresponsive. I'm not able to get a password prompt at the console either. | 17:21 |
| dmsimard | I'm going to give it a while longer | 17:21 |
| corvus | my screen is still attached; last output was 88.3% | 17:22 |
| dmsimard | corvus: in that screen, there is a window with a watch on a du -sh and the last printed timestamp is from ~20 minutes ago | 17:22 |
| *** tosky has joined #openstack-infra | 17:23 | |
| *** e0ne has quit IRC | 17:25 | |
| dmsimard | it looks like there is a spike in traffic right before it went unresponsive http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=311&rra_id=0&graph_start=1516901062&graph_end=1516987462 | 17:25 |
| *** salv-orlando has joined #openstack-infra | 17:25 | |
| dmsimard | outbound traffic too | 17:25 |
| *** e0ne has joined #openstack-infra | 17:26 | |
| corvus | maybo osla actually managed to process and serve a logfile | 17:26 |
| *** jamesmcarthur has quit IRC | 17:27 | |
| *** trown|rover is now known as trown|lunch | 17:29 | |
| *** jpena is now known as jpena|off | 17:30 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul templates for new project tatu-dashboard. https://review.openstack.org/538307 | 17:31 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul system-required template for new project Tatu. https://review.openstack.org/511339 | 17:32 |
| *** david-lyle has quit IRC | 17:32 | |
| dmsimard | It's weird because at least as far as http is concerned, logs.o.o is responding well.. I'm able to browse http://logs.openstack.org/15/536215/17/check/openstack-helm-multinode-ubuntu/cbaa1e4/ fairly quickly | 17:32 |
| *** caphrim007 has joined #openstack-infra | 17:32 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Zuul templates for new project python-tatuclient. https://review.openstack.org/538304 | 17:33 |
| *** jpich has quit IRC | 17:34 | |
| *** caphrim007 has quit IRC | 17:35 | |
| *** caphrim007 has joined #openstack-infra | 17:35 | |
| *** tesseract has quit IRC | 17:35 | |
| dmsimard | hmm, doesn't look like rax exposes the console log, "console log show" fails with a http 400 | 17:36 |
| *** agopi is now known as agopi|afk | 17:39 | |
| *** rkukura has joined #openstack-infra | 17:41 | |
| *** e0ne has quit IRC | 17:42 | |
| *** cshastri has quit IRC | 17:42 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: New project for Tatu (SSH as a Service) CLI and Python Client. https://review.openstack.org/537802 | 17:43 |
| dmsimard | infra-root: logs.o.o seems to have gotten entirely unresponsive.. I'm unable to use the screen, I'm seeing post_failures in jobs, we're able to browse logs over http surprisingly well but that's it. Last we know the fsck was at 88.3% in phase 2. Console over VNC doesn't give a password prompt. Do we set ourselves a "deadline" after which we would need to consider rebooting ? | 17:43 |
| *** e0ne has joined #openstack-infra | 17:44 | |
| *** slaweq has joined #openstack-infra | 17:46 | |
| pabelanger | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=137&rra_id=all | 17:46 |
| pabelanger | load is way up now | 17:46 |
| corvus | dmsimard: post_failures should only be for jobs which already failed, right? | 17:46 |
| dmsimard | corvus: I'm not sure I understand the question | 17:46 |
| corvus | dmsimard: because we're only uploading logs for failed jobs | 17:47 |
| *** gouthamr has quit IRC | 17:47 | |
| dmsimard | corvus: oh, that's a good point | 17:47 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Add new project for Tatu (SSH as a Service) Horizon Plugin. https://review.openstack.org/537653 | 17:47 |
| pabelanger | corvus: ya, I've only seen POST_FAILURES for jobs that have failed this morning | 17:47 |
| corvus | so isn't strictly speaking a significant further reduction in service | 17:47 |
| corvus | at least, it's not making the queue length worse | 17:48 |
| dmsimard | right cause the job was going to return FAILURE instead | 17:48 |
| pabelanger | tarballs.o.o is still responding to HTTP | 17:48 |
| dmsimard | pabelanger: the fact that there are new lines on the cacti graphs at all is encouraging, there was a wide gap | 17:48 |
| *** gouthamr has joined #openstack-infra | 17:49 | |
| *** slaweq_ has joined #openstack-infra | 17:49 | |
| dmsimard | pabelanger: like a 30 minute long gap | 17:49 |
| corvus | ah yep | 17:49 |
| *** slaweq has quit IRC | 17:50 | |
| *** b_bezak_ has joined #openstack-infra | 17:50 | |
| *** e0ne has quit IRC | 17:51 | |
| *** b_bezak has quit IRC | 17:51 | |
| *** esberglu has quit IRC | 17:53 | |
| *** kjackal has joined #openstack-infra | 17:53 | |
| dmsimard | The swap graphs for the period we just got back don't show an increase in swap usage or activity -- looks like around 60% i/o wait on the CPU graphs. | 17:54 |
| *** b_bezak_ has quit IRC | 17:55 | |
| *** jamesmcarthur has joined #openstack-infra | 17:55 | |
| *** jkilpatr has quit IRC | 17:56 | |
| *** yamahata has joined #openstack-infra | 17:56 | |
| dmsimard | Anyone know the partition scheme by heart? I put the scratch files on /opt which was it's own partition iirc.. both logs.o.o and tarballs.o.o seem to be working well over http. Could something be hammering the / partition or volume causing the OS to become unstable ? | 17:57 |
| *** florianf has quit IRC | 17:57 | |
| *** rkukura has quit IRC | 17:57 | |
| *** erlon_ has quit IRC | 17:58 | |
| *** yamamoto has quit IRC | 17:58 | |
| *** jamesmcarthur has quit IRC | 17:59 | |
| *** gouthamr_ has joined #openstack-infra | 17:59 | |
| *** yolanda has quit IRC | 18:01 | |
| *** links has joined #openstack-infra | 18:02 | |
| fungi | dmsimard: i'm pretty sure that /opt and swap both come from the same (ephemeral) drive? still travelling, sorry i can't be more help | 18:02 |
| *** gouthamr has quit IRC | 18:02 | |
| *** rlandy is now known as rlandy|brb | 18:02 | |
| dmsimard | fungi: it's alright, thanks | 18:03 |
| *** hongbin has joined #openstack-infra | 18:04 | |
| fungi | anyway, if fsck scratch activity monopolizes /opt then it will block swapping (or vice versa) in that scenario | 18:04 |
| dmsimard | that would be the case if both are on the same device, yes | 18:06 |
| dmsimard | need to step away for lunch before I digest myself, brb | 18:07 |
| corvus | fried_rolls: i'm waiting on a recheck to respond to your ml post, btw. the logs timed out from the last one and were deleted. of course, they may simply not upload this time. :( | 18:08 |
| *** agopi|afk is now known as agopi | 18:08 | |
| *** mriedem has quit IRC | 18:13 | |
| *** jamesmcarthur has joined #openstack-infra | 18:15 | |
| *** erlon_ has joined #openstack-infra | 18:18 | |
| *** yamamoto has joined #openstack-infra | 18:19 | |
| *** jamesmcarthur has quit IRC | 18:19 | |
| *** pramodrj07 has joined #openstack-infra | 18:24 | |
| *** pramodrj07 has quit IRC | 18:24 | |
| *** pramodrj07 has joined #openstack-infra | 18:25 | |
| *** caphrim007_ has joined #openstack-infra | 18:27 | |
| *** rlandy|brb is now known as rlandy | 18:28 | |
| *** caphrim007_ has quit IRC | 18:28 | |
| *** caphrim00_ has joined #openstack-infra | 18:28 | |
| *** electrofelix has quit IRC | 18:30 | |
| *** caphrim007 has quit IRC | 18:30 | |
| *** dtroyer has quit IRC | 18:31 | |
| *** links has quit IRC | 18:32 | |
| *** dtroyer has joined #openstack-infra | 18:32 | |
| *** trown|lunch is now known as trown|ruck | 18:36 | |
| *** trown|ruck is now known as trown|rover | 18:36 | |
| *** armax_ has joined #openstack-infra | 18:37 | |
| mrhillsman | i think logs for governance repos are broken | 18:39 |
| *** kjackal has quit IRC | 18:39 | |
| mrhillsman | example - http://logs.openstack.org/27/532627 | 18:40 |
| mrhillsman | http://logs.openstack.org/11/538311 | 18:40 |
| *** armax has quit IRC | 18:40 | |
| *** armax_ is now known as armax | 18:40 | |
| mrhillsman | or is this something folks running the repos need to fix? | 18:41 |
| mrhillsman | this is the full link - http://logs.openstack.org/11/538311/2/check/build-openstack-sphinx-docs/1699d9b/html/ | 18:41 |
| *** weshay is now known as weshay|ruck|brb | 18:43 | |
| *** xarses_ has joined #openstack-infra | 18:47 | |
| *** esberglu has joined #openstack-infra | 18:48 | |
| *** andreww has joined #openstack-infra | 18:49 | |
| *** lucasagomes is now known as lucas-afk | 18:49 | |
| *** yamamoto has quit IRC | 18:52 | |
| *** xarses_ has quit IRC | 18:52 | |
| *** tpsilva has joined #openstack-infra | 18:53 | |
| *** hjensas has joined #openstack-infra | 18:57 | |
| *** hjensas has joined #openstack-infra | 18:57 | |
| *** ralonsoh has quit IRC | 18:58 | |
| *** erlon_ has quit IRC | 19:03 | |
| *** b_bezak has joined #openstack-infra | 19:04 | |
| *** sambetts is now known as sambetts|afk | 19:04 | |
| *** mriedem has joined #openstack-infra | 19:05 | |
| *** slaweq_ has quit IRC | 19:05 | |
| *** caphrim00_ has quit IRC | 19:06 | |
| *** caphrim007 has joined #openstack-infra | 19:06 | |
| *** b_bezak has joined #openstack-infra | 19:06 | |
| Roamer` | mrhillsman, if this is a log for a successful build that happened within the last 24 hours, it was probably not uploaded because of yesterday's problems with the log server | 19:07 |
| *** yamamoto has joined #openstack-infra | 19:07 | |
| *** david-lyle has joined #openstack-infra | 19:08 | |
| *** jtomasek has quit IRC | 19:09 | |
| *** hjensas has quit IRC | 19:10 | |
| *** comstud has quit IRC | 19:10 | |
| *** yamamoto has quit IRC | 19:12 | |
| *** harlowja has quit IRC | 19:14 | |
| mrhillsman | i just pushed it not long ago | 19:16 |
| mrhillsman | https://review.openstack.org/#/c/538311/ | 19:16 |
| pabelanger | mrhillsman: yah, we've disabled uploads of successful jobs currently. Until we repair logs.o.o, failures will only be logged. | 19:17 |
| mrhillsman | ah ok | 19:18 |
| pabelanger | hopefully later today we can revert that change | 19:18 |
| mrhillsman | thx | 19:18 |
| *** makowals has joined #openstack-infra | 19:19 | |
| *** dprince has quit IRC | 19:21 | |
| *** links has joined #openstack-infra | 19:21 | |
| *** b_bezak has quit IRC | 19:25 | |
| *** jkilpatr has joined #openstack-infra | 19:26 | |
| *** weshay|ruck|brb is now known as weshay|ruck | 19:27 | |
| *** b_bezak has joined #openstack-infra | 19:30 | |
| *** b_bezak has quit IRC | 19:30 | |
| *** b_bezak has joined #openstack-infra | 19:31 | |
| *** pramodrj07 has quit IRC | 19:32 | |
| *** wolverineav has quit IRC | 19:35 | |
| *** b_bezak has quit IRC | 19:35 | |
| *** wolverineav has joined #openstack-infra | 19:35 | |
| *** wolverineav has quit IRC | 19:40 | |
| *** hjensas has joined #openstack-infra | 19:41 | |
| *** gouthamr_ has quit IRC | 19:44 | |
| *** salv-orlando has quit IRC | 19:45 | |
| *** salv-orlando has joined #openstack-infra | 19:45 | |
| *** harlowja has joined #openstack-infra | 19:46 | |
| *** pramodrj07 has joined #openstack-infra | 19:50 | |
| *** pcichy has joined #openstack-infra | 19:50 | |
| *** salv-orlando has quit IRC | 19:50 | |
| *** Pramod has joined #openstack-infra | 19:51 | |
| *** pramodrj07 has quit IRC | 19:55 | |
| *** dprince has joined #openstack-infra | 19:56 | |
| *** harlowja_ has joined #openstack-infra | 19:58 | |
| *** fried_rolls is now known as fried_rice | 19:58 | |
| *** matbu has joined #openstack-infra | 19:59 | |
| *** ihrachys has quit IRC | 19:59 | |
| fried_rice | corvus Roger that, thank you. | 19:59 |
| *** gouthamr has joined #openstack-infra | 20:00 | |
| *** harlowja has quit IRC | 20:00 | |
| dhellmann | dmsimard : how are things looking? | 20:01 |
| *** makowals has quit IRC | 20:02 | |
| *** Pramod has quit IRC | 20:02 | |
| *** pramodrj07 has joined #openstack-infra | 20:02 | |
| *** slaweq has joined #openstack-infra | 20:08 | |
| pabelanger | dhellmann: We're just talking in #openstack-infra-incident we're working on cloning volumes, hopefully that works in parallel (maybe another hour) then we can reformat and bring logs back online | 20:08 |
| pabelanger | then on different host fsck old filesystem | 20:08 |
| pabelanger | and rsync back | 20:08 |
| dhellmann | pabelanger : thanks, I'll follow the conversation over there | 20:09 |
| *** ldnunes has quit IRC | 20:15 | |
| *** yamamoto has joined #openstack-infra | 20:15 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoint add speaker https://review.openstack.org/538333 | 20:15 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added new endpoint add speaker https://review.openstack.org/538333 | 20:17 |
| *** agopi_ has joined #openstack-infra | 20:22 | |
| *** agopi has quit IRC | 20:25 | |
| *** camunoz has quit IRC | 20:28 | |
| *** yamamoto has quit IRC | 20:32 | |
| *** rfolco|ruck is now known as rfolco|off | 20:32 | |
| *** pcichy has quit IRC | 20:35 | |
| dmsimard | infra-root: rebooting logs.openstack.org now | 20:36 |
| *** makowals has joined #openstack-infra | 20:39 | |
| dmsimard | dhellmann: hi -- I'd like to try and rerun the blazarclient failure once the log server is back online.. but the script to generate the command that we need to run isn't going to help us because there are no logs for it so it might be a bit tricky to put the right things in the right places | 20:42 |
| *** e0ne has joined #openstack-infra | 20:43 | |
| *** gouthamr has quit IRC | 20:43 | |
| *** links has quit IRC | 20:43 | |
| dhellmann | dmsimard : how about if we test with a different tag then, using the release-test repo? | 20:44 |
| dhellmann | if that works, we can revert and reapply the blazarclient patch | 20:44 |
| dhellmann | oh, no, we can't | 20:44 |
| dhellmann | the tag was applied | 20:44 |
| dhellmann | hrm | 20:44 |
| corvus | dmsimard: i can walk you through making the command | 20:45 |
| dmsimard | corvus: I was going to pattern off of what the script wants/needs and the other commands it generated but if there's a better way.. | 20:46 |
| *** salv-orlando has joined #openstack-infra | 20:46 | |
| *** dprince has quit IRC | 20:47 | |
| corvus | dmsimard: well, it's less of a process and more of just knowing which arguments are needed. i can review what you come up with if you want. | 20:47 |
| dmsimard | corvus: sure, reviewing sounds great. | 20:48 |
| *** salv-orlando has quit IRC | 20:50 | |
| *** makowals has quit IRC | 20:50 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoints update and delete speaker https://review.openstack.org/538334 | 20:53 |
| *** bobh has quit IRC | 20:54 | |
| *** bobh has joined #openstack-infra | 20:55 | |
| *** rosmaita has quit IRC | 20:57 | |
| *** jkilpatr has quit IRC | 20:57 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added new endpoints update and delete speaker https://review.openstack.org/538334 | 21:00 |
| *** bobh has quit IRC | 21:00 | |
| dmsimard | corvus: so, according to the zuul build that failed (can't deeplink) the change that failed is https://git.openstack.org/cgit/openstack/python-blazarclient/commit/?id=1d46185bf1e0c18f69038adedd37cf6f6eaf06ab | 21:01 |
| dmsimard | based on that, I've drafted zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release --project=openstack/python-blazarclient --ref=refs/tags/1.0.0 --newrev=037f74de1f48b90dc25c5812b16543a6fccedfc1 | 21:01 |
| corvus | dmsimard: how did you find that change? | 21:02 |
| dmsimard | corvus: zuul dashboard, searching for the blazarclient job that failed https://i.imgur.com/fS7cbbM.png | 21:03 |
| dmsimard | ah, yeah I see 1d46185bf1e0c18f69038adedd37cf6f6eaf06ab matched for 1.0.0 | 21:04 |
| dmsimard | git show-ref has it 1d46185bf1e0c18f69038adedd37cf6f6eaf06ab refs/tags/1.0.0 | 21:04 |
| corvus | dmsimard: oh. turns out you can deeplink, you just have to do it manually -- "?project=...". tristanC is going to look into updating the url later. | 21:04 |
| *** agopi__ has joined #openstack-infra | 21:04 | |
| dmsimard | I guess that's also the commit id in the cgit url | 21:04 |
| dhellmann | that's the hash for the signed tag rather than the commit being tagged | 21:05 |
| dhellmann | and that's the hash you want, iiuc | 21:05 |
| dmsimard | alright so.... this sounds good to me: zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release --project=openstack/python-blazarclient --ref=refs/tags/1.0.0 --newrev=1d46185bf1e0c18f69038adedd37cf6f6eaf06ab | 21:06 |
| dhellmann | I believe that to be correct | 21:07 |
| *** agopi_ has quit IRC | 21:07 | |
| corvus | dhellmann, dmsimard: yep, the revised command looks correct. | 21:07 |
| dmsimard | ok, firing it up. | 21:07 |
| dmsimard | dhellmann: it's queued and I see it on zuul.o.o | 21:08 |
| dhellmann | yep, I see it | 21:08 |
| corvus | (you're telling zuul that "refs/tags/1.0.0" changed to the value "1d46185bf1e0c18f69038adedd37cf6f6eaf06ab" -- you always give the value of the ref itself) | 21:08 |
| dmsimard | dhellmann: was that the only one requiring attention ? | 21:08 |
| *** agopi__ is now known as agopi | 21:08 | |
| dhellmann | dmsimard : yes, the others failed at points where I could run commands by hand to recover (announcements and constraint updates) | 21:09 |
| dmsimard | okay | 21:10 |
| dhellmann | dmsimard, corvus : thanks for your help with these | 21:10 |
| dmsimard | I'll try and keep an eye on the job consoles if I can get hold of one | 21:10 |
| dhellmann | ditto | 21:10 |
| pabelanger | I'll be stepping away for a few hours now, I'll check back in on logs.o.o later this evening. | 21:11 |
| dhellmann | dmsimard : yesterday smcginnis gave me a link to a pastebin with a script that fungi uses to build these commands. do you have that? | 21:11 |
| dhellmann | I could probably modify it so I can run it locally and get the command without needing the logs | 21:11 |
| *** agopi is now known as agopi|out | 21:11 | |
| *** ihrachys has joined #openstack-infra | 21:12 | |
| dmsimard | dhellmann: yup, I found it and I formalized the need for something along these lines in a story here: https://storyboard.openstack.org/#!/story/2001500 | 21:12 |
| *** olaph1 has joined #openstack-infra | 21:12 | |
| dhellmann | cool, thanks | 21:12 |
| dhellmann | do we have to run the same command separately for the tag queue to trigger the release note build? | 21:13 |
| dhellmann | sorry, tag pipeline | 21:13 |
| corvus | dhellmann: yes | 21:13 |
| dhellmann | ok, so the same arguments but --pipeline=tag | 21:13 |
| corvus | yep | 21:13 |
| dhellmann | dmsimard , can you do that, too, please? | 21:13 |
| dmsimard | I can do that | 21:14 |
| dhellmann | ty | 21:14 |
| dmsimard | dhellmann: the python release job just started: http://zuul.openstack.org/stream.html?uuid=13aa058d83c34ab38bc2cf51c02bdd81&logfile=console.log | 21:14 |
| corvus | (it bypasses the pipeline requirements check, so you have to specify which pipelines it should end up in) | 21:14 |
| *** olaph has quit IRC | 21:14 | |
| corvus | (nb that means if you type it wrong, you could enqueue an unmerged change in post or something, so... danger zone) | 21:15 |
| dhellmann | yeah, I'm going to try to write a shell script that generates the commands given a repo name and a tag | 21:16 |
| corvus | dhellmann: sounds good | 21:16 |
| dmsimard | console for that job isn't working or is taking a long time to start :/ | 21:16 |
| *** agopi|out has quit IRC | 21:16 | |
| corvus | dmsimard: i believe due to the incorrect hostname problem, we only have log streaming functioning on 2 zuul executors. | 21:16 |
| dhellmann | I've noticed that sometimes these do take a while to start producing output, but I'm also not especially patient :-) | 21:17 |
| dhellmann | ah | 21:17 |
| dmsimard | corvus: incorrect hostname ? you mean versus zuulv3.o.o ? | 21:18 |
| dmsimard | I was able to identify which ze that build was running on and copied the build dir for later troubleshooting | 21:19 |
| corvus | dmsimard: no, the executors have the wrong hostname configured thanks to cloud-init. | 21:19 |
| dmsimard | corvus: oh, that :( | 21:19 |
| dmsimard | mordred: for the record I've got /home/dmsimard/13aa058d83c34ab38bc2cf51c02bdd81 on ze09 which is a requeue of the job that failed earlier today with the zuul_stream error | 21:21 |
| *** ganso has quit IRC | 21:21 | |
| *** slaweq has quit IRC | 21:22 | |
| *** slaweq has joined #openstack-infra | 21:22 | |
| *** Goneri has quit IRC | 21:23 | |
| *** bobh has joined #openstack-infra | 21:25 | |
| mordred | dmsimard: awesome. thanks! | 21:26 |
| dmsimard | dhellmann: I think it failed again :( | 21:26 |
| dmsimard | looking | 21:26 |
| *** slaweq has quit IRC | 21:27 | |
| dmsimard | we have logs this time \o/ http://logs.openstack.org/1d/1d46185bf1e0c18f69038adedd37cf6f6eaf06ab/release/release-openstack-python/13aa058/ | 21:27 |
| dmsimard | failed here http://logs.openstack.org/1d/1d46185bf1e0c18f69038adedd37cf6f6eaf06ab/release/release-openstack-python/13aa058/ara/result/26cee65c-b3cd-4267-9a03-1fe45be043d4/ | 21:27 |
| dhellmann | corvus , dmsimard : https://review.openstack.org/538339 is the script I came up with | 21:27 |
| dmsimard | dhellmann: it's the upload to pypi that is failing | 21:28 |
| dmsimard | Error: 'Development Status :: 3 - Alpha,' is not a valid choice for this field see https://packaging.python.org/specifications/core-metadata for url: https://upload.pypi.org/legacy/ | 21:28 |
| dhellmann | oh, joy | 21:28 |
| corvus | wow those are validated? | 21:28 |
| dmsimard | corvus: yeah they're not freeform, you can search with those tags on pypi | 21:29 |
| dhellmann | maybe that's new for warehouse | 21:29 |
| dhellmann | according to https://pypi.python.org/pypi?%3Aaction=list_classifiers that is a valid trove classifier | 21:29 |
| dmsimard | https://github.com/openstack/python-blazarclient/blob/master/setup.cfg#L11 hrm.... I think it would be the comma ? | 21:29 |
| dhellmann | oh, the comma | 21:29 |
| dhellmann | yeah | 21:29 |
| dmsimard | there's a comma on the line after as well | 21:29 |
| dmsimard | apparently it's had a comma for 4 years now | 21:30 |
| dhellmann | yay for api changes | 21:30 |
| dmsimard | we can try removing the commas and look at what happens, I guess we'll need a 1.0.1 or something like that. | 21:30 |
| dhellmann | yeah | 21:30 |
| dmsimard | I'll send a patch I already have the repo checked out | 21:31 |
| dhellmann | ok | 21:31 |
| *** Pramod has joined #openstack-infra | 21:31 | |
| dhellmann | I can prepare the release patch once you have that posted | 21:31 |
| dmsimard | https://review.openstack.org/538340 Remove commas in setup.cfg package classifiers | 21:32 |
| dhellmann | ah, crud | 21:33 |
| dhellmann | that repo is already branched | 21:33 |
| dmsimard | so it'll need a backport as well | 21:33 |
| dhellmann | yeah | 21:33 |
| dmsimard | ¯\_(ツ)_/¯ | 21:34 |
| dmsimard | I'll send it right now | 21:34 |
| dhellmann | thanks | 21:34 |
| *** e0ne has quit IRC | 21:34 | |
| *** matbu has quit IRC | 21:34 | |
| dhellmann | I guess we'll have to just let the blazaar team know so they can fast-approve things | 21:34 |
| dhellmann | I'll compose that email | 21:34 |
| *** pramodrj07 has quit IRC | 21:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use zuul-sudo-grep https://review.openstack.org/538342 | 21:35 |
| dmsimard | dhellmann: link to changeid https://review.openstack.org/#/q/Ieba2a4bec5ffc8e618501b38f73628e0b59c12a5 | 21:36 |
| *** rosmaita has joined #openstack-infra | 21:36 | |
| *** dave-mccowan has quit IRC | 21:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove now obsolete jenkins-sudo-grep lines https://review.openstack.org/538344 | 21:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Create a parent job to publish to the zuul website https://review.openstack.org/538345 | 21:39 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use zuul-sudo-grep https://review.openstack.org/538342 | 21:39 |
| AJaeger | mordred: https://review.openstack.org/538344 and https://review.openstack.org/538342 do the cleanup of jenkins-sudo-grep | 21:40 |
| dmsimard | mordred: yeah it caused a zuul_stream error again. It's this task that appears to trigger it: http://logs.openstack.org/1d/1d46185bf1e0c18f69038adedd37cf6f6eaf06ab/release/release-openstack-python/13aa058/ara/file/61efbd0f-a0e5-4a14-b567-f182269ee84e/#line-23 | 21:40 |
| dmsimard | mordred: error is here: http://logs.openstack.org/1d/1d46185bf1e0c18f69038adedd37cf6f6eaf06ab/release/release-openstack-python/13aa058/ara/result/26cee65c-b3cd-4267-9a03-1fe45be043d4/ | 21:40 |
| *** jcoufal has quit IRC | 21:41 | |
| dhellmann | dmsimard : http://lists.openstack.org/pipermail/openstack-dev/2018-January/126636.html | 21:41 |
| dmsimard | dhellmann: ok, hopefully when 1.0.1 is tagged everything should just work | 21:42 |
| dhellmann | oh, hey, look, I have +2 on their stable branch | 21:42 |
| *** david-lyle has quit IRC | 21:42 | |
| *** dklyle has joined #openstack-infra | 21:43 | |
| AJaeger | dhellmann: you should as releasemanager | 21:43 |
| dmsimard | dhellmann: yeah I almost got caught off guard cause their gitreview change isn't merged yet :) | 21:43 |
| *** esberglu has quit IRC | 21:44 | |
| *** PramodJ has joined #openstack-infra | 21:44 | |
| dhellmann | those should all be going into the queue to merge soon | 21:44 |
| *** PramodJ has quit IRC | 21:44 | |
| *** slaweq has joined #openstack-infra | 21:44 | |
| *** pramodrj07 has joined #openstack-infra | 21:45 | |
| dhellmann | interesting that their python 35 job is nonvoting | 21:45 |
| *** Pramod has quit IRC | 21:45 | |
| *** salv-orlando has joined #openstack-infra | 21:46 | |
| *** bobh has quit IRC | 21:47 | |
| *** rlandy has quit IRC | 21:49 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use override-checkout to select jobs https://review.openstack.org/537655 | 21:49 |
| dhellmann | dmsimard , smcginnis : while we wait for the blazarclient patches shall we try another release? | 21:50 |
| smcginnis | If we think the external issues are cleared up, might as well. | 21:51 |
| *** salv-orlando has quit IRC | 21:51 | |
| dhellmann | https://review.openstack.org/#/c/538118/ looks like a nice simple candidate | 21:51 |
| dhellmann | and I verified that they have no commas in their classifiers :-) | 21:51 |
| smcginnis | +W | 21:51 |
| dmsimard | dhellmann: what do you mean try another release ? | 21:51 |
| dmsimard | dhellmann: oh, merge something else you mean | 21:52 |
| dhellmann | dmsimard : we've been holding a batch of requests until issues were fixed | 21:52 |
| dhellmann | yes | 21:52 |
| dmsimard | dhellmann: the logserver is healthy right now so if there's a failure we'll have logs | 21:52 |
| dmsimard | here's to hoping commas aren't popular right now | 21:52 |
| smcginnis | Are we safe to do those kolla ones yet? I think they were the only ones that needed that infra patch merged, but I've lost sight of that one. | 21:52 |
| dhellmann | I suppose that's as strong a vote of confidence as we can expect right now | 21:52 |
| dhellmann | I'm going to approve the barbicanclient release | 21:53 |
| dhellmann | oh, smcginnis already did | 21:53 |
| smcginnis | We shall know shortly if things are OK. | 21:53 |
| *** kjackal has joined #openstack-infra | 21:56 | |
| *** trown|rover is now known as trown|outtypewww | 22:00 | |
| *** esberglu has joined #openstack-infra | 22:03 | |
| smcginnis | Made it to release-post at least. | 22:04 |
| *** amoralej is now known as amoralej|off | 22:10 | |
| *** amoralej|off is now known as amoralej | 22:11 | |
| *** esberglu has quit IRC | 22:11 | |
| *** salv-orlando has joined #openstack-infra | 22:12 | |
| *** maharg101 has joined #openstack-infra | 22:13 | |
| *** amoralej is now known as amoralej|off | 22:14 | |
| *** panda is now known as panda|off | 22:15 | |
| *** kgiusti has left #openstack-infra | 22:18 | |
| *** kukacz has joined #openstack-infra | 22:19 | |
| *** StaceyF has joined #openstack-infra | 22:22 | |
| *** tpsilva has quit IRC | 22:23 | |
| *** snapiri- has quit IRC | 22:24 | |
| StaceyF | hey everyone, I am trying to align gerrit as close as possible to the openstack gerrit. Is there any guidance on how this is done? | 22:25 |
| smcginnis | tag-releases done, publish-static and release-openstack-python in progress. | 22:26 |
| *** andreww has quit IRC | 22:27 | |
| fungi | StaceyF: we try to keep this current, but no guarantees... https://docs.openstack.org/infra/system-config/gerrit.html | 22:28 |
| fungi | it's current to my knowledge, at least | 22:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't override allowed-projects https://review.openstack.org/538353 | 22:29 |
| smcginnis | release-post jobs done, now just the release ones. | 22:31 |
| StaceyF | great, thanks! | 22:38 |
| *** esberglu has joined #openstack-infra | 22:41 | |
| smcginnis | Always seemed to me jobs like announce-release would make great candidates for container run jobs instead of full on VMs. | 22:42 |
| dhellmann | are we doing container jobs? | 22:42 |
| smcginnis | No, but I think it would be interesting if we did. | 22:43 |
| smcginnis | Though we have plenty of "interesting" things to keep us busy as is. | 22:43 |
| dhellmann | I wonder how many other jobs like that could be put into containers, and how much benefit we would get from doing it | 22:43 |
| dhellmann | the announce job does run python setup.py so we can't reuse the same host for more than one job | 22:44 |
| smcginnis | Not sure if there are really a lot, but it seems like it would reduce the load overall and be a lot faster. | 22:44 |
| smcginnis | If it's in a container we should be able to do that. | 22:44 |
| dhellmann | sure | 22:44 |
| dhellmann | I meant now | 22:44 |
| smcginnis | Yay, release job complete. | 22:44 |
| dhellmann | woo! | 22:45 |
| smcginnis | Oh, yeah. Not with how things are now. | 22:45 |
| dhellmann | time to switch to weekend mode here | 22:45 |
| corvus | if you have ideas about how to actually accomplish that, i expect we'll be talking about it at the ptg :) | 22:45 |
| smcginnis | corvus: Oh cool. If I came up with some ideas, I will certainly share. | 22:45 |
| smcginnis | Have a good weekend dhellmann. | 22:45 |
| dhellmann | corvus : maybe instead of containers we should look at qinling functions | 22:45 |
| smcginnis | Haha, there you go. | 22:45 |
| corvus | good, we need a shared understanding of the problem space and brainstorming | 22:46 |
| smcginnis | Or qinling functions running in kata containers. | 22:46 |
| *** esberglu has quit IRC | 22:46 | |
| *** fultonj has quit IRC | 22:52 | |
| *** esberglu has joined #openstack-infra | 22:53 | |
| *** esberglu has quit IRC | 22:58 | |
| *** StaceyF has quit IRC | 23:06 | |
| *** edmondsw has quit IRC | 23:07 | |
| *** slaweq has quit IRC | 23:09 | |
| *** slaweq has joined #openstack-infra | 23:09 | |
| *** lbragstad is now known as elbragstad | 23:11 | |
| *** esberglu has joined #openstack-infra | 23:13 | |
| *** slaweq has quit IRC | 23:13 | |
| *** maharg101 has quit IRC | 23:15 | |
| *** esberglu has quit IRC | 23:17 | |
| *** ihrachys has quit IRC | 23:19 | |
| *** bobh has joined #openstack-infra | 23:28 | |
| *** jgwentworth is now known as melwitt | 23:28 | |
| *** olaph1 is now known as olaph | 23:31 | |
| *** jbadiapa has quit IRC | 23:42 | |
| *** esberglu has joined #openstack-infra | 23:43 | |
| *** esberglu has quit IRC | 23:48 | |
| *** jbadiapa has joined #openstack-infra | 23:50 | |
| *** hongbin has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
