| mordred | corvus: look at http://logs.openstack.org/17/536517/1/check/swift-tox-func-post-as-copy/a48ce51/job-output.txt.gz#_2018-01-22_18_57_22_638659 | 00:00 |
|---|---|---|
| johnsom | Yep, thus the 400 on the second try | 00:00 |
| pabelanger | so I don't think we want to delete what was uploaded on pypi | 00:00 |
| mordred | corvus: and the list of refs/zuul/ 'new branch' entries | 00:00 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Rotate cloud launcher log files https://review.openstack.org/537706 | 00:00 |
| corvus | mordred: yeah, we need to finish dropping the zuul refs | 00:00 |
| corvus | they should be mostly harmless noise though | 00:00 |
| johnsom | Yeah, actually, looking through the log, the first run pretty much did what it needed except for that ssh error with keys. Maybe we just need to run the other two jobs (announce-release and propose-update-constraints) that were skipped on both runs | 00:02 |
| pabelanger | not the first time I have seen remove-build-sshkey role fail. I wonder it we could some how maybe zuul know about optional playbooks, not fail a job. I mean, it is nice we delete SSH key on nodes, but don't think we should fail job if it doesn't do it | 00:03 |
| pabelanger | corvus: mordred: ^thoughts? | 00:03 |
| corvus | pabelanger: i think there's an ansible option for that | 00:03 |
| pabelanger | k, let me check | 00:04 |
| johnsom | It probably aborted the release pipeline part as well | 00:05 |
| pabelanger | johnsom: yes, most likely | 00:05 |
| pabelanger | we won't be able to enqueue for pre-release again, package is uploaded but we might be able to do release, would need to see what job does | 00:05 |
| johnsom | So the announce and upper-constraint just gets dropped? | 00:06 |
| *** chandankumar has joined #openstack-infra | 00:07 | |
| corvus | timburke, mordred: i'm still stumped | 00:07 |
| mordred | corvus, timburke: yup. me too | 00:07 |
| mordred | timburke: good job- it's a good puzzle | 00:07 |
| timburke | glad i could help :-) | 00:08 |
| pabelanger | johnsom: yes, we are not able to enqueue single jobs currently | 00:08 |
| *** chkumar246 has quit IRC | 00:09 | |
| *** chandankumar has quit IRC | 00:12 | |
| corvus | timburke, mordred: oh i think i have a hypothesis. the merge commit did not register as an update to zuul's configuration, so it ran that change (which depended on the merge commit which removed the job) under the current 'live' config, which still had the job. | 00:13 |
| timburke | i can try adding an extra line to .zuul.yaml or something | 00:14 |
| corvus | timburke, mordred: now that the merge commit has landed -- the live config should no longer have the job. but only if another config change has landed to swift, or if we've done a full reconfig. the latter has probably happened. | 00:14 |
| corvus | timburke, mordred: so why don't we see if a recheck dtrt now... and i'll double check whether we've done a full reconfig to help verify. | 00:14 |
| johnsom | pabelanger Well, looking at things, I'm not sure announce and propose-upper matter for this milestone release, so maybe just picking it up at release and down will be fine. | 00:14 |
| mordred | yah. so a recheck of the followup patch should work now | 00:14 |
| corvus | (and if we haven't done a full reconfig, i can force one) | 00:15 |
| mordred | corvus: if the recheck works, then that gives credence to the theory and that there might be bug in config change detection when the change in question is itself a merge commit | 00:15 |
| timburke | rechecked! we'll see what happens :) | 00:16 |
| corvus | mordred: yeah, we base config change detection on the files gerrit tells us about -- knowing what we know about reviewing merge commits in gerrit, i'm pretty suspicious :) | 00:16 |
| *** larainema has joined #openstack-infra | 00:16 | |
| mordred | corvus: ++ | 00:16 |
| corvus | the last full reconfig was 2018-01-23 16:13:52,434 | 00:16 |
| pabelanger | johnsom: yah, I think we have a few options, not that we can fix right this moment. But we could make python-openstack-release job a little smarter, in the case if when we enqueue a change again. And don't actually fail the job in base/post-ssh.yaml playbook, I am looking into that. But getting late. | 00:16 |
| timburke | thanks corvus, mordred! | 00:17 |
| timburke | want me to write up a bug or anything about it? | 00:17 |
| corvus | timburke: i'll write it up, thanks | 00:17 |
| *** edmondsw has joined #openstack-infra | 00:19 | |
| *** jcoufal has quit IRC | 00:20 | |
| johnsom | pabelanger I'm not familiar with these pipelines, are those three steps in the pre-release the only thing that runs here? Is release and post-release for other purposes? | 00:20 |
| johnsom | If those were the only three steps required, I think I am fine for this MS3 release. It would be bad for the *actual* release since this is the first *release* for this repo, but for the MS3 I think it's not a big deal | 00:21 |
| corvus | timburke, mordred: on the status page, i don't see the 'copy' job, so i think we have confirmation | 00:22 |
| pabelanger | johnsom: right, you can see http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml for flow of the pipelines and when they are used, based on type of tag in this case | 00:22 |
| corvus | timburke, mordred: bug: https://storyboard.openstack.org/#!/story/2001496 | 00:23 |
| *** edmondsw has quit IRC | 00:23 | |
| *** chandankumar has joined #openstack-infra | 00:26 | |
| johnsom | pabelanger Ok, I think we are fine here. The root cause needs to get fixed, but I think this MS3 release for octavia-dashboard is probably ok. | 00:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: ignore_errors for post-ssh.yaml https://review.openstack.org/537711 | 00:28 |
| pabelanger | corvus: mordred: johnsom: ^ I think that is our first step to ignoring errors for remove-build-sshkeys role, however I am unsure if that will actually work when we get UNREACHABLE from the host. | 00:29 |
| pabelanger | I can dig more into testing that in the morning if we first want to confirm | 00:29 |
| mordred | corvus: EXCELLENT | 00:31 |
| *** edmondsw has joined #openstack-infra | 00:31 | |
| *** bobh has joined #openstack-infra | 00:34 | |
| *** jbadiapa has quit IRC | 00:34 | |
| *** edmondsw has quit IRC | 00:36 | |
| *** edmondsw has joined #openstack-infra | 00:37 | |
| ianw | mordred: so ... is there any way to debug an error in /etc/ansible/hosts/openstack other than to run it manually and wait? | 00:38 |
| ianw | the cron job currently fails with | 00:39 |
| ianw | raise exceptions.DiscoveryFailure('Could not determine a suitable URL ' | 00:39 |
| ianw | keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not determine a suitable URL for the plugin | 00:39 |
| ianw | but i'm not sure what cloud is causing that | 00:39 |
| ianw | i'm thinking of adding in the ansible contrib/ an environment variable to dump debug info to a file, which we can set for the cron job | 00:40 |
| ianw | but that doesn't help immediately | 00:40 |
| corvus | ianw: any chance it's infracloud? | 00:40 |
| ianw | "the cron job" being the cloud launcher cron job i should say | 00:40 |
| ianw | corvus: we removed those from the config yesterday, so don't think so | 00:40 |
| *** edmondsw has quit IRC | 00:41 | |
| pabelanger | is infracloud still in cloud launcher? | 00:42 |
| pabelanger | I never thought to remove that too | 00:42 |
| ianw | no it uses all-clouds.yaml, which doens't have it | 00:42 |
| *** dave-mccowan has joined #openstack-infra | 00:42 | |
| ianw | i'm running it against a copy with the [jenkins|zuul] clouds removed to avoid listing all the testing vms ... still haven't hit it | 00:44 |
| *** edmondsw has joined #openstack-infra | 00:44 | |
| pabelanger | ianw: maybe because it is still listed in http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/clouds_layouts.yml#n202 ? | 00:44 |
| ianw | ? maybe ... the bt in the logs is from the inventory lister,which i didn't think knew about that at all | 00:45 |
| *** edmondsw has quit IRC | 00:45 | |
| *** edmondsw has joined #openstack-infra | 00:46 | |
| mordred | ianw: do you have a full traceback? | 00:49 |
| *** edmondsw has quit IRC | 00:50 | |
| mordred | (mostly I'd like to at the very least log an error with the cloud name when that happens) | 00:50 |
| ianw | mordred: http://paste.openstack.org/show/653015/ | 00:51 |
| mordred | ianw: thanks! | 00:51 |
| ianw | ok, just caught it live ... http://paste.openstack.org/show/653018/ | 00:52 |
| ianw | ohh, i guess the layout in the file doesn't represent how it walks through the dict at all | 00:53 |
| mordred | ianw: that seems like it's a rax:DFW error :( | 00:56 |
| ianw | hmm, not 100% sure? would that be the last thing *before* the error? | 00:57 |
| ianw | it does seem the loop is in shade | 00:57 |
| mordred | yah. that'll be the 'get all the servers' loop in the inventory code | 00:57 |
| ianw | for server in inventory.list_hosts(**list_args): | 00:57 |
| ianw | File "/usr/local/lib/python2.7/dist-packages/shade/inventory.py", line 68, in list_hosts | 00:57 |
| ianw | for server in cloud.list_servers(detailed=expand): | 00:57 |
| ianw | let me reduce the cloud to just rax dfw and run it again | 00:58 |
| *** bobh has quit IRC | 00:59 | |
| ianw | oh, you know what, that has a blank password | 01:00 |
| ianw | oh thank god, i almost posted it | 01:00 |
| mordred | ianw: :) | 01:02 |
| mordred | ianw: but yes - I agree with you - there are no usernames, passwords or project_ids for openstack-rax in all-clouds.yaml | 01:02 |
| mordred | ianw: actually - what even is openstack-rax? | 01:03 |
| *** gk__ has joined #openstack-infra | 01:03 | |
| gk__ | fungi: Hi fungi, are you there? | 01:03 |
| mordred | ianw: openstackci-rax and openstackjenkins-rax both seem fine | 01:03 |
| ianw | yeah, openstack-rax seems to have been there forever | 01:04 |
| ianw | have we dropped it from hiera? | 01:04 |
| mordred | ianw: I see no openstack_rax_username in hiera | 01:05 |
| *** zhurong has joined #openstack-infra | 01:05 | |
| ianw | mordred: and as far as "git log -p" is concerned, there never was? | 01:07 |
| mordred | ianw: and that entry in the puppet manifest has been there since 2015 | 01:08 |
| mordred | ianw: you have uncovered a very confusing riddle | 01:08 |
| *** olaph1 has joined #openstack-infra | 01:09 | |
| *** cuongnv has joined #openstack-infra | 01:10 | |
| *** olaph has quit IRC | 01:11 | |
| mordred | ianw: I mean - the simple solution is to just remove the entry from all-clouds ... but I'd kind of like to understand how this ever worked | 01:11 |
| ianw | right, has the client updated and been ignoring it or something? | 01:12 |
| mordred | ianw: OH ... | 01:14 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove openstack-rax cloud https://review.openstack.org/537713 | 01:14 |
| mordred | ianw: this is cloud-launcher, so something is setting OS_CLIENT_CONFIG_FILE to /etc/openstack/all-clouds.yaml right? | 01:14 |
| mordred | ianw: that file doesn't normally get used for inventory | 01:14 |
| mordred | so it doesn't have the fail_on_errors: False setting that the normal clouds.yaml does | 01:15 |
| ianw | yep, /opt/system-config/production/run_cloud_launcher.sh | 01:15 |
| mordred | ianw: my hunch is that we got 'lucky' and ran cloud launcher at a time when the inventory cache needed to be invalidated and redone | 01:15 |
| mordred | so rather than using the cached inventory from the normal clouds.yaml run - it trashed that and tried to build one from all-clouds ... which is basically never a thing we want | 01:16 |
| mordred | ianw: but thing is - we don't need the dynamic inventory for cloud-launcher since it runs its tasks on localhost | 01:16 |
| mordred | ianw: so we should update that script to pass a -i hosts and point it to a hosts file that only has localhost - so that the OS_CLIENT_CONFIG_FILE settings don't compete with the normal puppetmaster inventory | 01:17 |
| mordred | ianw: we should also make sure that the inventory runs while all-clouds was active haven't been cached - and rebuild the inventory cache with clouds.yaml | 01:17 |
| mordred | otherwise we're going to have puppetmaster trying to run ansible on nodepool nodes | 01:17 |
| ianw | it also takes more than an hour to list everything i think! | 01:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Pass -i /dev/null to cloud launcher https://review.openstack.org/537714 | 01:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Pass -i /dev/null to cloud launcher https://review.openstack.org/537714 | 01:21 |
| mordred | ianw: there you go ... well, it shouldn't take more than on hour to list everything if we're using the right clouds.yaml file ... and if it does I should spend some time debugging why | 01:21 |
| mordred | ianw: I just tested on puppetmaster and -i /dev/null avoids the system configured dynamic inventory - and ansible has localhost as a built-in host entry when there is no inventory | 01:22 |
| *** dave-mccowan has quit IRC | 01:22 | |
| ianw | looking at the logs, seems to have been happening for quite a while. i guess we just never added new clouds | 01:23 |
| mordred | actually- we really need to overhaul how all that inventory is happening anyway - I'll try to tee that up for tomorrow | 01:23 |
| ianw | ok, just running that manually now | 01:24 |
| *** kiennt26 has joined #openstack-infra | 01:26 | |
| ianw | os_client_config.exceptions.OpenStackConfigException: Cloud admin-infracloud-vanilla was not found. | 01:27 |
| ianw | ahh, so that does use the other flie | 01:27 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove infracloud-vanilla and chocolate cloud launcher config https://review.openstack.org/537715 | 01:29 |
| ianw | isn't automation great! :) | 01:30 |
| mordred | :) | 01:33 |
| * mordred has to afk ... | 01:33 | |
| ianw | mordred: thanks for the help. i think we'll have the keys rolled out to linaro now, which is where i started all those hours ago | 01:35 |
| *** dhill_ has quit IRC | 01:35 | |
| *** liujiong has joined #openstack-infra | 01:36 | |
| *** bobh has joined #openstack-infra | 01:37 | |
| *** xarses has joined #openstack-infra | 01:49 | |
| *** xarses has quit IRC | 01:50 | |
| *** daidv has joined #openstack-infra | 01:52 | |
| *** dave-mccowan has joined #openstack-infra | 01:53 | |
| mnaser | getting intermittent 500s from gerrit | 01:55 |
| *** xarses has joined #openstack-infra | 01:57 | |
| *** gk__ has quit IRC | 01:58 | |
| ianw | hmm, memory usage isn't too high | 02:03 |
| *** salv-orl_ has joined #openstack-infra | 02:04 | |
| *** salv-orlando has quit IRC | 02:07 | |
| *** gibi has quit IRC | 02:07 | |
| *** tinwood has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-infra | 02:11 | |
| *** zhurong has quit IRC | 02:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove the custom release permissions for storyboardclient https://review.openstack.org/537647 | 02:14 |
| *** gcb has joined #openstack-infra | 02:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: Make py35 jobs voting for refstack-client https://review.openstack.org/537639 | 02:16 |
| openstackgerrit | Merged openstack-infra/project-config master: Add release notes job to octavia-dashboard https://review.openstack.org/537618 | 02:16 |
| openstackgerrit | Merged openstack-infra/project-config master: retire rack and python-rackclient project https://review.openstack.org/536672 | 02:16 |
| *** ekcs has quit IRC | 02:19 | |
| *** stakeda has joined #openstack-infra | 02:21 | |
| *** yolanda has quit IRC | 02:22 | |
| *** s-shiono has joined #openstack-infra | 02:23 | |
| *** slaweq has joined #openstack-infra | 02:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Publish release notes for osc-placement https://review.openstack.org/537465 | 02:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Python tarball job from tripleo-ui https://review.openstack.org/536631 | 02:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Fetch javascript output on publish jobs too https://review.openstack.org/536945 | 02:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy mistralclient jobs https://review.openstack.org/535812 | 02:23 |
| *** hongbin has joined #openstack-infra | 02:27 | |
| *** slaweq has quit IRC | 02:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-to-pypi job to tripleo-ipsec https://review.openstack.org/536785 | 02:33 |
| *** dave-mccowan has quit IRC | 02:33 | |
| *** jamesmcarthur has joined #openstack-infra | 02:37 | |
| *** jamesmcarthur has quit IRC | 02:41 | |
| *** jamesmcarthur has joined #openstack-infra | 02:42 | |
| *** jamesmcarthur has quit IRC | 02:43 | |
| *** jamesmcarthur has joined #openstack-infra | 02:43 | |
| *** rkukura has quit IRC | 02:45 | |
| *** jamesmcarthur has quit IRC | 02:48 | |
| *** bobh has quit IRC | 02:48 | |
| *** ianychoi has quit IRC | 02:49 | |
| *** ianychoi has joined #openstack-infra | 02:49 | |
| *** ganso has quit IRC | 02:52 | |
| *** larainema has quit IRC | 02:52 | |
| *** bobh has joined #openstack-infra | 02:54 | |
| *** dave-mccowan has joined #openstack-infra | 02:55 | |
| *** rlandy|bbl is now known as rlandy | 02:58 | |
| *** zhurong has joined #openstack-infra | 03:08 | |
| *** gema has quit IRC | 03:09 | |
| *** gema has joined #openstack-infra | 03:09 | |
| *** gema has quit IRC | 03:09 | |
| *** gema has joined #openstack-infra | 03:09 | |
| *** dougwig has quit IRC | 03:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Shift javascript publish jobs to post https://review.openstack.org/536952 | 03:15 |
| AJaeger | config-core, could you review https://review.openstack.org/536097 and https://review.openstack.org/536092 , please? the jobs are in-repo now and can be removed from project-config. | 03:20 |
| *** harlowja has quit IRC | 03:21 | |
| *** armax has joined #openstack-infra | 03:24 | |
| AJaeger | mnaser, ianw, https://review.openstack.org/535811 as well, please ^ | 03:26 |
| *** zhurong_ has joined #openstack-infra | 03:27 | |
| *** olaph1 has quit IRC | 03:30 | |
| *** olaph has joined #openstack-infra | 03:30 | |
| *** rlandy has quit IRC | 03:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add zuul-website repo https://review.openstack.org/537670 | 03:31 |
| *** openstackgerrit has quit IRC | 03:33 | |
| *** yamamoto has joined #openstack-infra | 03:37 | |
| *** armax_ has joined #openstack-infra | 03:43 | |
| *** armax has quit IRC | 03:43 | |
| *** armax_ is now known as armax | 03:43 | |
| AJaeger | ianw: could you also review https://review.openstack.org/536527 , please? That's the new python36 job. | 03:47 |
| *** openstackgerrit has joined #openstack-infra | 03:49 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Shift javascript publish jobs to post https://review.openstack.org/536952 | 03:49 |
| *** annp has joined #openstack-infra | 03:49 | |
| AJaeger | wow, zuul now uses 20 GB - good that we upgraded. http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all | 03:50 |
| *** olaph1 has joined #openstack-infra | 03:51 | |
| *** olaph has quit IRC | 03:52 | |
| *** namnh has joined #openstack-infra | 03:57 | |
| *** rosmaita has quit IRC | 04:06 | |
| *** janki has joined #openstack-infra | 04:09 | |
| *** links has joined #openstack-infra | 04:10 | |
| *** jamesmcarthur has joined #openstack-infra | 04:10 | |
| *** jamesmcarthur has quit IRC | 04:14 | |
| *** sree has joined #openstack-infra | 04:22 | |
| *** dave-mccowan has quit IRC | 04:24 | |
| *** bobh has quit IRC | 04:26 | |
| *** bobh has joined #openstack-infra | 04:27 | |
| *** bobh has quit IRC | 04:27 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: GPT partitioning support https://review.openstack.org/533490 | 04:28 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: bootloader: handle grub-efi better https://review.openstack.org/536600 | 04:28 |
| *** harlowja has joined #openstack-infra | 04:28 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: change default image into GPT with ESP and BSP https://review.openstack.org/536601 | 04:28 |
| *** zhurong_ has quit IRC | 04:39 | |
| *** psachin has joined #openstack-infra | 04:42 | |
| *** jamesmcarthur has joined #openstack-infra | 04:45 | |
| *** ramishra has joined #openstack-infra | 04:47 | |
| *** claudiub has joined #openstack-infra | 04:48 | |
| *** jamesmcarthur has quit IRC | 04:51 | |
| *** jamesmcarthur has joined #openstack-infra | 04:53 | |
| *** zhenguo has quit IRC | 04:54 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove remaining Mistral jobs https://review.openstack.org/535811 | 04:56 |
| *** jamesmcarthur has quit IRC | 04:58 | |
| *** sree has quit IRC | 05:10 | |
| *** hrubi has quit IRC | 05:11 | |
| *** sree has joined #openstack-infra | 05:12 | |
| *** hrubi has joined #openstack-infra | 05:13 | |
| *** sree has quit IRC | 05:16 | |
| *** harlowja has quit IRC | 05:17 | |
| *** olaph has joined #openstack-infra | 05:17 | |
| *** olaph1 has quit IRC | 05:19 | |
| EmilienM | ianw: http://logs.openstack.org/72/537572/5/check/puppet-openstack-integration-4-scenario002-tempest-ubuntu-xenial/2c308d8/logs/apt-cache-policy.txt.gz | 05:19 |
| EmilienM | ianw: the mirror works for on ubuntu btw | 05:19 |
| EmilienM | 500 http://mirror.sto2.citycloud.openstack.org/apt-puppetlabs xenial/PC1 amd64 Packages | 05:19 |
| EmilienM | see https://review.openstack.org/#/c/537572/ | 05:19 |
| *** askb has quit IRC | 05:21 | |
| *** abelur_ has quit IRC | 05:21 | |
| EmilienM | ianw: now waiting on the yum one | 05:21 |
| EmilienM | https://review.openstack.org/#/c/537633/ | 05:21 |
| *** jtomasek has joined #openstack-infra | 05:23 | |
| *** jaosorior has joined #openstack-infra | 05:23 | |
| *** hongbin has quit IRC | 05:25 | |
| *** salv-orlando has joined #openstack-infra | 05:31 | |
| *** aviau has quit IRC | 05:31 | |
| *** salv-orl_ has quit IRC | 05:31 | |
| *** aviau has joined #openstack-infra | 05:31 | |
| *** abelur_ has joined #openstack-infra | 05:31 | |
| *** sree has joined #openstack-infra | 05:33 | |
| *** askb has joined #openstack-infra | 05:33 | |
| *** sree has quit IRC | 05:37 | |
| *** jamesmcarthur has joined #openstack-infra | 05:40 | |
| *** gongysh has joined #openstack-infra | 05:41 | |
| *** jamesmcarthur has quit IRC | 05:46 | |
| *** jamesmcarthur has joined #openstack-infra | 05:48 | |
| *** cshastri has joined #openstack-infra | 05:49 | |
| *** jamesmcarthur has quit IRC | 05:53 | |
| ianw | EmilienM: me too, it's still doing the initial clone | 05:56 |
| ianw | :) | 05:56 |
| EmilienM | ok | 05:56 |
| ianw | EmilienM: there's a partial release at http://mirror.iad.rax.openstack.org/yum-puppetlabs/ | 05:56 |
| ianw | does the top level look ok? | 05:56 |
| ianw | it's died a few times due to afs errors (but the script continued on to release the volume anyway) | 05:57 |
| ianw | there's probably a lot we could prune ... it's doing fedora 20-> for example | 05:57 |
| EmilienM | ianw: what's '7' dir? | 05:58 |
| EmilienM | http://mirror.iad.rax.openstack.org/yum-puppetlabs/yum could be http://mirror.iad.rax.openstack.org/yum-puppetlabs/ I think | 05:58 |
| EmilienM | but I don't mind having an extra 'yum', it's just pointless | 05:58 |
| EmilienM | I guess I'll need to modify https://review.openstack.org/#/c/537633/4/modules/openstack_project/files/mirror/yum-puppetlabs-mirror-update.sh | 05:59 |
| ianw | hmm, would $mirror/yum/ (trailing slash) do it? | 06:00 |
| ianw | (always get so lost with rsync ... i'm sure the command line is turing complete) | 06:00 |
| *** pgadiya has joined #openstack-infra | 06:05 | |
| EmilienM | ianw: how could we deploy the mirror by default in the centos7 images? | 06:06 |
| EmilienM | is that something infra does? | 06:06 |
| EmilienM | or I have to pull it manually? | 06:06 |
| *** ianychoi has quit IRC | 06:08 | |
| *** ianychoi has joined #openstack-infra | 06:09 | |
| *** sree has joined #openstack-infra | 06:09 | |
| *** xinliang has quit IRC | 06:09 | |
| ianw | EmilienM: it would be an ansible role that pulls the info from /etc/ci/mirror_info.sh | 06:10 |
| ianw | i guess it will drop a repo file | 06:11 |
| EmilienM | ianw: I did rsync -rlptDvz rsync://rsync.puppet.com/packages/yum/* | 06:11 |
| EmilienM | and it worked | 06:11 |
| EmilienM | add the /* maybe | 06:11 |
| ianw | EmilienM: yeah, it just needs the / i think. i'll kill the current sync, move everything up, and restart it with that | 06:11 |
| EmilienM | ok | 06:12 |
| EmilienM | ianw: I'm doing early testing with https://review.openstack.org/537572 | 06:12 |
| EmilienM | I go to bed now | 06:12 |
| EmilienM | see you, and thanks again for the help | 06:12 |
| *** zhurong has quit IRC | 06:13 | |
| *** sree has quit IRC | 06:13 | |
| *** rcernin has quit IRC | 06:14 | |
| *** sree has joined #openstack-infra | 06:16 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Revert "Revert "Add Puppetlabs mirror for CentOS7"" https://review.openstack.org/537633 | 06:17 |
| *** sree has quit IRC | 06:20 | |
| *** xinliang has joined #openstack-infra | 06:22 | |
| *** xinliang has quit IRC | 06:22 | |
| *** xinliang has joined #openstack-infra | 06:22 | |
| *** zhurong has joined #openstack-infra | 06:22 | |
| *** dbecker has quit IRC | 06:30 | |
| ianw | infra-root: linaro cloud integrated into the launcher, 211.148.24.199 is a host with puppet at least deployed ok! i'll continue on and see if the node launcher works with it in the near future. | 06:32 |
| ianw | infra-root: https://review.openstack.org/537706 https://review.openstack.org/537713 https://review.openstack.org/537715 https://review.openstack.org/537714 for your perusal so the cloud-launcher can do it's thing when you have a chance | 06:33 |
| *** sree has joined #openstack-infra | 06:40 | |
| *** janki has quit IRC | 06:42 | |
| *** janki has joined #openstack-infra | 06:43 | |
| *** dbecker has joined #openstack-infra | 06:43 | |
| *** armax has quit IRC | 06:45 | |
| *** armax has joined #openstack-infra | 06:46 | |
| *** armax has quit IRC | 06:46 | |
| *** armax has joined #openstack-infra | 06:46 | |
| *** armax has quit IRC | 06:47 | |
| *** sree has quit IRC | 06:47 | |
| *** armax has joined #openstack-infra | 06:47 | |
| *** armax has quit IRC | 06:47 | |
| *** jamesmcarthur has joined #openstack-infra | 06:48 | |
| *** armax has joined #openstack-infra | 06:48 | |
| *** armax has quit IRC | 06:48 | |
| *** armax has joined #openstack-infra | 06:49 | |
| *** armax has quit IRC | 06:49 | |
| *** armax has joined #openstack-infra | 06:49 | |
| *** armax has quit IRC | 06:50 | |
| *** sree has joined #openstack-infra | 06:51 | |
| *** jamesmcarthur has quit IRC | 06:52 | |
| *** markvoelker has quit IRC | 06:52 | |
| *** markvoelker has joined #openstack-infra | 06:53 | |
| *** sree has quit IRC | 06:55 | |
| *** markvoelker has quit IRC | 06:57 | |
| *** CrayZee has joined #openstack-infra | 07:01 | |
| *** threestrands_ has quit IRC | 07:02 | |
| *** CrayZee is now known as snapiri- | 07:02 | |
| *** gouthamr has joined #openstack-infra | 07:02 | |
| *** dsariel has joined #openstack-infra | 07:08 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: GPT partitioning support https://review.openstack.org/533490 | 07:09 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: bootloader: handle grub-efi better https://review.openstack.org/536600 | 07:09 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: change default image into GPT with ESP and BSP https://review.openstack.org/536601 | 07:09 |
| *** larainema has joined #openstack-infra | 07:10 | |
| AJaeger | frickler: could you put these on your review queue, please? https://review.openstack.org/#/c/536097/ https://review.openstack.org/#/c/536092/ https://review.openstack.org/#/c/536527/ | 07:11 |
| *** liujiong has quit IRC | 07:18 | |
| openstackgerrit | megan guiney proposed openstack-infra/subunit2sql master: Add delete by uuid functions https://review.openstack.org/537775 | 07:18 |
| *** ethfci has quit IRC | 07:19 | |
| *** jtomasek has quit IRC | 07:26 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Default max pool resources to math.inf https://review.openstack.org/537776 | 07:26 |
| *** slaweq has joined #openstack-infra | 07:28 | |
| openstackgerrit | Ifat Afek proposed openstack-infra/project-config master: Update vitrage-dashboard publish job https://review.openstack.org/537781 | 07:29 |
| *** e0ne has joined #openstack-infra | 07:30 | |
| *** slaweq has quit IRC | 07:32 | |
| *** slaweq has joined #openstack-infra | 07:32 | |
| *** armaan has joined #openstack-infra | 07:47 | |
| *** verdurin_ has quit IRC | 07:47 | |
| *** stakeda has quit IRC | 07:47 | |
| *** jamesmcarthur has joined #openstack-infra | 07:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use openstack-tox-python36 https://review.openstack.org/537790 | 07:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use openstack-tox-python36 https://review.openstack.org/537790 | 07:50 |
| *** jamesmcarthur has quit IRC | 07:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use openstack-tox-py36 https://review.openstack.org/537790 | 07:53 |
| AJaeger | ianw: another bashate change https://review.openstack.org/537791 | 07:53 |
| *** sree has joined #openstack-infra | 07:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move legacy-cross-midonet in-tree https://review.openstack.org/536092 | 07:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove requests-mock legacy jobs https://review.openstack.org/536097 | 07:54 |
| AJaeger | frickler, ianw: Thanks for reviews. Could I bother you with https://review.openstack.org/537790 as well, please? | 07:55 |
| *** pcaruana has joined #openstack-infra | 07:55 | |
| *** e0ne has quit IRC | 07:56 | |
| *** sree has quit IRC | 07:59 | |
| *** verdurin_ has joined #openstack-infra | 07:59 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Create openstack-tox-py36 job https://review.openstack.org/536527 | 08:03 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete legacy jobs https://review.openstack.org/536100 | 08:06 |
| *** florianf has joined #openstack-infra | 08:13 | |
| *** s-shiono has quit IRC | 08:15 | |
| *** florianf has quit IRC | 08:16 | |
| *** florianf has joined #openstack-infra | 08:16 | |
| *** tesseract has joined #openstack-infra | 08:20 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: New project for Tatu (SSH as a Service) Horizon Plugin. https://review.openstack.org/537653 | 08:24 |
| *** vivsoni has quit IRC | 08:24 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: New project Tatu (SSH as a Service). https://review.openstack.org/511335 | 08:25 |
| *** gongysh has quit IRC | 08:26 | |
| *** gcb has quit IRC | 08:26 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Add new project for Tatu (SSH as a Service) Horizon Plugin. https://review.openstack.org/537653 | 08:27 |
| *** dingyichen has quit IRC | 08:27 | |
| *** caphrim007_ has quit IRC | 08:27 | |
| *** caphrim007 has joined #openstack-infra | 08:28 | |
| *** gcb has joined #openstack-infra | 08:29 | |
| *** xarses has quit IRC | 08:30 | |
| *** xarses has joined #openstack-infra | 08:30 | |
| *** janki has quit IRC | 08:31 | |
| *** gongysh has joined #openstack-infra | 08:33 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: New project for Tatu (SSH as a Service) CLI and Python Client. https://review.openstack.org/537802 | 08:37 |
| *** vivsoni has joined #openstack-infra | 08:38 | |
| *** jpena|off is now known as jpena | 08:44 | |
| *** sree has joined #openstack-infra | 08:45 | |
| *** janki has joined #openstack-infra | 08:47 | |
| *** ralonsoh has joined #openstack-infra | 08:48 | |
| *** sree has quit IRC | 08:50 | |
| *** shardy_afk is now known as shardy | 08:51 | |
| *** jpich has joined #openstack-infra | 08:51 | |
| *** shardy has quit IRC | 08:51 | |
| *** slaweq has quit IRC | 08:52 | |
| *** slaweq has joined #openstack-infra | 08:54 | |
| *** markvoelker has joined #openstack-infra | 08:54 | |
| *** shardy has joined #openstack-infra | 08:55 | |
| *** zhurong has quit IRC | 08:55 | |
| *** oidgar has joined #openstack-infra | 09:04 | |
| *** jamesmcarthur has joined #openstack-infra | 09:04 | |
| *** zoli is now known as zoli|brb | 09:04 | |
| *** zoli|brb is now known as zoli | 09:05 | |
| *** sree has joined #openstack-infra | 09:09 | |
| *** jamesmcarthur has quit IRC | 09:10 | |
| *** jbadiapa has joined #openstack-infra | 09:13 | |
| *** sree has quit IRC | 09:14 | |
| *** apetrich has joined #openstack-infra | 09:15 | |
| *** kopecmartin has joined #openstack-infra | 09:19 | |
| *** dbecker has quit IRC | 09:19 | |
| *** dbecker has joined #openstack-infra | 09:21 | |
| AJaeger | pabelanger: could you look at https://review.openstack.org/537789 , please? - seems windmill jobs are broken | 09:23 |
| *** shardy has quit IRC | 09:23 | |
| *** zhurong has joined #openstack-infra | 09:25 | |
| *** shardy has joined #openstack-infra | 09:26 | |
| *** markvoelker has quit IRC | 09:27 | |
| *** amoralej|off is now known as amoralej | 09:29 | |
| *** gibi_ has joined #openstack-infra | 09:29 | |
| *** derekh has joined #openstack-infra | 09:29 | |
| *** jamesmcarthur has joined #openstack-infra | 09:31 | |
| *** e0ne has joined #openstack-infra | 09:31 | |
| *** vivsoni has quit IRC | 09:34 | |
| *** vivsoni has joined #openstack-infra | 09:35 | |
| *** jamesmcarthur has quit IRC | 09:36 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 09:37 |
| *** apetrich has quit IRC | 09:39 | |
| openstackgerrit | Ifat Afek proposed openstack-infra/project-config master: Update vitrage-dashboard publish job https://review.openstack.org/537781 | 09:42 |
| *** vivsoni_ has joined #openstack-infra | 09:43 | |
| *** vivsoni has quit IRC | 09:47 | |
| *** jtomasek has joined #openstack-infra | 09:51 | |
| *** dsariel has quit IRC | 09:51 | |
| *** jtomasek has quit IRC | 09:54 | |
| *** jtomasek has joined #openstack-infra | 09:55 | |
| *** zhurong_ has joined #openstack-infra | 09:56 | |
| *** agopi has quit IRC | 09:57 | |
| *** dsariel has joined #openstack-infra | 10:00 | |
| *** electrofelix has joined #openstack-infra | 10:01 | |
| *** asilenkov has quit IRC | 10:03 | |
| *** asilenkov has joined #openstack-infra | 10:03 | |
| *** rakhmerov has quit IRC | 10:04 | |
| *** yamamoto has quit IRC | 10:04 | |
| *** rakhmerov has joined #openstack-infra | 10:04 | |
| *** jchhatbar has joined #openstack-infra | 10:10 | |
| *** florianf has quit IRC | 10:10 | |
| *** janki has quit IRC | 10:11 | |
| *** erlon has joined #openstack-infra | 10:12 | |
| *** jchhatbar has quit IRC | 10:14 | |
| *** apetrich has joined #openstack-infra | 10:15 | |
| *** florianf has joined #openstack-infra | 10:21 | |
| *** jtomasek has quit IRC | 10:21 | |
| *** markvoelker has joined #openstack-infra | 10:25 | |
| *** cuongnv has quit IRC | 10:26 | |
| d0ugal | Where does the build-openstack-api-ref job come from? https://review.openstack.org/#/c/537064/ | 10:32 |
| *** jpena is now known as jpena|off | 10:33 | |
| *** jpena|off is now known as jpena | 10:33 | |
| d0ugal | afaik, we don't enable it anywhere, it just started on that patch. | 10:34 |
| *** cshastri has quit IRC | 10:34 | |
| *** ldnunes has joined #openstack-infra | 10:35 | |
| *** cshastri has joined #openstack-infra | 10:35 | |
| *** pbourke has quit IRC | 10:36 | |
| *** florianf has quit IRC | 10:37 | |
| *** lucas-afk is now known as lucasagomes | 10:37 | |
| *** florianf has joined #openstack-infra | 10:37 | |
| *** florianf has quit IRC | 10:38 | |
| *** florianf_ has joined #openstack-infra | 10:38 | |
| *** pbourke has joined #openstack-infra | 10:38 | |
| *** zoli is now known as zoli|lunch | 10:39 | |
| *** zoli|lunch is now known as zoli | 10:39 | |
| frickler | d0ugal: https://review.openstack.org/536294 | 10:41 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 10:41 |
| *** jamesmcarthur has joined #openstack-infra | 10:41 | |
| d0ugal | frickler: oh! thanks | 10:42 |
| *** kjackal has joined #openstack-infra | 10:43 | |
| d0ugal | shame it is broken :) | 10:43 |
| frickler | d0ugal: seem we missed to check for PTL approval, do you want a revert? | 10:44 |
| d0ugal | frickler: Yeah, I guess we should for now. cc rakhmerov (PTL) | 10:44 |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config master: Revert "Add api-ref jobs for Workflow service (mistral)" https://review.openstack.org/537839 | 10:44 |
| AJaeger | d0ugal: could you update the commit message with the reason, please? We'll speed approve then... | 10:46 |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config master: Revert "Add api-ref jobs for Workflow service (mistral)" https://review.openstack.org/537839 | 10:46 |
| d0ugal | AJaeger: Done :) | 10:46 |
| *** annp has quit IRC | 10:46 | |
| AJaeger | d0ugal: approved | 10:46 |
| d0ugal | Thanks! | 10:47 |
| frickler | d0ugal: AJaeger: the error looks like it might also be easily fixed, but I'm no sphinx expert http://logs.openstack.org/64/537064/2/check/build-openstack-api-ref/43ea4e0/job-output.txt.gz#_2018-01-25_05_32_09_545399 | 10:47 |
| *** jamesmcarthur has quit IRC | 10:47 | |
| d0ugal | I assume we should re-enable this in our .zuul.yaml file? The v3 migration just merged. | 10:47 |
| AJaeger | d0ugal: api-ref should stay in project-config | 10:47 |
| d0ugal | ok | 10:47 |
| d0ugal | I'll look into fixing it. | 10:48 |
| AJaeger | d0ugal: but you can test locally that it works - so, add it temporarily as part of fixing it - and once it works, remove and prpose the revert of the revert... | 10:48 |
| AJaeger | d0ugal: I wonder whether that sphinx extension is needed at all - I suggest t oremove it from api-ref/source/conf.py first... | 10:49 |
| *** jistr is now known as jistr|mtg | 10:49 | |
| d0ugal | AJaeger: will do, thanks. | 10:50 |
| *** gongysh has quit IRC | 10:54 | |
| *** jbadiapa has quit IRC | 10:55 | |
| *** kiennt26 has quit IRC | 10:56 | |
| *** sree has joined #openstack-infra | 10:56 | |
| *** markvoelker has quit IRC | 10:58 | |
| *** vkmc has quit IRC | 10:58 | |
| *** sree has quit IRC | 11:01 | |
| *** yamamoto has joined #openstack-infra | 11:05 | |
| *** vkmc has joined #openstack-infra | 11:05 | |
| *** sree has joined #openstack-infra | 11:08 | |
| AJaeger | d0ugal: yes, remove wsmeext.sphinxext and sphinxcontrib.pecanwsme.rest - and then you run into the next problem ;( | 11:10 |
| AJaeger | d0ugal: api-ref is not ready and needs some more work | 11:11 |
| *** apetrich has quit IRC | 11:11 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Propose to move submit-log-processor-jobs and submit-logstash-jobs in zuul-jobs https://review.openstack.org/537847 | 11:11 |
| d0ugal | AJaeger: so it seems, I am looking into the next issue now :) | 11:11 |
| *** HenryG has quit IRC | 11:11 | |
| d0ugal | but given the api-ref is empty, there is no rush for CI to build it :) | 11:11 |
| *** HenryG has joined #openstack-infra | 11:13 | |
| *** sree has quit IRC | 11:13 | |
| AJaeger | ;/ | 11:14 |
| *** gcb has quit IRC | 11:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Add api-ref jobs for Workflow service (mistral)" https://review.openstack.org/537839 | 11:18 |
| *** yamamoto has quit IRC | 11:18 | |
| *** gcb has joined #openstack-infra | 11:20 | |
| *** jamesmcarthur has joined #openstack-infra | 11:24 | |
| *** apetrich has joined #openstack-infra | 11:28 | |
| *** jamesmcarthur has quit IRC | 11:29 | |
| ssbarnea | hi! does openstack has any "sensitive" repositories where test gates are not started until someone approves the CR to be safe for gating? | 11:29 |
| *** jamesmcarthur has joined #openstack-infra | 11:30 | |
| *** alexchadin has joined #openstack-infra | 11:31 | |
| *** yamamoto has joined #openstack-infra | 11:31 | |
| *** namnh has quit IRC | 11:31 | |
| *** abelur_ has quit IRC | 11:32 | |
| *** jamesmcarthur has quit IRC | 11:34 | |
| *** zhurong has quit IRC | 11:35 | |
| *** sshnaidm_ has joined #openstack-infra | 11:39 | |
| *** sshnaidm has quit IRC | 11:41 | |
| *** tpsilva has joined #openstack-infra | 11:48 | |
| ianw | ssbarnea: do you mean that changes don't merge until a particular person signs off on them? no; not more than obviously only cores can +2/w. we have "liasons" who by convention have responsibility over things | 11:53 |
| ianw | see https://wiki.openstack.org/wiki/CrossProjectLiaisons | 11:53 |
| ssbarnea | ianw: nope, different. this is about giving the ok for the gate to run. We have some gates running on intranet and we cannot run them using and random CR coming from the "internet". Has nothing to do with the real review of the code or the decision to merge it. this is before anything else. | 11:55 |
| *** zhurong has joined #openstack-infra | 11:55 | |
| *** markvoelker has joined #openstack-infra | 11:55 | |
| ssbarnea | ianw: imagine that someone adds a "rm -rf /" inside tox.ini (or something even worse). | 11:56 |
| ssbarnea | i am trying to find out if someone already has a working implementation that avoids risks like this without preventing anyone from raising CRs. | 11:56 |
| ssbarnea | someone devs around here are not very keen on having to manually add a label in order to start a gate. | 11:57 |
| *** jamesmcarthur has joined #openstack-infra | 11:58 | |
| ssbarnea | ianw: to be clear, I am talking about gerrtihub.io gerrit instance. now we limit CR opening only to github org members and we want to allow anyone to open CRs. | 11:59 |
| *** smatzek has joined #openstack-infra | 12:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use openstack-tox-py36 https://review.openstack.org/537790 | 12:03 |
| *** jamesmcarthur has quit IRC | 12:03 | |
| openstackgerrit | Ifat Afek proposed openstack-infra/project-config master: Update vitrage-dashboard python jobs and publish job https://review.openstack.org/537781 | 12:03 |
| *** jpena is now known as jpena|lunch | 12:03 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 12:05 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 12:05 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: /{tenant}/projects.json routes https://review.openstack.org/537870 | 12:05 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 12:06 |
| *** jkilpatr has quit IRC | 12:08 | |
| *** sree has joined #openstack-infra | 12:14 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/openstack-zuul-jobs master: n8g-bagpipe: add n8g-sfc to PROJECTS for tempest job https://review.openstack.org/537876 | 12:17 |
| *** sjmc7 has joined #openstack-infra | 12:18 | |
| *** janki has joined #openstack-infra | 12:19 | |
| *** gouthamr has quit IRC | 12:19 | |
| *** sree has quit IRC | 12:19 | |
| *** gouthamr has joined #openstack-infra | 12:20 | |
| sjmc7 | hey, wonder if anyone can help. we’ve got a set of tests that rely on running our API process and then running requests against it. the tests run under py27 and py35 and sometime after late Jan 17th the py27 tests stopped working. still trying to determine why but it doesn’t seem related to any requirements or code changes, so i was wondering if anyone knows of any changes to the tests or infra between jan 18th and jan 23rd that i could look at? e | 12:21 |
| sjmc7 | review is https://review.openstack.org/#/c/535137/. the error’s eventually raised when the API server can’t be reached with socket.connect on 127.0.0.1 | 12:21 |
| clarkb | ssbarnea: by definition gate jobs dont start until a reviewer has reviewed and approved the change. To handle things like rm -rf we rely on single use VMs to provide isolation | 12:22 |
| clarkb | ssbarnea: if that isnt sufficient for you I'm not sure how you get away from manually applying a value to give the ok | 12:22 |
| *** tesseract has quit IRC | 12:24 | |
| ssbarnea | clarkb: correct me if I didn't understand: gates do not start until someone adds Code-Review +2 ? (+1 can be added by any user) | 12:24 |
| clarkb | ssbarnea: gate jobs require a verified +1, code review +2, and workflow +1 in our pipeline config | 12:25 |
| ssbarnea | i had the impression that this was not needed, at least on jenkins-job-builder project, the gate starts right away. | 12:25 |
| clarkb | ssbarnea: the check jobs start right away | 12:25 |
| clarkb | but they only vote +/-1 | 12:25 |
| ssbarnea | clarkb: ahh, so two stages. | 12:25 |
| ssbarnea | clarkb: now my problem is how to secure the "check"? | 12:26 |
| clarkb | in our setup that id what we are more worried about. And for us that means single use VMs as well as not allowing check jobs to have access to zuul secrets | 12:27 |
| *** janki has quit IRC | 12:28 | |
| ssbarnea | clarkb: yep, i know that is the ideal way of doing it, but securing the build env is a very-very long shot, not possible now. | 12:28 |
| clarkb | in that case you may not want to run any jobs until some vote is made like code review +2 | 12:29 |
| *** markvoelker has quit IRC | 12:29 | |
| ssbarnea | clarkb: in this case I assume that the trick is to add a check job that is secure and run the sensitive gates only after the review. | 12:29 |
| ssbarnea | clarkb: thanks, this really helps. i will try to write it down and "disseminate" the knowledge. | 12:30 |
| *** yamamoto has quit IRC | 12:31 | |
| clarkb | sjmc7: I think you may want to add logging around unexpected server launch status for api | 12:31 |
| *** yamamoto has joined #openstack-infra | 12:31 | |
| clarkb | sjmc7: I'm guessing its failing to start due to a lort conflict or somethibg like that an you just need to see what that is and correct it | 12:31 |
| *** panda is now known as panda|lunch | 12:31 | |
| Roamer` | sjmc7, clarkb, I believe that mriedem and jroll were talking in #openstack-nova the other day about a new version of Python in Ubuntu, uploaded to the Ubuntu repo on January 18th | 12:32 |
| Roamer` | they were trying to track down something about failing ironic jobs | 12:33 |
| sjmc7 | yeah, i’ve been trying to capture stuff though it looks like stderr and stdout from the process have nothing in them, and the exitcode is apparently 0 | 12:33 |
| sjmc7 | Roamer`: that’s interesting. i’ll see if i can find the conversation, thanks | 12:34 |
| Roamer` | sjmc7, I'll try to find it for you, just a minute | 12:34 |
| *** sambetts|afk is now known as sambetts | 12:35 | |
| *** alexchadin has quit IRC | 12:35 | |
| Roamer` | sjmc7, mriedem found it here: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-01-24.log.html#t2018-01-24T14:53:44 | 12:36 |
| *** alexchadin has joined #openstack-infra | 12:36 | |
| sjmc7 | ah, interesting, thanks. i will see which py27 i’m using | 12:37 |
| Roamer` | ah, actually they were investigating some random segfaults, but later jroll showed up and said that the segfaults no longer happened and no one knew why :) | 12:37 |
| Roamer` | so it might not really be related to the Python version | 12:38 |
| *** apetrich has quit IRC | 12:38 | |
| sjmc7 | might as well start there, i’ve nothing better to try at the moment | 12:38 |
| *** jistr|mtg is now known as jistr | 12:38 | |
| *** sshnaidm_ has quit IRC | 12:38 | |
| *** chrisyang_0660 has quit IRC | 12:46 | |
| *** jkilpatr has joined #openstack-infra | 12:47 | |
| *** jkilpatr has quit IRC | 12:47 | |
| sjmc7 | it is suspect that the py35 tests still work | 12:47 |
| *** jkilpatr has joined #openstack-infra | 12:47 | |
| Roamer` | yeah, in your case it might really be the Python version... although it would be... weird... for a minor update to break working programs | 12:49 |
| Roamer` | weird, but not unheard of | 12:49 |
| sjmc7 | yeah. and python 2.7.12 is pretty elderly | 12:49 |
| *** yamamoto has quit IRC | 12:52 | |
| *** alexchadin has quit IRC | 12:52 | |
| *** alexchadin has joined #openstack-infra | 12:53 | |
| *** jtomasek has joined #openstack-infra | 12:53 | |
| *** sshnaidm_ has joined #openstack-infra | 12:54 | |
| *** makowals has quit IRC | 12:54 | |
| *** makowals has joined #openstack-infra | 12:56 | |
| *** rosmaita has joined #openstack-infra | 12:56 | |
| *** jamesmcarthur has joined #openstack-infra | 12:56 | |
| *** jamesmcarthur has quit IRC | 12:57 | |
| *** jamesmcarthur has joined #openstack-infra | 12:58 | |
| *** JasonCL has joined #openstack-infra | 12:59 | |
| *** snapiri- has quit IRC | 13:00 | |
| *** alexchadin has quit IRC | 13:00 | |
| *** abelur_ has joined #openstack-infra | 13:00 | |
| *** alexchadin has joined #openstack-infra | 13:00 | |
| *** CrayZee has joined #openstack-infra | 13:01 | |
| *** jbadiapa has joined #openstack-infra | 13:01 | |
| *** abelur_ has quit IRC | 13:01 | |
| *** jamesmcarthur has quit IRC | 13:02 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/536100/ to remove a few migrated legacy jobs from openstack-zuul-jobs, please? | 13:04 |
| *** CrayZee is now known as snapiri- | 13:04 | |
| *** dave-mccowan has joined #openstack-infra | 13:05 | |
| *** dprince has joined #openstack-infra | 13:05 | |
| *** florianf_ has quit IRC | 13:06 | |
| *** florianf_ has joined #openstack-infra | 13:06 | |
| *** yamamoto has joined #openstack-infra | 13:07 | |
| smcginnis | We had another release job fail with a timeout doing a git pull. | 13:09 |
| smcginnis | Could we get this re-enqueued? http://logs.openstack.org/32/323c387a2d1794e0679510657629470da8f7de92/release-post/tag-releases/b2091f1/job-output.txt.gz#_2018-01-25_07_05_34_073932 | 13:09 |
| *** sshnaidm_ is now known as sshnaidm | 13:11 | |
| *** zhurong has quit IRC | 13:16 | |
| *** gk__ has joined #openstack-infra | 13:17 | |
| *** alexchadin has quit IRC | 13:18 | |
| efried | Where is http://zuulv3.openstack.org/ ? | 13:18 |
| efried | Did I miss a memo? | 13:19 |
| *** gk__ has quit IRC | 13:19 | |
| *** makowals has quit IRC | 13:19 | |
| Roamer` | efried, after merging Zuul 3 it got renamed to zuul.openstack.org; it seems zuul3 was a temporary name | 13:20 |
| pabelanger | AJaeger: yes, I'm still working on the migration myself. But need to get some reviews on https://review.openstack.org/526708/ | 13:20 |
| efried | Roamer` Thanks | 13:20 |
| *** panda|lunch is now known as panda | 13:20 | |
| Roamer` | efried, and yeah, I need to retrain my fingers and my browser history, too :) | 13:20 |
| efried | This ^^ | 13:21 |
| AJaeger | pabelanger: I stepped away from that one, wanted somebody with better ansible knowledge like dmsimard review 526708 | 13:21 |
| openstackgerrit | Thomas Morin proposed openstack-infra/openstack-zuul-jobs master: n8g-bagpipe: add n8g-sfc to PROJECTS for tempest job https://review.openstack.org/537876 | 13:21 |
| AJaeger | pabelanger: +2, looks fine | 13:22 |
| *** armaan has quit IRC | 13:24 | |
| *** annp has joined #openstack-infra | 13:25 | |
| *** markvoelker has joined #openstack-infra | 13:26 | |
| *** makowals has joined #openstack-infra | 13:26 | |
| *** tesseract has joined #openstack-infra | 13:28 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Propose to move submit-log-processor-jobs and submit-logstash-jobs in zuul-jobs https://review.openstack.org/537847 | 13:28 |
| *** sree has joined #openstack-infra | 13:30 | |
| *** edmondsw_ has joined #openstack-infra | 13:30 | |
| *** liusheng has quit IRC | 13:32 | |
| *** kgiusti has joined #openstack-infra | 13:33 | |
| *** yolanda has joined #openstack-infra | 13:34 | |
| *** pgadiya has quit IRC | 13:34 | |
| *** jamesmcarthur has joined #openstack-infra | 13:35 | |
| smcginnis | Uh oh, something wrong with logs.o.o? | 13:36 |
| smcginnis | Finally loaded, but took a long time. | 13:37 |
| *** xarses has quit IRC | 13:37 | |
| *** rlandy has joined #openstack-infra | 13:37 | |
| *** armaan has joined #openstack-infra | 13:39 | |
| *** sree has quit IRC | 13:39 | |
| *** jamesmcarthur has quit IRC | 13:40 | |
| smcginnis | And now seeing post_failures on all the release-post tag-release jobs. | 13:41 |
| *** yamamoto has quit IRC | 13:42 | |
| *** tesseract has quit IRC | 13:42 | |
| *** jcoufal has joined #openstack-infra | 13:42 | |
| *** tesseract has joined #openstack-infra | 13:42 | |
| smcginnis | Why would a job have a finger:// link instead of a link to the logs? | 13:44 |
| smcginnis | - tag-releases finger://ze03.openstack.org/7a276329f4b245749479f8ea5d40526f : POST_FAILURE in 2m 49s | 13:44 |
| *** zhurong_ has quit IRC | 13:44 | |
| *** trown|outtypewww is now known as trown|rover | 13:45 | |
| *** apetrich has joined #openstack-infra | 13:46 | |
| *** sree has joined #openstack-infra | 13:48 | |
| *** edmondsw_ is now known as edmondsw | 13:48 | |
| *** alexchadin has joined #openstack-infra | 13:49 | |
| *** eharney has quit IRC | 13:49 | |
| *** sree has quit IRC | 13:52 | |
| *** alexchadin has quit IRC | 13:52 | |
| *** wolverineav has joined #openstack-infra | 13:55 | |
| *** esberglu has joined #openstack-infra | 13:55 | |
| pabelanger | sounds like networking issue, and jobs couldn't upload to logs.o.o | 13:57 |
| *** jamesmcarthur has joined #openstack-infra | 13:57 | |
| *** gongysh has joined #openstack-infra | 13:58 | |
| *** weshay|rover is now known as weshay|ruck | 13:58 | |
| *** Goneri has joined #openstack-infra | 13:59 | |
| *** markvoelker has quit IRC | 13:59 | |
| *** dougwig has joined #openstack-infra | 14:00 | |
| *** kzaitsev1pi has joined #openstack-infra | 14:01 | |
| *** david-lyle has quit IRC | 14:01 | |
| *** rfolco|rover is now known as rfolco|ruck | 14:01 | |
| *** agopi has joined #openstack-infra | 14:01 | |
| *** psachin has quit IRC | 14:02 | |
| *** yamamoto has joined #openstack-infra | 14:02 | |
| *** kzaitsev_pi has quit IRC | 14:03 | |
| *** Goneri has quit IRC | 14:05 | |
| *** kzaitsev_pi has joined #openstack-infra | 14:05 | |
| *** Goneri has joined #openstack-infra | 14:07 | |
| *** rfolco|ruck is now known as rfolco|ruck|brb | 14:07 | |
| *** kzaitsev1pi has quit IRC | 14:08 | |
| *** dhill_ has joined #openstack-infra | 14:08 | |
| *** esberglu has quit IRC | 14:08 | |
| efried | Here's something I *think* I noticed, but am not sure. | 14:10 |
| efried | https://review.openstack.org/#/c/526541/ - which is bottom of a long series - was in the gate and in progress (had blue bars inching forward) | 14:10 |
| efried | Patches after it in the series were red, so I started rechecking them. | 14:11 |
| efried | And after that, the blue bars on https://review.openstack.org/#/c/526541/ disappeared and it looks like it's just waiting again. | 14:11 |
| smcginnis | pabelanger: The last several release jobs all failed with those post_failures. I'm holding off on doing any more releases until I hear otherwise. | 14:12 |
| pabelanger | yah, looks like something is happening | 14:13 |
| pabelanger | looking now | 14:13 |
| sshnaidm | happens for tripleo too: ripleo-ci-centos-7-3nodes-multinode finger://ze04.openstack.org/d2893144d5ef4e62b7b764937230c21f : POST_FAILURE in 2h 31m 09s (non-voting) https://review.openstack.org/#/c/526414/ | 14:15 |
| *** erlon_ has joined #openstack-infra | 14:16 | |
| pabelanger | 2018-01-25 14:10:30,559 DEBUG zuul.AnsibleJob: [build: 5bb7c2d71cc44101b1f2905d6949c54e] as requested: [Errno 30] Read-only file system: ''/srv/static/logs/93/536793/1/gate/cross-horizon-py27/5bb7c2d''' | 14:16 |
| pabelanger | that isn't good | 14:16 |
| *** jamesmcarthur has quit IRC | 14:16 | |
| *** jamesmcarthur has joined #openstack-infra | 14:17 | |
| *** tesseract has quit IRC | 14:17 | |
| pabelanger | infra-root: ^I'm going to need some support with logs.o.o, our filesystem is read-only right now | 14:18 |
| erlon_ | @all, guys we are having problems with a missing bin-deps (libpcre3-dev), devstack is not installing it. how/were do we add it in order to be downloaded and not break all jobs?? | 14:18 |
| *** tesseract has joined #openstack-infra | 14:18 | |
| dmsimard | infra-root: let's move to infra-incident | 14:18 |
| dmsimard | pabelanger: how does this sound: #status notice We're currently experiencing issues with the logs.openstack.org server which will result in POST_FAILURE for jobs, please stand by and don't needlessly recheck jobs while we troubleshoot the problem. | 14:22 |
| efried | sounds good to me fwiw. sooner rather than later, cause recheck was my natural reaction | 14:23 |
| *** ganso has joined #openstack-infra | 14:23 | |
| *** hjensas has joined #openstack-infra | 14:23 | |
| *** smatzek has quit IRC | 14:24 | |
| dmsimard | #status notice We're currently experiencing issues with the logs.openstack.org server which will result in POST_FAILURE for jobs, please stand by and don't needlessly recheck jobs while we troubleshoot the problem. | 14:24 |
| openstackstatus | dmsimard: sending notice | 14:24 |
| *** smatzek has joined #openstack-infra | 14:24 | |
| *** smatzek has quit IRC | 14:24 | |
| *** esberglu has joined #openstack-infra | 14:24 | |
| -openstackstatus- NOTICE: We're currently experiencing issues with the logs.openstack.org server which will result in POST_FAILURE for jobs, please stand by and don't needlessly recheck jobs while we troubleshoot the problem. | 14:25 | |
| *** alexchadin has joined #openstack-infra | 14:25 | |
| *** lucasagomes is now known as lucas-hungry | 14:27 | |
| honza | when exactly are 'release' jobs run? on new tags? or is it manual? the zuul manual only says that 'gate', 'check', 'release' are arbitrary, and doesn't describe them | 14:27 |
| *** psachin has joined #openstack-infra | 14:27 | |
| openstackstatus | dmsimard: finished sending notice | 14:27 |
| AJaeger | honza: new tags | 14:27 |
| hjensas | We seem to have log issue, POST_FAILURE due to read only filesyste, See: http://paste.openstack.org/show/653416/ | 14:28 |
| honza | AJaeger: thanks | 14:28 |
| efried | hjensas known, being worked apace | 14:28 |
| AJaeger | honza: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml#n137 is definition | 14:28 |
| AJaeger | hjensas: see message by openstackstatus | 14:29 |
| honza | AJaeger: TIL about that file :) | 14:29 |
| *** gibi_ is now known as gibi | 14:29 | |
| hjensas | AJaeger: efried: yes, sorry missed the status update by dmsimard there. Thanks! | 14:30 |
| *** dmellado has quit IRC | 14:32 | |
| *** psachin has quit IRC | 14:32 | |
| *** jcoufal has quit IRC | 14:32 | |
| *** psachin has joined #openstack-infra | 14:33 | |
| *** stevebaker has quit IRC | 14:33 | |
| *** markvoelker has joined #openstack-infra | 14:34 | |
| *** eharney has joined #openstack-infra | 14:34 | |
| *** jpena|lunch is now known as jpena | 14:35 | |
| *** olaph1 has joined #openstack-infra | 14:35 | |
| *** jcoufal has joined #openstack-infra | 14:35 | |
| *** olaph has quit IRC | 14:36 | |
| *** cshastri has quit IRC | 14:36 | |
| *** cshastri has joined #openstack-infra | 14:37 | |
| *** agopi has quit IRC | 14:39 | |
| *** dmellado has joined #openstack-infra | 14:39 | |
| *** xarses has joined #openstack-infra | 14:39 | |
| *** dmellado has quit IRC | 14:39 | |
| *** xarses_ has joined #openstack-infra | 14:40 | |
| *** andreww has joined #openstack-infra | 14:40 | |
| *** jamesmcarthur has quit IRC | 14:41 | |
| *** sshnaidm_ has joined #openstack-infra | 14:41 | |
| *** dmellado has joined #openstack-infra | 14:41 | |
| *** dmellado has quit IRC | 14:42 | |
| *** stevebaker has joined #openstack-infra | 14:42 | |
| *** dmellado has joined #openstack-infra | 14:43 | |
| *** sshnaidm has quit IRC | 14:44 | |
| *** dmellado has quit IRC | 14:44 | |
| weshay|ruck | greetings, need some help figuring out why we have a job failing.. openstack-tox-linters in the tripleo gate queue on patch 531503,1 I try to get the logs but it's pointing to a finger:// address | 14:44 |
| weshay|ruck | afaik we've hit a few of these and it's reseting the gate | 14:44 |
| dmsimard | weshay|ruck: there was a notice sent out to all channels | 14:44 |
| *** xarses_ has quit IRC | 14:44 | |
| weshay|ruck | ah.. /me checks twitter | 14:44 |
| dmsimard | weshay|ruck: there are issues on logs.openstack.org, we are working on it | 14:44 |
| weshay|ruck | dmsimard, you are busy | 14:45 |
| weshay|ruck | k.. thanks | 14:45 |
| weshay|ruck | ah see it | 14:45 |
| *** ralonsoh_ has joined #openstack-infra | 14:46 | |
| *** stevebaker has quit IRC | 14:47 | |
| *** ralonsoh has quit IRC | 14:48 | |
| *** andreas_s has joined #openstack-infra | 14:50 | |
| *** dmellado has joined #openstack-infra | 14:53 | |
| *** dhill_ has quit IRC | 14:53 | |
| *** dmellado has quit IRC | 14:53 | |
| *** stevebaker has joined #openstack-infra | 14:55 | |
| *** stevebaker has quit IRC | 14:55 | |
| *** kjackal has quit IRC | 14:55 | |
| *** kopecmartin has quit IRC | 14:56 | |
| *** stevebaker has joined #openstack-infra | 14:56 | |
| *** stevebaker has quit IRC | 14:56 | |
| *** pcichy has quit IRC | 14:56 | |
| *** alexchadin has quit IRC | 14:57 | |
| *** dhill_ has joined #openstack-infra | 14:58 | |
| *** jamesmcarthur has joined #openstack-infra | 14:59 | |
| *** smatzek has joined #openstack-infra | 15:00 | |
| *** smatzek has quit IRC | 15:00 | |
| *** dmellado has joined #openstack-infra | 15:01 | |
| *** smatzek has joined #openstack-infra | 15:01 | |
| *** dmellado has quit IRC | 15:01 | |
| *** agopi has joined #openstack-infra | 15:02 | |
| *** r-daneel has joined #openstack-infra | 15:02 | |
| *** dmellado has joined #openstack-infra | 15:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Only upload logs when jobs fail https://review.openstack.org/537929 | 15:03 |
| *** dmellado has quit IRC | 15:03 | |
| *** jamesmcarthur has quit IRC | 15:03 | |
| *** agopi_ has joined #openstack-infra | 15:04 | |
| *** bobh has joined #openstack-infra | 15:06 | |
| *** agopi has quit IRC | 15:06 | |
| *** ihrachys_ is now known as ihrachys | 15:08 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Handle missing request during a decline. https://review.openstack.org/537932 | 15:09 |
| *** caphrim007 has quit IRC | 15:10 | |
| prometheanfire | bad FS again? | 15:11 |
| *** sshnaidm_ is now known as sshnaidm | 15:12 | |
| efried | prometheanfire Yes. Working in -incident | 15:13 |
| prometheanfire | neato :P | 15:14 |
| *** andreas_s has quit IRC | 15:15 | |
| *** andreas_s has joined #openstack-infra | 15:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Only upload logs when jobs fail https://review.openstack.org/537929 | 15:17 |
| *** andreas_s_ has joined #openstack-infra | 15:18 | |
| *** janki has joined #openstack-infra | 15:18 | |
| *** kopecmartin has joined #openstack-infra | 15:18 | |
| *** andreas_s has quit IRC | 15:18 | |
| frickler | sjmc7: seems neutron also has issue with the new py27, maybe this can help you, too: https://review.openstack.org/#/c/537863/1 | 15:18 |
| *** dmellado has joined #openstack-infra | 15:19 | |
| *** gongysh has quit IRC | 15:19 | |
| sjmc7 | thanks frickler . looks like glance is having the same problem as us too | 15:20 |
| *** pcaruana has quit IRC | 15:20 | |
| frickler | sjmc7: yeah, if the update is breaking eventlet, that will have pretty widespread impact | 15:22 |
| *** armax has joined #openstack-infra | 15:22 | |
| *** annp has quit IRC | 15:22 | |
| *** lucas-hungry is now known as lucasagomes | 15:22 | |
| *** alexchadin has joined #openstack-infra | 15:25 | |
| *** dmellado has quit IRC | 15:25 | |
| *** dmellado has joined #openstack-infra | 15:26 | |
| *** dmellado has quit IRC | 15:27 | |
| *** apetrich has quit IRC | 15:27 | |
| *** olaph1 has quit IRC | 15:29 | |
| *** dmellado has joined #openstack-infra | 15:29 | |
| *** dmellado has quit IRC | 15:29 | |
| *** olaph has joined #openstack-infra | 15:29 | |
| *** hjensas has quit IRC | 15:30 | |
| *** dave-mccowan has quit IRC | 15:31 | |
| *** d0ugal has quit IRC | 15:31 | |
| *** dave-mccowan has joined #openstack-infra | 15:31 | |
| *** numans has quit IRC | 15:34 | |
| *** numans has joined #openstack-infra | 15:34 | |
| *** david-lyle has joined #openstack-infra | 15:38 | |
| sjmc7 | that looks like it, frickler - thanks for the pointer | 15:39 |
| *** d0ugal has joined #openstack-infra | 15:40 | |
| openstackgerrit | melissaml proposed openstack-infra/jeepyb master: fix misspelling of 'password' https://review.openstack.org/537941 | 15:41 |
| *** adarazs is now known as adarazs_biab | 15:41 | |
| *** petevg has left #openstack-infra | 15:42 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Stop the PoolWorker thread when max-servers is 0 https://review.openstack.org/537942 | 15:44 |
| *** alexchadin has quit IRC | 15:45 | |
| *** yolanda has quit IRC | 15:47 | |
| *** oidgar has quit IRC | 15:48 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Allow for max-servers less than 0 https://review.openstack.org/537946 | 15:48 |
| *** sree has joined #openstack-infra | 15:49 | |
| *** larainema has quit IRC | 15:49 | |
| *** jbadiapa has quit IRC | 15:50 | |
| *** jamesmcarthur has joined #openstack-infra | 15:52 | |
| *** hongbin has joined #openstack-infra | 15:52 | |
| *** sree has quit IRC | 15:53 | |
| *** camunoz has joined #openstack-infra | 15:54 | |
| *** ramishra has quit IRC | 15:54 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Allow for max-servers less than 0 https://review.openstack.org/537946 | 15:55 |
| *** jamesmcarthur has quit IRC | 15:56 | |
| *** rfolco|ruck|brb is now known as rfolco|ruck | 15:57 | |
| *** slaweq has quit IRC | 15:57 | |
| dmsimard | infra-root: I'd like to send the following update: #status notice logs.openstack.org is stabilized and there should no longer be *new* POST_FAILURE errors. Logs for jobs that ran in the past weeks until earlier today are currently unavailable while a FSCK runs on the volume they are hosted on. We're going to temporarily disable *successful* jobs from uploading their logs to reduce strain on our | 15:57 |
| dmsimard | current limited capacity. Thanks for your patience ! | 15:57 |
| *** slaweq has joined #openstack-infra | 15:58 | |
| dmsimard | Oops, that might be a bit long.. let me shorten it up | 15:58 |
| corvus | dmsimard: looks good, modulo 512 bytes :) | 15:58 |
| openstackgerrit | Thomas Morin proposed openstack-infra/openstack-zuul-jobs master: n8g-bagpipe: add n8g-sfc to PROJECTS for tempest job https://review.openstack.org/537876 | 15:58 |
| *** e0ne has quit IRC | 15:59 | |
| dmsimard | v2 #status notice logs.openstack.org is stabilized and there should no longer be *new* POST_FAILURE errors. Logs for jobs that ran in the past weeks until earlier today are currently unavailable pending FSCK completion. We're going to temporarily disable *successful* jobs from uploading their logs to reduce strain on our current limited capacity. Thanks for your patience ! | 15:59 |
| *** adarazs_biab is now known as adarazs | 15:59 | |
| corvus | dmsimard: ++ | 16:00 |
| corvus | patch 2/4 of the mailman template series sucessfully applied, i'm approving 3/4: https://review.openstack.org/535852 this will switch the sites to use the new templates. there should be no change for openstack, and zuul-ci.org should stop using the openstack templates. | 16:00 |
| dmsimard | #status notice logs.openstack.org is stabilized and there should no longer be *new* POST_FAILURE errors. Logs for jobs that ran in the past weeks until earlier today are currently unavailable pending FSCK completion. We're going to temporarily disable *successful* jobs from uploading their logs to reduce strain on our current limited capacity. Thanks for your patience ! | 16:00 |
| openstackstatus | dmsimard: sending notice | 16:00 |
| pabelanger | dmsimard: thanks | 16:00 |
| *** armax_ has joined #openstack-infra | 16:01 | |
| dmsimard | pabelanger: 537929 has been queued for 45 mins in check.. | 16:01 |
| -openstackstatus- NOTICE: logs.openstack.org is stabilized and there should no longer be *new* POST_FAILURE errors. Logs for jobs that ran in the past weeks until earlier today are currently unavailable pending FSCK completion. We're going to temporarily disable *successful* jobs from uploading their logs to reduce strain on our current limited capacity. Thanks for your patience ! | 16:01 | |
| *** smatzek has left #openstack-infra | 16:02 | |
| pabelanger | dmsimard: 2 more pateches in front and will start running jobs, another option is to just enqueue directly into gate, once we have the votes | 16:02 |
| *** slaweq has quit IRC | 16:02 | |
| dhellmann | nice recovery time, folks | 16:03 |
| openstackstatus | dmsimard: finished sending notice | 16:03 |
| dhellmann | when things settle down, I have a few release jobs that need to be reenqueued | 16:03 |
| *** armax has quit IRC | 16:03 | |
| *** armax_ is now known as armax | 16:03 | |
| dmsimard | dhellmann: can you put them in an etherpad ? So we know which ones to do and strike which ones have been handled | 16:04 |
| *** trown|rover is now known as trown|brb | 16:06 | |
| dhellmann | dmsimard : all set: https://etherpad.openstack.org/p/6GloEgik8b | 16:06 |
| dhellmann | those SHA values are for the patches within openstack/releases | 16:07 |
| dmsimard | dhellmann: are any of those time sensitive ? | 16:07 |
| dmsimard | (I know ideally we'd run them all) | 16:07 |
| dmsimard | just want to prioritize if we need to | 16:07 |
| dhellmann | well, today's the library release freeze date so all of them are | 16:08 |
| dmsimard | dhellmann: fair | 16:08 |
| *** gema has quit IRC | 16:09 | |
| dhellmann | it's not the end of the world if we have to wait, of course, but none are more time sensitive than the others | 16:09 |
| Roamer` | (not complaining, just asking) does only uploading error logs mean we don't see the logs even while the jobs are running? | 16:09 |
| Roamer` | and thanks for the quick incident response! | 16:10 |
| dhellmann | dmsimard : now that I've given you the list, I realized I should check to see if any of those tags made it. If they do, we'll need different SHAs to retrigger those failed jobs | 16:10 |
| dmsimard | Roamer`: the job consoles that you can stream from http://zuul.openstack.org are not affected by today's problems | 16:10 |
| dhellmann | otherwise the tag-releases job will just say "already tagged" and not do anything | 16:11 |
| dmsimard | dhellmann: right -- so the jobs probably ran and did what they had to do -- and failed due to log upload is what you mean | 16:11 |
| *** yamahata has quit IRC | 16:11 | |
| dmsimard | dhellmann: I feel like that's something we need to address in some shape or form.. it's "okay" for check/gate jobs to fail due to log upload but release/tag/post are different since they "do" things | 16:12 |
| dmsimard | I don't have any great ideas lright now | 16:12 |
| Roamer` | dmsimard, thanks, those are exactly the ones I meant; it took some time for them to start producing output, so I thought I'd ask. Thanks! | 16:12 |
| Roamer` | but yes, they did start producing output | 16:12 |
| dhellmann | dmsimard : yes, it would be good to have them be more idempotent | 16:12 |
| dhellmann | let me paste the actual errors associated with each job into that etherpad | 16:13 |
| *** trown|brb is now known as trown|rover | 16:13 | |
| dmsimard | pabelanger: 60 minutes... :( | 16:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Enabled ssh retries for ansible https://review.openstack.org/537953 | 16:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Do not attempt to handle requests when disabled https://review.openstack.org/537954 | 16:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop spamming ansible/ansible https://review.openstack.org/537955 | 16:18 |
| dhellmann | dmsimard : ok, the etherpad now includes all of the actual jobs that failed or were skipped | 16:19 |
| dmsimard | dhellmann: thanks, I'm not super familiar with re-enqueuing post/tag/release jobs but at least we got the legwork done so we can eventually take care of it | 16:20 |
| dhellmann | yeah, usually fungi does this based on links to logs but some of these are only giving me finger: links and I don't know what to do with those | 16:21 |
| dmsimard | ok 537929 is starting to actually run jobs from check | 16:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop spamming ansible/ansible https://review.openstack.org/537955 | 16:22 |
| *** andreas_s_ has quit IRC | 16:23 | |
| *** andreas_s has joined #openstack-infra | 16:24 | |
| *** andreas_s_ has joined #openstack-infra | 16:25 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Do not attempt to handle requests when disabled https://review.openstack.org/537954 | 16:26 |
| *** bhavik1 has joined #openstack-infra | 16:27 | |
| *** andreas_s has quit IRC | 16:28 | |
| *** jamesmcarthur has joined #openstack-infra | 16:30 | |
| *** andreas_s_ has quit IRC | 16:30 | |
| openstackgerrit | Merged openstack-infra/statusbot master: Thanks & Success bot provide confirmation site url https://review.openstack.org/510699 | 16:30 |
| *** caphrim007 has joined #openstack-infra | 16:30 | |
| *** jamesmcarthur has quit IRC | 16:31 | |
| *** jamesmca_ has joined #openstack-infra | 16:31 | |
| *** dtantsur|afk is now known as dtantsur | 16:31 | |
| efried | That weird stalling thing I mentioned earlier - now I've seen it happen without anything nearby having been rechecked. | 16:33 |
| dmsimard | efried: I haven't witnessed that, what is it ? | 16:34 |
| efried | Viz: a patch like https://review.openstack.org/#/c/526541/ is sitting in the gate, then it expands and blue bars start to move, then it goes back to collapsed. | 16:34 |
| *** hjensas has joined #openstack-infra | 16:34 | |
| *** hjensas has quit IRC | 16:34 | |
| *** hjensas has joined #openstack-infra | 16:34 | |
| efried | dmsimard And next time it expands, the blue bars start over. | 16:34 |
| efried | or at least seem to. | 16:34 |
| *** larainema has joined #openstack-infra | 16:34 | |
| *** kjackal has joined #openstack-infra | 16:34 | |
| dmsimard | efried: you mean the zuul.openstack.org interface ? | 16:34 |
| efried | yes | 16:35 |
| mordred | efried: it's possible a change ahead of it in the gate queue failed causing it to get re-enqueued | 16:35 |
| pabelanger | yah, I think that is the issue | 16:35 |
| *** jamesmca_ has quit IRC | 16:35 | |
| efried | oh. How does a guy get to the front of the queue? :) | 16:35 |
| dmsimard | Either that or a retry on a pre playbook error | 16:35 |
| pabelanger | integrated queue has been resetting a lot over the last few days | 16:35 |
| mordred | dmsimard: yah - but that should only be one job at a time | 16:36 |
| dmsimard | mordred: I don't know, does a pre retry kick things out of the queue ? | 16:36 |
| mordred | dmsimard: no, it should not | 16:36 |
| mordred | dmsimard: the queue only gets reset if a job fails | 16:37 |
| efried | But just to be sure, if I issue recheck on things later in the series, that shouldn't affect the guy in the gate at all, right? | 16:37 |
| efried | or does it? | 16:37 |
| pabelanger | we just had a post_failure again | 16:37 |
| pabelanger | looking to see why | 16:37 |
| pabelanger | 2018-01-25 16:33:36,927 DEBUG zuul.AnsibleJob: [build: ce494f5752da448187ea73dd316abd59] msg: 'SSH Error: data could not be sent to remote host "logs.openstack.org". | 16:38 |
| openstackgerrit | Merged openstack-infra/puppet-mailman master: Use multisite template dir https://review.openstack.org/535852 | 16:39 |
| *** bhavik1 has quit IRC | 16:41 | |
| *** openstackstatus has quit IRC | 16:41 | |
| *** dsariel has quit IRC | 16:42 | |
| *** dmellado has joined #openstack-infra | 16:42 | |
| *** openstackstatus has joined #openstack-infra | 16:42 | |
| *** ChanServ sets mode: +v openstackstatus | 16:42 | |
| *** linkmark has joined #openstack-infra | 16:43 | |
| *** andreas_s has joined #openstack-infra | 16:45 | |
| *** dmellado has quit IRC | 16:46 | |
| TheJulia | pabelanger: would that happen to have been the ironic job in the gate? | 16:47 |
| pabelanger | TheJulia: 533312,2 | 16:48 |
| *** zoli is now known as zoli|gone | 16:48 | |
| *** zoli|gone is now known as zoli | 16:48 | |
| TheJulia | 535772,2 also has a post_failure :( | 16:49 |
| TheJulia | with-in the last 20 minutes | 16:49 |
| *** oidgar has joined #openstack-infra | 16:49 | |
| *** hamzy has quit IRC | 16:49 | |
| *** slaweq has joined #openstack-infra | 16:49 | |
| *** andreas_s has quit IRC | 16:50 | |
| *** rpittau has quit IRC | 16:53 | |
| *** slaweq has quit IRC | 16:53 | |
| *** dmellado has joined #openstack-infra | 16:53 | |
| *** jamesmcarthur has joined #openstack-infra | 16:54 | |
| pabelanger | TheJulia: yah, logs.o.o still limping along | 16:54 |
| *** jamesmcarthur has quit IRC | 16:54 | |
| pabelanger | trying to get it back online | 16:54 |
| *** jamesmcarthur has joined #openstack-infra | 16:54 | |
| *** jkilpatr has quit IRC | 16:54 | |
| TheJulia | okay, this is going to be a long day... | 16:54 |
| *** jkilpatr has joined #openstack-infra | 16:56 | |
| *** trown|rover is now known as trown|lunch | 16:56 | |
| *** kopecmartin has quit IRC | 16:56 | |
| *** links has quit IRC | 16:57 | |
| *** jtomasek has quit IRC | 16:57 | |
| *** tesseract has quit IRC | 16:58 | |
| *** slaweq has joined #openstack-infra | 17:00 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config master: translation-jobs for neutron-vpnaas-dashboard https://review.openstack.org/537970 | 17:01 |
| *** slaweq has quit IRC | 17:02 | |
| *** rkukura has joined #openstack-infra | 17:03 | |
| *** janki has quit IRC | 17:05 | |
| *** electrofelix has quit IRC | 17:06 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Only upload logs when jobs fail https://review.openstack.org/537929 | 17:07 |
| *** hjensas has quit IRC | 17:07 | |
| pabelanger | mordred: corvus: dmsimard: ^change to upload failed jobs landed | 17:08 |
| mordred | pabelanger: woot | 17:08 |
| *** jamesmcarthur has quit IRC | 17:08 | |
| rajinir | Is there a problem accessing the gate logs? "File Not Found" | 17:09 |
| pabelanger | rajinir: yes, we are working though an outage right now | 17:09 |
| pabelanger | rajinir: so some logs are not available ATM | 17:10 |
| rajinir | ok, thanks pabelanger | 17:10 |
| *** sree has joined #openstack-infra | 17:12 | |
| *** jistr is now known as jistr|conf | 17:13 | |
| *** yolanda has joined #openstack-infra | 17:15 | |
| *** dtantsur is now known as dtantsur|afk | 17:15 | |
| *** sree has quit IRC | 17:16 | |
| *** rkukura has quit IRC | 17:19 | |
| *** jpich has quit IRC | 17:25 | |
| *** jamesmcarthur has joined #openstack-infra | 17:26 | |
| EmilienM | corvus: hey, I'm trying to test "debug" flag in zuul but so far it didn't work on https://review.openstack.org/#/c/537981/1/.zuul.yaml - any idea what I did wrong? | 17:30 |
| *** jamesmcarthur has quit IRC | 17:31 | |
| *** panda is now known as panda|bbl | 17:33 | |
| corvus | EmilienM: you'll have to wait for the report when the jobs complete, sorry. | 17:33 |
| corvus | it'd be nice for it to show up earlier, we could probably do that, but it'll be a bit more work | 17:34 |
| *** kjackal has quit IRC | 17:35 | |
| *** shardy has quit IRC | 17:35 | |
| EmilienM | corvus: ok no worries | 17:35 |
| EmilienM | corvus: I'm debugging a similar issue we had if you remember with projects having custom branches wanted to run stable/ocata jobs for example | 17:36 |
| EmilienM | corvus: puppet-pacemaker has 0.6.x branch and we want the branch to run ocata & newton jobs | 17:36 |
| EmilienM | sounds like a new challenge :) | 17:36 |
| *** gouthamr has quit IRC | 17:38 | |
| *** gouthamr has joined #openstack-infra | 17:40 | |
| *** dmellado has quit IRC | 17:40 | |
| *** yamamoto has quit IRC | 17:40 | |
| *** slaweq has joined #openstack-infra | 17:41 | |
| *** cshastri has quit IRC | 17:42 | |
| *** esberglu has quit IRC | 17:42 | |
| corvus | EmilienM: cool, the stuff we did earlier may be sufficient, or we may need a new thing i've been working on: http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-January/000014.html -- i'm currently favoring https://review.openstack.org/537655 as the solution to that. | 17:43 |
| *** yamamoto has joined #openstack-infra | 17:43 | |
| corvus | EmilienM: let me know when you've poked at the problem and you want to discuss it. | 17:43 |
| *** yamamoto has quit IRC | 17:44 | |
| *** yamamoto has joined #openstack-infra | 17:44 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add support for tuning e2fsck for large filesystems https://review.openstack.org/537983 | 17:46 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Enable e2fsck tuning for large partitions on static.o.o https://review.openstack.org/537984 | 17:46 |
| *** slaweq has quit IRC | 17:46 | |
| dmsimard | infra-root: ^ | 17:46 |
| ganso | hey folks has anyone stumbled on devstack failing because it is unable to find openvswitch on centos? doing a simple yum install openvswitch fails. It worked this monday in the same env, now it doesn't work anymore? | 17:48 |
| dmsimard | ganso: openvswitch is not available in the base CentOS repositories -- I believe devstack is set up to retrieve it from the RDO repositories for CentOS.. at least in our CI jobs that's how it is. | 17:49 |
| ganso | dmsimard: I found an old commit showing that devstack is supposed to add the repo | 17:49 |
| dmsimard | ganso: but it's possible that the repository set up logic is not part of devstack proper and instead is bundled in our jobs or our images, I'm not super knowledgeable around these bits | 17:50 |
| *** florianf_ has quit IRC | 17:50 | |
| ganso | dmsimard: this is an old devstack commit https://review.openstack.org/#/c/141694/3/stack.sh | 17:50 |
| dmsimard | ganso: I mean, regardless of whether devstack does it or not, a good first step would be to check if you have any RDO repositories enabled on your environment | 17:50 |
| dmsimard | If you have it and it's still failing, then there's another problem | 17:50 |
| dmsimard | ganso: that patch is ancient and there's no guarantee anything like that still exists | 17:51 |
| ganso | dmsimard: this is the line that is failing https://github.com/openstack-dev/devstack/blob/master/stack.sh#L321 | 17:53 |
| *** tellesnobrega has joined #openstack-infra | 17:53 | |
| ganso | dmsimard: I copied the yum install command and it didn't have the rdo-release installed | 17:53 |
| ganso | dmsimard: for some reason it is failing to check that | 17:53 |
| dmsimard | ganso: there was an outage on rdoproject.org earlier, can you try again ? | 17:54 |
| *** derekh has quit IRC | 17:54 | |
| dmsimard | ianw: ^ maybe this should be mirrored elsewhere | 17:54 |
| *** slaweq has joined #openstack-infra | 17:54 | |
| ganso | dmsimard: I'll retry just a min | 17:54 |
| pabelanger | we have reverse apache proxy setup | 17:54 |
| dmsimard | pabelanger: not if people are running devstack from their personal environment | 17:55 |
| ganso | dmsimard: it worked, it was probably the outage | 17:55 |
| dmsimard | ganso: sorry about that :( | 17:55 |
| ganso | dmsimard: I removed the packages and run stack.sh again | 17:55 |
| ganso | dmsimard: np =) | 17:55 |
| pabelanger | dmsimard: maybe confused, but why would we mirror in that case? | 17:55 |
| ganso | dmsimard: thanks for the help! =D | 17:55 |
| dmsimard | pabelanger: I don't know, it's something devstack depends on | 17:56 |
| dmsimard | pabelanger: it's just one standalone rpm file which adds the mirror.centos.org mirrors for RDO | 17:57 |
| dmsimard | pabelanger: like.. I wonder if there could perhaps be a "meta" reverse proxy.. like mirror.openstack.org (which goes to any of the available mirrors/proxies ?) and then do like mirror.openstack.org/path/to/rdo-release.rpm | 17:58 |
| dmsimard | ¯\_(ツ)_/¯ | 17:58 |
| *** oidgar has quit IRC | 17:58 | |
| *** trown|lunch is now known as trown | 17:59 | |
| *** trown is now known as trown|rover | 17:59 | |
| pabelanger | seems like a lot of work for single RPM, especially if just adding .repo file to mirror.centos.org | 18:00 |
| EmilienM | corvus: ack | 18:00 |
| pabelanger | could likely just do that in devstack directly | 18:00 |
| pabelanger | or move rpm to mirror.centos.org | 18:01 |
| *** rkukura has joined #openstack-infra | 18:01 | |
| pabelanger | dmsimard: but, devstack shouldn't hit rdoproject.org directly, when running in gate, but use apache reverse proxy | 18:01 |
| *** slaweq has quit IRC | 18:02 | |
| dmsimard | pabelanger: ganso is not running devstack from the gate | 18:02 |
| *** ralonsoh_ has quit IRC | 18:02 | |
| pabelanger | right, just saying if rdoproject.org is down, devstack in gate will break | 18:02 |
| dmsimard | Depsite the proxy, right.. good point | 18:03 |
| *** SumitNaiksatam has joined #openstack-infra | 18:04 | |
| *** david-lyle has quit IRC | 18:08 | |
| *** weshay|ruck is now known as weshay|ruck|brb | 18:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:10 |
| *** gouthamr has quit IRC | 18:13 | |
| *** hamzy has joined #openstack-infra | 18:13 | |
| *** jistr|conf is now known as jistr | 18:14 | |
| *** jkilpatr has quit IRC | 18:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Set flag to turn off uploading logs on success https://review.openstack.org/537990 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove tripleo pipelines from zuul https://review.openstack.org/537611 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Disable tripleo-test-cloud-rh1 for nodepool https://review.openstack.org/537991 | 18:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove tripelo-test-cloud-rh1 https://review.openstack.org/537992 | 18:16 |
| clarkb | I know its been a busy day and fungi and I am not really here, but is infracloud properly gone? I'm with the people that can update the donations thank you page so may want to update that if it is gone gone | 18:17 |
| clarkb | pabelanger: ^ | 18:18 |
| *** hamzy has quit IRC | 18:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:18 |
| pabelanger | clarkb: yan, doesn't look like it is coming back | 18:18 |
| pabelanger | clarkb: we'd need to reach out to outside tech to debug more | 18:19 |
| pabelanger | clarkb: and given last email, I think removing it is likely best now | 18:19 |
| clarkb | ++ | 18:19 |
| clarkb | I'll let people know to update the thank you page | 18:19 |
| pabelanger | ++ | 18:19 |
| pabelanger | I'll push up patches to to finalize it | 18:19 |
| clarkb | ok | 18:20 |
| *** gouthamr has joined #openstack-infra | 18:20 | |
| *** hamzy has joined #openstack-infra | 18:21 | |
| *** sambetts is now known as sambetts|afk | 18:22 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove infracloud-chocolate from nodepool https://review.openstack.org/537995 | 18:24 |
| pabelanger | clarkb: ^ | 18:24 |
| *** naichuans_ has quit IRC | 18:27 | |
| *** jkilpatr has joined #openstack-infra | 18:27 | |
| *** agopi_ is now known as agopi | 18:27 | |
| *** weshay|ruck|brb is now known as weshay | 18:27 | |
| *** esberglu has joined #openstack-infra | 18:28 | |
| clarkb | pabelanger: +2, thanks. I think we should try to get broad consensus on that to make sure people are comfortable with it before approving | 18:28 |
| pabelanger | clarkb: agree | 18:28 |
| clarkb | but ya based on past emails I don't expect that 1) we'd get responses to our questions and 2) that its just been turned off to be used elsewhere by other people | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:28 |
| pabelanger | infra-root: When you have time, https://review.openstack.org/537995/ could use some discussion. I believe we are at the point where we want to start removing infracloud from project-config / system-config base on clarkb comments above. It was a fun run while it lasted | 18:30 |
| dmsimard | mordred: you know, while our intentions are probably noble and all.. if this is going to live in zuul jobs, the toggle might as well be "disable all job uploads" rather than selective ? If we want a selective logic, maybe we could look at what we did for ara -- http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/emit-ara-html/tasks/main.yaml | 18:31 |
| Shrews | pabelanger: ++ to removing it | 18:31 |
| dmsimard | mordred: in fact I think it probably /should/ be the same logic as what we're doing with ara -- because then all we need to do is to change the variable value on the base playbook | 18:31 |
| mordred | dmsimard: oh - yah - I agree- copying that logic seems like the right thing | 18:32 |
| *** gouthamr has quit IRC | 18:35 | |
| *** goutham__ has joined #openstack-infra | 18:35 | |
| *** camunoz has quit IRC | 18:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Set flag to turn off uploading logs on success https://review.openstack.org/537990 | 18:39 |
| mordred | pabelanger, dmsimard, corvus ^^ corresponding site-variables patch | 18:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Set flag to turn off uploading logs on success https://review.openstack.org/537990 | 18:40 |
| *** hjensas has joined #openstack-infra | 18:40 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop spamming ansible/ansible https://review.openstack.org/537955 | 18:41 |
| mordred | pabelanger, corvus, clarkb, dmsimard:^^ had to update that with another fix to the lint script | 18:42 |
| *** yamamoto has quit IRC | 18:42 | |
| *** amoralej is now known as amoralej|off | 18:43 | |
| *** jpena is now known as jpena|off | 18:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:46 |
| *** sshnaidm has quit IRC | 18:46 | |
| dhellmann | is there still something wrong with the logs volume? I'm finding jobs that pass and then have no logs. | 18:50 |
| *** lucasagomes is now known as lucas-afk | 18:50 | |
| *** jamesmcarthur has joined #openstack-infra | 18:50 | |
| dhellmann | the list-changes logs are missing from https://review.openstack.org/#/c/537965/ for example | 18:50 |
| dmsimard | dhellmann: yes, we've temporarily disabled successful job uploads to reduce pressure on the little interim storage volume that we have | 18:51 |
| dhellmann | and that finished about 10 minutes ago | 18:51 |
| dhellmann | oh | 18:51 |
| dhellmann | well, that's going to make releases inconvenient | 18:51 |
| dmsimard | dhellmann: failed jobs will upload their logs | 18:51 |
| *** armaan has quit IRC | 18:51 | |
| dhellmann | we rely on a job to produce a report for things that can't be checked automatically | 18:51 |
| dhellmann | is that a global switch or is it possible to toggle it for specific jobs? | 18:51 |
| dmsimard | dhellmann: we're setting up a better alternative right now which will allow us to selectively enable things on a more granular basis | 18:52 |
| dhellmann | ok | 18:52 |
| dmsimard | dhellmann: right now it's a global toggle but https://review.openstack.org/#/q/topic:log-toggle will give us that granularity | 18:52 |
| dhellmann | did I miss an announcement about that change? I mean, I know this is all in flux still | 18:52 |
| dhellmann | ok | 18:52 |
| dmsimard | dhellmann: It was mentioned in the last status notice | 18:52 |
| EmilienM | pabelanger: when you have time, please look https://review.openstack.org/#/c/537633/ thanks | 18:53 |
| dhellmann | oh, I did completely miss that in the scrollback. thanks dmsimard | 18:53 |
| dmsimard | dhellmann: it's easy to miss it with the amount of chatter, no worries. I try to advertise twitter and https://wiki.openstack.org/wiki/Infrastructure_Status as alternatives :D | 18:54 |
| *** jamesmcarthur has quit IRC | 18:55 | |
| dmsimard | dhellmann: I've still got that etherpad opened, we're not forgetting you btw... In what repository is the list-changes job definition ? | 18:55 |
| dhellmann | the job is releases-tox-list-changes so it's probably defined in the openstack/releases repo | 18:56 |
| dhellmann | yeah, it is | 18:56 |
| dmsimard | dhellmann: does the relevant content print out in the console ? or solely to a file that is eventually logged ? | 18:56 |
| dhellmann | we would want to save openstack-tox-validate too because that sometimes emits warnings | 18:56 |
| *** yamahata has joined #openstack-infra | 18:56 | |
| dhellmann | dmsimard : both. There's a file under the tox logs directory called list-changes-results.log with the useful output | 18:57 |
| dhellmann | that is less to filter through than the full console output | 18:57 |
| mordred | dmsimard: so in this particular case we'd want to set zuul_site_upload_logs in the base job instead of site-variables because we have a few jobs that need to override it | 18:57 |
| dmsimard | mordred: yeah, not unlike what we're doing with the ara variable | 18:58 |
| *** jamesmcarthur has joined #openstack-infra | 18:58 | |
| dmsimard | I mean, it allows people to shoot themselves in the foot but... | 18:58 |
| *** myoung is now known as myoung|biab | 18:58 | |
| dmsimard | mordred: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n47 | 18:59 |
| *** psachin has quit IRC | 19:02 | |
| *** snapiri- has quit IRC | 19:02 | |
| *** jamesmcarthur has quit IRC | 19:02 | |
| *** camunoz has joined #openstack-infra | 19:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Set flag to turn off uploading logs on success https://review.openstack.org/537990 | 19:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add setting for disabling log uploads to base-test https://review.openstack.org/538002 | 19:04 |
| dmsimard | mordred: comment on https://review.openstack.org/#/c/537990/ | 19:07 |
| dmsimard | mordred: oh nevermind it's two different patches | 19:07 |
| *** david-lyle has joined #openstack-infra | 19:09 | |
| *** david-lyle has quit IRC | 19:09 | |
| *** yamahata has quit IRC | 19:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: DNM Testing that log disabling works https://review.openstack.org/538007 | 19:10 |
| dmsimard | mordred: I'm not sure that's going to work until the project-config stuff has landed | 19:10 |
| mordred | corvus, tobiash, dmsimard: ^^ there is an o-z-j patch that we can use to test the log toggling | 19:11 |
| mordred | yah. we need to land the patch to toggle off logs in base-test first (which should be safe to land) | 19:11 |
| *** erlon has quit IRC | 19:11 | |
| mordred | then we can recheck the o-z-j patch and verify the change in behavior (before we land the project-config patch, both fake jobs should upload logs, after we land the patch only one should) | 19:12 |
| *** apetrich has joined #openstack-infra | 19:12 | |
| mordred | and if we're happy with that we can land the project-config patch to base and then the zuul-jobs patch | 19:13 |
| *** kjackal has joined #openstack-infra | 19:13 | |
| mordred | corvus, pabelanger, dmsimard: ^^ that sound like a decent plan? | 19:13 |
| dmsimard | mordred: wfm | 19:13 |
| corvus | mordred: yep. though you'll also need to revise your ozj patch; zuul wants a playbook. | 19:16 |
| *** SumitNaiksatam has quit IRC | 19:18 | |
| *** slaweq has joined #openstack-infra | 19:19 | |
| mordred | corvus: oh right. piddle | 19:19 |
| *** stevebaker has joined #openstack-infra | 19:20 | |
| *** myoung|biab is now known as myoung | 19:20 | |
| corvus | dmsimard: you want to +3 538002 | 19:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: DNM Testing that log disabling works https://review.openstack.org/538007 | 19:21 |
| dmsimard | corvus: done | 19:22 |
| mordred | corvus: there- also made them no-node jobs so we don't have to take node capacity (or wait for it) | 19:22 |
| dmsimard | as if it wasn't enough I'm simultaneously dealing with RDO infrastructure outages so please ping me if I miss anything -- I'm keeping an eye on the FSCK from time to time | 19:22 |
| dmsimard | We're hovering around 55% storage right now, the log maintenance script doesn't seem to be deleting anything just yet | 19:23 |
| mordred | dmsimard: ++ | 19:23 |
| corvus | it's set to 240m | 19:24 |
| *** david-lyle has joined #openstack-infra | 19:26 | |
| *** jamesmcarthur has joined #openstack-infra | 19:31 | |
| *** stevebaker has quit IRC | 19:31 | |
| *** kjackal has quit IRC | 19:31 | |
| *** stevebaker has joined #openstack-infra | 19:32 | |
| *** myoung is now known as myoung|cheesebur | 19:34 | |
| *** myoung|cheesebur is now known as myoung|food | 19:34 | |
| *** slaweq has quit IRC | 19:41 | |
| *** yamamoto has joined #openstack-infra | 19:43 | |
| *** jamesmcarthur has quit IRC | 19:44 | |
| *** harlowja has joined #openstack-infra | 19:44 | |
| *** jamesmcarthur has joined #openstack-infra | 19:45 | |
| *** onovy has quit IRC | 19:45 | |
| *** peterlisak has quit IRC | 19:45 | |
| *** e0ne has joined #openstack-infra | 19:49 | |
| *** peterlisak has joined #openstack-infra | 19:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add zuul-website project to Zuul https://review.openstack.org/538013 | 19:51 |
| *** yamamoto has quit IRC | 19:55 | |
| efried | Pursuant to http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-01-25.log.html#t2018-01-25T16:34:22 -- how many times could this keep happening? Eventually this patch has gotta bubble to the top of the queue, nah? | 19:57 |
| *** esberglu has quit IRC | 19:57 | |
| *** onovy has joined #openstack-infra | 19:58 | |
| *** goutham__ has quit IRC | 19:58 | |
| *** larainema has quit IRC | 19:59 | |
| *** myoung|food is now known as myoung | 20:02 | |
| mordred | efried: it could, in the worst-possible-case scenario, happen once for every patch ahead of it in the queue | 20:03 |
| efried | mordred But those guys will get kicked to the back whenever they fail, so eventually mine comes to the front and it'll finish (one way or the other) right? | 20:04 |
| mordred | yes | 20:04 |
| efried | okay, that's something. | 20:04 |
| *** agopi is now known as agopi|out | 20:04 | |
| mordred | efried: process ultimately degrades to what things would be like if we ran the patches in the gate one at a time waitingfor each to finish before starting the next one | 20:04 |
| mordred | efried: most of the time it's able to do things *much* more parallel than that, obviously | 20:04 |
| efried | okay. | 20:05 |
| efried | mordred And obviously this is the best possible week for it to "degrade" :) | 20:05 |
| efried | Not that there's ever a *good* time... | 20:05 |
| *** esberglu has joined #openstack-infra | 20:06 | |
| dhellmann | do we think it's safe to continue tagging releases? or should I hold off for a while longer? | 20:06 |
| mordred | dhellmann: I'd hold off - we've got patches in flight to provide you the ability to re-enable the publication of success logs for release jobs | 20:08 |
| dhellmann | ok | 20:08 |
| *** e0ne has quit IRC | 20:08 | |
| *** agopi|out has quit IRC | 20:08 | |
| mordred | dhellmann: https://review.openstack.org/#/q/topic:log-toggle fwiw | 20:08 |
| *** Goneri has quit IRC | 20:09 | |
| mordred | dhellmann: plan is to land https://review.openstack.org/538002, then use https://review.openstack.org/538007 to verify that it works, then land https://review.openstack.org/537990 to turn on the flag globally and https://review.openstack.org/537986 to switch to the flag-based impl | 20:10 |
| dhellmann | makes sense | 20:11 |
| mordred | dhellmann: you could probably go ahead and submit a patch to releases repo to add vars:\n zuul_site_upload_logs: true to your jobs where you need success logs | 20:11 |
| mordred | dhellmann: as it won't have any impact until the zuul-jobs patch lands, but you can be ready to rock and roll as soon as it does | 20:11 |
| dhellmann | ok | 20:11 |
| *** jamesmcarthur has quit IRC | 20:12 | |
| dhellmann | where do I set that? in the job definition? or where the job is associated with the repo? | 20:12 |
| mordred | dhellmann: either place will work | 20:13 |
| dhellmann | ok | 20:13 |
| dhellmann | mordred : like this? https://review.openstack.org/538019 | 20:14 |
| *** slaweq has joined #openstack-infra | 20:14 | |
| mordred | dhellmann: yes - that's perfect | 20:14 |
| corvus | mordred: i think we should gate-enqueue 538002 | 20:14 |
| mordred | corvus: agree | 20:15 |
| corvus | i will do so | 20:15 |
| mordred | corvus: thanks! | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| dhellmann | mordred : thanks | 20:15 |
| rajinir | Is this a known issue? src/pcremodule.c:32:18: fatal error: pcre.h: No such file or directory | 20:16 |
| mordred | corvus: I'm a little confused as to why we don't have results for 538007 yet - it doens't have ... UGH | 20:16 |
| mordred | corvus: nevermind. it has build-openstack-sphinx-docs defined in project-config | 20:16 |
| *** chason has quit IRC | 20:17 | |
| mordred | corvus: maybe I should submit the same patch to a different repo that doesn't have any project-config defined jobs? | 20:17 |
| corvus | mordred: ya... | 20:18 |
| rajinir | @all, anyone working in the Third Party pcre failures? Its affecting many | 20:18 |
| mordred | corvus: openstack-infra/zone-zuul-ci.org | 20:18 |
| *** apetrich has quit IRC | 20:18 | |
| mordred | corvus: has no project-config jobs | 20:18 |
| corvus | mordred: heh, wfm. | 20:18 |
| *** Jeffrey4l has quit IRC | 20:19 | |
| corvus | mordred: 538007 enqueued in gate | 20:19 |
| corvus | mordred: er, not that one, the other one. 538002 | 20:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Stop spamming ansible/ansible https://review.openstack.org/537955 | 20:20 |
| corvus | mordred: not urgent, but when you have a sec, could you review https://review.openstack.org/538013 just to get that going in the background? | 20:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zone-zuul-ci.org master: DNM Testing that log disabling works https://review.openstack.org/538020 | 20:21 |
| *** jkilpatr has quit IRC | 20:22 | |
| *** flwang has quit IRC | 20:23 | |
| mordred | corvus: days like today I wish we had support for cancelling builds when a patch is abandoned | 20:23 |
| efried | or when someone -Ws or -2s it | 20:23 |
| corvus | mordred: yeah. shouldn't be hard. | 20:23 |
| *** chason has joined #openstack-infra | 20:23 | |
| corvus | efried: that's no reason not to run tests :) | 20:23 |
| *** Jeffrey4l has joined #openstack-infra | 20:24 | |
| AJaeger | pabelanger: could you update grafana as well for tripleo, please? See https://review.openstack.org/#/c/537611/ | 20:24 |
| corvus | (in fact, ensuring patches work before asking people to review them is a big part of why the system exists) | 20:24 |
| rosmaita | could someone take a look at https://review.openstack.org/#/c/536733/ ? it's been sitting for 5 hours, doesn't seem to be getting any action | 20:24 |
| efried | corvus How about a special 'cancel check' comment, which we would of course all use as good stewards of the infra when we deemed checking unnecessary. | 20:24 |
| pabelanger | AJaeger: opted to do it in 537992 once we removed it from nodepool | 20:25 |
| corvus | rosmaita: you can check on the status at http://zuul.openstack.org/ | 20:26 |
| rosmaita | corvus yes, but there doesn't seem to be any info there other than it's been sitting 5hr 15min | 20:26 |
| corvus | rosmaita: if you mouse over the grey dot, it'll tell you why no jobs are running | 20:27 |
| efried | rosmaita Only 5:15? | 20:27 |
| efried | rosmaita Let me know when you get to 17:50 (536545,1) | 20:28 |
| mordred | corvus: job-that-does-nothing-with-no-logs and job-that-does-nothing-with-logs are running \o/ | 20:28 |
| *** slaweq has quit IRC | 20:28 | |
| rosmaita | efried you have my sympathies | 20:28 |
| efried | :) | 20:28 |
| mordred | not sure I've ever been excited about jobs that don't do anything | 20:28 |
| AJaeger | pabelanger: ok, thanks | 20:29 |
| *** guhcampos has joined #openstack-infra | 20:30 | |
| *** guhcampos has left #openstack-infra | 20:31 | |
| mordred | corvus: bah. the base-test approach to verifying this works did not work - because, of course, log publication is in the base job and therefore does not use the version of the role from the depends-on on zuul-jobs | 20:32 |
| mordred | corvus: so, what we have done is verified that the system for preventing speculative execution in a trusted context is alive and working properly | 20:33 |
| corvus | mordred: gimme a sec, that confuses me | 20:33 |
| *** dprince has quit IRC | 20:33 | |
| corvus | mordred: yeah, base-test needs to land, then we recheck 538020 | 20:33 |
| mordred | corvus: no - that still won't do it - because the functionality we're looking to test is in the role in zuul-jobs | 20:34 |
| corvus | mordred: oh, that hasn't landed. yes, i'm with you now. | 20:34 |
| corvus | mordred: the options are cowboy that in, or make a test role. | 20:35 |
| mordred | corvus: we'd need to land a copy of the role named somethig else, and then update base-test to use that copy | 20:35 |
| mordred | yah | 20:35 |
| rosmaita | corvus is there any way i can get the glance jobs out of the integrated queue without patching project-config/projects/yaml? | 20:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Add setting for disabling log uploads to base-test https://review.openstack.org/538002 | 20:35 |
| rosmaita | corvus also, i did not know that about the grey dot ... guess i need to RTFM and learn the interface better | 20:36 |
| corvus | mordred: i think given the urgency, we can continue to promote those changes in gate. | 20:36 |
| mordred | corvus: given gate length, I think I'm more inclined to just landing the variable setting and then the zuul-jobs change | 20:36 |
| *** e0ne has joined #openstack-infra | 20:36 | |
| *** flwang has joined #openstack-infra | 20:36 | |
| *** jtomasek has joined #openstack-infra | 20:36 | |
| corvus | mordred: i hear that. gate length is an equally valid reason to be cautious :) | 20:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Switch to python3 for tox -enodepool https://review.openstack.org/538023 | 20:36 |
| *** slaweq has joined #openstack-infra | 20:37 | |
| corvus | rosmaita: well, it shares a queue for a reason -- it interacts with the other services. | 20:37 |
| pabelanger | AJaeger: should fix tox -enodepool errors^ | 20:37 |
| *** jtomasek has quit IRC | 20:38 | |
| rosmaita | corvus i know, i'm just desperate ... i will shut up now and let you work ... thanks! | 20:38 |
| AJaeger | pabelanger: Ah! Great | 20:38 |
| AJaeger | pabelanger: You get my vote of confidence that this works ;) | 20:39 |
| pabelanger | pressure is on now | 20:40 |
| prometheanfire | can we recheck yet? | 20:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove tripleo pipelines from zuul https://review.openstack.org/537611 | 20:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Disable tripleo-test-cloud-rh1 for nodepool https://review.openstack.org/537991 | 20:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove tripelo-test-cloud-rh1 https://review.openstack.org/537992 | 20:45 |
| corvus | prometheanfire: yes, see entry at top of https://wiki.openstack.org/wiki/Infrastructure_Status | 20:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add modified upload-logs as test-upload-logs to base-test https://review.openstack.org/538025 | 20:46 |
| mordred | corvus: ^^ | 20:46 |
| prometheanfire | are we not sending notificatoins to channels? | 20:46 |
| corvus | prometheanfire: that was sent | 20:46 |
| prometheanfire | ah, k | 20:47 |
| mordred | corvus: roles/test-upload-logs was copied via cp -a ... so should be an exact duplicate of the role in zuul-jobs | 20:47 |
| prometheanfire | missed it | 20:47 |
| *** rlandy is now known as rlandy|brb | 20:47 | |
| corvus | mordred: lgtm. who else is reviewing these? | 20:48 |
| mordred | pabelanger, dmsimard: ^^ could you look at https://review.openstack.org/538025 real quick? | 20:48 |
| pabelanger | okay, I haven't had anything to eat yet, stepping away for a bit | 20:48 |
| mordred | pabelanger, dmsimard: tl;dr - thepreviousbase-test patch wouldn't work because speculative protection. this is a copy of the new version of the role in zuul-jobs so that we can land it and then recheck the test patch | 20:49 |
| corvus | mordred: i'll enqueue it now | 20:50 |
| *** jamesmcarthur has joined #openstack-infra | 20:51 | |
| AJaeger | corvus: also enque https://review.openstack.org/538023, please | 20:51 |
| mordred | corvus: cool. | 20:51 |
| corvus | AJaeger: what's the urgency there? | 20:51 |
| AJaeger | corvus: it fixes nodepool testing - in case we need to merge a chnage that triggers nodepool testing.. | 20:52 |
| *** andreas_s has joined #openstack-infra | 20:52 | |
| AJaeger | corvus: has lower priority than mordred's change for sure | 20:52 |
| corvus | AJaeger: i'd like to be conservative about things we direct-enqueue. | 20:52 |
| AJaeger | corvus: ok - then let's wait | 20:53 |
| corvus | the log changes are in service of fixing every zuulv3 installation in the world, including ours. :) | 20:53 |
| *** myoung is now known as myoung|pto | 20:53 | |
| AJaeger | ;) | 20:54 |
| *** e0ne has quit IRC | 20:54 | |
| *** jamesmcarthur has quit IRC | 20:55 | |
| dmsimard | mordred: was away, looking | 20:56 |
| dmsimard | mordred: that looks like the wrong version of the role | 20:56 |
| *** andreas_s has quit IRC | 20:56 | |
| corvus | dmsimard: yes i agree | 20:57 |
| *** agopi|out has joined #openstack-infra | 20:57 | |
| dmsimard | mordred, corvus: commented on https://review.openstack.org/538025 | 20:57 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config master: translation-jobs for neutron-vpnaas-dashboard https://review.openstack.org/537970 | 20:57 |
| dmsimard | corvus: -W prevents the patch from merging despite it being already in the gate queue right ? | 20:58 |
| corvus | dmsimard: yes, as does C-2 | 20:58 |
| dmsimard | corvus: ok, I'll remove -W | 20:58 |
| AJaeger | dmsimard: you can leave it in - a new patchset will reset it... | 20:58 |
| *** tosky has joined #openstack-infra | 20:58 | |
| dmsimard | AJaeger: sure, but it's better to rely on -2 | 20:59 |
| dmsimard | (in case the new patch doesn't land in time) | 20:59 |
| corvus | dmsimard: i think AJaeger is saying that -W is the best approach, since it will stop it from landing, and also be reset. | 20:59 |
| AJaeger | dmsimard: yes, I mean: A -W will be reset and no manual action is needed. Corvus needs to remove his -2 eventually... | 21:00 |
| dmsimard | oh, right -2's aren't removed on new patchset | 21:00 |
| *** sshnaidm has joined #openstack-infra | 21:00 | |
| *** kgiusti has left #openstack-infra | 21:01 | |
| *** Jeffrey4l has quit IRC | 21:02 | |
| *** chason has quit IRC | 21:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add modified upload-logs as test-upload-logs to base-test https://review.openstack.org/538025 | 21:04 |
| openstackgerrit | Mark Hamzy proposed openstack/diskimage-builder master: fix the grub root label when an XFS disk label is truncated https://review.openstack.org/532279 | 21:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add public key hosting to SSL site https://review.openstack.org/538031 | 21:04 |
| mordred | turns out copying a directory IS hard | 21:04 |
| mordred | corvus, AJaeger, dmsimard: patch updated | 21:05 |
| dmsimard | corvus: during the all but 5 minutes I used to ingest some nutrients earlier, I was thinking that the zuul executor "zoning" (is that what we'll call it?) could potentially allow us to shard executors by region/cloud -- and by doing so we could probably shard the logs into a logserver per region | 21:06 |
| dmsimard | So that we don't have this huge spof impacting all regions | 21:06 |
| mordred | dmsimard: yah - SO ... I'm working up a writeup write now on shifting to swift ... | 21:07 |
| dmsimard | corvus: I know the use case of the executor zoning was mostly to hop around network boundaries but I could see this being used to make each region/nodepool provider more independant | 21:07 |
| mordred | dmsimard: which incidentally includes provisions for using per-cloud swift | 21:07 |
| dmsimard | mordred: we're not expecting nodepool providers to carry swift right now though | 21:07 |
| mordred | all of our nodepool providers carry swift currently | 21:07 |
| mordred | the only one that didn't was, amusingly enough, infracloud | 21:08 |
| dmsimard | mordred: also, there's that posix over swift thing we never managed to get right | 21:08 |
| corvus | dmsimard: yes, though executor zones contribute only the ability to reduce internet traffic, which, tbh, is not generally a huge concern. it's not required for such changes. | 21:08 |
| mordred | yah - I've got thoughts on that - but they're document-sized rather than irc sized | 21:08 |
| *** jbadiapa has joined #openstack-infra | 21:08 | |
| mordred | dmsimard: which is to say - I think enough has changed that it's actually not hard now like it was before | 21:09 |
| corvus | mordred: we've been inching toward it for a while :) | 21:09 |
| dmsimard | corvus: well it's not so much about reducing traffic than, say, have (at least) two executors per region with their own logserver so that if a logserver explodes, the other regions are not impacted | 21:09 |
| mordred | dmsimard: I hope to have something for you to read by tomorrow | 21:09 |
| dmsimard | corvus: having the executors in the same region as the logserver has that bandwidth benefit, yes | 21:10 |
| corvus | dmsimard: i'm just saying the region doesn't matter for logservers. it's redundancy you are advocating. they could both be in dfw and it would still work. | 21:10 |
| dmsimard | hmm, so actually this isn't something we could do easily before with zuul v2 -- but with zuul v3's return for log_url, we could technically upload anywhere and it would just work | 21:11 |
| corvus | that's why it's there :) | 21:11 |
| dmsimard | all we could need to do is the plumbing to get the fqdn of the logserver through to the job so it uploads to the right place | 21:12 |
| dmsimard | but having all the executors in dfw makes the whole pull/push thing a bit inefficient, yeah | 21:12 |
| corvus | dmsimard: but there are many other pieces too, which i suspect mordred has included in his writeup, since we've talked about this in depth several times over the years | 21:12 |
| *** rlandy|brb is now known as rlandy | 21:12 | |
| *** chason has joined #openstack-infra | 21:13 | |
| dmsimard | I'm happy to make it happen, anything to prevent our disks from getting full and to avoid these recurring fscks -- it's a good investment | 21:13 |
| corvus | i think this is valuable, but i think all of our time will be best served by waiting for mordred to post his proposal, then we'll have a baseline for discussion and we don't have to retread | 21:13 |
| dmsimard | s/make it/help make it/ | 21:13 |
| dmsimard | sure | 21:13 |
| *** Jeffrey4l has joined #openstack-infra | 21:14 | |
| corvus | yes. also, for planning purposes, i think if we do make a change, we're likely to do it after the v3 release | 21:14 |
| dmsimard | is the zoning scoped for v3 ? or after ? | 21:15 |
| corvus | dmsimard: no one has signed up to implement it, but if they did, it would be post v3 | 21:15 |
| dmsimard | okay | 21:15 |
| mordred | dmsimard: feel like a +A on https://review.openstack.org/#/c/538025/ - and then corvus can promote in gate? | 21:20 |
| *** jkilpatr has joined #openstack-infra | 21:21 | |
| dmsimard | +3 | 21:21 |
| mordred | dmsimard: thanks! | 21:22 |
| mordred | corvus: you in a position to easily promote or should I hop on and do it? | 21:22 |
| corvus | mordred: done | 21:22 |
| mordred | woot | 21:22 |
| *** tpsilva has quit IRC | 21:22 | |
| *** apetrich has joined #openstack-infra | 21:23 | |
| *** sshnaidm is now known as sshnaidm|off | 21:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add zuul-website project to Zuul https://review.openstack.org/538013 | 21:25 |
| *** armaan has joined #openstack-infra | 21:25 | |
| *** pramodrj07 has joined #openstack-infra | 21:29 | |
| *** eharney has quit IRC | 21:31 | |
| *** esberglu has quit IRC | 21:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add zuul-website to #openstack-infra gerritbot https://review.openstack.org/538041 | 21:34 |
| *** stevebaker has quit IRC | 21:36 | |
| dmsimard | dhellmann reported some failed jobs that would need to be manually re-enqueued, is someone able to take care of that ? https://etherpad.openstack.org/p/6GloEgik8b | 21:39 |
| *** jheroux has joined #openstack-infra | 21:40 | |
| corvus | dmsimard: yep, do you need to hand that off? | 21:41 |
| dmsimard | corvus: not super familiar with requeuing things beyond the usual check/gate and I have my hands full with RDO infrastructure exploding, would appreciate if someone else could take it, yes :) | 21:42 |
| corvus | dhellmann, smcginnis: are either of you available to work with me on that ^? | 21:43 |
| *** jheroux has quit IRC | 21:43 | |
| *** hamzy has quit IRC | 21:43 | |
| *** stevebaker has joined #openstack-infra | 21:44 | |
| dhellmann | corvus : o/ | 21:45 |
| *** jamesmcarthur has joined #openstack-infra | 21:45 | |
| corvus | dhellmann: howdy. looking at the first entry, i see an openstack/releases sha on line 6 which corresponds to a commit which instructs us to tag tripleo-heat-templates version 6.2.9 (mentioned in line 9) | 21:46 |
| corvus | dhellmann: what are lines 7-9? | 21:46 |
| dhellmann | corvus : I have approved https://review.openstack.org/#/c/538017/1 as a test to make sure the jobs are working before we start with anything real | 21:46 |
| dhellmann | those lines are the contents of the failure emails indicating which jobs failed | 21:47 |
| dhellmann | some of these things failed in different ways than usual | 21:47 |
| dhellmann | because of the log issue, I don't always have log URLs to give | 21:47 |
| dhellmann | some of the jobs failed on different repos, too | 21:47 |
| corvus | dhellmann: why are instack-undrecloud and puppet-tripleo involved? | 21:47 |
| dhellmann | so line 7 is saying that the tag was applied to instack-undercloud but the jobs that ran failed in some way | 21:48 |
| dhellmann | those are the triplo deliverables that had failures | 21:48 |
| dhellmann | that tripleo release patch had a zillion deliverables in one patch | 21:48 |
| corvus | dhellmann: i think that's the thing i'm missing -- how does that work? | 21:48 |
| dhellmann | as a convenience we don't require a patch in openstack/releases to be limited to tagging one repo | 21:49 |
| corvus | dhellmann: oh i see now! | 21:49 |
| dhellmann | the patch can have N deliverable files and each deliverable can have M repos involved getting the same tag | 21:49 |
| dhellmann | the tag-releases job works out what tags need to be applied where and submits the tag | 21:49 |
| corvus | i did git show, but it just happened to line up in my terminal in a way where i didn't notice it tagged 3 things :) | 21:49 |
| dhellmann | then the regular per-repo process takes over from there | 21:49 |
| *** jamesmcarthur has quit IRC | 21:49 | |
| dhellmann | aha | 21:49 |
| dhellmann | oh, that one only had 3 things, another tripleo patch has many more but we haven't approved that one | 21:50 |
| dhellmann | sorry, I got those mixed up | 21:50 |
| openstackgerrit | Merged openstack-infra/project-config master: Add modified upload-logs as test-upload-logs to base-test https://review.openstack.org/538025 | 21:50 |
| dmsimard | mordred: ^ | 21:50 |
| corvus | dmsimard: okay, so if i re-enqueue 323... will the tag-releases job be idempotent and note that 6.instack-undercloud is already tagged, but the other 2 aren't and so add tags only to those? | 21:51 |
| dhellmann | corvus : so for the tripleo patch we need to reenqueue the jobs that should be run when a tag is applied to openstack/instack-undercloud and then we can re-run the job that happens when a patch merges into openstack/releases to retrigger the other 2 releases (since those tags were not applied) | 21:52 |
| corvus | (er, sorry about the stray "6." in that) | 21:52 |
| dhellmann | yes, that's right | 21:52 |
| corvus | dhellmann: the order doesn't matter there right, we can do instack-undercloud and openstack/releases in either order, yeah? | 21:53 |
| dhellmann | yeah, that should be fine | 21:53 |
| dmsimard | corvus: I'm not very familiar with the tag-releases job, was that for dhellmann ? | 21:53 |
| corvus | dmsimard: yep. dtab. | 21:53 |
| dmsimard | k | 21:53 |
| corvus | dhellmann: and finally -- do we need to wait for the successful logs changes for this, or is that for a different set of jobs? | 21:53 |
| dhellmann | I'm not 100% sure. I *need* logs for list-changes jobs but that won't be part of any of these. If we'll have logs for failures of these jobs, it should be ok to start them now. | 21:54 |
| dhellmann | list-changes runs in the check queue for openstack/releases | 21:55 |
| corvus | dhellmann: yep, we should always have logs for failures, unless logs.o.o blows up. okay, i'll start preparing commands for this; should take just a couple mins, i'll ping you before i run them. | 21:55 |
| dhellmann | corvus : sounds good. let me know if my notes for the others are confusing. I tried to collect the shas for the git tags in each repo for those. | 21:56 |
| mordred | corvus: I have rechecked the test patch | 21:56 |
| *** linkmark has quit IRC | 22:00 | |
| *** yamahata has joined #openstack-infra | 22:02 | |
| corvus | dhellmann: okay, do those two commands look reasonable to you? | 22:02 |
| dhellmann | corvus : I've never peeked behind this particular curtain with fungi before, so a very qualified yes | 22:03 |
| corvus | dhellmann: oh, should i also enqueue instack-undercloud into the 'tag' pipeline? i guess it would have matched that too.... | 22:03 |
| dhellmann | ah, yeah | 22:03 |
| dhellmann | oh, hrm | 22:04 |
| dhellmann | corvus : there is a 6.1.4 release for instack-undercloud on pypi | 22:04 |
| dhellmann | so maybe that release did work | 22:04 |
| corvus | oh neat | 22:04 |
| dhellmann | yeah, sorry, I should have checked that before | 22:04 |
| *** slaweq has quit IRC | 22:04 | |
| *** slaweq has joined #openstack-infra | 22:05 | |
| corvus | dhellmann: i bet we can use the dashboard to see if any instack release jobs failed | 22:05 |
| *** yamamoto has joined #openstack-infra | 22:05 | |
| dhellmann | the release notes look out of date | 22:05 |
| dhellmann | dashboard? | 22:05 |
| corvus | dhellmann: http://zuul.openstack.org/builds.html | 22:06 |
| corvus | dhellmann: unfortunately, it seems i can't deep link | 22:06 |
| corvus | dhellmann: but put 'release' in pipeline and 'openstack/instack-undercloud' in project and hit refresh | 22:07 |
| corvus | dhellmann: oh good, the 'obvious' way to deep link works: http://zuul.openstack.org/builds.html?pipeline=release&project=openstack%2Finstack-undercloud | 22:08 |
| corvus | dhellmann: and http://zuul.openstack.org/builds.html?pipeline=tag&project=openstack%2Finstack-undercloud | 22:09 |
| *** trown|rover is now known as trown|outtypewww | 22:09 | |
| corvus | that last one looks like the releasenotes job for that tag succeeded. | 22:09 |
| mordred | corvus, dmsimard: https://review.openstack.org/#/c/538020/ appropriately has logs and no-logs | 22:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to python3 for tox -enodepool https://review.openstack.org/538023 | 22:11 |
| mordred | corvus, dmsimard: so I think we can now land https://review.openstack.org/#/c/537990/ and then https://review.openstack.org/#/c/537986/ | 22:12 |
| dhellmann | corvus : ah, so I'm not seeing any failures there | 22:12 |
| corvus | mordred: do you want to address dmsimard's comment, or just ignore that due to its small contribution to log usage? | 22:13 |
| corvus | mordred: on 537990 | 22:13 |
| dhellmann | though the release notes are out of date so I wonder if that job was skipped instead of failing | 22:13 |
| dhellmann | we probably don't record skips | 22:13 |
| *** slaweq_ has joined #openstack-infra | 22:13 | |
| corvus | dhellmann: tbh, i'm not sure about skips there, but we apparently recorded success... | 22:14 |
| dhellmann | well, maybe there's a problem with the release notes configuration then | 22:14 |
| dhellmann | I'll figure that out | 22:14 |
| mordred | corvus: yah - I wanted to just ignore it - I also wasn't sure what impact it might have had on the tests | 22:14 |
| *** hrubi has quit IRC | 22:14 | |
| corvus | dhellmann: okay, so we're down to just the one equeue-ref for releases then, right? | 22:15 |
| mordred | which is probably nothing - but it didn't seem worth iterating on | 22:15 |
| dhellmann | corvus : for the tripleo things, yes | 22:15 |
| corvus | mordred: +2 from me | 22:15 |
| corvus | dhellmann: ready for me to run that now? | 22:15 |
| *** slaweq_ has quit IRC | 22:15 | |
| *** slaweq_ has joined #openstack-infra | 22:16 | |
| dhellmann | corvus : let me look at that release-test job first, just a sec | 22:16 |
| dhellmann | well, I'm not finding it in the dashboard | 22:17 |
| corvus | me neither | 22:17 |
| dhellmann | oh, it might not have been triggered yet | 22:18 |
| dhellmann | I see a few things in release-post | 22:18 |
| *** hrubi has joined #openstack-infra | 22:18 | |
| *** slaweq_ has quit IRC | 22:18 | |
| dhellmann | http://zuul.openstack.org/stream.html?uuid=04858d5fed8d46b3867270af47fa144a&logfile=console.log | 22:18 |
| dhellmann | that's the job that will start the job I want to watch | 22:18 |
| *** slaweq has quit IRC | 22:19 | |
| *** slaweq_ has joined #openstack-infra | 22:19 | |
| dhellmann | these jobs are hard to talk about | 22:19 |
| *** slaweq has joined #openstack-infra | 22:19 | |
| dhellmann | there we go | 22:20 |
| corvus | dhellmann: this is like when your gate job gates your job in the gate. | 22:20 |
| dhellmann | in a hole on the bottom of the sea | 22:21 |
| *** rossella_s has quit IRC | 22:22 | |
| *** slaweq_ has quit IRC | 22:23 | |
| dhellmann | it never seems to take this long when I'm not watching :-) | 22:24 |
| *** vaidy has quit IRC | 22:25 | |
| *** isviridov_away has quit IRC | 22:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set flag to turn off uploading logs on success https://review.openstack.org/537990 | 22:25 |
| *** rossella_s has joined #openstack-infra | 22:27 | |
| *** Goneri has joined #openstack-infra | 22:29 | |
| *** bobh has quit IRC | 22:30 | |
| dhellmann | corvus : ok, that job seems to have worked so we can rerun the other | 22:31 |
| corvus | dhellmann: okay, i'll execute the command on line 12 -- openstack/releases 323c38... right? | 22:32 |
| dhellmann | corvus : yes | 22:32 |
| corvus | dhellmann: done | 22:32 |
| dhellmann | ok | 22:33 |
| dhellmann | for the octavia dashboard thing on line 14 we need the constraint update job at least | 22:33 |
| dhellmann | I can do the announce one myself | 22:33 |
| pabelanger | infra-root: https://etherpad.openstack.org/p/2tAC4qI2eR is the current status of logs.o.o, if you have a moment to review please | 22:33 |
| *** rossella_s has quit IRC | 22:33 | |
| *** rossella_s has joined #openstack-infra | 22:35 | |
| corvus | dhellmann: iiuc, as long as the pypi upload didn't happen, we should be able to re-enqueue the ref for octavia-dashboard and get the whole set, yeah? | 22:37 |
| dhellmann | yes | 22:38 |
| dhellmann | corvus : stand by it looks like that did upload | 22:38 |
| dhellmann | meh, without the logs I should have checked all of this already | 22:39 |
| dhellmann | well now I don't know what actually failed for that job | 22:40 |
| dhellmann | maybe just uploading the logs? | 22:40 |
| dhellmann | all 4 deliverables appear on pypi just fine | 22:40 |
| dhellmann | oh, the constraints updates | 22:40 |
| dhellmann | well, these are all pre-releases so there wouldn't be a constraints update | 22:41 |
| corvus | dhellmann: hrm, it does run a propose-update-constraints job anyway | 22:42 |
| dhellmann | maybe that job ignores pre-releases. I forget now. | 22:42 |
| dhellmann | I thought we had it set up to not do that in different queues. | 22:42 |
| dhellmann | none of those things appear in the constraints list so it would be a no-op | 22:43 |
| corvus | (we need the ref to be included in the zuul dashboard table; it's really hard to deal with the tag pipelines without it) | 22:44 |
| dhellmann | I think I can do the rest of these by hands | 22:46 |
| dhellmann | hand | 22:46 |
| dhellmann | it's just announcements and constraint updates | 22:47 |
| *** jamesmcarthur has joined #openstack-infra | 22:47 | |
| corvus | dhellmann: oh wow, that's much less work than i thought i signed up for :) | 22:50 |
| dhellmann | yeah, I should have looked at pypi earlier | 22:51 |
| corvus | dhellmann: i'm assuming you have all the failures here already? http://zuul.openstack.org/builds.html?pipeline=release-post&project=openstack%2Freleases | 22:51 |
| dhellmann | I took it on faith that if the job reported failure it actually failed | 22:51 |
| *** jamesmcarthur has quit IRC | 22:51 | |
| dhellmann | I've been using http://lists.openstack.org/pipermail/release-job-failures/2018-January/thread.html as the list | 22:51 |
| corvus | dhellmann: the tripleo one from earlier is the bottom two failures | 22:52 |
| corvus | dhellmann: but there's 4 other failed tag-releases jobs after it | 22:52 |
| *** wolverineav has quit IRC | 22:53 | |
| dhellmann | corvus : yes, that list matches the list in the etherpad I built from the email failure reports | 22:53 |
| corvus | dhellmann: oh i see at the bottom there | 22:54 |
| corvus | dhellmann: so do we need to re-enqueue those as well? | 22:54 |
| dhellmann | those are just to update releases.o.o and we'll get that when we start approving other releases | 22:54 |
| corvus | pabelanger: could just reformat the partition | 22:54 |
| *** jcoufal has quit IRC | 22:54 | |
| dhellmann | so no | 22:54 |
| corvus | dhellmann: ok | 22:54 |
| mriedem | AJaeger: is there someone within suse that would care about this? https://bugs.launchpad.net/nova/+bug/1741329 | 22:55 |
| openstack | Launchpad bug 1741329 in OpenStack Compute (nova) "Install and configure controller node for openSUSE and SUSE Linux Enterprise in nova" [Undecided,New] | 22:55 |
| *** pcrews has joined #openstack-infra | 22:56 | |
| *** maharg101 has joined #openstack-infra | 22:58 | |
| dmsimard | mordred: we didn't consider the logstash and openstack-health post playbooks when disabling logs | 22:59 |
| dmsimard | the logstash workers are hitting 404s | 22:59 |
| pabelanger | corvus: if we run mkfs.ext4 over /dev/mapper/main-logs, that should be good to start fresh? Or do we have to go though the process of rebuilding the volume? | 22:59 |
| dmsimard | pabelanger: you mean you want to wipe the whole volume ? | 22:59 |
| corvus | pabelanger: mkfs is sufficient; it's not a raid volume, so there isn't really anything else to do | 22:59 |
| corvus | dmsimard: it was my suggestion from above | 22:59 |
| dhellmann | corvus : thanks for your help this afternoon! | 23:00 |
| *** armax has quit IRC | 23:00 | |
| *** camunoz has quit IRC | 23:00 | |
| corvus | dhellmann: you're welcome! we learned some things :) | 23:00 |
| dhellmann | yes! that dashboard looks like it will be useful when I have time to figure out how to use it | 23:00 |
| dmsimard | corvus: clarkb suggested trying to dirty mount the volume which we admittedly didn't try | 23:00 |
| corvus | dmsimard: that may just push the problem into an arbitrary point in the future | 23:01 |
| pabelanger | corvus: okay, clarkb also suggest we just bring mount back online, and if data is corrupt, it is only 4 week retention. But wasn't sure how to ensure that without finishing fsck | 23:01 |
| corvus | when ext4 belatedly realizes it's corrupt and switches it to read-only | 23:01 |
| *** tosky has quit IRC | 23:02 | |
| *** maharg101 has quit IRC | 23:02 | |
| *** rossella_s has quit IRC | 23:03 | |
| *** rossella_s has joined #openstack-infra | 23:06 | |
| *** slaweq has quit IRC | 23:06 | |
| clarkb | ah ok that wasnt what I was sure of | 23:06 |
| clarkb | in that case maybe not great idea | 23:06 |
| dhellmann | corvus : the vitrage team needs https://review.openstack.org/#/c/537781/ in order to have the release jobs they need to tag a release, can you take a look when you have a minute? | 23:07 |
| dmsimard | If we're going to wipe 4 weeks of logs we might as well spin off a new node somewhere we don't need to stripe 13 volumes | 23:07 |
| pabelanger | not sure I follow | 23:08 |
| *** pcrews has quit IRC | 23:08 | |
| *** yamahata has quit IRC | 23:08 | |
| *** stevebaker has quit IRC | 23:09 | |
| dmsimard | pabelanger: the mkfs.ext4 suggestion | 23:09 |
| *** Pramod has joined #openstack-infra | 23:09 | |
| pabelanger | right, I'm not following spin off a new node comment | 23:09 |
| corvus | clarkb: i'm not sure about that behavior, but i would not be surprised. | 23:09 |
| dmsimard | pabelanger: if we're going to start the server from scratch, we might as well do it on a server where we don't need to stripe 13 different volumes together | 23:10 |
| corvus | dmsimard: i advocate losing the data and reformatting only to get it back in service in a timely manner. i'm not sure rebuild the system from scratch meets those goals. | 23:10 |
| *** Aibot has joined #openstack-infra | 23:10 | |
| *** Aibot has quit IRC | 23:10 | |
| *** Pramod has quit IRC | 23:10 | |
| pabelanger | right, if we reformat, I'd not like to do more then that. | 23:10 |
| *** pramodrj07 has quit IRC | 23:10 | |
| *** pramodrj07 has joined #openstack-infra | 23:11 | |
| dmsimard | what is most expensive, keeping the gate on hold for ~8hrs and do the fsck with the usual way of doing it or reformatting ? | 23:11 |
| corvus | dhellmann: done | 23:12 |
| dhellmann | corvus : thanks again! | 23:12 |
| dmsimard | pabelanger's suggestion is worth considering as well to lower the capacity | 23:12 |
| dmsimard | so we don't overwhelm the log server while the fsck is running | 23:13 |
| pabelanger | clarkb: what are your thoughts on reformat? expire 4 weeks of data at once? | 23:13 |
| corvus | i think it would be better to stop all log uploads than to lower capacity. | 23:13 |
| dmsimard | so let jobs run but without uploads ? | 23:14 |
| corvus | (and, ftr, i don't think it's a good idea to stop all log uploads) | 23:14 |
| corvus | (just that lowering capacity is even less desirable) | 23:14 |
| *** andreww has quit IRC | 23:14 | |
| dmsimard | yeah.. none of the options sound great :( | 23:15 |
| dmsimard | If we keep chugging along, the fsck is slow but will eventually complete I guess | 23:15 |
| *** stevebaker has joined #openstack-infra | 23:16 | |
| dmsimard | Is there a way to get the logstash workers to stop trying an url ? We disabled log uploads but not the post gearman things so we have these 404's http://paste.openstack.org/raw/653519/ | 23:17 |
| dmsimard | Which might not help our current ram utilization | 23:17 |
| corvus | the swap activity is low, even though the usage is growing. it's possible that it may eventually complete without incident. it's a big experiment. | 23:17 |
| corvus | dmsimard: i doubt a 404 uses much ram | 23:18 |
| corvus | apache appears to be responsible for very little ram use | 23:18 |
| pabelanger | dmsimard: what % are we at now? | 23:21 |
| dmsimard | 61% phase 1 | 23:21 |
| *** wolverineav has joined #openstack-infra | 23:21 | |
| pabelanger | okay, I'm going to help to step away for a bit. #dadops, for now I guess we keep on this path and see how it goes. Current logs seem under control, but /opt is up to 10% | 23:22 |
| pabelanger | s/help/have | 23:22 |
| dmsimard | logs are indeed under control and the usage is actually reducing despite the fact that I bumped the timeout | 23:23 |
| dmsimard | it turns out that uploading only failed job logs aren't worth that much ? | 23:24 |
| *** wolverineav has quit IRC | 23:25 | |
| *** rlandy is now known as rlandy|biab | 23:26 | |
| pabelanger | okay, I'll check back shortly and see how we are progressing | 23:26 |
| *** ianychoi has quit IRC | 23:26 | |
| *** ianychoi has joined #openstack-infra | 23:27 | |
| *** edmondsw has quit IRC | 23:27 | |
| *** dave-mccowan has quit IRC | 23:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update vitrage-dashboard python jobs and publish job https://review.openstack.org/537781 | 23:28 |
| mriedem | is the "no logs in successful job runs" thing temporary or permanent? | 23:28 |
| dmsimard | mriedem: temporary | 23:28 |
| mriedem | whew | 23:29 |
| dmsimard | mriedem: it's a compromise to allow the gate to keep going despite our current degraded state | 23:29 |
| *** Goneri has quit IRC | 23:30 | |
| *** armax has joined #openstack-infra | 23:31 | |
| *** claudiub has quit IRC | 23:47 | |
| *** gongysh has joined #openstack-infra | 23:54 | |
| *** pcrews has joined #openstack-infra | 23:55 | |
| *** r-daneel has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!