| openstackgerrit | Merged openstack-infra/project-config master: adds missing required-projects to n8g-(bagpipe|bgppvn) https://review.openstack.org/510435 | 00:00 |
|---|---|---|
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron to the project dependecy list for zuul https://review.openstack.org/510347 | 00:00 |
| *** xarses has joined #openstack-infra | 00:00 | |
| *** xarses has quit IRC | 00:00 | |
| *** caphrim007 has quit IRC | 00:00 | |
| *** andreas_s has quit IRC | 00:01 | |
| *** felipemonteiro has joined #openstack-infra | 00:01 | |
| *** mancdaz has joined #openstack-infra | 00:01 | |
| *** isviridov_away has quit IRC | 00:01 | |
| *** vaidy has quit IRC | 00:01 | |
| pabelanger | done | 00:01 |
| fungi | clarkb: i usually only do that on the entry for the openid | 00:02 |
| *** greghaynes has quit IRC | 00:02 | |
| clarkb | fungi: so WHERE external_id = 'https://something' ; ? | 00:02 |
| *** felipemonteiro_ has joined #openstack-infra | 00:02 | |
| *** greghaynes has joined #openstack-infra | 00:02 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: zuul-cloner-shim: Use st_dev to check for filesystem https://review.openstack.org/511079 | 00:02 |
| pabelanger | clarkb: ya, tripleo should be able to revert and stop downloading directly from cirros.org | 00:02 |
| pabelanger | EmilienM: ^ | 00:02 |
| fungi | clarkb: yes | 00:02 |
| *** xarses has joined #openstack-infra | 00:02 | |
| fungi | clarkb: but even that can probably be skipped if you can get by with just marking the unused account inactive | 00:03 |
| fungi | i tend to reassign the openid in cases where it's possible the same person may be inadvertently switching between openids | 00:03 |
| fungi | but if you disable the account and leave the openid on it, then at least they still get an error trying to log in with it | 00:04 |
| clarkb | fungi: in this case I think it may be a single ubuntu account that had its email addr updated, would openid switching happen in that case? | 00:04 |
| clarkb | but I've asked that the person involved login normally and check which account they end up on so there is a chance we can just disable the extra account | 00:04 |
| fungi | it's hard to tell whether that's the case | 00:04 |
| fungi | because you can have multiple ubuntu sso accounts mapping to a single launchpad account | 00:04 |
| clarkb | there openids are definitely different in this case so I'm guessing it depends on what happens when they attempt to login | 00:05 |
| fungi | if they switch back and forth on the e-mail address they provide to login.ubuntu.com, i believe it goes back and forth between openids even if they both get assigned to one launchpad profile | 00:05 |
| clarkb | basically which account does gerrit give them | 00:05 |
| ianw | clarkb: hmm, how self testing is that? | 00:05 |
| clarkb | ianw: I don't think it is very :) | 00:05 |
| clarkb | ianw: I think if we get it to run tests involving zuul cloner on rax it will self test | 00:06 |
| *** vaidy has joined #openstack-infra | 00:06 | |
| *** isviridov_away has joined #openstack-infra | 00:06 | |
| *** felipemonteiro has quit IRC | 00:06 | |
| ianw | i'll try doing a bit more manual due diligence on it | 00:06 |
| clarkb | ianw: but that may require a depends on as I'm not sure we use z-c on those changes and then you'll have to reschedule until you end up on rax | 00:06 |
| *** Apoorva_ has quit IRC | 00:11 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add unbound role https://review.openstack.org/510734 | 00:11 |
| *** mat128 has quit IRC | 00:12 | |
| *** jascott1 has joined #openstack-infra | 00:15 | |
| *** yamahata has quit IRC | 00:18 | |
| *** dingyichen has joined #openstack-infra | 00:18 | |
| ianw | clarkb: hmm, https://git.openstack.org/cgit/openstack/python-openstackclient/tree/openstackclient/tests/functional/post_test_hook_tips.sh#n49 probably isn't helping | 00:18 |
| mnaser | i was thinking we should totally search the openstack codebase for "jenkins" | 00:19 |
| *** tpsilva has quit IRC | 00:20 | |
| ianw | "sudo -E -H -u jenkins tox -efunctional" seems to be a common trope | 00:20 |
| *** Apoorva has joined #openstack-infra | 00:20 | |
| *** felipemonteiro_ has quit IRC | 00:21 | |
| fungi | we could reset the uid and gid of jenkins at the beginning of legacy jobs? | 00:22 |
| *** xarses has quit IRC | 00:22 | |
| mnaser | fungi i can imagine this impacting other things like not finding things in HOMEDIR? | 00:23 |
| *** xinliang has quit IRC | 00:23 | |
| fungi | if they hardcode /home/jenkins maybe | 00:23 |
| fungi | but ~jenkins would be /home/zuul | 00:23 |
| mnaser | well i can imagine their code will start to write out things with user "zuul" then sudo into jenkins and not be able to read/write things | 00:24 |
| fungi | it would own the same files because it would have the same uid | 00:25 |
| mordred | ianw, fungi: well - also - basically everytihng about that script has new zuul replacements | 00:25 |
| fungi | basically if we reset the uid and gid of "jenkins" at the start of legacy jobs then they have no chance to create files as the former jenkins gid/uid, so would have interchangeable ownership between jenkins and zuul | 00:26 |
| fungi | for all files created as either | 00:27 |
| *** xarses has joined #openstack-infra | 00:28 | |
| *** andreas_s has joined #openstack-infra | 00:28 | |
| ianw | fungi: i like it ... but not sure if it's one of those "curse is worse than disease" -- will it create much harder to diagnose issues with mixed up homedirs for example? | 00:29 |
| *** pvaneck has quit IRC | 00:29 | |
| fungi | unknown unknowns are always the worst | 00:30 |
| fungi | is it a better solution that trying to patch all projects who cargo-culted that pattern around? | 00:30 |
| mordred | yah - I'd kind of prefer the opportunity to just actually fix the things | 00:30 |
| mordred | like, seriously - there is only one line in https://git.openstack.org/cgit/openstack/python-openstackclient/tree/openstackclient/tests/functional/post_test_hook_tips.sh#n49 that needs to stick around somehow | 00:31 |
| mordred | and that's line 44 | 00:31 |
| fungi | i'm fine with saying that the username the job starts as was never meant as a stable interface, and if you hard-code "jenkins" (or "zuul" for that matter) into your projects then that needs fixing | 00:31 |
| *** Hunner has quit IRC | 00:31 | |
| *** Hunner has joined #openstack-infra | 00:32 | |
| *** Hunner has quit IRC | 00:32 | |
| *** Hunner has joined #openstack-infra | 00:32 | |
| *** andreas_s has quit IRC | 00:32 | |
| mordred | line 49 can be a call to the openstack-tox role in the playbook - and the job can set required-projects: osc-lib, os-client-config, python-openstacksdk - and all the logic that script does is now taken care of | 00:33 |
| mordred | sourcing ~stack/devstack/openrc admin admin before running tox is a thing that needs to be handled - but that can likely just be done by setting an env var and updating the tox.ini to consume it and do the source | 00:33 |
| mordred | which is mostly me saying that fixing the problem jobs may be both easier AND better long-term BUT ... it is now time for the dinners ... | 00:35 |
| *** xinliang has joined #openstack-infra | 00:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 00:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 00:43 |
| EmilienM | pabelanger: revert what? | 00:44 |
| *** caphrim007 has joined #openstack-infra | 00:46 | |
| anupn | fungi, a quick question, i figured that i can use disabled_python3_packages to make swift run with python2 but can i directly call this function from file devstack-vm-gate-wrap.sh? | 00:51 |
| *** kiennt26 has joined #openstack-infra | 00:52 | |
| anupn | i see that from devstack plugin for ironic i can call function enable_python3_packages directly but wonder can the call be done directly from devstack-vm-gate-wrap.sh or i have to do it from ironic project | 00:53 |
| *** kiennt26_ has joined #openstack-infra | 00:53 | |
| *** kiennt26_ has quit IRC | 00:53 | |
| *** thorst has joined #openstack-infra | 00:53 | |
| *** Goneri has quit IRC | 00:53 | |
| *** kiennt26_ has joined #openstack-infra | 00:54 | |
| *** thorst has quit IRC | 00:54 | |
| *** LindaWang has joined #openstack-infra | 00:54 | |
| EmilienM | pabelanger: this one? | 00:56 |
| *** kiennt26_ has quit IRC | 01:00 | |
| *** kiennt26_ has joined #openstack-infra | 01:01 | |
| *** kiennt26 has quit IRC | 01:04 | |
| *** andreas_s has joined #openstack-infra | 01:04 | |
| *** Apoorva_ has joined #openstack-infra | 01:05 | |
| ianw | what's openstack-zuul-roles meant to hold? | 01:07 |
| *** Apoorva has quit IRC | 01:08 | |
| *** andreas_s has quit IRC | 01:09 | |
| *** Apoorva_ has quit IRC | 01:09 | |
| *** kiennt26_ has quit IRC | 01:10 | |
| *** mat128 has joined #openstack-infra | 01:10 | |
| *** jkilpatr has quit IRC | 01:10 | |
| *** kiennt26 has joined #openstack-infra | 01:10 | |
| *** eroux has quit IRC | 01:11 | |
| *** cuongnv has joined #openstack-infra | 01:11 | |
| *** kiennt26 has quit IRC | 01:12 | |
| *** kiennt26 has joined #openstack-infra | 01:14 | |
| *** kiennt26 has quit IRC | 01:17 | |
| *** jaypipes has quit IRC | 01:18 | |
| *** xarses has quit IRC | 01:18 | |
| openstackgerrit | Anup Navare proposed openstack-infra/devstack-gate master: [Test] DNM Checking if tinyIPA builds with py3 https://review.openstack.org/509641 | 01:19 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add logs for config-download https://review.openstack.org/510709 | 01:21 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Update document url and fix a spelling error https://review.openstack.org/505475 | 01:22 |
| *** kiennt26 has joined #openstack-infra | 01:25 | |
| fungi | ianw: prior discussion suggests not much if anything. we're likely to retire that after we're finished with the transition | 01:25 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Craton https://review.openstack.org/510820 | 01:26 |
| *** kiennt26 has quit IRC | 01:26 | |
| *** kiennt26 has joined #openstack-infra | 01:26 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Include gate jobs in the cistatus page https://review.openstack.org/482316 | 01:27 |
| *** kiennt26 has quit IRC | 01:27 | |
| *** kiennt26 has joined #openstack-infra | 01:27 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Craton https://review.openstack.org/510824 | 01:27 |
| *** hemna_ has quit IRC | 01:31 | |
| *** s-shiono has joined #openstack-infra | 01:32 | |
| openstackgerrit | Merged openstack-infra/system-config master: Allow all zuul servers to send to statsd https://review.openstack.org/511068 | 01:34 |
| *** hongbin has joined #openstack-infra | 01:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 01:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 01:35 |
| *** andreas_s has joined #openstack-infra | 01:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 01:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 01:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 01:38 |
| *** namnh has joined #openstack-infra | 01:40 | |
| *** andreas_s has quit IRC | 01:41 | |
| *** liujiong has joined #openstack-infra | 01:41 | |
| *** mat128 has quit IRC | 01:42 | |
| *** xarses has joined #openstack-infra | 01:50 | |
| *** psachin has joined #openstack-infra | 01:51 | |
| *** aeng has quit IRC | 01:51 | |
| *** aeng has joined #openstack-infra | 01:52 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Cyborg https://review.openstack.org/511087 | 01:56 |
| *** xarses has quit IRC | 01:57 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Cyborg https://review.openstack.org/511088 | 01:57 |
| openstackgerrit | Kien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Remove Kuryr-kubernetes legacy jobs https://review.openstack.org/510773 | 01:57 |
| *** dhinesh_ has quit IRC | 02:06 | |
| *** owalsh_ has joined #openstack-infra | 02:08 | |
| dmsimard | mordred, inc0: I'm curious, was there any other discussions regarding an eventual Docker Registry implementation in Infra ? I remember we discussed docker-registry/docker-distribution (which are kind of bad), some solution from CoreOS as well as OpenShift (Atomic) registry | 02:11 |
| *** owalsh has quit IRC | 02:11 | |
| inc0 | dmsimard: no we didn't move topic that much after ptg | 02:12 |
| inc0 | openshift has it's registry implementation? | 02:12 |
| dmsimard | inc0: yeah, that's what we're using for the RDO private registry. | 02:12 |
| dmsimard | inc0: so you know that openshift is basically k8s with bells and whistles right | 02:12 |
| inc0 | yeah | 02:13 |
| dmsimard | inc0: so there a internal docker-registry pod in openshift, it's docker-registry v2, but openshift (and openshift API) plugs a bunch of holes | 02:13 |
| inc0 | well from my perspective I don't really care which implementation we choose | 02:13 |
| dmsimard | It allows things like easy RBAC/ACL (we use oauth with github), easy image deletion (although this will get better in 3.7 which is not yet released), being able to list images remotely, being able to tag images without pulling them, etc. | 02:15 |
| inc0 | yeah, some of these would be cool | 02:15 |
| inc0 | especially when some of our images will have multiple tags | 02:15 |
| dmsimard | oh, and it provides a fancy UI to view things | 02:16 |
| dmsimard | http://www.projectatomic.io/registry/ | 02:16 |
| dmsimard | atomic registry is deprecated in favor of openshift standalone registry though | 02:16 |
| inc0 | cool | 02:16 |
| openstackgerrit | Kien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Remove Kuryr-kubernetes legacy jobs https://review.openstack.org/510773 | 02:16 |
| ianw | fungi: ok good, thought i might have missed a memo :) | 02:17 |
| dmsimard | it has that easy install atomic thing if you want to try it out, although you'll end up using openshift-ansible if you find you need something a bit out of the ordinary | 02:17 |
| *** thorst has joined #openstack-infra | 02:17 | |
| dmsimard | https://docs.openshift.com/container-platform/latest/install_config/install/stand_alone_registry.html | 02:17 |
| dmsimard | inc0: do you remember what that core os registry thing was ? | 02:17 |
| inc0 | well we'd still need puppet module for this stuff | 02:17 |
| inc0 | dmsimard: I tried to look for it - their stuff is quay.io | 02:18 |
| inc0 | but can't find opensource | 02:18 |
| *** thorst has quit IRC | 02:18 | |
| dmsimard | inc0: "try for free on-premise" ouch | 02:18 |
| dmsimard | yeah I don't see any quay.io things in https://github.com/coreos :/ | 02:19 |
| *** dave-mccowan has joined #openstack-infra | 02:20 | |
| *** fanzhang_ has joined #openstack-infra | 02:20 | |
| inc0 | well this openshift one looks good tho | 02:20 |
| *** fanzhang has quit IRC | 02:21 | |
| *** petems has quit IRC | 02:21 | |
| *** jlvillal has quit IRC | 02:21 | |
| *** jlvillal has joined #openstack-infra | 02:21 | |
| dmsimard | We've been fighting a bit with it, sent a few PRs and helped with feedback, they're pretty receptive. | 02:22 |
| *** dave-mcc_ has quit IRC | 02:22 | |
| dmsimard | We might upgrade to 3.7 ahead of release because we need the image pruning improvements | 02:22 |
| *** petems has joined #openstack-infra | 02:23 | |
| dmsimard | We're not "OpenStack scale" but we're still pushing all our kolla images there >8 times a day so it is a concern to avoid keeping images (and their blobs/layers) around forever to keep disk space usage under control | 02:24 |
| inc0 | does it have features for that? | 02:24 |
| dmsimard | inc0: yeah: https://docs.openshift.org/latest/admin_guide/pruning_resources.html#pruning-images | 02:25 |
| *** liusheng has joined #openstack-infra | 02:25 | |
| dmsimard | oh, it also natively supports openstack cinder for persistent volume claims :) | 02:31 |
| *** rgerganov has quit IRC | 02:33 | |
| *** fanzhang_ is now known as fanzhang | 02:33 | |
| *** thorst has joined #openstack-infra | 02:33 | |
| *** rgerganov has joined #openstack-infra | 02:33 | |
| *** ijw has quit IRC | 02:34 | |
| *** thorst has quit IRC | 02:38 | |
| *** ramishra has joined #openstack-infra | 02:39 | |
| *** mat128 has joined #openstack-infra | 02:42 | |
| *** lamt has quit IRC | 02:42 | |
| *** pleia2 has quit IRC | 02:43 | |
| *** armaan has quit IRC | 02:43 | |
| *** armaan has joined #openstack-infra | 02:43 | |
| *** _Cyclone_ has quit IRC | 02:44 | |
| *** _Cyclone_ has joined #openstack-infra | 02:46 | |
| *** lamt has joined #openstack-infra | 02:46 | |
| *** lamt is now known as Guest41495 | 02:46 | |
| *** pleia2 has joined #openstack-infra | 02:50 | |
| *** dave-mccowan has quit IRC | 02:51 | |
| *** erlon has quit IRC | 02:54 | |
| *** markvoelker has quit IRC | 02:54 | |
| *** xarses has joined #openstack-infra | 02:55 | |
| dmsimard | ianw: gj on the unbound stuff :) | 02:55 |
| *** links has joined #openstack-infra | 02:57 | |
| ianw | dmsimard: thanks for review! i do like finding ways that ansible has already figured out the hard bits for | 02:59 |
| *** caphrim007 has quit IRC | 03:00 | |
| dmsimard | ansible is crazy good :D | 03:00 |
| dmsimard | in-line jinja was the last thing I learned that made my head explode | 03:00 |
| dmsimard | ianw: doh, your typo in https://review.openstack.org/#/c/510734/5/roles/configure-unbound/tasks/main.yaml made it through "fowarding" :p | 03:01 |
| *** sree has joined #openstack-infra | 03:01 | |
| pabelanger | ianw: the idea for openstack-zuul-roles and zuul-roles, we to be a seperate projects to just hold roles, where openstack-zuul-jobs / zuul-jobs would have been just playbooks. But, in the end it results in a lot of cross repo depends-on, which made first round of review difficult. It worked, but we quickly decided to move things into 2 repos (over 4) to simplify things | 03:02 |
| dmsimard | pabelanger: +1 | 03:03 |
| *** nicolasbock has quit IRC | 03:06 | |
| *** sree has quit IRC | 03:06 | |
| ianw | dmsimard: boo; i did fix it but must have somehow dropped it in migration. we can fix when we find something wrong with it :) | 03:09 |
| *** mat128 has quit IRC | 03:13 | |
| *** gongysh has joined #openstack-infra | 03:16 | |
| *** yamahata has joined #openstack-infra | 03:17 | |
| mordred | dmsimard: quay.io is out - it's closed source | 03:17 |
| mordred | dmsimard: OpenShift Container Registry looks like it requires running OpenShift and/or kubernetes - which sounds a bit overkill | 03:20 |
| *** armaan has quit IRC | 03:24 | |
| mordred | dmsimard: atomic registry looks better - it at least has a non-k8s install method - but it *looks* like it still wants to be installed and run using docker rather than just being able to be installed like other things | 03:25 |
| *** yamamoto has quit IRC | 03:25 | |
| *** armaan has joined #openstack-infra | 03:25 | |
| dmsimard | mordred: atomic registry is deprecated | 03:25 |
| mordred | dmsimard: of course it is | 03:26 |
| dmsimard | ¯\_(ツ)_/¯ | 03:26 |
| pabelanger | I thought we were on board with docker-registry ubuntu package? | 03:26 |
| mordred | dmsimard: well - from what I can tell docker registry is still the winner in terms of not requiring a wholesale adoption of a completely different way of viewing hte world to be installed | 03:26 |
| mordred | pabelanger: yes. very much so | 03:26 |
| *** liujiong has quit IRC | 03:26 | |
| pabelanger | Ya, once zuulv3 is healthy, I have it on my list to play with again | 03:27 |
| pabelanger | based on what we talked about at PTG | 03:27 |
| dmsimard | All I'm saying is that docker-registry/docker-distribution have a lot of limitations | 03:27 |
| mordred | pabelanger: I think this was about looking at the other thing that provide docker registries to see if they provided compelling features or other reaons they would be the right choice | 03:27 |
| *** liujiong has joined #openstack-infra | 03:27 | |
| mordred | dmsimard: sure - but what they dont' have is a requirement that we run kubernetes | 03:27 |
| mordred | in fact, they don't even make us run docker | 03:27 |
| dmsimard | Little to no client side implementation, hard ACL/RBAC, hard image deletion | 03:27 |
| pabelanger | dmsimard: right, but I think our use case is pretty basic, isn't it? | 03:27 |
| mordred | dmsimard: yah - but easy to run - apt-get install works - we don't have to install and run an entire additoinal orchestration system with a bunch of networking overlay crap we don't need and a ton of moving parts that don't gain us anything | 03:28 |
| dmsimard | I'm on my phone and typing this much hurts but happy to pick up the discussion at a keyboard tomorrow or later :p | 03:28 |
| mordred | dmsimard: yes - much more fun conversation with a keyboard :) | 03:29 |
| pabelanger | +1 | 03:29 |
| dmsimard | mordred: our standalone registry in RDO is 3 pods with one container in each. One for registry, one for web console and one for the router (haproxy) | 03:30 |
| mordred | right. but that assumes a place where you can run pods | 03:30 |
| dmsimard | openshift-ansible takes care of setting everything up | 03:30 |
| ianw | dmsimard: re jeblairs comment in https://review.openstack.org/#/c/504788 ... does the integration test run on each host? | 03:30 |
| *** liujiong has quit IRC | 03:30 | |
| dmsimard | I hate to leave this discussion unfinished but I really need a keyboard :( | 03:31 |
| mordred | dmsimard: it's fine - it's honestly a better conversation for post-v3 week | 03:31 |
| dmsimard | ianw: yes, jeblair is right but I was on other things all day | 03:31 |
| pabelanger | np, I just wanted to make sure I didn't miss something. | 03:32 |
| ianw | that's ok, just going through the open reviews in the list | 03:32 |
| mordred | dmsimard, pabelanger: I should know better than to dive in to that particular can of worms at this time of night anyway | 03:32 |
| dmsimard | pabelanger: a spec where we evaluate the different options with pros and cons seems fair | 03:32 |
| pabelanger | dmsimard: IIRC, we have a spec up | 03:32 |
| pabelanger | https://review.openstack.org/447524/ is what I was thinking about | 03:41 |
| openstackgerrit | Merged openstack-infra/project-config master: Add post tarball gate to kuryr-tempest-plugin https://review.openstack.org/510786 | 03:42 |
| *** armaan has quit IRC | 03:47 | |
| *** armaan has joined #openstack-infra | 03:47 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add an origin remote for tag-releases for notes https://review.openstack.org/511004 | 03:48 |
| mordred | ianw: https://review.openstack.org/#/c/511079 lgtm - any reason not to go ahead and land it? | 03:55 |
| *** ganso has quit IRC | 03:56 | |
| ianw | mordred: i think it's ok ... i can keep an eye on the zuulv3 queue to ensure it's not causing undue issues | 03:56 |
| mordred | ianw: cool - I pulled the trigger on it | 03:57 |
| mordred | ianw: also - do you know off the top of your head if we got the new images built with the cache dir link thing? | 03:57 |
| ianw | mordred: i checked and at least xenial & centos were building a few hours ago | 03:57 |
| mordred | kk. I'll leave that entry in the list for now | 03:58 |
| ianw | yep, those are fresh | 03:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Pass root_marker_dir to write-root-marker https://review.openstack.org/511122 | 04:01 |
| mordred | ianw: thta's a pretty easy fix to one of the bugs ^^ :) | 04:02 |
| mordred | pabelanger: (if'n you're still around ^^) | 04:02 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job https://review.openstack.org/510237 | 04:04 |
| *** ykarel|afk has joined #openstack-infra | 04:04 | |
| *** sshnaidm|off is now known as sshnaidm | 04:06 | |
| sshnaidm | clarkb, the images in zuulv3 is in /opt/cache, not in /opt/cache/files | 04:07 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add upload-npm role https://review.openstack.org/510686 | 04:08 |
| ianw | is it just me, or is there no "hide ci" button on this page -> https://review.openstack.org/#/c/510237/ | 04:08 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: zuul-cloner-shim: Use st_dev to check for filesystem https://review.openstack.org/511079 | 04:08 |
| mordred | ianw: I see a "Toggle CI" button at the bottom | 04:09 |
| ianw | yeah, i do on chrome, but firefox | 04:09 |
| ianw | not firefox | 04:09 |
| ianw | sorry, looks like i messed up what i thought was a simple rebase | 04:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use zuul_legacy_vars for propose jobs for now https://review.openstack.org/511126 | 04:12 |
| *** hongbin has quit IRC | 04:12 | |
| *** mat128 has joined #openstack-infra | 04:12 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add release-openstack-javascript job https://review.openstack.org/502185 | 04:13 |
| *** yamamoto has joined #openstack-infra | 04:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add requirements to legacy-gate-placement-api-ref https://review.openstack.org/511127 | 04:14 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job https://review.openstack.org/510237 | 04:15 |
| ianw | mordred: pretty sure names just got backwards in zuul.d/project-templates.yaml ^ | 04:15 |
| *** zaneb has quit IRC | 04:17 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add python-openstackclient to legacy-check-osc-plugins https://review.openstack.org/511128 | 04:17 |
| ianw | also, i am an idiot and just had to resize my screen | 04:19 |
| mordred | ianw: ah - good catch on the javascript ones | 04:19 |
| mordred | ianw: ok - that's all the patches for the outstanding job bugs I know about (and can deal with) | 04:20 |
| mordred | the propose-updates one needs a complete rework - butI think the zuul_legacy_vars thing will tide us over for a day or two until we can get to it | 04:21 |
| *** jtomasek has joined #openstack-infra | 04:21 | |
| *** khamtamtun has joined #openstack-infra | 04:23 | |
| *** khamtamtun has quit IRC | 04:25 | |
| *** claudiub|2 has joined #openstack-infra | 04:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add configure-unbound role to base-test job https://review.openstack.org/510728 | 04:26 |
| pabelanger | sshnaidm: clarkb: I'm not sure how that is possible, both zuulv2.5 and zuulv3 are using the same images. And just confirmed it is /opt/cache/files | 04:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Pass root_marker_dir to write-root-marker https://review.openstack.org/511122 | 04:36 |
| pabelanger | EOD | 04:38 |
| *** caphrim007 has joined #openstack-infra | 04:38 | |
| *** thorst has joined #openstack-infra | 04:39 | |
| *** coolsvap has joined #openstack-infra | 04:41 | |
| *** zaneb has joined #openstack-infra | 04:41 | |
| *** yamamoto has quit IRC | 04:43 | |
| *** thorst has quit IRC | 04:44 | |
| *** mat128 has quit IRC | 04:44 | |
| sshnaidm | pabelanger, clarkb in zuulv2 it was in /opt/stack/cache/files/cirros-0.3.5-x86_64-disk.img | 04:44 |
| *** liusheng has quit IRC | 04:51 | |
| jlvillal | Anyone else noticing build failures with messages like: The repository 'http://mirror.bhs1.ovh.openstack.org/ubuntu xenial Release' is not signed. | 04:52 |
| *** gongysh has quit IRC | 04:52 | |
| ianw | jlvillal: no, but we don't sign them. the repos should be setup to ignore it | 04:54 |
| jlvillal | And then at the error: The following packages have unmet dependencies: | 04:54 |
| jlvillal | libcurl4-gnutls-dev : Depends: libcurl3-gnutls (= 7.47.0-1ubuntu2.2) but 7.47.0-1ubuntu2.3 is to be installed | 04:54 |
| jlvillal | http://logs.openstack.org/93/507793/2/check/gate-ironic-python35/00b403d/console.html#_2017-10-11_03_22_15_634972 | 04:54 |
| chandankumar | ianw: https://review.openstack.org/501231 and https://review.openstack.org/508502 need blessing on these reviews :-) | 04:54 |
| jlvillal | ianw, Thanks. I've seen a lot of failures in unit test jobs and the like | 04:54 |
| *** markvoelker has joined #openstack-infra | 04:55 | |
| jlvillal | As in every Ironic job in the gate as seen here: http://status.openstack.org/zuul/ | 04:56 |
| jlvillal | Has a failure. | 04:56 |
| *** armaan has quit IRC | 04:58 | |
| *** armaan has joined #openstack-infra | 04:59 | |
| ianw | jlvillal: i think it might be upstream | 05:00 |
| jlvillal | ianw, Okay. I'll check back in the morning. 22:00 here :) | 05:00 |
| jlvillal | ianw, Thanks! | 05:00 |
| *** armax has quit IRC | 05:01 | |
| *** gongysh has joined #openstack-infra | 05:07 | |
| *** sree_ has joined #openstack-infra | 05:11 | |
| *** gongysh has quit IRC | 05:11 | |
| *** sree_ is now known as Guest72138 | 05:11 | |
| *** gouthamr has quit IRC | 05:12 | |
| *** armaan has quit IRC | 05:13 | |
| *** armaan has joined #openstack-infra | 05:13 | |
| clarkb | sshnaidm: yes should still be there | 05:17 |
| clarkb | sshnaidm: same image is used in both places | 05:17 |
| sshnaidm | clarkb, afaik it moved to /opt/cache/files , without "Stack" | 05:18 |
| *** eumel8 has joined #openstack-infra | 05:18 | |
| sshnaidm | clarkb, anyway, scripts failed to find it there.. | 05:18 |
| clarkb | sshnaidm: the stack bit is a devstack-gate thing | 05:18 |
| clarkb | it copies from /home/zuul/cache to set that up which was broken | 05:18 |
| clarkb | was fixed today | 05:19 |
| sshnaidm | clarkb, let's fix it again after a migration, not a big deal | 05:19 |
| eumel8 | morning | 05:19 |
| *** Guest2849 is now known as sdake | 05:20 | |
| *** sdake is now known as Guest95430 | 05:21 | |
| *** Guest95430 is now known as sdake_fixing | 05:23 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix: legacy-sahara-cli is a "dsvm" job https://review.openstack.org/511037 | 05:24 |
| *** xarses has quit IRC | 05:26 | |
| *** sdake_fixing has quit IRC | 05:29 | |
| *** sdake_fixing has joined #openstack-infra | 05:29 | |
| *** markvoelker has quit IRC | 05:30 | |
| *** gildub has quit IRC | 05:33 | |
| *** sdake_fixing has quit IRC | 05:35 | |
| *** udesale has joined #openstack-infra | 05:38 | |
| *** armaan has quit IRC | 05:40 | |
| *** armaan has joined #openstack-infra | 05:41 | |
| *** mat128 has joined #openstack-infra | 05:43 | |
| *** yamamoto has joined #openstack-infra | 05:43 | |
| *** cshastri has joined #openstack-infra | 05:49 | |
| *** yamamoto has quit IRC | 05:49 | |
| *** csomerville has joined #openstack-infra | 05:50 | |
| *** cody-somerville has quit IRC | 05:50 | |
| *** sshnaidm is now known as sshnaidm|afk | 05:52 | |
| *** dhajare has joined #openstack-infra | 05:54 | |
| *** yamamoto has joined #openstack-infra | 05:54 | |
| *** yamamoto has quit IRC | 05:58 | |
| *** yamamoto has joined #openstack-infra | 06:02 | |
| *** yamamoto has quit IRC | 06:04 | |
| AJaeger | morning. ianw, are all zuul executors up and running? | 06:05 |
| *** andreas_s has joined #openstack-infra | 06:11 | |
| *** liusheng has joined #openstack-infra | 06:14 | |
| *** mat128 has quit IRC | 06:14 | |
| *** martinkopec has joined #openstack-infra | 06:15 | |
| *** dhajare has quit IRC | 06:19 | |
| *** dhajare has joined #openstack-infra | 06:19 | |
| *** martinkopec has quit IRC | 06:20 | |
| *** martinkopec has joined #openstack-infra | 06:21 | |
| *** yamamoto has joined #openstack-infra | 06:21 | |
| AJaeger | ah, seems we make progress on jobs - so, might be fine and just a long queue... | 06:23 |
| ianw | AJaeger: yeah, things were restarted about 10 hours ago | 06:25 |
| *** markvoelker has joined #openstack-infra | 06:26 | |
| *** spectr has quit IRC | 06:27 | |
| *** spectr has joined #openstack-infra | 06:28 | |
| *** vsaienk0 has joined #openstack-infra | 06:30 | |
| AJaeger | ianw: great, thanks | 06:31 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Trove https://review.openstack.org/511149 | 06:34 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: neutron-lbaas: Add neutron requirments for tox jobs https://review.openstack.org/509143 | 06:38 |
| *** thorst has joined #openstack-infra | 06:40 | |
| *** thorst has quit IRC | 06:44 | |
| AJaeger | morning yolanda! Please review https://review.openstack.org/510765 | 06:45 |
| yolanda | hi AJaeger | 06:46 |
| yolanda | sure | 06:46 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Trove legacy jobs https://review.openstack.org/511150 | 06:46 |
| AJaeger | thanks! If you didn't follow the announcements, today we switch fully to Zuul v3 ;) | 06:46 |
| openstackgerrit | RUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin https://review.openstack.org/504870 | 06:46 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Freezer https://review.openstack.org/511151 | 06:46 |
| yolanda | AJaeger, yes, i read the email | 06:46 |
| yolanda | really hard work on it... | 06:47 |
| *** ccamacho has joined #openstack-infra | 06:47 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Freezer https://review.openstack.org/511152 | 06:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Revert "Use new post pipelines" https://review.openstack.org/510887 | 06:48 |
| AJaeger | yeah - many changes in flight - and making good progress | 06:49 |
| *** owalsh_ is now known as owalsh | 06:49 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Trove https://review.openstack.org/511149 | 06:50 |
| dmellado | more changes? /me sighs xD | 06:50 |
| *** threestrands has quit IRC | 06:50 | |
| AJaeger | dmellado: all those projects that want to adjust their jobs ;) | 06:51 |
| openstackgerrit | RUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin https://review.openstack.org/504870 | 06:52 |
| dmellado | AJaeger: heh, I know, just kidding | 06:52 |
| dmellado | even I need to add quite a few more gates too | 06:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove project pylockfile from infrastructure systems https://review.openstack.org/508656 | 06:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Use zuul_legacy_vars for propose jobs for now https://review.openstack.org/511126 | 06:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Add new repo for magnum tempest plugin https://review.openstack.org/501231 | 06:55 |
| openstackgerrit | Merged openstack-infra/project-config master: neutron-lbaas: Add neutron requirments for tox jobs https://review.openstack.org/509143 | 06:55 |
| openstackgerrit | RUIJIE YUAN proposed openstack-infra/project-config master: add merge-check job template for senlin-tempest-plugin https://review.openstack.org/511154 | 06:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Add python-tempestconf project https://review.openstack.org/508502 | 06:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Adding check/gate jobs to Contributor Guide https://review.openstack.org/509937 | 06:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Add configure-unbound to base job https://review.openstack.org/511075 | 06:56 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job https://review.openstack.org/510237 | 06:56 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add requirements to legacy-gate-placement-api-ref https://review.openstack.org/511127 | 06:56 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove most of the oslo.messaging legacy tests https://review.openstack.org/510661 | 06:56 |
| SpamapS | well lookie there it's catching up. :) | 06:57 |
| AJaeger | SpamapS: That's yolanda's review breakfast ;) | 06:59 |
| AJaeger | dmellado: go for it - hope in-repo works fine for you ;) | 06:59 |
| yolanda | AJaeger, yep... trying to do some reviews before starting my day | 07:00 |
| dmellado | AJaeger: heh, I hope so! I'll check multinode things now so crossing fingers | 07:00 |
| *** markvoelker has quit IRC | 07:00 | |
| dmellado | yolanda: you just stack them before going to bed and do that in the morning | 07:00 |
| dmellado | so you can show smart and mighty xDDDDD | 07:00 |
| * dmellado hides | 07:00 | |
| *** sdake has joined #openstack-infra | 07:01 | |
| *** sdake has quit IRC | 07:01 | |
| *** sdake has joined #openstack-infra | 07:01 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: set neutron mtu to match system mtu https://review.openstack.org/509761 | 07:02 |
| *** tumbarka has quit IRC | 07:02 | |
| *** openstackgerrit has quit IRC | 07:03 | |
| *** openstackgerrit has joined #openstack-infra | 07:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs https://review.openstack.org/510765 | 07:03 |
| AJaeger | yolanda: could you +2 again, please? - I had to resolve a merge conflict ^ | 07:04 |
| *** rcernin has joined #openstack-infra | 07:05 | |
| strigazi | AJaeger: Hello, could you have a look when you can to magnum's zuulv3 migration patch? https://review.openstack.org/#/c/508676 | 07:06 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: grenade-dsvm-ironic: Move task 'legacy workspace dir' to first https://review.openstack.org/510795 | 07:06 |
| AJaeger | yolanda: and review https://review.openstack.org/#/c/510353 as well, please | 07:06 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Add merge-check for python-tempestconf https://review.openstack.org/510353 | 07:08 |
| AJaeger | strigazi: looks fine - and I missed your recheck ;) Let'S wait for Zuul +1 and then merge. I just looked at .zuul.yaml and overall structure | 07:08 |
| *** jpena|off is now known as jpena | 07:08 | |
| yolanda | AJaeger, 510353 fails | 07:09 |
| yolanda | i tried a rebase but still has -1 from Zuul | 07:09 |
| yolanda | may be one of the dependencies | 07:09 |
| AJaeger | yolanda: let'S wait a bit until the repo is properly created - thanks | 07:10 |
| AJaeger | will recheck later | 07:10 |
| *** mat128 has joined #openstack-infra | 07:12 | |
| *** pcaruana has joined #openstack-infra | 07:13 | |
| *** ykarel_ has joined #openstack-infra | 07:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "Add configure-unbound to base job" https://review.openstack.org/511157 | 07:14 |
| AJaeger | yolanda, ianw, I think unbound causes problem with IPv6, let's revert for now ^ | 07:14 |
| openstackgerrit | RUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin https://review.openstack.org/504870 | 07:15 |
| *** ykarel|afk has quit IRC | 07:16 | |
| *** ralonsoh has joined #openstack-infra | 07:18 | |
| *** tesseract has joined #openstack-infra | 07:19 | |
| *** ykarel_ is now known as ykarel | 07:21 | |
| *** florianf has joined #openstack-infra | 07:21 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Octavia-dashboard https://review.openstack.org/511160 | 07:21 |
| *** hashar has joined #openstack-infra | 07:24 | |
| openstackgerrit | RUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin https://review.openstack.org/504870 | 07:25 |
| *** neon_icon has joined #openstack-infra | 07:26 | |
| *** tinwood has quit IRC | 07:30 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Octavia-dashboard legacy jobs https://review.openstack.org/511165 | 07:30 |
| *** tinwood has joined #openstack-infra | 07:30 | |
| *** armaan has quit IRC | 07:33 | |
| *** nikhil has quit IRC | 07:34 | |
| *** armaan has joined #openstack-infra | 07:34 | |
| *** thorre has quit IRC | 07:34 | |
| *** armaan has quit IRC | 07:35 | |
| *** yamamoto has quit IRC | 07:35 | |
| *** thorre has joined #openstack-infra | 07:35 | |
| *** armaan has joined #openstack-infra | 07:35 | |
| *** yamamoto has joined #openstack-infra | 07:36 | |
| *** yamamoto has quit IRC | 07:36 | |
| *** jpich has joined #openstack-infra | 07:39 | |
| AJaeger | yolanda: can you force merge https://review.openstack.org/511157 , please? | 07:39 |
| AJaeger | WE run into RETRY_LIMITS all the time ;( | 07:39 |
| yolanda | ok i'll do | 07:40 |
| *** thorst has joined #openstack-infra | 07:41 | |
| AJaeger | thanks | 07:41 |
| *** mat128 has quit IRC | 07:44 | |
| kazsh | AJaeger: G'day, what's next action for this change ? Should I ask my PTL to check this ? or need to fix something (looks getting an error but not sure the cause) ? | 07:44 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Add configure-unbound to base job" https://review.openstack.org/511157 | 07:45 |
| *** thorst has quit IRC | 07:45 | |
| AJaeger | kazsh: which one? | 07:45 |
| kazsh | Oops sorry, https://review.openstack.org/#/c/509119/7 AJaeger | 07:46 |
| AJaeger | #status log Lots of RETRY_LIMIT errors due to unbound useage with Zuul v3, we reverted the change; recheck your changes | 07:46 |
| openstackstatus | AJaeger: finished logging | 07:46 |
| AJaeger | kazsh: you need PTL +1 - and let me comment | 07:47 |
| kazsh | AJaeger, ok will do, thanks for your help as alwasy :) | 07:47 |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** armaan has quit IRC | 07:51 | |
| *** armaan has joined #openstack-infra | 07:51 | |
| *** openstackgerrit has joined #openstack-infra | 07:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add new repo for manila tempest plugin https://review.openstack.org/507905 | 07:52 |
| *** csomerville has quit IRC | 07:54 | |
| *** csomerville has joined #openstack-infra | 07:57 | |
| *** markvoelker has joined #openstack-infra | 07:57 | |
| *** gildub has joined #openstack-infra | 08:02 | |
| *** florianf has quit IRC | 08:02 | |
| *** florianf has joined #openstack-infra | 08:08 | |
| *** alexchadin has joined #openstack-infra | 08:12 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test containers update https://review.openstack.org/511175 | 08:12 |
| *** florianf has quit IRC | 08:13 | |
| *** florianf has joined #openstack-infra | 08:13 | |
| *** gildub has quit IRC | 08:13 | |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project https://review.openstack.org/509119 | 08:17 |
| *** yamahata has quit IRC | 08:19 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add publish-openstack-sphinx-docs to Blazar https://review.openstack.org/511176 | 08:20 |
| *** alexchadin has quit IRC | 08:22 | |
| *** alexchadin has joined #openstack-infra | 08:22 | |
| *** e0ne has joined #openstack-infra | 08:23 | |
| *** yamamoto has joined #openstack-infra | 08:23 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove Kuryr-kubernetes legacy jobs https://review.openstack.org/510773 | 08:23 |
| *** claudiub|2 has quit IRC | 08:26 | |
| openstackgerrit | RUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin https://review.openstack.org/504870 | 08:27 |
| *** pcaruana has quit IRC | 08:27 | |
| *** kjackal_ has joined #openstack-infra | 08:29 | |
| *** shardy has joined #openstack-infra | 08:29 | |
| *** markvoelker has quit IRC | 08:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Revert "Add javascript tarball publication job" https://review.openstack.org/511178 | 08:31 |
| *** pcaruana has joined #openstack-infra | 08:31 | |
| *** dingyichen has quit IRC | 08:32 | |
| AJaeger | infra-root, I don't know what's going on with release-openstack-javascript, we have several jobs failing that it's undefined. As quick aid, I propose 511178. | 08:32 |
| *** openstackgerrit has quit IRC | 08:33 | |
| *** openstackgerrit has joined #openstack-infra | 08:33 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add documentation publish jobs to Blazar https://review.openstack.org/511176 | 08:33 |
| *** spectr has quit IRC | 08:40 | |
| AJaeger | infra-root, or is just one zuul-executor not updated? Some changes pass, others fail | 08:40 |
| *** dhajare has quit IRC | 08:40 | |
| fanzhang | Hi, jekins and zuul in trove are kind of broken, are you guys working on it ? | 08:40 |
| *** egonzalez has joined #openstack-infra | 08:41 | |
| AJaeger | fanzhang: better ask trove team on #openstack-trove and review the current status. The team is really in maintenance mode, so I would not spend any work on it unless you can help them fix issues. | 08:41 |
| *** ianychoi has quit IRC | 08:42 | |
| *** dhajare has joined #openstack-infra | 08:42 | |
| *** mat128 has joined #openstack-infra | 08:42 | |
| fanzhang | AJaeger: Thanks for your apply. So it's true that trove is in maintenance mode? I saw them discussed about it some days ago, but I am not sure what the final result is. I'll try to ask trove team, thanks anyway. | 08:44 |
| *** danpawlik has quit IRC | 08:44 | |
| *** cody-somerville has joined #openstack-infra | 08:44 | |
| *** cody-somerville has joined #openstack-infra | 08:44 | |
| *** electrofelix has joined #openstack-infra | 08:45 | |
| *** csomerville has quit IRC | 08:47 | |
| chandankumar | AJaeger: how to add initial core reviewers for python-tempestconf? https://review.openstack.org/#/admin/groups/1842,members | 08:50 |
| *** spectr has joined #openstack-infra | 08:50 | |
| *** derekh has joined #openstack-infra | 08:52 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added merge-check job template for magnum-tempest-plugin https://review.openstack.org/510529 | 08:55 |
| AJaeger | fanzhang: there was some discussion about that, best ask the trove devs | 08:55 |
| AJaeger | chandankumar: an infra-root can do this later (not me) | 08:55 |
| *** lucas-afk is now known as lucasagomes | 08:55 | |
| chandankumar | AJaeger: https://review.openstack.org/#/c/510529/ | 08:56 |
| *** ykarel is now known as ykarel|lunch | 08:56 | |
| *** yamamoto has quit IRC | 08:58 | |
| *** alexchadin has quit IRC | 08:58 | |
| *** alexchadin has joined #openstack-infra | 08:59 | |
| AJaeger | chandankumar: later... | 09:00 |
| *** yamamoto has joined #openstack-infra | 09:01 | |
| AJaeger | chandankumar: see 511178 for fix | 09:03 |
| *** yamamoto has quit IRC | 09:06 | |
| *** ianychoi has joined #openstack-infra | 09:06 | |
| *** danpawlik has joined #openstack-infra | 09:06 | |
| alexchadin | AJaeger: there is new issue with all kind of jenkins jobs: http://logs.openstack.org/84/510784/1/gate/gate-watcher-pep8-ubuntu-xenial/3c5f5c4/console.html#_2017-10-11_08_54_48_972653 | 09:08 |
| AJaeger | alexchadin: that looks like Ubuntu problem where their repo is broken ;( | 09:11 |
| *** sambetts|afk is now known as sambetts | 09:11 | |
| *** dtantsur|afk is now known as dtantsur | 09:11 | |
| AJaeger | infra-root, could you check this, please? ^ | 09:11 |
| AJaeger | alexchadin: can't help further here, hope others will later | 09:11 |
| alexchadin | AJaeger: :( thanks | 09:11 |
| AJaeger | alexchadin: I don't see this in infra jobs - could you recheck? Maybe it was temporary? | 09:12 |
| alexchadin | AJaeger: just rechecked - let's see what it brings | 09:13 |
| *** mat128 has quit IRC | 09:15 | |
| *** neon_icon has quit IRC | 09:18 | |
| *** armaan has quit IRC | 09:19 | |
| *** armaan has joined #openstack-infra | 09:20 | |
| *** priteau has joined #openstack-infra | 09:20 | |
| *** tosky has joined #openstack-infra | 09:21 | |
| *** fanzhang has quit IRC | 09:22 | |
| *** fanzhang has joined #openstack-infra | 09:25 | |
| *** markvoelker has joined #openstack-infra | 09:28 | |
| *** fanzhang_ has joined #openstack-infra | 09:31 | |
| *** quite has quit IRC | 09:32 | |
| *** ralonsoh_ has joined #openstack-infra | 09:32 | |
| eumel8 | lots of jobs are affected from broken package libcurl4-gnutls-dev | 09:34 |
| *** ralonsoh has quit IRC | 09:36 | |
| *** jaosorior has quit IRC | 09:36 | |
| AJaeger | ;( | 09:39 |
| *** s-shiono has quit IRC | 09:40 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added initial jobs for python-tempestconf https://review.openstack.org/511194 | 09:41 |
| *** thorst has joined #openstack-infra | 09:42 | |
| *** vsaienk0 has quit IRC | 09:42 | |
| *** vsaienk0 has joined #openstack-infra | 09:43 | |
| chandankumar | AJaeger: i didnot got about this change 511178 | 09:44 |
| *** quite has joined #openstack-infra | 09:44 | |
| AJaeger | chandankumar: 511178 should fix the issues with Zuul complaining about missing javascript job - and thus allow your change to merge... | 09:45 |
| AJaeger | to run tests on I mean | 09:45 |
| tosky | AJaeger: hi, what's going on with jessie here? https://review.openstack.org/#/c/511071/ | 09:47 |
| *** jaosorior has joined #openstack-infra | 09:47 | |
| *** thorst has quit IRC | 09:48 | |
| tosky | and also (I think I didn't ask it yet): is there a way to test the in-repo zuul v3 jobs locally without sending the review, or does the "merge all available jobs together" make this complicated? | 09:49 |
| *** hashar is now known as hasharAway | 09:50 | |
| AJaeger | tosky: regarding jessie: On our list - this sometimes files due to a test we have. It's not in production yet (unbound and IPv6) but testing to see whetehr we can enable it. | 09:50 |
| AJaeger | I'm not aware of a method to test locally - best to ask later here again | 09:51 |
| tosky | sure :) | 09:51 |
| tosky | so about that test, is there some review that fixes it that I should watch, or is it tracked in the zuul3-issues eitherpad? | 09:52 |
| AJaeger | I added to the etherpad already... | 09:52 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added initial jobs for python-tempestconf https://review.openstack.org/511194 | 09:54 |
| *** ykarel|lunch is now known as ykarel | 09:54 | |
| *** udesale has quit IRC | 09:58 | |
| *** alexchadin has quit IRC | 09:58 | |
| *** alexchadin has joined #openstack-infra | 09:58 | |
| *** markvoelker has quit IRC | 10:01 | |
| AJaeger | infra-root, change https://review.openstack.org/#/c/508656 reports about missing pylockfile but that was removed in https://review.openstack.org/#/c/508656 . Do we need to restart zuul so that it sees the removal of a project from it's config? Also, checking out github that change is not there - might as well be out of sync git mirrors. | 10:03 |
| *** LindaWang has quit IRC | 10:03 | |
| AJaeger | infra-root, could you check all our git instances whether those are fine and synced, please? | 10:03 |
| *** sdague has joined #openstack-infra | 10:03 | |
| * fungi just woke up but needs to do morning things before we start the outage | 10:04 | |
| AJaeger | Checking http://git.openstack.org/cgit/openstack-infra/project-config/log/ - the change is not there... | 10:05 |
| AJaeger | fungi, good morning - see above, our git mirrors are out of sync - I guess we need to fix that first ^ | 10:05 |
| AJaeger | fungi: get your morning things done first, please | 10:05 |
| fungi | #status notice The CI system will be offline starting at 11:00 UTC (in just under an hour) for Zuul v3 rollout: http://lists.openstack.org/pipermail/openstack-dev/2017-October/123337.html | 10:07 |
| openstackstatus | fungi: sending notice | 10:07 |
| fungi | wanted to get that ^ sent out as early as i could | 10:07 |
| AJaeger | ;) | 10:07 |
| * fungi goes to wake up and find coffee | 10:08 | |
| *** cuongnv has quit IRC | 10:08 | |
| -openstackstatus- NOTICE: The CI system will be offline starting at 11:00 UTC (in just under an hour) for Zuul v3 rollout: http://lists.openstack.org/pipermail/openstack-dev/2017-October/123337.html | 10:08 | |
| *** wolverineav has joined #openstack-infra | 10:08 | |
| *** andreas_s has quit IRC | 10:09 | |
| openstackstatus | fungi: finished sending notice | 10:10 |
| *** andreas_s has joined #openstack-infra | 10:10 | |
| *** dbecker has joined #openstack-infra | 10:12 | |
| *** pbourke has quit IRC | 10:12 | |
| *** cody-somerville has quit IRC | 10:12 | |
| *** kjackal_ has quit IRC | 10:13 | |
| *** pbourke has joined #openstack-infra | 10:14 | |
| *** mat128 has joined #openstack-infra | 10:14 | |
| *** andreas_s has quit IRC | 10:14 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server https://review.openstack.org/510362 | 10:15 |
| AJaeger | infra-root, why does https://git.openstack.org/cgit/openstack-infra/project-config/?h=review/andreas_jaeger/zuulv3-unbound exist? Was yolanda 's force merge done somehow wrongly? | 10:15 |
| * AJaeger needs lunch now... | 10:15 | |
| *** armaan has quit IRC | 10:19 | |
| *** rhallisey has joined #openstack-infra | 10:22 | |
| *** boden has joined #openstack-infra | 10:23 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Set right owner for /home/zuul/cache files https://review.openstack.org/511203 | 10:25 |
| *** LindaWang has joined #openstack-infra | 10:27 | |
| *** niska has quit IRC | 10:29 | |
| *** alexchadin has quit IRC | 10:31 | |
| *** gildub has joined #openstack-infra | 10:32 | |
| *** namnh has quit IRC | 10:32 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs https://review.openstack.org/510765 | 10:32 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove the legacy pylint jobs (replaced) https://review.openstack.org/511071 | 10:32 |
| *** Qiming has quit IRC | 10:32 | |
| *** niska has joined #openstack-infra | 10:34 | |
| ethfci | Hello gate-horizon-docs-ubuntu-xenial fails with The following packages have unmet dependencies: libcurl4-gnutls-dev : Depends: libcurl3-gnutls (= 7.47.0-1ubuntu2.2) but 7.47.0-1ubuntu2.3 is to be installed | 10:35 |
| *** adarazs has joined #openstack-infra | 10:35 | |
| ykarel | i saw the same ^^ here: http://logs.openstack.org/79/509079/3/gate/gate-tripleo-common-pep8-ubuntu-xenial/c155b9f/console.html in tripleo-common newton branch patch | 10:40 |
| *** mat128 has quit IRC | 10:45 | |
| *** dizquierdo has joined #openstack-infra | 10:45 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server https://review.openstack.org/510362 | 10:46 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Revert "Add javascript tarball publication job" https://review.openstack.org/511178 | 10:47 |
| AJaeger | fungi, I'm not sure whether we still need the above, see the etherpad ^ | 10:47 |
| *** jkilpatr has joined #openstack-infra | 10:49 | |
| *** Qiming has joined #openstack-infra | 10:54 | |
| fungi | okay, i'm at the computer for realz now | 10:57 |
| AJaeger | hi fungi | 10:57 |
| *** erlon has joined #openstack-infra | 10:57 | |
| ethfci | Hello Fungi | 10:58 |
| fungi | ethfci: ykarel: my first guess is some dependency conflict between xenial main and uca, but i can't look into it right now as we're about to undergo some rather delicate and time-sensitive maintenance activity | 10:58 |
| AJaeger | So, let me recap quickly: We had merged the unbound change this morning and that broke lots of jobs, so reverted (https://review.openstack.org/#/c/511157/) | 10:58 |
| *** markvoelker has joined #openstack-infra | 10:58 | |
| fungi | i can start checking out the suspected issue with git mirroring now if pabelanger and mordred are able to focus on the rollout | 10:58 |
| AJaeger | Looking at http://git.openstack.org/cgit/openstack-infra/project-config/log/ , I see the change only on a branch. yolanda force merged it | 10:58 |
| yolanda | AJaeger, looks like i did something wrong when merging yes... | 10:59 |
| yolanda | only exists there? | 10:59 |
| fungi | merging something how? | 10:59 |
| *** Guest72138 has quit IRC | 10:59 | |
| yolanda | force push | 10:59 |
| fungi | force-pushed what again? | 10:59 |
| yolanda | AJaeger's change for reverting unbound | 10:59 |
| AJaeger | We now get errors like https://review.openstack.org/#/c/508656 that report about missing pylockfile which was removed in https://review.openstack.org/#/c/508656 but does not show up on git.o.o | 10:59 |
| fungi | yolanda: there is a submit button in gerrit, just add yourself to the project bootstrappers group and then reload the page with the change | 11:00 |
| ykarel | fungi, Ok | 11:00 |
| AJaeger | fungi, 511157 | 11:00 |
| yolanda | fungi ah, i did manually. I added myself as project bootstrapper, but i git a manual git push --force, i guess it ended in the wrong branch | 11:01 |
| *** dbecker has quit IRC | 11:02 | |
| fungi | if this was a push --force resulting in a non-fast-forwardable state, i'll need to see if we can recover the old state from the reflog on disk in gerrit (or from backups), but odds are this has broken project-config pulling on, well, basically all our servers | 11:02 |
| AJaeger | another problem: (see https://review.openstack.org/#/c/511178/ for revert): after merge of https://review.openstack.org/#/c/510237/ we get errors that a job does not exist - that is was added as part of 510237. This happens with some openstack-zuul-job changes but apparently not all. I proposed the revert | 11:02 |
| fungi | we may need to cancel maintenance | 11:02 |
| fungi | infra-root: heads up ^ | 11:02 |
| *** mrhillsman has quit IRC | 11:03 | |
| *** nicolasbock has joined #openstack-infra | 11:03 | |
| fungi | yolanda: just to clear this up for me, can you explain what caused you to resort to using git push rather than the submit button in gerrit the rest of us have been using for similar circumstances? | 11:03 |
| *** masuberu has joined #openstack-infra | 11:05 | |
| yolanda | fungi, i usually do it with the force push | 11:05 |
| fungi | okay, good news is this looks like it's probably fast-forwardable | 11:05 |
| *** mrhillsman has joined #openstack-infra | 11:05 | |
| fungi | yolanda: please don't ever, ever git push directly into a live repo in gerrit, and whatever you do only ever add --force if fast-forward pushing breaks | 11:05 |
| *** ganso has joined #openstack-infra | 11:06 | |
| fungi | we rely on direct git push exclusively for importing outside repository state, but everything else should be driver through gerrit unless gerrit itself is entirely offline (in which case you're stuck merging locally from a shell on gerrit anyway) | 11:07 |
| fungi | s/driver/driven/ | 11:07 |
| *** masber has quit IRC | 11:07 | |
| yolanda | ok good to know, because the times i had to force a merge when something was broken, i always did with git push | 11:07 |
| AJaeger | fungi, so this merged in project-config according to gerrit but is not in git: https://review.openstack.org/#/c/508656 - just for double checking once you resolved this | 11:07 |
| fungi | in the future, when you have an absolute need to bypass the ci system, add yourself to project bootstrappers, remove any lingering -2 verify vote, add a +2 verify (and code review +2 and workflow +1 if missing) and then use the submit button to cause gerrit to merge ituse the submit button in gerrit | 11:08 |
| yolanda | ++ | 11:09 |
| *** andreas_s has joined #openstack-infra | 11:09 | |
| *** fanzhang_ has quit IRC | 11:13 | |
| *** fanzhang has quit IRC | 11:13 | |
| fungi | #status log deleted the errant review/andreas_jaeger/zuulv3-unbound branch from the openstack-infra/project-config repository (formerly at commit 2e8ae4da5d422df4de0b9325bd9c54e2172f79a0) | 11:13 |
| openstackstatus | fungi: finished logging | 11:13 |
| Shrews | fungi: i'm "awake" (note the quotes) if you need help with anything | 11:15 |
| *** sshnaidm|afk is now known as sshnaidm | 11:15 | |
| fungi | thanks Shrews! it's mostly gerrit surgery at the moment. know any java? ;) | 11:15 |
| fungi | infra-root: i'm going to work on piecing together the list of merged project-config changes which ended up parented to the 2e8ae4da5d422df4de0b9325bd9c54e2172f79a0 commit on the wrong branch | 11:15 |
| dirk | *shhh* run, if you can, run!! | 11:15 |
| Shrews | *grumble* | 11:16 |
| fungi | current state as best i can piece together is that a gerrit change targeted for master was directly pushed into a new branch, gerrit thought it was merged at that point and allowed other changes to merge but they're on some detached head and not actually master | 11:17 |
| Shrews | eek | 11:18 |
| *** shardy is now known as shardy_lunch | 11:19 | |
| fungi | https://review.openstack.org/507905 is the last change to actually merge to master as far as the replicas are concerned | 11:21 |
| fungi | and `git show master` on the filesystem on gerrit confirms | 11:21 |
| *** udesale has joined #openstack-infra | 11:21 | |
| fungi | so at least this doesn't appear to have broken replication, we're just unable to merge new changes to master in this state (instead they're merging into a black hole) | 11:22 |
| AJaeger | but changes after 510769 are missing | 11:22 |
| *** lucasagomes is now known as lucas-hungry | 11:23 | |
| AJaeger | So, 510769 merged, 510728 merged, 511157 merged, and then 507905 - according to git | 11:24 |
| AJaeger | according to http://git.openstack.org/cgit/openstack-infra/project-config/ | 11:24 |
| fungi | yolanda: roughly what time did you push the revert into gerrit? | 11:24 |
| AJaeger | compare https://review.openstack.org/#/q/project:openstack-infra/project-config+status:merged | 11:24 |
| fungi | looking now, i think what actually happened is that the branch got rewound and then this change merged on an earlier state | 11:25 |
| *** larainema has quit IRC | 11:25 | |
| fungi | in which case we _may_ simply be able to repropose all the previously merged changes and merge them normally | 11:25 |
| AJaeger | fungi: around 7:40 UTC (will do from here) | 11:25 |
| AJaeger | fungi: around 7:40 UTC (she said "will do" at that time) | 11:26 |
| yolanda | yep, that time, when AJaeger asked | 11:26 |
| yolanda | i did at the same moment | 11:26 |
| AJaeger | gerrit records 7:51 UTC | 11:26 |
| Shrews | fungi: could we cherry-pick the missing changes fromt he detached head onto master? | 11:26 |
| fungi | Shrews: they'll end up with a different commit id | 11:27 |
| Shrews | eh | 11:27 |
| fungi | pretty sure that'll confuse gerrit | 11:27 |
| Shrews | yeah | 11:27 |
| fungi | okay, so here's my revised theory now having looked at the situation: | 11:28 |
| fungi | changes merged up through 509937 at 06:56z | 11:29 |
| *** kjackal_ has joined #openstack-infra | 11:29 | |
| *** markvoelker has quit IRC | 11:30 | |
| *** boden has quit IRC | 11:30 | |
| *** ldnunes has joined #openstack-infra | 11:34 | |
| fungi | then yolanda checked out 511157 but left it parented on an earlier master branch state from where 511075 had merged and pushed that at 07:40z, rolling the branch back by 14(?) changes and various associated merge commits | 11:34 |
| fungi | after that, gerrit happily merged 507905 on top of the new branch state | 11:35 |
| *** alexchadin has joined #openstack-infra | 11:35 | |
| AJaeger | yes, 511075 merged which caused lots of RETRY_LIMIT errors and I used gerrit to propose a revert which is 511157 | 11:35 |
| fungi | so at this point, we have two choices: | 11:35 |
| fungi | (that i can think of) | 11:36 |
| fungi | 1. reset the master branch state to the last change that merged before this incident and then repropose the revert and the change which merged after it and merge them again | 11:36 |
| *** witek has joined #openstack-infra | 11:37 | |
| fungi | this will be a non-fast-forwardable change from the current branch state, but it will again become fast-forwardable from the prior branch state (which is what most of our systems are probably still stuck at) | 11:37 |
| fungi | or... | 11:37 |
| fungi | 2. leave the master branch as-is and repropose all the (14?) missing changes | 11:38 |
| fungi | this will be fast-forwardable from the _current_ branch state, but i think leaves our systems which had an earlier branch state from sometime between 06:56 and 07:40 stuck unable to update without manual assistance | 11:38 |
| fungi | i'm leaning toward option #1 on the odds that more of our systems are likely stuck unable to fast-forward right now | 11:39 |
| *** jpena is now known as jpena|lunch | 11:42 | |
| * AJaeger joins fungi in favoring #1 | 11:42 | |
| fungi | for #1, i need to work out what the commit id of the merge commit that stitched 509937 into master was, which i think i should be able to find from zuul's debug log since it will have seen the merge commit and enqueued post jobs | 11:43 |
| *** spectr has quit IRC | 11:44 | |
| fungi | or, actually, gerrit's on disk reflog probably has it. i'll check that first | 11:44 |
| AJaeger | fungi, I'll be offline for the next hour and cannot give you further encouragement. Thanks a lot for fixing this! | 11:45 |
| fungi | thanks for the detailed record AJaeger! | 11:45 |
| *** spectr has joined #openstack-infra | 11:46 | |
| fungi | nope, the reflog seems to omit the same commits as the git log. i'll go hunting in zuul's debug logs | 11:48 |
| AJaeger | fungi: once that is fixed, please recheck one of the changes that I collected in the etherpad for the issues "elease-openstack-javascript is undefined" - see line 73. I have a revert up if needed but I smell some connection between these issues, so let's handle these one by one... | 11:48 |
| AJaeger | fungi, ignore the above for now - and bookmark for later, please ;) | 11:48 |
| AJaeger | bbl | 11:48 |
| fungi | thanks | 11:48 |
| *** mat128 has joined #openstack-infra | 11:50 | |
| fungi | zuul to the rescue! the last "sane" branch state was the merge commit at 55da0b6986b546f4a80e043b4a80561c0a4bc797 | 11:52 |
| fungi | Merge "Adding check/gate jobs to Contributor Guide" | 11:52 |
| fungi | that has 53e230969e179e3c79a6a4cf8dc3b21b166614b6 (change 509937) as a parent | 11:53 |
| *** spectr-RH has joined #openstack-infra | 11:54 | |
| fungi | i'm going to resort to git push --force to reset the project-config master branch state from 5a7168229f0f73d6b2d5a3e8859d0a8974462792 (Merge "Add new repo for manila tempest plugin") to 55da0b6986b546f4a80e043b4a80561c0a4bc797 (Merge "Adding check/gate jobs to Contributor Guide") | 11:54 |
| *** stakeda has quit IRC | 11:55 | |
| *** claudiub|2 has joined #openstack-infra | 11:55 | |
| fungi | this will be a non-fast-forwardable change from the current branch state, but should be fast-forwardable from any time prior to 07:40z | 11:55 |
| *** tpsilva has joined #openstack-infra | 11:55 | |
| *** yamamoto has joined #openstack-infra | 11:55 | |
| *** gildub has quit IRC | 11:56 | |
| fungi | it _will_ completely orphan 511157 (Revert "Add configure-unbound to base job") and 507905 (Add new repo for manila tempest plugin) which i will repropose shortly parented on the corrected master branch | 11:56 |
| fungi | (using new gerrit change ids obviously) | 11:56 |
| *** eharney has joined #openstack-infra | 11:57 | |
| fungi | i'm also making a complete cp -ax of the current ~gerrit2/review_site/git/openstack-infra/project-config.git into ~fungi/project-config.git first | 11:57 |
| *** spectr has quit IRC | 11:57 | |
| *** jcoufal has joined #openstack-infra | 11:58 | |
| sshnaidm | infra-root, we have a problem with setup workspace in tripleo jobs: rm: cannot remove ‘/home/zuul/cache/files/cirros-0.3.5-x86_64-disk.vhd.tgz’: Permission denied | 11:58 |
| sshnaidm | http://logs.openstack.org/25/446525/19/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq/240206f/logs/devstack-gate-setup-workspace-new.txt | 11:58 |
| *** yamamoto has quit IRC | 11:59 | |
| fungi | sshnaidm: thanks for the heads up, can you see if it's already on https://etherpad.openstack.org/p/zuulv3-issues and add it if not? we haven't started the maintenance yet anyway, another emergency cropped up i'm trying to deal with first | 12:00 |
| sshnaidm | fungi, sure | 12:00 |
| *** dave-mccowan has joined #openstack-infra | 12:00 | |
| *** larainema has joined #openstack-infra | 12:00 | |
| *** lucas-hungry is now known as lucasagomes | 12:01 | |
| fungi | ug, gerrit won't let me pull 55da0b6986b546f4a80e043b4a80561c0a4bc797 unless it belongs to a current branch or tag i think. i'll make a temporary branch from it | 12:02 |
| *** andreas_s has quit IRC | 12:02 | |
| *** andreas_s has joined #openstack-infra | 12:02 | |
| *** Hal has joined #openstack-infra | 12:02 | |
| *** dprince has joined #openstack-infra | 12:03 | |
| *** Hal is now known as Guest61531 | 12:03 | |
| *** edmondsw has joined #openstack-infra | 12:04 | |
| *** thorst has joined #openstack-infra | 12:06 | |
| fungi | i have created temporary branch "sane-master" from 55da0b6986b546f4a80e043b4a80561c0a4bc797, fetched that, then reset --hard my local master branch to it | 12:06 |
| mnaser | good morning (and good luck!) infra :) | 12:07 |
| openstackgerrit | Merged openstack-infra/project-config master: neutron-lbaas: Add neutron requirments for tox jobs https://review.openstack.org/509143 | 12:08 |
| *** yamamoto has joined #openstack-infra | 12:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use zuul_legacy_vars for propose jobs for now https://review.openstack.org/511126 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Pass root_marker_dir to write-root-marker https://review.openstack.org/511122 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Add an origin remote for tag-releases for notes https://review.openstack.org/511004 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Add release-openstack-javascript job https://review.openstack.org/502185 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Adding check/gate jobs to Contributor Guide https://review.openstack.org/509937 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Add post tarball gate to kuryr-tempest-plugin https://review.openstack.org/510786 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: adds missing required-projects to n8g-(bagpipe|bgppvn) https://review.openstack.org/510435 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove project pylockfile from infrastructure systems https://review.openstack.org/508656 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove the legacy oslo.messaging project jobs https://review.openstack.org/510659 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron to the project dependecy list for zuul https://review.openstack.org/510347 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Add new repo for magnum tempest plugin https://review.openstack.org/501231 | 12:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Add python-tempestconf project https://review.openstack.org/508502 | 12:08 |
| fungi | apologies for the spam, but that was apparently the confirmation from gerritbot following my execution of `git push --force gerrit master` | 12:08 |
| fungi | those are as best as i can tell the commits which had previously been unwound | 12:09 |
| fungi | now i'll get to work on reproposing 511157 and 507905 | 12:09 |
| *** liusheng has quit IRC | 12:10 | |
| *** liusheng has joined #openstack-infra | 12:10 | |
| *** shardy_lunch is now known as shardy | 12:11 | |
| *** andreas_s has quit IRC | 12:11 | |
| *** spectr-RH has quit IRC | 12:13 | |
| *** spectr has joined #openstack-infra | 12:13 | |
| fungi | okay, so looking at the history, it looks like 511075 (Add configure-unbound to base job) merged immediately after what is now the new (restored) master branch state so i'll find its merge commit (if any) and wind forward to that with push (no --force should be needed now) | 12:14 |
| * fungi returns to the zuul debug log to check for merge commits | 12:14 | |
| *** spectr-RH has joined #openstack-infra | 12:14 | |
| fungi | yeah, there definitely would have been a separate merge commit for that one, owing to the missing commits which followed its direct parent | 12:15 |
| *** panda|off is now known as panda | 12:15 | |
| *** Guest61531 has quit IRC | 12:15 | |
| *** liusheng has quit IRC | 12:15 | |
| * Shrews watches fungi exercise his git-foo and is impressed | 12:16 | |
| *** andreas_s has joined #openstack-infra | 12:16 | |
| *** mat128_ has joined #openstack-infra | 12:17 | |
| *** spectr has quit IRC | 12:18 | |
| *** openstackgerrit has quit IRC | 12:18 | |
| *** markvoelker has joined #openstack-infra | 12:18 | |
| *** mat128 has quit IRC | 12:19 | |
| fungi | courtesy of zuul logs, 8685183d1c48afbb5e095d6c727c9acc199353f1 is the merge commit i want there. recreatnig my temporary "sane-master" from that now | 12:19 |
| *** andreas_s has quit IRC | 12:21 | |
| fungi | i have checked out 8685183d1c48afbb5e095d6c727c9acc199353f1 and reset --hard my local master branch to that | 12:21 |
| fungi | er, s/checked out/fetched/ | 12:21 |
| *** andreas_s has joined #openstack-infra | 12:21 | |
| *** openstackgerrit has joined #openstack-infra | 12:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add configure-unbound to base job https://review.openstack.org/511075 | 12:23 |
| fungi | and there's the other missing commit ^ | 12:24 |
| fungi | now to repropose its orphaned revert | 12:24 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Revert "Add configure-unbound to base job" https://review.openstack.org/511215 | 12:25 |
| fungi | that's ^ the reproposal of 511157 with a new change-id | 12:25 |
| *** andreas_s has quit IRC | 12:25 | |
| *** andreas_s has joined #openstack-infra | 12:26 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Add new repo for manila tempest plugin https://review.openstack.org/511216 | 12:27 |
| fungi | and that's ^ the reproposal of 507905 with a new change-id | 12:27 |
| fungi | i'll reapprove those now | 12:27 |
| mnaser | fungi maybe you've thought about this, but will puppet runs for zuul reconfig likely have a problem updating projet-config locally now (if they ran once inbetween?) | 12:27 |
| mnaser | assuming the servers consuming project-config do a git pull | 12:28 |
| *** rlandy has joined #openstack-infra | 12:28 | |
| fungi | the expectation is that they were already in that state. this surgery is putting the branch back into a fast-forwardable state from its earlier state | 12:29 |
| mnaser | git magic | 12:29 |
| fungi | not really magic. more like a really big hammer | 12:29 |
| *** trown|outtypewww is now known as trown | 12:29 | |
| fungi | this is really less like surgery and more like a butcher shop | 12:30 |
| mnaser | takes still to use a big hammer without breaking things :p | 12:30 |
| mnaser | s/still/skill/ | 12:30 |
| *** andreas_s has quit IRC | 12:30 | |
| *** priteau has quit IRC | 12:30 | |
| mnaser | also, configure-unbound, i think `ansible_default_ipv6` is undefined on machines that dont have ipv6, so instead of "when: ansible_default_ipv6.address is defined" we could probably do "ansible_default_ipv6.address" | 12:30 |
| mnaser | err | 12:31 |
| mnaser | "when ansible_default_ipv6 is defined" | 12:31 |
| *** andreas_s has joined #openstack-infra | 12:31 | |
| *** yamamoto has quit IRC | 12:31 | |
| mnaser | i'll pick that up and make a patch to add it to base-test to test out (unless someone doesnt pick it up), but coffee time | 12:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Add configure-unbound to base job" https://review.openstack.org/511215 | 12:31 |
| fungi | manually submitted via gerrit ^ since the parent it's reverting breaks testing it | 12:32 |
| persia | Shrews: Not that it matters much, but the usual transliteration is "fu" or "fū" | 12:32 |
| *** sshnaidm is now known as sshnaidm|mtg | 12:32 | |
| persia | fungi: For those of us following along at home, was there a push before the bot-reported merge of 511075? | 12:33 |
| fungi | persia: yep, when gerritbot reported merging 511075 that was the result of me running `git push gerrit master` (no --force) with my local master reset to 8685183d1c48afbb5e095d6c727c9acc199353f1 | 12:35 |
| *** jpena|lunch is now known as jpena | 12:35 | |
| *** wolverineav has quit IRC | 12:35 | |
| fungi | it was basically fast-forwarding the branch one change, since that change had actually merged normally previously after the state i had initially repaired master to | 12:35 |
| fungi | in short, i miscalculated the last sane branch state by one change (and its corresponding merge commit) so wound it forward to that one | 12:36 |
| fungi | finally, now i have single-core approved 511216 in hopes that it merges normally | 12:36 |
| fungi | i'm also deleting my temporary "sane-master" branch in gerrit again now | 12:37 |
| persia | fungi: Thanks for the confirmation. That's how I thought it worked, but you hadn't reported the specific push command (and had reported the first push command), so I wondered if some other magic had applied. | 12:38 |
| fungi | persia: nope, you're absolutely right thanks for reminding me i had failed to report one step there | 12:38 |
| *** thorst_ has joined #openstack-infra | 12:38 | |
| *** esberglu has quit IRC | 12:38 | |
| *** thorst has quit IRC | 12:39 | |
| *** andreas_s has quit IRC | 12:40 | |
| persia | Does project-config get special priority currently for zuulv3 reasons, or do we expect to wait ~4 hours for 511216 to safely merge? | 12:41 |
| fungi | there is a separate infra-check pipeline for project-config which has highest priority in zuulv3's config for now | 12:42 |
| persia | Does that help the merge? I thought it was still zuulv2.5 that was doing the actual merges (to make sure zuul+gerrit *can* merge 511216 safely atop the now sane master) | 12:43 |
| persia | Or does just performing a verify test show that the history rewrite was successful? | 12:43 |
| fungi | zuulv3 has authority over a (very few) infra config repos at the moment | 12:43 |
| *** Hal has joined #openstack-infra | 12:44 | |
| fungi | unlike for basically all other projects | 12:44 |
| *** kiennt26_ has joined #openstack-infra | 12:44 | |
| *** Hal is now known as Guest40292 | 12:44 | |
| persia | Ah, good. | 12:44 |
| fungi | zuulv3 just now noticed 511215 merged and has requeued (previously failing) check jobs for 511216, so fingers crossed | 12:44 |
| fungi | oh, it's already moved into infra-gate in fact | 12:44 |
| fungi | eta 4 minutes to normal merge | 12:45 |
| *** kiennt26 has quit IRC | 12:45 | |
| *** kiennt26_ has quit IRC | 12:45 | |
| *** andreas_s has joined #openstack-infra | 12:45 | |
| *** kiennt26 has joined #openstack-infra | 12:46 | |
| *** alexchadin has quit IRC | 12:46 | |
| *** jcoufal has quit IRC | 12:47 | |
| *** eumel8 has quit IRC | 12:47 | |
| *** jcoufal has joined #openstack-infra | 12:48 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add new repo for manila tempest plugin https://review.openstack.org/511216 | 12:49 |
| fungi | infra-root: ^ looks like project-config is working normally again now with no missing changes (discounting the orphaned two i had to repropose into gerrit). it's probably safe to proceed with the zuulv3 rollout maintenance now, that is if we still think it's not too late to begin at this point | 12:49 |
| fungi | pabelanger: mordred: you said you'd be awake to help... around yet? | 12:50 |
| AJaeger | thanks, fungi. I'm back online... | 12:50 |
| * cmurphy hi-fives fungi | 12:54 | |
| fungi | thanks cmurphy! | 12:54 |
| AJaeger | fungi, I rechecked some of the changes with the issue I mentioned - let's see whether it works now | 12:54 |
| fungi | AJaeger: appreciated | 12:54 |
| *** andreas_s has quit IRC | 12:55 | |
| fungi | AJaeger: note that 511216 went through the normal ci process (fast-reviewed, approved, passed check, passed gate, zuul merged it) | 12:55 |
| fungi | so in theory we should be free of any widespread issue with project-config at this point | 12:55 |
| *** alexchadin has joined #openstack-infra | 12:55 | |
| *** ykarel has quit IRC | 12:56 | |
| fungi | but always good to double-check ;) | 12:56 |
| AJaeger | fungi: I hope so as well... | 12:56 |
| *** ykarel has joined #openstack-infra | 12:56 | |
| *** esberglu has joined #openstack-infra | 12:57 | |
| *** dizquierdo has quit IRC | 12:58 | |
| AJaeger | fungi, yes looks good - zuul has started execution, so the errors are gone. I'll remove the content from etherpad and abandon the job. | 12:59 |
| AJaeger | We're green again... | 12:59 |
| fungi | thanks for confirming, AJaeger! | 12:59 |
| *** andreas_s has joined #openstack-infra | 12:59 | |
| fungi | i feel pretty confident we can push forward with the rollout finally, though we're two hours behind scheduled start time at this point and still missing pabelanger and mordred it seems | 13:00 |
| fungi | so we may need to do a mass batch-enqueue of changes from zuul v2 into v3 at this point | 13:00 |
| *** hasharAway is now known as hashar | 13:01 | |
| fungi | also the rollout steps we have in https://etherpad.openstack.org/p/zuulv3-cutover don't seem to be updated with the new change urls? | 13:02 |
| fungi | did a new version of the rollout plan get documented elsewhere? | 13:02 |
| *** openstackgerrit has quit IRC | 13:03 | |
| Shrews | fungi: i was looking for one a few moments ago and did not find one. not sure if there is an updated one | 13:03 |
| fungi | #status notice Due to unrelated emergencies, the Zuul v3 rollout has not started yet; stay tuned for further updates | 13:04 |
| openstackstatus | fungi: sending notice | 13:04 |
| mordred | fungi: morning! | 13:05 |
| fungi | a wayward mordred arrives in a puff of smoke | 13:05 |
| -openstackstatus- NOTICE: Due to unrelated emergencies, the Zuul v3 rollout has not started yet; stay tuned for further updates | 13:05 | |
| mordred | fungi: I have poorly timezoned - sorry - I lodged it in my brain for some stupid reason as 12:00 local time - which is clearly stupid | 13:06 |
| *** Guest40292 has quit IRC | 13:06 | |
| mordred | fungi: what's the tl;dr on the scrollback - or should I just read it? | 13:06 |
| fungi | heh, no problem. you got here just in time to miss all the project-config repair excitement | 13:06 |
| openstackstatus | fungi: finished sending notice | 13:07 |
| mordred | \o/ | 13:07 |
| mordred | did I break project-config? sounds lke something I'd be the cause of | 13:07 |
| *** bobh has joined #openstack-infra | 13:07 | |
| fungi | mordred: summary is that there was an unfortunate git push --force into project-config around 07:40 in a misguided attempt to bypass ci for reverting a deadlock on a job config | 13:07 |
| mordred | ah, nod | 13:08 |
| fungi | took ~2 hours to repair the repo into a sane state fast-forwardable from its earlier commits | 13:08 |
| mordred | the old 'everytime we bypass CI we break something' trope | 13:08 |
| fungi | once i woke up and got coffee | 13:08 |
| mordred | we're like a walking example of why our CI system exists :) | 13:08 |
| fungi | yeah, in this case it was less the bypassing ci merging broken content, and rather unsafe process/method followed which broke the repository itself | 13:09 |
| mordred | nod | 13:10 |
| fungi | granted there was a chain of incorrect process | 13:11 |
| *** sdake has quit IRC | 13:11 | |
| *** sdake has joined #openstack-infra | 13:11 | |
| *** sdake has quit IRC | 13:11 | |
| *** sdake has joined #openstack-infra | 13:11 | |
| fungi | the change which initially deadlocked our zuulv3 configuration was due to approving an untested addition to the base job | 13:11 |
| fungi | which we already know to be dangerous | 13:11 |
| fungi | but then the need to bypass ci when reverting that was met with a git push --force of an incorrect and non-fast-forwardable repo state, rather than using the submit button in gerrit | 13:12 |
| *** openstackgerrit has joined #openstack-infra | 13:12 | |
| openstackgerrit | Merged openstack-infra/project-config master: networking-bgpvpn: add translation jobs https://review.openstack.org/509178 | 13:12 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy watcher jobs https://review.openstack.org/510849 | 13:12 |
| *** kgiusti has joined #openstack-infra | 13:13 | |
| *** mriedem has joined #openstack-infra | 13:13 | |
| *** mat128_ has quit IRC | 13:14 | |
| mordred | fungi: submit button always better ... is there anything I can do to help anything at the moment or anything you'd like meto look at? | 13:15 |
| *** dbecker has joined #openstack-infra | 13:15 | |
| *** sdake has quit IRC | 13:15 | |
| fungi | nope, i think we're back on track, i was just looking for the updated rollout plan with the correct change urls | 13:15 |
| fungi | and waiting for you and pabelanger to appear | 13:15 |
| fungi | anyway, i'm beginning to worry that we lack the bandwidth to provide sufficient process documentation and mentoring to our sysadmins in apac and emea timezones, so they're not as aware of safe procedures the rest of us have established and end up having to guess (sometimes poorly) how to handle such situations | 13:17 |
| fungi | which is something we'll all need to work on/help fix | 13:17 |
| mordred | yah. I agree | 13:18 |
| mordred | fungi: I've only got the old etherpad plan ... | 13:18 |
| fungi | mordred: yeah, that's all i found as well. pabelanger proposed reverted reverts and related patches for those, right? | 13:19 |
| *** sdake has joined #openstack-infra | 13:19 | |
| fungi | i think i remember hmi saying he put them under a consistent review topic. checking | 13:19 |
| mordred | https://review.openstack.org/#/q/status:open+topic:zuulv3-rollout | 13:19 |
| mordred | fungi: ^^ | 13:19 |
| *** sdake has quit IRC | 13:19 | |
| *** sdake has joined #openstack-infra | 13:19 | |
| fungi | perfect--thanks! | 13:19 |
| * mordred willupdate etherpad real quick | 13:19 | |
| *** zhurong has joined #openstack-infra | 13:20 | |
| mordred | fungi: actually - I made a new one- https://etherpad.openstack.org/p/zuulv3-rollout | 13:20 |
| *** baoli has joined #openstack-infra | 13:22 | |
| fungi | even better. i've added that url to the old pad for those who come later | 13:22 |
| *** gouthamr has joined #openstack-infra | 13:23 | |
| *** markmcd has quit IRC | 13:23 | |
| *** mpaolino has joined #openstack-infra | 13:23 | |
| mordred | fungi: should we prepare revert reverts? | 13:25 |
| *** markmcd has joined #openstack-infra | 13:26 | |
| mordred | I guess they're easy enough to make if we need them | 13:26 |
| *** wolverineav has joined #openstack-infra | 13:27 | |
| fungi | yeah, if someone else wants to work on proposing those while maintenance is underway, that's pretty low-hanging fruit too | 13:28 |
| *** Guest41495 has quit IRC | 13:29 | |
| mordred | fungi: well - it's not that many steps to do - we could likely get the ball rolling without pabelanger | 13:31 |
| EmilienM | fungi: can we merge a patch in openstack/release? or should we wait? | 13:31 |
| *** lamt has joined #openstack-infra | 13:31 | |
| fungi | EmilienM: we haven't started yet, so should be fine to go ahead | 13:32 |
| *** vhosakot has joined #openstack-infra | 13:33 | |
| *** jaypipes has joined #openstack-infra | 13:35 | |
| *** priteau has joined #openstack-infra | 13:35 | |
| *** sree has joined #openstack-infra | 13:36 | |
| *** tosky_ has joined #openstack-infra | 13:36 | |
| mordred | AJaeger: did you do anything to fix the emacs missing on the debian-jessie integration test yet? | 13:37 |
| mordred | fungi: if you didn't see yet: http://logs.openstack.org/28/511128/1/infra-check/multinode-integration-debian-jessie/2e4a52c/job-output.txt.gz#_2017-10-11_08_29_09_050799 | 13:37 |
| *** tosky has quit IRC | 13:38 | |
| mordred | fungi: I think we need to consider that a blocker to rollout - since it'll block literally any patch to ozj ... can't decide if we should just disable the jessie test for a second or try to figure out what's up with emacs | 13:38 |
| *** jistr is now known as jistr|mtg | 13:38 | |
| *** pblaho has quit IRC | 13:38 | |
| fungi | mordred: i _think_ that's what the untested addition to the base job was attempting to address | 13:39 |
| mordred | gotcha | 13:40 |
| mordred | fungi: do you have a link happy to the untested addition? | 13:41 |
| fungi | so maybe worth spending a couple minutes revisiting what ianw had figured out and seeing if we can iterate on it more safely | 13:41 |
| AJaeger | mordred: a recheck healed it ;) | 13:41 |
| *** pblaho has joined #openstack-infra | 13:41 | |
| mordred | oh it did? | 13:41 |
| mordred | AJaeger: it looks still broken | 13:42 |
| AJaeger | mordred: indeed - too much going on ;( | 13:42 |
| fungi | mordred: ianw's attempt to fix is https://review.openstack.org/511075 | 13:42 |
| AJaeger | mordred: https://review.openstack.org/#/c/511071/ worked and was run later | 13:43 |
| *** udesale has quit IRC | 13:43 | |
| mordred | fungi, AJaeger: I think I'd like to sugges that we remove the jessie multinode integration test - and then we can iterate on fixing the problem with a patch re-adding the job | 13:44 |
| mordred | because iterating on that job, especially if we need fixes to the base job, could take all day | 13:45 |
| AJaeger | mordred: go for it... | 13:46 |
| *** mat128 has joined #openstack-infra | 13:46 | |
| *** srobert has joined #openstack-infra | 13:46 | |
| fungi | mordred: we have an updated failure example from after i fixed up the project-config repo? | 13:46 |
| fungi | just want to be absolutely certain the issue still remains before we disable what may be a now working job | 13:47 |
| mordred | fungi: AJaeger just rechecked the patch | 13:48 |
| * AJaeger rechecked it, not run yet | 13:48 | |
| *** mat128 has quit IRC | 13:48 | |
| fungi | if it fails, then i'm +3 on temporarily disabling that job | 13:48 |
| AJaeger | wait, multinode-itegration passed | 13:48 |
| AJaeger | http://logs.openstack.org/28/511128/1/infra-check/multinode-integration-debian-jessie/891ae5f/ | 13:49 |
| mordred | woot | 13:49 |
| fungi | so no need to disable anything there? | 13:49 |
| *** mat128 has joined #openstack-infra | 13:49 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci master: Enable repo for python-os-testr https://review.openstack.org/490032 | 13:49 |
| AJaeger | http://logs.openstack.org/28/511128/1/infra-check/base-integration-debian-jessie/8c6dfc5/ passed as well | 13:49 |
| *** armax has joined #openstack-infra | 13:49 | |
| AJaeger | fungi: I would say: Not. But we need to keep a close eye on these... | 13:53 |
| *** hashar is now known as hasharAway | 13:55 | |
| *** udesale has joined #openstack-infra | 13:55 | |
| *** links has quit IRC | 13:56 | |
| *** eumel8 has joined #openstack-infra | 13:57 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/storyboard-webclient master: Don't show underline when hovering on task titles https://review.openstack.org/511246 | 13:58 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/storyboard-webclient master: Use cursor when hovering over a read-only card https://review.openstack.org/511247 | 13:58 |
| *** alexchadin has quit IRC | 13:58 | |
| AJaeger | argh, faileld here - http://logs.openstack.org/48/510848/2/infra-check/multinode-integration-debian-jessie/d618c7e/ | 13:59 |
| AJaeger | mordred: ^ | 13:59 |
| fungi | so it's intermittent? | 13:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add documentation on force-merging a change https://review.openstack.org/511248 | 13:59 |
| fungi | thanks jeblair! | 14:00 |
| jeblair | good morning! | 14:00 |
| *** jistr|mtg is now known as jistr | 14:00 | |
| AJaeger | " Cannot initiate the connection to security.debian.org:80 (2001:a78:5:0:216:35ff:fe7f:be4f). - connect (101: Network is unreachable)" | 14:00 |
| AJaeger | fungi: random ;( | 14:00 |
| AJaeger | morning, jeblair | 14:01 |
| pabelanger | fungi: sorry, I am running late this morning | 14:01 |
| pabelanger | prepared now | 14:01 |
| pabelanger | had some issues with internets | 14:01 |
| AJaeger | jeblair: let me fix a typo in it quickly... | 14:01 |
| openstackgerrit | Andreas Scheuring proposed openstack/diskimage-builder master: Fix rendering issues in DIB "building_an_image" doc https://review.openstack.org/511250 | 14:02 |
| fungi | AJaeger: interesting... according to this it ran in ovh-gra1 and failed reaching a v6 global address even though it has no global v6 routes: http://logs.openstack.org/48/510848/2/infra-check/multinode-integration-debian-jessie/d618c7e/zuul-info/zuul-info.primary.txt | 14:02 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config master: Add documentation on force-merging a change https://review.openstack.org/511248 | 14:02 |
| fungi | pabelanger: no worries, this morning has been a bust all 'round | 14:02 |
| *** gmann is now known as gmann_afk | 14:03 | |
| pabelanger | fungi: :) Where should I be focusing? | 14:03 |
| AJaeger | fungi: so, that means we have problems with the test unbound change - that we only run in openstack-zuul-jobs? We could revert that as well... | 14:03 |
| *** spectr-RH has quit IRC | 14:04 | |
| pabelanger | I've removed my WIP from rolllout patches | 14:04 |
| jeblair | AJaeger: i thought unbound was completely reverted | 14:04 |
| AJaeger | fungi, mordred https://review.openstack.org/#/c/510728/ is still in | 14:04 |
| AJaeger | jeblair: ^ | 14:04 |
| *** spectr has joined #openstack-infra | 14:04 | |
| *** chlong has joined #openstack-infra | 14:04 | |
| jeblair | AJaeger: yes, but nothing should use base-test | 14:05 |
| fungi | AJaeger: i'm having trouble finding the error you quoted in http://logs.openstack.org/48/510848/2/infra-check/multinode-integration-debian-jessie/d618c7e/job-output.txt.gz | 14:06 |
| AJaeger | jeblair: indeed, now I'm confused | 14:06 |
| jeblair | AJaeger: those jobs inherit from base-minimal, but that shouldn't have unbound | 14:07 |
| AJaeger | fungi, http://logs.openstack.org/48/510848/2/infra-check/multinode-integration-debian-jessie/d618c7e/ara/ - and then go to multinode.yaml page 2 and see the failure | 14:07 |
| AJaeger | " | 14:07 |
| AJaeger | "configure-mirrors : Update apt cache" is the failure that gave the message | 14:07 |
| pabelanger | http://logs.openstack.org/48/510848/2/infra-check/multinode-integration-debian-jessie/d618c7e/job-output.txt.gz#_2017-10-11_13_50_49_650057 | 14:08 |
| pabelanger | ya, error doesn't make it way into job-output.txt | 14:08 |
| pabelanger | but ARA will show it | 14:08 |
| fungi | AJaeger: thanks, i can see in ara that playbook(?) failed but i'm still not familiar with navigating it to get to logged output | 14:08 |
| *** slaweq_ has quit IRC | 14:08 | |
| AJaeger | fungi, neither am I - I just learned that ara helps me locate them quickly ;) | 14:08 |
| fungi | there are no records for the playbook, apparently | 14:08 |
| fungi | ahh, under "tasks" | 14:09 |
| pabelanger | isn't the issue that debian-jessie isn't using our AFS mirrors? | 14:09 |
| pabelanger | I thought I added that to etherpad last time around | 14:09 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add python-openstackclient to legacy-check-osc-plugins https://review.openstack.org/511128 | 14:09 |
| pabelanger | looking | 14:09 |
| jeblair | yeah, then click the red failed button on the task | 14:09 |
| *** tosky_ is now known as tosky | 14:10 | |
| fungi | okay, so there's a very good chance that apt is falling back to attempting ipv6 after failures to reach security.debian.org over ipv4, even though it doesn't have any ipv6 route to be able to reach it (and then only the latter error bubbles up to stdout) | 14:11 |
| AJaeger | mordred: reviewed the etherpad, looks good from change perspective | 14:12 |
| jeblair | root@ubuntu-xenial-ovh-gra1-11333378:~# ping6 security.debian.org | 14:12 |
| jeblair | connect: Network is unreachable | 14:12 |
| mordred | jeblair: can it ping6 other v6 things and just not security.debian.org? | 14:13 |
| fungi | we don't have ipv6 routes | 14:13 |
| fungi | from there | 14:13 |
| *** eharney has quit IRC | 14:13 | |
| jeblair | mordred: nope | 14:13 |
| pabelanger | that is wrong | 14:13 |
| pabelanger | it should be: deb http://mirror.dfw.rax.openstack.org/debian jessie-security main for example | 14:13 |
| pabelanger | so configure-mirror is not correct | 14:13 |
| pabelanger | looking now | 14:14 |
| fungi | are you able to connect to port 80 on the v4 address for security.debian.org from ovh-gra1? | 14:14 |
| *** rajinir-afk is now known as rajinir | 14:14 | |
| jeblair | fungi: using nc yes | 14:14 |
| fungi | okay, so it's not entirely broken | 14:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Also add security.list for Debian on configure-mirror https://review.openstack.org/511253 | 14:17 |
| fungi | so either apt is only trying v6 (and failing because we have no v6 routes in ovh-gra1) or it's getting intermittent access failures over v4 and then falling back to trying v6 even though there's no point | 14:17 |
| pabelanger | jeblair: fungi: ^ should help with security updates | 14:17 |
| mordred | jeblair, fungi, pabelanger: I've added the issue with the apt error not making it into the log to the etherpad so I don't lose it | 14:17 |
| fungi | thanks mordred | 14:17 |
| pabelanger | mordred: 511253 should be our fix | 14:17 |
| *** rbrndt has joined #openstack-infra | 14:19 | |
| fungi | thanks pabelanger! | 14:19 |
| *** martinkopec has quit IRC | 14:20 | |
| AJaeger | So, ready to go forward with Zuul v3? | 14:20 |
| fungi | i'm still mildly weirded out by the errors about failing to reach security.d.o and not sure why it's trying over v6 from a server with no v6 global routes | 14:21 |
| fungi | but willing to let that go for now | 14:21 |
| kiennt26 | hi, sorry for interrupted. i had faced a gate issue related to /home/zuul/cache/files/ permissions : http://logs.openstack.org/15/377215/45/check/legacy-tempest-dsvm-py35/53d21c3/logs/devstack-gate-setup-workspace-new.txt | 14:22 |
| jeblair | fungi: me too, but it does appear that the fix matches the v2 behavior, so i guess best to leave that rabbit hole for later. :) | 14:22 |
| mordred | well - I'd like to see the ozj patch pass after thatconfigure mirrors patch before we do -cause if we can't land ozj patches that'sbad | 14:22 |
| dmsimard | AJaeger: you can also just search for "failed" in the search box of the task panel | 14:23 |
| jeblair | mordred: this may only happen if the debian job lands on ovh-gra1 | 14:24 |
| AJaeger | dmsimard: Would be better to do that at top-level ... | 14:24 |
| AJaeger | dmsimard: normally there's one page and no need to skip to second - but for those it's handy, thanks! | 14:24 |
| mordred | jeblair: oh. bother. k. let's just assume it's good then | 14:24 |
| dmsimard | AJaeger: It's a frontend/UX issue, not a backend one. The data is there, I'm just bad at frontend :p | 14:25 |
| mordred | jeblair, fungi, pabelanger: kiennt26's issue is on the etherpad - it's from 03:26:10.949 though - so might have been the old images without the /home/zuul/cache yeah? | 14:25 |
| jeblair | mordred: good question -- do we have new images? | 14:25 |
| dmsimard | AJaeger: there's a lot of different things we can eventually do with ara -- for example (pseudocode) "ara task list --all |grep failed |xargs |echo 'http://direct link to failure'" | 14:25 |
| pabelanger | mordred: yah, new images were built yesterday | 14:26 |
| mordred | jeblair: the images built last night | 14:26 |
| pabelanger | ~14 hrs ago | 14:26 |
| fungi | mordred: jeblair: i thought ianw built new images, but maybe they got rolled back? or maybe they went into use after 03:26z | 14:26 |
| jeblair | nodepool says it's uploading images to some providers | 14:26 |
| mordred | so it's still potentially working up updating the images | 14:27 |
| mordred | oh - wait | 14:27 |
| pabelanger | yah, nb03 is slow compared to nb04 | 14:27 |
| mordred | that is DeFINITELY using the new images | 14:27 |
| mordred | find /home/zuul/cache/files/ -mindepth 1 -maxdepth 1 -exec mv '{}' is the command it's running | 14:28 |
| mordred | followed by mv: cannot move '/home/zuul/cache/files/cirros-0.3.4-x86_64-disk.vhd.tgz' to '/opt/stack/cache/files/cirros-0.3.4-x86_64-disk.vhd.tgz': Permission denied | 14:28 |
| mordred | so is it possible we didn't get ownership set correctly for the cache files? | 14:28 |
| fungi | is it attempting to perform the mv as stack or something instead of zuul? | 14:28 |
| mordred | or do we need to add a pre-playbook task to chown things? | 14:29 |
| mordred | http://logs.openstack.org/15/377215/45/check/legacy-tempest-dsvm-py35/53d21c3/logs/devstack-gate-setup-workspace-new.txt | 14:29 |
| *** zhurong has quit IRC | 14:29 | |
| mordred | 2017-10-11 03:26:10.950 | 14:29 |
| mordred | (we should gzip that file, btw, so it'll get time links) | 14:29 |
| *** coolsvap has quit IRC | 14:31 | |
| fungi | ahh, judging from that log the files must be owned by not-zuul | 14:31 |
| pabelanger | lrwxrwxrwx 1 root root 16 Oct 11 00:16 files -> /opt/cache/files | 14:31 |
| pabelanger | that is on 15.184.65.134 | 14:31 |
| pabelanger | in /home/zuul | 14:32 |
| fungi | pabelanger: yeah, but who owns the files in there? | 14:32 |
| mordred | jenkins, I believe | 14:32 |
| pabelanger | jenkins because we still don't chown them | 14:32 |
| mordred | because we just made a symlink right? | 14:32 |
| fungi | it's not trying to move the directory/symlink, just trying to move the files out of it | 14:32 |
| fungi | yeah | 14:32 |
| pabelanger | http://nb04.openstack.org/dib.ubuntu-xenial.log | 14:33 |
| pabelanger | 2017-10-11 00:16:31,926 INFO nodepool.image.build.ubuntu-xenial: + chown zuul:zuul /home/zuul/cache | 14:33 |
| pabelanger | is all we do | 14:33 |
| pabelanger | then symlink | 14:33 |
| fungi | i bet that was supposed to have a -R | 14:33 |
| pabelanger | which, jenkins owns | 14:33 |
| jeblair | wait i'm really confused | 14:33 |
| jeblair | devstack-gate only does that if ~/cache/files exists | 14:33 |
| jeblair | so adding that to the images *broke* this | 14:33 |
| mordred | yup | 14:34 |
| jeblair | why did we add it to the images? | 14:34 |
| fungi | jeblair: this is all related to fixes clarkb and ianw were working on last night to try to get v3 working | 14:34 |
| *** udesale has quit IRC | 14:34 | |
| jeblair | yeah, i'm trying to find the issue that prompted it | 14:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Chown /home/zuul/cache before trying to modify it https://review.openstack.org/511255 | 14:34 |
| jeblair | i guess it's http://logs.openstack.org/86/499786/5/check/legacy-neutron-dsvm-api/08219f3/logs/devstacklog.txt.gz#_2017-10-06_15_43_54_924 that? | 14:35 |
| mordred | jeblair: the issue was the v3 legacy dsvm jobs weren't able to use the files cached on disk so were hitting the internet for all of them | 14:35 |
| *** vsaienk0 has quit IRC | 14:35 | |
| mordred | jeblair: yah. also tripleo was failing because of trying to use a cirros image that wasn't there | 14:35 |
| mordred | I think that patch above should 'fix' the issue - at least until we can fix the image builds properly | 14:36 |
| jeblair | i'm going to spend more time understanding this, because i'm currently confused by why the original fix was needed at all | 14:36 |
| pabelanger | Something is going to break, because we cannot have both zuul and jenkins owing /opt/cache/files: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/jenkins-slave/install.d/20-jenkins-slave unless we add them into a shared group? | 14:37 |
| jeblair | what is the difference between /opt/cache/files and /opt/stack/cache/files ? | 14:37 |
| mordred | jeblair: I believe /opt/cache/files is where we put things, and /opt/stack/cache/files is where devstack looks for them | 14:37 |
| fungi | jeblair: i find clarkb talking about it around 21:33z in scrollback | 14:37 |
| jeblair | mordred: agree | 14:37 |
| jeblair | so in http://logs.openstack.org/86/499786/5/check/legacy-neutron-dsvm-api/08219f3/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-10-06_15_42_34_993 | 14:38 |
| jeblair | (which is the "original" problem) | 14:38 |
| *** felipemonteiro has joined #openstack-infra | 14:38 | |
| jeblair | apparently /opt/stack/cache/files doesn't exist | 14:39 |
| pabelanger | doesn't something eventually delete it? | 14:39 |
| jeblair | so it almost looks like devstack-gate was never updated to deal with the /opt/cache/files directory existing | 14:39 |
| *** vsaienk0 has joined #openstack-infra | 14:40 | |
| mordred | "If it's still in home, move it to /opt, this will make sure we have the artifacts in the same filesystem as devstack." | 14:40 |
| *** felipemonteiro_ has joined #openstack-infra | 14:40 | |
| jeblair | mordred: yeah, that made me think someone was updating that to support them being in /opt, but there doesn't seem to be any code to handle them being at /opt/cache/files :/ | 14:40 |
| mordred | jeblair: I think we have a couple of transitions each half-way done - the ~/cache/files is the old place and is there for backwards compat, but then like you said d-g wasn't updated to just work from /opt/cache/files in thefirst place | 14:41 |
| jeblair | so do we rely on the symlink in current devstack as well? | 14:41 |
| mordred | oh wow | 14:41 |
| mordred | look at the next line | 14:41 |
| mordred | we move them from ~/cache to /opt/cache if ~/cache exists - then we copy them to $DEST/devstack/files/ | 14:41 |
| jeblair | well, hardlink but yeah | 14:42 |
| mordred | yah | 14:42 |
| *** spectr has quit IRC | 14:43 | |
| *** felipemonteiro has quit IRC | 14:44 | |
| jeblair | mordred: okay, i think your change will fix it; i think it's a patch on a patch, but unwinding that down to just using /opt/cache is probably something that will take 3 tries to get right :| | 14:44 |
| mordred | jeblair: well - I'm gonna push up a patch for that too | 14:44 |
| mordred | so that we don't forget | 14:44 |
| jeblair | mordred: cool, thx | 14:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files https://review.openstack.org/511260 | 14:45 |
| fungi | okay, so we want 511255 in this case at least for now? | 14:47 |
| *** shardy has quit IRC | 14:47 | |
| jeblair | yeah i think so | 14:48 |
| fungi | cool, lgtm anyway | 14:48 |
| fungi | if it passes then it should be safe | 14:48 |
| jeblair | i updated etherpad | 14:48 |
| fungi | thanks! | 14:49 |
| AJaeger | what about stopping (starting at rollback) our special nodes like proposal slave? We forgot that last time with rollback... | 14:50 |
| fungi | also, i _think_ https://etherpad.openstack.org/p/zuulv3-rollout is ready to go at this point, when we're comfortable proceeding | 14:50 |
| fungi | AJaeger: i thought the issue was we had stopped zlstatic01 and forgot to start it again | 14:50 |
| fungi | do we need to stop the v2 launchers? | 14:51 |
| mordred | fungi: they should be fine - quota will be 0 | 14:52 |
| mordred | infra-root: I'm comfortable as soon as that d-g patch lands - I'm pretty sure ALL v3 d-g jobs will fail until it does :) | 14:52 |
| pabelanger | we might want to eventually to avoid logs from spamming | 14:52 |
| fungi | yeah, that's what i thought... just couldn't recall why we would have stopped them in the previous try | 14:52 |
| mordred | kiennt26: btw - thank you for reporting that issue - turns out it was massively widely important :) | 14:52 |
| pabelanger | fungi: I think we did, but I wasn't here that day | 14:52 |
| AJaeger | fungi: might have been zlstatic01, don't remember which one it was... | 14:53 |
| jeblair | i think that means we have at least 2 hours before we can do anything | 14:53 |
| kiennt26 | oh, my pleasure. Thank all for working on it! | 14:54 |
| pabelanger | think so too | 14:54 |
| jeblair | quite likely more like 3 | 14:54 |
| pabelanger | maybe longer, check is 203 ATM | 14:54 |
| jeblair | well, it's *running* jobs, so the check length doesn't have a huge impact on the time we're waiting. | 14:55 |
| fungi | sshnaidm|mtg: https://review.openstack.org/511255 hopefully also addresses the issue you added to the etherpad | 14:55 |
| AJaeger | we should promote that change once it reaches gate... | 14:56 |
| *** fnordahl has joined #openstack-infra | 14:56 | |
| fungi | we could enqueue directly into the gate pipeline if we're in a hurry | 14:57 |
| fungi | and then promote it, yes | 14:57 |
| *** kiennt26 has quit IRC | 14:57 | |
| *** xarses has joined #openstack-infra | 14:57 | |
| fungi | i'm happy to do that now | 14:57 |
| *** srobert has quit IRC | 14:58 | |
| AJaeger | let's do it, there's just one nova job in integrated gate currently | 14:58 |
| *** spectr has joined #openstack-infra | 14:58 | |
| jeblair | i worry about doing that just a little bit since a likely failure mode is that things continue to work, but with much more network traffic (ie, the original v3 problem) | 14:58 |
| *** LindaWang has quit IRC | 14:58 | |
| fungi | that's a fair concern | 14:58 |
| AJaeger | And that nova hcange is update from global requirements | 14:58 |
| *** LindaWang has joined #openstack-infra | 14:58 | |
| jeblair | i guess we could do it and then carefully example the logs after merging and be fast on the revert gun | 14:58 |
| *** dbecker has quit IRC | 14:59 | |
| *** srobert has joined #openstack-infra | 14:59 | |
| *** mat128 has quit IRC | 14:59 | |
| jeblair | but regardless, we'll need to check | 14:59 |
| *** cshastri has quit IRC | 15:01 | |
| *** rhallisey has quit IRC | 15:02 | |
| *** psachin has quit IRC | 15:02 | |
| *** rosmaita has joined #openstack-infra | 15:03 | |
| *** larainema has quit IRC | 15:05 | |
| mordred | the one question mark I have on the patch is whether or not chown -R traverses symlinks | 15:05 |
| *** chlong has quit IRC | 15:06 | |
| fungi | is failure of tests.unit.test_scheduler.TestScheduler.test_double_live_reconfiguration_shared_queue.test_job_aborted in tox-py35-on-zuul for openstack-infra/zuul-jobs a known flaky result? | 15:06 |
| *** sree has quit IRC | 15:06 | |
| fungi | 511253 passed check but then bombed in the gate on it | 15:06 |
| *** sree has joined #openstack-infra | 15:07 | |
| *** eharney has joined #openstack-infra | 15:07 | |
| rosmaita | when someone has a minute, i have a question about a review whose dependency has merged, but zuul doesn't seem to be picking up the patch; i looked through the docs and gave it a new workflow +1, but still no action ... not sure what i'm doing wrong. https://review.openstack.org/#/c/508887/ | 15:10 |
| *** dizquierdo has joined #openstack-infra | 15:10 | |
| *** sree has quit IRC | 15:11 | |
| *** links has joined #openstack-infra | 15:12 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add documentation on force-merging a change https://review.openstack.org/511248 | 15:12 |
| openstackgerrit | Sam Betts proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs https://review.openstack.org/511264 | 15:15 |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config master: Remove ironic legacy jobs https://review.openstack.org/511265 | 15:15 |
| *** efried has joined #openstack-infra | 15:17 | |
| *** dhinesh has joined #openstack-infra | 15:17 | |
| *** ralonsoh_ has quit IRC | 15:18 | |
| fungi | infra-root: fyi, i have a dental cleaning and will be disappearing on you for about 1.5 hours from 18:45-20:15 | 15:18 |
| fungi | a few hours from now | 15:18 |
| pabelanger | ack | 15:20 |
| AJaeger | rosmaita: that change has no +1 by Jenkins | 15:20 |
| AJaeger | rosmaita: and without a +1 by Jenkins, Zuul v2 will not merge it - the +1 by Zuul is not enough | 15:20 |
| fungi | yeah, needs a recheck | 15:21 |
| *** dhinesh has quit IRC | 15:21 | |
| fungi | which seems to have been tried previously as well | 15:21 |
| AJaeger | fungi, jeblair, so push the devstack-gate change into gate - or not? Didn't understand the conclusion | 15:21 |
| fungi | so i'm guessing there's something else broken... | 15:21 |
| fungi | glace-specs... was that the repo in the corrupted state we needed to repair? | 15:22 |
| fungi | i wonder if zuul is having a hard time pulling from it | 15:22 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: New Summit public endpoints https://review.openstack.org/511270 | 15:22 |
| clarkb | nova specs was the repo | 15:22 |
| *** dhinesh has joined #openstack-infra | 15:23 | |
| fungi | oh, okay. i'll dig around in zuul debug logs and see if i can spot the problem | 15:23 |
| AJaeger | maybe rebase? It's not current... | 15:23 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: New Summit public endpoints https://review.openstack.org/511270 | 15:23 |
| *** LindaWang has quit IRC | 15:23 | |
| rosmaita | AJaeger fungi : i will try a rebase and see what happens | 15:23 |
| openstackgerrit | Sam Betts proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs https://review.openstack.org/511264 | 15:24 |
| AJaeger | sambetts: for the ironic changes: Normal process is: Move the changes to your repo, then sent project-config change that removes them with depends-on set. If we now merge your change, you have none of these jobs left... | 15:25 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Marketplace API https://review.openstack.org/498102 | 15:26 |
| sambetts | AJaeger: so project-config depends on ironic change? | 15:26 |
| fungi | rosmaita: looks like that kicked it into gear. my bet is that you had a cross-repo dependency loop which prevented the change from being enqueued back on october 2nd, but it has since been resolved | 15:27 |
| AJaeger | sambetts: yes. Isn't that explained properly in https://docs.openstack.org/infra/manual/zuulv3.html#howto-update-legacy-jobs | 15:27 |
| rosmaita | fungi ty | 15:28 |
| sambetts | AJaeger: I knew that the names had to be unique, so that why I had the removal before the addition | 15:28 |
| sambetts | AJaeger: but if I have to have all the jobs run twice for a time then ok | 15:29 |
| AJaeger | sambetts: please follow the guide | 15:29 |
| *** andreas_s has quit IRC | 15:29 | |
| fungi | rosmaita: oh! actually the issue is that zuul v2 ("jenkins") was offline back on the 2nd and we were using zuul v3 ("zuul") which did leave +1 votes, just needed a recheck with v2 running now to get it enqueued there | 15:29 |
| AJaeger | sambetts: we can quickly merge your change after the ironic one has merged... | 15:29 |
| *** andreas_s has joined #openstack-infra | 15:29 | |
| AJaeger | sambetts: alternative is that you have no testing for ironic for some time and need to freeze repo | 15:29 |
| rosmaita | fungi: i understand now ... also thank you AJaeger | 15:30 |
| *** ldnunes has quit IRC | 15:32 | |
| *** ldnunes has joined #openstack-infra | 15:32 | |
| openstackgerrit | Sam Betts proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs https://review.openstack.org/511264 | 15:34 |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config master: Remove ironic legacy jobs https://review.openstack.org/511265 | 15:34 |
| AJaeger | sambetts: commented, this should work fine... | 15:37 |
| *** chlong has joined #openstack-infra | 15:38 | |
| *** andreas_s has quit IRC | 15:38 | |
| AJaeger | fungi, did you want to promote the devstack-gate job into gate pipeline - or wait? | 15:39 |
| fungi | AJaeger: sounds like it was safer to wait and get some additional results | 15:41 |
| sambetts | AJaeger: thanks for the help :) | 15:41 |
| AJaeger | fungi: ok | 15:42 |
| jeblair | fungi, AJaeger, mordred: if that's the plan, what should we do about the v3 rollout? spend today stabilizing more things and roll out tomorrow morning? | 15:43 |
| sambetts | AJaeger: qq does normal yaml syntax work in zuul v3 configs? e.g. can I use anchors etc?? | 15:43 |
| pabelanger | AJaeger: we also need to get back a zuulv3 job too I htink | 15:43 |
| jeblair | pabelanger: let's restart zuulv3 then and recheck. | 15:43 |
| pabelanger | jeblair: wfm | 15:43 |
| *** hongbin has joined #openstack-infra | 15:43 | |
| jeblair | sambetts: yes, but please consider carefully whether it improves legibility or not, and whether what you're doing may be better accomplished some other way (for instance, job inhertance, or variance...) | 15:44 |
| *** eumel8 has quit IRC | 15:45 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 15:45 |
| AJaeger | mordred: looking at https://review.openstack.org/#/c/511160/1/zuul.d/projects.yaml - the PTI includes some javascript jobs, should those stay in? | 15:45 |
| jeblair | sambetts: feel free to send me a link to an example and i can check to see if there are any alternatives that may help | 15:45 |
| dmsimard | AJaeger, jeblair: ^ fixed firewall tests | 15:45 |
| jeblair | i'm restarting zuulv3 to dump the queues | 15:45 |
| pabelanger | ack | 15:46 |
| fungi | roger | 15:46 |
| sambetts | jeblair: the use case I am considering is that every one of our jobs has the same list of "irrelevent-files" and using an anchor I can define it once put a &common-files and then everywhere else use *common-files instead of duplicating the list | 15:46 |
| AJaeger | sambetts: alternative would be to have one common base job -and use inheritance as jeblair commented | 15:46 |
| *** srobert has quit IRC | 15:47 | |
| *** srobert has joined #openstack-infra | 15:47 | |
| sambetts | makes sense, thanks | 15:47 |
| inc0 | good morning, I understand you've kicked off zuulv3 migration? | 15:48 |
| AJaeger | sambetts: so, instead of basing on legacy-dsvm-base, you create a legacy-ironic-base job with all the irrrelevant files and then use that everywhre | 15:48 |
| sambetts | yeah, seems like the nicest way to do it | 15:48 |
| dmsimard | sambetts: like others have said, you can set those as a common base job with the parameters that you want and then have other jobs inherit from that, here's an example: https://github.com/openstack-infra/openstack-zuul-jobs/commit/05327069abec9734ea0b6597f6c3d719330aeca9#diff-29a997266a1a8f64b10e1aac7f90d176 | 15:48 |
| dmsimard | 'job' is a bit of a misnomer in that context because it doesn't end up running anything by itself | 15:49 |
| jeblair | AJaeger, sambetts: yeah, either would work. if you have at least one other thing in common with those files (eg, required-projects or something...) i'd say that would tip the scales toward inheritance. | 15:49 |
| jeblair | s/files/jobs/ | 15:51 |
| *** chlong has quit IRC | 15:51 | |
| AJaeger | and ironic files have long list of required-project, so this fits... | 15:52 |
| *** rbrndt has quit IRC | 15:52 | |
| mordred | AJaeger: well- we don't have good new PTI versions of javascript jobs (although we're getting there) - let's keep them in and get ourselves a nice PTI project-template for javascript | 15:53 |
| * mordred reviews | 15:53 | |
| AJaeger | thanks, mordred | 15:54 |
| jeblair | inc0: no we have not | 15:54 |
| AJaeger | regarding devstack-gate - " libcurl4-gnutls-dev : Depends: libcurl3-gnutls (= 7.47.0-1ubuntu2.2) but 7.47.0-1ubuntu2.3 is to be installed" hits us on the first job already | 15:54 |
| *** egonzalez has quit IRC | 15:54 | |
| AJaeger | http://logs.openstack.org/55/511255/1/check/legacy-devstack-gate-tox-py3-run-tests/8134612/ | 15:54 |
| *** ralonsoh has joined #openstack-infra | 15:54 | |
| inc0 | jeblair: still plannig for today? | 15:55 |
| jeblair | inc0: i don't know what the plan is. i have an outstanding query to other folks in this channel to formulate a plan, but have not received a response. | 15:55 |
| clarkb | AJaeger: lets check upstream ubuntu and see if that is broken there /me looks | 15:56 |
| inc0 | :) | 15:56 |
| fungi | jeblair: oh, i thought we were waiting to see whether the devstack-gate patch for cache ownership works before deciding | 15:56 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: New current summit public endpoint https://review.openstack.org/511279 | 15:56 |
| pabelanger | AJaeger: just see that | 15:56 |
| *** Rockyg has joined #openstack-infra | 15:56 | |
| pabelanger | looks like ubuntu is broken | 15:56 |
| *** nicolasbock has quit IRC | 15:56 | |
| *** coolsvap has joined #openstack-infra | 15:56 | |
| AJaeger | pabelanger: yeah, first reports 7+ hours ago or so | 15:57 |
| *** slaweq has joined #openstack-infra | 15:57 | |
| fungi | AJaeger: clarkb: pabelanger: my initial guess is that it's a conflict between xenial main and uca... do we unconditionally enable uca? | 15:57 |
| clarkb | AJaeger: pabelanger https://packages.ubuntu.com/xenial/libcurl4-gnutls-dev that says 2.3 now | 15:57 |
| clarkb | fungi: for devstack we do yes | 15:57 |
| *** vsaienk0 has quit IRC | 15:57 | |
| pabelanger | AJaeger: fungi: clarkb: looks like all jobs with bindep-fallback are breaking | 15:57 |
| mordred | infra-root: I think we should not roll out with devstack-gate in its current state - so I thnk we should continue to work the d-g problem until it is solved. If we're comfortable with the time left in the day after it is solved I think we can roll forward - otherwise we should regroup | 15:58 |
| *** sileht has quit IRC | 15:58 | |
| mordred | zomg uca conflicting with xenial wtf | 15:58 |
| clarkb | pabelanger: AJaeger fungi ok if its all bindep fallback then likely not completely UCAs fault as we should only uca with devstack | 15:58 |
| *** sileht has joined #openstack-infra | 15:58 | |
| *** pcaruana has quit IRC | 15:59 | |
| *** rcernin has quit IRC | 15:59 | |
| fungi | yeah | 15:59 |
| mnaser | libcurl4-gnutls-dev doesnt seem to be in uca? | 15:59 |
| jeblair | mordred: ack. i can't imagine that happening in less than 3h without the uca issue, and the uca issue will likely add several more hours. so i'm not optimistic we'll be doing it today. | 15:59 |
| clarkb | ubuntu package listing above says it should dep on 2.3 so I'm guessing we've synced some incomplete or slow to update mirror state | 15:59 |
| mordred | I DO think we should send out a quick update to the list (I can write one real quick) explaining there are 2 different devstack-gate issues we're working at the moment and so we're not going to roll out in th emiddle of that | 15:59 |
| fungi | so did we end up with an in-between mirror state? | 15:59 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: New current summit public endpoint https://review.openstack.org/511279 | 15:59 |
| pabelanger | clarkb: Ya, security update seems to have done it | 15:59 |
| mordred | jeblair: I'm also not optimistic - mostly just wanted to be complete about my thinking | 16:00 |
| jeblair | mordred: should we set a go/no-go time for today? | 16:00 |
| fungi | new mirror pulse should be kicking off right now, correct? | 16:00 |
| pabelanger | ya | 16:00 |
| pabelanger | every 2 hours | 16:00 |
| pabelanger | checking logs | 16:00 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Marketplace API https://review.openstack.org/498102 | 16:00 |
| mnaser | oh it could be an rsync failing mid-sync | 16:00 |
| fungi | mnaser: shouldn't be. if the mirror process fails to complete we don't release the volume | 16:01 |
| mnaser | i remember once we had a similar issue regarding permissions (i believe fungi helped resolve that) | 16:01 |
| fungi | mnaser: but also... mirroring these via reprepro not rsync | 16:01 |
| mnaser | oh you're right | 16:01 |
| mnaser | ill boot up a xenial vm, apt-get update and apt-get install libcurl4-gnutls-dev | 16:02 |
| pabelanger | okay, I see some errors in mirror-update | 16:02 |
| pabelanger | working to fix it now | 16:02 |
| fungi | thanks pabelanger | 16:02 |
| *** lucasagomes is now known as lucas-afk | 16:02 | |
| mnaser | ^ awesome | 16:02 |
| clarkb | jeblair: mordred: being on the west coast I'm good with as late as early afternoon, but realize that may no work great for others (re go no go) | 16:02 |
| fungi | mnaser: thanks, would be nice having a control group to confirm that the general situation with ubuntu is expected working for those packages | 16:02 |
| mnaser | i was able to install libcurl4-gnutls-dev successfully (from our mirrors on our cloud) | 16:03 |
| mnaser | Setting up libcurl4-gnutls-dev:amd64 (7.47.0-1ubuntu2.3) ... | 16:03 |
| mnaser | so that installed without any issues | 16:04 |
| fungi | ooh, this _could_ i think be caused by newer packages on our images than in the mirrors | 16:04 |
| fungi | so easily explained by whatever pabelanger found broken | 16:04 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on public endpoints https://review.openstack.org/511281 | 16:04 |
| *** slaweq has quit IRC | 16:05 | |
| pabelanger | fungi: mnaser: clarkb: ya, our mirror-update for ubuntu is out of sync. manually fixing, then will see what happened | 16:06 |
| *** rhallisey has joined #openstack-infra | 16:06 | |
| clarkb | pabelanger: was the cron erroring? | 16:06 |
| AJaeger | 511255 has lots of failures currently ;( | 16:06 |
| AJaeger | no time to investigate | 16:06 |
| mnaser | yeah usually package X will depends on Y 2.2 (did not update with mirror-update), but package Y was already sync'd up to 2.3 and 2.2 was removed | 16:06 |
| fungi | we've gotten into this state in the past, owing to the fact that our image build process doesn't use our mirrors | 16:07 |
| pabelanger | clarkb: I think we killed reprepro, with timeout and it left lockfile | 16:07 |
| pabelanger | so, next runs couldn't properly sync | 16:07 |
| clarkb | pabelanger: gotcha | 16:07 |
| mnaser | AJaeger oh that sa lot of red | 16:07 |
| *** dbecker has joined #openstack-infra | 16:07 | |
| * mnaser will try looking | 16:07 | |
| pabelanger | clarkb: possible there was a large update from ubuntu, which look longer then 30mins to download packages | 16:07 |
| mnaser | 2017-10-11 16:05:20.611 | rm: cannot remove '/home/zuul/cache/files/cirros-0.3.4-x86_64-disk.vhd.tgz': Permission denied | 16:08 |
| mnaser | i think a fix for this was merged? | 16:08 |
| mnaser | or was proposed | 16:08 |
| pabelanger | mnaser: not yet | 16:08 |
| pabelanger | 511255 | 16:08 |
| *** jpich has quit IRC | 16:08 | |
| *** wolverineav has quit IRC | 16:09 | |
| mnaser | looks like that fix didn't work in the gate as AJaeger mentioned | 16:09 |
| *** nicolasbock has joined #openstack-infra | 16:09 | |
| *** bhavik1 has joined #openstack-infra | 16:09 | |
| clarkb | mnaser: its failing on the tls thing I think | 16:09 |
| clarkb | mnaser: so have to fix one item at a time :) | 16:09 |
| mnaser | clarkb: i dont think so -- at least here - http://logs.openstack.org/55/511255/1/check/legacy-grenade-dsvm-neutron-multinode/aace3f3/logs/devstack-gate-setup-workspace-new.txt | 16:09 |
| *** hemna_ has joined #openstack-infra | 16:10 | |
| efried | oh, no cutover yet, so ignore bogus merge conflicts from v3? | 16:10 |
| pabelanger | clarkb: ya, we are downloading new kernels from ubuntu security, that would have pushed us over 30min timeout | 16:10 |
| AJaeger | efried: tell us about those, please | 16:10 |
| efried | AJaeger e.g. https://review.openstack.org/#/c/490057/23 | 16:10 |
| mordred | infra-root: draft status update email | 16:10 |
| *** chlong has joined #openstack-infra | 16:11 | |
| clarkb | mnaser: chown -R must not follow symlinks? | 16:11 |
| AJaeger | efried: why bogus merge conflict? | 16:11 |
| clarkb | mordred: ^ | 16:11 |
| efried | AJaeger PS23 is on the tip of the master branch. No further rebase is possible. | 16:11 |
| efried | And no dependencies | 16:11 |
| mnaser | clarkb i guess time to recreate a small experiment to test this out | 16:11 |
| AJaeger | efried: but gerrit says "Merge conflict" | 16:12 |
| efried | AJaeger Just so :) | 16:12 |
| mordred | clarkb: ah - so perhaps https://review.openstack.org/#/c/511260/1 is actually the better option? or we can respin that one to chwon the files too | 16:12 |
| efried | AJaeger It should not :) | 16:12 |
| jeblair | mordred: was there more to that msg about status update? | 16:12 |
| clarkb | mordred: mnaser ya we need -L -R | 16:12 |
| AJaeger | anybody here has time to help efried? | 16:12 |
| mordred | jeblair: whoops | 16:12 |
| clarkb | the default iwht -R is to not traverse links | 16:12 |
| jeblair | efried, AJaeger: yep, will look | 16:12 |
| mordred | infra-root: https://etherpad.openstack.org/p/1LkJ6Tar8I | 16:12 |
| efried | AJaeger I don't need to be helped, necessarily, assuming this is just a zuulv3 snafu and I can ignore it for now. | 16:13 |
| *** hemna_ has quit IRC | 16:13 | |
| mordred | jeblair: I *think* that might be an instance of the issue I had on the etherpad with no specific examples | 16:13 |
| mnaser | clarkb i think: sudo chown -R zuul:zuul /opt/stack/cache/files/ (the trailing slash) makes it not chown the root dir itself so while the file permissions update, the directory is still likely unacessible? | 16:13 |
| efried | Y'all have enough to worry about | 16:13 |
| *** wolverineav has joined #openstack-infra | 16:13 | |
| mnaser | FYI i like to add -v to get a verbose output of whats happening | 16:13 |
| clarkb | mnaser: according to the manpage -P is default with -R which means don't traverse any links | 16:13 |
| clarkb | mnaser: i htink we just have to add -L | 16:13 |
| AJaeger | efried: if there's a bug, let's investigate so that we can access the siutation... | 16:14 |
| mnaser | clarkb im all for doing that (and maybe adding -v to see what it is touching exactly for easier debugging? | 16:14 |
| clarkb | mnaser: sure | 16:14 |
| clarkb | mnaser: do you want ot push that patchset? | 16:14 |
| efried | AJaeger Sure, just let me know what you need from me. | 16:14 |
| AJaeger | efried: jeblair is looking -thanks | 16:14 |
| mnaser | clarkb sure, i think mordred is busy keeping up with other stuff now | 16:14 |
| mordred | mnaser, clarkb: I think https://review.openstack.org/#/c/511260 is a more complete solution - if we're gonna push new patchsets - perhaps we should either squash them - or we should rebase 511260 so as not to be parented on the other and race to see which fixes first? | 16:15 |
| fungi | mordred: lgtm, rabbie burns included | 16:16 |
| efried | jeblair I have a few examples of the same thing. It actually seems like I'm getting dinged with merge conflicts in all my nova patches (and none of them look like real merge conflicts afaict). | 16:16 |
| efried | jeblair Here's another: https://review.openstack.org/#/c/511261/ | 16:16 |
| *** hemna_ has joined #openstack-infra | 16:16 | |
| *** sshnaidm|mtg is now known as sshnaidm | 16:16 | |
| mnaser | mordred i think that's much cleaner | 16:17 |
| pabelanger | aptmethod got 'http://security.ubuntu.com/ubuntu/pool/main/c/curl/libcurl4-gnutls-dev_7.47.0-1ubuntu2.3_amd64.deb' | 16:17 |
| pabelanger | that is better | 16:17 |
| mnaser | im rechecking that change | 16:17 |
| mordred | mnaser, clarkb: should we reparent it so they're not tied together? | 16:17 |
| clarkb | mordred: mnaser oh ya maybe lets just use that fix then | 16:17 |
| clarkb | and I don't think we need both so squashing to a single thing should be fine | 16:17 |
| mnaser | mordred clarkb yeah the chown would be useless at that point | 16:18 |
| clarkb | the only problemwith that is going to be things like tripleo | 16:18 |
| clarkb | becuse they don't use devstack and so look for things in places | 16:18 |
| mnaser | we could fix that in pre.yaml for their jobs | 16:18 |
| mordred | clarkb: that's a good point - however, they've temporarily disabled consuming from cache | 16:19 |
| mordred | pending our cache fixes making it in to the images | 16:19 |
| clarkb | ya | 16:19 |
| mordred | so we could just make sure when we revert their disable patch that we update it to pull from /opt/cache/files instead | 16:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files https://review.openstack.org/511260 | 16:19 |
| mordred | there's a squashed/rebased version of that ^^ | 16:19 |
| mordred | EmilienM: your patch that disabled consuming cirros images from the cache hasn't been reverted yet,right? | 16:20 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy job from openstackdocstheme https://review.openstack.org/511287 | 16:20 |
| fungi | mordred: i think maybe it did because sshnaidm reported it failing earlier today | 16:20 |
| mnaser | could we bump mordred's change to top of zuulv3 queue? | 16:21 |
| sshnaidm | fungi, mordred shouldn't be disabled, I pushed it yesterday lately | 16:21 |
| pabelanger | mnaser: no point until we fix ubuntu AFS | 16:21 |
| sshnaidm | we shouldn't consume this image right now | 16:21 |
| mordred | cool | 16:21 |
| pabelanger | which should be in the next 20mins I think | 16:21 |
| fungi | at 11:58z in here he said "we have a problem with setup workspace in tripleo jobs: rm: cannot remove /home/zuul/cache/files/cirros-0.3.5-x86_64-disk.vhd.tgz Permission denied" | 16:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy job from openstackdocstheme https://review.openstack.org/511287 | 16:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: remove openstackdocstheme-tox-doc-publish-buildpdf https://review.openstack.org/511289 | 16:22 |
| EmilienM | sshnaidm: thanks, good to know | 16:23 |
| AJaeger | now post failure on https://review.openstack.org/511253 ;( I'll recheck | 16:25 |
| *** dizquierdo has quit IRC | 16:25 | |
| mordred | fungi, EmilienM, sshnaidm: ok - so the d-g patch will actually fix tripleo too because tripleo-ci consumes the function that is fixed in that patch | 16:26 |
| mordred | clarkb: ^^ | 16:26 |
| clarkb | neat | 16:26 |
| *** dhajare has quit IRC | 16:26 | |
| mordred | so it's set up to consume from /opt/stack/cache/files/cirros-0.3.5-x86_64-disk.img | 16:27 |
| clarkb | will tripleo look to copy the files from that location though? | 16:27 |
| clarkb | ah ok | 16:27 |
| mordred | so no additional vector to worry about | 16:27 |
| sshnaidm | which patch, sorry? | 16:27 |
| tosky | question: the jobs live in the same namespace (so job names should not collide), but this is not valid for role names, because you need to explicitely specify the repository - is it correct? | 16:27 |
| mordred | sshnaidm: the one fromyesterday - we're making a d-g change and wantedto amke sure it wouldn't break you | 16:27 |
| sshnaidm | mordred, ok, thanks | 16:27 |
| mordred | sshnaidm: it should be fine (should actualy fix a current issue) | 16:27 |
| AJaeger | tosky: correct - but you can reference different repos | 16:27 |
| pabelanger | mordred: did we land your zuul patch for post playbook logs? 511253 might have just hit it | 16:27 |
| AJaeger | tosky: use "roles: - zuul: openstack-infra/zuul-jobs" for example to use roles fro mthere | 16:28 |
| mordred | tosky: that is correct. I mean- roles also live in the same namespace- but just making a role with a conflicting name will not necessarily break anyone becaues they would have to explicitly request that role to be added | 16:28 |
| AJaeger | tosky: see openstack-manuals/.zuul.yaml for an example | 16:28 |
| tosky | ack: the problem I'm facing is the name of a role, if I should try to make it unique or I can rely on the name of the repository (and it seems to be the latter) | 16:28 |
| mnaser | i think base jobs use openstack-zuul-jobs or zuul-jobs | 16:29 |
| mnaser | for the roles, which means i'd avoid any roles that have the same names there | 16:29 |
| *** yamahata has joined #openstack-infra | 16:30 | |
| *** Goneri has joined #openstack-infra | 16:31 | |
| dmsimard | mnaser: there is a notion of precedence | 16:31 |
| dmsimard | I'd need to test it, but first path in the role path wins | 16:31 |
| *** weshay is now known as weshay|ruck | 16:32 | |
| *** panda is now known as panda|rover | 16:32 | |
| jeblair | dmsimard: feel free to test it, though you could also look at the docs: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.roles | 16:33 |
| jeblair | tosky, mnaser: ^ | 16:33 |
| dmsimard | Something that we could do, also, is to make the role paths relative | 16:33 |
| dmsimard | So instead of "upload-logs", it'd be "project-config/upload-logs" or something like that to avoid name clashing | 16:34 |
| dmsimard | You can do that by putting your role path one depth lower (or higher, depending on how you look at it) | 16:34 |
| jeblair | dmsimard: okay please do read the docs. | 16:34 |
| *** d0ugal has quit IRC | 16:35 | |
| mordred | tosky: yah - you only need to worry about it if you're planning on other people using them | 16:35 |
| dmsimard | jeblair: I know about the docs and Zuul, I'm talking about Ansible here | 16:35 |
| tosky | ok, I guess I will invest a bit more time | 16:35 |
| tosky | future proof is better done now | 16:35 |
| *** Swami has joined #openstack-infra | 16:35 | |
| *** mpaolino has quit IRC | 16:36 | |
| *** e0ne has quit IRC | 16:38 | |
| *** ralonsoh has quit IRC | 16:39 | |
| *** dtantsur is now known as dtantsur|afk | 16:39 | |
| *** dizquierdo has joined #openstack-infra | 16:39 | |
| dmsimard | jeblair: I'm saying that Ansible provides the ability to add a role path that is not a directory of roles -- for example 'zuul-jobs' could be a role path (instead of zuul-jobs/roles) and then when you need a role from there, such as upload-logs, you specify 'zuul-jobs/roles/upload-logs' instead of just 'upload-logs'. | 16:39 |
| *** felipemonteiro_ has quit IRC | 16:40 | |
| mordred | dmsimard: right- we talked about doing that - but the problem is that it gets weird when wanting to be able to test playbooks or roles that people have written for things that aren't zuul | 16:40 |
| clarkb | mordred: if you hvaen't sent that email yet I think you need to update the change number for the cache fix | 16:41 |
| mordred | dmsimard: because although you can set up your role path that way - the default behavior out in the ansible world is also a flat namespace, so we've been wary about introducing zuul-specific semantics for such things | 16:41 |
| clarkb | mordred: I've updated it | 16:41 |
| mordred | clarkb: cool - and wll send now | 16:41 |
| dmsimard | mordred: sure, I was pointing out that the possibility was there if we were interested in addressing potential name clashing | 16:42 |
| dmsimard | whether or not we want to do that, I don't have a strong opinion | 16:42 |
| mordred | dmsimard: nod | 16:42 |
| dmsimard | knowing that it's possible and we consciously don't want to do it is totally fine | 16:42 |
| *** epende has joined #openstack-infra | 16:42 | |
| *** rbrndt has joined #openstack-infra | 16:43 | |
| *** bhavik1 has quit IRC | 16:43 | |
| epende | I've created a server group using ServerGroupAntiAffinityFilter. I'd like to have nodepool use that group for certain images. Is there a way to do this? | 16:44 |
| clarkb | epende: is that exposed via nova boot api to non admin users? | 16:45 |
| dmsimard | epende: server group ? is that like a host aggregate ? | 16:45 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Convert legacy-api-ref and -guide to native v3 https://review.openstack.org/511290 | 16:45 |
| mnaser | server groups are a tenant user-facing concept for users | 16:45 |
| dmsimard | mnaser: TIL | 16:45 |
| mnaser | you can create a server group which has (soft)-affinity and (soft)-anti-affinity | 16:46 |
| epende | dmsimard server group as in this example: https://docs.openstack.org/ocata/config-reference/compute/schedulers.html#servergroupantiaffinityfilter | 16:46 |
| mnaser | the filter is simply a tool in the scheduler which enforces it | 16:46 |
| AJaeger | mordred: what do you think of 511290? It's just converting, so low priority... | 16:46 |
| mnaser | if you dont have the filter, the affinity settings wont be enforced, so yeah | 16:46 |
| epende | I have the group created and set to use the filter, but I would like nodepool to use the new group | 16:46 |
| AJaeger | bbl | 16:46 |
| *** larivee has joined #openstack-infra | 16:46 | |
| clarkb | epende: in that example you linked it appears that you use a specific flavor to end up in the group | 16:46 |
| clarkb | epende: in the nodepool case you should be able to just tell nodepool to use that group | 16:47 |
| fungi | by telling it to use that flavor? | 16:47 |
| epende | clarkb: Yes, I have the image I need, but I do not see where in nodepool's settings to configure that gruop | 16:47 |
| clarkb | fungi: er ya sorry | 16:47 |
| clarkb | epende: what fungi says, tell nodepool to use the flavor associated with the group | 16:47 |
| dmsimard | epende: how are you hinting at the group ? through the image ? through the flavor ? or really through the group ? | 16:47 |
| *** chlong has quit IRC | 16:48 | |
| *** dprince has quit IRC | 16:48 | |
| clarkb | oh wait | 16:48 |
| mnaser | scheduler hints | 16:48 |
| clarkb | its a hint | 16:48 |
| clarkb | bah | 16:48 |
| * clarkb rages at the nova api | 16:48 | |
| fungi | so we'd need special hints support in nodepool i suppose? | 16:48 |
| clarkb | fungi: ya I think so | 16:48 |
| mordred | aroo? | 16:48 |
| fungi | and plumb that through shade | 16:49 |
| * mordred reads | 16:49 | |
| epende | Ok, I thought that might be the case. Thanks clarkb and mnaser and dmsimard | 16:49 |
| *** dprince has joined #openstack-infra | 16:50 | |
| clarkb | why isn't that just a flvaor | 16:50 |
| clarkb | we have too many pizza cutters | 16:50 |
| dmsimard | I wonder if a flavor extra spec could be used for this | 16:50 |
| mordred | well - if the right way to use such a thing is to pass things directly in the create call (like it is for azs) I don't see any reason to not support it | 16:50 |
| clarkb | mordred: ya we should support it, mostly just grumpy that nova has like 50 ways to slice the pizza | 16:51 |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/irc-meetings master: Create alternate time for Neutron Drivers meeting https://review.openstack.org/511293 | 16:51 |
| *** sshnaidm is now known as sshnaidm|off | 16:52 | |
| mordred | k. I have found thedocs for how it works - adding support will not be hard | 16:52 |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/irc-meetings master: Create alternate time for Neutron Drivers meeting https://review.openstack.org/511293 | 16:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Also add security.list for Debian on configure-mirror https://review.openstack.org/511253 | 16:54 |
| clarkb | epende: out of curiousity why would nodepool nodes need anti affinity? is it just to keep them from bad neighboring each other? | 16:54 |
| *** links has quit IRC | 16:54 | |
| clarkb | in theory they are completely ephemeral instances so doesn't really matter where they schedule to | 16:54 |
| epende | clarkb: just a primative way to prevent internal fragmentation by having too many really large nodes from going to one compute host | 16:55 |
| dmsimard | epende: are you an operator of that cloud ? | 16:55 |
| epende | yes | 16:55 |
| epende | If there's another, better way to solve the problem I'm open to it | 16:56 |
| epende | Really the problem is, what if too many smaller nodes spawn on a compute host and crowd out all the larger nodes, leaving a lot of unused RAM & disk | 16:57 |
| *** d0ugal has joined #openstack-infra | 16:57 | |
| dmsimard | so a typical bin packing problem | 16:57 |
| epende | tetris | 16:57 |
| dmsimard | yes :) | 16:57 |
| *** cody-somerville has joined #openstack-infra | 16:58 | |
| *** cody-somerville has joined #openstack-infra | 16:58 | |
| clarkb | pabelanger: are mirrors still updating? | 16:58 |
| *** derekh has quit IRC | 16:58 | |
| epende | dmisard: We may be able to accomplish something similar with AZs, but it's less ideal | 16:58 |
| dmsimard | epende: I would say that leaving a lot of unused ram and disk is usually a consequence of mismatching compute hardware, flavors and overcommit ratios | 16:58 |
| dirk | infra-root: could somebody remove the jenkins-2 here: https://review.openstack.org/#/c/511210 and we'll wait for zuul to vote on it | 16:58 |
| pabelanger | clarkb: ya, I'm rechecking checksums for the indexes just to be sure | 16:59 |
| pabelanger | clarkb: taking a little time | 16:59 |
| *** andreas_s has joined #openstack-infra | 16:59 | |
| AJaeger | dirk: we still merge using jenkins... | 16:59 |
| *** baoli has quit IRC | 16:59 | |
| dirk | what? I thought we have the big flip over today? | 16:59 |
| pabelanger | not yet, see ML and status notice | 17:00 |
| fungi | dirk: nope, we got sidelined by various unrelated problems | 17:00 |
| AJaeger | dirk: remember the problem we talked about? Took us longer and then some more problems came up... | 17:00 |
| *** felipemonteiro has joined #openstack-infra | 17:00 | |
| dmsimard | epende: for example, a 32 core node with 64GB RAM works fairly okay with a 1:1 vcpu allocation ratio and flavors which are usually 1vcpu to 2GB of RAM | 17:00 |
| *** baoli has joined #openstack-infra | 17:00 | |
| *** eumel8 has joined #openstack-infra | 17:00 | |
| dmsimard | epende: but if you have a 32 core node with 128GB RAM, you'll likely want to increase your cpu overcommit to 2:1 so that you can use that RAM, you'll end up allocating 64 vcpus instead of 32. | 17:00 |
| epende | dmsimard: we're talking 110GB RAM, 16 VCPU flavors for the biggest ones, all the way down to 2G. | 17:01 |
| AJaeger | dirk: so, wait for the Jenkins recheck, please | 17:01 |
| *** Apoorva has joined #openstack-infra | 17:01 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:01 | |
| clarkb | epende: my biggest concern with the anti affinity approach is it requires your users to opt in, so won't affect anyone not in the know | 17:01 |
| clarkb | epende: this may be ok if you ar ethe only user but otherwise you'll likely to continue to have problems | 17:02 |
| dmsimard | epende: and the scheduler currently sends VMs to the compute node with the least instances ? with the most available ram ? | 17:02 |
| epende | clarkb: that is ok for our use case as we're the only users | 17:02 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on public endpoints https://review.openstack.org/511281 | 17:02 |
| mordred | clarkb: there are definitely private cloud usage patterns that are a bit different from how we consume the public clouds | 17:03 |
| epende | dmsimard: We haven't observed enough scheduler behavior to know yet, as we only have one compute host capable of accepting mroe than one large instance right now | 17:03 |
| clarkb | mordred: oh sure but if you have more than one person hitting the boot api all of a sudden this pattern falls over | 17:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Image should be optional https://review.openstack.org/511299 | 17:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add a call for setting volume bootable flag https://review.openstack.org/511300 | 17:04 |
| jeblair | fungi, clarkb, mordred: looking into the random erroneous merge conflict error, i found some interesting information. | 17:04 |
| clarkb | anti affinity is something you do when you know you want the 3 database servers on 3 different hypervisors so its opt in | 17:04 |
| mordred | jeblair: oh yeah? | 17:04 |
| clarkb | but ya if its single user it will probably work fine | 17:04 |
| *** kjackal_ has quit IRC | 17:04 | |
| epende | The use case is that we're running a private CI/CD cloud where nodepool instances are used as needed by the tests. The remaining instances are static. | 17:04 |
| *** gouthamr has quit IRC | 17:04 | |
| epende | clarkb, mordred: we mainly want to prevent starvation of the really large images if the scheduler does not pack the other instances optimally | 17:05 |
| *** dhinesh has quit IRC | 17:05 | |
| *** andreas_s has quit IRC | 17:05 | |
| *** felipemonteiro has quit IRC | 17:06 | |
| *** dizquierdo has quit IRC | 17:06 | |
| dirk | fungi: pabelanger : AJaeger : alright, thanks | 17:06 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Marketplace API https://review.openstack.org/498102 | 17:07 |
| *** andreas_s has joined #openstack-infra | 17:07 | |
| *** ganso has left #openstack-infra | 17:07 | |
| *** jascott1 has quit IRC | 17:09 | |
| *** jpena is now known as jpena|off | 17:09 | |
| *** trown is now known as trown|lunch | 17:10 | |
| efried | jeblair Do tell | 17:10 |
| persia | Could someone point me at the place to start debugging a post-gate publication job that may not have run? | 17:13 |
| AJaeger | persia: check status.openstack.org/zuul and see whether it has run | 17:14 |
| persia | AJaeger: It is no longer in queue | 17:14 |
| AJaeger | persia: if it has finished all post jobs already, check the log files, let me get you a link... | 17:14 |
| smcginnis | Unfortunately I have to step away, but just saw a zuul RETRY_LIMIT failure. If anyone has a moment to check out why, that would be awesome. Or I'll try later. | 17:14 |
| smcginnis | http://logs.openstack.org/22/505422/2/check/build-openstack-sphinx-docs/9fda1a5/ | 17:14 |
| *** rkukura has quit IRC | 17:14 | |
| *** rkukura_ has joined #openstack-infra | 17:14 | |
| AJaeger | persia: https://docs.openstack.org/infra/manual/developers.html#post-processing has information on how to get a link | 17:14 |
| *** nikhil has joined #openstack-infra | 17:14 | |
| *** tesseract has quit IRC | 17:15 | |
| jeblair | efried: i switched to #openstack-infra-incident because it was busy here. of course then everyone stopped talking. anyway, here's the log: http://eavesdrop.openstack.org/irclogs/%23openstack-infra-incident/%23openstack-infra-incident.2017-10-11.log | 17:15 |
| fungi | persia: if this is the election repo, i'm almost certain you hit a publication race between changes which merged at nearly the same time. we don't enforce serial ordering for jobs running on individual commits in the post pipeline | 17:15 |
| pabelanger | smcginnis: known issues, we are working to fix AFS mirrors for ubuntu | 17:15 |
| pabelanger | hopefully not much longer | 17:15 |
| persia | AJaeger: Thank you. | 17:15 |
| *** andreas_s has quit IRC | 17:15 | |
| fungi | persia: merging another change in that repo will get you to the state you expect, or one of us can manually retrigger the one which was expected to run last (but didn't because it probably ran on a slightly slower node than the one for the change which came before it) | 17:16 |
| persia | fungi: Ah, so if two jobs merged within a few minutes, it is possible that the post job only ran once, against a stale repo? | 17:16 |
| *** andreas_s has joined #openstack-infra | 17:16 | |
| *** larivee has left #openstack-infra | 17:16 | |
| fungi | persia: the post job ran for both commits, but may have published the result for the penultimate commit after the one for the ultimate commit | 17:17 |
| persia | Ah, no, it ran twice, but the older change overwrote the newer one. I understand. Yes, I'll just push a meaningless change to fix that (or find a real change to be fixed) | 17:17 |
| fungi | due to the one which was enqueued slightly sooner running slightly longer | 17:17 |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/irc-meetings master: Update Neutron team meeting chairperson https://review.openstack.org/511303 | 17:17 |
| *** sambetts is now known as sambetts|afk | 17:17 | |
| fungi | persia: we have some ideas for a new queue type in zuul which could solve this specific problem, but wanted to hold off implementation until v3 is fully baked | 17:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add group parameter to create_server https://review.openstack.org/511305 | 17:18 |
| mordred | clarkb, epende:^^ there's the shade support for setting a group on a create_server | 17:18 |
| jeblair | ya, i've totally written that patch in my head. just haven't typed it out yet :) | 17:18 |
| persia | fungi: I am entirely in favour of holding off on things until zuulv3 is fully baked :) | 17:19 |
| *** dizquierdo has joined #openstack-infra | 17:19 | |
| jeblair | there's one thing i haven't figured out yet -- what to call it. i'm thinking the 'supercedent' pipeline manager (to go along with 'dependent', and 'independent'). | 17:20 |
| *** rbrndt has quit IRC | 17:20 | |
| *** caphrim007_ has joined #openstack-infra | 17:20 | |
| clarkb | yesterday I brought up a duplicate gerrit account situation. User has confirmed that when they login they get the "new" one so I think we need to associate the new openid with the old account. | 17:21 |
| clarkb | does `UPDATE account_external_ids SET account_id = 21561 WHERE external_id = 'https://login.ubuntu.com/+id/yAnAdLz';` look correct for doing that? | 17:21 |
| clarkb | mordred: fungi ^ | 17:21 |
| jeblair | as discussed in #openstack-infra-incident i'm going to update all the origin remotes on the v3 system | 17:21 |
| jeblair | somehow | 17:21 |
| *** panda|rover is now known as panda|rover|off | 17:22 | |
| fungi | clarkb: yes, that is what i would normally run under those circumstances | 17:22 |
| fungi | jeblair: a for loop with sed over the .git/config files is how i'd probably do it | 17:23 |
| persia | AJaeger: Intrestingly, although the parent of the change has logs at that location, there seem to be none for d55c590ad189904043ef0496a9cb6d8eed8211d2 in openstack/elections - would this indicate the job didn't run, or is there some other log that shows which jobs did run? | 17:23 |
| clarkb | fungi: ok I'm going to run that query now | 17:23 |
| *** caphrim007 has quit IRC | 17:24 | |
| fungi | persia: if the change wasn't a fast-forward merge onto the branch, then there will be a merge commit and _that's_ what post jobs get run for | 17:24 |
| clarkb | and I've set account 27031 inactive | 17:24 |
| persia | fungi: Thank you. | 17:25 |
| fungi | persia: so c722a78bea5d1a75cb204cc783b2480131bd5bc4 based on the git log | 17:26 |
| clarkb | #status log move Gerrit account 27031s' openid to account 21561 and marked 27031 inactive | 17:26 |
| openstackstatus | clarkb: finished logging | 17:26 |
| persia | fungi: Right, which seems to have bumped into an issue with apt mirroring, which I understand is currently being fixed, so I should wait a bit before re-running. | 17:27 |
| *** jascott1 has joined #openstack-infra | 17:27 | |
| fungi | persia: oh, neat, so a job failure in this case? inconvenient timing | 17:27 |
| *** links has joined #openstack-infra | 17:28 | |
| persia | Yep. There are 10 hours before it needs to be sorted, so it isn't catastrophic, but definitely inconvenient. | 17:28 |
| fungi | but yeah, pabelanger is on the case where that issue is concerned | 17:28 |
| pabelanger | yes, just rechecking checksums on mirror, taking a little bit | 17:29 |
| pabelanger | once done, we can release AFS volume and things back to working order | 17:29 |
| *** andreas_s has quit IRC | 17:29 | |
| fungi | thanks pabelanger! | 17:30 |
| mordred | jeblair: my $0.02 is to do a for loop with git remote remove origin ; git remote add origin ssh://zuul@review.openstack.org:29418/$PROJECT.git | 17:30 |
| jeblair | mordred, fungi: this is what i came up with: http://paste.openstack.org/show/623377/ | 17:31 |
| AJaeger | jeblair: will that work with the cd? | 17:32 |
| clarkb | jeblair: lgtm and probably atomic and won't race other ops | 17:32 |
| mordred | jeblair: yes - however, I'd suggest making line 7 and 8 (cd $root/$org/$repo && git remote set-url origin ssh://zuul@review.openstack.org:29418/$org/$repo) | 17:32 |
| clarkb | AJaeger: ya $root is rooted | 17:32 |
| AJaeger | Iw would have written "(cd somewhere;do-stuff) | 17:32 |
| AJaeger | clarkb, Ah, then it works | 17:32 |
| mordred | oh - cool | 17:32 |
| fungi | jeblair: that works. the reason i normally use sed is that i can just use a single loop over $root/*/*/.git/config and pass them to sed -i | 17:32 |
| mordred | I also retract my thought | 17:32 |
| mordred | jeblair: so - that lgtm | 17:33 |
| AJaeger | mordred: could you review https://review.openstack.org/511290 , please? Can we do this more compact? | 17:33 |
| jeblair | fungi: that sounds nice... though i think with the git command we actually get locking! | 17:33 |
| *** baoli_ has joined #openstack-infra | 17:33 | |
| fungi | jeblair: oh! i had no idea git locked around the config file. sounds ideal then | 17:34 |
| jeblair | fungi: i didn't either until i ran that command as the wrong user! | 17:34 |
| fungi | hah | 17:34 |
| fungi | nice ;) | 17:34 |
| *** tosky has quit IRC | 17:34 | |
| mordred | AJaeger: you say "This is only run on master branch of a project" - just to be clear, do you mean that it should only be triggered by patches to master and patches to other branches should be ignored? | 17:35 |
| *** baoli has quit IRC | 17:35 | |
| AJaeger | mordred: correct | 17:35 |
| mordred | AJaeger: or that no matter which branch triggers it, the job should run on master (like reno) | 17:35 |
| mordred | AJaeger: cool | 17:35 |
| AJaeger | mordred: the first - only changes against master should trigger it. Can update the wording... | 17:36 |
| mordred | AJaeger: no - that's fine - I just wantedto make sure | 17:36 |
| mordred | AJaeger: there are still uses of legacy-api-guide and legacy-api-ref in post in that project-template | 17:37 |
| *** inc0 has quit IRC | 17:37 | |
| AJaeger | mordred: yes - that would be a followup. Want to do it one step at a time | 17:37 |
| mordred | AJaeger: k. however, you deleted the post playbooks in this patch ... | 17:38 |
| melwitt | hi, in nova we're seeing zuul v3 report false merge conflicts on patches. is that expected to be seen? | 17:38 |
| *** ykarel is now known as ykarel|afk | 17:38 | |
| pabelanger | melwitt: I believe we're working on a fix now. See the above discussions | 17:38 |
| mordred | melwitt: yah- it's not 'correct' behavior - but we're looking in to it | 17:39 |
| melwitt | cool, thank you | 17:39 |
| jeblair | okay i've updated remotes on zm05-zm08 and ze01-ze10 | 17:39 |
| mordred | jeblair: we think that the merger issue you're fixing is the cause of the merge issues yeah? (and not just a thing that you happened to find while looking) | 17:39 |
| jeblair | mordred: yep | 17:39 |
| mordred | cool | 17:39 |
| mordred | that's exciting | 17:40 |
| jeblair | melwitt, efried: zuul v3 should not report any false merge conflicts after this point. | 17:40 |
| AJaeger | mordred: that template is broken, post job and build job should not be the same - ok, will fix together. | 17:41 |
| *** e0ne has joined #openstack-infra | 17:41 | |
| efried | jeblair Thanks! | 17:41 |
| mordred | AJaeger: oh - nevermind then | 17:41 |
| mordred | AJaeger: that patch looks fine | 17:41 |
| *** dhajare has joined #openstack-infra | 17:41 | |
| AJaeger | mordred: ok, will do the publish then as well, thanks! | 17:41 |
| mordred | AJaeger: speaking of - did you see my email on the infra list about "Change publication interface ..." | 17:42 |
| mordred | AJaeger: (it's partially inspired by the extra work we have to do for these jobs) | 17:42 |
| jeblair | moved to fixed section of etherpad | 17:42 |
| *** andreas_s has joined #openstack-infra | 17:43 | |
| jeblair | mordred: what's the current devstack-gate change? i saw something about squashing? | 17:44 |
| clarkb | jeblair: 511260 | 17:44 |
| jeblair | that's still parented on the other change, isn't it? | 17:44 |
| clarkb | I think the newer patchset unparented it | 17:44 |
| mordred | jeblair: no - I squashed it | 17:44 |
| jeblair | derp, i was looking at an outdated copy, thanks | 17:45 |
| *** dhinesh has joined #openstack-infra | 17:47 | |
| *** andreas_s has quit IRC | 17:47 | |
| *** andreas_s has joined #openstack-infra | 17:48 | |
| *** slaweq has joined #openstack-infra | 17:48 | |
| pabelanger | checksums still running :( | 17:49 |
| pabelanger | taking much longer then expected | 17:49 |
| jeblair | dmsimard: what do you mean "tripleo has their own graphite server" ? | 17:49 |
| mordred | wow. if only there was some way that we could collectively work on a solution to collect stats on tasks run that could be shared effectively amongst all the users | 17:49 |
| jeblair | EmilienM: ^? | 17:49 |
| rajinir | Some basic issue after we upgraded to the latest project config, gearman server not starting .., How to debug this issue? | 17:50 |
| pabelanger | jeblair: private infrastructure that lives in tripleo-test-cloud-rh1 | 17:50 |
| dmsimard | pabelanger beat me to it | 17:50 |
| *** rosmaita has quit IRC | 17:50 | |
| jeblair | what in the *world* are we doing here? are we working together on a project, or are we building little fiefdoms? | 17:50 |
| jeblair | i know what i thought we were doing | 17:50 |
| EmilienM | jeblair: you might want to talk with sshnaidm | 17:50 |
| EmilienM | I haven't worked or contributed to that parts | 17:50 |
| fungi | graphing stats for the underlying hardware in the test cloud? | 17:50 |
| EmilienM | and yes we should work together | 17:51 |
| EmilienM | weshay|ruck: you might want to see ^ | 17:51 |
| *** rosmaita has joined #openstack-infra | 17:51 | |
| dmsimard | jeblair: the topic came about as EmilienM and I were discussing opportunities for graphing different task/ansible mtrics | 17:51 |
| EmilienM | yes I can't agree more than that - we should build a common tooling for our needs | 17:52 |
| pabelanger | jeblair: 480121 is relevant when you have time | 17:52 |
| mordred | I think this goes hand in hand with the 'shift-tripleo-ci-to-third-party' - a third-party ci running a local graphite server seems perfectly fine, while things run centrally but reporting out to an external graphite server seem much weirder | 17:52 |
| EmilienM | and stop doing tripleo-only things | 17:52 |
| EmilienM | jeblair, mordred: unfortunately, you can't force people to stop what they're doing sometimes | 17:52 |
| EmilienM | I guess it comes by education and showing example | 17:52 |
| weshay|ruck | EmilienM, dmsimard afaik we were measuring the performance of rh1 and ovb jobs | 17:52 |
| jeblair | dmsimard: yeah, that's a great topic. SpamapS was doing some work on measuring and tracking tasks within jobs and reporting to graphite. | 17:52 |
| EmilienM | weshay|ruck: have we engaged work to do it at an higher level? | 17:53 |
| EmilienM | weshay|ruck: e.g. in OpenStack Infra? | 17:53 |
| pabelanger | mordred: yah, I've helf off on commenting until 3rd party CI move | 17:53 |
| weshay|ruck | isn't that one step removed from upstream infra | 17:53 |
| pabelanger | held* | 17:53 |
| weshay|ruck | ya.. pabelanger btw.. we just started to migrate jobs off rh1 | 17:53 |
| *** slaweq has quit IRC | 17:53 | |
| jeblair | EmilienM, dmsimard, weshay|ruck: see http://specs.openstack.org/openstack/qa-specs/specs/devstack/counter-inspection.html | 17:53 |
| dmsimard | weshay|ruck mentions that it's for OVB jobs which will definitely be third party | 17:53 |
| *** links has quit IRC | 17:53 | |
| pabelanger | weshay|ruck: ++ | 17:54 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Create publish-api-ref/guide jobs https://review.openstack.org/511315 | 17:54 |
| weshay|ruck | before everyone gets outraged.. ya consider the context a bit | 17:54 |
| fungi | pabelanger: it looked like the discussion in #openstack-dev suggested yesterday that the rh1 test cloud was going away, and that (i think?) we can probably remove it from nodepool configuration in the next week or so | 17:54 |
| dmsimard | fungi: I'm not sure a week or so is realistic, but it's been a WIP several weeks in the making. We are making good progress. | 17:54 |
| pabelanger | fungi: great, I'll have to look at backlog for that | 17:55 |
| weshay|ruck | fungi, ya.. no | 17:55 |
| *** electrofelix has quit IRC | 17:55 | |
| weshay|ruck | it's not going away that quickly | 17:55 |
| dmsimard | fungi: and FWIW this third party CI will actually run out of review.rdoproject.org's Zuul | 17:55 |
| mordred | weshay|ruck: I'm not outraged - I mostly want to point out that there are (now, with v3) some potential integration points for such work that could both be leveraged in a 3rd party context and also potentially used directly upstream ... | 17:55 |
| weshay|ruck | mordred, great! that is good news | 17:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Create publish-api-ref/guide jobs https://review.openstack.org/511315 | 17:56 |
| mordred | so if this third party CI is going to be running ot of review.rdoproject.org's Zuul it might not be terrible to chat about ways in which the goals that are trying to be accomplished might be done in a way that's shareable and not project-specific | 17:56 |
| fungi | dmsimard: pabelanger: http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2017-10-10.log.html#t2017-10-10T16:11:35 | 17:56 |
| mordred | (probably not a good discussion topic to get into in detail today of course) | 17:56 |
| jeblair | there's also prior work begun on a very similar project. it has stalled, but there's a bunch of work done about how to actually integrate with graphite | 17:56 |
| jeblair | that work is applicable for upstream and downstream jobs / servers | 17:56 |
| weshay|ruck | there were no grand plans around that graphana server anyway, it's just an engineer doing a thing on their own w/o much direction | 17:56 |
| weshay|ruck | on it | 17:56 |
| *** andreas_s has quit IRC | 17:56 | |
| pabelanger | fungi: thanks | 17:57 |
| *** ijw has joined #openstack-infra | 17:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Convert legacy-api-ref and -guide to native v3 https://review.openstack.org/511290 | 17:58 |
| AJaeger | mordred: https://review.openstack.org/511315 pushed and used in updated https://review.openstack.org/511290 | 17:58 |
| *** tosky has joined #openstack-infra | 17:58 | |
| AJaeger | mordred: saw your email - looking forward to it. | 17:59 |
| weshay|ruck | mordred, jeblair pabelanger ok.. we'll probably have more to talk about in about two weeks once we have rh1->rdo-cloud and software factory well in progress for ovb jobs | 17:59 |
| mordred | AJaeger: https://review.openstack.org/#/c/511315/ - -1 in there with a question - also there's some trailing whitespace that will anger the whitespace gods | 18:01 |
| jeblair | weshay|ruck, sshnaidm|off: cool. in the mean time, if you have a moment, please give http://specs.openstack.org/openstack/qa-specs/specs/devstack/counter-inspection.html a perusal for background info | 18:01 |
| AJaeger | whitespace? Shame on me... thanks, mordred | 18:02 |
| weshay|ruck | k.. thanks | 18:02 |
| *** dhajare has quit IRC | 18:02 | |
| mordred | weshay|ruck: looking forward to chatting about it | 18:02 |
| jeblair | ++ | 18:02 |
| *** hrybacki is now known as hrybacki|trainin | 18:03 | |
| AJaeger | mordred: good find, thanks | 18:03 |
| dmsimard | jeblair: the scope of that spec has nothing about ansible, though, should there be ? | 18:03 |
| anupn | Hi all, if i want to disable a specific service from running with py3 i see there is a function disable_python3_package that exists so is that has to be called from project's plugin.sh only? | 18:03 |
| jeblair | dmsimard: it's not a new spec. :) | 18:03 |
| *** ijw has quit IRC | 18:04 | |
| jeblair | it just had its 2nd birthday | 18:04 |
| *** caphrim007_ has quit IRC | 18:04 | |
| *** ijw has joined #openstack-infra | 18:05 | |
| * SpamapS peers into the past | 18:05 | |
| jeblair | dmsimard: i'm definitely not suggesting it's what we should implement -- i'm saying we should be aware of previous work before we invent new things | 18:05 |
| pabelanger | ++ | 18:05 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Create publish-api-ref/guide jobs https://review.openstack.org/511315 | 18:05 |
| AJaeger | mordred: fixed ^ | 18:05 |
| SpamapS | Unfortunately I never got back to the counter inspection spec. | 18:05 |
| SpamapS | But it was pretty close to done. | 18:05 |
| SpamapS | Just needed to go the last mile to get those subunit counter attachments into graphing systems. | 18:06 |
| *** e0ne has quit IRC | 18:06 | |
| SpamapS | (and of course after some data collection, add more data points and maybe get more granular) | 18:06 |
| *** andreas_s has joined #openstack-infra | 18:06 | |
| clarkb | re 511260, if a devstack or grenade job runs and passes is that sufficient info to know we covered the cache handling and can enqueue to the gate at that point? | 18:06 |
| clarkb | and if so is that something we want to do? | 18:07 |
| *** dizquierdo has quit IRC | 18:07 | |
| *** trown|lunch is now known as trown | 18:07 | |
| mordred | clarkb: I think we want to see a legacy-devstack job run on top of that in v3 - but if it runs and passes in v2, then I don't think landing it will break v3 legacy dsvm jobs any MORE than they are broken right now | 18:08 |
| clarkb | oh right its the v3 side that we want to check on | 18:08 |
| clarkb | I guess we don't want to regress v2 though | 18:08 |
| AJaeger | mordred: there're still some variants of api-ref but my changes should cover the base... | 18:09 |
| jeblair | yeah. we do need to inspect logs to verify we haven't fallen back on network behavior | 18:09 |
| mordred | AJaeger: ++ | 18:09 |
| jeblair | we only need one job complete for that though | 18:09 |
| mordred | jeblair: agree | 18:09 |
| mordred | AJaeger: sorry - caught one more thing | 18:11 |
| mordred | AJaeger: but otherwise looks spot-on | 18:11 |
| *** andreas_s has quit IRC | 18:11 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 18:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 18:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 18:14 |
| *** hasharAway is now known as hashar | 18:14 | |
| *** rbrndt has joined #openstack-infra | 18:15 | |
| jeblair | i'm going to afk for about an hour for lunch and things | 18:16 |
| *** efried is now known as efried_nomnom | 18:17 | |
| smcginnis | pabelanger: Thanks! | 18:22 |
| *** kjackal_ has joined #openstack-infra | 18:23 | |
| *** baoli_ has quit IRC | 18:24 | |
| *** d0ugal has quit IRC | 18:24 | |
| *** baoli has joined #openstack-infra | 18:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Create publish-api-ref/guide jobs https://review.openstack.org/511315 | 18:26 |
| AJaeger | mordred: Thanks for catching that! Next try ^ | 18:26 |
| *** armaan has joined #openstack-infra | 18:27 | |
| mordred | AJaeger: looks awesome! clarkb, pabelanger wanna +3 https://review.openstack.org/#/c/511315 ? | 18:27 |
| AJaeger | thanks! | 18:27 |
| *** yolanda has quit IRC | 18:28 | |
| mordred | https://review.openstack.org/#/c/511290 also looks great clarkb pabelanger | 18:29 |
| clarkb | mordred: AJaeger 315 is approved | 18:30 |
| honza | mordred: is there a place where i can track any progress on the UCA mirrors issue you mentioned on the ML? | 18:30 |
| honza | http://lists.openstack.org/pipermail/openstack-dev/2017-October/123464.html | 18:31 |
| mordred | honza: not anywhere great - I believe right now pabelanger has something running that is verifying checksums - and is taking its sweet time doing it | 18:32 |
| dmsimard | fungi: did I hallucinate telling you about rh1 not closing next week ? I somehow remember typing about it but not finding it in my buffers :/ | 18:32 |
| dmsimard | fungi: oh nevermind, it's in -incident instead of -infra, my bad. | 18:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use build-openstack-api-ref for openstackdocstheme https://review.openstack.org/511318 | 18:32 |
| honza | mordred: cool, thanks, i shall attempt to follow along here | 18:33 |
| pabelanger | mordred: honza: ya, we are currently limited by rax bandwidth, reprepro wants to ensure proper checksums for all packages | 18:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Create publish-api-ref/guide jobs https://review.openstack.org/511315 | 18:35 |
| AJaeger | clarkb: thanks | 18:36 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: remove legacy openstackdocstheme-tox-doc-publish-buildpdf an https://review.openstack.org/511289 | 18:36 |
| clarkb | now we recheck 290? | 18:36 |
| mordred | yup | 18:37 |
| mordred | also - ... I love the new error message | 18:37 |
| dmsimard | Question about elastic-recheck -- it turns out it's not straightforward to support zuul v2 and zuul v3 simultaneously. Do we care about v2 support in elastic-recheck post roll-forward ? | 18:37 |
| clarkb | dmsimard: what is the issue with supporting both? | 18:38 |
| dmsimard | I think if there are external/third party CI users they can pin to a commit before v3 is added in or something like that | 18:38 |
| dmsimard | clarkb: mainly the expectation of a file named either console.html or job-output.txt | 18:38 |
| clarkb | dmsimard: and we can't query for both? | 18:38 |
| clarkb | dmsimard: maybe we should make it a configurable file name and then others can configure to use whatever they need | 18:38 |
| clarkb | so default can be job-output.txt, but allow it to be set to console.html | 18:39 |
| *** armaan has quit IRC | 18:39 | |
| dmsimard | clarkb: it's for the 'required_files' method here: https://review.openstack.org/#/c/509313/2/elastic_recheck/elasticRecheck.py | 18:39 |
| clarkb | gotcha, ya in that case maybe we can make that filename configurable? | 18:40 |
| fungi | okay, i have to step out to the aforementioned dental checkup for the next ~1.5 hours but will rejoin you all soonish | 18:40 |
| * fungi disappears into the aether | 18:41 | |
| * AJaeger waves good night | 18:41 | |
| *** dbecker has quit IRC | 18:41 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: switch whereto to publish docs on docs.o.o https://review.openstack.org/507243 | 18:42 |
| dmsimard | clarkb: I'd like to, yes, but I was also interested in keeping the changes to a minimum. Ideally, I'd really prefer another hint at whether a job is running on v2 or v3. I tried to find where the 'type' field for logs were set, it seems like it's always jenkins, even on v3 jobs | 18:42 |
| *** vsaienk0 has joined #openstack-infra | 18:42 | |
| clarkb | dmsimard: I think logstash rules may set that, we can possibly have the jobs that zuulv3 submits send that info | 18:44 |
| dmsimard | clarkb: I looked in gearman log processors, in the new submit roles/modules, etc.. couldn't find anywhere | 18:44 |
| clarkb | dmsimard: its in the logstash rules themselves iirc /me finds a link | 18:45 |
| dmsimard | clarkb: do you know where the logstash rules are ? system-config ? | 18:45 |
| mnaser | was the ML backed up by any chance? | 18:45 |
| mnaser | i just got a whooole bunch of emails all of a sudden | 18:46 |
| clarkb | dmsimard: https://git.openstack.org/cgit/openstack-infra/logstash-filters/tree/filters/openstack-filters.conf is the main set of rules, but the jenkin sstuff is set at https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/logstash/input.conf.erb | 18:46 |
| *** Rockyg has quit IRC | 18:46 | |
| dmsimard | clarkb: oh man that repo was way too obvious | 18:47 |
| clarkb | mnaser: I haven't noticed that | 18:48 |
| clarkb | mnaser: possible your mx was away for a bit? | 18:48 |
| mnaser | maybe... we use (please don't bite) google apps for our email so | 18:48 |
| mnaser | i dont understand what's going on, it might be an issue at their side, it's emails i've already read.. shown arrived a 2:17 am | 18:49 |
| mnaser | but headers show them from days ago | 18:49 |
| mnaser | curious | 18:49 |
| clarkb | dmsimard: wat we could do is change the type in that input file when we change to v3 | 18:50 |
| clarkb | dmsimard: and have e-r key off of that | 18:51 |
| clarkb | I'm not sure we actually use the type anywhere for anything important but we should check that first | 18:51 |
| *** rwsu has quit IRC | 18:51 | |
| dmsimard | clarkb: I was thinking something slightly different -- have the type be part of the data we push when we submit to gearman | 18:52 |
| clarkb | dmsimard: ya we could do that too | 18:52 |
| clarkb | dmsimard: I think you can have logstash set a default if there isn't data on the input either | 18:52 |
| dmsimard | clarkb: like here or something https://github.com/openstack-infra/project-config/blob/master/roles/submit-log-processor-jobs/library/submit_log_processor_jobs.py#L101 | 18:52 |
| clarkb | so jenkins could be the default then we override when we submit the jobs | 18:52 |
| *** yolanda has joined #openstack-infra | 18:54 | |
| *** dizquierdo has joined #openstack-infra | 18:54 | |
| clarkb | end of the day though that may be overthinking it | 18:57 |
| *** efried_nomnom is now known as efried | 18:58 | |
| rajinir | zuul issue: With the latest project-config, the old layout.yaml doesn't work. Is there a simple layout.yaml we can use? Like https://git.openstack.org/cgit/openstack-infra/project-config-example/ | 18:58 |
| clarkb | mordred: jeblair the v2 side of 511260 lgtm http://logs.openstack.org/60/511260/2/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/49d90ae/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-10-11_17_19_20_372 | 18:59 |
| clarkb | http://logs.openstack.org/60/511260/2/check/legacy-tempest-dsvm-neutron-full/133e3c0/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-10-11_17_16_11_691 but still not happy about something on v3 | 19:01 |
| *** masuberu has quit IRC | 19:01 | |
| clarkb | does that need more sudo? | 19:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 19:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 19:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 19:02 |
| pabelanger | clarkb: yah, zuul user won't be able to override jenkins | 19:03 |
| mordred | clarkb, pabelanger: it seems like that wants to run as stack or something? | 19:04 |
| clarkb | I think the stack user owns the destination side yes | 19:05 |
| jeblair | so why did this work when run as jenkins? | 19:05 |
| clarkb | sudo chown -R $USER:$USER $cache_dir <- is what the old code did /me looks at new change again | 19:06 |
| mordred | oh - we missed a chown | 19:07 |
| mordred | yay for gating | 19:07 |
| clarkb | ya I think you need the chown on the source side possibly | 19:07 |
| jeblair | $cache_dir is different before and after this change | 19:07 |
| clarkb | and I think dib may already make them jenkins'd | 19:07 |
| jeblair | i think we were effectively depending on the copy to change the perms for us before | 19:08 |
| mordred | yah - I agree | 19:08 |
| jeblair | okay, so yeah, we just need something new here :) | 19:08 |
| clarkb | ya copy was from /opt/cache/files -> /opt/stack/cache/files -> /opt/stack/new/devstack/files | 19:09 |
| jeblair | a chown on cache dir right before our "cp -l"? is that what you're suggesting? | 19:09 |
| clarkb | we remoed that first copy which I think must have figured out perms for us | 19:09 |
| clarkb | jeblair: yes | 19:09 |
| mordred | is ... | 19:09 |
| clarkb | chown /opt/cache/files to $USER:$USER | 19:09 |
| jeblair | this all sounds reasonable to me | 19:09 |
| mordred | wait though ... | 19:09 |
| mordred | we're copying them to $DEST/devstack/files/ ... permissions and ownership of that dir hasn't changed ... so are the files in /opt/cache/files not readable? | 19:10 |
| clarkb | mordred: it is possible we have perissions issues on both sides | 19:11 |
| mordred | I don't see any code that would affect write permissions in $DEST/devstack/files/ ... unless that just happens to pre-exist and is owned by jenkins | 19:11 |
| mordred | BUT - I guess we could chown both $DEST/devstack/files/ and /opt/cache/files to $USER:$USER | 19:12 |
| clarkb | I want to say we chown all the repos after all this is done? | 19:12 |
| clarkb | ya we chown $BASE to stack after the fact | 19:12 |
| clarkb | s oI think at that point its all owned by zuul | 19:13 |
| clarkb | except the cache which is owned by jenkins during the image build | 19:13 |
| clarkb | sudo chown -R $USER:$USER $DEST | 19:13 |
| clarkb | that makes the devstack repo owned by zuul | 19:13 |
| clarkb | and happens before we do cache things so I think its just the one side we need to worry about | 19:14 |
| *** caphrim007 has joined #openstack-infra | 19:14 | |
| mordred | clarkb: ok. shall I push up another patch? | 19:15 |
| mordred | clarkb: why we can't *read* from the cache dir is mind-boggling | 19:15 |
| clarkb | mordred: so I think its actually a write problem | 19:15 |
| clarkb | mordred: due to how hardlinks work | 19:15 |
| clarkb | you have to update the counter on the inode or something | 19:16 |
| mordred | clarkb: OH --- we don't have write permissions in the cache | 19:16 |
| mordred | yah | 19:16 |
| mordred | that makes much more sense to my brain hole - thank you | 19:16 |
| *** inc0 has joined #openstack-infra | 19:16 | |
| clarkb | there is definitley a write to say there is another "copy" of this thing | 19:16 |
| clarkb | I think on the inode record and the original file must control access ot that? this is actually a corner of unix file perms that I'm not super familiar with | 19:16 |
| *** vsaienk0 has quit IRC | 19:16 | |
| mordred | clarkb: same here- ready for meto push up a new PS? | 19:17 |
| clarkb | ya I think so | 19:17 |
| jeblair | cp -l /etc/passwd . | 19:17 |
| jeblair | cp: cannot create hard link './passwd' to '/etc/passwd': Operation not permitted | 19:17 |
| jeblair | ^ seems to support that theory :) | 19:17 |
| *** slaweq has joined #openstack-infra | 19:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files https://review.openstack.org/511260 | 19:19 |
| mordred | there we go | 19:19 |
| clarkb | mordred: that should go in the if check | 19:20 |
| clarkb | left comment on change | 19:20 |
| jeblair | are we running as $USER there? | 19:20 |
| rajinir | clarkb: zuul issue: With the latest project-config, the old layout.yaml doesn't work. Is there a simple layout.yaml we can use? Like https://git.openstack.org/cgit/openstack-infra/project-config-example/ | 19:20 |
| jeblair | or are we going to chown it to stack and then try to hardlink it as zuul? | 19:20 |
| mordred | clarkb: doh | 19:21 |
| clarkb | pretty sure it is running as zuul there | 19:21 |
| clarkb | jeblair: ^ | 19:21 |
| clarkb | jeblair: since it happens in -wrap and there isn't an explicit sudo | 19:22 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: stop updating rtfd for whereto https://review.openstack.org/511324 | 19:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files https://review.openstack.org/511260 | 19:22 |
| clarkb | rajinir: can you be more specific, what version of zuul and how does it not work anymore? we are still using our layout.yaml for example | 19:22 |
| rajinir | clarkb: using this https://git.openstack.org/cgit/openstack-infra/project-config-example/tree/zuul/layout.yaml didn't work | 19:23 |
| clarkb | rajinir: how did it fail? | 19:23 |
| AJaeger | https://review.openstack.org/#/c/511290 and https://review.openstack.org/511287 are ready for merge if anybody has time for a quick review | 19:23 |
| rajinir | :wq | 19:24 |
| rajinir | clarkb: the new queues don't show up. no proper clue in the logs | 19:25 |
| mordred | AJaeger: both look great - https://review.openstack.org/#/c/511318 has an issue | 19:25 |
| clarkb | rajinir: can you run the layout checker against it? | 19:25 |
| jeblair | clarkb, mordred: so do we need to chown to zuul, then cp -l, then chown to stack? | 19:26 |
| rajinir | clarkb: with the default layout from the latest project-config, all the queues show up | 19:26 |
| clarkb | jeblair: yes, however the chown to stack happens for us already in devstack-vm-gate.sh | 19:26 |
| rajinir | clarkb: https://usercontent.irccloud-cdn.com/file/AWlMyf9s/image.png | 19:26 |
| clarkb | rajinir: probably the best way to diagnose it is to run the layout checker command against it | 19:26 |
| rajinir | clarkb: is there an example how to run it? | 19:27 |
| jeblair | clarkb: ok | 19:27 |
| clarkb | rajinir: zuul-server -c tools/zuul.conf-sample -l zuul/layout.yaml -t {envdir}/tmp/job-list.txt is how the gate jobs do it. The -t flag is what tells it to validate | 19:29 |
| jeblair | rajinir: can you check the debug log? if zuul didn't start up properly, there should be something there. | 19:29 |
| *** jamesdenton has joined #openstack-infra | 19:29 | |
| rajinir | clarkb: what is job-list.txt, is that new? | 19:31 |
| *** mat128 has joined #openstack-infra | 19:31 | |
| clarkb | rajinir: its a list of job names to check against, the command help output says it is optional though | 19:31 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: remove legacy openstackdocstheme jobs https://review.openstack.org/511289 | 19:31 |
| rajinir | clarkb: jeblair: all these have changed quiet a bit. But the https://git.openstack.org/cgit/openstack-infra/project-config-example/tree/zuul/ is not updated | 19:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use build-openstack-api-ref for openstackdocstheme https://review.openstack.org/511318 | 19:33 |
| *** vhosakot has quit IRC | 19:33 | |
| *** vhosakot has joined #openstack-infra | 19:34 | |
| *** coolsvap has quit IRC | 19:36 | |
| rajinir | jebliar: clarkb: WARNING zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. Can not get account information. | 19:36 |
| rajinir | jebliar: clarkb: may be gearman issue? | 19:37 |
| clarkb | rajinir: that is mostly expected | 19:37 |
| clarkb | (depends on your version of gerrit but doesn't indicate a problem) | 19:38 |
| jeblair | rajinir: nothing in the master branch of zuul has changed to cause the project-config-example to break | 19:38 |
| jeblair | we should probably update the case of the label tags, but that wouldn't cause zuul not to start up | 19:38 |
| jeblair | i mean nothing should cause it to break | 19:39 |
| *** makowals_ has joined #openstack-infra | 19:39 | |
| rajinir | clarkb: jebliar: good to know. | 19:39 |
| jeblair | obviously something is broken, but there have been almost no changes to zuul master for months, aside from the case issue. | 19:39 |
| *** e0ne has joined #openstack-infra | 19:40 | |
| rajinir | clarkb: jebliar: gearman is rinning, I tested the connection from jenkins | 19:43 |
| rajinir | jebliar: clarkb: But the pipeline eques check and silient are not showing up in the zuul web UI | 19:43 |
| rajinir | https://www.irccloud.com/pastebin/9JqMdD9P/status | 19:44 |
| rajinir | clarkb: jebliar: status of the gearman server | 19:44 |
| *** ykarel|afk has quit IRC | 19:50 | |
| clarkb | I've got to go grab lunch now back in a bit | 19:50 |
| *** Apoorva_ has joined #openstack-infra | 19:58 | |
| *** harlowja has quit IRC | 20:00 | |
| openstackgerrit | Benjamin Navaro proposed openstack-infra/project-config master: Online inap-mtl01 region https://review.openstack.org/511328 | 20:02 |
| *** Apoorva has quit IRC | 20:02 | |
| *** ociuhandu has quit IRC | 20:02 | |
| benj_ | Fyi. The inap-mtl01 region is back online. Hence this review. | 20:04 |
| rajinir | clarkb>: jebliar: Solved it. function single_use_node is missing in the openstackfunctions.py, adding it resolved the issue | 20:05 |
| *** apetrich has quit IRC | 20:07 | |
| *** sree has joined #openstack-infra | 20:07 | |
| *** edmondsw_ has joined #openstack-infra | 20:08 | |
| pabelanger | benj_: great | 20:08 |
| *** edmondsw has quit IRC | 20:08 | |
| *** apetrich has joined #openstack-infra | 20:10 | |
| *** edmondsw_ is now known as edmondsw | 20:12 | |
| *** sree has quit IRC | 20:12 | |
| pabelanger | clarkb: mordred: ^511328 bring inap backonline, if you want to =# | 20:12 |
| pabelanger | +3 | 20:12 |
| *** gildub has joined #openstack-infra | 20:13 | |
| rajinir | clarkb>: jebliar: thanks for the tips | 20:14 |
| dansmith | are we mid-transition now? | 20:15 |
| dansmith | the last I saw this morning made it sound like it was on hold until further notice, but I just saw a job with two POST_FAILURE jobs... | 20:15 |
| pabelanger | dansmith: no, we haven't started as of yet | 20:15 |
| eumel8 | pabelanger: something news from the mirror front? | 20:15 |
| pabelanger | eumel8: not yet, sadly | 20:16 |
| jeblair | dansmith: link to failures? | 20:16 |
| dansmith | jeblair: http://logs.openstack.org/02/509002/7/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/e352110/ | 20:16 |
| *** cody-somerville has quit IRC | 20:16 | |
| dansmith | that was reported as POST_FAILURE, but it seems fine in the console log | 20:16 |
| mordred | jeblair: https://review.openstack.org/#/c/511328/ has tw POST_FAILURES from v3 - I'm just now starting to look | 20:17 |
| jeblair | dansmith: that one looks like a v2 job | 20:17 |
| dansmith | it is | 20:17 |
| dansmith | that's why I was asking if we were cutting over and things running in v2 should be expecting to fail | 20:17 |
| pabelanger | we are getting close on filling /srv/static/log: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 20:17 |
| eumel8 | pabelanger: thx, then I'll schedule the rechecks for tomorrow | 20:18 |
| pabelanger | maybe we should start a purne just to be safe | 20:18 |
| mordred | failed: No space left on device (28) | 20:18 |
| mordred | pabelanger: we'r enot just close -we're getting errors | 20:18 |
| dansmith | mordred: where did you see that? | 20:18 |
| jeblair | well, there we go. that will affect both. | 20:18 |
| pabelanger | mordred: eep | 20:18 |
| mordred | dansmith: debug logs on ze04.openstack.org | 20:18 |
| dansmith | oh sorry, thought that was related to my thing | 20:19 |
| jeblair | dansmith: very likely it is | 20:19 |
| mordred | dansmith: well- if we're having space issues on logs.o.o - that would totally cause POST_FAILURES | 20:19 |
| dansmith | ah okay | 20:19 |
| jeblair | probably failed to copy something (though obviously copied some stuff) | 20:20 |
| mordred | pabelanger: did you start a prune? | 20:20 |
| dansmith | was trying to figure out where in my logs I might've seen a failure | 20:20 |
| pabelanger | mordred: I have not | 20:20 |
| mordred | /dev/mapper/main-logs 12T 12T 713G 95% /srv/static/logs | 20:20 |
| pabelanger | we look to have 5% left, | 20:20 |
| pabelanger | yah | 20:20 |
| jeblair | that will be the root allocation | 20:20 |
| jeblair | so we're at 0 space available for non-root users | 20:20 |
| * dansmith runs /nick nagios | 20:20 | |
| mordred | ah | 20:20 |
| mordred | dansmith: you're the best nagios we know :) | 20:21 |
| jeblair | i don't think we need any root space on that partition. bit of an oversight there | 20:21 |
| jeblair | infra-root: any objections to me freeing the root reserved blocks on the logs partition? | 20:21 |
| mordred | jeblair: nope. none from me | 20:21 |
| pabelanger | none from me | 20:22 |
| jeblair | we should *also* prune, because that's not going to be enough or last very long | 20:22 |
| fungi | okay, i have returned | 20:22 |
| fungi | ~100 lines of scrollback... i'll be with you all in a moment ;) | 20:22 |
| mordred | infra-root: /usr/local/sbin/log_archive_maintenance.sh is running - and has been since Sept 30 | 20:23 |
| *** slaweq has quit IRC | 20:24 | |
| jeblair | wow, erm, apparently we already don't have reserved blocks on that filesystem: Reserved block count: 0 | 20:25 |
| *** armax has quit IRC | 20:25 | |
| jeblair | we may have hit the inode limit slightly early | 20:25 |
| jeblair | /dev/mapper/main-logs 805306368 805305720 648 100% /srv/static/logs | 20:25 |
| jeblair | (on the plus side -- wow we have tuned those really close :) | 20:25 |
| clarkb | all that /etc logging | 20:25 |
| mtreinish | hmm, wonder if the missing console log: http://logs.openstack.org/83/511283/1/check/gate-tempest-dsvm-py35-ubuntu-xenial/ed1e53b/ is related to that? | 20:26 |
| mordred | mtreinish: certainly could be | 20:27 |
| openstackgerrit | Anup Navare proposed openstack-infra/devstack-gate master: [Test] DNM Checking if tinyIPA builds with py3 https://review.openstack.org/509641 | 20:27 |
| mordred | man - we're having ALL of the fun issues today | 20:27 |
| mordred | also - running double jobs isn't just stressful for our node quota - tunrs out it'stwice the logging ... | 20:27 |
| dmsimard | who would've thought :( | 20:28 |
| jeblair | well, 1.25x the jobs, but yeah | 20:28 |
| *** andreas_s has joined #openstack-infra | 20:30 | |
| *** slaweq has joined #openstack-infra | 20:32 | |
| fungi | 1.25x the job volume when you were over 80% used to start with is, well... yeah i guess we should have seen this coming | 20:33 |
| mordred | jeblair: fwiw, I'm not doing anything on that server because you're on and doing things and double-rooting is likely counter-productive, so please let me know if I can make the help directly | 20:33 |
| jeblair | mordred: oh i'm not doing anything | 20:33 |
| mordred | jeblair: darn. I don't have any excuses to not do anything | 20:34 |
| jeblair | there is apparently nothing to do regarding reserved blocks | 20:34 |
| fungi | df -h | 20:34 |
| * fungi grumbles at wrong window | 20:34 | |
| jeblair | maybe we can increase the inode count though? | 20:35 |
| *** andreas_s has quit IRC | 20:35 | |
| pabelanger | okay, I think AFS mirror for ubuntu is ready | 20:35 |
| fungi | afaik that involves a complete reformat | 20:35 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Revert "neutron: Make grenade-neutron-dvr-multinode job non-voting" https://review.openstack.org/505318 | 20:35 |
| pabelanger | just checking another thing and will be able to release | 20:35 |
| jeblair | fungi: i think *something* about that changed in ext4, but i don't recall details | 20:35 |
| mordred | should we stop the current cleaner cron and restart it it with a more aggressive mtime ? | 20:35 |
| fungi | jeblair: yeah, i'm currently pouring over docs to find out | 20:36 |
| jeblair | mordred: is it still doing things, or is it stuck? | 20:36 |
| mordred | jeblair: tough to say - it's a single find command | 20:36 |
| jlvillal | With zuul v3 and in project job definitions. I see a proposed patch that uses a zuul.d/ directory. Will that work? | 20:36 |
| mordred | jlvillal: yup | 20:36 |
| jlvillal | Also, can we use a different directory name then playbook/ ? | 20:36 |
| jlvillal | Can we put the playbooks under zuul.d/ ? | 20:36 |
| jeblair | mordred: yes, the find command is proceeding | 20:36 |
| jlvillal | mordred: thanks | 20:36 |
| mordred | jlvillal: don't put paybooks in zuul.d | 20:36 |
| * jlvillal slightly sad about that | 20:37 | |
| mordred | jlvillal: zuul will try to read them as config :) | 20:37 |
| jlvillal | mordred: Can we put them in zuul_playbooks/ ? | 20:37 |
| mordred | jlvillal: you can put themanywhere else you want - the path is explicitin the run: pre-run: post-run: in the job configs | 20:37 |
| jlvillal | Or does it have to be playbooks/ | 20:37 |
| mordred | jlvillal: just not in zuul.d - zuul.d is special | 20:37 |
| jlvillal | mordred: Thanks! | 20:37 |
| dmsimard | mordred: roles might need to be in roles/, though ? | 20:37 |
| mordred | jlvillal: roles must be in roles | 20:38 |
| dmsimard | I can see how/why users might want to centralize zuul/ci things in a single directory though | 20:38 |
| jlvillal | mordred: Uh, as in a top level roles/ directory ? | 20:38 |
| *** priteau has quit IRC | 20:38 | |
| jlvillal | Or as in playbooks/roles/ zuul_playbooks/roles ? | 20:38 |
| tosky | yep, I asked the same during the PTG | 20:39 |
| tosky | I understand that zuul.d is special, but it's just a convention | 20:39 |
| tosky | I mean, technically nothing prevents for having a zuul directory with zuul.d for jobs, and then playbooks and roles | 20:39 |
| jlvillal | tosky: Yep to what? | 20:39 |
| pabelanger | clarkb: I've started vos release mirror.ubuntu | 20:39 |
| mordred | jlvillal: roles is special because it has to do with how zuul checks out and puts roles into the role path for jobs to share roles across repos | 20:39 |
| *** dprince has quit IRC | 20:40 | |
| tosky | jlvillal: yep to "people may want to centralize all zuul stuff" | 20:40 |
| mnaser | by the way do we have to manually specify the name of the repo itself inside roles .. i think that was an issue i ran into a few days ago | 20:40 |
| fungi | jeblair: the manpage for mkfs.ext4 seems to be shared for ext2 and ext3 but still claims the inode ratio can't be adjusted after the filesystem is created | 20:40 |
| dmsimard | mnaser: the name of the repo itself ? | 20:40 |
| mordred | mnaser: no - you should not need to specify the nameof the repo inside of roles | 20:40 |
| tosky | mordred: but is there a reason why role must be top level? Couldn't zuul check it from, say zuul/roles instead of roles? | 20:40 |
| jlvillal | mordred: Okay. So does that mean roles need to be in a top-level roles/ directory in the project? Or is it stored in zuul.d/? Or somewhere else? | 20:40 |
| mnaser | ex: if my playbook is playbooks/test.yaml, then it would look for roles in playbooks/roles/<xyz> but not roles/<xyz> | 20:41 |
| mnaser | let me show an example of a failure i had | 20:41 |
| mnaser | ah you know what, nevermind, i was running ansible-playbook adhoc and it wasnt zuul calling it (little "compatibility"/"migration" layer while we have zuulv2) | 20:41 |
| dmsimard | mnaser: sounds like a default ansible behavior to seek out roles nested under the directory in which the playbook is | 20:41 |
| mordred | mnaser: right. that's a diffrent thing | 20:41 |
| dmsimard | well there you go | 20:41 |
| jlvillal | The patch I am looking at doesn't have roles, so maybe I shouldn't worry about it :) | 20:41 |
| jeblair | jlvillal, mordred, dmsimard: roles only need to be in roles/ if you are sharing them with other repos | 20:41 |
| fungi | jeblair: mordred: we should move discussion about the logserver outage to the incident channel since it's hard to wade through the ansible discussions in here at the moment | 20:41 |
| mnaser | yeah, my bad, just kinda clicked as i was talking out loud | 20:41 |
| mnaser | :p | 20:42 |
| mordred | jlvillal: nothing can go in zuul.d except zuul config | 20:42 |
| jeblair | fungi: agreed | 20:42 |
| mordred | butyes - what jeblair just said - if the roles are not intended to be shared with other repos, they can totally go adjacent to your playbooks | 20:42 |
| jlvillal | jeblair: mordred: thanks | 20:42 |
| mnaser | https://review.openstack.org/#/c/510611/ <== pattern for migrating really big jobs imho | 20:42 |
| pabelanger | Released volume mirror.ubuntu successfully | 20:43 |
| tosky | jeblair: so there is no way to change zuul to have shared roles, by default under another special directory? | 20:43 |
| jeblair | tosky: no -- we're trying not to add anything to the standard ansible ways of doing things. most ansible roles are either a single repo, or have a roles/ directory at the top level. | 20:44 |
| tosky | jeblair: but openstack repositories are not only ansible repositores; I disagree with this choice | 20:45 |
| tosky | but that's as much as I can say and do | 20:46 |
| pabelanger | fungi: I'm not sure why 511328 wasn't enqueued into gate | 20:47 |
| pabelanger | oh, maybe -1 from zuul | 20:48 |
| jeblair | tosky: if you have no intention of sharing the ansible roles you write with anyone outside of your repository, please put them wherever you want! | 20:48 |
| pabelanger | clarkb: sure we add nodepool.o.o to emergency and manually apply 511328 to bring inap-mtl01 back online? | 20:49 |
| tosky | jeblair: sure, so another repository for the tests for each component | 20:49 |
| jeblair | that's what i was trying to say earlier -- there is no requirement that roles be under roles/ -- unless they are to be shared outside of the repo | 20:49 |
| clarkb | pabelanger: maybe make sure logs is happy first? | 20:49 |
| *** kgiusti has left #openstack-infra | 20:49 | |
| clarkb | otherwise its just more pressure on that fs | 20:49 |
| pabelanger | clarkb: ya, I _think_ mordred is running clean up? | 20:50 |
| mordred | nope | 20:50 |
| *** jkilpatr has quit IRC | 20:50 | |
| mordred | pabelanger, clarkb: we've shifted to incident channel | 20:50 |
| pabelanger | 1k | 20:50 |
| jeblair | tosky: consider though that if done carefully, roles that you write to aid in zuul testing your project may be useful to people outside the context of zuul. in that case, it may be appropriate to have them in a top level roles/ directory. | 20:53 |
| jeblair | (even if the project itself isn't otherwise strongly ansible focused. think of it like having a docker or vagrant file, etc...) | 20:54 |
| *** thorst_ has quit IRC | 20:55 | |
| tosky | jeblair: but then they are probably candidate to be moved somewhere else | 20:55 |
| tosky | in a really native ansible (maybe galaxy) way | 20:55 |
| *** ccamacho has quit IRC | 20:55 | |
| jeblair | tosky: possibly so, especially for larger multi-repo projects | 20:55 |
| jeblair | tosky: well, this is all designed to be compatible with galaxy | 20:55 |
| jeblair | (with the added benefit that if the repo is hosted in a system zuul knows about, you can use depends-on) | 20:56 |
| *** thorst has joined #openstack-infra | 20:58 | |
| *** eumel8 has quit IRC | 20:59 | |
| *** trown is now known as trown|outtypewww | 20:59 | |
| *** makowals_ has quit IRC | 21:00 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Adding new project Tatu (SSH as a Service). https://review.openstack.org/511335 | 21:01 |
| *** dprince has joined #openstack-infra | 21:02 | |
| *** dprince has quit IRC | 21:02 | |
| *** esberglu has quit IRC | 21:03 | |
| *** thorst has quit IRC | 21:03 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Adding new project Tatu (SSH as a Service). https://review.openstack.org/511335 | 21:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 21:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 21:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 21:06 |
| *** dprince has joined #openstack-infra | 21:06 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add documentation on manually upgrading a StoryBoard instance https://review.openstack.org/510669 | 21:07 |
| *** edmondsw has quit IRC | 21:07 | |
| *** harlowja has joined #openstack-infra | 21:07 | |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Adding new project Tatu (SSH as a Service). https://review.openstack.org/511335 | 21:11 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Add Zuul merge-check template. https://review.openstack.org/511339 | 21:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Stop mirroring openSUSE 42.2 https://review.openstack.org/502316 | 21:12 |
| *** florianf has quit IRC | 21:12 | |
| pabelanger | ianw: clarkb: ^mind reviews, and approves. I've manually applied that to mirror-update.o.o already | 21:13 |
| clarkb | pabelanger: there is already a change for that | 21:13 |
| clarkb | from dirk iirc | 21:13 |
| pabelanger | clarkb: that is the change, I just removed the rm -rf | 21:13 |
| clarkb | oh got it | 21:13 |
| *** jtomasek has quit IRC | 21:14 | |
| clarkb | I've just finished eating some bún will review shortly | 21:14 |
| *** edmondsw has joined #openstack-infra | 21:14 | |
| jeblair | clarkb: eating some what? | 21:15 |
| SamYaple | exactly | 21:15 |
| SamYaple | what | 21:15 |
| efried | https://en.wikipedia.org/wiki/Bun_cha | 21:15 |
| jeblair | mm that looks tasty | 21:16 |
| SamYaple | mmmmm | 21:16 |
| clarkb | its excellent stuff | 21:16 |
| SamYaple | i would eat that | 21:16 |
| pabelanger | now I want that | 21:16 |
| mordred | yah. me too | 21:16 |
| mordred | I'm very sad that I'm not eating that rightnow | 21:16 |
| efried | mtreinish I'm about to restack - what's the status of the glance-filling-up-/tmp thing? Want me to try anything special? | 21:17 |
| *** dprince has quit IRC | 21:19 | |
| *** edmondsw has quit IRC | 21:19 | |
| clarkb | pabelanger: did vos release complete or is that what precipitated needing to clean out suse 42.2? | 21:19 |
| *** slaweq has quit IRC | 21:20 | |
| pabelanger | clarkb: for ubuntu mirror, ya. I've also done it on opensuse mirror too | 21:22 |
| pabelanger | clarkb: I'm trying to validate if fallback-bindep is working again | 21:22 |
| clarkb | thanks | 21:22 |
| *** slaweq has joined #openstack-infra | 21:22 | |
| mtreinish | efried: you need to pull those 2 patches to fix it. It's blocked on glance reviews (no +2s on it at all) | 21:23 |
| efried | mtreinish Roger wilco, pulling now. | 21:23 |
| rajinir | Jenkins fails to launch the slaves: the image built by the disk-image-builder is missing java. Any idea of how to add that element? | 21:23 |
| pabelanger | clarkb: okay, http://zuulv3.openstack.org/static/stream.html?uuid=aa007097e6a54b26a04986ffd0077c13&logfile=console.log looks to have worked | 21:24 |
| pabelanger | which appears to be using bindep-fallback.txt | 21:24 |
| clarkb | v2 looks happy with 511260, http://logs.openstack.org/60/511260/4/check/gate-tempest-dsvm-multinode-full-ubuntu-xenial-nv/feec631/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-10-11_19_34_49_021 I don't think we have any logs on the v3 side yet due to the log server fun | 21:24 |
| *** slaweq has quit IRC | 21:27 | |
| pabelanger | okay, I have to step away to eat. I'll return shortly | 21:27 |
| *** armax has joined #openstack-infra | 21:28 | |
| *** jkilpatr has joined #openstack-infra | 21:28 | |
| *** mat128 has quit IRC | 21:28 | |
| *** cody-somerville has joined #openstack-infra | 21:29 | |
| *** Apoorva_ has quit IRC | 21:29 | |
| *** Apoorva has joined #openstack-infra | 21:30 | |
| *** ociuhandu has joined #openstack-infra | 21:30 | |
| *** e0ne has quit IRC | 21:31 | |
| *** wolverineav has quit IRC | 21:31 | |
| *** esberglu has joined #openstack-infra | 21:34 | |
| *** ldnunes has quit IRC | 21:37 | |
| *** hashar has quit IRC | 21:37 | |
| clarkb | fungi: ianw pabelanger I've started to put info on https://etherpad.openstack.org/p/infra-sydney-summit-project-update | 21:38 |
| dmsimard | you guys have fun in sydney, I'm not going, it's too damn far :( | 21:39 |
| clarkb | dmsimard: can confirm it is not a short trip. Really makes you appreciate how far people in australia have to travel for anything | 21:40 |
| dmsimard | definitely | 21:40 |
| *** thorst has joined #openstack-infra | 21:40 | |
| dmsimard | and the jet lag, oh boy the jet lag | 21:40 |
| dmsimard | I've seen people with worse expected flights but for me it'd be a minimum of 24h of flight/connecting and it's +14hrs there (5:42PM here, 8:42AM there). I would be totally messed up. | 21:42 |
| dmsimard | Going to Tokyo was fine, coming back took 3 weeks to recover | 21:43 |
| tosky | (or -10hrs?) | 21:44 |
| jeblair | my favorite part is that going there i put a day on deposit, but it's returned to me when i come back. | 21:45 |
| mnaser | im considering extending my stay and discovering australia | 21:45 |
| mnaser | might as well if im down there | 21:45 |
| * mordred is going early - will be vacationing for a bit in new zealand, then going to scuba the great barrier reef ... I shoujld be quite happy by the time I make it to sydney | 21:46 | |
| jlvillal | Any issue with putting playbooks in "zuul/playbooks/", not "zuul.d/" Or is "zuul/" also a special directory? | 21:47 |
| inc0 | isn't great barrier reef dead? | 21:47 |
| clarkb | inc0: I think there are portions of it that are unhappy but it is quite massive. | 21:47 |
| *** eharney has quit IRC | 21:47 | |
| clarkb | also apparently certain species are thriving in the changing conditions | 21:47 |
| inc0 | that'd be good, I've heard some grim scenarios about i | 21:48 |
| inc0 | t | 21:48 |
| mnaser | maybe mordred can tell us how it looks like at the summit | 21:49 |
| mnaser | :) | 21:49 |
| *** ldnunes has joined #openstack-infra | 21:49 | |
| clarkb | I tried to scheme up a way I could visit my parents on my way to sydney | 21:50 |
| clarkb | but it makes the plane tickets about 3x more and I don't really want to fly air niugini | 21:50 |
| clarkb | or air nauru | 21:50 |
| jlvillal | clarkb: Are your parents retired and doing low-cost living? | 21:51 |
| * jlvillal dreams of that :) | 21:51 | |
| clarkb | jlvillal: they live in micronesia (I grew up there) and happen to be semi retired | 21:52 |
| pabelanger | jlvillal: ansible playbooks can go into any directory, you point to them in pre-run, run, post-run. However, zuul jobs need to be zuul.yaml / .zuul.yaml / zuul.d / .zuul.d | 21:52 |
| jlvillal | clarkb: Ah cool. At one time I was thinking about trying to get a job on the Marshall Islands. But it didn't happen. | 21:52 |
| jlvillal | pabelanger: Thanks. Good to know "zuul/" isn't special. | 21:53 |
| clarkb | jlvillal: neat, on kwaj? | 21:53 |
| clarkb | the marshalles are really hurting with the whole ocean levels are rising thing | 21:53 |
| jlvillal | clarkb: yeah Kwajalein. I was chatting with a guy on the internet who worked there. I did a TSR program for DOS at the time to help him out. | 21:53 |
| jlvillal | Good old "Spontaneous Assembly" library. Made writing TSR code pretty easy. | 21:54 |
| clarkb | kwaj is an interesting place, I've never gotten off of the plane there but flown through many times | 21:54 |
| jlvillal | clarkb: Nice :) | 21:54 |
| clarkb | ebeye is one of the most densely populated places in the world | 21:55 |
| *** threestrands has joined #openstack-infra | 21:56 | |
| jlvillal | clarkb: I didn't know. I do think I would probably get island fever after awhile. Not sure how I would react to living somewhere so small. | 21:56 |
| dmsimard | clarkb: wow TIL about Micronesia, which happens to hold the TLD ".fm". Turns out I live on an island about half it's size. Looks super pretty on pictures :) | 21:56 |
| dmsimard | oh, nevermind the size, wolfram alpha and wikipedia disagree | 21:57 |
| *** jkilpatr has quit IRC | 21:57 | |
| *** jkilpatr has joined #openstack-infra | 21:57 | |
| clarkb | dmsimard: the island my parents live on is about 80km in circumference | 21:58 |
| jlvillal | dmsimard: I don't think any one island is very big. Though you make me curious which island you live on :) I lived on an island before, Great Britain, but it was pretty big. | 21:58 |
| dmsimard | jlvillal: Montreal :) | 21:58 |
| jlvillal | Any one island in Micronesia that is. | 21:58 |
| clarkb | jlvillal: ya I was super ready to get off the island by the time I was done with high school | 21:58 |
| jlvillal | :) | 21:59 |
| dmsimard | jlvillal: we have like 13 bridges and tunnels to go in an out of the island, it's a big city :) | 22:00 |
| jlvillal | dmsimard: Ah, an island in between rivers :) | 22:00 |
| *** ijw has quit IRC | 22:00 | |
| jlvillal | dmsimard: I was thinking in the ocean. | 22:00 |
| dmsimard | oh, yeah, nothing like out in the middle of the pacific ocena | 22:00 |
| *** ijw has joined #openstack-infra | 22:00 | |
| *** cody-somerville has quit IRC | 22:01 | |
| dmsimard | Living on one of those islands must be special and sounds like where I'd end up retiring one day. Away from all the craziness of the cities :D | 22:01 |
| bkero | I'd like a place on a tropical island | 22:01 |
| bkero | A little bungalow retreat when it's cold and awful in Portland. | 22:01 |
| bkero | Some satellite internet, solar panels, rainwater catchment, and fishing pole | 22:02 |
| jlvillal | bkero: We have sunshine today here in the Portland area. | 22:03 |
| * jlvillal waves from Hillsboro | 22:03 | |
| fungi | my island is only ~10mi offshore, and we have two bridges to the mainland. i call it a "starter island" | 22:03 |
| *** jaypipes has quit IRC | 22:03 | |
| fungi | maybe in a few years i'll graduate to one you can only reach by air and sea | 22:04 |
| *** claudiub|2 has quit IRC | 22:04 | |
| bkero | jlvillal: Yeah, I saw a few glimpses | 22:08 |
| *** rlandy has quit IRC | 22:08 | |
| mnaser | i remember reading about this island which is the farthest point/island from land ever | 22:11 |
| *** baoli has quit IRC | 22:11 | |
| mnaser | sits on an active volcano | 22:11 |
| *** bobh has quit IRC | 22:11 | |
| jlvillal | mnaser: Oh yeah. That is a British colony. I saw a video about it. | 22:12 |
| mnaser | apparently only reachable by boat, which goes 6-8 times per year, it has no airport | 22:12 |
| jlvillal | mnaser: Tristan da Cunha | 22:12 |
| ianw | well instead of walking into my study, i'm going to have to catch a bus to the city. we all have our pain :) | 22:12 |
| mnaser | jlvillal: yep! | 22:13 |
| clarkb | re 511260 on v3 http://logs.openstack.org/60/511260/4/check/legacy-grenade-dsvm-neutron-multinode/23d78bd/logs/devstack-gate-setup-workspace-old.txt.gz#_2017-10-11_21_06_18_231 lgtm | 22:13 |
| clarkb | jeblair: mordred ^ do we want to enqueue that in the gate once logs.o.o is happier? | 22:13 |
| clarkb | I guess we should see where it grabs get-pip.py | 22:13 |
| fungi | that will be a good litmus test | 22:14 |
| clarkb | http://logs.openstack.org/60/511260/4/check/legacy-grenade-dsvm-neutron-multinode/23d78bd/logs/grenade.sh.txt.gz#_2017-10-11_21_08_19_486 that looks good too | 22:14 |
| clarkb | it seems to skip the download and sets the file path to the cache location | 22:14 |
| jeblair | clarkb: ya; sounds like as soon as we think it has a snowball's chance of uploading logs, we should gate enqueue it | 22:15 |
| jeblair | though.... | 22:15 |
| *** kjackal_ has quit IRC | 22:15 | |
| jeblair | clarkb, fungi, mordred: probably we should at least respond to ianw's comment first :) | 22:15 |
| * clarkb looks | 22:16 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Collect mistral-ansible execution files from /tmp" https://review.openstack.org/511347 | 22:16 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Collect mistral-ansible execution files from /tmp" https://review.openstack.org/511347 | 22:16 |
| clarkb | jeblair: ianw aha re that its what we've done all along so its possible it doesn't work in every case | 22:16 |
| ianw | jeblair: i may be wrong? | 22:16 |
| clarkb | ianw: jeblair but I don't think that bit is a regression | 22:16 |
| clarkb | but also we put all of /opt under the ephemeral drive on rax so at least for us I think it will work | 22:17 |
| clarkb | third party ci is where we'd have to be most concerned | 22:17 |
| ianw | so previously it was "if ~/cache/files then move it to $BASE/cache/files" | 22:20 |
| ianw | then symlink $BASE/cache/files to $DEST/devstack/files | 22:20 |
| jeblair | ya, i think we control enough of this in infra to know that it's okay (/opt is always a single fs). but 3rd party ci could be different. i think that's fairly unlikely though and think it's reasonable for us to merge this under that assumption. | 22:21 |
| clarkb | s/sym/hard/ ? | 22:21 |
| ianw | sorry hard, yep | 22:21 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Add Zuul merge-check template. https://review.openstack.org/511339 | 22:21 |
| mordred | infra-root, EmilienM: what do people think of potentially force-merging https://review.openstack.org/511347 to stop the influx of inodes to the log server? | 22:21 |
| EmilienM | mordred: ++ | 22:21 |
| clarkb | mordred: its probably a good idea since everything is affected right now | 22:21 |
| dmsimard | +1 | 22:21 |
| EmilienM | there is no way this patch breaks our CI | 22:22 |
| EmilienM | (more than it is right now :D) | 22:22 |
| EmilienM | mordred: go ahead | 22:22 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Revert "Collect mistral-ansible execution files from /tmp" https://review.openstack.org/511347 | 22:22 |
| mordred | done | 22:22 |
| mordred | EmilienM, weshay|ruck thanks! | 22:22 |
| weshay|ruck | np | 22:22 |
| EmilienM | mordred: hey can you force merge all my patches in tripleo? lol | 22:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/tripleo-ci master: Stop collecting ephemeral temp dirs https://review.openstack.org/511348 | 22:23 |
| mordred | EmilienM: hehe. I promise - you don't actually want that :) | 22:23 |
| mordred | EmilienM, weshay|ruck: ^^ there's a followup | 22:23 |
| mordred | I don't know that it's strictly required - it's possible those .~tmp~ dirs were leftovers from an upload that ran out of inodes | 22:23 |
| fungi | yup, sounds good to me (and already done) | 22:23 |
| EmilienM | mordred: go ahead, merge this one too | 22:24 |
| mordred | BUT - they are certainly not things that are useful | 22:24 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: remove the collection of /tmp/* from upstream jobs https://review.openstack.org/511349 | 22:24 |
| EmilienM | I agree | 22:24 |
| jeblair | late ++ for force merge :) | 22:24 |
| EmilienM | weshay|ruck: +2 +A on ^ | 22:24 |
| mordred | I'm a little more reluctant to force-merge the .~tmp~ change without seeing it work being passed through yaml to rsync | 22:24 |
| EmilienM | mordred: https://review.openstack.org/511349 too | 22:24 |
| EmilienM | mordred: ok we can wait on this one | 22:25 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: remove the collection of /tmp/* from upstream jobs https://review.openstack.org/511349 | 22:25 |
| *** edmondsw has joined #openstack-infra | 22:25 | |
| fungi | mordred: want to go ahead and do a separate find -name to remove existing ones? | 22:26 |
| mordred | fungi: not a bad idea - or we could update the curent script and restart it - which do you think is better? | 22:26 |
| fungi | i would leave it running so it doesn't lose its place, and start a second pass targeting the tmpfiles specifically | 22:27 |
| *** wolverineav has joined #openstack-infra | 22:27 | |
| fungi | which should in theory complete much more quickly | 22:27 |
| fungi | if they're dirs we can match on -type d and then use deltree or whatever option that is | 22:28 |
| * fungi checks | 22:28 | |
| mordred | yah - they're all dirs | 22:28 |
| *** dizquierdo has quit IRC | 22:29 | |
| clarkb | EmilienM: mordred there was also a place where we were copying all of /etc from the test node | 22:29 |
| clarkb | I don't have any links of that handy though | 22:30 |
| mordred | fungi: https://etherpad.openstack.org/p/ri90YWCIWn | 22:30 |
| *** edmondsw has quit IRC | 22:30 | |
| mordred | clarkb: /etc looks ore specific now I think | 22:30 |
| clarkb | mordred: it is in some places not others | 22:30 |
| clarkb | I'm trying to find examples | 22:30 |
| fungi | i guess i was thinking of -execdir rm -rf {} \; | 22:32 |
| fungi | or + | 22:32 |
| fungi | the latter is probably faster | 22:32 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Adding new project Tatu (SSH as a Service). https://review.openstack.org/511335 | 22:32 |
| openstackgerrit | Pino de Candia proposed openstack-infra/project-config master: Add Zuul merge-check template. https://review.openstack.org/511339 | 22:32 |
| clarkb | http://logs.openstack.org/14/508614/1/gate/gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container/4fd5466/logs/undercloud/etc/ its copying things like shadow | 22:33 |
| mordred | fungi: yah - I think we'll need to execdir | 22:33 |
| *** felipemonteiro_ has quit IRC | 22:34 | |
| *** felipemonteiro_ has joined #openstack-infra | 22:34 | |
| fungi | for some reason i thought there was a directory-specific equivalent of -delete | 22:34 |
| mordred | find -O3 /srv/static/logs/ -depth -not -name robots.txt -not -name lost+found \ | 22:34 |
| mordred | -not -wholename /srv/static/logs/help/\* \ | 22:34 |
| mordred | \( -type d -name .~tmp. -execdir rm -rf {} \; \) \) | 22:34 |
| pabelanger | clarkb: ya, next up in /etc. I'd like to see that get blocked, and back only needed files | 22:34 |
| mordred | fungi: I think that's one too many ()'s ... | 22:35 |
| clarkb | overall though it does look a lot better than before | 22:35 |
| mordred | honestly - does this one need any ()'s ? | 22:35 |
| mordred | find -O3 /srv/static/logs/ -depth -not -name robots.txt -not -name lost+found \ | 22:35 |
| mordred | -not -wholename /srv/static/logs/help/\* \ | 22:36 |
| mordred | \( -type d -name .~tmp. -execdir rm -rf {} \; \) | 22:36 |
| mordred | how's that ?? | 22:36 |
| fungi | mordred: you can skip most of what's there if you're matching on a specific name | 22:36 |
| mordred | oh - that's a realy good point | 22:36 |
| mordred | fungi: find logs -type d -name .~tmp~ -execdir rm -rf {} \; | 22:37 |
| mordred | yea? | 22:37 |
| fungi | so something like `find /srv/static/logs/ -name '.~tmp.' -execdir rm -rf {} + | 22:37 |
| fungi | ` | 22:37 |
| fungi | oh, i missed -type d | 22:37 |
| fungi | but yeah | 22:37 |
| mordred | fungi: what does the + do? | 22:37 |
| fungi | think of + like the xargs version of \; | 22:38 |
| clarkb | mordred: I think it does like xargs and lines up a bunch of them | 22:38 |
| mordred | cool | 22:38 |
| * mordred tries it out in a single directory to make sure | 22:38 | |
| fungi | i would also quote that filename | 22:38 |
| fungi | so that ~ doesn't get a surprise shell-expansion | 22:38 |
| *** priteau has joined #openstack-infra | 22:39 | |
| mordred | fungi: it's deleting things - but I'm also getting a lot of file not found error | 22:40 |
| fungi | mordred: likely because of deleting the parents of things it found | 22:40 |
| fungi | as long as the file not found errors are for things you expected it to match, then i would just ignore them | 22:41 |
| mordred | fungi: no - they're still there ... | 22:41 |
| *** tpsilva has quit IRC | 22:41 | |
| mordred | ok | 22:41 |
| fungi | oh, hrm | 22:41 |
| SamYaple | sounds like a quoting issue | 22:41 |
| fungi | mordred: what does one of the directory names look like? | 22:41 |
| fungi | wondering if you're trying to regex instead of glob | 22:41 |
| mordred | find: `logs/undercloud/tmp/ansible/lib/python2.7/site-packages/pbr/.~tmp~': No such file or directory | 22:41 |
| mordred | the find is matching them properly | 22:41 |
| jamespage | o/ | 22:42 |
| ianw | clarkb / jeblair: ok, i think i talked myself into 511260 being ok | 22:42 |
| mordred | sudo find logs -type d -name .~tmp~ | 22:42 |
| SamYaple | mordred: can you run a rm on that directly? | 22:42 |
| mordred | does the right thing | 22:42 |
| SamYaple | or does something need escaping | 22:42 |
| jamespage | seem to be hitting alot of installability issues with check jobs on reviews for charms - https://review.openstack.org/#/c/511333/ | 22:42 |
| fungi | mordred: okay, try quoting "{}" but that may mean we need to switch to \; if it still goes wierd | 22:42 |
| ianw | i'd like to update the commit msg but don't want to kill ci on it | 22:42 |
| clarkb | ianw: I think we'll likely enqueue directly to the gate so updating commit message is likely fine | 22:43 |
| jamespage | "libcurl4-gnutls-dev : Depends: libcurl3-gnutls (= 7.47.0-1ubuntu2.2) but 7.47.0-1ubuntu2.3 is to be installed" | 22:43 |
| clarkb | mordred: jeblair ^ that sound right? | 22:43 |
| clarkb | jamespage: our ubuntu mirror ended up out of sync, it was recently resynced and fixed | 22:43 |
| fungi | jamespage: should be fixed in the last hour or so. our xenial mirror was stale and had older packages than our latest images | 22:43 |
| jamespage | great - I guessed it was probably something like that - I'll get folk to submite rechecks | 22:43 |
| jamespage | cheers | 22:43 |
| clarkb | jamespage: sorry for the trouble | 22:44 |
| *** priteau has quit IRC | 22:44 | |
| pabelanger | clarkb: spacex launch in 8mins | 22:44 |
| jamespage | clarkb: no worries :) | 22:44 |
| clarkb | pabelanger: ooh | 22:45 |
| jeblair | clarkb: wfm | 22:45 |
| mordred | fungi: well - no combinations made the errors go away - but also it totally deleted the correct files | 22:45 |
| pabelanger | clarkb: I missed the launch on Monday | 22:45 |
| fungi | good thing this one's not going up from the mid-atlantic regional spaceport. we've got rain for the past hour or so | 22:47 |
| mordred | infra-root: I have started a new screen session called 'remove-tmp-files' | 22:47 |
| mordred | and i'm going to run: | 22:48 |
| mordred | find -O3 /srv/static/logs/ -type d -name .~tmp. -execdir rm -rf {} + | 22:48 |
| mordred | unless anyone screams | 22:48 |
| *** rbrndt has quit IRC | 22:48 | |
| *** rbrndt has joined #openstack-infra | 22:48 | |
| *** edmondsw has joined #openstack-infra | 22:48 | |
| *** rbrndt has quit IRC | 22:48 | |
| fungi | mordred: your example earlier had .~tmp~ | 22:49 |
| mordred | fungi: yes - thank you | 22:49 |
| mordred | find -O3 /srv/static/logs/ -type d -name '.~tmp~' -execdir rm -rf {} + | 22:49 |
| clarkb | pabelanger: this is a recycled first stage too right? | 22:49 |
| mordred | is what I'm actually going to run | 22:49 |
| clarkb | mordred: no screaming here | 22:49 |
| fungi | mordred: lgtm | 22:50 |
| mordred | k. it's running | 22:50 |
| fungi | mordred: the -O3 is probably not useful in this case (due to no nested logic) but shouldn't hurt | 22:51 |
| mordred | ok. while those run I'm going to go find food | 22:52 |
| mordred | not that this isn't exciting and all | 22:52 |
| *** jcoufal has quit IRC | 22:53 | |
| *** edmondsw has quit IRC | 22:54 | |
| *** wolverineav has quit IRC | 22:54 | |
| *** wolverineav has joined #openstack-infra | 22:55 | |
| *** wolverineav has quit IRC | 22:59 | |
| clarkb | ianw: I think you can go ahead and update the commit message on 511260 | 23:00 |
| clarkb | then we'll enqueue it once mordreds find has given us breathing room | 23:00 |
| *** priteau has joined #openstack-infra | 23:01 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files https://review.openstack.org/511260 | 23:01 |
| ianw | clarkb / mordred: ^ ok, i am convinced that is how we got into this situation | 23:02 |
| pabelanger | clarkb: yah | 23:02 |
| ianw | i think after this, we could probably remove the ~/cache/files symlink all together, since the "historical" reason for having it is, i think, just this copy | 23:02 |
| pabelanger | clarkb: not sure which mission it flew before | 23:02 |
| *** xarses has quit IRC | 23:06 | |
| *** priteau has quit IRC | 23:06 | |
| *** ldnunes has quit IRC | 23:08 | |
| *** ldnunes has joined #openstack-infra | 23:09 | |
| *** hongbin has quit IRC | 23:11 | |
| *** felipemonteiro_ has quit IRC | 23:12 | |
| *** masber has joined #openstack-infra | 23:14 | |
| *** ttx has quit IRC | 23:20 | |
| *** ttx has joined #openstack-infra | 23:20 | |
| *** caphrim007 has quit IRC | 23:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 23:28 |
| *** yamamoto has joined #openstack-infra | 23:28 | |
| clarkb | pabelanger: fyi http://logs.openstack.org/16/502316/3/check/gate-openstackci-beaker-ubuntu-trusty/ca58a5a/console.html | 23:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 23:31 |
| pabelanger | clarkb: python package broken? | 23:32 |
| pabelanger | http://logs.openstack.org/16/502316/3/check/gate-openstackci-beaker-ubuntu-trusty/ca58a5a/console.html#_2017-10-11_23_24_09_576597 | 23:32 |
| *** thorst has quit IRC | 23:33 | |
| clarkb | what is keyring for | 23:35 |
| pabelanger | I don't know, I'd have to look | 23:36 |
| clarkb | apparently os-loganalyze deps on it | 23:37 |
| clarkb | maybe? | 23:37 |
| clarkb | I wonder if that was for the swift stuff (whcih we stopped using) | 23:37 |
| tonyb | Sorry to come late to the party but is the gnutls issues fixed? http://logs.openstack.org/c7/c722a78bea5d1a75cb204cc783b2480131bd5bc4/post/static-election-publish/d11a220/console.html#_2017-10-11_01_54_31_755588 | 23:42 |
| clarkb | tonyb: yes it should be fixed at this point, but now battling the logs server is out of inodes issue | 23:43 |
| *** tosky has quit IRC | 23:43 | |
| clarkb | tonyb: our ubuntu mirror got out of sync, pabelanger resynced it and made it happy again about 2 hours ago at this point | 23:43 |
| tonyb | clarkb: Thanks. Good luck with the inode issue. | 23:43 |
| clarkb | when it rains it pours :) | 23:43 |
| tonyb | :) | 23:44 |
| pabelanger | Yah, it _should_ be fixed | 23:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use weakref for change cache https://review.openstack.org/511355 | 23:45 |
| *** mriedem has quit IRC | 23:48 | |
| *** vhosakot has quit IRC | 23:53 | |
| *** vhosakot has joined #openstack-infra | 23:53 | |
| *** ldnunes has quit IRC | 23:56 | |
| smcginnis | The logs on here look like another out of sync issue: https://review.openstack.org/#/c/511356/ | 23:57 |
| *** vhosakot has quit IRC | 23:58 | |
| pabelanger | Hmm | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
