Wednesday, 2017-10-11

openstackgerritMerged openstack-infra/project-config master: adds missing required-projects to n8g-(bagpipe|bgppvn)
openstackgerritMerged openstack-infra/project-config master: Add neutron to the project dependecy list for zuul
*** xarses has joined #openstack-infra00:00
*** xarses has quit IRC00:00
*** caphrim007 has quit IRC00:00
*** andreas_s has quit IRC00:01
*** felipemonteiro has joined #openstack-infra00:01
*** mancdaz has joined #openstack-infra00:01
*** isviridov_away has quit IRC00:01
*** vaidy has quit IRC00:01
fungiclarkb: i usually only do that on the entry for the openid00:02
*** greghaynes has quit IRC00:02
clarkbfungi: so WHERE external_id = 'https://something' ; ?00:02
*** felipemonteiro_ has joined #openstack-infra00:02
*** greghaynes has joined #openstack-infra00:02
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: zuul-cloner-shim: Use st_dev to check for filesystem
pabelangerclarkb: ya, tripleo should be able to revert and stop downloading directly from cirros.org00:02
pabelangerEmilienM: ^00:02
fungiclarkb: yes00:02
*** xarses has joined #openstack-infra00:02
fungiclarkb: but even that can probably be skipped if you can get by with just marking the unused account inactive00:03
fungii tend to reassign the openid in cases where it's possible the same person may be inadvertently switching between openids00:03
fungibut if you disable the account and leave the openid on it, then at least they still get an error trying to log in with it00:04
clarkbfungi: in this case I think it may be a single ubuntu account that had its email addr updated, would openid switching happen in that case?00:04
clarkbbut I've asked that the person involved login normally and check which account they end up on so there is a chance we can just disable the extra account00:04
fungiit's hard to tell whether that's the case00:04
fungibecause you can have multiple ubuntu sso accounts mapping to a single launchpad account00:04
clarkbthere openids are definitely different in this case so I'm guessing it depends on what happens when they attempt to login00:05
fungiif they switch back and forth on the e-mail address they provide to, i believe it goes back and forth between openids even if they both get assigned to one launchpad profile00:05
clarkbbasically which account does gerrit give them00:05
ianwclarkb: hmm, how self testing is that?00:05
clarkbianw: I don't think it is very :)00:05
clarkbianw: I think if we get it to run tests involving zuul cloner on rax it will self test00:06
*** vaidy has joined #openstack-infra00:06
*** isviridov_away has joined #openstack-infra00:06
*** felipemonteiro has quit IRC00:06
ianwi'll try doing a bit more manual due diligence on it00:06
clarkbianw: but that may require a depends on as I'm not sure we use z-c on those changes and then you'll have to reschedule until you end up on rax00:06
*** Apoorva_ has quit IRC00:11
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add unbound role
*** mat128 has quit IRC00:12
*** jascott1 has joined #openstack-infra00:15
*** yamahata has quit IRC00:18
*** dingyichen has joined #openstack-infra00:18
ianwclarkb: hmm, probably isn't helping00:18
mnaseri was thinking we should totally search the openstack codebase for "jenkins"00:19
*** tpsilva has quit IRC00:20
ianw"sudo -E -H -u jenkins tox -efunctional" seems to be a common trope00:20
*** Apoorva has joined #openstack-infra00:20
*** felipemonteiro_ has quit IRC00:21
fungiwe could reset the uid and gid of jenkins at the beginning of legacy jobs?00:22
*** xarses has quit IRC00:22
mnaser fungi i can imagine this impacting other things like not finding things in HOMEDIR?00:23
*** xinliang has quit IRC00:23
fungiif they hardcode /home/jenkins maybe00:23
fungibut ~jenkins would be /home/zuul00:23
mnaserwell i can imagine their code will start to write out things with user "zuul" then sudo into jenkins and not be able to read/write things00:24
fungiit would own the same files because it would have the same uid00:25
mordredianw, fungi: well - also - basically everytihng about that script has new zuul replacements00:25
fungibasically if we reset the uid and gid of "jenkins" at the start of legacy jobs then they have no chance to create files as the former jenkins gid/uid, so would have interchangeable ownership between jenkins and zuul00:26
fungifor all files created as either00:27
*** xarses has joined #openstack-infra00:28
*** andreas_s has joined #openstack-infra00:28
ianwfungi: i like it ... but not sure if it's one of those "curse is worse than disease" -- will it create much harder to diagnose issues with mixed up homedirs for example?00:29
*** pvaneck has quit IRC00:29
fungiunknown unknowns are always the worst00:30
fungiis it a better solution that trying to patch all projects who cargo-culted that pattern around?00:30
mordredyah - I'd kind of prefer the opportunity to just actually fix the things00:30
mordredlike, seriously - there is only one line in that needs to stick around somehow00:31
mordredand that's line 4400:31
fungii'm fine with saying that the username the job starts as was never meant as a stable interface, and if you hard-code "jenkins" (or "zuul" for that matter) into your projects then that needs fixing00:31
*** Hunner has quit IRC00:31
*** Hunner has joined #openstack-infra00:32
*** Hunner has quit IRC00:32
*** Hunner has joined #openstack-infra00:32
*** andreas_s has quit IRC00:32
mordredline 49 can be a call to the openstack-tox role in the playbook - and the job can set required-projects: osc-lib, os-client-config, python-openstacksdk - and all the logic that script does is now taken care of00:33
mordredsourcing ~stack/devstack/openrc admin admin before running tox is a thing that needs to be handled - but that can likely just be done by setting an env var and updating the tox.ini to consume it and do the source00:33
mordredwhich is mostly me saying that fixing the problem jobs may be both easier AND better long-term BUT ... it is now time for the dinners ...00:35
*** xinliang has joined #openstack-infra00:35
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor
EmilienMpabelanger: revert what?00:44
*** caphrim007 has joined #openstack-infra00:46
anupnfungi, a quick question, i figured that i can use disabled_python3_packages to make swift run with python2 but can i directly call this function from file
*** kiennt26 has joined #openstack-infra00:52
anupni see that from devstack plugin for ironic i can call function enable_python3_packages directly but wonder can the call be done directly from or i have to do it from ironic project00:53
*** kiennt26_ has joined #openstack-infra00:53
*** kiennt26_ has quit IRC00:53
*** thorst has joined #openstack-infra00:53
*** Goneri has quit IRC00:53
*** kiennt26_ has joined #openstack-infra00:54
*** thorst has quit IRC00:54
*** LindaWang has joined #openstack-infra00:54
EmilienMpabelanger: this one?00:56
*** kiennt26_ has quit IRC01:00
*** kiennt26_ has joined #openstack-infra01:01
*** kiennt26 has quit IRC01:04
*** andreas_s has joined #openstack-infra01:04
*** Apoorva_ has joined #openstack-infra01:05
ianwwhat's openstack-zuul-roles meant to hold?01:07
*** Apoorva has quit IRC01:08
*** andreas_s has quit IRC01:09
*** Apoorva_ has quit IRC01:09
*** kiennt26_ has quit IRC01:10
*** mat128 has joined #openstack-infra01:10
*** jkilpatr has quit IRC01:10
*** kiennt26 has joined #openstack-infra01:10
*** eroux has quit IRC01:11
*** cuongnv has joined #openstack-infra01:11
*** kiennt26 has quit IRC01:12
*** kiennt26 has joined #openstack-infra01:14
*** kiennt26 has quit IRC01:17
*** jaypipes has quit IRC01:18
*** xarses has quit IRC01:18
openstackgerritAnup Navare proposed openstack-infra/devstack-gate master: [Test] DNM Checking if tinyIPA builds with py3
openstackgerritMerged openstack-infra/tripleo-ci master: Add logs for config-download
openstackgerritMerged openstack-infra/tripleo-ci master: Update document url and fix a spelling error
*** kiennt26 has joined #openstack-infra01:25
fungiianw: prior discussion suggests not much if anything. we're likely to retire that after we're finished with the transition01:25
openstackgerritDuong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Craton
*** kiennt26 has quit IRC01:26
*** kiennt26 has joined #openstack-infra01:26
openstackgerritMerged openstack-infra/tripleo-ci master: Include gate jobs in the cistatus page
*** kiennt26 has quit IRC01:27
*** kiennt26 has joined #openstack-infra01:27
openstackgerritDuong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Craton
*** hemna_ has quit IRC01:31
*** s-shiono has joined #openstack-infra01:32
openstackgerritMerged openstack-infra/system-config master: Allow all zuul servers to send to statsd
*** hongbin has joined #openstack-infra01:35
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats
*** andreas_s has joined #openstack-infra01:37
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats
*** namnh has joined #openstack-infra01:40
*** andreas_s has quit IRC01:41
*** liujiong has joined #openstack-infra01:41
*** mat128 has quit IRC01:42
*** xarses has joined #openstack-infra01:50
*** psachin has joined #openstack-infra01:51
*** aeng has quit IRC01:51
*** aeng has joined #openstack-infra01:52
openstackgerritDuong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Cyborg
*** xarses has quit IRC01:57
openstackgerritDuong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Cyborg
openstackgerritKien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Remove Kuryr-kubernetes legacy jobs
*** dhinesh_ has quit IRC02:06
*** owalsh_ has joined #openstack-infra02:08
dmsimardmordred, inc0: I'm curious, was there any other discussions regarding an eventual Docker Registry implementation in Infra ? I remember we discussed docker-registry/docker-distribution (which are kind of bad), some solution from CoreOS as well as OpenShift (Atomic) registry02:11
*** owalsh has quit IRC02:11
inc0dmsimard: no we didn't move topic that much after ptg02:12
inc0openshift has it's registry implementation?02:12
dmsimardinc0: yeah, that's what we're using for the RDO private registry.02:12
dmsimardinc0: so you know that openshift is basically k8s with bells and whistles right02:12
dmsimardinc0: so there a internal docker-registry pod in openshift, it's docker-registry v2, but openshift (and openshift API) plugs a bunch of holes02:13
inc0well from my perspective I don't really care which implementation we choose02:13
dmsimardIt allows things like easy RBAC/ACL (we use oauth with github), easy image deletion (although this will get better in 3.7 which is not yet released), being able to list images remotely, being able to tag images without pulling them, etc.02:15
inc0yeah, some of these would be cool02:15
inc0especially when some of our images will have multiple tags02:15
dmsimardoh, and it provides a fancy UI to view things02:16
dmsimardatomic registry is deprecated in favor of openshift standalone registry though02:16
openstackgerritKien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Remove Kuryr-kubernetes legacy jobs
ianwfungi: ok good, thought i might have missed a memo :)02:17
dmsimardit has that easy install atomic thing if you want to try it out, although you'll end up using openshift-ansible if you find you need something a bit out of the ordinary02:17
*** thorst has joined #openstack-infra02:17
dmsimardinc0: do you remember what that core os registry thing was ?02:17
inc0well we'd still need puppet module for this stuff02:17
inc0dmsimard: I tried to look for it - their stuff is quay.io02:18
inc0but can't find opensource02:18
*** thorst has quit IRC02:18
dmsimardinc0: "try for free on-premise" ouch02:18
dmsimardyeah I don't see any things in :/02:19
*** dave-mccowan has joined #openstack-infra02:20
*** fanzhang_ has joined #openstack-infra02:20
inc0well this openshift one looks good tho02:20
*** fanzhang has quit IRC02:21
*** petems has quit IRC02:21
*** jlvillal has quit IRC02:21
*** jlvillal has joined #openstack-infra02:21
dmsimardWe've been fighting a bit with it, sent a few PRs and helped with feedback, they're pretty receptive.02:22
*** dave-mcc_ has quit IRC02:22
dmsimardWe might upgrade to 3.7 ahead of release because we need the image pruning improvements02:22
*** petems has joined #openstack-infra02:23
dmsimardWe're not "OpenStack scale" but we're still pushing all our kolla images there >8 times a day so it is a concern to avoid keeping images (and their blobs/layers) around forever to keep disk space usage under control02:24
inc0does it have features for that?02:24
dmsimardinc0: yeah:
*** liusheng has joined #openstack-infra02:25
dmsimardoh, it also natively supports openstack cinder for persistent volume claims :)02:31
*** rgerganov has quit IRC02:33
*** fanzhang_ is now known as fanzhang02:33
*** thorst has joined #openstack-infra02:33
*** rgerganov has joined #openstack-infra02:33
*** ijw has quit IRC02:34
*** thorst has quit IRC02:38
*** ramishra has joined #openstack-infra02:39
*** mat128 has joined #openstack-infra02:42
*** lamt has quit IRC02:42
*** pleia2 has quit IRC02:43
*** armaan has quit IRC02:43
*** armaan has joined #openstack-infra02:43
*** _Cyclone_ has quit IRC02:44
*** _Cyclone_ has joined #openstack-infra02:46
*** lamt has joined #openstack-infra02:46
*** lamt is now known as Guest4149502:46
*** pleia2 has joined #openstack-infra02:50
*** dave-mccowan has quit IRC02:51
*** erlon has quit IRC02:54
*** markvoelker has quit IRC02:54
*** xarses has joined #openstack-infra02:55
dmsimardianw: gj on the unbound stuff :)02:55
*** links has joined #openstack-infra02:57
ianwdmsimard: thanks for review!  i do like finding ways that ansible has already figured out the hard bits for02:59
*** caphrim007 has quit IRC03:00
dmsimardansible is crazy good :D03:00
dmsimardin-line jinja was the last thing I learned that made my head explode03:00
dmsimardianw: doh, your typo in made it through "fowarding" :p03:01
*** sree has joined #openstack-infra03:01
pabelangerianw: the idea for openstack-zuul-roles and zuul-roles, we to be a seperate projects to just hold roles, where openstack-zuul-jobs / zuul-jobs would have been just playbooks. But, in the end it results in a lot of cross repo depends-on, which made first round of review difficult.  It worked, but we quickly decided to move things into 2 repos (over 4) to simplify things03:02
dmsimardpabelanger: +103:03
*** nicolasbock has quit IRC03:06
*** sree has quit IRC03:06
ianwdmsimard: boo; i did fix it but must have somehow dropped it in migration.  we can fix when we find something wrong with it :)03:09
*** mat128 has quit IRC03:13
*** gongysh has joined #openstack-infra03:16
*** yamahata has joined #openstack-infra03:17
mordreddmsimard: is out - it's closed source03:17
mordreddmsimard: OpenShift Container Registry looks like it requires running OpenShift and/or kubernetes - which sounds a bit overkill03:20
*** armaan has quit IRC03:24
mordreddmsimard: atomic registry looks better - it at least has a non-k8s install method - but it *looks* like it still wants to be installed and run using docker rather than just being able to be installed like other things03:25
*** yamamoto has quit IRC03:25
*** armaan has joined #openstack-infra03:25
dmsimardmordred: atomic registry is deprecated03:25
mordreddmsimard: of course it is03:26
pabelangerI thought we were on board with docker-registry ubuntu package?03:26
mordreddmsimard: well - from what I can tell docker registry is still the winner in terms of not requiring a wholesale adoption of a completely different way of viewing hte world to be installed03:26
mordredpabelanger: yes. very much so03:26
*** liujiong has quit IRC03:26
pabelangerYa, once zuulv3 is healthy, I have it on my list to play with again03:27
pabelangerbased on what we talked about at PTG03:27
dmsimardAll I'm saying is that docker-registry/docker-distribution have a lot of limitations03:27
mordredpabelanger: I think this was about looking at the other thing that provide docker registries to see if they provided compelling features or other reaons they would be the right choice03:27
*** liujiong has joined #openstack-infra03:27
mordreddmsimard: sure - but what they dont' have is a requirement that we run kubernetes03:27
mordredin fact, they don't even make us run docker03:27
dmsimardLittle to no client side implementation, hard ACL/RBAC, hard image deletion03:27
pabelangerdmsimard: right, but I think our use case is pretty basic, isn't it?03:27
mordreddmsimard: yah - but easy to run - apt-get install works - we don't have to install and run an entire additoinal orchestration system with a bunch of networking overlay crap we don't need and a ton of moving parts that don't gain us anything03:28
dmsimardI'm on my phone and typing this much hurts but happy to pick up the discussion at a keyboard tomorrow or later :p03:28
mordreddmsimard: yes - much more fun conversation with a keyboard :)03:29
dmsimardmordred: our standalone registry in RDO is 3 pods with one container in each. One for registry, one for web console and one for the router (haproxy)03:30
mordredright. but that assumes a place where you can run pods03:30
dmsimardopenshift-ansible takes care of setting everything up03:30
ianwdmsimard: re jeblairs comment in ... does the integration test run on each host?03:30
*** liujiong has quit IRC03:30
dmsimardI hate to leave this discussion unfinished but I really need a keyboard :(03:31
mordreddmsimard: it's fine - it's honestly a better conversation for post-v3 week03:31
dmsimardianw: yes, jeblair is right but I was on other things all day03:31
pabelangernp, I just wanted to make sure I didn't miss something.03:32
ianwthat's ok, just going through the open reviews in the list03:32
mordreddmsimard, pabelanger: I should know better than to dive in to that particular can of worms at this time of night anyway03:32
dmsimardpabelanger: a spec where we evaluate the different options with pros and cons seems fair03:32
pabelangerdmsimard: IIRC, we have a spec up03:32
pabelanger is what I was thinking about03:41
openstackgerritMerged openstack-infra/project-config master: Add post tarball gate to kuryr-tempest-plugin
*** armaan has quit IRC03:47
*** armaan has joined #openstack-infra03:47
openstackgerritMerged openstack-infra/project-config master: Add an origin remote for tag-releases for notes
mordredianw: lgtm - any reason not to go ahead and land it?03:55
*** ganso has quit IRC03:56
ianwmordred: i think it's ok ... i can keep an eye on the zuulv3 queue to ensure it's not causing undue issues03:56
mordredianw: cool - I pulled the trigger on it03:57
mordredianw: also - do you know off the top of your head if we got the new images built with the cache dir link thing?03:57
ianwmordred: i checked and at least xenial & centos were building a few hours ago03:57
mordredkk. I'll leave that entry in the list for now03:58
ianwyep, those are fresh03:58
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Pass root_marker_dir to write-root-marker
mordredianw: thta's a pretty easy fix to one of the bugs ^^ :)04:02
mordredpabelanger: (if'n you're still around ^^)04:02
openstackgerritIan Wienand proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job
*** ykarel|afk has joined #openstack-infra04:04
*** sshnaidm|off is now known as sshnaidm04:06
sshnaidmclarkb, the images in zuulv3 is in /opt/cache, not in /opt/cache/files04:07
openstackgerritMerged openstack-infra/zuul-jobs master: Add upload-npm role
ianwis it just me, or is there no "hide ci" button on this page ->
openstackgerritMerged openstack-infra/zuul-jobs master: zuul-cloner-shim: Use st_dev to check for filesystem
mordredianw: I see a "Toggle CI" button at the bottom04:09
ianwyeah, i do on chrome, but firefox04:09
ianwnot firefox04:09
ianwsorry, looks like i messed up what i thought was a simple rebase04:09
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Use zuul_legacy_vars for propose jobs for now
*** hongbin has quit IRC04:12
*** mat128 has joined #openstack-infra04:12
openstackgerritMerged openstack-infra/project-config master: Add release-openstack-javascript job
*** yamamoto has joined #openstack-infra04:13
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add requirements to legacy-gate-placement-api-ref
openstackgerritIan Wienand proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job
ianwmordred: pretty sure names just got backwards in zuul.d/project-templates.yaml ^04:15
*** zaneb has quit IRC04:17
openstackgerritMonty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add python-openstackclient to legacy-check-osc-plugins
ianwalso, i am an idiot and just had to resize my screen04:19
mordredianw: ah - good catch on the javascript ones04:19
mordredianw: ok - that's all the patches for the outstanding job bugs I know about (and can deal with)04:20
mordredthe propose-updates one needs a complete rework - butI think the zuul_legacy_vars thing will tide us over for a day or two until we can get to it04:21
*** jtomasek has joined #openstack-infra04:21
*** khamtamtun has joined #openstack-infra04:23
*** khamtamtun has quit IRC04:25
*** claudiub|2 has joined #openstack-infra04:25
openstackgerritMerged openstack-infra/project-config master: Add configure-unbound role to base-test job
pabelangersshnaidm: clarkb: I'm not sure how that is possible, both zuulv2.5 and zuulv3 are using the same images.  And just confirmed it is /opt/cache/files04:36
openstackgerritMerged openstack-infra/project-config master: Pass root_marker_dir to write-root-marker
*** caphrim007 has joined #openstack-infra04:38
*** thorst has joined #openstack-infra04:39
*** coolsvap has joined #openstack-infra04:41
*** zaneb has joined #openstack-infra04:41
*** yamamoto has quit IRC04:43
*** thorst has quit IRC04:44
*** mat128 has quit IRC04:44
sshnaidmpabelanger, clarkb in zuulv2 it was in /opt/stack/cache/files/cirros-0.3.5-x86_64-disk.img04:44
*** liusheng has quit IRC04:51
jlvillalAnyone else noticing build failures with messages like: The repository ' xenial Release' is not signed.04:52
*** gongysh has quit IRC04:52
ianwjlvillal: no, but we don't sign them.  the repos should be setup to ignore it04:54
jlvillalAnd then at the error: The following packages have unmet dependencies:04:54
jlvillallibcurl4-gnutls-dev : Depends: libcurl3-gnutls (= 7.47.0-1ubuntu2.2) but 7.47.0-1ubuntu2.3 is to be installed04:54
chandankumarianw: and need blessing on these reviews :-)04:54
jlvillalianw, Thanks. I've seen a lot of failures in unit test jobs and the like04:54
*** markvoelker has joined #openstack-infra04:55
jlvillalAs in every Ironic job in the gate as seen here:
jlvillalHas a failure.04:56
*** armaan has quit IRC04:58
*** armaan has joined #openstack-infra04:59
ianwjlvillal: i think it might be upstream05:00
jlvillalianw, Okay. I'll check back in the morning. 22:00 here :)05:00
jlvillalianw, Thanks!05:00
*** armax has quit IRC05:01
*** gongysh has joined #openstack-infra05:07
*** sree_ has joined #openstack-infra05:11
*** gongysh has quit IRC05:11
*** sree_ is now known as Guest7213805:11
*** gouthamr has quit IRC05:12
*** armaan has quit IRC05:13
*** armaan has joined #openstack-infra05:13
clarkbsshnaidm: yes  should still be there05:17
clarkbsshnaidm: same image is used in both places05:17
sshnaidmclarkb, afaik it moved to /opt/cache/files , without "Stack"05:18
*** eumel8 has joined #openstack-infra05:18
sshnaidmclarkb, anyway, scripts failed to find it there..05:18
clarkbsshnaidm: the stack bit is a devstack-gate thing05:18
clarkbit copies from /home/zuul/cache to set that up which was broken05:18
clarkbwas fixed today05:19
sshnaidmclarkb, let's fix it again after a migration, not a big deal05:19
*** Guest2849 is now known as sdake05:20
*** sdake is now known as Guest9543005:21
*** Guest95430 is now known as sdake_fixing05:23
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Fix: legacy-sahara-cli is a "dsvm" job
*** xarses has quit IRC05:26
*** sdake_fixing has quit IRC05:29
*** sdake_fixing has joined #openstack-infra05:29
*** markvoelker has quit IRC05:30
*** gildub has quit IRC05:33
*** sdake_fixing has quit IRC05:35
*** udesale has joined #openstack-infra05:38
*** armaan has quit IRC05:40
*** armaan has joined #openstack-infra05:41
*** mat128 has joined #openstack-infra05:43
*** yamamoto has joined #openstack-infra05:43
*** cshastri has joined #openstack-infra05:49
*** yamamoto has quit IRC05:49
*** csomerville has joined #openstack-infra05:50
*** cody-somerville has quit IRC05:50
*** sshnaidm is now known as sshnaidm|afk05:52
*** dhajare has joined #openstack-infra05:54
*** yamamoto has joined #openstack-infra05:54
*** yamamoto has quit IRC05:58
*** yamamoto has joined #openstack-infra06:02
*** yamamoto has quit IRC06:04
AJaegermorning. ianw, are all zuul executors up and running?06:05
*** andreas_s has joined #openstack-infra06:11
*** liusheng has joined #openstack-infra06:14
*** mat128 has quit IRC06:14
*** martinkopec has joined #openstack-infra06:15
*** dhajare has quit IRC06:19
*** dhajare has joined #openstack-infra06:19
*** martinkopec has quit IRC06:20
*** martinkopec has joined #openstack-infra06:21
*** yamamoto has joined #openstack-infra06:21
AJaegerah, seems we make progress on jobs - so, might be fine and just a long queue...06:23
ianwAJaeger: yeah, things were restarted about 10 hours ago06:25
*** markvoelker has joined #openstack-infra06:26
*** spectr has quit IRC06:27
*** spectr has joined #openstack-infra06:28
*** vsaienk0 has joined #openstack-infra06:30
AJaegerianw: great, thanks06:31
openstackgerritNam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Trove
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: neutron-lbaas: Add neutron requirments for tox jobs
*** thorst has joined #openstack-infra06:40
*** thorst has quit IRC06:44
AJaegermorning yolanda! Please review
yolandahi AJaeger06:46
openstackgerritNam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Trove legacy jobs
AJaegerthanks! If you didn't follow the announcements, today we switch fully to Zuul v3 ;)06:46
openstackgerritRUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin
openstackgerritDuong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Freezer
yolandaAJaeger, yes, i read the email06:46
yolandareally hard work on it...06:47
*** ccamacho has joined #openstack-infra06:47
openstackgerritDuong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Freezer
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Revert "Use new post pipelines"
AJaegeryeah - many changes in flight - and making good progress06:49
*** owalsh_ is now known as owalsh06:49
openstackgerritNam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Trove
dmelladomore changes? /me sighs xD06:50
*** threestrands has quit IRC06:50
AJaegerdmellado: all those projects that want to adjust their jobs ;)06:51
openstackgerritRUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin
dmelladoAJaeger: heh, I know, just kidding06:52
dmelladoeven I need to add quite a few more gates too06:52
openstackgerritMerged openstack-infra/project-config master: Remove project pylockfile from infrastructure systems
openstackgerritMerged openstack-infra/project-config master: Use zuul_legacy_vars for propose jobs for now
openstackgerritMerged openstack-infra/project-config master: Add new repo for magnum tempest plugin
openstackgerritMerged openstack-infra/project-config master: neutron-lbaas: Add neutron requirments for tox jobs
openstackgerritRUIJIE YUAN proposed openstack-infra/project-config master: add merge-check job template for senlin-tempest-plugin
openstackgerritMerged openstack-infra/project-config master: Add python-tempestconf project
openstackgerritMerged openstack-infra/project-config master: Adding check/gate jobs to Contributor Guide
openstackgerritMerged openstack-infra/project-config master: Add configure-unbound to base job
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add requirements to legacy-gate-placement-api-ref
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove most of the oslo.messaging legacy tests
SpamapSwell lookie there it's catching up. :)06:57
AJaegerSpamapS: That's yolanda's review breakfast ;)06:59
AJaegerdmellado: go for it - hope in-repo works fine for you ;)06:59
yolandaAJaeger, yep... trying to do some reviews before starting my day07:00
dmelladoAJaeger: heh, I hope so! I'll check multinode things now so crossing fingers07:00
*** markvoelker has quit IRC07:00
dmelladoyolanda: you just stack them before going to bed and do that in the morning07:00
dmelladoso you can show smart and mighty xDDDDD07:00
* dmellado hides07:00
*** sdake has joined #openstack-infra07:01
*** sdake has quit IRC07:01
*** sdake has joined #openstack-infra07:01
openstackgerritMerged openstack-infra/tripleo-ci master: set neutron mtu to match system mtu
*** tumbarka has quit IRC07:02
*** openstackgerrit has quit IRC07:03
*** openstackgerrit has joined #openstack-infra07:03
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs
AJaegeryolanda: could you +2 again, please? - I had to resolve a merge conflict ^07:04
*** rcernin has joined #openstack-infra07:05
strigaziAJaeger: Hello, could you have a look when you can to magnum's zuulv3 migration patch?
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: grenade-dsvm-ironic: Move task 'legacy workspace dir' to first
AJaegeryolanda: and review as well, please07:06
openstackgerrityolanda.robla proposed openstack-infra/project-config master: Add merge-check for python-tempestconf
AJaegerstrigazi: looks fine - and I missed your recheck ;) Let'S wait for Zuul +1 and then merge. I just looked at .zuul.yaml and overall structure07:08
*** jpena|off is now known as jpena07:08
yolandaAJaeger, 510353 fails07:09
yolandai tried a rebase but still has -1 from Zuul07:09
yolandamay be one of the dependencies07:09
AJaegeryolanda: let'S wait a bit until the repo is properly created - thanks07:10
AJaegerwill recheck later07:10
*** mat128 has joined #openstack-infra07:12
*** pcaruana has joined #openstack-infra07:13
*** ykarel_ has joined #openstack-infra07:14
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Revert "Add configure-unbound to base job"
AJaegeryolanda, ianw, I think unbound causes problem with IPv6, let's revert for now ^07:14
openstackgerritRUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin
*** ykarel|afk has quit IRC07:16
*** ralonsoh has joined #openstack-infra07:18
*** tesseract has joined #openstack-infra07:19
*** ykarel_ is now known as ykarel07:21
*** florianf has joined #openstack-infra07:21
openstackgerritNam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Octavia-dashboard
*** hashar has joined #openstack-infra07:24
openstackgerritRUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin
*** neon_icon has joined #openstack-infra07:26
*** tinwood has quit IRC07:30
openstackgerritNam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Octavia-dashboard legacy jobs
*** tinwood has joined #openstack-infra07:30
*** armaan has quit IRC07:33
*** nikhil has quit IRC07:34
*** armaan has joined #openstack-infra07:34
*** thorre has quit IRC07:34
*** armaan has quit IRC07:35
*** yamamoto has quit IRC07:35
*** thorre has joined #openstack-infra07:35
*** armaan has joined #openstack-infra07:35
*** yamamoto has joined #openstack-infra07:36
*** yamamoto has quit IRC07:36
*** jpich has joined #openstack-infra07:39
AJaegeryolanda: can you force merge , please?07:39
AJaegerWE run into RETRY_LIMITS all the time ;(07:39
yolandaok i'll do07:40
*** thorst has joined #openstack-infra07:41
*** mat128 has quit IRC07:44
kazshAJaeger: G'day, what's next action for this change ?  Should I ask my PTL to check this ? or need to fix something (looks getting an error but not sure the cause) ?07:44
openstackgerritMerged openstack-infra/project-config master: Revert "Add configure-unbound to base job"
*** thorst has quit IRC07:45
AJaegerkazsh: which one?07:45
kazshOops sorry,  AJaeger07:46
AJaeger#status log Lots of RETRY_LIMIT errors due to unbound useage with Zuul v3, we reverted the change; recheck your changes07:46
openstackstatusAJaeger: finished logging07:46
AJaegerkazsh: you need PTL +1 - and let me comment07:47
kazshAJaeger, ok will do, thanks for your help as alwasy :)07:47
*** openstackgerrit has quit IRC07:48
*** armaan has quit IRC07:51
*** armaan has joined #openstack-infra07:51
*** openstackgerrit has joined #openstack-infra07:52
openstackgerritMerged openstack-infra/project-config master: Add new repo for manila tempest plugin
*** csomerville has quit IRC07:54
*** csomerville has joined #openstack-infra07:57
*** markvoelker has joined #openstack-infra07:57
*** gildub has joined #openstack-infra08:02
*** florianf has quit IRC08:02
*** florianf has joined #openstack-infra08:08
*** alexchadin has joined #openstack-infra08:12
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test containers update
*** florianf has quit IRC08:13
*** florianf has joined #openstack-infra08:13
*** gildub has quit IRC08:13
openstackgerritKazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project
*** yamahata has quit IRC08:19
openstackgerritHiroaki Kobayashi proposed openstack-infra/project-config master: Add publish-openstack-sphinx-docs to Blazar
*** alexchadin has quit IRC08:22
*** alexchadin has joined #openstack-infra08:22
*** e0ne has joined #openstack-infra08:23
*** yamamoto has joined #openstack-infra08:23
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove Kuryr-kubernetes legacy jobs
*** claudiub|2 has quit IRC08:26
openstackgerritRUIJIE YUAN proposed openstack-infra/project-config master: Add new repo for Senlin tempest plugin
*** pcaruana has quit IRC08:27
*** kjackal_ has joined #openstack-infra08:29
*** shardy has joined #openstack-infra08:29
*** markvoelker has quit IRC08:30
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Revert "Add javascript tarball publication job"
*** pcaruana has joined #openstack-infra08:31
*** dingyichen has quit IRC08:32
AJaegerinfra-root, I don't know what's going on with release-openstack-javascript, we have several jobs failing that it's undefined. As quick aid, I propose 511178.08:32
*** openstackgerrit has quit IRC08:33
*** openstackgerrit has joined #openstack-infra08:33
openstackgerritHiroaki Kobayashi proposed openstack-infra/project-config master: Add documentation publish jobs to Blazar
*** spectr has quit IRC08:40
AJaegerinfra-root, or is just one zuul-executor not updated? Some changes pass, others fail08:40
*** dhajare has quit IRC08:40
fanzhangHi, jekins and zuul in trove are kind of broken, are you guys working on it ?08:40
*** egonzalez has joined #openstack-infra08:41
AJaegerfanzhang: better ask trove team on #openstack-trove and review the current status. The team is really in maintenance mode, so I would not spend any work on it unless you can help them fix issues.08:41
*** ianychoi has quit IRC08:42
*** dhajare has joined #openstack-infra08:42
*** mat128 has joined #openstack-infra08:42
fanzhangAJaeger: Thanks for your apply. So it's true that trove is in maintenance mode? I saw them discussed about it some days ago, but I am not sure what the final result is. I'll try to ask trove team, thanks anyway.08:44
*** danpawlik has quit IRC08:44
*** cody-somerville has joined #openstack-infra08:44
*** cody-somerville has joined #openstack-infra08:44
*** electrofelix has joined #openstack-infra08:45
*** csomerville has quit IRC08:47
chandankumarAJaeger: how to add initial core reviewers for python-tempestconf?,members08:50
*** spectr has joined #openstack-infra08:50
*** derekh has joined #openstack-infra08:52
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Added merge-check job template for magnum-tempest-plugin
AJaegerfanzhang: there was some discussion about that, best ask the trove devs08:55
AJaegerchandankumar: an infra-root can do this later (not me)08:55
*** lucas-afk is now known as lucasagomes08:55
*** ykarel is now known as ykarel|lunch08:56
*** yamamoto has quit IRC08:58
*** alexchadin has quit IRC08:58
*** alexchadin has joined #openstack-infra08:59
AJaegerchandankumar: later...09:00
*** yamamoto has joined #openstack-infra09:01
AJaegerchandankumar: see 511178 for fix09:03
*** yamamoto has quit IRC09:06
*** ianychoi has joined #openstack-infra09:06
*** danpawlik has joined #openstack-infra09:06
alexchadinAJaeger: there is new issue with all kind of jenkins jobs:
AJaegeralexchadin: that looks like Ubuntu problem where their repo is broken ;(09:11
*** sambetts|afk is now known as sambetts09:11
*** dtantsur|afk is now known as dtantsur09:11
AJaegerinfra-root, could you check this, please? ^09:11
AJaegeralexchadin: can't help further here, hope others will later09:11
alexchadinAJaeger: :( thanks09:11
AJaegeralexchadin: I don't see this in infra jobs - could you recheck? Maybe it was temporary?09:12
alexchadinAJaeger: just rechecked - let's see what it brings09:13
*** mat128 has quit IRC09:15
*** neon_icon has quit IRC09:18
*** armaan has quit IRC09:19
*** armaan has joined #openstack-infra09:20
*** priteau has joined #openstack-infra09:20
*** tosky has joined #openstack-infra09:21
*** fanzhang has quit IRC09:22
*** fanzhang has joined #openstack-infra09:25
*** markvoelker has joined #openstack-infra09:28
*** fanzhang_ has joined #openstack-infra09:31
*** quite has quit IRC09:32
*** ralonsoh_ has joined #openstack-infra09:32
eumel8lots of jobs are affected from broken package libcurl4-gnutls-dev09:34
*** ralonsoh has quit IRC09:36
*** jaosorior has quit IRC09:36
*** s-shiono has quit IRC09:40
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Added initial jobs for python-tempestconf
*** thorst has joined #openstack-infra09:42
*** vsaienk0 has quit IRC09:42
*** vsaienk0 has joined #openstack-infra09:43
chandankumarAJaeger: i didnot got about this change 51117809:44
*** quite has joined #openstack-infra09:44
AJaegerchandankumar: 511178 should fix the issues with Zuul complaining about missing javascript job - and thus allow your change to merge...09:45
AJaegerto run tests on I mean09:45
toskyAJaeger: hi, what's going on with jessie here?
*** jaosorior has joined #openstack-infra09:47
*** thorst has quit IRC09:48
toskyand also (I think I didn't ask it yet): is there a way to test the in-repo zuul v3 jobs locally without sending the review, or does the "merge all available jobs together" make this complicated?09:49
*** hashar is now known as hasharAway09:50
AJaegertosky: regarding jessie: On our list - this sometimes files due to a test we have. It's not in production yet (unbound and IPv6) but testing to see whetehr we can enable it.09:50
AJaegerI'm not aware of a method to test locally - best to ask later here again09:51
toskysure :)09:51
toskyso about that test, is there some review that fixes it that I should watch, or is it tracked in the zuul3-issues eitherpad?09:52
AJaegerI added to the etherpad already...09:52
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Added initial jobs for python-tempestconf
*** ykarel|lunch is now known as ykarel09:54
*** udesale has quit IRC09:58
*** alexchadin has quit IRC09:58
*** alexchadin has joined #openstack-infra09:58
*** markvoelker has quit IRC10:01
AJaegerinfra-root, change reports about missing pylockfile but that was removed in . Do we need to restart zuul so that it sees the removal of a project from it's config? Also, checking out github that change is not there - might as well be out of sync git mirrors.10:03
*** LindaWang has quit IRC10:03
AJaegerinfra-root, could you check all our git instances whether those are fine and synced, please?10:03
*** sdague has joined #openstack-infra10:03
* fungi just woke up but needs to do morning things before we start the outage10:04
AJaegerChecking - the change is not there...10:05
AJaegerfungi, good morning - see above, our git mirrors are out of sync - I guess we need to fix that first ^10:05
AJaegerfungi: get your morning things done first, please10:05
fungi#status notice The CI system will be offline starting at 11:00 UTC (in just under an hour) for Zuul v3 rollout:
openstackstatusfungi: sending notice10:07
fungiwanted to get that ^ sent out as early as i could10:07
* fungi goes to wake up and find coffee10:08
*** cuongnv has quit IRC10:08
-openstackstatus- NOTICE: The CI system will be offline starting at 11:00 UTC (in just under an hour) for Zuul v3 rollout:
*** wolverineav has joined #openstack-infra10:08
*** andreas_s has quit IRC10:09
openstackstatusfungi: finished sending notice10:10
*** andreas_s has joined #openstack-infra10:10
*** dbecker has joined #openstack-infra10:12
*** pbourke has quit IRC10:12
*** cody-somerville has quit IRC10:12
*** kjackal_ has quit IRC10:13
*** pbourke has joined #openstack-infra10:14
*** mat128 has joined #openstack-infra10:14
*** andreas_s has quit IRC10:14
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server
AJaegerinfra-root, why does exist? Was yolanda 's force merge done somehow wrongly?10:15
* AJaeger needs lunch now...10:15
*** armaan has quit IRC10:19
*** rhallisey has joined #openstack-infra10:22
*** boden has joined #openstack-infra10:23
openstackgerritKien Nguyen proposed openstack-infra/project-config master: Set right owner for /home/zuul/cache files
*** LindaWang has joined #openstack-infra10:27
*** niska has quit IRC10:29
*** alexchadin has quit IRC10:31
*** gildub has joined #openstack-infra10:32
*** namnh has quit IRC10:32
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove the legacy pylint jobs (replaced)
*** Qiming has quit IRC10:32
*** niska has joined #openstack-infra10:34
ethfciHello gate-horizon-docs-ubuntu-xenial fails with  The following packages have unmet dependencies: libcurl4-gnutls-dev : Depends: libcurl3-gnutls (= 7.47.0-1ubuntu2.2) but 7.47.0-1ubuntu2.3 is to be installed10:35
*** adarazs has joined #openstack-infra10:35
ykareli saw the same ^^ here: in tripleo-common newton branch patch10:40
*** mat128 has quit IRC10:45
*** dizquierdo has joined #openstack-infra10:45
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Revert "Add javascript tarball publication job"
AJaegerfungi, I'm not sure whether we still need the above, see the etherpad ^10:47
*** jkilpatr has joined #openstack-infra10:49
*** Qiming has joined #openstack-infra10:54
fungiokay, i'm at the computer for realz now10:57
AJaegerhi fungi10:57
*** erlon has joined #openstack-infra10:57
ethfciHello Fungi10:58
fungiethfci: ykarel: my first guess is some dependency conflict between xenial main and uca, but i can't look into it right now as we're about to undergo some rather delicate and time-sensitive maintenance activity10:58
AJaegerSo, let me recap quickly: We had merged the unbound change this morning and that broke lots of jobs, so reverted (
*** markvoelker has joined #openstack-infra10:58
fungii can start checking out the suspected issue with git mirroring now if pabelanger and mordred are able to focus on the rollout10:58
AJaegerLooking at , I see the change only on a branch. yolanda force merged it10:58
yolandaAJaeger, looks like i did something wrong when merging yes...10:59
yolandaonly exists there?10:59
fungimerging something how?10:59
*** Guest72138 has quit IRC10:59
yolandaforce push10:59
fungiforce-pushed what again?10:59
yolandaAJaeger's change for reverting unbound10:59
AJaegerWe now get errors like  that report about missing pylockfile  which was removed in but does not show up on git.o.o10:59
fungiyolanda: there is a submit button in gerrit, just add yourself to the project bootstrappers group and then reload the page with the change11:00
ykarelfungi, Ok11:00
AJaegerfungi, 51115711:00
yolandafungi ah, i did manually. I added myself as project bootstrapper, but i git a manual git push --force, i guess it ended in the wrong branch11:01
*** dbecker has quit IRC11:02
fungiif this was a push --force resulting in a non-fast-forwardable state, i'll need to see if we can recover the old state from the reflog on disk in gerrit (or from backups), but odds are this has broken project-config pulling on, well, basically all our servers11:02
AJaegeranother problem: (see for revert): after merge of we get errors that a job does not exist - that is was added as part of 510237. This happens with some openstack-zuul-job changes but apparently not all. I proposed the revert11:02
fungiwe may need to cancel maintenance11:02
fungiinfra-root: heads up ^11:02
*** mrhillsman has quit IRC11:03
*** nicolasbock has joined #openstack-infra11:03
fungiyolanda: just to clear this up for me, can you explain what caused you to resort to using git push rather than the submit button in gerrit the rest of us have been using for similar circumstances?11:03
*** masuberu has joined #openstack-infra11:05
yolandafungi, i usually do it with the force push11:05
fungiokay, good news is this looks like it's probably fast-forwardable11:05
*** mrhillsman has joined #openstack-infra11:05
fungiyolanda: please don't ever, ever git push directly into a live repo in gerrit, and whatever you do only ever add --force if fast-forward pushing breaks11:05
*** ganso has joined #openstack-infra11:06
fungiwe rely on direct git push exclusively for importing outside repository state, but everything else should be driver through gerrit unless gerrit itself is entirely offline (in which case you're stuck merging locally from a shell on gerrit anyway)11:07
*** masber has quit IRC11:07
yolandaok good to know, because the times i had to force a merge when something was broken, i always did with git push11:07
AJaegerfungi, so this merged in project-config according to gerrit but is not in git: - just for double checking once you resolved this11:07
fungiin the future, when you have an absolute need to bypass the ci system, add yourself to project bootstrappers, remove any lingering -2 verify vote, add a +2 verify (and code review +2 and workflow +1 if missing) and then use the submit button to cause gerrit to merge ituse the submit button in gerrit11:08
*** andreas_s has joined #openstack-infra11:09
*** fanzhang_ has quit IRC11:13
*** fanzhang has quit IRC11:13
fungi#status log deleted the errant review/andreas_jaeger/zuulv3-unbound branch from the openstack-infra/project-config repository (formerly at commit 2e8ae4da5d422df4de0b9325bd9c54e2172f79a0)11:13
openstackstatusfungi: finished logging11:13
Shrewsfungi: i'm "awake" (note the quotes) if you need help with anything11:15
*** sshnaidm|afk is now known as sshnaidm11:15
fungithanks Shrews! it's mostly gerrit surgery at the moment. know any java? ;)11:15
fungiinfra-root: i'm going to work on piecing together the list of merged project-config changes which ended up parented to the 2e8ae4da5d422df4de0b9325bd9c54e2172f79a0 commit on the wrong branch11:15
dirk*shhh* run, if you can, run!!11:15
fungicurrent state as best i can piece together is that a gerrit change targeted for master was directly pushed into a new branch, gerrit thought it was merged at that point and allowed other changes to merge but they're on some detached head and not actually master11:17
*** shardy is now known as shardy_lunch11:19
fungi is the last change to actually merge to master as far as the replicas are concerned11:21
fungiand `git show master` on the filesystem on gerrit confirms11:21
*** udesale has joined #openstack-infra11:21
fungiso at least this doesn't appear to have broken replication, we're just unable to merge new changes to master in this state (instead they're merging into a black hole)11:22
AJaegerbut changes after 510769 are missing11:22
*** lucasagomes is now known as lucas-hungry11:23
AJaegerSo, 510769 merged, 510728 merged, 511157 merged, and then 507905 - according to git11:24
AJaegeraccording to
fungiyolanda: roughly what time did you push the revert into gerrit?11:24
fungilooking now, i think what actually happened is that the branch got rewound and then this change merged on an earlier state11:25
*** larainema has quit IRC11:25
fungiin which case we _may_ simply be able to repropose all the previously merged changes and merge them normally11:25
AJaegerfungi: around 7:40 UTC (will do from here)11:25
AJaegerfungi: around 7:40 UTC (she said "will do" at that time)11:26
yolandayep, that time, when  AJaeger asked11:26
yolandai did at the same moment11:26
AJaegergerrit records 7:51 UTC11:26
Shrewsfungi: could we cherry-pick the missing changes fromt he detached head onto master?11:26
fungiShrews: they'll end up with a different commit id11:27
fungipretty sure that'll confuse gerrit11:27
fungiokay, so here's my revised theory now having looked at the situation:11:28
fungichanges merged up through 509937 at 06:56z11:29
*** kjackal_ has joined #openstack-infra11:29
*** markvoelker has quit IRC11:30
*** boden has quit IRC11:30
*** ldnunes has joined #openstack-infra11:34
fungithen yolanda checked out 511157 but left it parented on an earlier master branch state from where 511075 had merged and pushed that at 07:40z, rolling the branch back by 14(?) changes and various associated merge commits11:34
fungiafter that, gerrit happily merged 507905 on top of the new branch state11:35
*** alexchadin has joined #openstack-infra11:35
AJaegeryes, 511075 merged which caused lots of RETRY_LIMIT errors and I used gerrit to propose a revert which is 51115711:35
fungiso at this point, we have two choices:11:35
fungi(that i can think of)11:36
fungi1. reset the master branch state to the last change that merged before this incident and then repropose the revert and the change which merged after it and merge them again11:36
*** witek has joined #openstack-infra11:37
fungithis will be a non-fast-forwardable change from the current branch state, but it will again become fast-forwardable from the prior branch state (which is what most of our systems are probably still stuck at)11:37
fungi2. leave the master branch as-is and repropose all the (14?) missing changes11:38
fungithis will be fast-forwardable from the _current_ branch state, but i think leaves our systems which had an earlier branch state from sometime between 06:56 and 07:40 stuck unable to update without manual assistance11:38
fungii'm leaning toward option #1 on the odds that more of our systems are likely stuck unable to fast-forward right now11:39
*** jpena is now known as jpena|lunch11:42
* AJaeger joins fungi in favoring #111:42
fungifor #1, i need to work out what the commit id of the merge commit that stitched 509937 into master was, which i think i should be able to find from zuul's debug log since it will have seen the merge commit and enqueued post jobs11:43
*** spectr has quit IRC11:44
fungior, actually, gerrit's on disk reflog probably has it. i'll check that first11:44
AJaegerfungi, I'll be offline for the next hour and cannot give you further encouragement. Thanks a lot for fixing this!11:45
fungithanks for the detailed record AJaeger!11:45
*** spectr has joined #openstack-infra11:46
funginope, the reflog seems to omit the same commits as the git log. i'll go hunting in zuul's debug logs11:48
AJaegerfungi: once that is fixed, please recheck one of the changes that I collected in the etherpad for the issues "elease-openstack-javascript is undefined" - see line 73. I have a revert up if needed but I smell some connection between these issues, so let's handle these one by one...11:48
AJaegerfungi, ignore the above for now - and bookmark for later, please ;)11:48
*** mat128 has joined #openstack-infra11:50
fungizuul to the rescue! the last "sane" branch state was the merge commit at 55da0b6986b546f4a80e043b4a80561c0a4bc79711:52
fungiMerge "Adding check/gate jobs to Contributor Guide"11:52
fungithat has 53e230969e179e3c79a6a4cf8dc3b21b166614b6 (change 509937) as a parent11:53
*** spectr-RH has joined #openstack-infra11:54
fungii'm going to resort to git push --force to reset the project-config master branch state from 5a7168229f0f73d6b2d5a3e8859d0a8974462792 (Merge "Add new repo for manila tempest plugin") to 55da0b6986b546f4a80e043b4a80561c0a4bc797 (Merge "Adding check/gate jobs to Contributor Guide")11:54
*** stakeda has quit IRC11:55
*** claudiub|2 has joined #openstack-infra11:55
fungithis will be a non-fast-forwardable change from the current branch state, but should be fast-forwardable from any time prior to 07:40z11:55
*** tpsilva has joined #openstack-infra11:55
*** yamamoto has joined #openstack-infra11:55
*** gildub has quit IRC11:56
fungiit _will_ completely orphan 511157 (Revert "Add configure-unbound to base job") and 507905 (Add new repo for manila tempest plugin) which i will repropose shortly parented on the corrected master branch11:56
fungi(using new gerrit change ids obviously)11:56
*** eharney has joined #openstack-infra11:57
fungii'm also making a complete cp -ax of the current ~gerrit2/review_site/git/openstack-infra/project-config.git into ~fungi/project-config.git first11:57
*** spectr has quit IRC11:57
*** jcoufal has joined #openstack-infra11:58
sshnaidminfra-root, we have a problem with setup workspace in tripleo jobs: rm: cannot remove ‘/home/zuul/cache/files/cirros-0.3.5-x86_64-disk.vhd.tgz’: Permission denied11:58
*** yamamoto has quit IRC11:59
fungisshnaidm: thanks for the heads up, can you see if it's already on and add it if not? we haven't started the maintenance yet anyway, another emergency cropped up i'm trying to deal with first12:00
sshnaidmfungi, sure12:00
*** dave-mccowan has joined #openstack-infra12:00
*** larainema has joined #openstack-infra12:00
*** lucas-hungry is now known as lucasagomes12:01
fungiug, gerrit won't let me pull 55da0b6986b546f4a80e043b4a80561c0a4bc797 unless it belongs to a current branch or tag i think. i'll make a temporary branch from it12:02
*** andreas_s has quit IRC12:02
*** andreas_s has joined #openstack-infra12:02
*** Hal has joined #openstack-infra12:02
*** dprince has joined #openstack-infra12:03
*** Hal is now known as Guest6153112:03
*** edmondsw has joined #openstack-infra12:04
*** thorst has joined #openstack-infra12:06
fungii have created temporary branch "sane-master" from 55da0b6986b546f4a80e043b4a80561c0a4bc797, fetched that, then reset --hard my local master branch to it12:06
mnasergood morning (and good luck!) infra :)12:07
openstackgerritMerged openstack-infra/project-config master: neutron-lbaas: Add neutron requirments for tox jobs
*** yamamoto has joined #openstack-infra12:08
openstackgerritMerged openstack-infra/project-config master: Use zuul_legacy_vars for propose jobs for now
openstackgerritMerged openstack-infra/project-config master: Pass root_marker_dir to write-root-marker
openstackgerritMerged openstack-infra/project-config master: Add an origin remote for tag-releases for notes
openstackgerritMerged openstack-infra/project-config master: Add release-openstack-javascript job
openstackgerritMerged openstack-infra/project-config master: Adding check/gate jobs to Contributor Guide
openstackgerritMerged openstack-infra/project-config master: Add post tarball gate to kuryr-tempest-plugin
openstackgerritMerged openstack-infra/project-config master: adds missing required-projects to n8g-(bagpipe|bgppvn)
openstackgerritMerged openstack-infra/project-config master: Remove project pylockfile from infrastructure systems
openstackgerritMerged openstack-infra/project-config master: Remove the legacy oslo.messaging project jobs
openstackgerritMerged openstack-infra/project-config master: Add neutron to the project dependecy list for zuul
openstackgerritMerged openstack-infra/project-config master: Add new repo for magnum tempest plugin
openstackgerritMerged openstack-infra/project-config master: Add python-tempestconf project
fungiapologies for the spam, but that was apparently the confirmation from gerritbot following my execution of `git push --force gerrit master`12:08
fungithose are as best as i can tell the commits which had previously been unwound12:09
funginow i'll get to work on reproposing 511157 and 50790512:09
*** liusheng has quit IRC12:10
*** liusheng has joined #openstack-infra12:10
*** shardy_lunch is now known as shardy12:11
*** andreas_s has quit IRC12:11
*** spectr-RH has quit IRC12:13
*** spectr has joined #openstack-infra12:13
fungiokay, so looking at the history, it looks like 511075 (Add configure-unbound to base job) merged immediately after what is now the new (restored) master branch state so i'll find its merge commit (if any) and wind forward to that with push (no --force should be needed now)12:14
* fungi returns to the zuul debug log to check for merge commits12:14
*** spectr-RH has joined #openstack-infra12:14
fungiyeah, there definitely would have been a separate merge commit for that one, owing to the missing commits which followed its direct parent12:15
*** panda|off is now known as panda12:15
*** Guest61531 has quit IRC12:15
*** liusheng has quit IRC12:15
* Shrews watches fungi exercise his git-foo and is impressed12:16
*** andreas_s has joined #openstack-infra12:16
*** mat128_ has joined #openstack-infra12:17
*** spectr has quit IRC12:18
*** openstackgerrit has quit IRC12:18
*** markvoelker has joined #openstack-infra12:18
*** mat128 has quit IRC12:19
fungicourtesy of zuul logs, 8685183d1c48afbb5e095d6c727c9acc199353f1 is the merge commit i want there. recreatnig my temporary "sane-master" from that now12:19
*** andreas_s has quit IRC12:21
fungii have checked out 8685183d1c48afbb5e095d6c727c9acc199353f1 and reset --hard my local master branch to that12:21
fungier, s/checked out/fetched/12:21
*** andreas_s has joined #openstack-infra12:21
*** openstackgerrit has joined #openstack-infra12:23
openstackgerritMerged openstack-infra/project-config master: Add configure-unbound to base job
fungiand there's the other missing commit ^12:24
funginow to repropose its orphaned revert12:24
openstackgerritJeremy Stanley proposed openstack-infra/project-config master: Revert "Add configure-unbound to base job"
fungithat's ^ the reproposal of 511157 with a new change-id12:25
*** andreas_s has quit IRC12:25
*** andreas_s has joined #openstack-infra12:26
openstackgerritJeremy Stanley proposed openstack-infra/project-config master: Add new repo for manila tempest plugin
fungiand that's ^ the reproposal of 507905 with a new change-id12:27
fungii'll reapprove those now12:27
mnaserfungi maybe you've thought about this, but will puppet runs for zuul reconfig likely have a problem updating projet-config locally now (if they ran once inbetween?)12:27
mnaserassuming the servers consuming project-config do a git pull12:28
*** rlandy has joined #openstack-infra12:28
fungithe expectation is that they were already in that state. this surgery is putting the branch back into a fast-forwardable state from its earlier state12:29
mnasergit magic12:29
funginot really magic. more like a really big hammer12:29
*** trown|outtypewww is now known as trown12:29
fungithis is really less like surgery and more like a butcher shop12:30
mnasertakes still to use a big hammer without breaking things :p12:30
*** andreas_s has quit IRC12:30
*** priteau has quit IRC12:30
mnaseralso, configure-unbound, i think `ansible_default_ipv6` is undefined on machines that dont have ipv6, so instead of "when: ansible_default_ipv6.address is defined" we could probably do "ansible_default_ipv6.address"12:30
mnaser"when ansible_default_ipv6 is defined"12:31
*** andreas_s has joined #openstack-infra12:31
*** yamamoto has quit IRC12:31
mnaseri'll pick that up and make a patch to add it to base-test to test out (unless someone doesnt pick it up), but coffee time12:31
openstackgerritMerged openstack-infra/project-config master: Revert "Add configure-unbound to base job"
fungimanually submitted via gerrit ^ since the parent it's reverting breaks testing it12:32
persiaShrews: Not that it matters much, but the usual transliteration is "fu" or "fū"12:32
*** sshnaidm is now known as sshnaidm|mtg12:32
persiafungi: For those of us following along at home, was there a push before the bot-reported merge of 511075?12:33
fungipersia: yep, when gerritbot reported merging 511075 that was the result of me running `git push gerrit master` (no --force) with my local master reset to 8685183d1c48afbb5e095d6c727c9acc199353f112:35
*** jpena|lunch is now known as jpena12:35
*** wolverineav has quit IRC12:35
fungiit was basically fast-forwarding the branch one change, since that change had actually merged normally previously after the state i had initially repaired master to12:35
fungiin short, i miscalculated the last sane branch state by one change (and its corresponding merge commit) so wound it forward to that one12:36
fungifinally, now i have single-core approved 511216 in hopes that it merges normally12:36
fungii'm also deleting my temporary "sane-master" branch in gerrit again now12:37
persiafungi: Thanks for the confirmation.  That's how I thought it worked, but you hadn't reported the specific push command (and had reported the first push command), so I wondered if some other magic had applied.12:38
fungipersia: nope, you're absolutely right thanks for reminding me i had failed to report one step there12:38
*** thorst_ has joined #openstack-infra12:38
*** esberglu has quit IRC12:38
*** thorst has quit IRC12:39
*** andreas_s has quit IRC12:40
persiaDoes project-config get special priority currently for zuulv3 reasons, or do we expect to wait ~4 hours for 511216 to safely merge?12:41
fungithere is a separate infra-check pipeline for project-config which has highest priority in zuulv3's config for now12:42
persiaDoes that help the merge?  I thought it was still zuulv2.5 that was doing the actual merges (to make sure zuul+gerrit *can* merge 511216 safely atop the now sane master)12:43
persiaOr does just performing a verify test show that the history rewrite was successful?12:43
fungizuulv3 has authority over a (very few) infra config repos at the moment12:43
*** Hal has joined #openstack-infra12:44
fungiunlike for basically all other projects12:44
*** kiennt26_ has joined #openstack-infra12:44
*** Hal is now known as Guest4029212:44
persiaAh, good.12:44
fungizuulv3 just now noticed 511215 merged and has requeued (previously failing) check jobs for 511216, so fingers crossed12:44
fungioh, it's already moved into infra-gate in fact12:44
fungieta 4 minutes to normal merge12:45
*** kiennt26 has quit IRC12:45
*** kiennt26_ has quit IRC12:45
*** andreas_s has joined #openstack-infra12:45
*** kiennt26 has joined #openstack-infra12:46
*** alexchadin has quit IRC12:46
*** jcoufal has quit IRC12:47
*** eumel8 has quit IRC12:47
*** jcoufal has joined #openstack-infra12:48
openstackgerritMerged openstack-infra/project-config master: Add new repo for manila tempest plugin
fungiinfra-root: ^ looks like project-config is working normally again now with no missing changes (discounting the orphaned two i had to repropose into gerrit). it's probably safe to proceed with the zuulv3 rollout maintenance now, that is if we still think it's not too late to begin at this point12:49
fungipabelanger: mordred: you said you'd be awake to help... around yet?12:50
AJaegerthanks, fungi. I'm back online...12:50
* cmurphy hi-fives fungi12:54
fungithanks cmurphy!12:54
AJaegerfungi, I rechecked some of the changes with the issue I mentioned - let's see whether it works now12:54
fungiAJaeger: appreciated12:54
*** andreas_s has quit IRC12:55
fungiAJaeger: note that 511216 went through the normal ci process (fast-reviewed, approved, passed check, passed gate, zuul merged it)12:55
fungiso in theory we should be free of any widespread issue with project-config at this point12:55
*** alexchadin has joined #openstack-infra12:55
*** ykarel has quit IRC12:56
fungibut always good to double-check ;)12:56
AJaegerfungi: I hope so as well...12:56
*** ykarel has joined #openstack-infra12:56
*** esberglu has joined #openstack-infra12:57
*** dizquierdo has quit IRC12:58
AJaegerfungi, yes looks good - zuul has started execution, so the errors are gone. I'll remove the content from etherpad and abandon the job.12:59
AJaegerWe're green again...12:59
fungithanks for confirming, AJaeger!12:59
*** andreas_s has joined #openstack-infra12:59
fungii feel pretty confident we can push forward with the rollout finally, though we're two hours behind scheduled start time at this point and still missing pabelanger and mordred it seems13:00
fungiso we may need to do a mass batch-enqueue of changes from zuul v2 into v3 at this point13:00
*** hasharAway is now known as hashar13:01
fungialso the rollout steps we have in don't seem to be updated with the new change urls?13:02
fungidid a new version of the rollout plan get documented elsewhere?13:02
*** openstackgerrit has quit IRC13:03
Shrewsfungi: i was looking for one a few moments ago and did not find one. not sure if there is an updated one13:03
fungi#status notice Due to unrelated emergencies, the Zuul v3 rollout has not started yet; stay tuned for further updates13:04
openstackstatusfungi: sending notice13:04
mordredfungi: morning!13:05
fungia wayward mordred arrives in a puff of smoke13:05
-openstackstatus- NOTICE: Due to unrelated emergencies, the Zuul v3 rollout has not started yet; stay tuned for further updates13:05
mordredfungi: I have poorly timezoned - sorry - I lodged it in my brain for some stupid reason as 12:00 local time - which is clearly stupid13:06
*** Guest40292 has quit IRC13:06
mordredfungi: what's the tl;dr on the scrollback - or should I just read it?13:06
fungiheh, no problem. you got here just in time to miss all the project-config repair excitement13:06
openstackstatusfungi: finished sending notice13:07
mordreddid I break project-config? sounds lke something I'd be the cause of13:07
*** bobh has joined #openstack-infra13:07
fungimordred: summary is that there was an unfortunate git push --force into project-config around 07:40 in a misguided attempt to bypass ci for reverting a deadlock on a job config13:07
mordredah, nod13:08
fungitook ~2 hours to repair the repo into a sane state fast-forwardable from its earlier commits13:08
mordredthe old 'everytime we bypass CI we break something' trope13:08
fungionce i woke up and got coffee13:08
mordredwe're like a walking example of why our CI system exists :)13:08
fungiyeah, in this case it was less the bypassing ci merging broken content, and rather unsafe process/method followed which broke the repository itself13:09
fungigranted there was a chain of incorrect process13:11
*** sdake has quit IRC13:11
*** sdake has joined #openstack-infra13:11
*** sdake has quit IRC13:11
*** sdake has joined #openstack-infra13:11
fungithe change which initially deadlocked our zuulv3 configuration was due to approving an untested addition to the base job13:11
fungiwhich we already know to be dangerous13:11
fungibut then the need to bypass ci when reverting that was met with a git push --force of an incorrect and non-fast-forwardable repo state, rather than using the submit button in gerrit13:12
*** openstackgerrit has joined #openstack-infra13:12
openstackgerritMerged openstack-infra/project-config master: networking-bgpvpn: add translation jobs
openstackgerritMerged openstack-infra/project-config master: Remove legacy watcher jobs
*** kgiusti has joined #openstack-infra13:13
*** mriedem has joined #openstack-infra13:13
*** mat128_ has quit IRC13:14
mordredfungi: submit button always better ... is there anything I can do to help anything at the moment or anything you'd like meto look at?13:15
*** dbecker has joined #openstack-infra13:15
*** sdake has quit IRC13:15
funginope, i think we're back on track, i was just looking for the updated rollout plan with the correct change urls13:15
fungiand waiting for you and pabelanger to appear13:15
fungianyway, i'm beginning to worry that we lack the bandwidth to provide sufficient process documentation and mentoring to our sysadmins in apac and emea timezones, so they're not as aware of safe procedures the rest of us have established and end up having to guess (sometimes poorly) how to handle such situations13:17
fungiwhich is something we'll all need to work on/help fix13:17
mordredyah. I agree13:18
mordredfungi: I've only got the old etherpad plan ...13:18
fungimordred: yeah, that's all i found as well. pabelanger proposed reverted reverts and related patches for those, right?13:19
*** sdake has joined #openstack-infra13:19
fungii think i remember hmi saying he put them under a consistent review topic. checking13:19
mordredfungi: ^^13:19
*** sdake has quit IRC13:19
*** sdake has joined #openstack-infra13:19
* mordred willupdate etherpad real quick13:19
*** zhurong has joined #openstack-infra13:20
mordredfungi: actually - I made a new one-
*** baoli has joined #openstack-infra13:22
fungieven better. i've added that url to the old pad for those who come later13:22
*** gouthamr has joined #openstack-infra13:23
*** markmcd has quit IRC13:23
*** mpaolino has joined #openstack-infra13:23
mordredfungi: should we prepare revert reverts?13:25
*** markmcd has joined #openstack-infra13:26
mordredI guess they're easy enough to make if we need them13:26
*** wolverineav has joined #openstack-infra13:27
fungiyeah, if someone else wants to work on proposing those while maintenance is underway, that's pretty low-hanging fruit too13:28
*** Guest41495 has quit IRC13:29
mordredfungi: well - it's not that many steps to do - we could likely get the ball rolling without pabelanger13:31
EmilienMfungi: can we merge a patch in openstack/release? or should we wait?13:31
*** lamt has joined #openstack-infra13:31
fungiEmilienM: we haven't started yet, so should be fine to go ahead13:32
*** vhosakot has joined #openstack-infra13:33
*** jaypipes has joined #openstack-infra13:35
*** priteau has joined #openstack-infra13:35
*** sree has joined #openstack-infra13:36
*** tosky_ has joined #openstack-infra13:36
mordredAJaeger: did you do anything to fix the emacs missing on the debian-jessie integration test yet?13:37
mordredfungi: if you didn't see yet:
*** tosky has quit IRC13:38
mordredfungi: I think we need to consider that a blocker to rollout - since it'll block literally any patch to ozj ... can't decide if we should just disable the jessie test for a second or try to figure out what's up with emacs13:38
*** jistr is now known as jistr|mtg13:38
*** pblaho has quit IRC13:38
fungimordred: i _think_ that's what the untested addition to the base job was attempting to address13:39
mordredfungi: do you have a link happy to the untested addition?13:41
fungiso maybe worth spending a couple minutes revisiting what ianw had figured out and seeing if we can iterate on it more safely13:41
AJaegermordred: a recheck healed it ;)13:41
*** pblaho has joined #openstack-infra13:41
mordredoh it did?13:41
mordredAJaeger: it looks still broken13:42
AJaegermordred: indeed - too much going on ;(13:42
fungimordred: ianw's attempt to fix is
AJaegermordred: worked and was run later13:43
*** udesale has quit IRC13:43
mordredfungi, AJaeger: I think I'd like to sugges that we remove the jessie multinode integration test - and then we can iterate on fixing the problem with a patch re-adding the job13:44
mordredbecause iterating on that job, especially if we need fixes to the base job, could take all day13:45
AJaegermordred: go for it...13:46
*** mat128 has joined #openstack-infra13:46
*** srobert has joined #openstack-infra13:46
fungimordred: we have an updated failure example from after i fixed up the project-config repo?13:46
fungijust want to be absolutely certain the issue still remains before we disable what may be a now working job13:47
mordredfungi: AJaeger just rechecked the patch13:48
* AJaeger rechecked it, not run yet13:48
*** mat128 has quit IRC13:48
fungiif it fails, then i'm +3 on temporarily disabling that job13:48
AJaegerwait, multinode-itegration passed13:48
fungiso no need to disable anything there?13:49
*** mat128 has joined #openstack-infra13:49
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci master: Enable repo for python-os-testr
AJaeger passed as well13:49
*** armax has joined #openstack-infra13:49
AJaegerfungi: I would say: Not. But we need to keep a close eye on these...13:53
*** hashar is now known as hasharAway13:55
*** udesale has joined #openstack-infra13:55
*** links has quit IRC13:56
*** eumel8 has joined #openstack-infra13:57
openstackgerritStephen Finucane proposed openstack-infra/storyboard-webclient master: Don't show underline when hovering on task titles
openstackgerritStephen Finucane proposed openstack-infra/storyboard-webclient master: Use cursor when hovering over a read-only card
*** alexchadin has quit IRC13:58
AJaegerargh, faileld here -
AJaegermordred: ^13:59
fungiso it's intermittent?13:59
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add documentation on force-merging a change
fungithanks jeblair!14:00
jeblairgood morning!14:00
*** jistr|mtg is now known as jistr14:00
AJaeger" Cannot initiate the connection to (2001:a78:5:0:216:35ff:fe7f:be4f). - connect (101: Network is unreachable)"14:00
AJaegerfungi: random ;(14:00
AJaegermorning, jeblair14:01
pabelangerfungi: sorry, I am running late this morning14:01
pabelangerprepared now14:01
pabelangerhad some issues with internets14:01
AJaegerjeblair: let me fix a typo in it quickly...14:01
openstackgerritAndreas Scheuring proposed openstack/diskimage-builder master: Fix rendering issues in DIB "building_an_image" doc
fungiAJaeger: interesting... according to this it ran in ovh-gra1 and failed reaching a v6 global address even though it has no global v6 routes:
openstackgerritAndreas Jaeger proposed openstack-infra/system-config master: Add documentation on force-merging a change
fungipabelanger: no worries, this morning has been a bust all 'round14:02
*** gmann is now known as gmann_afk14:03
pabelangerfungi: :) Where should I be focusing?14:03
AJaegerfungi: so, that means we have problems with the test unbound change - that we only run in openstack-zuul-jobs? We could revert that as well...14:03
*** spectr-RH has quit IRC14:04
pabelangerI've removed my WIP from rolllout patches14:04
jeblairAJaeger: i thought unbound was completely reverted14:04
AJaegerfungi, mordred is still in14:04
AJaegerjeblair: ^14:04
*** spectr has joined #openstack-infra14:04
*** chlong has joined #openstack-infra14:04
jeblairAJaeger: yes, but nothing should use base-test14:05
fungiAJaeger: i'm having trouble finding the error you quoted in
AJaegerjeblair: indeed, now I'm confused14:06
jeblairAJaeger: those jobs inherit from base-minimal, but that shouldn't have unbound14:07
AJaegerfungi, - and then go to multinode.yaml page 2 and see the failure14:07
AJaeger"configure-mirrors : Update apt cache" is the failure that gave the message14:07
pabelangerya, error doesn't make it way into job-output.txt14:08
pabelangerbut ARA will show it14:08
fungiAJaeger: thanks, i can see in ara that playbook(?) failed but i'm still not familiar with navigating it to get to logged output14:08
*** slaweq_ has quit IRC14:08
AJaegerfungi, neither am I - I just learned that ara helps me locate them quickly ;)14:08
fungithere are no records for the playbook, apparently14:08
fungiahh, under "tasks"14:09
pabelangerisn't the issue that debian-jessie isn't using our AFS mirrors?14:09
pabelangerI thought I added that to etherpad last time around14:09
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add python-openstackclient to legacy-check-osc-plugins
jeblairyeah, then click the red failed button on the task14:09
*** tosky_ is now known as tosky14:10
fungiokay, so there's a very good chance that apt is falling back to attempting ipv6 after failures to reach over ipv4, even though it doesn't have any ipv6 route to be able to reach it (and then only the latter error bubbles up to stdout)14:11
AJaegermordred: reviewed the etherpad, looks good from change perspective14:12
jeblairroot@ubuntu-xenial-ovh-gra1-11333378:~# ping6 security.debian.org14:12
jeblairconnect: Network is unreachable14:12
mordredjeblair: can it ping6 other v6 things and just not
fungiwe don't have ipv6 routes14:13
fungifrom there14:13
*** eharney has quit IRC14:13
jeblairmordred: nope14:13
pabelangerthat is wrong14:13
pabelangerit should be: deb jessie-security main for example14:13
pabelangerso configure-mirror is not correct14:13
pabelangerlooking now14:14
fungiare you able to connect to port 80 on the v4 address for from ovh-gra1?14:14
*** rajinir-afk is now known as rajinir14:14
jeblairfungi: using nc yes14:14
fungiokay, so it's not entirely broken14:15
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Also add security.list for Debian on configure-mirror
fungiso either apt is only trying v6 (and failing because we have no v6 routes in ovh-gra1) or it's getting intermittent access failures over v4 and then falling back to trying v6 even though there's no point14:17
pabelangerjeblair: fungi: ^ should help with security updates14:17
mordredjeblair, fungi, pabelanger: I've added the issue with the apt error not making it into the log to the etherpad so I don't lose it14:17
fungithanks mordred14:17
pabelangermordred: 511253 should be our fix14:17
*** rbrndt has joined #openstack-infra14:19
fungithanks pabelanger!14:19
*** martinkopec has quit IRC14:20
AJaegerSo, ready to go forward with Zuul v3?14:20
fungii'm still mildly weirded out by the errors about failing to reach security.d.o and not sure why it's trying over v6 from a server with no v6 global routes14:21
fungibut willing to let that go for now14:21
kiennt26hi, sorry for interrupted. i had faced a gate issue related to /home/zuul/cache/files/ permissions :
jeblairfungi: me too, but it does appear that the fix matches the v2 behavior, so i guess best to leave that rabbit hole for later.  :)14:22
mordredwell - I'd like to see the ozj patch pass after thatconfigure mirrors patch before we do -cause if we can't land ozj patches that'sbad14:22
dmsimardAJaeger: you can also just search for "failed" in the search box of the task panel14:23
jeblairmordred: this may only happen if the debian job lands on ovh-gra114:24
AJaegerdmsimard: Would be better to do that at top-level ...14:24
AJaegerdmsimard: normally there's one page and no need to skip to second - but for those it's handy, thanks!14:24
mordredjeblair: oh. bother. k. let's just assume it's good then14:24
dmsimardAJaeger: It's a frontend/UX issue, not a backend one. The data is there, I'm just bad at frontend :p14:25
mordredjeblair, fungi, pabelanger: kiennt26's issue is on the etherpad - it's from 03:26:10.949 though - so might have been the old images without the /home/zuul/cache yeah?14:25
jeblairmordred: good question -- do we have new images?14:25
dmsimardAJaeger: there's a lot of different things we can eventually do with ara -- for example (pseudocode) "ara task list --all |grep failed |xargs |echo 'http://direct link to failure'"14:25
pabelangermordred: yah, new images were built yesterday14:26
mordredjeblair: the images built last night14:26
pabelanger~14 hrs ago14:26
fungimordred: jeblair: i thought ianw built new images, but maybe they got rolled back? or maybe they went into use after 03:26z14:26
jeblairnodepool says it's uploading images to some providers14:26
mordredso it's still potentially working up updating the images14:27
mordredoh - wait14:27
pabelangeryah, nb03 is slow compared to nb0414:27
mordredthat is DeFINITELY using the new images14:27
mordredfind /home/zuul/cache/files/ -mindepth 1 -maxdepth 1 -exec mv '{}' is the command it's running14:28
mordredfollowed by mv: cannot move '/home/zuul/cache/files/cirros-0.3.4-x86_64-disk.vhd.tgz' to '/opt/stack/cache/files/cirros-0.3.4-x86_64-disk.vhd.tgz': Permission denied14:28
mordredso is it possible we didn't get ownership set correctly for the cache files?14:28
fungiis it attempting to perform the mv as stack or something instead of zuul?14:28
mordredor do we need to add a pre-playbook task to chown things?14:29
*** zhurong has quit IRC14:29
mordred2017-10-11 03:26:10.95014:29
mordred(we should gzip that file, btw, so it'll get time links)14:29
*** coolsvap has quit IRC14:31
fungiahh, judging from that log the files must be owned by not-zuul14:31
pabelangerlrwxrwxrwx 1 root root   16 Oct 11 00:16 files -> /opt/cache/files14:31
pabelangerthat is on
pabelangerin /home/zuul14:32
fungipabelanger: yeah, but who owns the files in there?14:32
mordredjenkins, I believe14:32
pabelangerjenkins because we still don't chown them14:32
mordredbecause we just made a symlink right?14:32
fungiit's not trying to move the directory/symlink, just trying to move the files out of it14:32
pabelanger2017-10-11 00:16:31,926 INFO + chown zuul:zuul /home/zuul/cache14:33
pabelangeris all we do14:33
pabelangerthen symlink14:33
fungii  bet that was supposed to have a -R14:33
pabelangerwhich, jenkins owns14:33
jeblairwait i'm really confused14:33
jeblairdevstack-gate only does that if ~/cache/files exists14:33
jeblairso adding that to the images *broke* this14:33
jeblairwhy did we add it to the images?14:34
fungijeblair: this is all related to fixes clarkb and ianw were working on last night to try to get v3 working14:34
*** udesale has quit IRC14:34
jeblairyeah, i'm trying to find the issue that prompted it14:34
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Chown /home/zuul/cache before trying to modify it
jeblairi guess it's that?14:35
mordredjeblair: the issue was the v3 legacy dsvm jobs weren't able to use the files cached on disk so were hitting the internet for all of them14:35
*** vsaienk0 has quit IRC14:35
mordredjeblair: yah. also tripleo was failing because of trying to use a cirros image that wasn't there14:35
mordredI think that patch above should 'fix' the issue - at least until we can fix the image builds properly14:36
jeblairi'm going to spend more time understanding this, because i'm currently confused by why the original fix was needed at all14:36
pabelangerSomething is going to break, because we cannot have both zuul and jenkins owing /opt/cache/files: unless we add them into a shared group?14:37
jeblairwhat is the difference between /opt/cache/files and /opt/stack/cache/files ?14:37
mordredjeblair: I believe /opt/cache/files is where we put things, and /opt/stack/cache/files is where devstack looks for them14:37
fungijeblair: i find clarkb talking about it around 21:33z in scrollback14:37
jeblairmordred: agree14:37
jeblairso in
jeblair(which is the "original" problem)14:38
*** felipemonteiro has joined #openstack-infra14:38
jeblairapparently /opt/stack/cache/files doesn't exist14:39
pabelangerdoesn't something eventually delete it?14:39
jeblairso it almost looks like devstack-gate was never updated to deal with the /opt/cache/files directory existing14:39
*** vsaienk0 has joined #openstack-infra14:40
mordred"If it's still in home, move it to /opt, this will make sure we have the artifacts in the same filesystem as devstack."14:40
*** felipemonteiro_ has joined #openstack-infra14:40
jeblairmordred: yeah, that made me think someone was updating that to support them being in /opt, but there doesn't seem to be any code to handle them being at /opt/cache/files :/14:40
mordredjeblair: I think we have a couple of transitions each half-way done - the ~/cache/files is the old place and is there for backwards compat, but then like you said d-g wasn't updated to just work from /opt/cache/files in thefirst place14:41
jeblairso do we rely on the symlink in current devstack as well?14:41
mordredoh wow14:41
mordredlook at the next line14:41
mordredwe move them from ~/cache to /opt/cache if ~/cache exists - then we copy them to $DEST/devstack/files/14:41
jeblairwell, hardlink but yeah14:42
*** spectr has quit IRC14:43
*** felipemonteiro has quit IRC14:44
jeblairmordred: okay, i think your change will fix it; i think it's a patch on a patch, but unwinding that down to just using /opt/cache is probably something that will take 3 tries to get right :|14:44
mordredjeblair: well - I'm gonna push up a patch for that too14:44
mordredso that we don't forget14:44
jeblairmordred: cool, thx14:45
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files
fungiokay, so we want 511255 in this case at least for now?14:47
*** shardy has quit IRC14:47
jeblairyeah i think so14:48
fungicool, lgtm anyway14:48
fungiif it passes then it should be safe14:48
jeblairi updated etherpad14:48
AJaegerwhat about stopping (starting at rollback) our special nodes like proposal slave? We forgot that last time with rollback...14:50
fungialso, i _think_ is ready to go at this point, when we're comfortable proceeding14:50
fungiAJaeger: i thought the issue was we had stopped zlstatic01 and forgot to start it again14:50
fungido we need to stop the v2 launchers?14:51
mordredfungi: they should be fine - quota will be 014:52
mordredinfra-root: I'm comfortable as soon as that d-g patch lands - I'm pretty sure ALL v3 d-g jobs will fail until it does :)14:52
pabelangerwe might want to eventually to avoid logs from spamming14:52
fungiyeah, that's what i thought... just couldn't recall why we would have stopped them in the previous try14:52
mordredkiennt26: btw - thank you for reporting that issue - turns out it was massively widely important :)14:52
pabelangerfungi: I think we did, but I wasn't here that day14:52
AJaegerfungi: might have been zlstatic01, don't remember which one it was...14:53
jeblairi think that means we have at least 2 hours before we can do anything14:53
kiennt26oh, my pleasure. Thank all for working on it!14:54
pabelangerthink so too14:54
jeblairquite likely more like 314:54
pabelangermaybe longer, check is 203 ATM14:54
jeblairwell, it's *running* jobs, so the check length doesn't have a huge impact on the time we're waiting.14:55
fungisshnaidm|mtg: hopefully also addresses the issue you added to the etherpad14:55
AJaegerwe should promote that change once it reaches gate...14:56
*** fnordahl has joined #openstack-infra14:56
fungiwe could enqueue directly into the gate pipeline if we're in a hurry14:57
fungiand then promote it, yes14:57
*** kiennt26 has quit IRC14:57
*** xarses has joined #openstack-infra14:57
fungii'm happy to do that now14:57
*** srobert has quit IRC14:58
AJaegerlet's do it, there's just one nova job in integrated gate currently14:58
*** spectr has joined #openstack-infra14:58
jeblairi worry about doing that just a little bit since a likely failure mode is that things continue to work, but with much more network traffic (ie, the original v3 problem)14:58
*** LindaWang has quit IRC14:58
fungithat's a fair concern14:58
AJaegerAnd that nova hcange is update from global requirements14:58
*** LindaWang has joined #openstack-infra14:58
jeblairi guess we could do it and then carefully example the logs after merging and be fast on the revert gun14:58
*** dbecker has quit IRC14:59
*** srobert has joined #openstack-infra14:59
*** mat128 has quit IRC14:59
jeblairbut regardless, we'll need to check14:59
*** cshastri has quit IRC15:01
*** rhallisey has quit IRC15:02
*** psachin has quit IRC15:02
*** rosmaita has joined #openstack-infra15:03
*** larainema has quit IRC15:05
mordredthe one question mark I have on the patch is whether or not chown -R traverses symlinks15:05
*** chlong has quit IRC15:06
fungiis failure of tests.unit.test_scheduler.TestScheduler.test_double_live_reconfiguration_shared_queue.test_job_aborted in tox-py35-on-zuul for openstack-infra/zuul-jobs a known flaky result?15:06
*** sree has quit IRC15:06
fungi511253 passed check but then bombed in the gate on it15:06
*** sree has joined #openstack-infra15:07
*** eharney has joined #openstack-infra15:07
rosmaitawhen someone has a minute, i have a question about a review whose dependency has merged, but zuul doesn't seem to be picking up the patch; i looked through the docs and gave it a new workflow +1, but still no action ... not sure what i'm doing wrong.
*** dizquierdo has joined #openstack-infra15:10
*** sree has quit IRC15:11
*** links has joined #openstack-infra15:12
openstackgerritJeremy Stanley proposed openstack-infra/system-config master: Add documentation on force-merging a change
openstackgerritSam Betts proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs
openstackgerritSam Betts proposed openstack-infra/project-config master: Remove ironic legacy jobs
*** efried has joined #openstack-infra15:17
*** dhinesh has joined #openstack-infra15:17
*** ralonsoh_ has quit IRC15:18
fungiinfra-root: fyi, i have a dental cleaning and will be disappearing on you for about 1.5 hours from 18:45-20:1515:18
fungia few hours from now15:18
AJaegerrosmaita: that change has no +1 by Jenkins15:20
AJaegerrosmaita: and without a +1 by Jenkins, Zuul v2 will not merge it - the +1 by Zuul is not enough15:20
fungiyeah, needs a recheck15:21
*** dhinesh has quit IRC15:21
fungiwhich seems to have been tried previously as well15:21
AJaegerfungi, jeblair, so push the devstack-gate change into gate - or not? Didn't understand the conclusion15:21
fungiso i'm guessing there's something else broken...15:21
fungiglace-specs... was that the repo in the corrupted state we needed to repair?15:22
fungii wonder if zuul is having a hard time pulling from it15:22
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: New Summit public endpoints
clarkbnova specs was the repo15:22
*** dhinesh has joined #openstack-infra15:23
fungioh, okay. i'll dig around in zuul debug logs and see if i can spot the problem15:23
AJaegermaybe rebase? It's not current...15:23
openstackgerritMerged openstack-infra/openstackid-resources master: New Summit public endpoints
*** LindaWang has quit IRC15:23
rosmaitaAJaeger fungi : i will try a rebase and see what happens15:23
openstackgerritSam Betts proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs
AJaegersambetts: for the ironic changes: Normal process is: Move the changes to your repo, then sent project-config change that removes them with depends-on set. If we now merge your change, you have none of these jobs left...15:25
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Marketplace API
sambettsAJaeger: so project-config depends on ironic change?15:26
fungirosmaita: looks like that kicked it into gear. my bet is that you had a cross-repo dependency loop which prevented the change from being enqueued back on october 2nd, but it has since been resolved15:27
AJaegersambetts: yes. Isn't that explained properly in
rosmaitafungi ty15:28
sambettsAJaeger: I knew that the names had to be unique, so that why I had the removal before the addition15:28
sambettsAJaeger: but if I have to have all the jobs run twice for a time then ok15:29
AJaegersambetts: please follow the guide15:29
*** andreas_s has quit IRC15:29
fungirosmaita: oh! actually the issue is that zuul v2 ("jenkins") was offline back on the 2nd and we were using zuul v3 ("zuul") which did leave +1 votes, just needed a recheck with v2 running now to get it enqueued there15:29
AJaegersambetts: we can quickly merge your change after the ironic one has merged...15:29
*** andreas_s has joined #openstack-infra15:29
AJaegersambetts: alternative is that you have no testing for ironic for some time and need to freeze repo15:29
rosmaitafungi: i understand now ... also thank you AJaeger15:30
*** ldnunes has quit IRC15:32
*** ldnunes has joined #openstack-infra15:32
openstackgerritSam Betts proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs
openstackgerritSam Betts proposed openstack-infra/project-config master: Remove ironic legacy jobs
AJaegersambetts: commented, this should work fine...15:37
*** chlong has joined #openstack-infra15:38
*** andreas_s has quit IRC15:38
AJaegerfungi, did you want to promote the devstack-gate job into gate pipeline - or wait?15:39
fungiAJaeger: sounds like it was safer to wait and get some additional results15:41
sambettsAJaeger: thanks for the help :)15:41
AJaegerfungi: ok15:42
jeblairfungi, AJaeger, mordred: if that's the plan, what should we do about the v3 rollout?  spend today stabilizing more things and roll out tomorrow morning?15:43
sambettsAJaeger: qq does normal yaml syntax work in zuul v3 configs? e.g. can I use anchors etc??15:43
pabelangerAJaeger: we also need to get back a zuulv3 job too I htink15:43
jeblairpabelanger: let's restart zuulv3 then and recheck.15:43
pabelangerjeblair: wfm15:43
*** hongbin has joined #openstack-infra15:43
jeblairsambetts: yes, but please consider carefully whether it improves legibility or not, and whether what you're doing may be better accomplished some other way (for instance, job inhertance, or variance...)15:44
*** eumel8 has quit IRC15:45
openstackgerritDavid Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall
AJaegermordred: looking at - the PTI includes some javascript jobs, should those stay in?15:45
jeblairsambetts: feel free to send me a link to an example and i can check to see if there are any alternatives that may help15:45
dmsimardAJaeger, jeblair: ^ fixed firewall tests15:45
jeblairi'm restarting zuulv3 to dump the queues15:45
sambettsjeblair: the use case I am considering is that every one of our jobs has the same list of "irrelevent-files" and using an anchor I can define it once put a &common-files and then everywhere else use *common-files instead of duplicating the list15:46
AJaegersambetts: alternative would be to have one common base job -and use inheritance as jeblair commented15:46
*** srobert has quit IRC15:47
*** srobert has joined #openstack-infra15:47
sambettsmakes sense, thanks15:47
inc0good morning, I understand you've kicked off zuulv3 migration?15:48
AJaegersambetts: so, instead of basing on legacy-dsvm-base, you create a legacy-ironic-base job with all the irrrelevant files and then use that everywhre15:48
sambettsyeah, seems like the nicest way to do it15:48
dmsimardsambetts: like others have said, you can set those as a common base job with the parameters that you want and then have other jobs inherit from that, here's an example:
dmsimard'job' is a bit of a misnomer in that context because it doesn't end up running anything by itself15:49
jeblairAJaeger, sambetts: yeah, either would work.  if you have at least one other thing in common with those files (eg, required-projects or something...) i'd say that would tip the scales toward inheritance.15:49
*** chlong has quit IRC15:51
AJaegerand ironic files have long list of required-project, so this fits...15:52
*** rbrndt has quit IRC15:52
mordredAJaeger: well- we don't have good new PTI versions of javascript jobs (although we're getting there) - let's keep them in and get ourselves a nice PTI project-template for javascript15:53
* mordred reviews15:53
AJaegerthanks, mordred15:54
jeblairinc0: no we have not15:54
AJaegerregarding devstack-gate - " libcurl4-gnutls-dev : Depends: libcurl3-gnutls (= 7.47.0-1ubuntu2.2) but 7.47.0-1ubuntu2.3 is to be installed" hits us on the first job already15:54
*** egonzalez has quit IRC15:54
*** ralonsoh has joined #openstack-infra15:54
inc0jeblair: still plannig for today?15:55
jeblairinc0: i don't know what the plan is.  i have an outstanding query to other folks in this channel to formulate a plan, but have not received a response.15:55
clarkbAJaeger: lets check upstream ubuntu and see if that is broken there /me looks15:56
fungijeblair: oh, i thought we were waiting to see whether the devstack-gate patch for cache ownership works before deciding15:56
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: New current summit public endpoint
pabelangerAJaeger: just see that15:56
*** Rockyg has joined #openstack-infra15:56
pabelangerlooks like ubuntu is broken15:56
*** nicolasbock has quit IRC15:56
*** coolsvap has joined #openstack-infra15:56
AJaegerpabelanger: yeah, first reports 7+ hours ago or so15:57
*** slaweq has joined #openstack-infra15:57
fungiAJaeger: clarkb: pabelanger: my initial guess is that it's a conflict between xenial main and uca... do we unconditionally enable uca?15:57
clarkbAJaeger: pabelanger that says 2.3 now15:57
clarkbfungi: for devstack we do yes15:57
*** vsaienk0 has quit IRC15:57
pabelangerAJaeger: fungi: clarkb: looks like all jobs with bindep-fallback are breaking15:57
mordredinfra-root: I think we should not roll out with devstack-gate in its current state - so I thnk we should continue to work the d-g problem until it is solved. If we're comfortable with the time left in the day after it is solved I think we can roll forward - otherwise we should regroup15:58
*** sileht has quit IRC15:58
mordredzomg uca conflicting with xenial wtf15:58
clarkbpabelanger: AJaeger fungi ok if its all bindep fallback then likely not completely UCAs fault as we should only uca with devstack15:58
*** sileht has joined #openstack-infra15:58
*** pcaruana has quit IRC15:59
*** rcernin has quit IRC15:59
mnaserlibcurl4-gnutls-dev doesnt seem to be in uca?15:59
jeblairmordred: ack.  i can't imagine that happening in less than 3h without the uca issue, and the uca issue will likely add several more hours.  so i'm not optimistic we'll be doing it today.15:59
clarkbubuntu package listing above says it should dep on 2.3 so I'm guessing we've synced some incomplete or slow to update mirror state15:59
mordredI DO think we should send out a quick update to the list (I can write one real quick) explaining there are 2 different devstack-gate issues we're working at the moment and so we're not going to roll out in th emiddle of that15:59
fungiso did we end up with an in-between mirror state?15:59
openstackgerritMerged openstack-infra/openstackid-resources master: New current summit public endpoint
pabelangerclarkb: Ya, security update seems to have done it15:59
mordredjeblair: I'm also not optimistic - mostly just wanted to be complete about my thinking16:00
jeblairmordred: should we set a go/no-go time for today?16:00
funginew mirror pulse should be kicking off right now, correct?16:00
pabelangerevery 2 hours16:00
pabelangerchecking logs16:00
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Marketplace API
mnaseroh it could be an rsync failing mid-sync16:00
fungimnaser: shouldn't be. if the mirror process fails to complete we don't release the volume16:01
mnaseri remember once we had a similar issue regarding permissions (i believe fungi helped resolve that)16:01
fungimnaser: but also... mirroring these via reprepro not rsync16:01
mnaseroh you're right16:01
mnaserill boot up a xenial vm, apt-get update and apt-get install libcurl4-gnutls-dev16:02
pabelangerokay, I see some errors in mirror-update16:02
pabelangerworking to fix it now16:02
fungithanks pabelanger16:02
*** lucasagomes is now known as lucas-afk16:02
mnaser^ awesome16:02
clarkbjeblair: mordred: being on the west coast I'm good with as late as early afternoon, but realize that may no work great for others (re go no go)16:02
fungimnaser: thanks, would be nice having a control group to confirm that the general situation with ubuntu is expected working for those packages16:02
mnaseri was able to install libcurl4-gnutls-dev successfully (from our mirrors on our cloud)16:03
mnaserSetting up libcurl4-gnutls-dev:amd64 (7.47.0-1ubuntu2.3) ...16:03
mnaserso that installed without any issues16:04
fungiooh, this _could_ i think be caused by newer packages on our images than in the mirrors16:04
fungiso easily explained by whatever pabelanger found broken16:04
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Fix on public endpoints
*** slaweq has quit IRC16:05
pabelangerfungi: mnaser: clarkb: ya, our mirror-update for ubuntu is out of sync. manually fixing, then will see what happened16:06
*** rhallisey has joined #openstack-infra16:06
clarkbpabelanger: was the cron erroring?16:06
AJaeger511255 has lots of failures currently ;(16:06
AJaegerno time to investigate16:06
mnaseryeah usually package X will depends on Y 2.2 (did not update with mirror-update), but package Y was already sync'd up to 2.3 and 2.2 was removed16:06
fungiwe've gotten into this state in the past, owing to the fact that our image build process doesn't use our mirrors16:07
pabelangerclarkb: I think we killed reprepro, with timeout and it left lockfile16:07
pabelangerso, next runs couldn't properly sync16:07
clarkbpabelanger: gotcha16:07
mnaserAJaeger oh that sa lot of red16:07
*** dbecker has joined #openstack-infra16:07
* mnaser will try looking16:07
pabelangerclarkb: possible there was a large update from ubuntu, which look longer then 30mins to download packages16:07
mnaser2017-10-11 16:05:20.611 | rm: cannot remove '/home/zuul/cache/files/cirros-0.3.4-x86_64-disk.vhd.tgz': Permission denied16:08
mnaseri think a fix for this was merged?16:08
mnaseror was proposed16:08
pabelangermnaser: not yet16:08
*** jpich has quit IRC16:08
*** wolverineav has quit IRC16:09
mnaserlooks like that fix didn't work in the gate as AJaeger mentioned16:09
*** nicolasbock has joined #openstack-infra16:09
*** bhavik1 has joined #openstack-infra16:09
clarkbmnaser: its failing on the tls thing I think16:09
clarkbmnaser: so have to fix one item at a time :)16:09
mnaserclarkb: i dont think so -- at least here -
*** hemna_ has joined #openstack-infra16:10
efriedoh, no cutover yet, so ignore bogus merge conflicts from v3?16:10
pabelangerclarkb: ya, we are downloading new kernels from ubuntu security, that would have pushed us over 30min timeout16:10
AJaegerefried: tell us about those, please16:10
efriedAJaeger e.g.
mordredinfra-root: draft status update email16:10
*** chlong has joined #openstack-infra16:11
clarkbmnaser: chown -R must not follow symlinks?16:11
AJaegerefried: why bogus merge conflict?16:11
clarkbmordred: ^16:11
efriedAJaeger PS23 is on the tip of the master branch.  No further rebase is possible.16:11
efriedAnd no dependencies16:11
mnaserclarkb i guess time to recreate a small experiment to test this out16:11
AJaegerefried: but gerrit says "Merge conflict"16:12
efriedAJaeger Just so :)16:12
mordredclarkb: ah - so perhaps is actually the better option? or we can respin that one to chwon the files too16:12
efriedAJaeger It should not :)16:12
jeblairmordred: was there more to that msg about status update?16:12
clarkbmordred: mnaser ya we need -L -R16:12
AJaegeranybody here has time to help efried?16:12
mordredjeblair: whoops16:12
clarkbthe default iwht -R is to not traverse links16:12
jeblairefried, AJaeger: yep, will look16:12
efriedAJaeger I don't need to be helped, necessarily, assuming this is just a zuulv3 snafu and I can ignore it for now.16:13
*** hemna_ has quit IRC16:13
mordredjeblair: I *think* that might be an instance of the issue I had on the etherpad with no specific examples16:13
mnaserclarkb i think: sudo chown -R zuul:zuul /opt/stack/cache/files/ (the trailing slash) makes it not chown the root dir itself so while the file permissions update, the directory is still likely unacessible?16:13
efriedY'all have enough to worry about16:13
*** wolverineav has joined #openstack-infra16:13
mnaserFYI i like to add -v to get a verbose output of whats happening16:13
clarkbmnaser: according to the manpage -P is default with -R which means don't traverse any links16:13
clarkbmnaser: i htink we just have to add -L16:13
AJaegerefried: if there's a bug, let's investigate so that we can access the siutation...16:14
mnaserclarkb im all for doing that (and maybe adding -v to see what it is touching exactly for easier debugging?16:14
clarkbmnaser: sure16:14
clarkbmnaser: do you want ot push that patchset?16:14
efriedAJaeger Sure, just let me know what you need from me.16:14
AJaegerefried: jeblair is looking  -thanks16:14
mnaserclarkb sure, i think mordred is busy keeping up with other stuff now16:14
mordredmnaser, clarkb: I think is a more complete solution  - if we're gonna push new patchsets - perhaps we should either squash them - or we should rebase 511260 so as not to be parented on the other and race to see which fixes first?16:15
fungimordred: lgtm, rabbie burns included16:16
efriedjeblair I have a few examples of the same thing.  It actually seems like I'm getting dinged with merge conflicts in all my nova patches (and none of them look like real merge conflicts afaict).16:16
efriedjeblair Here's another:
*** hemna_ has joined #openstack-infra16:16
*** sshnaidm|mtg is now known as sshnaidm16:16
mnasermordred i think that's much cleaner16:17
pabelangeraptmethod got ''16:17
pabelangerthat is better16:17
mnaserim rechecking that change16:17
mordredmnaser, clarkb: should we reparent it so they're not tied together?16:17
clarkbmordred: mnaser oh ya maybe lets just use that fix then16:17
clarkband I don't think we need both so squashing to a single thing should be fine16:17
mnasermordred clarkb yeah the chown would be useless at that point16:18
clarkbthe only problemwith that is going to be things like tripleo16:18
clarkbbecuse they don't use devstack and so look for things in places16:18
mnaserwe could fix that in pre.yaml for their jobs16:18
mordredclarkb: that's a good point - however, they've temporarily disabled consuming from cache16:19
mordredpending our cache fixes making it in to the images16:19
mordredso we could just make sure when we revert their disable patch that we update it to pull from /opt/cache/files instead16:19
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files
mordredthere's a squashed/rebased version of that ^^16:19
mordredEmilienM: your patch that disabled consuming cirros images from the cache hasn't been reverted yet,right?16:20
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove legacy job from openstackdocstheme
fungimordred: i think maybe it did because sshnaidm reported it failing earlier today16:20
mnasercould we bump mordred's change to top of zuulv3 queue?16:21
sshnaidmfungi, mordred shouldn't be disabled, I pushed it yesterday lately16:21
pabelangermnaser: no point until we fix ubuntu AFS16:21
sshnaidmwe shouldn't consume this image right now16:21
pabelangerwhich should be in the next 20mins I think16:21
fungiat 11:58z in here he said "we have a problem with setup workspace in tripleo jobs: rm: cannot remove /home/zuul/cache/files/cirros-0.3.5-x86_64-disk.vhd.tgz Permission denied"16:22
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove legacy job from openstackdocstheme
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: remove openstackdocstheme-tox-doc-publish-buildpdf
EmilienMsshnaidm: thanks, good to know16:23
AJaegernow post failure on ;( I'll recheck16:25
*** dizquierdo has quit IRC16:25
mordredfungi, EmilienM, sshnaidm: ok - so the d-g patch will actually fix tripleo too because tripleo-ci consumes the function that is fixed in that patch16:26
mordredclarkb: ^^16:26
*** dhajare has quit IRC16:26
mordredso it's set up to consume from /opt/stack/cache/files/cirros-0.3.5-x86_64-disk.img16:27
clarkbwill tripleo look to copy the files from that location though?16:27
clarkbah ok16:27
mordredso no additional vector to worry about16:27
sshnaidmwhich patch, sorry?16:27
toskyquestion: the jobs live in the same namespace (so job names should not collide), but this is not valid for role names, because you need to explicitely specify the repository - is it correct?16:27
mordredsshnaidm: the one fromyesterday - we're making a d-g change and wantedto amke sure it wouldn't break you16:27
sshnaidmmordred, ok, thanks16:27
mordredsshnaidm: it should be fine (should actualy fix a current issue)16:27
AJaegertosky: correct - but you can reference different repos16:27
pabelangermordred: did we land your zuul patch for post playbook logs? 511253 might have just hit it16:27
AJaegertosky: use "roles: - zuul: openstack-infra/zuul-jobs" for example to use roles fro mthere16:28
mordredtosky: that is correct. I mean- roles also live in the same namespace- but just making a role with a conflicting name will not necessarily break anyone becaues they would have to explicitly request that role to be added16:28
AJaegertosky: see openstack-manuals/.zuul.yaml for an example16:28
toskyack: the problem I'm facing is the name of a role, if I should try to make it unique or I can rely on the name of the repository (and it seems to be the latter)16:28
mnaseri think base jobs use openstack-zuul-jobs or zuul-jobs16:29
mnaserfor the roles, which means i'd avoid any roles that have the same names there16:29
*** yamahata has joined #openstack-infra16:30
*** Goneri has joined #openstack-infra16:31
dmsimardmnaser: there is a notion of precedence16:31
dmsimardI'd need to test it, but first path in the role path wins16:31
*** weshay is now known as weshay|ruck16:32
*** panda is now known as panda|rover16:32
jeblairdmsimard: feel free to test it, though you could also look at the docs:
jeblairtosky, mnaser: ^16:33
dmsimardSomething that we could do, also, is to make the role paths relative16:33
dmsimardSo instead of "upload-logs", it'd be "project-config/upload-logs" or something like that to avoid name clashing16:34
dmsimardYou can do that by putting your role path one depth lower (or higher, depending on how you look at it)16:34
jeblairdmsimard: okay please do read the docs.16:34
*** d0ugal has quit IRC16:35
mordredtosky: yah - you only need to worry about it if you're planning on other people using them16:35
dmsimardjeblair: I know about the docs and Zuul, I'm talking about Ansible here16:35
toskyok, I guess I will invest a bit more time16:35
toskyfuture proof is better done now16:35
*** Swami has joined #openstack-infra16:35
*** mpaolino has quit IRC16:36
*** e0ne has quit IRC16:38
*** ralonsoh has quit IRC16:39
*** dtantsur is now known as dtantsur|afk16:39
*** dizquierdo has joined #openstack-infra16:39
dmsimardjeblair: I'm saying that Ansible provides the ability to add a role path that is not a directory of roles -- for example 'zuul-jobs' could be a role path (instead of zuul-jobs/roles) and then when you need a role from there, such as upload-logs, you specify 'zuul-jobs/roles/upload-logs' instead of just 'upload-logs'.16:39
*** felipemonteiro_ has quit IRC16:40
mordreddmsimard: right- we talked about doing that - but the problem is that it gets weird when wanting to be able to test playbooks or roles that people have written for things that aren't zuul16:40
clarkbmordred: if you hvaen't sent that email yet I think you need to update the change number for the cache fix16:41
mordreddmsimard: because although you can set up your role path that way - the default behavior out in the ansible world is also a flat namespace, so we've been wary about introducing zuul-specific semantics for such things16:41
clarkbmordred: I've updated it16:41
mordredclarkb: cool - and wll send now16:41
dmsimardmordred: sure, I was pointing out that the possibility was there if we were interested in addressing potential name clashing16:42
dmsimardwhether or not we want to do that, I don't have a strong opinion16:42
mordreddmsimard: nod16:42
dmsimardknowing that it's possible and we consciously don't want to do it is totally fine16:42
*** epende has joined #openstack-infra16:42
*** rbrndt has joined #openstack-infra16:43
*** bhavik1 has quit IRC16:43
ependeI've created a server group using ServerGroupAntiAffinityFilter.  I'd like to have nodepool use that group for certain images.  Is there a way to do this?16:44
clarkbepende: is that exposed via nova boot api to non admin users?16:45
dmsimardepende: server group ? is that like a host aggregate ?16:45
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Convert legacy-api-ref and -guide to native v3
mnaserserver groups are a tenant user-facing concept for users16:45
dmsimardmnaser: TIL16:45
mnaseryou can create a server group which has (soft)-affinity and (soft)-anti-affinity16:46
ependedmsimard server group as in this example:
mnaserthe filter is simply a tool in the scheduler which enforces it16:46
AJaegermordred: what do you think of 511290? It's just converting, so low priority...16:46
mnaserif you dont have the filter, the affinity settings wont be enforced, so yeah16:46
ependeI have the group created and set to use the filter, but I would like nodepool to use the new group16:46
*** larivee has joined #openstack-infra16:46
clarkbepende: in that example you linked it appears that you use a specific flavor to end up in the group16:46
clarkbepende: in the nodepool case you should be able to just tell nodepool to use that group16:47
fungiby telling it to use that flavor?16:47
ependeclarkb: Yes, I have the image I need, but I do not see where in nodepool's settings to configure that gruop16:47
clarkbfungi: er ya sorry16:47
clarkbepende: what fungi says, tell nodepool to use the flavor associated with the group16:47
dmsimardepende: how are you hinting at the group ? through the image ? through the flavor ? or really through the group ?16:47
*** chlong has quit IRC16:48
*** dprince has quit IRC16:48
clarkboh wait16:48
mnaserscheduler hints16:48
clarkbits a hint16:48
* clarkb rages at the nova api16:48
fungiso we'd need special hints support in nodepool i suppose?16:48
clarkbfungi: ya I think so16:48
fungiand plumb that through shade16:49
* mordred reads16:49
ependeOk, I thought that might be the case.  Thanks clarkb and mnaser and dmsimard16:49
*** dprince has joined #openstack-infra16:50
clarkbwhy isn't that just a flvaor16:50
clarkbwe have too many pizza cutters16:50
dmsimardI wonder if a flavor extra spec could be used for this16:50
mordredwell - if the right way to use such a thing is to pass things directly in the create call (like it is for azs) I don't see any reason to not support it16:50
clarkbmordred: ya we should support it, mostly just grumpy that nova has like 50 ways to slice the pizza16:51
openstackgerritMiguel Lavalle proposed openstack-infra/irc-meetings master: Create alternate time for Neutron Drivers meeting
*** sshnaidm is now known as sshnaidm|off16:52
mordredk. I have found thedocs for how it works - adding support will not be hard16:52
openstackgerritMiguel Lavalle proposed openstack-infra/irc-meetings master: Create alternate time for Neutron Drivers meeting
openstackgerritMerged openstack-infra/zuul-jobs master: Also add security.list for Debian on configure-mirror
clarkbepende: out of curiousity why would nodepool nodes need anti affinity? is it just to keep them from bad neighboring each other?16:54
*** links has quit IRC16:54
clarkbin theory they are completely ephemeral instances so doesn't really matter where they schedule to16:54
ependeclarkb:  just a primative way to prevent internal fragmentation by having too many really large nodes from going to one compute host16:55
dmsimardepende: are you an operator of that cloud ?16:55
ependeIf there's another, better way to solve the problem I'm open to it16:56
ependeReally the problem is, what if too many smaller nodes spawn on a compute host and crowd out all the larger nodes, leaving a lot of unused RAM & disk16:57
*** d0ugal has joined #openstack-infra16:57
dmsimardso a typical bin packing problem16:57
dmsimardyes :)16:57
*** cody-somerville has joined #openstack-infra16:58
*** cody-somerville has joined #openstack-infra16:58
clarkbpabelanger: are mirrors still updating?16:58
*** derekh has quit IRC16:58
ependedmisard: We may be able to accomplish something similar with AZs, but it's less ideal16:58
dmsimardepende: I would say that leaving a lot of unused ram and disk is usually a consequence of mismatching compute hardware, flavors and overcommit ratios16:58
dirkinfra-root: could somebody remove the jenkins-2 here: and we'll wait for zuul to vote on it16:58
pabelangerclarkb: ya, I'm rechecking checksums for the indexes just to be sure16:59
pabelangerclarkb: taking a little time16:59
*** andreas_s has joined #openstack-infra16:59
AJaegerdirk: we still merge using jenkins...16:59
*** baoli has quit IRC16:59
dirkwhat? I thought we have the big flip over today?16:59
pabelangernot yet, see ML and status notice17:00
fungidirk: nope, we got sidelined by various unrelated problems17:00
AJaegerdirk: remember the problem we talked about? Took us longer and then some more problems came up...17:00
*** felipemonteiro has joined #openstack-infra17:00
dmsimardepende: for example, a 32 core node with 64GB RAM works fairly okay with a 1:1 vcpu allocation ratio and flavors which are usually 1vcpu to 2GB of RAM17:00
*** baoli has joined #openstack-infra17:00
*** eumel8 has joined #openstack-infra17:00
dmsimardepende: but if you have a 32 core node with 128GB RAM, you'll likely want to increase your cpu overcommit to 2:1 so that you can use that RAM, you'll end up allocating 64 vcpus instead of 32.17:00
ependedmsimard: we're talking 110GB RAM, 16 VCPU flavors for the biggest ones, all the way down to 2G.17:01
AJaegerdirk: so, wait for the Jenkins recheck, please17:01
*** Apoorva has joined #openstack-infra17:01
*** felipemonteiro_ has joined #openstack-infra17:01
clarkbepende: my biggest concern with the anti affinity approach is it requires your users to opt in, so won't affect anyone not in the know17:01
clarkbepende: this may be ok if you ar ethe only user but otherwise you'll likely to continue to have problems17:02
dmsimardepende: and the scheduler currently sends VMs to the compute node with the least instances ? with the most available ram ?17:02
ependeclarkb: that is ok for our use case as we're the only users17:02
openstackgerritMerged openstack-infra/openstackid-resources master: Fix on public endpoints
mordredclarkb: there are definitely private cloud usage patterns that are a bit different from how we consume the public clouds17:03
ependedmsimard:  We haven't observed enough scheduler behavior to know yet, as we only have one compute host capable of accepting mroe than one large instance right now17:03
clarkbmordred: oh sure but if you have more than one person hitting the boot api all of a sudden this pattern falls over17:03
openstackgerritMonty Taylor proposed openstack-infra/shade master: Image should be optional
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add a call for setting volume bootable flag
jeblairfungi, clarkb, mordred: looking into the random erroneous merge conflict error, i found some interesting information.17:04
clarkbanti affinity is something you do when you know you want the 3 database servers on 3 different hypervisors so its opt in17:04
mordredjeblair: oh yeah?17:04
clarkbbut ya if its single user it will probably work fine17:04
*** kjackal_ has quit IRC17:04
ependeThe use case is that we're running a private CI/CD cloud where nodepool instances are used as needed by the tests.  The remaining instances are static.17:04
*** gouthamr has quit IRC17:04
ependeclarkb, mordred:  we mainly want to prevent starvation of the really large images if the scheduler does not pack the other instances optimally17:05
*** dhinesh has quit IRC17:05
*** andreas_s has quit IRC17:05
*** felipemonteiro has quit IRC17:06
*** dizquierdo has quit IRC17:06
dirkfungi: pabelanger : AJaeger : alright, thanks17:06
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Marketplace API
*** andreas_s has joined #openstack-infra17:07
*** ganso has left #openstack-infra17:07
*** jascott1 has quit IRC17:09
*** jpena is now known as jpena|off17:09
*** trown is now known as trown|lunch17:10
efriedjeblair Do tell17:10
persiaCould someone point me at the place to start debugging a post-gate publication job that may not have run?17:13
AJaegerpersia: check and see whether it has run17:14
persiaAJaeger: It is no longer in queue17:14
AJaegerpersia: if it has finished all post jobs already, check the log files, let me get you a link...17:14
smcginnisUnfortunately I have to step away, but just saw a zuul RETRY_LIMIT failure. If anyone has a moment to check out why, that would be awesome. Or I'll try later.17:14
*** rkukura has quit IRC17:14
*** rkukura_ has joined #openstack-infra17:14
AJaegerpersia: has information on how to get a link17:14
*** nikhil has joined #openstack-infra17:14
*** tesseract has quit IRC17:15
jeblairefried: i switched to #openstack-infra-incident because it was busy here.  of course then everyone stopped talking.  anyway, here's the log:
fungipersia: if this is the election repo, i'm almost certain you hit a publication race between changes which merged at nearly the same time. we don't enforce serial ordering for jobs running on individual commits in the post pipeline17:15
pabelangersmcginnis: known issues, we are working to fix AFS mirrors for ubuntu17:15
pabelangerhopefully not much longer17:15
persiaAJaeger: Thank you.17:15
*** andreas_s has quit IRC17:15
fungipersia: merging another change in that repo will get you to the state you expect, or one of us can manually retrigger the one which was expected to run last (but didn't because it probably ran on a slightly slower node than the one for the change which came before it)17:16
persiafungi: Ah, so if two jobs merged within a few minutes, it is possible that the post job only ran once, against a stale repo?17:16
*** andreas_s has joined #openstack-infra17:16
*** larivee has left #openstack-infra17:16
fungipersia: the post job ran for both commits, but may have published the result for the penultimate commit after the one for the ultimate commit17:17
persiaAh, no, it ran twice, but the older change overwrote the newer one.  I understand.  Yes, I'll just push a meaningless change to fix that (or find a real change to be fixed)17:17
fungidue to the one which was enqueued slightly sooner running slightly longer17:17
openstackgerritMiguel Lavalle proposed openstack-infra/irc-meetings master: Update Neutron team meeting chairperson
*** sambetts is now known as sambetts|afk17:17
fungipersia: we have some ideas for a new queue type in zuul which could solve this specific problem, but wanted to hold off implementation until v3 is fully baked17:18
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add group parameter to create_server
mordredclarkb, epende:^^ there's the shade support for setting a group on a create_server17:18
jeblairya, i've totally written that patch in my head.  just haven't typed it out yet :)17:18
persiafungi: I am entirely in favour of holding off on things until zuulv3 is fully baked :)17:19
*** dizquierdo has joined #openstack-infra17:19
jeblairthere's one thing i haven't figured out yet -- what to call it.  i'm thinking the 'supercedent' pipeline manager (to go along with 'dependent', and 'independent').17:20
*** rbrndt has quit IRC17:20
*** caphrim007_ has joined #openstack-infra17:20
clarkbyesterday I brought up a duplicate gerrit account situation. User has confirmed that when they login they get the "new" one so I think we need to associate the new openid with the old account.17:21
clarkbdoes `UPDATE account_external_ids SET account_id = 21561 WHERE external_id = '';` look correct for doing that?17:21
clarkbmordred: fungi  ^17:21
jeblairas discussed in #openstack-infra-incident i'm going to update all the origin remotes on the v3 system17:21
*** panda|rover is now known as panda|rover|off17:22
fungiclarkb: yes, that is what i would normally run under those circumstances17:22
fungijeblair: a for loop with sed over the .git/config files is how i'd probably do it17:23
persiaAJaeger: Intrestingly, although the parent of the change has logs at that location, there seem to be none for d55c590ad189904043ef0496a9cb6d8eed8211d2 in openstack/elections - would this indicate the job didn't run, or is there some other log that shows which jobs did run?17:23
clarkbfungi: ok I'm going to run that query now17:23
*** caphrim007 has quit IRC17:24
fungipersia: if the change wasn't a fast-forward merge onto the branch, then there will be a merge commit and _that's_ what post jobs get run for17:24
clarkband I've set account 27031 inactive17:24
persiafungi: Thank you.17:25
fungipersia: so c722a78bea5d1a75cb204cc783b2480131bd5bc4 based on the git log17:26
clarkb#status log move Gerrit account 27031s' openid to account 21561 and marked 27031 inactive17:26
openstackstatusclarkb: finished logging17:26
persiafungi: Right, which seems to have bumped into an issue with apt mirroring, which I understand is currently being fixed, so I should wait a bit before re-running.17:27
*** jascott1 has joined #openstack-infra17:27
fungipersia: oh, neat, so a job failure in this case? inconvenient timing17:27
*** links has joined #openstack-infra17:28
persiaYep.  There are 10 hours before it needs to be sorted, so it isn't catastrophic, but definitely inconvenient.17:28
fungibut yeah, pabelanger is on the case where that issue is concerned17:28
pabelangeryes, just rechecking checksums on mirror, taking a little bit17:29
pabelangeronce done, we can release AFS volume and things back to working order17:29
*** andreas_s has quit IRC17:29
fungithanks pabelanger!17:30
mordredjeblair: my $0.02 is to do a for loop with git remote remove origin ; git remote add origin ssh://$PROJECT.git17:30
jeblairmordred, fungi: this is what i came up with:
AJaegerjeblair: will that work with the cd?17:32
clarkbjeblair: lgtm and probably atomic and won't race other ops17:32
mordredjeblair: yes - however, I'd suggest making line 7 and 8 (cd $root/$org/$repo && git remote set-url origin ssh://$org/$repo)17:32
clarkbAJaeger: ya $root is rooted17:32
AJaegerIw would have written "(cd somewhere;do-stuff)17:32
AJaegerclarkb, Ah, then it works17:32
mordredoh - cool17:32
fungijeblair: that works. the reason i normally use sed is that i can just use a single loop over $root/*/*/.git/config and pass them to sed -i17:32
mordredI also retract my thought17:32
mordredjeblair: so - that lgtm17:33
AJaegermordred: could you review , please? Can we do this more compact?17:33
jeblairfungi: that sounds nice... though i think with the git command we actually get locking!17:33
*** baoli_ has joined #openstack-infra17:33
fungijeblair: oh! i had no idea git locked around the config file. sounds ideal then17:34
jeblairfungi: i didn't either until i ran that command as the wrong user!17:34
funginice ;)17:34
*** tosky has quit IRC17:34
mordredAJaeger: you say "This is only run on master branch of a project" - just to be clear, do you mean that it should only be triggered by patches to master and patches to other branches should be ignored?17:35
*** baoli has quit IRC17:35
AJaegermordred: correct17:35
mordredAJaeger: or that no matter which branch triggers it, the job should run on master (like reno)17:35
mordredAJaeger: cool17:35
AJaegermordred: the first - only changes against master should trigger it. Can update the wording...17:36
mordredAJaeger: no - that's fine - I just wantedto make sure17:36
mordredAJaeger: there are still uses of legacy-api-guide and legacy-api-ref in post in that project-template17:37
*** inc0 has quit IRC17:37
AJaegermordred: yes - that would be a followup. Want to do it one step at a time17:37
mordredAJaeger: k. however, you deleted the post playbooks in this patch ...17:38
melwitthi, in nova we're seeing zuul v3 report false merge conflicts on patches. is that expected to be seen?17:38
*** ykarel is now known as ykarel|afk17:38
pabelangermelwitt: I believe we're working on a fix now. See the above discussions17:38
mordredmelwitt: yah- it's not 'correct' behavior - but we're looking in to it17:39
melwittcool, thank you17:39
jeblairokay i've updated remotes on zm05-zm08 and ze01-ze1017:39
mordredjeblair: we think that the merger issue you're fixing is the cause of the merge issues yeah? (and not just a thing that you happened to find while looking)17:39
jeblairmordred: yep17:39
mordredthat's exciting17:40
jeblairmelwitt, efried: zuul v3 should not report any false merge conflicts after this point.17:40
AJaegermordred: that template is broken, post job and build job should not be the same - ok, will fix together.17:41
*** e0ne has joined #openstack-infra17:41
efriedjeblair Thanks!17:41
mordredAJaeger: oh - nevermind then17:41
mordredAJaeger: that patch looks fine17:41
*** dhajare has joined #openstack-infra17:41
AJaegermordred: ok, will do the publish then as well, thanks!17:41
mordredAJaeger: speaking of - did you see my email on the infra list about "Change publication interface ..."17:42
mordredAJaeger: (it's partially inspired by the extra work we have to do for these jobs)17:42
jeblairmoved to fixed section of etherpad17:42
*** andreas_s has joined #openstack-infra17:43
jeblairmordred: what's the current devstack-gate change?  i saw something about squashing?17:44
clarkbjeblair: 51126017:44
jeblairthat's still parented on the other change, isn't it?17:44
clarkbI think the newer patchset unparented it17:44
mordredjeblair: no - I squashed it17:44
jeblairderp, i was looking at an outdated copy, thanks17:45
*** dhinesh has joined #openstack-infra17:47
*** andreas_s has quit IRC17:47
*** andreas_s has joined #openstack-infra17:48
*** slaweq has joined #openstack-infra17:48
pabelangerchecksums still running :(17:49
pabelangertaking much longer then expected17:49
jeblairdmsimard: what do you mean "tripleo has their own graphite server" ?17:49
mordredwow. if only there was some way that we could collectively work on a solution to collect stats on tasks run that could be shared effectively amongst all the users17:49
jeblairEmilienM: ^?17:49
rajinirSome basic issue after we upgraded to the latest project config, gearman server not starting .., How to debug this issue?17:50
pabelangerjeblair: private infrastructure that lives in tripleo-test-cloud-rh117:50
dmsimardpabelanger beat me to it17:50
*** rosmaita has quit IRC17:50
jeblairwhat in the *world* are we doing here?  are we working together on a project, or are we building little fiefdoms?17:50
jeblairi know what i thought we were doing17:50
EmilienMjeblair: you might want to talk with sshnaidm17:50
EmilienMI haven't worked or contributed to that parts17:50
fungigraphing stats for the underlying hardware in the test cloud?17:50
EmilienMand yes we should work together17:51
EmilienMweshay|ruck: you might want to see ^17:51
*** rosmaita has joined #openstack-infra17:51
dmsimardjeblair: the topic came about as EmilienM and I were discussing opportunities for graphing different task/ansible mtrics17:51
EmilienMyes I can't agree more than that - we should build a common tooling for our needs17:52
pabelangerjeblair: 480121 is relevant when you have time17:52
mordred I think this goes hand in hand with the 'shift-tripleo-ci-to-third-party' - a third-party ci running a local graphite server seems perfectly fine, while things run centrally but reporting out to an external graphite server seem much weirder17:52
EmilienMand stop doing tripleo-only things17:52
EmilienMjeblair, mordred: unfortunately, you can't force people to stop what they're doing sometimes17:52
EmilienMI guess it comes by education and showing example17:52
weshay|ruckEmilienM, dmsimard afaik we were measuring the performance of rh1 and ovb jobs17:52
jeblairdmsimard: yeah, that's a great topic.  SpamapS was doing some work on measuring and tracking tasks within jobs and reporting to graphite.17:52
EmilienMweshay|ruck: have we engaged work to do it at an higher level?17:53
EmilienMweshay|ruck: e.g. in OpenStack Infra?17:53
pabelangermordred: yah, I've helf off on commenting until 3rd party CI move17:53
weshay|ruckisn't that one step removed from upstream infra17:53
weshay|ruckya.. pabelanger btw.. we just started to migrate jobs off rh117:53
*** slaweq has quit IRC17:53
jeblairEmilienM, dmsimard, weshay|ruck: see
dmsimardweshay|ruck mentions that it's for OVB jobs which will definitely be third party17:53
*** links has quit IRC17:53
pabelangerweshay|ruck: ++17:54
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Create publish-api-ref/guide jobs
weshay|ruckbefore everyone gets outraged.. ya consider the context a bit17:54
fungipabelanger: it looked like the discussion in #openstack-dev suggested yesterday that the rh1 test cloud was going away, and that (i think?) we can probably remove it from nodepool configuration in the next week or so17:54
dmsimardfungi: I'm not sure a week or so is realistic, but it's been a WIP several weeks in the making. We are making good progress.17:54
pabelangerfungi: great, I'll have to look at backlog for that17:55
weshay|ruckfungi, ya.. no17:55
*** electrofelix has quit IRC17:55
weshay|ruckit's not going away that quickly17:55
dmsimardfungi: and FWIW this third party CI will actually run out of's Zuul17:55
mordredweshay|ruck: I'm not outraged - I mostly want to point out that there are (now, with v3) some potential integration points for such work that could both be leveraged in a 3rd party context and also potentially used directly upstream ...17:55
weshay|ruckmordred, great! that is good news17:55
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Create publish-api-ref/guide jobs
mordredso if this third party CI is going to be running ot of's Zuul it might not be terrible to chat about ways in which the goals that are trying to be accomplished might be done in a way that's shareable and not project-specific17:56
fungidmsimard: pabelanger:
mordred(probably not a good discussion topic to get into in detail today of course)17:56
jeblairthere's also prior work begun on a very similar project.  it has stalled, but there's a bunch of work done about how to actually integrate with graphite17:56
jeblairthat work is applicable for upstream and downstream jobs / servers17:56
weshay|ruckthere were no grand plans around that graphana server anyway, it's just an engineer doing a thing on their own w/o much direction17:56
weshay|ruckon it17:56
*** andreas_s has quit IRC17:56
pabelangerfungi: thanks17:57
*** ijw has joined #openstack-infra17:57
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Convert legacy-api-ref and -guide to native v3
AJaegermordred: pushed and used in updated
*** tosky has joined #openstack-infra17:58
AJaegermordred: saw your email - looking forward to it.17:59
weshay|ruckmordred, jeblair pabelanger ok.. we'll probably have more to talk about in about two weeks once we have rh1->rdo-cloud and software factory well in progress for ovb jobs17:59
mordredAJaeger: - -1 in there with a question - also there's some trailing whitespace that will anger the whitespace gods18:01
jeblairweshay|ruck, sshnaidm|off: cool.  in the mean time, if you have a moment, please give a perusal for background info18:01
AJaegerwhitespace? Shame on me... thanks, mordred18:02
weshay|ruckk.. thanks18:02
*** dhajare has quit IRC18:02
mordredweshay|ruck: looking forward to chatting about it18:02
*** hrybacki is now known as hrybacki|trainin18:03
AJaegermordred: good find, thanks18:03
dmsimardjeblair: the scope of that spec has nothing about ansible, though, should there be ?18:03
anupnHi all, if i want to disable a specific service from running with py3 i see there is a function disable_python3_package that exists so is that has to be called from project's only?18:03
jeblairdmsimard: it's not a new spec.  :)18:03
*** ijw has quit IRC18:04
jeblairit just had its 2nd birthday18:04
*** caphrim007_ has quit IRC18:04
*** ijw has joined #openstack-infra18:05
* SpamapS peers into the past18:05
jeblairdmsimard: i'm definitely not suggesting it's what we should implement -- i'm saying we should be aware of previous work before we invent new things18:05
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Create publish-api-ref/guide jobs
AJaegermordred: fixed ^18:05
SpamapSUnfortunately I never got back to the counter inspection spec.18:05
SpamapSBut it was pretty close to done.18:05
SpamapSJust needed to go the last mile to get those subunit counter attachments into graphing systems.18:06
*** e0ne has quit IRC18:06
SpamapS(and of course after some data collection, add more data points and maybe get more granular)18:06
*** andreas_s has joined #openstack-infra18:06
clarkbre 511260, if a devstack or grenade job runs and passes is that sufficient info to know we covered the cache handling and can enqueue to the gate at that point?18:06
clarkband if so is that something we want to do?18:07
*** dizquierdo has quit IRC18:07
*** trown|lunch is now known as trown18:07
mordredclarkb: I think we want to see a legacy-devstack job run on top of that in v3 - but if it runs and passes in v2, then I don't think landing it will break v3 legacy dsvm jobs any MORE than they are broken right now18:08
clarkboh right its the v3 side that we want to check on18:08
clarkbI guess we don't want to regress v2 though18:08
AJaegermordred: there're still some variants of api-ref but my changes should cover the base...18:09
jeblairyeah.  we do need to inspect logs to verify we haven't fallen back on network behavior18:09
mordredAJaeger: ++18:09
jeblairwe only need one job complete for that though18:09
mordredjeblair: agree18:09
mordredAJaeger: sorry - caught one more thing18:11
mordredAJaeger: but otherwise looks spot-on18:11
*** andreas_s has quit IRC18:11
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats
*** hasharAway is now known as hashar18:14
*** rbrndt has joined #openstack-infra18:15
jeblairi'm going to afk for about an hour for lunch and things18:16
*** efried is now known as efried_nomnom18:17
smcginnispabelanger: Thanks!18:22
*** kjackal_ has joined #openstack-infra18:23
*** baoli_ has quit IRC18:24
*** d0ugal has quit IRC18:24
*** baoli has joined #openstack-infra18:24
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Create publish-api-ref/guide jobs
AJaegermordred: Thanks for catching that! Next try ^18:26
*** armaan has joined #openstack-infra18:27
mordredAJaeger: looks awesome! clarkb, pabelanger wanna +3 ?18:27
*** yolanda has quit IRC18:28
mordred also looks great clarkb pabelanger18:29
clarkbmordred: AJaeger 315 is approved18:30
honzamordred: is there a place where i can track any progress on the UCA mirrors issue you mentioned on the ML?18:30
mordredhonza: not anywhere great - I believe right now pabelanger has something running that is verifying checksums - and is taking its sweet time doing it18:32
dmsimardfungi: did I hallucinate telling you about rh1 not closing next week ? I somehow remember typing about it but not finding it in my buffers :/18:32
dmsimardfungi: oh nevermind, it's in -incident instead of -infra, my bad.18:32
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Use build-openstack-api-ref for openstackdocstheme
honzamordred: cool, thanks, i shall attempt to follow along here18:33
pabelangermordred: honza: ya, we are currently limited by rax bandwidth, reprepro wants to ensure proper checksums for all packages18:33
openstackgerritMerged openstack-infra/project-config master: Create publish-api-ref/guide jobs
AJaegerclarkb: thanks18:36
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: remove legacy openstackdocstheme-tox-doc-publish-buildpdf an
clarkbnow we recheck 290?18:36
mordredalso - ... I love the new error message18:37
dmsimardQuestion about elastic-recheck -- it turns out it's not straightforward to support zuul v2 and zuul v3 simultaneously. Do we care about v2 support in elastic-recheck post roll-forward ?18:37
clarkbdmsimard: what is the issue with supporting both?18:38
dmsimardI think if there are external/third party CI users they can pin to a commit before v3 is added in or something like that18:38
dmsimardclarkb: mainly the expectation of a file named either console.html or job-output.txt18:38
clarkbdmsimard: and we can't query for both?18:38
clarkbdmsimard: maybe we should make it a configurable file name and then others can configure to use whatever they need18:38
clarkbso default can be job-output.txt, but allow it to be set to console.html18:39
*** armaan has quit IRC18:39
dmsimardclarkb: it's for the 'required_files' method here:
clarkbgotcha, ya in that case maybe we can make that filename configurable?18:40
fungiokay, i have to step out to the aforementioned dental checkup for the next ~1.5 hours but will rejoin you all soonish18:40
* fungi disappears into the aether18:41
* AJaeger waves good night18:41
*** dbecker has quit IRC18:41
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: switch whereto to publish docs on docs.o.o
dmsimardclarkb: I'd like to, yes, but I was also interested in keeping the changes to a minimum. Ideally, I'd really prefer another hint at whether a job is running on v2 or v3. I tried to find where the 'type' field for logs were set, it seems like it's always jenkins, even on v3 jobs18:42
*** vsaienk0 has joined #openstack-infra18:42
clarkbdmsimard: I think logstash rules may set that, we can possibly have the jobs that zuulv3 submits send that info18:44
dmsimardclarkb: I looked in gearman log processors, in the new submit roles/modules, etc.. couldn't find anywhere18:44
clarkbdmsimard: its in the logstash rules themselves iirc /me finds a link18:45
dmsimardclarkb: do you know where the logstash rules are ? system-config ?18:45
mnaserwas the ML backed up by any chance?18:45
mnaseri just got a whooole bunch of emails all of a sudden18:46
clarkbdmsimard: is the main set of rules, but the jenkin sstuff is set at
*** Rockyg has quit IRC18:46
dmsimardclarkb: oh man that repo was way too obvious18:47
clarkbmnaser: I haven't noticed that18:48
clarkbmnaser: possible your mx was away for a bit?18:48
mnasermaybe... we use (please don't bite) google apps for our email so18:48
mnaseri dont understand what's going on, it might be an issue at their side, it's emails i've already read.. shown arrived a 2:17 am18:49
mnaserbut headers show them from days ago18:49
clarkbdmsimard: wat we could do is change the type in that input file when we change to v318:50
clarkbdmsimard: and have e-r key off of that18:51
clarkbI'm not sure we actually use the type anywhere for anything important but we should check that first18:51
*** rwsu has quit IRC18:51
dmsimardclarkb: I was thinking something slightly different -- have the type be part of the data we push when we submit to gearman18:52
clarkbdmsimard: ya we could do that too18:52
clarkbdmsimard: I think you can have logstash set a default if there isn't data on the input either18:52
dmsimardclarkb: like here or something
clarkbso jenkins could be the default then we override when we submit the jobs18:52
*** yolanda has joined #openstack-infra18:54
*** dizquierdo has joined #openstack-infra18:54
clarkbend of the day though that may be overthinking it18:57
*** efried_nomnom is now known as efried18:58
rajinirzuul issue: With the latest project-config, the old layout.yaml doesn't work. Is there a simple layout.yaml we can use? Like
clarkbmordred: jeblair the v2 side of 511260 lgtm
clarkb but still not happy about something on v319:01
*** masuberu has quit IRC19:01
clarkbdoes that need more sudo?19:01
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats
pabelangerclarkb: yah, zuul user won't be able to override jenkins19:03
mordredclarkb, pabelanger: it seems like that wants to run as stack or something?19:04
clarkbI think the stack user owns the destination side yes19:05
jeblairso why did this work when run as jenkins?19:05
clarkbsudo chown -R $USER:$USER $cache_dir <- is what the old code did /me looks at new change again19:06
mordredoh - we missed a chown19:07
mordredyay for gating19:07
clarkbya I think you need the chown on the source side possibly19:07
jeblair$cache_dir is different before and after this change19:07
clarkband I think dib may already make them jenkins'd19:07
jeblairi think we were effectively depending on the copy to change the perms for us before19:08
mordredyah - I agree19:08
jeblairokay, so yeah, we just need something new here :)19:08
clarkbya copy was from /opt/cache/files -> /opt/stack/cache/files -> /opt/stack/new/devstack/files19:09
jeblaira chown on cache dir right before our "cp -l"?  is that what you're suggesting?19:09
clarkbwe remoed that first copy which I think must have figured out perms for us19:09
clarkbjeblair: yes19:09
mordredis ...19:09
clarkbchown /opt/cache/files to $USER:$USER19:09
jeblairthis all sounds reasonable to me19:09
mordredwait though ...19:09
mordredwe're copying them to $DEST/devstack/files/ ... permissions and ownership of that dir hasn't changed  ... so are the files in /opt/cache/files not readable?19:10
clarkbmordred: it is possible we have perissions issues on both sides19:11
mordredI don't see any code that would affect write permissions in $DEST/devstack/files/ ... unless that just happens to pre-exist and is owned by jenkins19:11
mordredBUT - I guess we could chown both $DEST/devstack/files/ and /opt/cache/files to $USER:$USER19:12
clarkbI want to say we chown all the repos after all this is done?19:12
clarkbya we chown $BASE to stack after the fact19:12
clarkbs oI think at that point its all owned by zuul19:13
clarkbexcept the cache which is owned by jenkins during the image build19:13
clarkbsudo chown -R $USER:$USER $DEST19:13
clarkbthat makes the devstack repo owned by zuul19:13
clarkband happens before we do cache things so I think its just the one side we need to worry about19:14
*** caphrim007 has joined #openstack-infra19:14
mordredclarkb: ok. shall I push up another patch?19:15
mordredclarkb: why we can't *read* from the cache dir is mind-boggling19:15
clarkbmordred: so I think its actually a write problem19:15
clarkbmordred: due to how hardlinks work19:15
clarkbyou have to update the counter on the inode or something19:16
mordredclarkb: OH --- we don't have write permissions in the cache19:16
mordredthat makes much more sense to my brain hole - thank you19:16
*** inc0 has joined #openstack-infra19:16
clarkbthere is definitley a write to say there is another "copy" of this thing19:16
clarkbI think on the inode record and the original file must control access ot that? this is actually a corner of unix file perms that I'm not super familiar with19:16
*** vsaienk0 has quit IRC19:16
mordredclarkb: same here- ready for meto push up a new PS?19:17
clarkbya I think so19:17
jeblaircp -l /etc/passwd .19:17
jeblaircp: cannot create hard link './passwd' to '/etc/passwd': Operation not permitted19:17
jeblair^ seems to support that theory :)19:17
*** slaweq has joined #openstack-infra19:18
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files
mordredthere we go19:19
clarkbmordred: that should go in the if check19:20
clarkbleft comment on change19:20
jeblairare we running as $USER there?19:20
rajinirclarkb: zuul issue: With the latest project-config, the old layout.yaml doesn't work. Is there a simple layout.yaml we can use? Like
jeblairor are we going to chown it to stack and then try to hardlink it as zuul?19:20
mordredclarkb: doh19:21
clarkbpretty sure it is running as zuul there19:21
clarkbjeblair: ^19:21
clarkbjeblair: since it happens in -wrap and there isn't an explicit sudo19:22
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: stop updating rtfd for whereto
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files
clarkbrajinir: can you be more specific, what version of zuul and how does it not work anymore? we are still using our layout.yaml for example19:22
rajinirclarkb: using this didn't work19:23
clarkbrajinir: how did it fail?19:23
AJaeger and are ready for merge if anybody has time for a quick review19:23
rajinirclarkb: the new queues don't show up. no proper clue in the logs19:25
mordredAJaeger: both look great -  has an issue19:25
clarkbrajinir: can you run the layout checker against it?19:25
jeblairclarkb, mordred: so do we need to chown to zuul, then cp -l, then chown to stack?19:26
rajinirclarkb: with the default layout from the latest project-config, all the queues show up19:26
clarkbjeblair: yes, however the chown to stack happens for us already in devstack-vm-gate.sh19:26
clarkbrajinir: probably the best way to diagnose it is to run the layout checker command against it19:26
rajinirclarkb: is there an example how to run it?19:27
jeblairclarkb: ok19:27
clarkbrajinir: zuul-server -c tools/zuul.conf-sample -l zuul/layout.yaml -t {envdir}/tmp/job-list.txt is how the gate jobs do it. The -t flag is what tells it to validate19:29
jeblairrajinir: can you check the debug log?  if zuul didn't start up properly, there should be something there.19:29
*** jamesdenton has joined #openstack-infra19:29
rajinirclarkb: what is job-list.txt, is that new?19:31
*** mat128 has joined #openstack-infra19:31
clarkbrajinir: its a list of job names to check against, the command help output says it is optional though19:31
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: remove legacy openstackdocstheme jobs
rajinirclarkb: jeblair: all these have changed quiet a bit. But the is not updated19:32
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Use build-openstack-api-ref for openstackdocstheme
*** vhosakot has quit IRC19:33
*** vhosakot has joined #openstack-infra19:34
*** coolsvap has quit IRC19:36
rajinirjebliar: clarkb:  WARNING zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit.                    Can not get account information.19:36
rajinirjebliar: clarkb:  may be gearman issue?19:37
clarkbrajinir: that is mostly expected19:37
clarkb(depends on your version of gerrit but doesn't indicate a problem)19:38
jeblairrajinir: nothing in the master branch of zuul has changed to cause the project-config-example to break19:38
jeblairwe should probably update the case of the label tags, but that wouldn't cause zuul not to start up19:38
jeblairi mean nothing should cause it to break19:39
*** makowals_ has joined #openstack-infra19:39
rajinirclarkb: jebliar: good to know.19:39
jeblairobviously something is broken, but there have been almost no changes to zuul master for months, aside from the case issue.19:39
*** e0ne has joined #openstack-infra19:40
rajinirclarkb: jebliar: gearman is rinning, I tested the connection from jenkins19:43
rajinirjebliar: clarkb:  But the pipeline eques check and silient are not showing up in the zuul web UI19:43
rajinirclarkb: jebliar: status of the gearman server19:44
*** ykarel|afk has quit IRC19:50
clarkbI've got to go grab lunch now back in a bit19:50
*** Apoorva_ has joined #openstack-infra19:58
*** harlowja has quit IRC20:00
openstackgerritBenjamin Navaro proposed openstack-infra/project-config master: Online inap-mtl01 region
*** Apoorva has quit IRC20:02
*** ociuhandu has quit IRC20:02
benj_Fyi. The inap-mtl01 region is back online. Hence this review.20:04
rajinirclarkb>: jebliar: Solved it.  function single_use_node is missing in the, adding it resolved the issue20:05
*** apetrich has quit IRC20:07
*** sree has joined #openstack-infra20:07
*** edmondsw_ has joined #openstack-infra20:08
pabelangerbenj_: great20:08
*** edmondsw has quit IRC20:08
*** apetrich has joined #openstack-infra20:10
*** edmondsw_ is now known as edmondsw20:12
*** sree has quit IRC20:12
pabelangerclarkb: mordred: ^511328 bring inap backonline, if you want to =#20:12
*** gildub has joined #openstack-infra20:13
rajinirclarkb>: jebliar: thanks for the tips20:14
dansmithare we mid-transition now?20:15
dansmiththe last I saw this morning made it sound like it was on hold until further notice, but I just saw a job with two POST_FAILURE jobs...20:15
pabelangerdansmith: no, we haven't started as of yet20:15
eumel8pabelanger: something news from the mirror front?20:15
pabelangereumel8: not yet, sadly20:16
jeblairdansmith: link to failures?20:16
*** cody-somerville has quit IRC20:16
dansmiththat was reported as POST_FAILURE, but it seems fine in the console log20:16
mordredjeblair: has tw POST_FAILURES from v3 - I'm just now starting to look20:17
jeblairdansmith: that one looks like a v2 job20:17
dansmithit is20:17
dansmiththat's why I was asking if we were cutting over and things running in v2 should be expecting to fail20:17
pabelangerwe are getting close on filling /srv/static/log:
eumel8pabelanger: thx, then I'll schedule the rechecks for tomorrow20:18
pabelangermaybe we should start a purne just to be safe20:18
mordredfailed: No space left on device (28)20:18
mordredpabelanger: we'r enot just close -we're getting errors20:18
dansmithmordred: where did you see that?20:18
jeblairwell, there we go.  that will affect both.20:18
pabelangermordred: eep20:18
mordreddansmith: debug logs on ze04.openstack.org20:18
dansmithoh sorry, thought that was related to my thing20:19
jeblairdansmith: very likely it is20:19
mordreddansmith: well- if we're having space issues on logs.o.o - that would totally cause POST_FAILURES20:19
dansmithah okay20:19
jeblairprobably failed to copy something (though obviously copied some stuff)20:20
mordredpabelanger: did you start a prune?20:20
dansmithwas trying to figure out where in my logs I might've seen a failure20:20
pabelangermordred: I have not20:20
mordred/dev/mapper/main-logs          12T   12T  713G  95% /srv/static/logs20:20
pabelangerwe look to have 5% left,20:20
jeblairthat will be the root allocation20:20
jeblairso we're at 0 space available for non-root users20:20
* dansmith runs /nick nagios20:20
mordreddansmith: you're the best nagios we know :)20:21
jeblairi don't think we need any root space on that partition.  bit of an oversight there20:21
jeblairinfra-root: any objections to me freeing the root reserved blocks on the logs partition?20:21
mordredjeblair: nope. none from me20:21
pabelangernone from me20:22
jeblairwe should *also* prune, because that's not going to be enough or last very long20:22
fungiokay, i have returned20:22
fungi~100 lines of scrollback... i'll be with you all in a moment ;)20:22
mordredinfra-root: /usr/local/sbin/ is running - and has been since Sept 3020:23
*** slaweq has quit IRC20:24
jeblairwow, erm, apparently we already don't have reserved blocks on that filesystem: Reserved block count:     020:25
*** armax has quit IRC20:25
jeblairwe may have hit the inode limit slightly early20:25
jeblair/dev/mapper/main-logs        805306368 805305720       648  100% /srv/static/logs20:25
jeblair(on the plus side -- wow we have tuned those really close :)20:25
clarkball that /etc logging20:25
mtreinishhmm, wonder if the missing console log: is related to that?20:26
mordredmtreinish: certainly could be20:27
openstackgerritAnup Navare proposed openstack-infra/devstack-gate master: [Test] DNM Checking if tinyIPA builds with py3
mordredman - we're having ALL of the fun issues today20:27
mordredalso - running double jobs isn't just stressful for our node quota - tunrs out it'stwice the logging ...20:27
dmsimardwho would've thought :(20:28
jeblairwell, 1.25x the jobs, but yeah20:28
*** andreas_s has joined #openstack-infra20:30
*** slaweq has joined #openstack-infra20:32
fungi1.25x the job volume when you were over 80% used to start with is, well... yeah i guess we should have seen this coming20:33
mordredjeblair: fwiw, I'm not doing anything on that server because you're on and doing things and double-rooting is likely counter-productive, so please let me know if I can make the help directly20:33
jeblairmordred: oh i'm not doing anything20:33
mordredjeblair: darn. I don't have any excuses to not do anything20:34
jeblairthere is apparently nothing to do regarding reserved blocks20:34
fungidf -h20:34
* fungi grumbles at wrong window20:34
jeblairmaybe we can increase the inode count though?20:35
*** andreas_s has quit IRC20:35
pabelangerokay, I think AFS mirror for ubuntu is ready20:35
fungiafaik that involves a complete reformat20:35
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config master: Revert "neutron: Make grenade-neutron-dvr-multinode job non-voting"
pabelangerjust checking another thing and will be able to release20:35
jeblairfungi: i think *something* about that changed in ext4, but i don't recall details20:35
mordredshould we stop the current cleaner cron and restart it it with a more aggressive mtime ?20:35
fungijeblair: yeah, i'm currently pouring over docs to find out20:36
jeblairmordred: is it still doing things, or is it stuck?20:36
mordredjeblair: tough to say - it's a single find command20:36
jlvillalWith zuul v3 and in project job definitions. I see a proposed patch that uses a zuul.d/ directory. Will that work?20:36
mordredjlvillal: yup20:36
jlvillalAlso, can we use a different directory name then playbook/ ?20:36
jlvillalCan we put the playbooks under zuul.d/ ?20:36
jeblairmordred: yes, the find command is proceeding20:36
jlvillalmordred: thanks20:36
mordredjlvillal: don't put paybooks in zuul.d20:36
* jlvillal slightly sad about that20:37
mordredjlvillal: zuul will try to read them as config :)20:37
jlvillalmordred: Can we put them in zuul_playbooks/ ?20:37
mordredjlvillal: you can put themanywhere else you want - the path is explicitin the run: pre-run: post-run: in the job configs20:37
jlvillalOr does it have to be playbooks/20:37
mordredjlvillal: just not in zuul.d - zuul.d is special20:37
jlvillalmordred: Thanks!20:37
dmsimardmordred: roles might need to be in roles/, though ?20:37
mordredjlvillal: roles must be in roles20:38
dmsimardI can see how/why users might want to centralize zuul/ci things in a single directory though20:38
jlvillalmordred: Uh, as in a top level roles/ directory ?20:38
*** priteau has quit IRC20:38
jlvillalOr as in playbooks/roles/ zuul_playbooks/roles ?20:38
toskyyep, I asked the same during the PTG20:39
toskyI understand that zuul.d is special, but it's just a convention20:39
toskyI mean, technically nothing prevents for having a zuul directory with zuul.d for jobs, and then playbooks and roles20:39
jlvillaltosky: Yep to what?20:39
pabelangerclarkb: I've started vos release mirror.ubuntu20:39
mordredjlvillal: roles is special because it has to do with how zuul checks out and puts roles into the role path for jobs to share roles across repos20:39
*** dprince has quit IRC20:40
toskyjlvillal: yep to "people may want to centralize all zuul stuff"20:40
mnaserby the way do we have to manually specify the name of the repo itself inside roles .. i think that was an issue i ran into a few days ago20:40
fungijeblair: the manpage for mkfs.ext4 seems to be shared for ext2 and ext3 but still claims the inode ratio can't be adjusted after the filesystem is created20:40
dmsimardmnaser: the name of the repo itself ?20:40
mordredmnaser: no - you should not need to specify the nameof the repo inside of roles20:40
toskymordred: but is there a reason why role must be top level? Couldn't zuul check it from, say zuul/roles instead of roles?20:40
jlvillalmordred: Okay. So does that mean roles need to be in a top-level roles/ directory in the project? Or is it stored in zuul.d/? Or somewhere else?20:40
mnaserex: if my playbook is playbooks/test.yaml, then it would look for roles in playbooks/roles/<xyz> but not roles/<xyz>20:41
mnaserlet me show an example of a failure i had20:41
mnaserah you know what, nevermind, i was running ansible-playbook adhoc and it wasnt zuul calling it (little "compatibility"/"migration" layer while we have zuulv2)20:41
dmsimardmnaser: sounds like a default ansible behavior to seek out roles nested under the directory in which the playbook is20:41
mordredmnaser: right. that's a diffrent thing20:41
dmsimardwell there you go20:41
jlvillalThe patch I am looking at doesn't have roles, so maybe I shouldn't worry about it :)20:41
jeblairjlvillal, mordred, dmsimard: roles only need to be in roles/ if you are sharing them with other repos20:41
fungijeblair: mordred: we should move discussion about the logserver outage to the incident channel since it's hard to wade through the ansible discussions in here at the moment20:41
mnaseryeah, my bad, just kinda clicked as i was talking out loud20:41
mordredjlvillal: nothing can go in zuul.d except zuul config20:42
jeblairfungi: agreed20:42
mordredbutyes - what jeblair just said - if the roles are not intended to be shared with other repos, they can totally go adjacent to your playbooks20:42
jlvillaljeblair: mordred: thanks20:42
mnaser <== pattern for migrating really big jobs imho20:42
pabelangerReleased volume mirror.ubuntu successfully20:43
toskyjeblair: so there is no way to change zuul to have shared roles, by default under another special directory?20:43
jeblairtosky: no -- we're trying not to add anything to the standard ansible ways of doing things.  most ansible roles are either a single repo, or have a roles/ directory at the top level.20:44
toskyjeblair: but openstack repositories are not only ansible repositores; I disagree with this choice20:45
toskybut that's as much as I can say and do20:46
pabelangerfungi: I'm not sure why 511328 wasn't enqueued into gate20:47
pabelangeroh, maybe -1 from zuul20:48
jeblairtosky: if you have no intention of sharing the ansible roles you write with anyone outside of your repository, please put them wherever you want!20:48
pabelangerclarkb: sure we add nodepool.o.o to emergency and manually apply 511328 to bring inap-mtl01 back online?20:49
toskyjeblair: sure, so another repository for the tests for each component20:49
jeblairthat's what i was trying to say earlier -- there is no requirement that roles be under roles/ -- unless they are to be shared outside of the repo20:49
clarkbpabelanger: maybe make sure logs is happy first?20:49
*** kgiusti has left #openstack-infra20:49
clarkbotherwise its just more pressure on that fs20:49
pabelangerclarkb: ya, I _think_ mordred is running clean up?20:50
*** jkilpatr has quit IRC20:50
mordredpabelanger, clarkb: we've shifted to incident channel20:50
jeblairtosky: consider though that if done carefully, roles that you write to aid in zuul testing your project may be useful to people outside the context of zuul.  in that case, it may be appropriate to have them in a top level roles/ directory.20:53
jeblair(even if the project itself isn't otherwise strongly ansible focused.  think of it like having a docker or vagrant file, etc...)20:54
*** thorst_ has quit IRC20:55
toskyjeblair: but then they are probably candidate to be moved somewhere else20:55
toskyin a really native ansible (maybe galaxy) way20:55
*** ccamacho has quit IRC20:55
jeblairtosky: possibly so, especially for larger multi-repo projects20:55
jeblairtosky: well, this is all designed to be compatible with galaxy20:55
jeblair(with the added benefit that if the repo is hosted in a system zuul knows about, you can use depends-on)20:56
*** thorst has joined #openstack-infra20:58
*** eumel8 has quit IRC20:59
*** trown is now known as trown|outtypewww20:59
*** makowals_ has quit IRC21:00
openstackgerritPino de Candia proposed openstack-infra/project-config master: Adding new project Tatu (SSH as a Service).
*** dprince has joined #openstack-infra21:02
*** dprince has quit IRC21:02
*** esberglu has quit IRC21:03
*** thorst has quit IRC21:03
openstackgerritPino de Candia proposed openstack-infra/project-config master: Adding new project Tatu (SSH as a Service).
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats
*** dprince has joined #openstack-infra21:06
openstackgerritAdam Coldrick proposed openstack-infra/storyboard master: Add documentation on manually upgrading a StoryBoard instance
*** edmondsw has quit IRC21:07
*** harlowja has joined #openstack-infra21:07
openstackgerritPino de Candia proposed openstack-infra/project-config master: Adding new project Tatu (SSH as a Service).
openstackgerritPino de Candia proposed openstack-infra/project-config master: Add Zuul merge-check template.
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Stop mirroring openSUSE 42.2
*** florianf has quit IRC21:12
pabelangerianw: clarkb: ^mind reviews, and approves. I've manually applied that to mirror-update.o.o already21:13
clarkbpabelanger: there is already a change for that21:13
clarkbfrom dirk iirc21:13
pabelangerclarkb: that is the change, I just removed the rm -rf21:13
clarkboh got it21:13
*** jtomasek has quit IRC21:14
clarkbI've just finished eating some bún will review shortly21:14
*** edmondsw has joined #openstack-infra21:14
jeblairclarkb: eating some what?21:15
jeblairmm that looks tasty21:16
clarkbits excellent stuff21:16
SamYaplei would eat that21:16
pabelangernow I want that21:16
mordredyah. me too21:16
mordredI'm very sad that I'm not eating that rightnow21:16
efriedmtreinish I'm about to restack - what's the status of the glance-filling-up-/tmp thing?  Want me to try anything special?21:17
*** dprince has quit IRC21:19
*** edmondsw has quit IRC21:19
clarkbpabelanger: did vos release complete or is that what precipitated needing to clean out suse 42.2?21:19
*** slaweq has quit IRC21:20
pabelangerclarkb: for ubuntu mirror, ya. I've also done it on opensuse mirror too21:22
pabelangerclarkb: I'm trying to validate if fallback-bindep is working again21:22
*** slaweq has joined #openstack-infra21:22
mtreinishefried: you need to pull those 2 patches to fix it. It's blocked on glance reviews (no +2s on it at all)21:23
efriedmtreinish Roger wilco, pulling now.21:23
rajinirJenkins fails to launch the slaves:  the image built by the disk-image-builder is missing java. Any idea of how to add that element?21:23
pabelangerclarkb: okay, looks to have worked21:24
pabelangerwhich appears to be using bindep-fallback.txt21:24
clarkbv2 looks happy with 511260, I don't think we have any logs on the v3 side yet due to the log server fun21:24
*** slaweq has quit IRC21:27
pabelangerokay, I have to step away to eat. I'll return shortly21:27
*** armax has joined #openstack-infra21:28
*** jkilpatr has joined #openstack-infra21:28
*** mat128 has quit IRC21:28
*** cody-somerville has joined #openstack-infra21:29
*** Apoorva_ has quit IRC21:29
*** Apoorva has joined #openstack-infra21:30
*** ociuhandu has joined #openstack-infra21:30
*** e0ne has quit IRC21:31
*** wolverineav has quit IRC21:31
*** esberglu has joined #openstack-infra21:34
*** ldnunes has quit IRC21:37
*** hashar has quit IRC21:37
clarkbfungi: ianw pabelanger I've started to put info on
dmsimardyou guys have fun in sydney, I'm not going, it's too damn far :(21:39
clarkbdmsimard: can confirm it is not a short trip. Really makes you appreciate how far people in australia have to travel for anything21:40
*** thorst has joined #openstack-infra21:40
dmsimardand the jet lag, oh boy the jet lag21:40
dmsimardI've seen people with worse expected flights but for me it'd be a minimum of 24h of flight/connecting and it's +14hrs there (5:42PM here, 8:42AM there). I would be totally messed up.21:42
dmsimardGoing to Tokyo was fine, coming back took 3 weeks to recover21:43
tosky(or -10hrs?)21:44
jeblairmy favorite part is that going there i put a day on deposit, but it's returned to me when i come back.21:45
mnaserim considering extending my stay and discovering australia21:45
mnasermight as well if im down there21:45
* mordred is going early - will be vacationing for a bit in new zealand, then going to scuba the great barrier reef ... I shoujld be quite happy by the time I make it to sydney21:46
jlvillalAny issue with putting playbooks in "zuul/playbooks/", not "zuul.d/"   Or is "zuul/" also a special directory?21:47
inc0isn't great barrier reef dead?21:47
clarkbinc0: I think there are portions of it that are unhappy but it is quite massive.21:47
*** eharney has quit IRC21:47
clarkbalso apparently certain species are thriving in the changing conditions21:47
inc0that'd be good, I've heard some grim scenarios about i21:48
mnasermaybe mordred can tell us how it looks like at the summit21:49
*** ldnunes has joined #openstack-infra21:49
clarkbI tried to scheme up a way I could visit my parents on my way to sydney21:50
clarkbbut it makes the plane tickets about 3x more and I don't really want to fly air niugini21:50
clarkbor air nauru21:50
jlvillalclarkb: Are your parents retired and doing low-cost living?21:51
* jlvillal dreams of that :)21:51
clarkbjlvillal: they live in micronesia (I grew up there) and happen to be semi retired21:52
pabelangerjlvillal: ansible playbooks can go into any directory, you point to them in pre-run, run, post-run. However, zuul jobs need to be zuul.yaml / .zuul.yaml / zuul.d / .zuul.d21:52
jlvillalclarkb: Ah cool. At one time I was thinking about trying to get a job on the Marshall Islands. But it didn't happen.21:52
jlvillalpabelanger: Thanks. Good to know "zuul/" isn't special.21:53
clarkbjlvillal: neat, on kwaj?21:53
clarkbthe marshalles are really hurting with the whole ocean levels are rising thing21:53
jlvillalclarkb: yeah Kwajalein. I was chatting with a guy on the internet who worked there. I did a TSR program for DOS at the time to help him out.21:53
jlvillalGood old "Spontaneous Assembly" library. Made writing TSR code pretty easy.21:54
clarkbkwaj is an interesting place, I've never gotten off of the plane there but flown through many times21:54
jlvillalclarkb: Nice :)21:54
clarkbebeye is one of the most densely populated places in the world21:55
*** threestrands has joined #openstack-infra21:56
jlvillalclarkb: I didn't know. I do think I would probably get island fever after awhile. Not sure how I would react to living somewhere so small.21:56
dmsimardclarkb: wow TIL about Micronesia, which happens to hold the TLD ".fm". Turns out I live on an island about half it's size. Looks super pretty on pictures :)21:56
dmsimardoh, nevermind the size, wolfram alpha and wikipedia disagree21:57
*** jkilpatr has quit IRC21:57
*** jkilpatr has joined #openstack-infra21:57
clarkbdmsimard: the island my parents live on is about 80km in circumference21:58
jlvillaldmsimard: I don't think any one island is very big. Though you make me curious which island you live on :) I lived on an island before, Great Britain, but it was pretty big.21:58
dmsimardjlvillal: Montreal :)21:58
jlvillalAny one island in Micronesia that is.21:58
clarkbjlvillal: ya I was super ready to get off the island by the time I was done with high school21:58
dmsimardjlvillal: we have like 13 bridges and tunnels to go in an out of the island, it's a big city :)22:00
jlvillaldmsimard: Ah, an island in between rivers :)22:00
*** ijw has quit IRC22:00
jlvillaldmsimard: I was thinking in the ocean.22:00
dmsimardoh, yeah, nothing like out in the middle of the pacific ocena22:00
*** ijw has joined #openstack-infra22:00
*** cody-somerville has quit IRC22:01
dmsimardLiving on one of those islands must be special and sounds like where I'd end up retiring one day. Away from all the craziness of the cities :D22:01
bkeroI'd like a place on a tropical island22:01
bkeroA little bungalow retreat when it's cold and awful in Portland.22:01
bkeroSome satellite internet, solar panels, rainwater catchment, and fishing pole22:02
jlvillalbkero: We have sunshine today here in the Portland area.22:03
* jlvillal waves from Hillsboro22:03
fungimy island is only ~10mi offshore, and we have two bridges to the mainland. i call it a "starter island"22:03
*** jaypipes has quit IRC22:03
fungimaybe in a few years i'll graduate to one you can only reach by air and sea22:04
*** claudiub|2 has quit IRC22:04
bkerojlvillal: Yeah, I saw a few glimpses22:08
*** rlandy has quit IRC22:08
mnaseri remember reading about this island which is the farthest point/island from land ever22:11
*** baoli has quit IRC22:11
mnasersits on an active volcano22:11
*** bobh has quit IRC22:11
jlvillalmnaser: Oh yeah. That is a British colony. I saw a video about it.22:12
mnaserapparently only reachable by boat, which goes 6-8 times per year, it has no airport22:12
jlvillalmnaser: Tristan da Cunha22:12
ianwwell instead of walking into my study, i'm going to have to catch a bus to the city.  we all have our pain :)22:12
mnaserjlvillal: yep!22:13
clarkbre 511260 on v3 lgtm22:13
clarkbjeblair: mordred ^ do we want to enqueue that in the gate once logs.o.o is happier?22:13
clarkbI guess we should see where it grabs get-pip.py22:13
fungithat will be a good litmus test22:14
clarkb that looks good too22:14
clarkbit seems to skip the download and sets the file path to the cache location22:14
jeblairclarkb: ya; sounds like as soon as we think it has a snowball's chance of uploading logs, we should gate enqueue it22:15
*** kjackal_ has quit IRC22:15
jeblairclarkb, fungi, mordred: probably we should at least respond to ianw's comment first :)22:15
* clarkb looks22:16
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Collect mistral-ansible execution files from /tmp"
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Collect mistral-ansible execution files from /tmp"
clarkbjeblair: ianw aha re that its what we've done all along so its possible it doesn't work in every case22:16
ianwjeblair: i may be wrong?22:16
clarkbianw: jeblair but I don't think that bit is a regression22:16
clarkbbut also we put all of /opt under the ephemeral drive on rax so at least for us I think it will work22:17
clarkbthird party ci is where we'd have to be most concerned22:17
ianwso previously it was "if ~/cache/files then move it to $BASE/cache/files"22:20
ianwthen symlink $BASE/cache/files to $DEST/devstack/files22:20
jeblairya, i think we control enough of this in infra to know that it's okay (/opt is always a single fs).  but 3rd party ci could be different.  i think that's fairly unlikely though and think it's reasonable for us to merge this under that assumption.22:21
clarkbs/sym/hard/ ?22:21
ianwsorry hard, yep22:21
openstackgerritPino de Candia proposed openstack-infra/project-config master: Add Zuul merge-check template.
mordredinfra-root, EmilienM: what do people think of potentially force-merging to stop the influx of inodes to the log server?22:21
EmilienMmordred: ++22:21
clarkbmordred: its probably a good idea since everything is affected right now22:21
EmilienMthere is no way this patch breaks our CI22:22
EmilienM(more than it is right now :D)22:22
EmilienMmordred: go ahead22:22
openstackgerritMerged openstack-infra/tripleo-ci master: Revert "Collect mistral-ansible execution files from /tmp"
mordredEmilienM, weshay|ruck thanks!22:22
EmilienMmordred: hey can you force merge all my patches in tripleo? lol22:22
openstackgerritMonty Taylor proposed openstack-infra/tripleo-ci master: Stop collecting ephemeral temp dirs
mordredEmilienM: hehe. I promise - you don't actually want that :)22:23
mordredEmilienM, weshay|ruck: ^^ there's a followup22:23
mordredI don't know that it's strictly required - it's possible those .~tmp~ dirs were leftovers from an upload that ran out of inodes22:23
fungiyup, sounds good to me (and already done)22:23
EmilienMmordred: go ahead, merge this one too22:24
mordredBUT - they are certainly not things that are useful22:24
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: remove the collection of /tmp/* from upstream jobs
EmilienMI agree22:24
jeblairlate ++ for force merge :)22:24
EmilienMweshay|ruck: +2 +A on ^22:24
mordredI'm a little more reluctant to force-merge the .~tmp~ change without seeing it work being passed through yaml to rsync22:24
EmilienMmordred: too22:24
EmilienMmordred: ok we can wait on this one22:25
openstackgerritMerged openstack-infra/tripleo-ci master: remove the collection of /tmp/* from upstream jobs
*** edmondsw has joined #openstack-infra22:25
fungimordred: want to go ahead and do a separate find -name to remove existing ones?22:26
mordredfungi: not a bad idea - or we could update the curent script and restart it - which do you think is better?22:26
fungii would leave it running so it doesn't lose its place, and start a second pass targeting the tmpfiles specifically22:27
*** wolverineav has joined #openstack-infra22:27
fungiwhich should in theory complete much more quickly22:27
fungiif they're dirs we can match on -type d and then use deltree or whatever option that is22:28
* fungi checks22:28
mordredyah - they're all dirs22:28
*** dizquierdo has quit IRC22:29
clarkbEmilienM: mordred there was also a place where we were copying all of /etc from the test node22:29
clarkbI don't have any links of that handy though22:30
*** edmondsw has quit IRC22:30
mordredclarkb: /etc looks ore specific now I think22:30
clarkbmordred: it is in some places not others22:30
clarkbI'm trying to find examples22:30
fungii guess i was thinking of -execdir rm -rf {} \;22:32
fungior +22:32
fungithe latter is probably faster22:32
openstackgerritPino de Candia proposed openstack-infra/project-config master: Adding new project Tatu (SSH as a Service).
openstackgerritPino de Candia proposed openstack-infra/project-config master: Add Zuul merge-check template.
clarkb its copying things like shadow22:33
mordredfungi: yah - I think we'll need to execdir22:33
*** felipemonteiro_ has quit IRC22:34
*** felipemonteiro_ has joined #openstack-infra22:34
fungifor some reason i thought there was a directory-specific equivalent of -delete22:34
mordredfind -O3 /srv/static/logs/ -depth -not -name robots.txt -not -name lost+found \22:34
mordred        -not -wholename /srv/static/logs/help/\*  \22:34
mordred    \( -type d -name .~tmp. -execdir rm -rf {} \; \) \)22:34
pabelangerclarkb: ya, next up in /etc. I'd like to see that get blocked, and back only needed files22:34
mordredfungi: I think that's one too many ()'s ...22:35
clarkboverall though it does look a lot better than before22:35
mordredhonestly - does this one need any ()'s ?22:35
mordredfind -O3 /srv/static/logs/ -depth -not -name robots.txt -not -name lost+found \22:35
mordred        -not -wholename /srv/static/logs/help/\*  \22:36
mordred    \( -type d -name .~tmp. -execdir rm -rf {} \;  \)22:36
mordredhow's that ??22:36
fungimordred: you can skip most of what's there if you're matching on a specific name22:36
mordredoh - that's a realy good point22:36
mordredfungi: find logs -type d -name .~tmp~ -execdir rm -rf {} \;22:37
fungiso something like `find /srv/static/logs/ -name '.~tmp.' -execdir rm -rf {} +22:37
fungioh, i missed -type d22:37
fungibut yeah22:37
mordredfungi: what does the + do?22:37
fungithink of + like the xargs version of \;22:38
clarkbmordred: I think it does like xargs and lines up a bunch of them22:38
* mordred tries it out in a single directory to make sure22:38
fungii would also quote that filename22:38
fungiso that ~ doesn't get a surprise shell-expansion22:38
*** priteau has joined #openstack-infra22:39
mordredfungi: it's deleting things - but I'm also getting a lot of file not found error22:40
fungimordred: likely because of deleting the parents of things it found22:40
fungias long as the file not found errors are for things you expected it to match, then i would just ignore them22:41
mordredfungi: no - they're still there ...22:41
*** tpsilva has quit IRC22:41
fungioh, hrm22:41
SamYaplesounds like a quoting issue22:41
fungimordred: what does one of the directory names look like?22:41
fungiwondering if you're trying to regex instead of glob22:41
mordredfind: `logs/undercloud/tmp/ansible/lib/python2.7/site-packages/pbr/.~tmp~': No such file or directory22:41
mordredthe find is matching them properly22:41
ianwclarkb / jeblair: ok, i think i talked myself into 511260 being ok22:42
mordredsudo find logs -type d -name .~tmp~22:42
SamYaplemordred: can you run a rm on that directly?22:42
mordreddoes the right thing22:42
SamYapleor does something need escaping22:42
jamespageseem to be hitting alot of installability issues with check jobs on reviews for charms -
fungimordred: okay, try quoting "{}" but that may mean we need to switch to \; if it still goes wierd22:42
ianwi'd like to update the commit msg but don't want to kill ci on it22:42
clarkbianw: I think we'll likely enqueue directly to the gate so updating commit message is likely fine22:43
jamespage"libcurl4-gnutls-dev : Depends: libcurl3-gnutls (= 7.47.0-1ubuntu2.2) but 7.47.0-1ubuntu2.3 is to be installed"22:43
clarkbmordred: jeblair ^ that sound right?22:43
clarkbjamespage: our ubuntu mirror ended up out of sync, it was recently resynced and fixed22:43
fungijamespage: should be fixed in the last hour or so. our xenial mirror was stale and had older packages than our latest images22:43
jamespagegreat - I guessed it was probably something like that - I'll get folk to submite rechecks22:43
clarkbjamespage: sorry for the trouble22:44
*** priteau has quit IRC22:44
pabelangerclarkb: spacex launch in 8mins22:44
jamespageclarkb: no worries :)22:44
clarkbpabelanger: ooh22:45
jeblairclarkb: wfm22:45
mordredfungi: well - no combinations made the errors go away - but also it totally deleted the correct files22:45
pabelangerclarkb: I missed the launch on Monday22:45
fungigood thing this one's not going up from the mid-atlantic regional spaceport. we've got rain for the past hour or so22:47
mordredinfra-root: I have started a new screen session called 'remove-tmp-files'22:47
mordredand i'm going to run:22:48
mordredfind -O3 /srv/static/logs/ -type d -name .~tmp. -execdir rm -rf {} +22:48
mordredunless anyone screams22:48
*** rbrndt has quit IRC22:48
*** rbrndt has joined #openstack-infra22:48
*** edmondsw has joined #openstack-infra22:48
*** rbrndt has quit IRC22:48
fungimordred: your example earlier had .~tmp~22:49
mordredfungi: yes - thank you22:49
mordredfind -O3 /srv/static/logs/ -type d -name '.~tmp~' -execdir rm -rf {} +22:49
clarkbpabelanger: this is a recycled first stage too right?22:49
mordredis what I'm actually going to run22:49
clarkbmordred: no screaming here22:49
fungimordred: lgtm22:50
mordredk. it's running22:50
fungimordred: the -O3 is probably not useful in this case (due to no nested logic) but shouldn't hurt22:51
mordredok. while those run I'm going to go find food22:52
mordrednot that this isn't exciting and all22:52
*** jcoufal has quit IRC22:53
*** edmondsw has quit IRC22:54
*** wolverineav has quit IRC22:54
*** wolverineav has joined #openstack-infra22:55
*** wolverineav has quit IRC22:59
clarkbianw: I think you can go ahead and update the commit message on 51126023:00
clarkbthen we'll enqueue it once mordreds find has given us breathing room23:00
*** priteau has joined #openstack-infra23:01
openstackgerritIan Wienand proposed openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files
ianwclarkb / mordred: ^ ok, i am convinced that is how we got into this situation23:02
pabelangerclarkb: yah23:02
ianwi think after this, we could probably remove the ~/cache/files symlink all together, since the "historical" reason for having it is, i think, just this copy23:02
pabelangerclarkb: not sure which mission it flew before23:02
*** xarses has quit IRC23:06
*** priteau has quit IRC23:06
*** ldnunes has quit IRC23:08
*** ldnunes has joined #openstack-infra23:09
*** hongbin has quit IRC23:11
*** felipemonteiro_ has quit IRC23:12
*** masber has joined #openstack-infra23:14
*** ttx has quit IRC23:20
*** ttx has joined #openstack-infra23:20
*** caphrim007 has quit IRC23:25
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance
*** yamamoto has joined #openstack-infra23:28
clarkbpabelanger: fyi
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance
pabelangerclarkb: python package broken?23:32
*** thorst has quit IRC23:33
clarkbwhat is keyring for23:35
pabelangerI don't know, I'd have to look23:36
clarkbapparently os-loganalyze deps on it23:37
clarkbI wonder if that was for the swift stuff (whcih we stopped using)23:37
tonybSorry to come late to the party but is the gnutls issues fixed?
clarkbtonyb: yes it should be fixed at this point, but now battling the logs server is out of inodes issue23:43
*** tosky has quit IRC23:43
clarkbtonyb: our ubuntu mirror got out of sync, pabelanger resynced it and made it happy again about 2 hours ago at this point23:43
tonybclarkb: Thanks.  Good luck with the inode issue.23:43
clarkbwhen it rains it pours :)23:43
pabelangerYah, it _should_ be fixed23:44
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use weakref for change cache
*** mriedem has quit IRC23:48
*** vhosakot has quit IRC23:53
*** vhosakot has joined #openstack-infra23:53
*** ldnunes has quit IRC23:56
smcginnisThe logs on here look like another out of sync issue:
*** vhosakot has quit IRC23:58

Generated by 2.15.3 by Marius Gedminas - find it at!