| asselin | fungi, mattoliverau it is a 'recent' change...sometime in the past few hours because I was getting past that before....so after that step backwards, I'm now back to the issue I was resolving. Thanks. | 00:04 |
|---|---|---|
| mattoliverau | asselin: yup, I think the puppetlabs repo was updated a few hours ago.. causing this headache | 00:04 |
| *** SumitNaiksatam has quit IRC | 00:08 | |
| *** Steap has quit IRC | 00:09 | |
| clarkb | fungi: https://jenkins01.openstack.org/job/gate-nova-docker-requirements/3/console any idea why that still fails? (the change being tested adds the newlines) | 00:10 |
| *** Steap has joined #openstack-infra | 00:10 | |
| *** pcrews has quit IRC | 00:10 | |
| fungi | clarkb: https://review.openstack.org/84590 | 00:10 |
| clarkb | fungi: I am not sure I inderstand after reading that change | 00:11 |
| clarkb | oh wait, we check before and after? | 00:11 |
| fungi | clarkb: there's a function to read requirements files. it gets used to read the old and new reqs so it can compare them, but it performs a style check both times | 00:12 |
| clarkb | is that to get a diff against whcih we check for matching global requirements? | 00:12 |
| clarkb | fungi: aha! thanks | 00:12 |
| fungi | i already updated the bug with a rudimentary explanation of that | 00:12 |
| fungi | but was probably not terribly clear | 00:12 |
| clarkb | fungi: the update was what prompted my nova-docker patch | 00:12 |
| fungi | ahh | 00:13 |
| fungi | clarkb: part of the problem is that cookiecutter likes to create files without trailing newlines | 00:13 |
| fungi | or at least this is what i've heard | 00:13 |
| clarkb | huh | 00:14 |
| fungi | i vaguely remember lifeless saying it hit a tripleo and/or gantt projects which were cookiecutter'd | 00:14 |
| *** dstanek has joined #openstack-infra | 00:14 | |
| *** dcramer_ has joined #openstack-infra | 00:14 | |
| *** sdake_1 has quit IRC | 00:15 | |
| *** zns has quit IRC | 00:15 | |
| lifeless | so yeah I opened a bug that the check whinged on stuff pip accepts | 00:15 |
| lifeless | but it was closed WONTFIX | 00:15 |
| lifeless | and yes cookiecutter makes every new project hit tis | 00:16 |
| clarkb | :( http://logs.openstack.org/90/84590/2/check/gate-config-layout/e4e6cec/console.html I am rechecking that change | 00:16 |
| * fungi shakes his fist at rackspace dns blacklists | 00:17 | |
| fungi | lifeless: the trailing newline thing? was it affecting hacking jobs on python files too i guess, not just requirements checks? | 00:18 |
| dhellmann | fungi: we hit that on oslotest, too, iirc | 00:19 |
| fungi | dhellmann: too awesome | 00:19 |
| lifeless | fungi: everything | 00:19 |
| fungi | well, at least that will hopefully fix the requirements check | 00:19 |
| lifeless | fungi: its about as annoying as it can possibly be | 00:19 |
| dhellmann | fungi: I thought it was our template, to be honest | 00:19 |
| lifeless | dhellmann: no, cookiecutter trims the strings. | 00:20 |
| dhellmann | lifeless: and audrey said no to fixing it? | 00:20 |
| lifeless | no idea | 00:20 |
| dhellmann | oh, I thought you said closed wontfix, but maybe that was something else | 00:20 |
| *** thuc has quit IRC | 00:20 | |
| lifeless | the bug I opened about the infra check on requirements files | 00:20 |
| dhellmann | lifeless: ah | 00:21 |
| marun | clarkb: still looking for some love for adding neutron's non-voting functional job to the devstack check queue | 00:21 |
| fungi | lifeless: i thought that was a bug against hacking, not the requirements job | 00:21 |
| jeblair | https://bugs.launchpad.net/openstack-ci/+bug/1286695 | 00:21 |
| uvirtbot | Launchpad bug 1286695 in openstack-ci "requirements newline" [Undecided,Won't fix] | 00:21 |
| *** thuc has joined #openstack-infra | 00:21 | |
| marun | https://review.openstack.org/#/c/82631/ | 00:21 |
| fungi | ahh, so it was! | 00:21 |
| dhellmann | I don't see cookiecutter explicitly chopping the file, so maybe it's the template tool it's using | 00:21 |
| jeblair | lifeless, dhellmann, fungi: yep. i marked that wontfix. i mean, we explicitly wrote that into the check. | 00:22 |
| clarkb | the bug is in cookiecutter not infra | 00:22 |
| fungi | i guess 1300843 is a dupe of 1286695, sort of, except the job is also brokenish | 00:22 |
| jeblair | clarkb: i agree, that's what i wrote in the bug when i closed it | 00:22 |
| lifeless | yep, I agree that cookiecutter is broken | 00:23 |
| clarkb | fungi: yes, I think your fix makes sense | 00:23 |
| fungi | jeblair: well, there is also a bug in infra, which is that the script is enforcing what amounts to a style check on an existing file, rather than just to the proposed version, making the file impossible to fix | 00:23 |
| dhellmann | yes, we had to turn that job off to fix oslo.test's repo | 00:23 |
| *** pcrews has joined #openstack-infra | 00:23 | |
| *** CaptTofu has joined #openstack-infra | 00:23 | |
| fungi | (without temporarily making the job non-voting anyway, and then setting it back after approving the change) | 00:24 |
| dhellmann | this looks relevant: https://github.com/audreyr/cookiecutter/pull/135 | 00:24 |
| clarkb | marun: done | 00:25 |
| marun | clarkb: danke! | 00:25 |
| clarkb | marun: but it needs a second +2 and +A | 00:25 |
| mattoliverau | I'm off to lunch, meeting the wife for some dumplings! bbl | 00:25 |
| marun | fungi: your turn :) https://review.openstack.org/#/c/82631/ | 00:25 |
| *** thuc has quit IRC | 00:26 | |
| *** matsuhashi has joined #openstack-infra | 00:26 | |
| clarkb | fungi: should todays images include jeblair's use google everywhere change assuming we can get them all built? | 00:27 |
| jeblair | that's https://review.openstack.org/#/c/84230/ and i think it should get another +2 | 00:28 |
| fungi | clarkb: good question--i have one i can check | 00:28 |
| fungi | oh, right, it's not approved yet | 00:28 |
| clarkb | oh I thought that had merged too for some reason | 00:28 |
| *** CaptTofu has quit IRC | 00:28 | |
| *** derekh has quit IRC | 00:28 | |
| *** SumitNaiksatam has joined #openstack-infra | 00:28 | |
| fungi | it's approved now | 00:29 |
| *** openstackgerrit has joined #openstack-infra | 00:30 | |
| fungi | also openstackgerrit is back now | 00:30 |
| fungi | marun: approved. looks like sdague gave his blessing on having it as a non-voting check for devstack on an earlier patchset as well | 00:32 |
| marun | fungi: awesome, danke! | 00:33 |
| *** mspreitz has quit IRC | 00:34 | |
| openstackgerrit | A change was merged to openstack-infra/config: Always forward to 8.8.8.8 on test nodes https://review.openstack.org/84230 | 00:34 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Detect neutron net-changes on reconfigure. https://review.openstack.org/84267 | 00:35 |
| *** mspreitz has joined #openstack-infra | 00:39 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add neutron functional job to devstack check queue https://review.openstack.org/82631 | 00:39 |
| *** bhuvan_ has joined #openstack-infra | 00:40 | |
| *** esker has joined #openstack-infra | 00:42 | |
| *** esker has quit IRC | 00:42 | |
| *** bhuvan_ is now known as bhuvan-afk | 00:42 | |
| *** esker has joined #openstack-infra | 00:43 | |
| *** bhuvan has quit IRC | 00:43 | |
| *** bhuvan-afk has quit IRC | 00:47 | |
| *** matsuhashi has quit IRC | 00:47 | |
| *** locke105 has quit IRC | 00:48 | |
| *** mrodden has quit IRC | 00:48 | |
| *** matsuhashi has joined #openstack-infra | 00:48 | |
| *** Ryan_Lane has quit IRC | 00:48 | |
| *** sdake has joined #openstack-infra | 00:48 | |
| *** sdake has quit IRC | 00:48 | |
| *** sdake has joined #openstack-infra | 00:48 | |
| *** mspreitz has quit IRC | 00:50 | |
| *** e0ne has joined #openstack-infra | 00:52 | |
| *** matsuhashi has quit IRC | 00:52 | |
| *** locke105 has joined #openstack-infra | 00:55 | |
| *** e0ne has quit IRC | 00:56 | |
| *** matsuhashi has joined #openstack-infra | 00:57 | |
| *** mrodden has joined #openstack-infra | 00:58 | |
| *** Sukhdev has joined #openstack-infra | 00:58 | |
| fungi | huh... https://jitsi.org/Projects/JitMeet | 01:01 |
| fungi | i wonder if something like that would finally sway the proponents of google-hangtime or whatever | 01:02 |
| pleia2 | might be worth a try, see if it's stable, easy to get going on linux and mac and the bandwidth requirements are reasonable | 01:04 |
| lifeless | we have 12 jobs running atm | 01:04 |
| lifeless | but 19 instances in ACtiVE | 01:04 |
| lifeless | what are the other 7 doing ? | 01:04 |
| lifeless | fungi: ^ ? | 01:04 |
| fungi | lifeless: sneering in the corner? probably still being set up and registered with jenkins would be my guess... checking | 01:05 |
| fungi | lifeless: http://paste.openstack.org/show/74787/ | 01:07 |
| *** jaypipes has quit IRC | 01:08 | |
| *** esker has quit IRC | 01:09 | |
| lifeless | fungi: thanks | 01:09 |
| fungi | my guess would either be in delete state but not deleted yet or in building state because they're being finalized once the instances have booted | 01:10 |
| fungi | that's a lot in delete though--i suspect some may not be real nodes? | 01:11 |
| fungi | a lot of them have been in delete for more than 30 minutes, but most are under an hour | 01:12 |
| *** mrodden has quit IRC | 01:15 | |
| lifeless | running reset-state on the ERROR state VMs | 01:18 |
| *** mfer has joined #openstack-infra | 01:18 | |
| *** zns has joined #openstack-infra | 01:19 | |
| *** yaguang has joined #openstack-infra | 01:19 | |
| fungi | clarkb: you determined that having only a non-voting job in a voting zuul pipeline like in https://review.openstack.org/83761 is fine because the absence of voting job results will cause it to default to a success vote on the change, right? | 01:19 |
| clarkb | fungi: yes | 01:20 |
| clarkb | fungi: watch I will eb wrong about that, but am 99% sure about it | 01:20 |
| lifeless | grr | ee1eda63-b4fd-4b93-9627-56d3f78bb83b | tripleo-precise-tripleo-test-cloud-3381790.slave.openstack.org | ERROR | deleting | Running | default-net=10.0.26.147; tripleo-bm-test=192.168.1.13 | | 01:20 |
| fungi | now i'm just trying to spelunk in d-g to make sure that $BASE/new/heat-templates will actually exist when that job tries to access it | 01:21 |
| fungi | i don't see heat-templates in the projects array in devstack-vm-gate-wrap.sh | 01:22 |
| fungi | probably needs to be added in an envvar in that job, i'm guessing | 01:22 |
| lifeless | hypervisors are still checking in ok | 01:23 |
| lifeless | but we're getting | fault | {"message": "No valid host was found. Exceeded max scheduling attempts 3 for instance 08d9d440-e951-4aa8-9174-066cb4e6e17d", "code": 500, "details": " File \"/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/scheduler/manager.py\", line 140, in run_instance | | 01:23 |
| lifeless | | | legacy_bdm_in_spec) | | 01:24 |
| lifeless | I think I'm going to blame the ram -1 weight | 01:24 |
| *** zns has quit IRC | 01:24 | |
| *** zns has joined #openstack-infra | 01:25 | |
| lifeless | and this is why some vms are failing... http://paste.openstack.org/show/74788/ | 01:27 |
| fungi | yeesh | 01:27 |
| fungi | who wrote that openstack stuff anyway? | 01:28 |
| jogo | I tried pushing up a patch to milestone-proposed and hit a bug | 01:28 |
| fungi | jogo: do go on | 01:28 |
| jogo | http://logs.openstack.org/69/84569/2/check/check-tempest-dsvm-full/efb5a9b/logs/testr_results.html.gz | 01:28 |
| jogo | http://logs.openstack.org/69/84569/2/check/check-tempest-dsvm-full/8c6e1aa/logs/testr_results.html.gz | 01:29 |
| jogo | both have same swift failure | 01:29 |
| *** yongli has joined #openstack-infra | 01:29 | |
| fungi | i assume you're merely expressing surprise that we're about to release software with a bug? | 01:30 |
| jogo | fungi: I am confused to why that is only hitting milestone-proposed and not trunk | 01:31 |
| fungi | or do you think it's something related to branch selection in devstack-gate? | 01:31 |
| jogo | as there is a neutron patch I need to land on milestone-proposed | 01:31 |
| jogo | fungi: I haven't dug into the root cause yet, was partially wondering if that has been seen before | 01:31 |
| *** thuc has joined #openstack-infra | 01:32 | |
| *** zhiyan_ is now known as zhiyan | 01:32 | |
| fungi | not that i'm aware of... i'm pouring over http://logs.openstack.org/69/84569/2/check/check-tempest-dsvm-full/8c6e1aa/logs/devstack-gate-setup-workspace-new.txt.gz now just to make sure it dtrt | 01:32 |
| fungi | looks like it's successfully checking out milestone-proposed for projects which have that branch, and master for projects which don't | 01:35 |
| clarkb | is it possible a change was made to milestone proposed branches that broke it? | 01:36 |
| clarkb | they are not necessesarily static (as your proposed change indicates) | 01:36 |
| *** thuc has quit IRC | 01:36 | |
| fungi | looks like it checked out the right neutron change too via a zuul ref, so i don't think there's likely any infra issue targeting m-p branch changes specifically | 01:37 |
| fungi | at least branch selection behavior looks like how i expect it to | 01:37 |
| jogo | no milestone proposed patches are passing | 01:38 |
| jogo | https://review.openstack.org/#/q/status:open+branch:milestone-proposed,n,z | 01:38 |
| fungi | or could we have a milestone-proposed hole in one of the branch-specific conditionals in the d-g scripts maybe? | 01:38 |
| fungi | maybe some service or option we normally disabled on master is falling through to a non-master default? | 01:39 |
| lifeless | right finally stabbed them hard enough to clear it up | 01:40 |
| fungi | lifeless: bravo | 01:40 |
| lifeless | 42 instances for nodepool | 01:41 |
| clarkb | jogo: the failure is that that method isn't raising an exception. Instead http 200 is returned and everything is happy | 01:42 |
| lifeless | I'll give it a few minutes then we should see more slaves | 01:42 |
| jogo | yeah maybe the config is wrong? | 01:42 |
| lifeless | fungi: that said I see 6 slaves but 32 active services | 01:43 |
| lifeless | fungi: which is odd... | 01:43 |
| *** saschpe has quit IRC | 01:43 | |
| fungi | lifeless: me too... http://paste.openstack.org/show/74789/ | 01:43 |
| *** saschpe has joined #openstack-infra | 01:44 | |
| clarkb | the time for expiration is Tue Apr 1 23:36:00 UTC 2014, 2014-04-01 23:36:01,365 when the response was logged. | 01:45 |
| lifeless | so thats 24 | 01:45 |
| lifeless | so I guess nodepool will notice they are gone soonish | 01:45 |
| clarkb | jogo: so that may be a legit regression in swift | 01:45 |
| clarkb | jogo: but also possibly in tempest | 01:45 |
| *** mrodden has joined #openstack-infra | 01:46 | |
| lifeless | fungi: nova list --all-tenants | grep cloud | grep delet | wc -l | 01:46 |
| lifeless | 3 | 01:46 |
| *** nosnos has joined #openstack-infra | 01:47 | |
| jogo | how come its only being seen milestone-proposed? | 01:47 |
| clarkb | jogo: swifts milestone proposed looks really old. is it possible this is related? | 01:47 |
| jogo | me checks if that is the case | 01:47 |
| *** changbl has joined #openstack-infra | 01:47 | |
| clarkb | jogo: swift milestone proposed is more than a month old | 01:47 |
| clarkb | maybe there is a fix that should be backported? | 01:48 |
| clarkb | or possibly swift's milestone proposed branch should be killed | 01:48 |
| clarkb | notmyname: ^ ideas? | 01:48 |
| fungi | lifeless: yeah, i think most of these are deletes which nova didn't act on, leaving the instances in a running state (or possibly the instances never existed and the error response was vague such that nodepool didn't think it successfully deleted anything_ | 01:48 |
| fungi | lifeless: i'll try deleting some of them through nodepool myself and see what it does | 01:49 |
| *** dcramer_ has quit IRC | 01:49 | |
| jogo | where is the logic that checks out milestone proposed? | 01:49 |
| lifeless | fungi: thanks | 01:50 |
| clarkb | jogo: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n195 | 01:52 |
| *** e0ne has joined #openstack-infra | 01:52 | |
| jogo | aha! https://review.openstack.org/#/c/79094/ | 01:53 |
| clarkb | jogo: it will check out the zuul ref for all projects that have it, then checkout the same branch as the proposed change in all projects that have it and finally fall bac kt omaster iirc | 01:53 |
| clarkb | oh the comment above that line has a list | 01:53 |
| clarkb | jogo: right so I think the fix here is maybe swift needs to remove its milestone proposed branch | 01:53 |
| clarkb | because it is old | 01:54 |
| clarkb | and the tip of the milestone proposed branch is the 1.13.0 release | 01:54 |
| fungi | clarkb: like from last cycle? | 01:54 |
| clarkb | fungi: well 1.5 months ago | 01:54 |
| fungi | oh, do they use m-p for their out-of-integrated-cycle released? | 01:54 |
| clarkb | since swift does cycles weird | 01:54 |
| jogo | ahh I was looking at the wrong file devstck-gate-setup-workspace | 01:54 |
| clarkb | fungi: yes | 01:54 |
| fungi | releases | 01:54 |
| fungi | got it. yeah, so m-p branches should be deleted as soon as a release is tagged/stable-ized | 01:55 |
| clarkb | yup | 01:55 |
| clarkb | but swift makes this weird | 01:55 |
| clarkb | because swift does haev stable/* branches too | 01:55 |
| lifeless | fungi: ok so 34 active vms, still only 6 jobs running | 01:56 |
| clarkb | not sure what the proper fix is here. It may also be backporting 79094 to milestone-proposed and releasing a point release | 01:56 |
| jogo | so that milestone is from Feb 24th | 01:56 |
| fungi | notmyname: if you're around, is your current milestone-proposed branch okay to be deleted? it seems to be outdated and getting tested with new milestone-proposed branches for integrated release projects | 01:56 |
| jogo | fungi: I think you have it backwards | 01:56 |
| jogo | they cut the milestone-propsed really really early | 01:56 |
| clarkb | jogo: yes, then did a release | 01:56 |
| fungi | lifeless: the node i picked to try to delete is probably going to time out on me--the delete call has been hanging for about 5 minutes, but i'll hopefully have a paste for you in a bit | 01:57 |
| clarkb | jogo: but for some reason didn't get rid of the milestone proposed branch | 01:57 |
| *** dims_ has quit IRC | 01:57 | |
| *** e0ne has quit IRC | 01:57 | |
| jogo | clarkb: on Feb 24th? | 01:57 |
| clarkb | jogo: see https://wiki.openstack.org/wiki/Icehouse_Release_Schedule | 01:57 |
| clarkb | jogo: no I think the last commit in the stablization was created on the 24th. tagging happened march 6th ish | 01:57 |
| lifeless | fungi: I'd really love to be churning through our backlog 5 * faster :) | 01:57 |
| lifeless | 5 trying to delete | 01:58 |
| jogo | anyway I am gonna propse a patch to there stable and see what happens | 01:58 |
| jogo | and notmyname can sort it out | 01:58 |
| *** dims_ has joined #openstack-infra | 01:58 | |
| clarkb | jogo: they don't have a stable for icehouse yet | 01:58 |
| clarkb | which complicates this further | 01:58 |
| clarkb | doing a release without a stable branch is weird | 01:58 |
| jogo | I think that is what milestone-propsed is | 01:58 |
| jogo | they have stable branches for every other release | 01:58 |
| jogo | https://review.openstack.org/84602 | 01:59 |
| jogo | notmyname: ^ | 01:59 |
| jogo | portante: ping^ | 01:59 |
| clarkb | jogo: that is not what milestone proposed is for | 01:59 |
| * jogo is confused | 01:59 | |
| clarkb | milestone proposed is for the work leading up to a release, once you ahve a release you make a real branch | 01:59 |
| jogo | clarkb: I thought the branch gets renamed or the equivalent of that | 02:00 |
| clarkb | jogo: yes | 02:00 |
| clarkb | jogo: but swift didn't/doesn't do that apparentkly | 02:00 |
| jogo | http://git.openstack.org/cgit/openstack/swift/?h=stable%2Fhavana | 02:00 |
| jogo | whgat is that then? | 02:00 |
| clarkb | that is havana, which happened 6 months ago | 02:01 |
| fungi | jogo: that's a stable branch from the last swift release prior to the integrated havana release, i believe | 02:01 |
| *** dcramer_ has joined #openstack-infra | 02:01 | |
| clarkb | jogo: since then they have done ~3 releases | 02:01 |
| fungi | jogo: the most recent swift release is 1.13.0 from four weeks ago | 02:01 |
| jogo | havana? | 02:01 |
| *** wenlock has joined #openstack-infra | 02:01 | |
| jogo | I thought they do both numbers and openstack releases | 02:01 |
| jogo | they just map them up | 02:01 |
| clarkb | jogo: 1.11.0, 1.12.0 and 1.13.0 do not have a stable branch associated with them | 02:02 |
| jogo | clarkb: no but stable/havana does | 02:02 |
| clarkb | so I am not sure how you can map them up | 02:02 |
| fungi | they created the current milestone-proposed branch one week before they tagged 1.13.0 | 02:02 |
| clarkb | jogo: right which is 1.10.0? | 02:02 |
| * clarkb checks | 02:02 | |
| jogo | right because 1.13.0 will turn into icehouse | 02:02 |
| clarkb | yup havana is 1.10.0 | 02:03 |
| jogo | anyway hopefully notmyname or portante or someone can sort this out | 02:03 |
| clarkb | jogo: right so how do you backport a fix to 1.12.0? | 02:03 |
| clarkb | jogo: you can't | 02:03 |
| jogo | clarkb: heh right | 02:03 |
| clarkb | jogo: but it also means the lack of a stable/icehouse for 1.13.0 is currently breaking everyone else | 02:03 |
| jogo | unless they retag ... | 02:03 |
| clarkb | jogo: no you can't do that | 02:03 |
| jogo | anyway we will see what swift folks say about https://review.openstack.org/#/c/84602/ and all that | 02:03 |
| jogo | time to go AFK o/ | 02:03 |
| fungi | not sure whether they expect a 1.14.0 around the icehouse release date or what, but regardless i think the current swift milestone-proposed branch was probably meant to be deleted when 1.13.0 was tagged | 02:04 |
| portante | jogo: not sure I understand what is up | 02:04 |
| portante | thought that fix was already posted to master | 02:04 |
| jogo | portante: what is milestone-proposed in swift for? | 02:04 |
| clarkb | portante: swift milestone proposed is more than a month old | 02:04 |
| clarkb | portante: so any testing of openstack milestone proposed fails against month old swift | 02:05 |
| jogo | portante: it is but we use milestone-proposed branches assuming they are all in prep for icehouse | 02:05 |
| *** yjiang has joined #openstack-infra | 02:05 | |
| jogo | portante: so if milestone-proposed becomes stable/icehouse then you need that patch otherwise just delete milestone-proposed | 02:05 |
| portante | I think notmyname has not updated milestone-proposed to be for 1.13.0 yet, probably still 1.12.0 | 02:05 |
| clarkb | portante: no it is 1.13.0 | 02:05 |
| clarkb | but 1.13.0 doesn't include the fix | 02:06 |
| clarkb | master does | 02:06 |
| portante | but a month old? that does not seem right | 02:06 |
| jogo | portante: what is milestone-proposed for? | 02:06 |
| portante | I don't know what notmyname using it for, so it is best to wait for him to answer | 02:06 |
| portante | I am curious as well | 02:06 |
| clarkb | portante: commit is from Feb 24th and was tagged march 3 | 02:06 |
| fungi | portante: i suspect it's just an oversight from the last swift release | 02:06 |
| jogo | http://git.openstack.org/cgit/openstack/swift/log/?h=milestone-proposed | 02:06 |
| *** CaptTofu has joined #openstack-infra | 02:07 | |
| fungi | it looks like m-p for swift gets created a week before the release, and then probably deleted again immediately after the release under normal circumstances, but wasn't this last time | 02:07 |
| portante | clarkb, jogo, fungi gonna guess here, but I think back at the end of february there was a consideration of making a 1.13.0 release, but we decided it was not ready, and just left the tag there | 02:08 |
| portante | I am almost positive that much of what is on master is going to be milestone-proposed, and including that pipeline patch, but notmyname will have to confirm | 02:08 |
| fungi | lifeless: http://paste.openstack.org/show/74791/ | 02:10 |
| lifeless | fungi: we're down to 5 slaves now :( | 02:10 |
| *** amotoki has joined #openstack-infra | 02:11 | |
| lifeless | fungi: so those two are more old ones, let me stab them. I'm more worried about the 19 or so spawned and active vms that aren't running jobs | 02:11 |
| jogo | portante: sounds good, well lets get to the bottom of this soon,as its blocking any RC2s from being cut | 02:11 |
| jogo | fungi clarkb: the more I look the more I think I am wrong, and your right | 02:12 |
| * jogo wonders off to find food | 02:12 | |
| fungi | portante: got it. swift's definition of what constitutes a release differs from most of ours i guess. even though http://tarballs.openstack.org/swift/swift-1.13.0.tar.gz is tagged and published, it wasn't a release because there was no announcement i guess | 02:12 |
| *** jhesketh has quit IRC | 02:12 | |
| notmyname | holy buffer playback, batman | 02:13 |
| notmyname | portante: clarkb: what's up? | 02:13 |
| notmyname | I see my name a lot | 02:13 |
| fungi | notmyname: just wondering if you meant to leave that milestone-proposed branch around | 02:13 |
| portante | notmyname: sorry, just texted you | 02:13 |
| fungi | notmyname: integration tests are using it for m-p changes for the integrated projectys | 02:13 |
| notmyname | I don't manage that (I don't have perms). ttx does. | 02:14 |
| notmyname | the milestone-proposed branch is used for swift releases | 02:14 |
| fungi | notmyname: he probably forgot to delete it. if you aren't using it, i'll prod him | 02:14 |
| portante | notmyname: kewl | 02:14 |
| clarkb | notmyname: tl;dr is swift milestone proposed does not work with the rest of openstack milestone proposed | 02:14 |
| notmyname | we're not. we haven't done an RC yet for icehouse, so if it's still there its from the 1.13.0 release | 02:14 |
| clarkb | notmyname: so we either need to back port or remove branch | 02:14 |
| notmyname | clarkb: which is actually a different question. why not? | 02:14 |
| notmyname | what's broken? | 02:14 |
| lifeless | fungi: ok no error state VMS | 02:15 |
| clarkb | notmyname: tempurl isn't run by default in 1.13.0's config | 02:15 |
| notmyname | clarkb: remove the branch | 02:15 |
| notmyname | ? | 02:15 |
| clarkb | notmyname: and apparently stuff depends on tempurl in the rest of openstack milestone proposed | 02:15 |
| notmyname | clarkb: was it earlier? when did it get removed if it was? | 02:15 |
| notmyname | clarkb: in devstack? | 02:15 |
| lifeless | fungi: 34 ACTIVE, 5 in state delet | 02:15 |
| clarkb | notmyname: it wasn't, but 1.13.0 is more than a month old | 02:15 |
| clarkb | notmyname: no in swift itself | 02:15 |
| clarkb | notmyname: devstack just copy pasta's your config | 02:15 |
| portante | sample config, that is | 02:16 |
| lifeless | fungi: so thats 29 vms, 4 slaves. | 02:16 |
| lifeless | fungi: there is something very wrong here :( | 02:16 |
| lifeless | nova list --all-tenants | grep cloud | grep ACTIVE.*unning | wc -l | 02:16 |
| lifeless | 31 | 02:16 |
| fungi | lifeless: trying to delete 3384049 now | 02:16 |
| fungi | looks like that worked | 02:17 |
| *** signed8bit has quit IRC | 02:17 | |
| notmyname | clarkb: looks like it was added to the sample config on mar 7 | 02:17 |
| notmyname | and that is after 1.13.0 | 02:18 |
| clarkb | that would do it | 02:18 |
| fungi | notmyname: yeah, i think if ttx deletes the milestone-proposed branch for swift, this problem goes away | 02:18 |
| clarkb | I don't think this is necessarily very urgent, it was just very weird due to swift's release process being different | 02:18 |
| notmyname | whatever is depending on it /should/ be using the discoverable capabilities. that's what it's there for. you can't count on tempurl being there in any swift deployment. | 02:19 |
| clarkb | but that depends on the nature of the fixes going into other projects' milestone proposed branches | 02:19 |
| clarkb | notmyname: its the testing suite | 02:19 |
| clarkb | notmyname: we need to test that | 02:19 |
| notmyname | test the testing suite? | 02:19 |
| clarkb | no, test suite needs to test tempurl | 02:20 |
| notmyname | clarkb: ah ok, so tempest (?) is relying on tempurl being there | 02:20 |
| clarkb | notmyname: yes | 02:20 |
| notmyname | clarkb: yes, agreed. but raise a SkipTest if it's not enabled | 02:20 |
| notmyname | same with a ton of other swift features | 02:20 |
| clarkb | notmyname: that wuold have masked the problem | 02:20 |
| fungi | lifeless: i'm re-trying to delete a6e2c5de-2906-44f6-b12f-31a07e5e9175 and it's timing out again | 02:20 |
| *** mriedem has quit IRC | 02:20 | |
| clarkb | notmyname: which is arguably no better | 02:20 |
| fungi | lifeless: should it be doing that? | 02:20 |
| notmyname | no, it's arguable better. it wouldn't block RCs | 02:20 |
| clarkb | notmyname: but it would allow potentially broken swift + openstack to release | 02:21 |
| lifeless | fungi: thats _ovs_vsctl(['del-port', bridge, dev]) | 02:21 |
| *** amcrn has quit IRC | 02:21 | |
| clarkb | we should fix tests not skip them | 02:21 |
| notmyname | clarkb: we should not continue this discussion about the stability of swift releases and just agree that testing is good ;-) | 02:22 |
| notmyname | point being, for now, we need the m-p branch deleted | 02:22 |
| lifeless | fungi: stabbed | 02:22 |
| fungi | lifeless: and it returned immediately after you did. cool | 02:23 |
| fungi | lifeless: need the same for 48cb454d-175f-48ae-83e5-b266571b3fd9 looks like | 02:23 |
| lifeless | different hypervisors | 02:24 |
| lifeless | so not one bad hypervisor | 02:24 |
| lifeless | fungi: try now | 02:24 |
| fungi | lifeless: i bet that just makes your day | 02:24 |
| lifeless | neutron vif plugging glue failed under load? news @ 11 | 02:24 |
| lifeless | though the stack trace is weird | 02:25 |
| *** Alexandra_ has quit IRC | 02:25 | |
| lifeless | it looks like we're still using the hybrid stuff | 02:25 |
| fungi | "load" meaning a few dozen instances with one tenant | 02:25 |
| lifeless | so there may be newer better stuff | 02:25 |
| lifeless | fungi: load refers to churn | 02:25 |
| lifeless | fungi: not scale :P | 02:25 |
| fungi | ahh, okay yes we do churn pretty heavily, granted | 02:25 |
| fungi | okay, that one deleted | 02:26 |
| *** jhesketh has joined #openstack-infra | 02:26 | |
| fungi | nova boot seems to be whacked out though | 02:28 |
| fungi | which is why you're not getting new nodes... | 02:28 |
| fungi | ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-43db38fd-5b65-40be-960f-d00bf74e23b9) | 02:28 |
| lifeless | fungi: well we have 29 ACTIVE that aren't acting as slaves | 02:29 |
| lifeless | fungi: this is what concerns me :) | 02:29 |
| *** dcramer_ has quit IRC | 02:29 | |
| clarkb | lifeless: fungi did those 29 return http 500 then go active? | 02:29 |
| clarkb | iirc we have seen that before | 02:29 |
| fungi | i see 29 "building" | 02:29 |
| lifeless | I see that they don't have floating-ips | 02:29 |
| lifeless | http://paste.ubuntu.com/7192570/ | 02:30 |
| lifeless | nova floating-ip-list | pastebinit | 02:30 |
| lifeless | http://paste.ubuntu.com/7192574/ | 02:30 |
| lifeless | fungi: are there errors in nodepool about floating ips ? | 02:31 |
| lifeless | all those with - seem to be leaked, to me | 02:31 |
| lifeless | let me zap em for you | 02:31 |
| *** yamahata has joined #openstack-infra | 02:31 | |
| lifeless | nova floating-ip-list | awk '$5=="-" { print $2}' | xargs -n1 nova floating-ip-delete | 02:31 |
| lifeless | running now | 02:31 |
| fungi | lifeless: yeah, the HTTP 500 i quoted above was attempting to do floating-ip create | 02:31 |
| lifeless | /var/log/upstart/nova-api.log:2014-04-02 02:30:03.375 7736 ERROR nova.api.openstack [req-43db38fd-5b65-40be-960f-d00bf74e23b9 d5af62d2183d431796d74c5bb119ec9f e01e473a9250498883955b80966a1e58] Caught error: 409-{u'NeutronError': {u'message': u"Quota exceeded for resources: ['floatingip']", u'type': u'OverQuota', u'detail': u''}} | 02:32 |
| fungi | stellar | 02:32 |
| lifeless | you might want to file a bug about how thats a useless feedback to the user | 02:32 |
| fungi | what, a message saying "either something went wrong or else it went right but wasn't what you wanted" is useless feedback? ;) | 02:33 |
| lifeless | your floatingip quota is 60 | 02:34 |
| lifeless | which is = instance quota | 02:34 |
| fungi | that ought to be sufficient given that max-servers is now 55 | 02:34 |
| fungi | but why were we leaking those, i wonder | 02:34 |
| lifeless | right, max-servers = 55 + 1 for te broke + 2 for image buidlds + 2 for slack | 02:34 |
| lifeless | = 60 | 02:35 |
| lifeless | clearly there is a nodepool bug | 02:35 |
| fungi | i just witnessed a node go ready, so i think this is working | 02:35 |
| lifeless | yay | 02:35 |
| lifeless | will you file the bug ? | 02:35 |
| clarkb | s/nodepool/novaclient/ | 02:36 |
| lifeless | or shall I ? | 02:36 |
| clarkb | nodepool just says give me a thing | 02:36 |
| lifeless | clarkb: leaking floatingips? nodepool - it is responsible for deleting them | 02:36 |
| fungi | and yeah, i know we plugged at least one floating-ip leak, but not sure where others might lurk | 02:36 |
| lifeless | clarkb: since its responsible for creating them | 02:36 |
| clarkb | lifeless: no, novaclient does it | 02:36 |
| clarkb | unless I missed something in the use of floating ips its just an arg to novaclient | 02:37 |
| lifeless | clarkb: nodepool/provider_manager.py line 136 | 02:37 |
| fungi | mordred has a wip change to reuse created floating-ips which aren't assigned to an instance any longer | 02:37 |
| *** rlandy has quit IRC | 02:37 | |
| lifeless | fungi: that should help | 02:37 |
| fungi | nodepool normally just tries to delete/create | 02:37 |
| fungi | lifeless: well, it would help more if it wasn't wip i think ;) | 02:37 |
| clarkb | fungi: I think that is an anti feature | 02:38 |
| lifeless | clarkb: what is? | 02:38 |
| clarkb | fungi: he wrote it to solve a percieved problem that HP had but isn't necessary to solve that problem | 02:38 |
| fungi | ahh | 02:38 |
| clarkb | lifeless: nodepool reusing particular floating IPs | 02:38 |
| clarkb | lifeless: mordred wanted it so that he could have static firewall rules | 02:38 |
| lifeless | clarkb: well I like it because its less churn | 02:38 |
| fungi | i figured it was just an efficiency measure to keep from deleting and creating the same floating-ips over and over | 02:38 |
| lifeless | clarkb: so rather than create + associate + delete | 02:38 |
| lifeless | its create + associate, + associate ... + delete. | 02:39 |
| clarkb | fungi: nope, it was purely because lol network firewalls | 02:39 |
| fungi | lolercoaster | 02:39 |
| *** gyee has quit IRC | 02:39 | |
| fungi | okay then | 02:39 |
| lifeless | ok we have much slaves now | 02:39 |
| clarkb | lifeless: my problem with it is it shifts the management of 1k floating IPs to humans | 02:39 |
| lifeless | either way | 02:39 |
| *** mspreitz has joined #openstack-infra | 02:39 | |
| lifeless | clarkb: huh why? | 02:39 |
| clarkb | lifeless: because someone needs to preallocate them | 02:39 |
| lifeless | clarkb: god no | 02:39 |
| *** SumitNaiksatam has quit IRC | 02:39 | |
| lifeless | clarkb: thats a totally different 'feature' | 02:39 |
| clarkb | lifeless: right its an anti feature | 02:40 |
| lifeless | clarkb: allocate when there are none available | 02:40 |
| lifeless | clarkb: delete if more than (say) 50 are unused | 02:40 |
| lifeless | clarkb: reuse when possible | 02:40 |
| clarkb | lifeless: ah so you are talking about something different | 02:40 |
| clarkb | lifeless: mordred's change depends on preallocating all that you need so that you can write firewall rules for them | 02:40 |
| clarkb | then only let nodepool use that particular pool | 02:40 |
| *** mfer has quit IRC | 02:40 | |
| lifeless | clarkb: yes, so I think we agree thats entirely different | 02:40 |
| lifeless | clarkb: its not about reuse per se, its about staticness | 02:41 |
| clarkb | ya what you describe I can see doing | 02:41 |
| clarkb | but mordred's change doesn't do that | 02:41 |
| fungi | BadRequest: Error. Unable to associate floating ip (HTTP 400) (Request-ID: req-1491460b-3ca8-415e-b31f-fd06ea00fbd4) | 02:42 |
| fungi | came immediately on the heels of an occurrence of the other error, which we're also still getting | 02:43 |
| lifeless | http://paste.openstack.org/show/74792/ | 02:46 |
| *** signed8bit has joined #openstack-infra | 02:46 | |
| *** signed8bit has quit IRC | 02:46 | |
| fungi | lifeless: bug 1301111 | 02:46 |
| uvirtbot | Launchpad bug 1301111 in nodepool "nodepoold provider manager leaks floating-ips" [High,Triaged] https://launchpad.net/bugs/1301111 | 02:46 |
| lifeless | fungi: cool. should I file one for nova on the 500 inscrutable error ? | 02:47 |
| lifeless | fungi: that VM appears to have its ip addresses the wrong way around | 02:48 |
| fungi | lifeless: you might be able to muster more snark than me--it hasn't been daylight here for about 6 hours now | 02:48 |
| *** alff has quit IRC | 02:48 | |
| *** alff_ has quit IRC | 02:48 | |
| fungi | 5 hours. something like that | 02:48 |
| * fungi is running out of steam | 02:48 | |
| fungi | so i'm guessing this is an multiple interface order + floating ip issue | 02:49 |
| fungi | some instances ending up with their interfaces swapped, causing floating ip allocation to get tried on a non-matching network | 02:50 |
| lifeless | nwcache order issue perhaps | 02:50 |
| lifeless | but yes thats exactly it | 02:50 |
| fungi | and then nodepoold not dealing gracefully with that failure mode | 02:50 |
| clarkb | lifeless: fungi I think https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/provider_manager.py#n451 is why the floating ips leak | 02:50 |
| lifeless | fungi: so actually nodepool should use neutron for floating ips when using neutron nics | 02:50 |
| lifeless | fungi: more code - need to query nova and neutron find the port then associate | 02:51 |
| lifeless | but it would avoid the failure mode entirely | 02:51 |
| clarkb | going back to lifeless' paste the server field for the ip is blank | 02:51 |
| clarkb | which would make that check always fail | 02:51 |
| clarkb | is that a nova/neutron bug at heart? | 02:51 |
| clarkb | or is there something nodepool can do to assocaite a floating ip that hasn't been associated? | 02:52 |
| *** e0ne has joined #openstack-infra | 02:52 | |
| fungi | right, so we're iterating through the list of allocated floating-ips looking for the one which says its assigned to the instance we're deleting | 02:52 |
| clarkb | yup but none are associated so we leak | 02:52 |
| *** wenlock_ has joined #openstack-infra | 02:52 | |
| fungi | but if the floating-ip gets orphaned, it hangs around indefinitely then | 02:52 |
| fungi | yeah, ugly | 02:53 |
| clarkb | so, we may want a thing that says if not intance id or instance if == node id | 02:53 |
| lifeless | fungi: can I get the error it threw? | 02:53 |
| lifeless | fungi: the quota error | 02:53 |
| clarkb | that will then remove unassociated IPs, but could possibly race against allocations that haven't been allocated yet | 02:54 |
| fungi | lifeless: it's in bug 1301111 | 02:54 |
| uvirtbot | Launchpad bug 1301111 in nodepool "nodepoold provider manager leaks floating-ips" [High,Triaged] https://launchpad.net/bugs/1301111 | 02:54 |
| fungi | ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-bb4664c5-614a-4242-97c0-c927135e5d04) | 02:54 |
| fungi | that one was the wrong-interface issue, but the error message is the same as when there was no quota | 02:54 |
| fungi | because, you know, if you're going to have a vague and meaningless error, you really ought to use it in every possible situation you can | 02:55 |
| clarkb | updated bug with what I found | 02:55 |
| *** david-lyle has joined #openstack-infra | 02:56 | |
| fungi | clarkb: thanks. i think that about does it for me tonight anyway | 02:56 |
| lifeless | clarkb: I've never seen ServerId filled in | 02:56 |
| lifeless | filed https://bugs.launchpad.net/nova/+bug/1301117 | 02:57 |
| uvirtbot | Launchpad bug 1301117 in nova "floating ip quota error neutron" [Undecided,New] | 02:57 |
| *** e0ne has quit IRC | 02:57 | |
| clarkb | lifeless: it apparently works with hpcloud unless we are leaking ips there (but the quota is 600 so we would know) | 02:57 |
| fungi | lifeless: thanks--that should be entertaining | 02:57 |
| clarkb | maybe that comes from a different table? | 02:57 |
| *** Sukhdev has quit IRC | 02:58 | |
| *** dcramer_ has joined #openstack-infra | 03:00 | |
| *** Alexandra_ has joined #openstack-infra | 03:01 | |
| lifeless | the floatingip code hasnt' changed in ages | 03:02 |
| lifeless | so not a nodepool change | 03:02 |
| lifeless | perhaps a nova change | 03:02 |
| lifeless | I may be misremembering 'never seen it filled out' | 03:02 |
| lifeless | jogo: ^ | 03:02 |
| lifeless | this completely breaks nodepool | 03:02 |
| lifeless | presumably with neutron | 03:03 |
| clarkb | lifeless: your idea of keeping track of how many are in the pool would work around this I think. As we could synchronize creates and deletes more easily | 03:04 |
| clarkb | however I would expect serverid to be populated | 03:04 |
| lifeless | you might expect that | 03:04 |
| lifeless | while true; do nova floating-ip-list | awk '$5=="-" { print $2}' | xargs -n1 nova floating-ip-delete; done | 03:07 |
| lifeless | I'm going to run that indefinitely | 03:07 |
| lifeless | filing a bug on the blank server id | 03:07 |
| clarkb | oh wait, everything is already synchronized per ProviderManager | 03:10 |
| clarkb | I think | 03:10 |
| clarkb | so making the check if not instanceid or instanceid == serverid should work | 03:10 |
| * clarkb will write that change and jeblair can -1 it tomorrow :) | 03:10 | |
| *** CaptTofu has quit IRC | 03:12 | |
| *** maxbit has joined #openstack-infra | 03:12 | |
| clarkb | oh nevermind I see the race. Create and add are distinct | 03:13 |
| *** david-lyle has quit IRC | 03:14 | |
| *** dims_ has quit IRC | 03:15 | |
| *** jhesketh has quit IRC | 03:16 | |
| lifeless | filed https://bugs.launchpad.net/nova/+bug/1301128 | 03:17 |
| uvirtbot | Launchpad bug 1301128 in nova "nova floating-ip-list has blank server id field" [Undecided,New] | 03:17 |
| lifeless | clarkb: can't we just check on fixed ip address ? | 03:17 |
| *** SumitNaiksatam has joined #openstack-infra | 03:17 | |
| lifeless | clarkb: that *is* filled in | 03:17 |
| clarkb | lifeless: maybe? | 03:17 |
| *** dims_ has joined #openstack-infra | 03:18 | |
| * clarkb tries to think of cases where you wouldn't have a fixed IP and can't come up with anything | 03:18 | |
| *** matsuhashi has quit IRC | 03:21 | |
| *** harlowja is now known as harlowja_away | 03:24 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Synchronize floating IP ops to prevent leaks. https://review.openstack.org/84611 | 03:24 |
| clarkb | lifeless: ^ is not great but may solve the problem in nodepool | 03:24 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Synchronize floating IP ops to prevent leaks. https://review.openstack.org/84611 | 03:25 |
| clarkb | now with more bug tracking | 03:25 |
| clarkb | oh wait, needs more fixing | 03:27 |
| *** jhesketh has joined #openstack-infra | 03:27 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Synchronize floating IP ops to prevent leaks. https://review.openstack.org/84611 | 03:28 |
| clarkb | and the more code I read the more I think there is a better way to fix this | 03:31 |
| lifeless | clarkb: I don' tthink you've thought that patch through entirely | 03:32 |
| lifeless | clarkb: (reviewed) | 03:32 |
| lifeless | bbiaw, C pickup time | 03:32 |
| clarkb | lifeless: ya, I see a completely different approach that is better and actually correct | 03:32 |
| *** morganfainberg is now known as morganfainberg_Z | 03:34 | |
| *** amotoki has quit IRC | 03:34 | |
| *** nosnos has quit IRC | 03:37 | |
| *** _nadya_ has joined #openstack-infra | 03:41 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Immediately delete a floating IP if doesn't attach https://review.openstack.org/84614 | 03:42 |
| clarkb | there | 03:42 |
| *** maxbit has quit IRC | 03:46 | |
| *** e0ne has joined #openstack-infra | 03:52 | |
| *** e0ne has quit IRC | 03:56 | |
| mgagne | ./generate_sample.sh --help => generate_sample.sh: invalid project package name Try `generate_sample.sh --help' for more information. | 03:58 |
| mgagne | I'm trying :O | 03:58 |
| clarkb | mgagne: what is that rom? | 03:58 |
| clarkb | *from | 03:58 |
| *** _nadya_ has quit IRC | 03:59 | |
| *** basha has joined #openstack-infra | 03:59 | |
| mgagne | clarkb: trying to use generate_sample.sh on an other project, looks to not be a turn-key solution :D | 03:59 |
| *** Alexandra_ has quit IRC | 04:00 | |
| mgagne | clarkb: --help looks to not be of any help. :D | 04:00 |
| mgagne | clarkb: don't worry, just ranting about how funny it is | 04:00 |
| clarkb | :) | 04:00 |
| mgagne | clarkb: I won't get pbr for now, but I'll probably get oslo.config :D | 04:01 |
| clarkb | mgagne: could you use a passphraseless key to sign tags, you'd be getting functional pbr but not necessarily secure tags | 04:02 |
| clarkb | mgagne: maybe note that in the tag commit thing | 04:02 |
| mgagne | clarkb: that's what I'm thinking of. I'm not that familiar with GPG keys yet, even less gpg-agent/passwordless stuff so that would be something to learn first. | 04:03 |
| *** mspreitz has quit IRC | 04:07 | |
| jogo | clarkb: I see milestone proposed is still there | 04:09 |
| clarkb | jogo: yes, we have to wait for ttx to fix it | 04:10 |
| jogo | clarkb: oh PTLs can't delete branches? | 04:10 |
| jogo | does sdague have have power? | 04:11 |
| clarkb | I think release managers have the power | 04:11 |
| clarkb | ttx will be around before sdague | 04:11 |
| jogo | https://review.openstack.org/#/admin/groups/11,members | 04:11 |
| *** thuc has joined #openstack-infra | 04:12 | |
| clarkb | yup and ttx will be the next one to be awake | 04:12 |
| *** thuc_ has joined #openstack-infra | 04:13 | |
| jogo | clarkb: that sounds like a good reason to go AFK | 04:14 |
| *** thuc has quit IRC | 04:17 | |
| *** mgagne has quit IRC | 04:17 | |
| lifeless | clarkb: ok, so now we're down to 3 slaves | 04:21 |
| lifeless | clarkb: whats nodepool seeing? | 04:21 |
| clarkb | lifeless: no idea, I don't ahve access right now | 04:22 |
| clarkb | lifeless: this is my way of separating real life from work >_> | 04:22 |
| StevenK | You forcibly revoke keys? | 04:22 |
| lifeless | clarkb: when can we get non-US ops please? | 04:23 |
| clarkb | StevenK: I leave keys on other machines | 04:23 |
| *** mrmartin has joined #openstack-infra | 04:23 | |
| StevenK | clarkb: Yeah, my way is rather more difficult to recover from. :-) | 04:24 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Don't include recheck instructions when unclassified failures https://review.openstack.org/84617 | 04:25 |
| *** pcrews has quit IRC | 04:27 | |
| *** nosnos has joined #openstack-infra | 04:31 | |
| lifeless | clarkb: how would infra feel about slaves calling home rather than being sshed into ? | 04:32 |
| lifeless | clarkb: no need for floatingips at all then | 04:32 |
| lifeless | clarkb: as an option anyhow, maybe not the only mode | 04:33 |
| clarkb | lifeless: it would break gozer | 04:33 |
| clarkb | lifeless: but possibly could be a configurable thing | 04:33 |
| lifeless | gozers control node is behind the firewall? | 04:33 |
| clarkb | yup | 04:33 |
| lifeless | silly gozer :) | 04:33 |
| wenlock_ | anyone tried install_puppet.sh tonight? | 04:35 |
| mattoliverau | wenlock_: yup, what's up? | 04:35 |
| wenlock_ | puppet-common : Depends: facter (< 2.0) but 2.0.1-1puppetlabs1 is to be installed | 04:35 |
| *** pcrews has joined #openstack-infra | 04:35 | |
| wenlock_ | wondering if something changed with apt repos from puppetlabs | 04:36 |
| mattoliverau | wenlock_: A patch has been merged which fixes it, just update the repo. facter needed to be pinned | 04:36 |
| *** matsuhashi has joined #openstack-infra | 04:36 | |
| wenlock_ | do you have the change number? | 04:37 |
| mattoliverau | wenlock_: http://is.gd/NMrFhP | 04:37 |
| wenlock_ | awesome, ty! | 04:38 |
| mattoliverau | no probs | 04:38 |
| lifeless | I'm seeing lots of | 04:38 |
| lifeless | 2014-04-02 04:32:53.331 | git.openstack.org: Temporary failure in name resolution | 04:38 |
| lifeless | failures in tripleo-ci jobs | 04:38 |
| lifeless | host git.openstack.org | 04:38 |
| lifeless | git.openstack.org has address 192.237.223.224 | 04:38 |
| lifeless | git.openstack.org has IPv6 address 2001:4800:7813:516:3bc3:d7f6:ff04:aacb | 04:38 |
| lifeless | from within another VM in that cloud with the same two interfaces | 04:38 |
| *** mgagne has joined #openstack-infra | 04:39 | |
| mattoliverau | lifeless: what do you use as the DNS resover in the hpcloud, is it down/over loaded? Can you dig it directly from the working server? | 04:45 |
| lifeless | mattoliverau: this isn't hpcloud | 04:45 |
| lifeless | mattoliverau: this is tripleo's test cloud's HP region | 04:45 |
| mattoliverau | isn't it. I though trible 0 stuff was | 04:45 |
| mattoliverau | oh | 04:45 |
| lifeless | mattoliverau: https://wiki.openstack.org/wiki/TripleO/TripleOCloud/Regions | 04:46 |
| lifeless | mattoliverau: anyhow, infras jenkins slaves futz with DNS | 04:46 |
| lifeless | mattoliverau: and I'm not sure what the current setup is | 04:46 |
| clarkb | lifeless: they should all use unbound to forward requests to 8.8.8.8 | 04:46 |
| lifeless | clarkb: for instance https://jenkins06.openstack.org/job/check-tripleo-ironic-seed-precise/126/console | 04:47 |
| clarkb | lifeless: can you check how old the tripleo precise image is? | 04:48 |
| lifeless | clarkb: built fresh on tuesday | 04:48 |
| lifeless | clarkb: oh you mean the base image? pulled from ubuntu's cloud site my monday afternoon | 04:49 |
| clarkb | no the nodepool created snapshot | 04:49 |
| clarkb | lifeless: https://review.openstack.org/#/c/84230/ merged this afternoon PST | 04:49 |
| lifeless | clarkb: | dcf09ac5-00a9-4137-9d99-6fcd407c8258 | tripleo-precise-1396313323.template.openstack.org | qcow2 | bare | 5164957696 | active | | 04:50 |
| clarkb | which applies the google dns unbound forwarding to all slaves, wondering if maybe that had unexpected sideeffects in tripelo land | 04:50 |
| lifeless | clarkb: well it broke due to the two-networks the first time you tried to use it | 04:50 |
| lifeless | clarkb: jeblair had a fix, I thought | 04:50 |
| *** matsuhashi has quit IRC | 04:50 | |
| lifeless | | updated_at | 2014-04-01T01:30:49 | | 04:50 |
| lifeless | helpfully there is no TZ on that time | 04:51 |
| clarkb | ok, so that rules out 84320 assuming the TZ thing is sane (UTC) | 04:51 |
| *** e0ne has joined #openstack-infra | 04:52 | |
| lifeless | so the neutron-local resolve thing | 04:52 |
| lifeless | or what if tripleo clouds had good DNS guaranteed (seems like a fairly important deployment feature....) | 04:53 |
| clarkb | if you had good DNS guaranteed, we could probably special case you and rely on the neutron/tripleo provided resolver | 04:54 |
| lifeless | mattoliverau: our current deployment strategy is 8.8.8.8 as the DNS server in the default overlay network | 04:54 |
| lifeless | clarkb: so we want to make these builds fly | 04:54 |
| lifeless | clarkb: and are planning cloud local caches of ubuntu, fedora, pypi, etc. | 04:55 |
| lifeless | clarkb: including dns | 04:55 |
| lifeless | clarkb: the question for tripleo-CI is whether those things are part of the cloud, or part of openstack-infras responsibility. | 04:55 |
| clarkb | lifeless: our experience so far has been that our clouds cannot provide working DNS (not talking about tripleo cloud here) so we have had to assume responsibility | 04:56 |
| lifeless | clarkb: the bigger picture - given we want basically gozer-as-a-service as part of the deployed cloud, is whether we can codeshare with infra on the deployment (given we're not about to use puppet for the baseline deployed infrastructure) | 04:56 |
| clarkb | lifeless: the cost of doing that for one more cloud (tripleo) is really low | 04:56 |
| *** basha has quit IRC | 04:56 | |
| lifeless | clarkb: yes, I get that :0 | 04:56 |
| clarkb | lifeless: ideally, we infra, want no part in this :) clouds should give us working dns | 04:57 |
| lifeless | clarkb: (we want gozer-as-a-service not to do development on openstack, but to CI validate things we're about to deploy) | 04:57 |
| *** e0ne has quit IRC | 04:57 | |
| *** thuc_ has quit IRC | 04:57 | |
| lifeless | clarkb: which means - building images quickly and efficiently etc | 04:57 |
| *** thuc has joined #openstack-infra | 04:58 | |
| *** matsuhashi has joined #openstack-infra | 05:00 | |
| *** thuc has quit IRC | 05:02 | |
| *** talluri has joined #openstack-infra | 05:05 | |
| *** ianw has quit IRC | 05:08 | |
| *** ianw has joined #openstack-infra | 05:08 | |
| *** ianw has quit IRC | 05:09 | |
| *** ianw has joined #openstack-infra | 05:09 | |
| *** LinuxJedi has quit IRC | 05:10 | |
| *** basha has joined #openstack-infra | 05:14 | |
| *** mrmartin has quit IRC | 05:16 | |
| *** basha has quit IRC | 05:19 | |
| *** LinuxJedi has joined #openstack-infra | 05:20 | |
| *** ildikov_ has quit IRC | 05:23 | |
| *** alexandra_ has joined #openstack-infra | 05:24 | |
| *** e0ne has joined #openstack-infra | 05:24 | |
| *** alff has joined #openstack-infra | 05:26 | |
| *** alff_ has joined #openstack-infra | 05:26 | |
| *** alff has quit IRC | 05:28 | |
| *** alff_ has quit IRC | 05:28 | |
| *** amotoki has joined #openstack-infra | 05:31 | |
| *** mrmartin has joined #openstack-infra | 05:38 | |
| *** Ryan_Lane has joined #openstack-infra | 05:43 | |
| *** e0ne has quit IRC | 05:45 | |
| *** e0ne has joined #openstack-infra | 05:45 | |
| *** basha has joined #openstack-infra | 05:49 | |
| *** e0ne has quit IRC | 05:50 | |
| *** VijayTripathi has joined #openstack-infra | 05:52 | |
| lifeless | dstufft: http://pip.readthedocs.org/en/latest/development.html probably wants to supply the VCS location for pip :) | 05:54 |
| *** zns has quit IRC | 05:55 | |
| *** jamielennox is now known as jamielennox|away | 05:55 | |
| *** e0ne has joined #openstack-infra | 06:03 | |
| *** e0ne has quit IRC | 06:04 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: add timeout-var option to the timeout module https://review.openstack.org/84630 | 06:06 |
| *** mrmartin has quit IRC | 06:07 | |
| *** thuc has joined #openstack-infra | 06:08 | |
| *** mgagne has quit IRC | 06:09 | |
| *** jamielennox|away is now known as jamielennox | 06:11 | |
| *** ArxCruz has quit IRC | 06:12 | |
| *** mihgen has joined #openstack-infra | 06:12 | |
| *** thuc has quit IRC | 06:13 | |
| *** ildikov_ has joined #openstack-infra | 06:13 | |
| *** mrda is now known as mrda_away | 06:14 | |
| *** dcramer_ has quit IRC | 06:15 | |
| *** dcramer_ has joined #openstack-infra | 06:31 | |
| *** denis_makogon has joined #openstack-infra | 06:31 | |
| *** shardy_afk is now known as shardy | 06:33 | |
| *** wenlock_ has quit IRC | 06:34 | |
| *** saju_m has joined #openstack-infra | 06:39 | |
| *** vogxn has joined #openstack-infra | 06:41 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: add build-timeout macro https://review.openstack.org/84637 | 06:42 |
| *** yolanda has joined #openstack-infra | 06:43 | |
| *** thomasbiege has joined #openstack-infra | 06:45 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: add timeout-var option to the timeout module https://review.openstack.org/84630 | 06:47 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: add timeout-var option to the timeout module https://review.openstack.org/84630 | 06:54 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: add build-timeout macro https://review.openstack.org/84637 | 06:55 |
| openstackgerrit | Thomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates https://review.openstack.org/83761 | 07:09 |
| *** mrmartin has joined #openstack-infra | 07:10 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Make hacking Python 3 compatible https://review.openstack.org/77585 | 07:10 |
| *** vogxn has quit IRC | 07:11 | |
| *** flaper87|afk is now known as flaper87 | 07:14 | |
| *** thomasbiege has quit IRC | 07:16 | |
| *** jcoufal has joined #openstack-infra | 07:17 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack-infra/reviewstats: Add oslo.messaging project https://review.openstack.org/84641 | 07:18 |
| *** jlibosva has joined #openstack-infra | 07:26 | |
| ttx | jogo: stale milestone-proposed branch ? which repo ? | 07:28 |
| *** afazekas is now known as __afazekas | 07:30 | |
| *** julienvey1 has joined #openstack-infra | 07:31 | |
| ttx | ah, swift | 07:31 |
| ttx | notmyname, jogo, clarkb, fungi: stale milestone-proposed branch for swift removed | 07:31 |
| *** julienvey1 is now known as julienvey | 07:31 | |
| ttx | sorry about that, but branch deletion is the only thing we could not automate in current gerrit | 07:32 |
| * ttx proposes a new motto: "if it's not automated, it's broken" | 07:33 | |
| *** zhiyan is now known as zhiyan_ | 07:35 | |
| *** jgallard has joined #openstack-infra | 07:40 | |
| *** saju_m has quit IRC | 07:42 | |
| *** roeyc has joined #openstack-infra | 07:44 | |
| StevenK | ttx: Your motto could use some improvement: "if it's not automated, tested, used or documented, it's broken" | 07:44 |
| *** boris-42 has quit IRC | 07:50 | |
| *** boris-42 has joined #openstack-infra | 07:51 | |
| *** saju_m has joined #openstack-infra | 07:52 | |
| *** jpich has joined #openstack-infra | 07:58 | |
| *** SumitNaiksatam has quit IRC | 08:04 | |
| *** dizquierdo has joined #openstack-infra | 08:07 | |
| *** skraynev_afk is now known as skraynev | 08:08 | |
| *** _afazekas has joined #openstack-infra | 08:13 | |
| *** e0ne has joined #openstack-infra | 08:16 | |
| *** VijayTripathi has quit IRC | 08:18 | |
| *** wayneeseguin has quit IRC | 08:28 | |
| *** derekh has joined #openstack-infra | 08:28 | |
| *** wayneeseguin has joined #openstack-infra | 08:28 | |
| *** jcoufal_ has joined #openstack-infra | 08:29 | |
| *** jcoufal has quit IRC | 08:30 | |
| *** jcoufal_ is now known as jcoufal | 08:30 | |
| *** Ryan_Lane has quit IRC | 08:30 | |
| *** jhesketh has quit IRC | 08:33 | |
| *** jhesketh_ has joined #openstack-infra | 08:36 | |
| *** jooools has joined #openstack-infra | 08:36 | |
| *** yassine has joined #openstack-infra | 08:38 | |
| *** saju_m has quit IRC | 08:48 | |
| *** saju_m has joined #openstack-infra | 08:49 | |
| *** dizquierdo has quit IRC | 08:53 | |
| *** che-arne has joined #openstack-infra | 08:53 | |
| *** jcoufal has quit IRC | 08:54 | |
| lifeless | clarkb: I had another simpler idea | 08:56 |
| lifeless | clarkb: the server.addresses list contains the floating ip its using. | 08:56 |
| lifeless | clarkb: just matching the floating ip against the servers ip addresses should be sufficient | 08:56 |
| *** matrohon_ has joined #openstack-infra | 08:57 | |
| *** matrohon_ has quit IRC | 08:57 | |
| *** chandankumar_ has joined #openstack-infra | 08:59 | |
| *** dizquierdo has joined #openstack-infra | 09:08 | |
| *** thomasbiege has joined #openstack-infra | 09:10 | |
| *** jp_at_hp has joined #openstack-infra | 09:12 | |
| *** thomasbiege has quit IRC | 09:12 | |
| *** ominakov has joined #openstack-infra | 09:21 | |
| *** yamahata has quit IRC | 09:24 | |
| *** saju_m has quit IRC | 09:24 | |
| *** saju_m has joined #openstack-infra | 09:26 | |
| *** ryanpetrello has quit IRC | 09:26 | |
| *** hashar has joined #openstack-infra | 09:26 | |
| *** dcramer_ has quit IRC | 09:27 | |
| *** dcramer_ has joined #openstack-infra | 09:27 | |
| *** ProfFalken has quit IRC | 09:27 | |
| *** proffalken has joined #openstack-infra | 09:28 | |
| *** pcrews_ has joined #openstack-infra | 09:30 | |
| *** saju_m has quit IRC | 09:31 | |
| *** ryanpetrello has joined #openstack-infra | 09:31 | |
| *** pcrews has quit IRC | 09:33 | |
| *** alexpilotti has joined #openstack-infra | 09:34 | |
| *** yaguang has quit IRC | 09:39 | |
| *** saju_m has joined #openstack-infra | 09:39 | |
| *** pblaho has joined #openstack-infra | 09:41 | |
| *** dizquierdo has quit IRC | 09:42 | |
| *** denis_makogon_ has joined #openstack-infra | 09:48 | |
| *** matsuhashi has quit IRC | 09:50 | |
| *** denis_makogon has quit IRC | 09:51 | |
| *** matsuhashi has joined #openstack-infra | 09:57 | |
| *** YorikSar_ has joined #openstack-infra | 10:01 | |
| *** YorikSar_ is now known as YorikSar | 10:06 | |
| *** dizquierdo has joined #openstack-infra | 10:06 | |
| *** dizquierdo_ has joined #openstack-infra | 10:07 | |
| *** dizquierdo has quit IRC | 10:07 | |
| *** dizquierdo_ is now known as dizquierdo | 10:08 | |
| *** jcoufal has joined #openstack-infra | 10:10 | |
| *** chandankumar_ has quit IRC | 10:11 | |
| *** markmc has joined #openstack-infra | 10:12 | |
| *** wenlock has quit IRC | 10:14 | |
| *** e0ne_ has joined #openstack-infra | 10:23 | |
| *** ildikov_ has quit IRC | 10:26 | |
| *** jgallard has quit IRC | 10:27 | |
| *** e0ne has quit IRC | 10:27 | |
| *** denis_makogon_ is now known as denis_makogon | 10:28 | |
| *** azneita has joined #openstack-infra | 10:34 | |
| *** katyafervent is now known as katyafervent_awa | 10:42 | |
| *** katyafervent_awa is now known as katyafervent | 10:42 | |
| *** e0ne_ has quit IRC | 10:44 | |
| *** azneita has quit IRC | 10:44 | |
| openstackgerrit | A change was merged to openstack-infra/config: Join gerritbot to the satori IRC channel https://review.openstack.org/79058 | 10:51 |
| openstackgerrit | Cyril Roelandt proposed a change to openstack-infra/config: Hacking: enable the Python 3 gate. https://review.openstack.org/84690 | 10:54 |
| *** hashar has quit IRC | 10:56 | |
| *** yjiang has quit IRC | 10:58 | |
| *** matsuhashi has quit IRC | 11:00 | |
| *** nosnos has quit IRC | 11:04 | |
| *** Guest86798 is now known as BobBall | 11:07 | |
| *** BobBall is now known as Guest50977 | 11:07 | |
| *** openstackgerrit has quit IRC | 11:08 | |
| *** openstackgerrit has joined #openstack-infra | 11:08 | |
| *** e0ne has joined #openstack-infra | 11:16 | |
| *** matsuhashi has joined #openstack-infra | 11:16 | |
| *** julim has joined #openstack-infra | 11:16 | |
| *** e0ne_ has joined #openstack-infra | 11:24 | |
| *** rlandy has joined #openstack-infra | 11:24 | |
| *** pblaho has quit IRC | 11:26 | |
| *** e0ne has quit IRC | 11:27 | |
| *** Guest50977 is now known as GnomeKing | 11:33 | |
| *** CaptTofu has joined #openstack-infra | 11:33 | |
| *** amotoki has quit IRC | 11:56 | |
| *** hashar has joined #openstack-infra | 12:00 | |
| *** pdmars has joined #openstack-infra | 12:03 | |
| *** basha has quit IRC | 12:04 | |
| *** dcramer_ has quit IRC | 12:04 | |
| *** yolanda has quit IRC | 12:06 | |
| *** rfolco has joined #openstack-infra | 12:07 | |
| *** dprince has joined #openstack-infra | 12:12 | |
| *** dims_ has quit IRC | 12:13 | |
| *** bknudson has quit IRC | 12:13 | |
| *** YorikSar has quit IRC | 12:15 | |
| *** thomasbiege has joined #openstack-infra | 12:15 | |
| *** saju_m has quit IRC | 12:17 | |
| *** esker has joined #openstack-infra | 12:17 | |
| *** YorikSar has joined #openstack-infra | 12:18 | |
| *** CaptTofu has quit IRC | 12:20 | |
| *** maxbit has joined #openstack-infra | 12:20 | |
| *** e0ne has joined #openstack-infra | 12:22 | |
| *** esker has quit IRC | 12:22 | |
| *** e0ne_ has quit IRC | 12:25 | |
| *** dims_ has joined #openstack-infra | 12:27 | |
| *** weshay has joined #openstack-infra | 12:32 | |
| *** pfallenop has quit IRC | 12:32 | |
| *** bknudson has joined #openstack-infra | 12:33 | |
| *** jhesketh_ has quit IRC | 12:34 | |
| *** pfallenop has joined #openstack-infra | 12:35 | |
| *** ildikov_ has joined #openstack-infra | 12:36 | |
| sdague | trippleo cloud hung again? | 12:36 |
| fungi | ttx: thanks! | 12:38 |
| *** chandankumar_ has joined #openstack-infra | 12:39 | |
| *** adalbas has joined #openstack-infra | 12:39 | |
| *** yolanda has joined #openstack-infra | 12:39 | |
| *** pfallenop has quit IRC | 12:43 | |
| roeyc | Hi, I'm having problem running evstack | 12:44 |
| *** dkranz has joined #openstack-infra | 12:44 | |
| roeyc | Devstack | 12:44 |
| roeyc | the problem is originating in the oslo.config module, I believe. | 12:45 |
| roeyc | Cant install oslo.config properly: http://paste.openstack.org/show/74816/ | 12:46 |
| *** pfallenop has joined #openstack-infra | 12:47 | |
| *** saschpe has quit IRC | 12:47 | |
| *** saju_m has joined #openstack-infra | 12:47 | |
| roeyc | Ideas, anyone? | 12:49 |
| *** aysyd has joined #openstack-infra | 12:49 | |
| *** dcramer_ has joined #openstack-infra | 12:50 | |
| *** xchu has joined #openstack-infra | 12:50 | |
| dhellmann | roeyc: this problem is caused by having 2 copies of an oslo library installed (probably not oslo.config), one using "pip -e" and another using "pip" | 12:50 |
| dhellmann | or if not "pip" sometimes a system package | 12:51 |
| *** chandankumar_ has quit IRC | 12:51 | |
| *** saschpe has joined #openstack-infra | 12:52 | |
| *** thuc has joined #openstack-infra | 12:54 | |
| *** thuc_ has joined #openstack-infra | 12:55 | |
| roeyc | dhellmann: Ok, thanks, this was helpful | 12:55 |
| *** chandan_kumar has quit IRC | 12:55 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Extract experimental-tripleo jobs to template https://review.openstack.org/84026 | 12:56 |
| *** thuc has quit IRC | 12:59 | |
| *** YorikSar_ has joined #openstack-infra | 12:59 | |
| *** _afazekas has quit IRC | 13:01 | |
| *** yamahata has joined #openstack-infra | 13:02 | |
| *** matsuhashi has quit IRC | 13:02 | |
| *** matsuhashi has joined #openstack-infra | 13:02 | |
| *** YorikSar has quit IRC | 13:02 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add rally-scenarios job https://review.openstack.org/84719 | 13:04 |
| openstackgerrit | A change was merged to openstack-infra/config: Add the check-tripleo-overcloud-f20 jenkins job https://review.openstack.org/81793 | 13:04 |
| *** prad_ has joined #openstack-infra | 13:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add tripleo overcloud job to check queue https://review.openstack.org/81794 | 13:05 |
| *** matsuhashi has quit IRC | 13:07 | |
| *** unicell has quit IRC | 13:07 | |
| *** dkranz has quit IRC | 13:07 | |
| *** ociuhandu has quit IRC | 13:07 | |
| *** jcoufal has quit IRC | 13:08 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add tox env for testing jenkins-job-builder https://review.openstack.org/83776 | 13:09 |
| *** matsuhashi has joined #openstack-infra | 13:10 | |
| *** jcoufal has joined #openstack-infra | 13:11 | |
| *** mriedem has joined #openstack-infra | 13:11 | |
| *** pdmars has quit IRC | 13:14 | |
| *** pdmars_ has joined #openstack-infra | 13:14 | |
| *** signed8bit has joined #openstack-infra | 13:14 | |
| *** esker has joined #openstack-infra | 13:14 | |
| *** dcramer_ has quit IRC | 13:17 | |
| *** eharney has joined #openstack-infra | 13:17 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add rally-scenarios job https://review.openstack.org/84719 | 13:17 |
| *** jgallard has joined #openstack-infra | 13:18 | |
| *** CaptTofu has joined #openstack-infra | 13:20 | |
| *** dkranz has joined #openstack-infra | 13:20 | |
| *** YorikSar has joined #openstack-infra | 13:22 | |
| *** YorikSar_ has quit IRC | 13:25 | |
| *** unicell has joined #openstack-infra | 13:34 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add rally-scenarios job https://review.openstack.org/84719 | 13:35 |
| *** saju_m has quit IRC | 13:36 | |
| *** denis_makogon has quit IRC | 13:37 | |
| *** dstanek has quit IRC | 13:40 | |
| *** thomasbiege has quit IRC | 13:42 | |
| *** thuc_ has quit IRC | 13:44 | |
| *** thuc has joined #openstack-infra | 13:44 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add rally-scenarios job https://review.openstack.org/84719 | 13:48 |
| *** chandan_kumar has joined #openstack-infra | 13:49 | |
| *** thuc has quit IRC | 13:49 | |
| *** jpeeler1 is now known as jpeeler | 13:50 | |
| *** jpeeler has joined #openstack-infra | 13:50 | |
| sdague | roeyc: how current is your devstack? | 13:53 |
| sdague | because we were supposed to have fixed that yesterday | 13:53 |
| *** denis_makogon has joined #openstack-infra | 13:56 | |
| *** vhoward- has joined #openstack-infra | 13:56 | |
| *** jgrimm has joined #openstack-infra | 13:59 | |
| *** zns has joined #openstack-infra | 14:04 | |
| *** zns has quit IRC | 14:05 | |
| *** zns has joined #openstack-infra | 14:05 | |
| *** ameade has joined #openstack-infra | 14:05 | |
| *** dstanek has joined #openstack-infra | 14:05 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Parameterize library cross-test job https://review.openstack.org/83759 | 14:09 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Enable cross-testing of oslo.test with dib and t-i-e https://review.openstack.org/83885 | 14:09 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements https://review.openstack.org/83886 | 14:09 |
| roeyc | sdague: Devstack install as part of our 3rd party CI, we clone from master | 14:12 |
| roeyc | The problem started yesterday, so I believe the fix has something to do with it | 14:13 |
| *** alff_ has joined #openstack-infra | 14:15 | |
| *** alff has joined #openstack-infra | 14:15 | |
| *** jpeeler has quit IRC | 14:17 | |
| *** jpeeler has joined #openstack-infra | 14:18 | |
| *** jpeeler has quit IRC | 14:18 | |
| *** jpeeler has joined #openstack-infra | 14:18 | |
| *** hashar has quit IRC | 14:22 | |
| *** hashar has joined #openstack-infra | 14:23 | |
| *** e0ne_ has joined #openstack-infra | 14:26 | |
| dhellmann | jeblair, fungi, clarkb, sdague: given that yesterday we had another case of an oslo lib release breaking unit tests in another project, it would be really good if we could merge the chain starting with https://review.openstack.org/83759 soon | 14:26 |
| sdague | roeyc: you are reusing nodes? | 14:28 |
| fungi | dhellmann: agreed... re-reviewing now | 14:28 |
| dhellmann | fungi: thanks | 14:28 |
| dhellmann | fungi: once that chain is in, I'll go through and figure out which cross-tests we need and submit those separately (bnemec proposes some in that chain, but not all) | 14:29 |
| fungi | dhellmann: yeah, i expect it to be a sizeable list, but the volume of changes triggering those is relatively small compared to some of the higher-volume projects we have so it should even out | 14:30 |
| dhellmann | fungi: right | 14:30 |
| *** e0ne has quit IRC | 14:30 | |
| dhellmann | fungi: are unit test nodes reused, or is this going to trigger a lot more VM consumption? | 14:30 |
| roeyc | sdague: yep, we need to use physical-host in-order to properly test our plugins | 14:30 |
| fungi | they're not reused these days, but don't worry too much about consumption (we're already burning through many thousands a day, this will probably be a drop in the ocean) | 14:31 |
| sdague | roeyc: right, you'll probably need to reset those nodes fully. Or at lest rm -rf /opt/stack/ | 14:31 |
| dhellmann | fungi: heh, ok | 14:31 |
| fungi | dhellmann: my bigger effect concern is that it will significantly slow the pace of development on library projects, since an unrelated gating bug on any one consuming project will effectively block you from merging changes | 14:32 |
| fungi | however, hopefully that won't be an issue, and i still see the benefits of this outweighing the drawbacks | 14:32 |
| dhellmann | fungi: yeah, I have some concern about that, too, but the current situation is demonstrably bad so maybe this will make it better | 14:33 |
| dhellmann | fungi: it's possible we would be able to dial down the cross-testing by doing more alpha releases | 14:34 |
| bnemec | dhellmann: fungi: Gate bugs shouldn't be as much of a problem for this since it's only running unit tests, not tempest. | 14:36 |
| bnemec | And since most of our unit test bugs are due to missing cross-testing... :-) | 14:36 |
| dhellmann | bnemec: we're adding tempest gates to the libs, too | 14:36 |
| dhellmann | bnemec: that "integrated-gate" job template | 14:37 |
| fungi | bnemec: keep in mind that those other projects unit tests are suddenly becoming integration tests for the libraries though, so it will actually be similar (for you) to failing tempest tests for example | 14:37 |
| openstackgerrit | Victor Boivie proposed a change to openstack-infra/jenkins-job-builder: Support for matrix axis as a space-delimited string https://review.openstack.org/84733 | 14:37 |
| openstackgerrit | Victor Boivie proposed a change to openstack-infra/jenkins-job-builder: triggered-builds: Support property file injection https://review.openstack.org/84734 | 14:37 |
| fungi | at least from a pace of gating perspective | 14:37 |
| openstackgerrit | Victor Boivie proposed a change to openstack-infra/jenkins-job-builder: Job name override - used when storing https://review.openstack.org/84736 | 14:37 |
| bnemec | dhellmann: Oh, yeah that will be interesting. :-) | 14:39 |
| bnemec | fungi: Right, but the unit tests generally don't have as many heisenbugs, do they? | 14:39 |
| openstackgerrit | Victor Boivie proposed a change to openstack-infra/jenkins-job-builder: Support for matrix axis as a space-delimited string https://review.openstack.org/84733 | 14:39 |
| *** chandankumar_ has joined #openstack-infra | 14:40 | |
| openstackgerrit | Victor Boivie proposed a change to openstack-infra/jenkins-job-builder: triggered-builds: Support property file injection https://review.openstack.org/84734 | 14:40 |
| fungi | bnemec: you would (er, will!) be surprised | 14:41 |
| openstackgerrit | Victor Boivie proposed a change to openstack-infra/jenkins-job-builder: Job name override - used when storing https://review.openstack.org/84736 | 14:41 |
| fungi | no, probably not as many, but you're spreading the surface area for bugs to collect on by cross-testing against all projects | 14:42 |
| *** rcleere has joined #openstack-infra | 14:42 | |
| *** david-lyle has joined #openstack-infra | 14:43 | |
| bnemec | fungi: Yeah, I say that and then I think about the number of intermittent failures we've had in incubator since enabling the parallel testing and I start getting kind of nervous. :-) | 14:43 |
| bnemec | Between tripleo and oslo I think I'm going to see just about every bug that happens anywhere in OpenStack. ;-) | 14:44 |
| fungi | and then it will be time to join the release cycle management team ;) | 14:44 |
| dhellmann | well, maybe we should start slow and only add cross-test jobs for projects we consider to have stable unit tests? | 14:44 |
| * dhellmann wonders which projects those might be | 14:45 | |
| fungi | i think it should be fairly easy to turn different ones on and off while fine-tuning this in the beginning | 14:45 |
| fungi | probably easier to just start and identify the pain points rather than speculating | 14:45 |
| dhellmann | fungi: good point | 14:46 |
| dhellmann | we can make the unstable ones non-voting fairly easily | 14:46 |
| *** xchu has quit IRC | 14:46 | |
| *** wenlock has joined #openstack-infra | 14:47 | |
| *** ociuhandu has joined #openstack-infra | 14:49 | |
| *** thedodd has joined #openstack-infra | 14:50 | |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Report pecan changesets, merges, and +2s to #pecanpy https://review.openstack.org/83564 | 14:50 |
| *** thuc_ has joined #openstack-infra | 14:51 | |
| *** matsuhashi has quit IRC | 14:52 | |
| *** doddstack has joined #openstack-infra | 14:54 | |
| *** mgagne has joined #openstack-infra | 14:57 | |
| *** thuc has joined #openstack-infra | 14:57 | |
| *** thedodd has quit IRC | 14:57 | |
| fungi | bnemec: 83885 seems to have a layout issue | 14:57 |
| *** thuc_ has quit IRC | 14:58 | |
| *** xchu has joined #openstack-infra | 14:58 | |
| *** xchu has quit IRC | 14:58 | |
| fungi | bnemec: looks like oslo.test vs oslotest in the job names is the issue | 14:59 |
| bnemec | fungi: Whoops, yeah. I probably missed that in the rebase. I'll get a new patch set pushed. | 15:00 |
| openstackgerrit | A change was merged to openstack-infra/config: Parameterize library cross-test job https://review.openstack.org/83759 | 15:00 |
| *** GnomeKing is now known as BobBall | 15:00 | |
| *** Sukhdev has joined #openstack-infra | 15:00 | |
| openstackgerrit | A change was merged to openstack-infra/config: Report pecan changesets, merges, and +2s to #pecanpy https://review.openstack.org/83564 | 15:01 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Enable cross-testing of oslo.test with dib and t-i-e https://review.openstack.org/83885 | 15:02 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements https://review.openstack.org/83886 | 15:02 |
| fungi | bnemec: that looks better--thanks! | 15:03 |
| *** ArxCruz has joined #openstack-infra | 15:03 | |
| bnemec | fungi: np, thanks for looking at these | 15:03 |
| fungi | bnemec: thanks for writing them! | 15:04 |
| *** _TheDodd_ has joined #openstack-infra | 15:05 | |
| bnemec | It's been a learning experience. :-) | 15:05 |
| fungi | those are the best (only?) kind | 15:06 |
| *** dcramer_ has joined #openstack-infra | 15:06 | |
| dhellmann | fungi: thanks! | 15:06 |
| sdague | fungi: BRANCH_OVERRIDE question | 15:07 |
| *** david_lyle_ has joined #openstack-infra | 15:07 | |
| *** doddstack has quit IRC | 15:07 | |
| fungi | yup | 15:08 |
| *** dklyle has joined #openstack-infra | 15:08 | |
| *** hogepodge has joined #openstack-infra | 15:08 | |
| sdague | fungi: how would I specify in a job that we should be running master tempest with stable/havana everything else? | 15:08 |
| fungi | sdague: hmm, we may not have a variable yet for that exactly | 15:09 |
| fungi | oh, you know what, branch override should actually do that | 15:10 |
| *** dklyle has quit IRC | 15:10 | |
| mtaylor | yes. I think that it should | 15:10 |
| fungi | if you have a job with a branch override to havana and only run it for changes to tempest master, it ought to get you what you need | 15:10 |
| *** dklyle has joined #openstack-infra | 15:10 | |
| *** david-lyle has quit IRC | 15:10 | |
| fungi | the trick is when you want to test a nova change on havana with master tempest where i think this gets hairy | 15:10 |
| *** dklyle is now known as david-lyle | 15:11 | |
| *** e0ne_ has quit IRC | 15:11 | |
| fungi | so i think you may want to run it non-voting until you're ready to delete the stable/havana branch from tempest officially | 15:11 |
| sdague | fungi: so that's ok | 15:11 |
| *** david_lyle_ has quit IRC | 15:11 | |
| *** signed8bit has quit IRC | 15:11 | |
| fungi | otherwise it won't be easy to set up symmetric gating | 15:11 |
| sdague | this would be mostly a sniff test | 15:11 |
| *** signed8bit has joined #openstack-infra | 15:12 | |
| fungi | yep, then you can probably just reuse the jobs we have for testing d-g | 15:12 |
| *** skraynev is now known as skraynev_afk | 15:13 | |
| mtaylor | sdague: oh - are you working on one-tempest-to-rule-them-all? | 15:13 |
| fungi | mtaylor: see scrollback in -qa for lots of detail | 15:14 |
| fungi | also, you're looking very official today | 15:14 |
| sdague | mtaylor: pondering it | 15:14 |
| *** thuc has quit IRC | 15:15 | |
| phschwartz | fungi: agreed. I was confused at first to might be impersonating mtaylor | 15:15 |
| phschwartz | *who might | 15:15 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Immediately delete a floating IP if doesn't attach https://review.openstack.org/84614 | 15:15 |
| *** jaypipes has joined #openstack-infra | 15:15 | |
| *** thuc has joined #openstack-infra | 15:15 | |
| *** maxbit has quit IRC | 15:15 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tempest stable non-voting jobs https://review.openstack.org/84749 | 15:16 |
| sdague | fungi: so that? | 15:16 |
| *** ArxCruz has quit IRC | 15:16 | |
| *** vhoward- has left #openstack-infra | 15:16 | |
| *** mihgen has quit IRC | 15:18 | |
| fungi | sdague: very close | 15:18 |
| fungi | see comments | 15:19 |
| *** vponomaryov has quit IRC | 15:19 | |
| *** mtaylor is now known as mordred | 15:19 | |
| *** mordred has quit IRC | 15:19 | |
| *** mordred has joined #openstack-infra | 15:19 | |
| *** thuc has quit IRC | 15:20 | |
| *** dstanek has quit IRC | 15:20 | |
| mordred | ah. thatś better | 15:20 |
| *** hogepodge has quit IRC | 15:20 | |
| *** signed8bit has quit IRC | 15:20 | |
| *** signed8bit has joined #openstack-infra | 15:21 | |
| fungi | mordred: clearly an imposter. the mordred i know doesn't have an ś key on his keyboard | 15:21 |
| *** openstackgerrit has quit IRC | 15:21 | |
| *** openstackgerrit has joined #openstack-infra | 15:22 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tempest stable non-voting jobs https://review.openstack.org/84749 | 15:22 |
| sdague | fungi: cool, fixed | 15:22 |
| fungi | ḱ yes, but not ś | 15:22 |
| mordred | fungi: for some reason, my keyboard has decided to shift to compose keystrokes | 15:22 |
| sdague | if we could land that quickly, we'd like data today on this | 15:22 |
| fungi | mordred: that's always superfun | 15:22 |
| sdague | to help inform how terribad things are | 15:23 |
| *** SumitNaiksatam has joined #openstack-infra | 15:26 | |
| *** e0ne has joined #openstack-infra | 15:26 | |
| *** Ryan_Lane has joined #openstack-infra | 15:27 | |
| *** VijayTripathi has joined #openstack-infra | 15:28 | |
| *** maxbit has joined #openstack-infra | 15:29 | |
| *** hogepodge has joined #openstack-infra | 15:29 | |
| *** UtahDave has joined #openstack-infra | 15:29 | |
| *** gokrokve has joined #openstack-infra | 15:30 | |
| sdague | mordred: you want to send it through - https://review.openstack.org/#/c/84749/ ? | 15:30 |
| sdague | passed layout, and it's non voting jobs | 15:31 |
| openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Add support for URLTrigger plugin https://review.openstack.org/83524 | 15:31 |
| *** amotoki has joined #openstack-infra | 15:32 | |
| *** hogepodge has quit IRC | 15:32 | |
| *** maxbit has quit IRC | 15:34 | |
| zaro | morning | 15:35 |
| *** mrodden has quit IRC | 15:37 | |
| *** dkliban has quit IRC | 15:38 | |
| *** lcostantino has joined #openstack-infra | 15:39 | |
| zaro | mordred: is this what you wanted? http://paste.openstack.org/show/74863/ | 15:40 |
| mtreinish | sdague: does 84749 just work because the zuul ref is from tempest master? because we use the same jobs for the stable stuff too. | 15:41 |
| zaro | mordred: inline diff in email notifications. it's already in gerrit 2.9 | 15:41 |
| *** hashar has quit IRC | 15:42 | |
| fungi | mtreinish: yes, we use those same jobs to test devstack-gate master against stable/havana of the various projects. the expectation is that it will check out the zuul ref for the tempest master change, which will prevent tempest stable/havana from being used to run the tests | 15:43 |
| fungi | mtreinish: i say "hope" because we haven't tried before to make devstack-gate not use an existing branch which matches the override branch, afaik | 15:44 |
| *** jlibosva has quit IRC | 15:44 | |
| fungi | but if it doesn't dtrt there, i think we consider that a bug and solve it | 15:44 |
| mtreinish | fungi: ok that's what I thought, just wanted to confirm | 15:45 |
| sdague | mtreinish: yeh this isn't the full solution by any means | 15:45 |
| sdague | having this run on stable/havana devstack would be real work | 15:47 |
| sdague | that part of the matrix isn't supported in current d-g | 15:47 |
| sdague | but lets see the terribad | 15:47 |
| *** signed8bit has quit IRC | 15:48 | |
| *** talluri has quit IRC | 15:48 | |
| *** mrodden has joined #openstack-infra | 15:49 | |
| mordred | zaro: well... that's a good step forward - but I was really hoping for the whole diff - not just the parts with inline comments | 15:50 |
| zaro | mordred: isn't that excessive? | 15:54 |
| * fungi imagines the mailbomb you get from a many-thousand-line translation update proposal change | 15:55 | |
| *** hogepodge has joined #openstack-infra | 15:55 | |
| mordred | zaro: launchpad does it for merge requests and it's fine | 15:57 |
| *** thuc has joined #openstack-infra | 15:57 | |
| *** mdenny has joined #openstack-infra | 15:57 | |
| mordred | I mention it because sometimes I'm on my phone reading an email, and I might want to read through the content of the change - but I don't really want to launch gerrit in my phone | 15:57 |
| fungi | clearly google is missing a key integration opportunity there, by writing your code review system and your phone operating system... | 15:58 |
| *** dkliban has joined #openstack-infra | 15:59 | |
| *** zhiyan_ is now known as zhiyan | 15:59 | |
| mordred | :) | 16:00 |
| zaro | can't imagine that i would like to see entire diff on my phone but hey whatever floats your boat. | 16:00 |
| zaro | it's probably already there as an option, but if not shouldn't be too difficult to add that option. | 16:00 |
| * anteaya notes zaro is copping some attitude | 16:01 | |
| *** freyes has joined #openstack-infra | 16:01 | |
| * anteaya applaudes | 16:01 | |
| zaro | that's what 4yr olds will do to you ;) | 16:02 |
| anteaya | ha ha ha | 16:03 |
| *** thomasem has joined #openstack-infra | 16:03 | |
| mordred | zaro: to counfound you even more - I'd like to be able to reply to the email and have my comments added as comments on the change, and any in-line comments I made go to the appropriate place as in-line comments :) | 16:04 |
| * fungi needs to step out and run an errand, hopefully back in less than an hour | 16:04 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/reviewstats: Add the ironic-python-agent project https://review.openstack.org/84773 | 16:04 |
| devananda | jroll: ^ proposal to start counting stats for IPA in the Ironic program, now that Juno is open | 16:05 |
| mordred | devananda: you did that on purpose didn't you? | 16:05 |
| devananda | mordred: yes | 16:05 |
| devananda | mordred: because I cant drink beer | 16:05 |
| mordred | jeblair: ^^ I beleive devananda is beating turbo-hipster for most hipster still | 16:05 |
| devananda | LOL | 16:06 |
| *** yamahata has quit IRC | 16:06 | |
| zaro | mordred: now that would be nice! | 16:07 |
| pleia2 | good morning | 16:08 |
| *** e0ne has quit IRC | 16:09 | |
| *** e0ne has joined #openstack-infra | 16:09 | |
| *** dkranz has quit IRC | 16:09 | |
| *** alff has quit IRC | 16:09 | |
| *** alff_ has quit IRC | 16:09 | |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Allow the `test` command to write to stdout https://review.openstack.org/84778 | 16:10 |
| devananda | mordred: reply to gerrit-review-email? holy smokes, yes! | 16:10 |
| *** alff_ has joined #openstack-infra | 16:11 | |
| *** alff has joined #openstack-infra | 16:11 | |
| *** e0ne has quit IRC | 16:12 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: add build-timeout macro https://review.openstack.org/84637 | 16:12 |
| *** hogepodge has quit IRC | 16:13 | |
| jeblair | devananda: nicely done. cheers! | 16:13 |
| anteaya | pleia2: morning | 16:13 |
| *** lcostantino has quit IRC | 16:13 | |
| *** jgallard has quit IRC | 16:14 | |
| *** alff has quit IRC | 16:15 | |
| *** alff_ has quit IRC | 16:15 | |
| *** jooools has quit IRC | 16:15 | |
| *** coolsvap has joined #openstack-infra | 16:17 | |
| *** e0ne has joined #openstack-infra | 16:17 | |
| *** jpich has quit IRC | 16:18 | |
| openstackgerrit | A change was merged to openstack-infra/config: add tempest stable non-voting jobs https://review.openstack.org/84749 | 16:18 |
| *** coolsvap has quit IRC | 16:18 | |
| *** yolanda has quit IRC | 16:18 | |
| *** chandankumar_ has quit IRC | 16:20 | |
| *** e0ne has quit IRC | 16:20 | |
| *** e0ne has joined #openstack-infra | 16:21 | |
| zaro | SergeyLukjanov: do you have time for quick review? https://review.openstack.org/#/c/82921 | 16:23 |
| *** dkranz has joined #openstack-infra | 16:23 | |
| *** mrmartin has quit IRC | 16:25 | |
| *** e0ne has quit IRC | 16:25 | |
| *** zns has quit IRC | 16:27 | |
| *** ominakov has quit IRC | 16:31 | |
| *** derekh has quit IRC | 16:32 | |
| clarkb | morning | 16:33 |
| *** e0ne has joined #openstack-infra | 16:33 | |
| clarkb | dhellmann: to be fair the unit tests were already broken :) oslo only tickled that | 16:33 |
| anteaya | morning clarkb | 16:34 |
| *** e0ne_ has joined #openstack-infra | 16:34 | |
| clarkb | this table is not level | 16:34 |
| clarkb | and it bothers me | 16:34 |
| pleia2 | heh | 16:34 |
| *** e0ne has quit IRC | 16:35 | |
| anteaya | folded paper is my table leveler of choice | 16:36 |
| anteaya | junk mail and flyers come in handy for that | 16:36 |
| clarkb | anteaya: the low end is bolted to the wall and the legs don't appear to get shorter | 16:36 |
| clarkb | this is a very curious installation | 16:37 |
| anteaya | wow | 16:37 |
| anteaya | where are you? | 16:37 |
| clarkb | anteaya: seattle office, I can show you if you are ever here | 16:37 |
| anteaya | who bolts a table to the wall without leveling it first | 16:37 |
| *** bhuvan has joined #openstack-infra | 16:37 | |
| anteaya | ah the incentive to visit the mothershiop | 16:37 |
| anteaya | I was actually thinking of showing up for a bit of time in august | 16:38 |
| anteaya | I bet the unlevel table will still be in existance then | 16:38 |
| *** alff has joined #openstack-infra | 16:38 | |
| *** alff_ has joined #openstack-infra | 16:38 | |
| *** blamar_ has joined #openstack-infra | 16:41 | |
| *** blamar has quit IRC | 16:42 | |
| *** blamar_ is now known as blamar | 16:42 | |
| *** markmcclain has joined #openstack-infra | 16:45 | |
| *** yolanda has joined #openstack-infra | 16:46 | |
| dhellmann | clarkb: well, true, but we keep finding problems like that after a lib is released rather than before | 16:46 |
| *** CaptTofu has quit IRC | 16:47 | |
| jroll | devananda: on https://review.openstack.org/#/c/84773/ does it matter that the core team is different for API? | 16:47 |
| devananda | jroll: ooh. good question | 16:47 |
| *** bknudson has quit IRC | 16:47 | |
| jroll | er, s/API/IPA | 16:47 |
| devananda | question for folks re: reviewstats repo. when a program has different core teams for different projects, how should that be represented? | 16:48 |
| *** basha has joined #openstack-infra | 16:51 | |
| *** zhiyan is now known as zhiyan_ | 16:51 | |
| *** gokrokve has quit IRC | 16:52 | |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: jenkins-job-builder: Allow `test` write to stdout https://review.openstack.org/84778 | 16:55 |
| fungi | devananda: mordred: at the summit we can have pbr vs ipa drink-offs. i think i'll have to side with the ironic crowd for that event | 16:55 |
| devananda | fungi: we need a whiskey project next | 16:56 |
| *** Ryan_Lane1 has joined #openstack-infra | 16:56 | |
| fungi | devananda: it's pronounced "wsgi" but yes | 16:57 |
| devananda | ah, of course | 16:57 |
| *** talluri has joined #openstack-infra | 16:58 | |
| *** Sukhdev has quit IRC | 17:00 | |
| devananda | fungi: question on core team things. do any programs have different core teams for different projects? is there an easy way for me to check this? | 17:00 |
| *** BobBall is now known as BobBallAway | 17:01 | |
| jogo | ttx: thanks | 17:01 |
| mordred | devananda: usually the program core team will be included in a specialied core team | 17:02 |
| *** basha has quit IRC | 17:02 | |
| fungi | devananda: there are times when they are completely separate, i think, though no good examples spring to mind | 17:02 |
| fungi | mostly not at least, it's more of what mordred describes | 17:03 |
| fungi | devananda: so, like, ipa-core might have ironic-core as an included group | 17:03 |
| devananda | ah, gotcha | 17:04 |
| fungi | devananda: and ipa-ptl might include both the ipa lead developer and the ironic ptl for oversight | 17:04 |
| devananda | how is taht modeled in reviewstats? just lump it all together? | 17:04 |
| fungi | devananda: i think so, yes. russellb could probably tell you with more certainty | 17:04 |
| devananda | i'm looking eg. at projects/tripleo.json, and it's just one list for core-team | 17:05 |
| *** jcoufal has quit IRC | 17:05 | |
| JayF | devananda: http://omissionbeer.com/ this group makes a damn good gf beer, although I haven't tried the IPA because I'm more of a lager guy :D | 17:05 |
| mordred | yeah - it doesn't get broken out by subteam | 17:05 |
| JayF | </late_response> | 17:05 |
| devananda | JayF: so i've heard. haven't tried their beers yet, tho | 17:06 |
| *** gokrokve has joined #openstack-infra | 17:06 | |
| fungi | devananda: got it. i've got this https://review.openstack.org/80437 script which could calculate the actual core reviewers (approvers) per repository, but it relies on mysql queries against the gerrit backend db at the moment | 17:06 |
| JayF | devananda: All gf beers I've had have been terribad, except those. They actually taste like beer :) | 17:06 |
| *** talluri_ has joined #openstack-infra | 17:06 | |
| devananda | JayF: "All gf beers I've had have been terribad" <-- yes. | 17:07 |
| fungi | JayF: devananda: i rather like echigo koshihikari (they make a good rice lager and rice stout) | 17:07 |
| devananda | hence why i stick to wsgi | 17:07 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add a fingerprint for bug 1301027 https://review.openstack.org/84558 | 17:07 |
| uvirtbot | Launchpad bug 1301027 in neutron/havana "Permission denied error in unit tests" [High,In progress] https://launchpad.net/bugs/1301027 | 17:07 |
| *** talluri has quit IRC | 17:08 | |
| devananda | fungi: thanks, but i think folks have answered my question | 17:08 |
| devananda | fungi: i'll update my proposal to reviewstats shortly | 17:08 |
| jogo | clarkb: is https://review.openstack.org/#/c/84558/ bette rnow? | 17:09 |
| *** roeyc has quit IRC | 17:10 | |
| *** dcramer_ has quit IRC | 17:11 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/reviewstats: Add the ironic-python-agent project https://review.openstack.org/84773 | 17:12 |
| devananda | jroll: ^ updated to include ya'll in the core stats | 17:13 |
| devananda | jroll: please verify that I got all the gerrit account names correct | 17:13 |
| JayF | I'll take a gander | 17:14 |
| devananda | thanks | 17:14 |
| *** Sukhdev has joined #openstack-infra | 17:14 | |
| JayF | these should be gerrit user ids? | 17:14 |
| jroll | devananda: ^ | 17:16 |
| devananda | I *think* so. russelb? | 17:16 |
| JayF | jroll: assuming yours as jroll is accurate? | 17:16 |
| jroll | gerrit ID? no, it's like Jim-Rollenhagen, I think | 17:17 |
| jroll | JayF: and yours is Jason-Oldos iirc? | 17:17 |
| JayF | in /gerrit/ it's not the same as in /launchpad/ | 17:17 |
| JayF | in gerrit mine is JayF | 17:17 |
| devananda | right | 17:17 |
| JayF | jroll: review.openstack.org -> login -> "settings" -> What's the user name in the top row of the table? | 17:18 |
| jroll | ahhhh | 17:18 |
| jroll | jimrollenhagen | 17:18 |
| jroll | I'll post a comment | 17:18 |
| JayF | I've got it | 17:18 |
| devananda | if you 'git remote -v | grep gerrit', what's the user name | 17:18 |
| jroll | yeah, jimrollenhagen | 17:19 |
| devananda | cool | 17:19 |
| *** signed8bit has joined #openstack-infra | 17:20 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:23 | |
| * SergeyLukjanov was commuting to home from office | 17:24 | |
| SergeyLukjanov | zaro, looking on https://review.openstack.org/#/c/82921 | 17:24 |
| russellb | devananda: yes | 17:25 |
| devananda | thanks | 17:25 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/reviewstats: Add the ironic-python-agent project https://review.openstack.org/84773 | 17:25 |
| *** thuc_ has joined #openstack-infra | 17:27 | |
| *** amotoki has quit IRC | 17:28 | |
| JayF | devananda: ^ what does that do, exactly? I know about stackalytics, but didn't know about reviewstats | 17:28 |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Treat input filename of "-" to mean stdin https://review.openstack.org/84803 | 17:28 |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Document using "-" to mean stdin or stdout https://review.openstack.org/84804 | 17:28 |
| *** e0ne has joined #openstack-infra | 17:29 | |
| *** dizquierdo has quit IRC | 17:29 | |
| *** medieval1_ has joined #openstack-infra | 17:29 | |
| devananda | JayF: reviewstats tells stackalytics what to base its stats on, I beleive | 17:30 |
| devananda | JayF: so eg, in Icehouse, IPA wasn't part of the ironic program, so it's contributions were not counted in stackalytics.com/?release=icehouse&metric=commits&project_type=openstack&module=ironic-group | 17:30 |
| *** SumitNaiksatam_ has joined #openstack-infra | 17:30 | |
| *** e0ne has quit IRC | 17:31 | |
| *** thuc has quit IRC | 17:31 | |
| mordred | devananda: reviewstats and stackalytics are not related | 17:31 |
| *** markmcclain has quit IRC | 17:31 | |
| devananda | mordred: oh? | 17:31 |
| devananda | humm | 17:31 |
| devananda | mordred: where does stackalytics get it's list of core reviewers from? | 17:31 |
| SpamapS | Ok, ci-overcloud _seems_ to be working | 17:31 |
| mordred | devananda: from people who cast a +2 or a +A or a -2 vote | 17:31 |
| devananda | mordred: and where does it get the list of projects in a "project group" from? | 17:32 |
| fungi | devananda: i think they scrape gitdm and a few other sources to piece something together (possibly the gerrit webui) | 17:32 |
| SpamapS | not sure who to ping to ask to have nodepool try again. :-P | 17:32 |
| devananda | hm, ok | 17:32 |
| mordred | devananda: there isa json file in the stackalytics repo | 17:32 |
| mordred | fungi: they do not | 17:32 |
| *** ociuhandu_ has joined #openstack-infra | 17:32 | |
| mordred | they pull info from gerrit, and then they infer core status as above | 17:32 |
| fungi | mordred: i thought they did initially to put together that json file when they started | 17:33 |
| devananda | mordred: and projects in a group? | 17:33 |
| mordred | in a json file | 17:33 |
| *** nibalize1 has joined #openstack-infra | 17:33 | |
| devananda | ah | 17:33 |
| mordred | etc/default_data.json | 17:33 |
| fungi | though ideally they should fix it to identify project->program mappings from the governance repo | 17:33 |
| *** markmcclain has joined #openstack-infra | 17:33 | |
| fungi | now that we have that | 17:33 |
| *** signed8bit has quit IRC | 17:34 | |
| devananda | reviewstats probably should pull from there too :) | 17:34 |
| devananda | JayF: to answer your question -- http://russellbryant.net/openstack-stats/ironic-reviewers-30.txt | 17:34 |
| mordred | fungi: ++ | 17:34 |
| JayF | I appear to be losing to josh and jroll, I'd better review more code ;) | 17:35 |
| rcarrillocruz | fungi: hi, do you have +2 powers? if so, and if you are not busy now, could you please have a look when you have a minute https://review.openstack.org/#/c/79205/ ? | 17:35 |
| *** harlowja_away is now known as harlowja | 17:35 | |
| *** german_ has joined #openstack-infra | 17:36 | |
| fungi | rcarrillocruz: ahh, yep--i already had that one open in a tab. just a sec and i should be done reviewing it | 17:36 |
| *** adarazs_ has joined #openstack-infra | 17:37 | |
| *** gokrokve_ has joined #openstack-infra | 17:37 | |
| *** talluri_ has quit IRC | 17:38 | |
| *** gokrokve has quit IRC | 17:38 | |
| *** e0ne_ has quit IRC | 17:38 | |
| *** UtahDave has quit IRC | 17:38 | |
| *** SumitNaiksatam has quit IRC | 17:38 | |
| *** ociuhandu has quit IRC | 17:38 | |
| *** adarazs has quit IRC | 17:38 | |
| *** matrohon has quit IRC | 17:38 | |
| *** nprivalova has quit IRC | 17:38 | |
| *** tchaypo has quit IRC | 17:38 | |
| *** medieval1 has quit IRC | 17:38 | |
| *** nibalizer has quit IRC | 17:38 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 17:38 | |
| *** ociuhandu_ is now known as ociuhandu | 17:38 | |
| fungi | rcarrillocruz: lgtm | 17:38 |
| rcarrillocruz | sweet , thx sir | 17:39 |
| german_ | Hi, | 17:39 |
| german_ | I am new to all this and have a few questions | 17:39 |
| SergeyLukjanov | fungi, devananda, stackalytics is already using governance repo for program -> project mapping | 17:40 |
| german_ | I am trying to update a branch for the libra stack-forge project by cherry picking things from the main branch. But zuul id failing them since pep8 got more strict | 17:40 |
| fungi | SergeyLukjanov: excellent! | 17:41 |
| *** alff_ has quit IRC | 17:41 | |
| *** alff has quit IRC | 17:41 | |
| *** jcoufal has joined #openstack-infra | 17:41 | |
| german_ | is there a way to bypass the pep8 checks? | 17:41 |
| fungi | german_: you might want to pin pep8 in the test-requirements.txt on the stable branch to an acceptable version from when that branch was cut | 17:42 |
| SergeyLukjanov | german_, you need to fix the hacking version (pep8/flake8 probably too) for stable branch | 17:42 |
| SergeyLukjanov | to avoid addiitional checks | 17:43 |
| fungi | german_: or if there are specific pep8 errors you want to silence, you can add an exclusion list in the tox.ini for those specifically | 17:43 |
| mordred | SergeyLukjanov: oh! I didn't realize it had started using that. neat | 17:43 |
| SergeyLukjanov | mordred, I think it's using gov for a few month | 17:43 |
| mordred | SergeyLukjanov: neat | 17:44 |
| german_ | well, git cherry-pick makes the system commit things before I can add changes | 17:44 |
| *** tchaypo has joined #openstack-infra | 17:44 | |
| SergeyLukjanov | german_, you need to make a separated change for stable branch that'll fix gating and make you able to cherry-pick stuff | 17:45 |
| *** basha_ has joined #openstack-infra | 17:45 | |
| mordred | devananda: you know there is a similar page here, yeah: http://stackalytics.com/report/contribution/ironic-group/30 | 17:45 |
| *** VijayTripathi has quit IRC | 17:46 | |
| german_ | thanks SergeyLukjanov I will give that a try | 17:46 |
| zaro | fungi: would like to get this thru so i can do further testing, just a quick review. https://review.openstack.org/#/c/82921 | 17:46 |
| zaro | SergeyLukjanov: thanks for the review. | 17:46 |
| fungi | sdague: i assume you meant to ping me here and not in the vmwareapi meeting | 17:47 |
| sdague | oops | 17:47 |
| SergeyLukjanov | zaro, np, I hope that version-properties are already in place | 17:47 |
| sdague | my tabs switched locations | 17:47 |
| *** nprivalova has joined #openstack-infra | 17:47 | |
| fungi | heh | 17:47 |
| *** dcramer_ has joined #openstack-infra | 17:47 | |
| *** matrohon has joined #openstack-infra | 17:47 | |
| sdague | this is my 3rd openstack... tab, meetings is 4th | 17:48 |
| zaro | SergeyLukjanov: yes, many existing jobs are currently using it. | 17:48 |
| devananda | SergeyLukjanov: so, we added ironic-python-agent to governance repo, and its in the ironic group now, but its core reviewers aren't (*)'d | 17:48 |
| SergeyLukjanov | zaro, okay | 17:49 |
| sdague | fungi: http://logs.openstack.org/86/84786/1/check/check-tempest-dsvm-full-havana/717dcd0/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-04-02_17_01_16_901 | 17:49 |
| asselin | Hi, I have a question about the nodepool puppet script. I'm running behind a proxy and working through those issues. There's a 'catalog' script that runs for 4550 seconds (75 minutes) constantly failing due to dns issue. Sure enough the settings in /etc/resolv.conf were no longer valid. Then, all of the sudden it works. The contents of this file were restored by the script. Any ideas why it would do this? http://paste.openstack.org/sho | 17:49 |
| asselin | w/74884/ | 17:49 |
| asselin | http://paste.openstack.org/show/74884/ | 17:49 |
| openstackgerrit | A change was merged to openstack-infra/config: gerrit plugin builds need to use version-properties.sh https://review.openstack.org/82921 | 17:50 |
| SergeyLukjanov | devananda, I've checked all CRs for your agent and it looks like http://stackalytics.com/report/contribution/ironic-python-agent/30 shows all cores | 17:52 |
| SergeyLukjanov | devananda, I mean all folks how placed +2 in this repo | 17:52 |
| devananda | SergeyLukjanov: yes, but http://www.stackalytics.com/report/contribution/ironic-group/30 does not | 17:52 |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Add stable filters for neutron functional+api jobs https://review.openstack.org/84815 | 17:53 |
| devananda | SergeyLukjanov: even though, in governance repo, both python-ironicclient and ironic-puthon-agent are part of the program | 17:53 |
| devananda | SergeyLukjanov: maybe I misunderstand something | 17:53 |
| openstackgerrit | A change was merged to openstack-infra/gerritbot: Add support for notification of releases on IRC https://review.openstack.org/79205 | 17:53 |
| SergeyLukjanov | devananda, oh, got it, I'll ping stackalytics maintainers tomorrow | 17:53 |
| *** Ryan_Lane1 has quit IRC | 17:54 | |
| *** Ryan_Lane1 has joined #openstack-infra | 17:54 | |
| devananda | SergeyLukjanov: thanks! fwiw,w e're discussing merging the review teams now. so the separate core group may go away (or may become a subgroup?) at some point | 17:55 |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: jenkins-job-builder: Allow read/write stdin/stdout https://review.openstack.org/84816 | 17:55 |
| *** _TheDodd_ has quit IRC | 17:55 | |
| zaro | sdague: with these three changes you should be able to modify your jobs to access a 'BUILD_TIMEOUT' env var: 84512, 84630, 84637 | 17:55 |
| *** nicedice has joined #openstack-infra | 17:55 | |
| SergeyLukjanov | devananda, FWIW IMO core review teams for subprojects should contain main core review team | 17:55 |
| fungi | sdague: okay, i see why... http://logs.openstack.org/86/84786/1/check/check-tempest-dsvm-full-havana/717dcd0/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-04-02_17_01_16_810 | 17:56 |
| fungi | sdague: zuul builds refs based on branch+change | 17:56 |
| JayF | devananda: this exists https://github.com/stackforge/stackalytics/blob/master/etc/test_programs.yaml in that stackalytics repo, but is named test. hmm. markwash might have had to solve this same problem for Glance? | 17:56 |
| fungi | sdague: and so devstack-gate is trying to fetch refs/zuul/stable/havana/Zb19b0edd1f1c46b188b5e1cd8bc5f69b instead of refs/zuul/master/Zb19b0edd1f1c46b188b5e1cd8bc5f69b | 17:56 |
| fungi | sdague: i think we can fix this in the setup script. just a minute i'll see if i can work out what variable needs swapping | 17:57 |
| *** UtahDave has joined #openstack-infra | 17:57 | |
| sdague | fungi: well, I wonder if what we should do is provide a TEMPEST_BRANCH_OVERRIDE | 17:57 |
| JayF | devananda: aha, it appears to be using the governace stuff. So it should be right? https://github.com/stackforge/stackalytics/blob/016b39237a2f2a425da88456ce36e2bc6e69931f/stackalytics/processor/config.py#L43 | 17:57 |
| sdague | because BRANCH_OVERRIDE won't be able to make us have tempest master on stable/havana devstack | 17:58 |
| sdague | which we'll need pretty quickly to fix issues | 17:58 |
| fungi | sdague: well, it will *as soon as* we delete tempest's stable/havana branch | 17:58 |
| sdague | because most of those will be devstack fixes for generating the tempest config | 17:58 |
| *** pdmars_ is now known as pdmars | 17:58 | |
| fungi | sdague: but agreed, if we want to test it from non-tempest proposed changes before deleting that branch, then we need a separate variable | 17:59 |
| sdague | fungi: well, I expect there will be a long interim period for fixes | 17:59 |
| *** thunder_ has joined #openstack-infra | 17:59 | |
| fungi | sdague: however, this has still exposed what i think is technically a bug in devstack-gate, that it tries to assemble the zuul ref for the zuul project based on the override branch rather than the change branch | 18:00 |
| fungi | which will never work afaik | 18:00 |
| fungi | since zuul builds that ref to encode the starting branch name for the change being merged | 18:00 |
| fungi | we don't normally notice it because the places where we rely on that behavior we just have a master branch on the project, or at least no branch matching the name of the override branch | 18:01 |
| sdague | fungi: ok, how would you like to proceed? | 18:02 |
| *** bknudson has joined #openstack-infra | 18:02 | |
| sdague | I feel like we'll need a TEMPEST_OVERRIDE_BRANCH rsn anyway | 18:02 |
| sdague | so my instinct is to do that | 18:03 |
| fungi | sdague: two-phase. fix this logic bug (which without having looked yet seems like it should be fairly trivial) and then introduce a separate variable like tempest_branch | 18:03 |
| *** maxbit has joined #openstack-infra | 18:03 | |
| fungi | i'll try to whip out the bug fix real quick | 18:03 |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: jenkins-job-builder: Allow read/write stdin/stdout https://review.openstack.org/84820 | 18:03 |
| fungi | working on it now | 18:03 |
| sdague | fungi: ok, I'll look at the larger override | 18:03 |
| fungi | sdague: great | 18:03 |
| *** dcramer_ has quit IRC | 18:05 | |
| *** mihgen has joined #openstack-infra | 18:06 | |
| *** zns has joined #openstack-infra | 18:07 | |
| *** signed8bit has joined #openstack-infra | 18:07 | |
| *** dstanek has joined #openstack-infra | 18:08 | |
| *** zul has quit IRC | 18:09 | |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: jenkins-job-builder: Allow read/write stdin/stdout https://review.openstack.org/84778 | 18:10 |
| *** ociuhandu has quit IRC | 18:11 | |
| *** andrew_plunk has joined #openstack-infra | 18:12 | |
| *** zul has joined #openstack-infra | 18:12 | |
| *** melwitt has joined #openstack-infra | 18:12 | |
| andrew_plunk | hello all. I am trying to set up third party testing for the openstack heat project at Rackspace, I have been following the documentation here: http://ci.openstack.org/third_party.html#the-jenkins-gerrit-trigger-plugin-way | 18:14 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: OVERRIDE_BRANCH should not apply to ZUUL_PROJECT https://review.openstack.org/84828 | 18:15 |
| fungi | sdague: i *think* the fix is that ^ trivial | 18:15 |
| *** denis_makogon_ has joined #openstack-infra | 18:15 | |
| andrew_plunk | I would like to set up a service account, I sent an email with the required information to the mailing list, but it looks like that post is waiting for moderation, would anyone mind taking a look at that? | 18:15 |
| fungi | andrew_plunk: this one? http://lists.openstack.org/pipermail/openstack-infra/2014-March/001126.html | 18:16 |
| andrew_plunk | correct fungi | 18:16 |
| sdague | fungi: does that break other parts of stable? | 18:16 |
| sdague | like if we are actually testing stable nova | 18:17 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: allow for generic branch overrides https://review.openstack.org/84829 | 18:17 |
| fungi | andrew_plunk: sorry for the silent treatment--we tend to batch these up and i hadn't gotten to it yet. that one looks like it has all the info i need, so just a sec and i'll take care of it | 18:17 |
| sdague | fungi: can you look at that? ^^^ | 18:17 |
| andrew_plunk | I really appreciate it fungi! | 18:18 |
| sdague | because I think it will give us the flexibility to do this on any project | 18:18 |
| fungi | sdague: how are you thinking about that fix breaking tests of changes to stable nova? | 18:18 |
| *** denis_makogon has quit IRC | 18:18 | |
| sdague | fungi: I'm not sure, my brain is vaguely jumbled on the branch flips right now | 18:18 |
| *** dcramer_ has joined #openstack-infra | 18:18 | |
| sdague | so probably ignore me :) | 18:18 |
| fungi | no, not ignoring you--my thinkmeat gets overcooked just trying to ponder it as well | 18:19 |
| fungi | your concern may be legit | 18:19 |
| fungi | i'm just not spotting what it would be | 18:19 |
| jeblair | fungi, sdague: are you considering deleting branches from tempest? | 18:20 |
| *** e0ne has joined #openstack-infra | 18:20 | |
| sdague | jeblair: we are collecting data | 18:20 |
| sdague | jeblair: mostly there is an interesting question on whether or not we should create an icehouse branch for tempest | 18:20 |
| jeblair | sdague: is there previous discussion you can point me to where i can catch up? | 18:20 |
| sdague | -qa from this morning | 18:21 |
| fungi | jeblair: right, lots of scrollback on it earlier in #-qa | 18:21 |
| jeblair | i'll try to do that real quick like | 18:21 |
| sdague | cool | 18:21 |
| fungi | jeblair: summary is, wouldn't it be nice if tempest didn't have branches and just got backward-compat tested against everything. the trouble is in the getting there | 18:21 |
| jeblair | fungi, sdague: yeah, the top layer of my thinkmeat likes the way those words sound. :) | 18:22 |
| sdague | right, because openstack versions end up being kind of senseless in deployed clouds anyway | 18:23 |
| sdague | the api should be the api | 18:23 |
| jeblair | ++ | 18:23 |
| fungi | the idea behind it is that it would make tempest more usable by third parties without having to run a tempest branch closely-related to the deployed cloud release (and how you even figure that out as a typical cloud admin) | 18:24 |
| *** alff_ has joined #openstack-infra | 18:24 | |
| *** alff has joined #openstack-infra | 18:24 | |
| sdague | right, you'd just run tempest, the newest verison | 18:24 |
| *** maxbit has quit IRC | 18:24 | |
| sdague | and we could even have tempest versions for convenience, but they wouldn't be branch versions, because it should be always working like the clients | 18:25 |
| sdague | anyway, step #1 is data | 18:25 |
| sdague | how bad would it be to get tempest master to work with stable/havana | 18:25 |
| fungi | but also, we're in a very briefly useful window where we have only one supported stable branch (havana) so if the delta in making tempest master work with havana isn't huge-huge, then yeah | 18:25 |
| sdague | to figure out our existing drift | 18:25 |
| jeblair | gotcha (in fact, that's exactly where i am in my scrollback reading :) | 18:25 |
| sdague | :) | 18:26 |
| fungi | sdague: interesting dynamic with the eval in 84829. looks useful and would allow us to also skirt the current bug if we don't want to fix it (but might make the fix for it more complex if we do) | 18:28 |
| sdague | fungi: yeh, I decided that it was probably worth trying to do that dynamically so if we had other overrides we wanted the infra would already be there | 18:29 |
| sdague | and some eval stuff crept into devstack recently | 18:29 |
| fungi | sdague: it's basically the inverse of the OVERRIDE_BRANCH envvar | 18:29 |
| sdague | yes... I think that's right | 18:30 |
| fungi | which should override the branch of everything except the project tested | 18:30 |
| *** maxbit has joined #openstack-infra | 18:30 | |
| fungi | yours should override only the branch of the project tested | 18:30 |
| mtreinish | krtaylor: your powerkvm ci is screwed up on stable branches: http://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/84755/2/gate-ibm-tempest-dsvm-full/42f205a | 18:30 |
| fungi | er, only the branch of a specific project | 18:30 |
| *** denis_makogon_ is now known as denis_makogon | 18:30 | |
| mtreinish | krtaylor: it's not pulling any of the stable branches for the other projects so you get tons of failures | 18:30 |
| fungi | (which i think due to the other bug, also cannot be the project being tested) | 18:30 |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Add .travis.yml for Travis CI https://review.openstack.org/84834 | 18:31 |
| fungi | sdague: though i think trying to shoehorn that into the existing dsvm job templates might be hairy. we probably are stuck doing project-specific branch overrides in separate job definitions instead | 18:32 |
| mordred | uhm | 18:32 |
| mordred | why would someone add .travis.yaml files? | 18:32 |
| fungi | mordred: isn't travis used to test everything? | 18:33 |
| mordred | I just -2'd that | 18:33 |
| fungi | we're replacing zuul with it | 18:33 |
| sdague | mordred: it wasn't even Apr 1 | 18:33 |
| mordred | sdague: right? | 18:33 |
| fungi | i figured they were just late with that one | 18:33 |
| jeblair | mordred: lol | 18:33 |
| * fungi proposes a patch to add vim style comment lines into emacs sources | 18:34 | |
| sdague | fungi: so you are probably right that we'd need new jobs for this | 18:35 |
| sdague | however, I'm ok with that for now to get it working, because we'll minimally need this on devstack + tempest | 18:35 |
| pabelanger | so, are you guys having a problem with puppetlabs apt repo too? | 18:35 |
| pabelanger | puppet-common : Depends: facter (< 2.0) but 2.0.1-1puppetlabs1 is to be installed | 18:35 |
| fungi | pabelanger: see fixes to openstack-infra/config yesterday, but yes | 18:36 |
| pabelanger | fungi, danke | 18:36 |
| sdague | there is also a really interesting side effect of master only tempest | 18:36 |
| fungi | sdague: right, these jobs are transitional, but the feature which would support them is nice to keep as a parting gift | 18:36 |
| sdague | in that people would notice when stable branches were broken much faster | 18:36 |
| *** thunder_ has quit IRC | 18:37 | |
| *** ociuhandu has joined #openstack-infra | 18:37 | |
| *** hashar has joined #openstack-infra | 18:38 | |
| jeblair | sdague, fungi: i think i'm caught up | 18:38 |
| *** mihgen has quit IRC | 18:39 | |
| fungi | andrew_plunk: any objection to me making the human readable name "RAX Heat CI" and the ssh username "raxheatci"? | 18:39 |
| sdague | and... | 18:39 |
| sdague | jeblair: curious on your thoughts | 18:39 |
| andrew_plunk | sounds good to me fungi | 18:39 |
| jeblair | sdague: what do you think the invocation will look like? changes to tempest master set OVERRIDE_ZUUL_BRANCH=stable/havana and then set OVERRIDE_TEMPEST_BRANCH to master? | 18:40 |
| jeblair | (for that single job, obviously) | 18:40 |
| sdague | yes | 18:40 |
| sdague | well, actually, we'd probably carve a new job | 18:41 |
| jeblair | yeah | 18:41 |
| sdague | which would override_tempest_branch=master | 18:41 |
| *** __afazekas is now known as afazekas | 18:41 | |
| *** chandankumar_ has joined #openstack-infra | 18:41 | |
| sdague | and put it nv on tempest and devstack | 18:41 |
| fungi | sdague: jeblair: i *think* 84828 will at least get the current devstack-gate backward-compat jobs working to glean initial results on tempest master changes | 18:42 |
| jeblair | so it's a hierarchical override: the general variable overrides the preferred branch for all projects, and the more specific variable takes precedence over that | 18:42 |
| sdague | yes | 18:42 |
| sdague | jeblair: https://review.openstack.org/#/c/84829/1/functions.sh | 18:42 |
| jeblair | sdague: yep, just making sure i understand the intended use correctly | 18:43 |
| sdague | that's my first take on the inner override | 18:43 |
| sdague | yep | 18:43 |
| jeblair | sdague: okay, i like the idea and i think it will work. i'm going to drop a few code organization type comments in there in a sec | 18:44 |
| sdague | sure | 18:44 |
| *** yassine has quit IRC | 18:45 | |
| *** _nadya_ has joined #openstack-infra | 18:45 | |
| *** thuc_ has quit IRC | 18:45 | |
| *** thuc has joined #openstack-infra | 18:46 | |
| *** nibalize1 is now known as nibalizer | 18:46 | |
| *** thuc has quit IRC | 18:48 | |
| *** thuc has joined #openstack-infra | 18:48 | |
| *** thuc has quit IRC | 18:48 | |
| *** thuc has joined #openstack-infra | 18:49 | |
| *** alexpilotti has quit IRC | 18:50 | |
| *** _nadya_ has quit IRC | 18:52 | |
| adam_g | got a quick one that could use another peek if anyone has a minute. https://review.openstack.org/#/c/84543/ | 18:52 |
| adam_g | interested to see if the UCA icehouse pocket will really break the world | 18:52 |
| fungi | adam_g: well, there are other ways, have you sync'd up with dims on the test jobs he's been running with tempest? | 18:53 |
| adam_g | fungi, alternatives to DEVSTACK_GATE_TEMPEST_REGEX? | 18:53 |
| krtaylor | mtreinish, thanks, I'm on it | 18:54 |
| *** chandankumar_ has quit IRC | 18:55 | |
| *** _TheDodd_ has joined #openstack-infra | 18:56 | |
| *** gokrokve_ has quit IRC | 18:56 | |
| fungi | adam_g: what dims was doing to test uca was proposing a wip devstack change which added the repo early in setup, and then he checked the tempest test results from rerunning it a bunch | 18:57 |
| jeblair | sdague: okay, reviewed. not as many code org comments as i thought actually. mostly just "please add docs and tests" by the end of it. :) | 18:57 |
| *** marun is now known as maru_afk | 18:57 | |
| fungi | adam_g: which didn't require temporarily modifying a job to get some idea of whether or not it worked | 18:57 |
| *** Sukhdev has quit IRC | 18:58 | |
| adam_g | fungi, oh, well.. im mostly more curious than anything, we actually need a newer libvirt regardless. we are setting up the havana UCA currently, but that package is slow to get the bug updates we need. the trusty/icehouse UCA has it and other fixes released | 18:58 |
| *** zehicle_at_dell has joined #openstack-infra | 18:58 | |
| fungi | adam_g: ahh, got it | 18:58 |
| jeblair | mordred: your -2 got a response | 18:59 |
| *** zehicle has quit IRC | 19:00 | |
| clarkb | wow sb | 19:01 |
| jeblair | sdague: also, it looks like there's a technical error in your patch since it doesn't pass the current unit tests: https://jenkins01.openstack.org/job/gate-devstack-gate-unittests/64/console | 19:01 |
| *** rwsu has quit IRC | 19:01 | |
| clarkb | why are we wanting to test havana against master tempest? I think i miseed that | 19:01 |
| jeblair | clarkb: the long answer is in -qa scrollback; the short answer is that we want to see if we can drop all branches from tempest and have it master-only (since the api is the api) | 19:02 |
| *** mrmartin has joined #openstack-infra | 19:02 | |
| clarkb | jeblair: the swift failure last night is a clear indication that we can't | 19:02 |
| clarkb | (not that those issues can't be fixed) | 19:02 |
| sdague | clarkb: so yeh, you have such sb | 19:02 |
| *** talluri_ has joined #openstack-infra | 19:02 | |
| sdague | jeblair: ok, all valid points in review, I'll circle around on that shortly | 19:03 |
| sdague | I also mostly made it lc because the project name was going to be lc | 19:03 |
| jeblair | sdague: awesome; i'll get lunch! :) | 19:03 |
| sdague | and it seemed less confusing in some way | 19:03 |
| jeblair | sdague: yeah, no great answer there -- i will say that we went with the (possibly) mixed case with the zuul parameters for swift log uploads | 19:04 |
| sdague | it's also a local variable | 19:04 |
| sdague | so lc seems ok | 19:04 |
| sdague | at least that was the convention we use in devstack | 19:05 |
| jeblair | that's a similar situation (all current ZUUL_* params are uppercase, but swift container names might be lower, so they could end up as SWIFT_something_FOO | 19:05 |
| sdague | UC_VAR for globals, and lc_var for locals | 19:05 |
| *** jaypipes has quit IRC | 19:05 | |
| jeblair | sdague: but there is an input var, right? the actual override_tempest_branch variable? | 19:05 |
| jeblair | sdague: yeah, i like that convention, i think d-g mostly follows it too | 19:05 |
| *** mrmartin has quit IRC | 19:06 | |
| sdague | did I miss one? | 19:06 |
| jeblair | sdague: so it's just the var that we would end up setting in the jjb defn, which aiui is override_tempest_branch (or OVERRIDE_tempest_BRANCH), right? | 19:06 |
| sdague | I thought I localized the all | 19:06 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Only run jobs with neutron on neutron changes https://review.openstack.org/84840 | 19:06 |
| sdague | jeblair: yep | 19:06 |
| mtreinish | markmcclain: ^^^ I think that should fix it | 19:06 |
| *** talluri_ has quit IRC | 19:06 | |
| sdague | export override_tempest_branch=master | 19:06 |
| sdague | that was the intent | 19:07 |
| *** mrda_away is now known as mrda | 19:07 | |
| jeblair | sdague: yeah, so that's the one i'm thiking of | 19:07 |
| fungi | clarkb: the failure last night is the sort of thing we want to see a gap analysis on so we know what tempest master needs to be able to run against havana servers | 19:07 |
| jeblair | project_branch_var and project_branch i agree should be lc | 19:07 |
| sdague | right | 19:07 |
| *** rwsu has joined #openstack-infra | 19:07 | |
| *** _nadya_ has joined #openstack-infra | 19:07 | |
| sdague | ok, gotcha | 19:07 |
| sdague | so if we want OVERRIDE_TEMPEST_BRANCH | 19:08 |
| markmcclain | mtreinish: awesome… hopefully this is only temporary as we're really close to enabling the full job and grenade | 19:08 |
| sdague | we could case transform | 19:08 |
| jeblair | for the exported one, i have a weak preference for OVERRIDE_tempest_BRANCH because on average, it's more like the other input variables and matches the arbitrary decision we made in a similar situation | 19:08 |
| sdague | I can add that to the var changes | 19:09 |
| mtreinish | markmcclain: I think that their will still be separates jobs for neutron grenade though right? | 19:09 |
| *** mwagner_afk has quit IRC | 19:10 | |
| markmcclain | I'm assuming we're going to want to run the full and grenade jobs w and w/o neutron for nova once we're ready right? | 19:10 |
| clarkb | ok lunch for me too | 19:10 |
| jeblair | sdague: i could probably live with case transform too if you like that better | 19:10 |
| mtreinish | markmcclain: yeah that's what I figured | 19:10 |
| mtreinish | markmcclain: http://summit.openstack.org/cfp/details/198 | 19:11 |
| markmcclain | mtreinish: cool | 19:11 |
| *** gokrokve has joined #openstack-infra | 19:12 | |
| markmcclain | mtreinish: yeah happy that session was proposed | 19:12 |
| *** basha_ has quit IRC | 19:13 | |
| sdague | jeblair: PROJECT=`echo $PROJECT | tr [:lower:] [:upper:]` | 19:13 |
| sdague | seems simple enought | 19:13 |
| krtaylor | mtreinish, y, it looks like we are not handling stable branches correctly, rfolco and I are discussing (while we are in a meeting) | 19:14 |
| *** CaptTofu has joined #openstack-infra | 19:18 | |
| *** nkinder has quit IRC | 19:19 | |
| *** Sukhdev has joined #openstack-infra | 19:20 | |
| devananda | hi guys! question on the nova.virt.ironic driver. specifically for sdague, russellb, and mordred -- if/when it is added to nova, can tests for the nova.virt.ironic driver be voting in nova? does that change if/when ironic graduates, or would they still be considered third-party tests (as it's not the libvirt driver) | 19:22 |
| *** dwalleck has joined #openstack-infra | 19:22 | |
| russellb | 3rd party, or at least non-voting tests for now i think | 19:23 |
| devananda | russellb: after graduation, does that change? | 19:23 |
| devananda | that's the key piece i need to understand | 19:23 |
| russellb | depends how it's implemented | 19:23 |
| russellb | right now, nothing 3rd party is gating | 19:24 |
| devananda | our tests are all run by infra today | 19:24 |
| devananda | so the tests aren't third-party | 19:24 |
| russellb | OK, then presumably we could look at making it gating after integrated, yes | 19:24 |
| russellb | i think that's the general process we've been following | 19:24 |
| russellb | i think? | 19:24 |
| devananda | none of the other hypervisor drivers are gating, tho | 19:25 |
| fungi | right, if we can run it in official infra (for example, if you get tripleo to the stability requirements necessary to support that) then we could technically have those tests gating, at which point it's more up to nova to decide whether or not they should be | 19:25 |
| *** gokrokve has quit IRC | 19:25 | |
| russellb | devananda: right, but they're all 3rd party CI | 19:25 |
| devananda | we can run a subset of tempest on ironic today, within HP/RS cloud | 19:25 |
| devananda | doesn't need tripleo-ci | 19:25 |
| russellb | cool | 19:25 |
| devananda | though taht'll be great, once it's up | 19:25 |
| russellb | so in that case, it'd be up to nova to decide when we're ready to gate on it | 19:25 |
| devananda | ok | 19:25 |
| devananda | so | 19:26 |
| fungi | devananda: oh, if they can run without nested virt or anything then it's possible we could run them upstream now even (again, voting/gating would still be up to nova in that case though) | 19:26 |
| devananda | can we avoid having ironic's developers suffer while the nova.virt.ironic driver is gating for *us* but not for nova? | 19:26 |
| devananda | fungi: we already do runthem upstream. uses cirros images. it'snot fast, and adam_g has been working on teh tempest stuff | 19:27 |
| *** gokrokve has joined #openstack-infra | 19:27 | |
| devananda | to filter out things ironic doesn't support (eg, snapshot) | 19:27 |
| fungi | devananda: right, assuming 84543 is what you're talking about, so long as it runs correctly you could in theory gate ironic on it now (and other non-integrated projects in your program for that matter, if it helps) | 19:27 |
| devananda | basically, can we manage a large chunk of ironic's code (which is gating for ironic) but have it in the nova tree (where it would not be gating) | 19:28 |
| devananda | for all of juno | 19:28 |
| devananda | and possibly longer, if nova doesn't want to gate on ironic | 19:28 |
| adam_g | fungi, that one + a baremetal scenario test for tempest | 19:28 |
| mtreinish | clarkb: I had a question about: https://review.openstack.org/#/c/83172/ how exactly do I use the branch designator to just change the job name? | 19:28 |
| mtreinish | I wasn't aware that was an option that's why I duplicated the job definition | 19:28 |
| *** gokrokve has quit IRC | 19:29 | |
| fungi | adam_g: btw, lgtm, clarkb's concern about uca aside | 19:29 |
| *** fbo is now known as fbo_away | 19:30 | |
| fungi | mtreinish: just add additional branch designators for it on a project in the jjb projects.yaml | 19:30 |
| devananda | frankly, this seems like a nightmare to me | 19:30 |
| fungi | mtreinish: you could simply embed the 2 into the branch designator parameter | 19:31 |
| mtreinish | fungi: ok, I'll give that a shot | 19:31 |
| adam_g | fungi, cool. ive been doing a bit of testing with it on RAX and HP and seems to work fine for the ironic case. and thanks for tipping me off to https://review.openstack.org/#/c/74889/ , will be keeping an eye on that | 19:31 |
| fungi | mtreinish: there should already be examples in the jjb projects file to draw from | 19:31 |
| openstackgerrit | A change was merged to openstack-infra/config: Limit Tempest tests in virutal Ironic devstack job https://review.openstack.org/84543 | 19:31 |
| dwalleck | devanada: The filtering part? That actually shouldn't be too bad, especially since it's clear what works and what doesn't | 19:32 |
| fungi | adam_g: yeah, i think he's very close now with the patched libvirt debs | 19:32 |
| devananda | dwalleck: filtering? | 19:32 |
| *** nkinder has joined #openstack-infra | 19:33 | |
| dwalleck | Filtering out of tests for features a driver/hypervisor doesn't support. | 19:33 |
| devananda | dwalleck: sorry, i mean the previous discussion | 19:33 |
| adam_g | dwalleck, https://review.openstack.org/#/c/84543/ <- i'm hoping this will filter out enough to get the current virtual-ironic job passing (http://logs.openstack.org/72/84472/1/check/check-tempest-dsvm-virtual-ironic/7ef049d/logs/testr_results.html.gz) | 19:34 |
| *** yolanda has quit IRC | 19:34 | |
| devananda | dwalleck: adding the nova.virt.ironic driver to nova's tree and having it not vote in nova | 19:34 |
| fungi | devananda: not vote (+1/-1 in check) or not gate (+2/-2 in gate)? | 19:35 |
| devananda | fungi: there's no longer a difference, afaict | 19:35 |
| devananda | fungi: if a job is voting in check, and -1's, it doesn't merge | 19:36 |
| fungi | devananda: oh, right you mean non-voting jobs in upstream ci, not third-party ci | 19:36 |
| devananda | right | 19:36 |
| dwalleck | Oh wow. Test filters are regex's? That sounds painful | 19:36 |
| adam_g | are there any other non-libvirt drivers being tested as part of the upstream CI? | 19:36 |
| devananda | adam_g: afaik, no. but imbw. | 19:36 |
| *** e0ne has quit IRC | 19:37 | |
| fungi | adam_g: not yet--i think the xenserver and docker crowds are working on stuff | 19:37 |
| devananda | fungi: i expect ironic tohave thirdparty CI once we get upstream CI working | 19:37 |
| devananda | fungi: and I think that'll be voting, not gating, on ironic project only | 19:37 |
| devananda | fungi: but having a job which is gating in ironic, non-gating in nova, and the code that it's testing is in nova, seems ... awkward. | 19:39 |
| devananda | fungi: so i'm seeking suggestions on how we can move forward with adding the nova.virt.ironic driver back to nova | 19:39 |
| devananda | s/back// | 19:40 |
| fungi | devananda: i agree. it does sound prone to breakage and interruption/loss of development inertia | 19:40 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs https://review.openstack.org/83172 | 19:40 |
| mtreinish | fungi: ^^^ so something like that? | 19:40 |
| *** dwalleck has quit IRC | 19:41 | |
| *** SnowDust has joined #openstack-infra | 19:41 | |
| *** jp_at_hp has quit IRC | 19:42 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs https://review.openstack.org/83172 | 19:42 |
| fungi | mtreinish: i think you don't need the additional entries in the job group, since that's just the list of templates against which parameters should be enumerated to define actual jobs | 19:42 |
| fungi | mtreinish: but otherwise, yes | 19:43 |
| adam_g | seems there needs to be a discussion about graduating from non-gating third party CI to integrated CI that runs as part of upstream infra and has the ability to -2 | 19:43 |
| mtreinish | fungi: ok, I'll remove that bit then | 19:43 |
| devananda | fungi: russellb: worth a summit slot? | 19:43 |
| jeblair | adam_g: i think some terms may be being confused here... | 19:44 |
| adam_g | i can't imagine ironic is alone in your nightmare, devananda | 19:44 |
| *** jcoufal has quit IRC | 19:44 | |
| adam_g | jeblair, probably :) | 19:44 |
| mtreinish | fungi: yeah those entries wouldn't do anything anyway since -2 is the branch designator | 19:44 |
| jeblair | the only reason to run a third-party ci system is because it's testing something that _can't_ be tested upstream (usually something proprietary) | 19:44 |
| jeblair | we'll happily run anything we can to test open source software upstream -- we want to test as much as possible | 19:45 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs https://review.openstack.org/83172 | 19:45 |
| devananda | jeblair: right. and we're doing all the testing of ironic upstream today. | 19:45 |
| jeblair | adam_g: so anyway, short version is that 3rd-party-ci vs upstream ci, and gating vs non-gating are separate questions | 19:45 |
| devananda | jeblair: my question is largely directed to nova. will they allow nova to be gated on the nova.virt.ironic driver upstream tests? | 19:46 |
| *** _nadya_ has quit IRC | 19:46 | |
| adam_g | jeblair, sure | 19:46 |
| fungi | they form a 2x2 matrix, where the "third-party x gating" corner is "never" | 19:46 |
| devananda | jeblair: russellb's answer was, " it'd be up to nova to decide when we're ready to gate on it" | 19:46 |
| jeblair | devananda: agreed, i think that's the right question | 19:46 |
| devananda | jeblair: which is reasonable, but puts ironic in a strange position | 19:47 |
| devananda | in that we could end up being gated on code we don't control, in a project fo rwhcih that code is *not* gating | 19:47 |
| fungi | devananda: fwiw, some library projects do that today by choice | 19:48 |
| devananda | fungi: interesting. examples? | 19:48 |
| jeblair | devananda: i agree -- normally we have an aversion to gating integrated projects on integrated ones because they are less mature, more prone to breakage, and still in a period where we expect them to be coming from outside of the project and not fully up to speed on processes, etc. | 19:48 |
| jeblair | devananda: i don't think that all applies to ironic though | 19:48 |
| fungi | devananda: running unit tests of a consuming project to ensure changes to the library don't break the project, but the project can change in ways which potentially prevent that library from merging code as a result | 19:48 |
| fungi | devananda: pecan/wsme are a couple examples | 19:49 |
| jeblair | devananda: and i think the argument that ironic is more tightly coupled with nova than our typical incubated project is also very reasonable | 19:49 |
| jeblair | devananda: so i think you make a good case, but it's true we really need to get russelb/nova on board with it | 19:49 |
| devananda | fungi: ah, yep, that's fair. and fwiw, pecan/wsme added ironic to their gate because of asymmetric breakage a while back ;) | 19:50 |
| sdague | I agree chasing is not fun, however, there is also fairness in play about what projects prevent nova from making behavior changes | 19:51 |
| fungi | devananda: and oslo in general is about to probably start doing a ton of that too | 19:51 |
| *** e0ne has joined #openstack-infra | 19:51 | |
| devananda | jeblair: right. the typical asymmetric gating between incubated/integrated projects makes a lot of sense, even for ironic, with respect to our neutron/glance/keystone/etc bindings | 19:51 |
| devananda | the difficulty (at least in my mind) is around teh nova.virt.ironic driver code, which is really very tightly coupled to ironic | 19:51 |
| *** e0ne has quit IRC | 19:52 | |
| devananda | because it places ironic in the position of being dependent upon an _internal_ api | 19:54 |
| devananda | instead of a public one | 19:54 |
| *** dripton has quit IRC | 19:55 | |
| *** dkliban has quit IRC | 19:57 | |
| *** gokrokve has joined #openstack-infra | 20:00 | |
| *** imcsk8 has joined #openstack-infra | 20:01 | |
| *** talluri has joined #openstack-infra | 20:03 | |
| *** mrodden has quit IRC | 20:04 | |
| *** mrodden has joined #openstack-infra | 20:05 | |
| sdague | jeblair: so the test-functions.sh is super dependant on order? | 20:05 |
| *** talluri has quit IRC | 20:08 | |
| hdd | does something like "recheck turbo-hipster" exist? I have a change that jenkins passed, but t-h failed, and don't want to recheck everything unnecessarily | 20:08 |
| fungi | mikal: ^ ? (if you're around) | 20:09 |
| *** dripton has joined #openstack-infra | 20:09 | |
| fungi | hdd: http://lists.openstack.org/pipermail/openstack-dev/2014-January/023345.html suggests "recheck migrations" may work? | 20:11 |
| pleia2 | jeblair: are we planning on having a nodepool docs site? (I've been pointing folks to the .rst files in cgit) | 20:11 |
| fungi | pleia2: i bet if you added tooling like we use to publish zuul and jenkins-job-builder docs to ci.o.o, it would be welcomed | 20:12 |
| fungi | at this point we're probably just missing the publish jobs? | 20:12 |
| *** Ryan_Lane2 has joined #openstack-infra | 20:12 | |
| fungi | jeblair's recent documentation additions were fairly comprehensive | 20:13 |
| pleia2 | fungi: yeah, I think we can probably just dump it at ci.openstack.org/nodepool/ and the job will be similar to zuul's | 20:13 |
| fungi | agreed | 20:13 |
| *** Ryan_Lane1 has quit IRC | 20:13 | |
| fungi | i assume that was his intent | 20:13 |
| pleia2 | thanks, I'll poke around | 20:14 |
| hdd | fungi: thanks, I'll give it a whirl | 20:14 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: allow for generic branch overrides https://review.openstack.org/84829 | 20:14 |
| sdague | jeblair: ok, I think that's good now | 20:14 |
| sdague | tests and all | 20:15 |
| sdague | tests found real issues as well, so tests good | 20:15 |
| *** dprince has quit IRC | 20:15 | |
| *** markmcclain has quit IRC | 20:15 | |
| jeblair | sdague: cool; i don't think they should be dependent on order, but i might have messed up some bash scoping thing :/ | 20:16 |
| sdague | the fact that all the functions use global vars means they stomp | 20:17 |
| sdague | and sometimes carry over | 20:17 |
| fungi | i'm still marginally worried that we have a subtle way a job can be constructed which ends up not testing the proposed change, and i think the failing unit test on 84828 actually gets it wrong... looking closer | 20:18 |
| *** Ryan_Lane2 has quit IRC | 20:18 | |
| *** russell_h has joined #openstack-infra | 20:18 | |
| *** fifieldt has quit IRC | 20:20 | |
| sdague | fungi: so I actually suspect the unit tests might not being doing the right things based on the ordering | 20:20 |
| *** fifieldt_ has joined #openstack-infra | 20:20 | |
| devananda | jeblair, fungi: fwiw, summary of what we jsut talked about -- http://summit.openstack.org/cfp/details/215 | 20:20 |
| *** zns has quit IRC | 20:22 | |
| SpamapS | hey, I don't see any nodes coming from nodepool on ci-overcloud.tripleo.org .. is it not trying or getting some errors? | 20:24 |
| SpamapS | fungi: ^? | 20:25 |
| SpamapS | clarkb: ^? | 20:25 |
| SpamapS | jeblair: ^? | 20:25 |
| fungi | SpamapS: looking | 20:25 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: allow for generic branch overrides https://review.openstack.org/84829 | 20:26 |
| SpamapS | fungi: ty | 20:26 |
| sdague | oops, there was a stray echo in there | 20:26 |
| fungi | SpamapS: well, it's not error nodes or leaked floating ips, digging into logs | 20:27 |
| SpamapS | fungi: I do see 3 in BUILD state | 20:27 |
| *** zns has joined #openstack-infra | 20:27 | |
| fungi | SpamapS: we have a ton in delete state in nodepool which don't show up in nova list | 20:28 |
| *** imcsk8_lap has quit IRC | 20:28 | |
| SpamapS | also nova hypervisor-list shows numbers that don't jive with nova list --all-tenants | 20:28 |
| fungi | SpamapS: most going back 9-10 hours | 20:28 |
| SpamapS | hypervisor-stats I should say | 20:28 |
| SpamapS | | running_vms | 67 | | 20:28 |
| *** imcsk8_lap has joined #openstack-infra | 20:28 | |
| fungi | those don't seem to be in our tenant anyway | 20:29 |
| clarkb | reuben consumed that was good | 20:29 |
| fungi | we just have a handful according to nova list | 20:29 |
| *** wenlock_ has joined #openstack-infra | 20:29 | |
| clarkb | mtreinish: did you get an answer about the branch thing? | 20:29 |
| *** medieval1_ is now known as medieval1 | 20:29 | |
| mikal | hdd: "recheck migrations" is what you're looking for | 20:29 |
| fungi | SpamapS: assuming the delete nodes are hanging around because there was some broken around 10 hours ago, i'll try to delete some through nodepool and see what happens | 20:29 |
| SpamapS | fungi: ok cool. I'm trying to figure out why my stats show 67 running | 20:30 |
| fungi | SpamapS: one of the instances, fbc7943f-6ea8-4c3e-ad2a-277fdb6dc714, i tried to delete just now in nodepool and it was successfully removed | 20:31 |
| *** maru_afk is now known as marun | 20:31 | |
| fungi | that one had been in a delete state previously for 9.75 hours | 20:31 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add a fingerprint for bug 1301027 https://review.openstack.org/84558 | 20:31 |
| uvirtbot | Launchpad bug 1301027 in neutron/havana "Permission denied error in unit tests" [High,Fix committed] https://launchpad.net/bugs/1301027 | 20:31 |
| *** markmcclain has joined #openstack-infra | 20:32 | |
| jogo | has this been seen recent;y: http://logs.openstack.org/90/80690/5/gate/gate-requirements-integration-dsvm/92586e0/console.html#_2014-04-02_18_16_19_892 | 20:33 |
| SpamapS | fungi: I see that one with an odd traceback error | 20:33 |
| jogo | pypi.python.org timeout | 20:33 |
| SpamapS | fungi: failed to spawn about 10 hours ago | 20:33 |
| *** wenlock_ has quit IRC | 20:33 | |
| *** Ryan_Lane1 has joined #openstack-infra | 20:33 | |
| fungi | SpamapS: yeah, we have/had about 50 like that | 20:34 |
| SpamapS | fungi: but they're not ERROR on our side, so.. weird | 20:34 |
| fungi | i suspect they're all in the same boat, but i can give you more uuids | 20:34 |
| *** blamar_ has joined #openstack-infra | 20:34 | |
| jeblair | jogo: the requirements gate jobs don't use our local mirror (on purpose) so they are at the mercy of pypi.python.org | 20:34 |
| fungi | SpamapS: they may have leaked into the nodepool database and not been deleted because of the particular way in which they failed to spawn | 20:35 |
| fungi | SpamapS: i'm checking the logs for an example of one now | 20:35 |
| SpamapS | fungi: gotchya | 20:35 |
| SpamapS | we have in fact found every way to fail | 20:35 |
| *** blamar has quit IRC | 20:36 | |
| *** blamar_ is now known as blamar | 20:36 | |
| SpamapS | and we're not working on inventing new ways to fail | 20:36 |
| fungi | SpamapS: you can only hope, but i doubt it | 20:36 |
| SpamapS | s/not/now/ | 20:36 |
| fungi | there are plenty of new ways to fail, as of yet undiscovered | 20:36 |
| SpamapS | fungi: do you have 151c7f69-b3d1-4682-8fae-259c90e0084b ? | 20:36 |
| jogo | fungi: have you thought about using devpi-server to help with that? | 20:36 |
| *** dkliban has joined #openstack-infra | 20:36 | |
| jogo | although that is pretty low priority | 20:37 |
| fungi | SpamapS: i do. nodepool id is 3392282 | 20:37 |
| jogo | https://pypi.python.org/pypi/devpi-server/1.2.1 | 20:37 |
| jogo | I use it locally with great results | 20:37 |
| fungi | SpamapS: i'll get the log entries for it now | 20:37 |
| SpamapS | fungi: ok, that is stuck in BUILD | 20:37 |
| SpamapS | fungi: has no hypervisor host assigned | 20:37 |
| fungi | SpamapS: we see it in a nodepool delete state | 20:37 |
| *** thomasem_ has joined #openstack-infra | 20:38 | |
| fungi | for 2.28 hours | 20:38 |
| *** thomasem has left #openstack-infra | 20:38 | |
| *** thomasem_ is now known as thomasem | 20:38 | |
| SpamapS | fungi: manually deleted it | 20:38 |
| SpamapS | fungi: nodepool agree it is gone yet? | 20:38 |
| fungi | SpamapS: not yet, but it retries deletes periodically | 20:40 |
| fungi | SpamapS: http://paste.openstack.org/show/74899/ | 20:42 |
| fungi | SpamapS: that's all we had in the debug log for that node | 20:42 |
| *** denis_makogon has quit IRC | 20:42 | |
| fungi | looks like it never left a building state | 20:42 |
| fungi | until nodepool gave up, anyway | 20:42 |
| mtreinish | clarkb: yeah, fungi gave me the answer | 20:42 |
| *** blamar has quit IRC | 20:44 | |
| *** blamar has joined #openstack-infra | 20:44 | |
| lifeless | I've reset state on the two stuck builds | 20:47 |
| *** blamar has quit IRC | 20:47 | |
| lifeless | they're now gone | 20:47 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add jobs to publish nodepool docs https://review.openstack.org/84857 | 20:48 |
| *** marun is now known as maru_afk | 20:48 | |
| fungi | nodepool doesn't seem to have retried the delete on 151c7f69-b3d1-4682-8fae-259c90e0084b yet and it's been over 10 minutes | 20:49 |
| fungi | i'll go ahead and delete the others from that timeframe but leave that one untouched to see if it does eventually get properly reclaimed | 20:50 |
| *** blamar has joined #openstack-infra | 20:50 | |
| *** freyes has quit IRC | 20:50 | |
| *** mrmartin has joined #openstack-infra | 20:51 | |
| fungi | the building count on the tripleo cloud should be spiking in our tenant now | 20:53 |
| mrmartin | hi | 20:54 |
| fungi | hi mrmartin | 20:54 |
| mrmartin | guys, may I get a review on this patch: https://review.openstack.org/#/c/83466/ | 20:54 |
| mrmartin | this was posted 5 days ago, and it is related to openid project | 20:55 |
| rfolco | mtreinish, you know if I have to repeat "branch: ^(stable/).*$" for every project name or there is a place I can put just once ? I was thinking to add here https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml#L310 | 20:55 |
| fungi | mrmartin: lgtm (i reviewed it last week), but i'm guessing we're all a bit behind on reviews. it looks like it can really only impact the openstack-id dev server right now, so i'm okay approving it and letting you keep tabs on http://puppetboard.openstack.org/ to see if it throws any errors | 20:56 |
| fungi | mrmartin: http://puppetboard.openstack.org/node/openstackid-dev.openstack.org | 20:56 |
| mrmartin | it will affect to prod server also, because I have a patch waiting there: https://review.openstack.org/#/c/83489/ | 20:57 |
| fungi | mrmartin: right, but the prod server doesn't exist yet | 20:57 |
| fungi | so it can't (yet!) be impacted by that change ;) | 20:57 |
| *** pdmars has quit IRC | 20:57 | |
| mrmartin | yeah, it is our goal to deliver one :) | 20:58 |
| fungi | SpamapS: lifeless i show 52 nodes building | 20:58 |
| *** CaptTofu has quit IRC | 20:58 | |
| fungi | mrmartin: agreed. i'm quite excited about it | 20:58 |
| mrmartin | but it requires prod ssl cert, instance creation, puppet config. | 20:59 |
| * fungi is going to disappear to grab an early dinner, then be back in a little while | 20:59 | |
| *** ok_delta has joined #openstack-infra | 20:59 | |
| *** aysyd has quit IRC | 21:00 | |
| clarkb | jeblair: fungi: was my nodepool change applied to running nodepool? curious if it is making things better | 21:02 |
| *** blamar has quit IRC | 21:02 | |
| clarkb | also we should double check that we aren't leaking in region-b | 21:02 |
| *** afazekas has quit IRC | 21:03 | |
| *** VijayTripathi has joined #openstack-infra | 21:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: Openstackid site-update feature https://review.openstack.org/83466 | 21:04 |
| lifeless | fungi: nope | 21:04 |
| lifeless | fungi: no building VMs | 21:05 |
| lifeless | clarkb: ^ | 21:05 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tempest-master jobs https://review.openstack.org/84864 | 21:05 |
| jeblair | clarkb: i have not restarted nodepool | 21:05 |
| sdague | alright, I'm off for the evening, lug meeting tonight, however I think that layout change + the d-g change should give us the expected results | 21:06 |
| *** Ryan_Lane1 has quit IRC | 21:06 | |
| *** Ryan_Lane1 has joined #openstack-infra | 21:06 | |
| sdague | and if there is any possibility for quick review, it would be great to get data on these as soon as we can, so we can make a decision about skipping the icehouse tag for tempest (and even what it would take to delete the havana one) | 21:07 |
| *** mbacchi has quit IRC | 21:08 | |
| *** _TheDodd_ has quit IRC | 21:09 | |
| mtreinish | rfolco: no I don't know that sorry. jeblair will what rfolco suggested above work for him? | 21:13 |
| mattoliverau | Morning all | 21:18 |
| *** hashar has quit IRC | 21:18 | |
| *** mrodden1 has joined #openstack-infra | 21:19 | |
| sdague | mtreinish: I think it will | 21:20 |
| *** Sukhdev has quit IRC | 21:20 | |
| *** Sukhdev has joined #openstack-infra | 21:21 | |
| *** mrodden has quit IRC | 21:22 | |
| lifeless | clarkb: / jeblair: any errors in nodepool log w.r.t. tripleo-cloud and/or how many nodes does nodepool think you have? | 21:24 |
| zaro | mordred: yo, looks like email notification with diff is actually in 2.8 and it looks like there's an option to include entire diff. https://gerrit-documentation.storage.googleapis.com/Documentation/2.8.2/config-gerrit.html#__a_id_sendemail_a_section_sendemail | 21:28 |
| *** ameade has quit IRC | 21:29 | |
| zaro | fungi: there's even an option to limit the size. so no email bombs, or well limited bombs at least. | 21:30 |
| *** markmc has quit IRC | 21:32 | |
| *** mrmartin has quit IRC | 21:34 | |
| *** freyes has joined #openstack-infra | 21:37 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Add access control https://review.openstack.org/84873 | 21:46 |
| *** thomasem has quit IRC | 21:47 | |
| *** mriedem has quit IRC | 21:48 | |
| *** sc68cal has quit IRC | 21:48 | |
| *** jhesketh has joined #openstack-infra | 21:50 | |
| mordred | zaro: awesome! | 21:50 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Add access control https://review.openstack.org/84873 | 21:50 |
| jhesketh | Morning | 21:50 |
| *** blamar has joined #openstack-infra | 21:51 | |
| lifeless | clarkb: jeblair: fungi: still no sign of nodepool | 21:52 |
| jeblair | clarkb, fungi, mordred, SpamapS, jhesketh: ^ i think with https://review.openstack.org/84873 , all of the pieces for the move to post-jenkins are on the board | 21:52 |
| jeblair | (i mean, still quite a bit to do, but hopefully we can start to see how it all fits together now) | 21:53 |
| jeblair | lifeless: it looks like i'm on deck to triage that; will take a look | 21:54 |
| lifeless | jeblair: thank you | 21:54 |
| jhesketh | jeblair: awesome :-) | 21:54 |
| *** UtahDave has quit IRC | 21:57 | |
| mordred | jeblair: ooh, really? neat | 21:57 |
| jeblair | lifeless: i think the connection is half-dead | 21:58 |
| mordred | jeblair: I've been thinking - what if we replaced gearman with an email queue that we had a bunch of humans respond to and take manual steps? | 21:58 |
| jeblair | mordred: this gozer thing you guys are working on sounds swell. :) | 21:59 |
| jeblair | lifeless: http://paste.openstack.org/show/74908/ | 21:59 |
| mordred | jeblair: :) | 21:59 |
| *** bhuvan_ has joined #openstack-infra | 21:59 | |
| jeblair | istr that we have had a discussion about the fact that this stack (novaclient, requests, httplib, ssl) doesn't seem to have a keepalive | 22:00 |
| *** thuc has quit IRC | 22:00 | |
| jeblair | and that it was desirable for it to have one. but i don't recall if someone had done the work to figure out where it should go and how to get it in there. or volunteered to do the work. | 22:00 |
| *** thuc has joined #openstack-infra | 22:01 | |
| jeblair | mordred, clarkb, lifeless, fungi, SpamapS: ^ does that ring a bell? | 22:01 |
| mordred | jeblair: well, I'd characterize it more like it holds open connections and doesn't handle them going away well | 22:01 |
| mordred | but yes | 22:01 |
| clarkb | jeblair: I remember it dying and people fiddleing with tcp tools to kill it but I don't remember if there was a plan to fix it | 22:02 |
| *** bhuvan has quit IRC | 22:02 | |
| lifeless | jeblair: ahha! yes | 22:03 |
| lifeless | jeblair: and we found and used a tool to reset it last time | 22:03 |
| jeblair | mordred: you are right. i mentioned keepalive because that was the proposed fix (not for the side effect of keeping established connections alive, but for the side effect of actively noticing when they fail) | 22:03 |
| lifeless | now what was the dang tool | 22:03 |
| lifeless | jeblair: ahha - http://killcx.sourceforge.net/ | 22:04 |
| jeblair | lifeless: i'm not really interested in manually fixing tcp connections in the long run. i think we need to get a real fix for this bug. | 22:04 |
| lifeless | jeblair: thats already installed on the nodepool box | 22:04 |
| lifeless | jeblair: I agree. | 22:04 |
| lifeless | jeblair: however, I'd like to not wait for that to get service back today, if thats ok | 22:04 |
| *** ArxCruz has joined #openstack-infra | 22:04 | |
| lifeless | jeblair: someone from the sprint was looking into fixing requests to have a keepalive | 22:04 |
| lifeless | jeblair: it might even have been SpamapS | 22:05 |
| jeblair | lifeless: i restarted nodepool | 22:05 |
| *** thuc has quit IRC | 22:05 | |
| lifeless | thank you! | 22:05 |
| jeblair | (we're not in a critical stage like we were last time, we can handle a restart now) | 22:05 |
| *** imcsk8 has quit IRC | 22:06 | |
| jeblair | SpamapS: we're all rootin for ya. :) | 22:06 |
| openstackgerrit | Andrew Grimberg proposed a change to openstack-infra/jenkins-job-builder: Allow custom settings configs for maven projects https://review.openstack.org/84879 | 22:07 |
| lifeless | I'll add a card to trello | 22:08 |
| markmcclain | can request that this be retried in the gate without another 1.5 hr detour through the check queue? https://review.openstack.org/#/c/84367/ | 22:08 |
| lifeless | jeblair: any errors? I'm not seeing either templates or nodes being spawned | 22:09 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Fix neutron configuration error https://review.openstack.org/84880 | 22:10 |
| jeblair | lifeless: yes there are errors ^ | 22:10 |
| lifeless | heh | 22:10 |
| *** david-lyle has quit IRC | 22:11 | |
| lifeless | mordred: ^ please review asap | 22:11 |
| *** tykeal has joined #openstack-infra | 22:11 | |
| lifeless | jeblair: \o/ nodes. | 22:12 |
| jeblair | lifeless: i've restarted it with that change manually applied | 22:12 |
| jeblair | 2014-04-02 22:11:25,708 DEBUG nodepool.NodePool: <AllocationGrant of 55 of tripleo-precise from tripleo-test-cloud> | 22:12 |
| lifeless | jeblair: thank you | 22:12 |
| jeblair | np | 22:13 |
| *** rcleere has quit IRC | 22:17 | |
| openstackgerrit | Andrew Grimberg proposed a change to openstack-infra/jenkins-job-builder: Allow custom settings configs for maven projects https://review.openstack.org/84879 | 22:19 |
| SpamapS | keepalives in requests made my brain explode | 22:20 |
| SpamapS | there are so many places the timeout may be set and/or lost | 22:20 |
| jeblair | SpamapS: wow. that's a really terrible thing for a fundamental library like requests to get wrong. | 22:21 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Tightened up story detail UI https://review.openstack.org/84882 | 22:21 |
| jeblair | SpamapS: is it hopeless? | 22:21 |
| openstackgerrit | Andrew Grimberg proposed a change to openstack-infra/jenkins-job-builder: Allow custom settings configs for maven projects https://review.openstack.org/84879 | 22:22 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add jobs to publish nodepool docs https://review.openstack.org/84857 | 22:23 |
| *** jlk has quit IRC | 22:24 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix neutron configuration error https://review.openstack.org/84880 | 22:24 |
| *** alff has quit IRC | 22:26 | |
| *** alff_ has quit IRC | 22:26 | |
| *** flaper87 is now known as flaper87|afk | 22:26 | |
| *** andrew_plunk has quit IRC | 22:27 | |
| lifeless | jeblair: clarkb had a fix for floating ip leaks right ? | 22:28 |
| *** signed8bit has quit IRC | 22:29 | |
| SpamapS | jeblair: no, it is just spaghetti | 22:29 |
| SpamapS | jeblair: I may have been expecting it to be easy. | 22:30 |
| *** signed8bit has joined #openstack-infra | 22:30 | |
| *** jlk has joined #openstack-infra | 22:30 | |
| clarkb | lifeless: I did, I think it would've just been applied with the recent service restart | 22:30 |
| *** jlk has quit IRC | 22:30 | |
| *** jlk has joined #openstack-infra | 22:30 | |
| lifeless | ok I'll remove my workaround | 22:32 |
| *** enikanorov_ has joined #openstack-infra | 22:32 | |
| lifeless | (while true; do nova floating-ip-list | awk '$5=="-" { print $2}' | xargs --no-run-if-empty -n1 nova floating-ip-delete; sleep 300; done) | 22:32 |
| *** signed8bit has quit IRC | 22:34 | |
| *** enikanorov has quit IRC | 22:35 | |
| *** harlowja has quit IRC | 22:35 | |
| clarkb | jeblair: what prompted the new gear feature? is that something gearman supports? | 22:36 |
| *** harlowja has joined #openstack-infra | 22:36 | |
| *** dstanek has quit IRC | 22:38 | |
| jogo | anteaya: am I reading this right: https://wiki.openstack.org/wiki/PTL_Elections_March/April_2014 glance and marconi have no one running | 22:38 |
| *** dstanek has joined #openstack-infra | 22:39 | |
| *** mgagne has quit IRC | 22:40 | |
| *** dims_ has quit IRC | 22:40 | |
| *** tchaypo has quit IRC | 22:40 | |
| pleia2 | jogo: yeah, she sent a mail out yesterday to encourage applicants | 22:41 |
| *** ogelbukh has quit IRC | 22:42 | |
| jogo | pleia2: haha | 22:42 |
| mtreinish | jogo: go for it | 22:42 |
| mtreinish | jogo for the first double ptl | 22:42 |
| jogo | mtreinish: oh i have a commit in glance on Jan 10th | 22:43 |
| fungi | okay, back | 22:43 |
| *** ogelbukh has joined #openstack-infra | 22:43 | |
| jogo | nothing in marconi | 22:43 |
| fungi | seems if i just go to dinner, everyone solves all the problems while i'm gone | 22:43 |
| fungi | perfection | 22:43 |
| mtreinish | jogo: you've got >24hrs :) | 22:43 |
| jogo | if no one else runs, I can run with the platform: I can technically run so I am | 22:44 |
| jogo | markwash: ^ | 22:44 |
| markwash | deadlines are made to be stared down | 22:45 |
| fungi | jogo: it's a good platform... if nothing else, it'll teach people the value of not ignoring the election schedule | 22:45 |
| jogo | markwash: haha if you slip up I am gonna run ;) | 22:45 |
| fungi | platform: i can read a calendar | 22:45 |
| * markwash forces UTC to become PST by force of will | 22:45 | |
| *** freyes has quit IRC | 22:46 | |
| *** imcsk8 has joined #openstack-infra | 22:47 | |
| *** prad_ has quit IRC | 22:48 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Adds metadata to story detail page https://review.openstack.org/84888 | 22:48 |
| lifeless | clarkb: looks like its still leaking ips | 22:50 |
| fungi | lifeless: i'll double-check the floating-ips | 22:50 |
| clarkb | mordred: jeblair: so this jshint/jslint thing going on in the mail list has me wondering why storyboard (and possibly zuul) uses jshint at all. We use open source tools. jshint isn't erally open source | 22:50 |
| clarkb | lifeless: its possible my fix was only a partial one. It handles cases where add floating ip returns an error and deletes that floating ip immediately (this was what appeared to appen in fungi's stacktrace in the bug) | 22:51 |
| lifeless | fungi: I see 24 VMs ACTIVE with no public IP | 22:51 |
| lifeless | clarkb: oh the leak is on VM delete | 22:51 |
| lifeless | clarkb: not on create | 22:51 |
| clarkb | lifeless: not according to fungi's stacktrace | 22:51 |
| fungi | clarkb: ZOMG it's THAT license | 22:51 |
| lifeless | clarkb: fungi's stacktrace shows what happens once you've leaked to quota | 22:51 |
| clarkb | lifeless: it did + it showed a leak | 22:52 |
| * jogo addes a note to my calendar for 5:59 UTC APril 4th | 22:52 | |
| clarkb | lifeless: there were two stacktraces showing both halves | 22:52 |
| lifeless | clarkb: ack | 22:52 |
| lifeless | clarkb: is your fix in review or landed? | 22:52 |
| clarkb | lifeless: it is landed. | 22:52 |
| fungi | i had entirely spaced on the fact that jshint uses the json license | 22:52 |
| mordred | clarkb: jshint or jslint? | 22:52 |
| clarkb | fungi: I agree with peopel saying it isn't a legal issue for us to test with it (or shouldn't) but it isn't open either | 22:53 |
| fungi | mordred: it's the "must be used for good, not evil license from json" | 22:53 |
| clarkb | mordred: both apparently. jslint is the bad bad one and jshint uses jslint's parser | 22:53 |
| *** eharney has quit IRC | 22:53 | |
| *** tchaypo has joined #openstack-infra | 22:54 | |
| *** alff has joined #openstack-infra | 22:54 | |
| *** alff_ has joined #openstack-infra | 22:54 | |
| mtreinish | mordred: https://raw.github.com/douglascrockford/JSLint/master/jslint.js | 22:54 |
| fungi | there was a kilothread (or nearly anyway) on debian-devel about it months back | 22:54 |
| *** harlowja has quit IRC | 22:54 | |
| *** harlowja has joined #openstack-infra | 22:55 | |
| fungi | it's borrowing from http://www.json.org/license.html | 22:55 |
| mordred | hrm | 22:55 |
| fungi | "...The Software shall be used for Good, not Evil..." | 22:55 |
| clarkb | and my right to be evil must be preserved | 22:55 |
| fungi | clarkb: while i may not agree with your evil, i will fight to the death to preserve your right to be evil ;) | 22:56 |
| mordred | I do not feel strongly enough about this to have a real opinion. I think the line in the license is silly and would be laughed out of a courtroom | 22:56 |
| *** dims_ has joined #openstack-infra | 22:56 | |
| mordred | I'm probably in the minority on that | 22:56 |
| mordred | so I'll defer to others | 22:56 |
| *** ok_delta has quit IRC | 22:56 | |
| fungi | as a decidedly silly person, in the mpfc sense of the term, i think that software licenses are not at all the appropriate place to be silly | 22:57 |
| clarkb | mordred: forget silly, it isn't a recognized open source license | 22:58 |
| lifeless | I'm seeing something very odd | 22:58 |
| fungi | now, nobody likes a good laugh more than i do... except, perhaps my wife... and some of her friends. oh, yes, and captain johnson. come to think of it, most people like a good laugh more than i do, but that's beside the point | 22:58 |
| lifeless | on e.g. 78787,3 | 22:59 |
| lifeless | I saw a job 25% through and then it toggled to 'queued' | 22:59 |
| lifeless | what would cause that ? | 22:59 |
| *** weshay has quit IRC | 22:59 | |
| fungi | lifeless: if the jenkins job gets terminated in certain ways, zuul will consider that a reason to requeue it (primarily if it doesn't return a job status) | 22:59 |
| *** saschpe has quit IRC | 23:00 | |
| clarkb | the slave connection dying is a common cause of ^ | 23:00 |
| *** mrodden1 has quit IRC | 23:00 | |
| *** yamahata has joined #openstack-infra | 23:01 | |
| lifeless | so this is a little worrying ;) | 23:01 |
| lifeless | SpamapS: could I ask you a favor? | 23:01 |
| lifeless | SpamapS: can you prep a kernel upgrade for the network node? | 23:01 |
| fungi | BadRequest: Error. Unable to associate floating ip (HTTP 400) (Request-ID: req-2432d252-14df-49d1-b6c0-1e15a0d39e84) | 23:01 |
| fungi | getting a lot of those | 23:02 |
| fungi | digging further into logs now | 23:02 |
| fungi | but that was confirmed to be a tripleo provider request | 23:02 |
| *** jerryz has joined #openstack-infra | 23:02 | |
| lifeless | ahh | 23:03 |
| lifeless | gro is on | 23:03 |
| lifeless | let me turn it off | 23:03 |
| lifeless | now that api | 23:03 |
| lifeless | /var/log/upstart/nova-api.log:2014-04-02 23:02:40.934 29859 WARNING nova.api.openstack.compute.contrib.floating_ips [req-2432d252-14df-49d1-b6c0-1e15a0d39e84 d5af62d2183d431796d74c5bb119ec9f e01e473a9250498883955b80966a1e58] multiple fixed_ips exist, using the first: 192.168.1.181 | 23:03 |
| fungi | "gro" is an acronym with which i am entirely unfamiliar | 23:03 |
| lifeless | getting the wrong port order | 23:03 |
| lifeless | fungi: generic-receive-offload: on | 23:03 |
| fungi | sounds like openstack stuff | 23:04 |
| mordred | clarkb: I hear taht argument. I'm still meh on it. it does not offend my open source sensibilities. I see if might offend a different person's open source sensibilities. I don't think our use of it is particularly troubling, nor do I think it, itself, is particularly troubling | 23:04 |
| lifeless | fungi: hardware accelerated TCP | 23:04 |
| fungi | oh, kernel | 23:04 |
| mordred | but as I said - my opinion here is not strong eough that I will fight hard against consensus to the contrary | 23:04 |
| fungi | yep, that knob | 23:04 |
| clarkb | mordred: I think you will find that debian will flip tables | 23:04 |
| lifeless | SpamapS: is there a bug for the timeout thing ? | 23:04 |
| mordred | clarkb: I don't care | 23:05 |
| clarkb | mordred: which considering it is a node thing probably doesn't matter | 23:05 |
| jeblair | clarkb: er, what thread on the list? | 23:05 |
| clarkb | mordred: right, don't forget we already package zuul in debian | 23:05 |
| mordred | but you don't need jshint to pacakge zuul for debia | 23:05 |
| clarkb | jeblair: openstack-dev and let me find the subject for you | 23:05 |
| clarkb | mordred: well if you do it the way zigo does it you do | 23:05 |
| fungi | mordred: i think licenses like that irk lawyers, and we might want to keep a few in our pocket for when things turn "bad" | 23:05 |
| clarkb | see many threads that have worn out this topic for details | 23:05 |
| mordred | that is zigo's problem | 23:06 |
| mordred | he has chosen to make life hard for himself | 23:06 |
| lifeless | I'm going to try mass-updating nova to have that ordering fix | 23:06 |
| clarkb | jeblair: 'Javascript linter' | 23:06 |
| lifeless | and see if it makes things better | 23:06 |
| lifeless | una momento while I prep that | 23:06 |
| fungi | lifeless: sounds like a guaranteed win | 23:06 |
| mordred | it it not important that the debian packaging scripts run a javascript linter to be able to make a package | 23:06 |
| clarkb | mordred: personally I find licenses that limit what I can use software for to be bothersome | 23:06 |
| *** SnowDust has quit IRC | 23:06 | |
| mordred | sure. I'm just saying - how bothering, vs. what we're using it for. for me, it's a judgement call and it does not bother me | 23:07 |
| clarkb | mordred: I am on board with restrictions about how things must be distributed and so on (to make sure stuff stays open) but saying I can't use X for Y is terribad | 23:07 |
| fungi | clarkb: i have a hard time thinking of javascript as software, though i know in the back of my mind it is, but for the most part i agree | 23:07 |
| mordred | it WOULD probably bother me if we were shipping it as a depend | 23:07 |
| clarkb | regardless of the values of X and Y | 23:07 |
| mordred | but as a code linting tool, the weirdness thing is not vexing to me | 23:07 |
| fungi | projects like node took the joke of treating javascript as a programming language as a personal challenge | 23:08 |
| *** saschpe has joined #openstack-infra | 23:08 | |
| openstackgerrit | A change was merged to openstack/requirements: If dependency not in global reqs exit with a 1 https://review.openstack.org/80690 | 23:08 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Altering the > to -gt to check number of processors https://review.openstack.org/83141 | 23:08 |
| jeblair | clarkb: i did not realise jshint was not free software. or open source software. | 23:09 |
| jeblair | i see that it is not. | 23:09 |
| fungi | jeblair: clearly you were not evil enough to discover you should not use it | 23:09 |
| jeblair | mordred: it violates the free software definition _and_ the open source definition. i don't think we should use it. | 23:10 |
| mordred | jeblair: ok. like I said, I defer to the consensus | 23:10 |
| fungi | from a humor standpoint, i found it amusing that the json author went on the record as offering exemptions to the license, free of charge, to anyone who truly wished to do evil with it | 23:10 |
| mordred | I disagree, because I think the sentence is a no-op | 23:10 |
| lifeless | rolling deploy in progress | 23:11 |
| *** yamahata has quit IRC | 23:11 | |
| mordred | and if the sentence is a no-op, then the license does not violate either definition | 23:11 |
| *** thuc has joined #openstack-infra | 23:11 | |
| mordred | but I'll make a patch to disable | 23:11 |
| jeblair | mordred: it's worth noting that zuul is pure evil, so it's pretty much a non-starter. :/ | 23:13 |
| mordred | :) | 23:13 |
| lifeless | fungi: ok, so - gro is now off, which should make tcp more reliable | 23:13 |
| lifeless | fungi: AFAIK we may go down again soon due to wrong kernel, but hopefully SpamapS is on that | 23:13 |
| lifeless | fungi: I've restarted all the nova-computes with a new neutron api.py | 23:13 |
| lifeless | fungi: and they are still up | 23:14 |
| lifeless | fungi: hopefully all new VMs will have their interface order as requested, not python dictionary order de jour | 23:14 |
| fungi | lifeless: okay, i'll see if things get any better and, if not, delete some nodes | 23:14 |
| lifeless | nova list --all-tenants | grep ACTIVE | grep -v 138.35 | wc -l | 23:14 |
| lifeless | 5 | 23:14 |
| lifeless | thats better | 23:14 |
| fungi | seems realistic anyway | 23:15 |
| lifeless | though I really didn't expect it to work that quickly | 23:15 |
| fungi | i have 15 building and 38 used | 23:15 |
| fungi | something tells me some of those are a lie | 23:15 |
| *** thuc has quit IRC | 23:16 | |
| lifeless | I'm running my while loop again to trim floating ips every 300 seconds | 23:16 |
| SpamapS | lifeless: mellanox? | 23:16 |
| lifeless | SpamapS: yeah | 23:16 |
| lifeless | SpamapS: saucy image | 23:16 |
| SpamapS | lifeless: I'll whip up an element so we don't forget again | 23:16 |
| lifeless | SpamapS: the reason it was rebooted | 23:16 |
| SpamapS | since it will likely be ~ 6 weeks until we get trusty | 23:16 |
| lifeless | SpamapS: was cmd_alloc fails and TCP sessions not. | 23:16 |
| lifeless | SpamapS: \o/ | 23:16 |
| SpamapS | lifeless: yeah. I have to run get kids but the driver is here if you are feeling intrepid: http://www.mellanox.com/page/products_dyn?product_family=27 | 23:17 |
| lifeless | SpamapS: I'm so not ;) | 23:17 |
| clarkb | out of curiousity, is mellanox a side effect of hand me down hardware or is it intentionally used to make your lives painful | 23:18 |
| lifeless | mellanox make very very high end network adapters | 23:18 |
| clarkb | yes, but their drivers are bad | 23:18 |
| lifeless | the problem is with great power comes great ability to fail | 23:18 |
| *** thuc has joined #openstack-infra | 23:18 | |
| lifeless | no, this is the hardware spec HP cloud 0.1 or whatever rolled out | 23:18 |
| SpamapS | clarkb: the new drivers are really nice | 23:18 |
| lifeless | I'm fairly sure newer stuff uses mellanox too | 23:19 |
| *** thuc has quit IRC | 23:19 | |
| SpamapS | yeah it's more that it just takes a lot of time and bug reports to get things right | 23:19 |
| clarkb | SpamapS: or you could get intel 10Gbe nics and have no problems | 23:19 |
| *** thuc has joined #openstack-infra | 23:19 | |
| lifeless | clarkb: why do you say that? | 23:19 |
| SpamapS | clarkb: There's some special sauce in the mellanox versions | 23:19 |
| SpamapS | like really | 23:19 |
| SpamapS | they pour thousand island dressing into the card | 23:19 |
| SpamapS | ok really | 23:20 |
| * SpamapS disappears | 23:20 | |
| lifeless | clarkb: as in what makes you think intel 10Gbps cards are fault free ? | 23:20 |
| clarkb | lifeless: Intel's non wireless drivers are pretty solid (I have never had trouble with them but maybe that is confirmation bias from working at Intel) | 23:20 |
| *** thuc has quit IRC | 23:20 | |
| lifeless | clarkb: ah - inside knowledge | 23:20 |
| clarkb | lifeless: they aren't fault free, but I have never seen the significant number of issues tat you have had with mellanox using intel nics | 23:20 |
| *** thuc has joined #openstack-infra | 23:20 | |
| clarkb | and we had thousands of them | 23:20 |
| lifeless | clarkb: you were using 10Gbps with GRO etc etc? | 23:20 |
| *** Sukhdev has quit IRC | 23:21 | |
| lifeless | clarkb: so we've had basically one issue, its fixed upstream its fixed in trusty (they updated) but trusty isn't released and we hadn't baked 'pull mellanox stuff down' into automation | 23:21 |
| *** mrodden has joined #openstack-infra | 23:22 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update the jjb timeout global defaults and add a build-timeout macro https://review.openstack.org/84637 | 23:22 |
| *** sc68cal has joined #openstack-infra | 23:23 | |
| fungi | yeah, we used both mellanox and intel 10gbps nics at $oldhell with vmware esxi and the intels were far more stable, but i'm not entirely sure that tweaked drivers on a hacked-up/trimmed-down and outdated rhel really provides much perspective on driver quality | 23:23 |
| fungi | we had similar experiences with nic drivers on checkpoint splat, but again same story | 23:24 |
| clarkb | lifeless: we were using X * 10Gbe LACP links and never had trouble, not sure what all options were used server side | 23:24 |
| jogo | mtreinish: https://review.openstack.org/84896 ;) | 23:25 |
| fungi | having seen, over the years, things that proprietary linux kernel embedding vendors do to hardware drivers, i take anything which isn't an off-the-shelf distro install with a dump truck of salt | 23:26 |
| fungi | nodepool says, miraculously, that at this moment there are 55 of the 55 max-servers for the tripleo provider in use | 23:28 |
| fungi | lifeless: SpamapS: ^ possible? | 23:29 |
| *** esker has quit IRC | 23:29 | |
| *** dcramer_ has quit IRC | 23:30 | |
| *** krotscheck has quit IRC | 23:30 | |
| lifeless | fungi: like I say though, trusty fixes this, OTS :) | 23:30 |
| lifeless | fungi: I see lots of slaves on the zuul status page so perhaps yes :) | 23:31 |
| lifeless | fungi: now, the question is whether the DNS issue we saw last night will come back | 23:31 |
| fungi | lifeless: not arguing... once trusty releases i'm very eager to get it into production asap | 23:31 |
| *** krotscheck has joined #openstack-infra | 23:31 | |
| openstackgerrit | Andrew Grimberg proposed a change to openstack-infra/jenkins-job-builder: Allow custom settings configs for maven projects https://review.openstack.org/84879 | 23:31 |
| zaro | clarkb: would you mind moving this along? https://review.openstack.org/#/c/69768 | 23:32 |
| fungi | more just questioning the ability to really compare nic driver stability and performance apples-to-apples unless all parties are on a level (vanilla kernel) playing field | 23:32 |
| *** thuc has quit IRC | 23:33 | |
| *** thuc has joined #openstack-infra | 23:33 | |
| lifeless | fungi: vanilla kernel doesn't claim to be usable | 23:34 |
| lifeless | fungi: years back they explicitly moved to 'and distros will stabilise and ship' as a model | 23:34 |
| fungi | lifeless: true, but comparing drivers/hardware without a level playing field is questionable at best | 23:34 |
| lifeless | sure | 23:35 |
| lifeless | but define level playing field | 23:35 |
| lifeless | I'd say 'what the hardware vendor recommends' is a level playing field | 23:35 |
| *** thuc_ has joined #openstack-infra | 23:35 | |
| fungi | and i'm taking "vanilla kernel" in the less traditionally strict sense by meaning "not including random out-of-tree/proprietary lkms and patches" | 23:35 |
| *** imcsk8 has quit IRC | 23:36 | |
| *** thuc_ has quit IRC | 23:36 | |
| fungi | i think the hardware vendors who actually bother to work with the kernel developer community on upstreaming support deserve a home-field advantage, as it were | 23:36 |
| *** thuc has quit IRC | 23:36 | |
| *** thuc has joined #openstack-infra | 23:36 | |
| jogo | markwash: now my only hope is marconi ;) | 23:37 |
| openstackgerrit | Andrew Grimberg proposed a change to openstack-infra/jenkins-job-builder: Allow custom settings configs for maven projects https://review.openstack.org/84879 | 23:38 |
| fungi | jogo: itym s/marconi/macaroni/ | 23:38 |
| mtreinish | jogo: hahaha | 23:38 |
| mtreinish | jogo: now I need to get a marconi commit in so I can vote for you :) | 23:38 |
| fungi | jogo: now is the time to announce your candidacy for ptl of macaroni | 23:38 |
| jogo | mtreinish: I just posted two | 23:38 |
| jogo | fungi: that would be my platform | 23:39 |
| mtreinish | jogo: that's 1/2 the marconi review queue!? | 23:39 |
| mtreinish | https://review.openstack.org/#/q/status:open+project:openstack/marconi,n,z | 23:39 |
| jogo | mtreinish: woot! good odds of me landing a patch in time | 23:40 |
| lifeless | fungi: homefield is not level :) | 23:40 |
| fungi | jogo: you should go nag anyone at all in their irc channel for reviews too ;) | 23:40 |
| *** Ryan_Lane2 has joined #openstack-infra | 23:40 | |
| jogo | fungi: I'll wait for jenkins to run first | 23:41 |
| *** masayukig has joined #openstack-infra | 23:41 | |
| fungi | lifeless: i can only hope the homefield slopes toward freedom | 23:41 |
| *** andreaf2 has joined #openstack-infra | 23:41 | |
| *** Ryan_Lane1 has quit IRC | 23:42 | |
| mtreinish | jogo: you won't have to worry about passing tempest though it's nonvoting :) | 23:42 |
| jogo | mtreinish: haha | 23:42 |
| openstackgerrit | Andrew Grimberg proposed a change to openstack-infra/jenkins-job-builder: Allow custom settings configs for maven projects https://review.openstack.org/84879 | 23:42 |
| markwash | you guys probably know how to do this: I want to run just one file of keystone unit tests with tox or testr or whatever | 23:42 |
| jogo | tox -epy27 keystone.tests.test_file_name | 23:43 |
| mtreinish | jogo: isn't there a '--' for the posargs? | 23:44 |
| jogo | mtreinish: its not always needed for some reason | 23:44 |
| jogo | but it can't hurt | 23:44 |
| jogo | tox -epy27 -- keystone.tests.test_file_name | 23:44 |
| fungi | probably only needed if you need to get dashopts down into testr from tox | 23:44 |
| mtreinish | fungi: ah, that makes sense | 23:44 |
| ianw | sdague: sorry about the silly comments from redhatci, i've fixed the issue | 23:44 |
| ianw | on your recent changes | 23:45 |
| openstackgerrit | Andrew Grimberg proposed a change to openstack-infra/jenkins-job-builder: Allow custom settings configs for maven projects https://review.openstack.org/84879 | 23:47 |
| mtreinish | jogo: http://logs.openstack.org/96/84896/1/check/check-tempest-dsvm-marconi/af3315b/console.html#_2014-04-02_23_44_56_714 | 23:47 |
| mtreinish | haha, it looks like marconi doesn't even start properly in the tempest marconi jobs :) | 23:47 |
| *** jerryz has quit IRC | 23:48 | |
| fungi | zaro: on 69768 if you can "gerrit plugin ls" then you're new enough to use the "replication start" syntax (implying replication plug-in) and otherwise use the old "gerrit replicate" syntax? what is the getVersion() method used for there (the commit message mentions adding it, but seems unused so far)? | 23:50 |
| openstackgerrit | Andrew Grimberg proposed a change to openstack-infra/jenkins-job-builder: Allow custom settings configs for maven projects https://review.openstack.org/84879 | 23:51 |
| mtreinish | jogo: hmm looks like a config thing or devstack thing. The server seems to be started during the run... | 23:51 |
| tykeal | haha! success... only after 8 attempts at that silly thing. that's what I get for rarely doing python ;) | 23:52 |
| fungi | tykeal: apparently indentation is semantic or something. at least that's what i heard | 23:53 |
| tykeal | fungi: lol no, what was biting me the most was the pep8 line length requirements | 23:54 |
| tykeal | when I did some refactoring to shorten up the lines I kept screwing up variable renames and such :D | 23:54 |
| fungi | yeah, there aren't many people still around who code on 80-column terminals. probably just me | 23:54 |
| tykeal | well... in my case I just was running into some rather unfortunate strings that didn't want to break apart nicely | 23:55 |
| fungi | and for everything else, there's the #noqa tag | 23:55 |
| tykeal | #noqa tag? umm... I don't think I want to know ;) | 23:56 |
| fungi | from the flake8 docs... "lines that contain a # noqa comment at the end will not issue warnings" | 23:57 |
| davidlenwell | lol my commit from 5 minutes ago says its 12 days old in github.. | 23:57 |
| tykeal | committing backwards in time. I like that | 23:58 |
| fungi | davidlenwell: did you amend an older commit? | 23:58 |
| davidlenwell | fungi: nope | 23:58 |
| fungi | wow--that's exciting. keeping spare chronotons in a mason jar behind the monitor? | 23:58 |
| nibalizer | davidlenwell: if you look at my github page i have a commit from the future which breaks their js visualizer | 23:59 |
| fungi | nibalizer: clearly you use your software for evil, not good | 23:59 |
| davidlenwell | its happening on a lot of files .. like https://github.com/stackforge/refstack/blob/master/doc/refstack.md says its 13 days old .. but that was changed and merged today | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!