fungi | isviridov: yes, they're not synchronized or related in any way. i've added you as the first member for each of those groups | 00:00 |
---|---|---|
fungi | isviridov: https://review.openstack.org/#/admin/groups/279,members https://review.openstack.org/#/admin/groups/278,members https://review.openstack.org/#/admin/groups/280,members | 00:00 |
*** cadenzajon has quit IRC | 00:00 | |
*** valentinbud has joined #openstack-infra | 00:01 | |
isviridov | fungi, awesome. Already adding )) | 00:01 |
*** dkliban has joined #openstack-infra | 00:01 | |
fungi | so, i simply re-ran manage-projects for that stackforge/ospurge without doing any cleanup, and got http://paste.openstack.org/show/70806 | 00:05 |
*** jeckersb is now known as jeckersb_gone | 00:05 | |
fungi | that's very gnarly | 00:05 |
*** zns has joined #openstack-infra | 00:05 | |
fungi | then i deleted the repositories for it from /var/lib/jeepyb, /var/lib/git and /home/gerrit2/review_site/git and ran it again, which got me http://paste.openstack.org/show/70807 | 00:06 |
fungi | yet still more and different fun! | 00:06 |
*** ok_delta has joined #openstack-infra | 00:07 | |
*** jcooley_ has quit IRC | 00:08 | |
fungi | mordred: i think it's balking at 'git remote remove' in your new patch | 00:09 |
fungi | "error: Unknown subcommand: remove" | 00:09 |
clarkb | it needs to be 'remote remove' | 00:09 |
clarkb | oh it is | 00:09 |
clarkb | hrm | 00:09 |
fungi | mordred: looks like it needs to be "rm" rather than "remove". i'll leave a comment on the review | 00:09 |
clarkb | remove must be in a newer version of git, I have it | 00:10 |
anteaya | what is happening when Failed to add groups file for project is the output? | 00:11 |
fungi | clarkb: git version 1.7.9.5 on review.o.o | 00:12 |
fungi | anteaya: it's leaving the repository half-configured and not adding the acl | 00:12 |
anteaya | for reasons unknown? | 00:12 |
*** Ryan_Lane2 has quit IRC | 00:13 | |
fungi | presently unknown, yes. that's why i'm investigating | 00:13 |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Fix logstash watchdog cron. https://review.openstack.org/77306 | 00:13 |
clarkb | fungi: ^ I think I derped let me test that really quickly | 00:13 |
clarkb | thats what I get for copy pasta | 00:13 |
anteaya | fungi: k | 00:13 |
* anteaya wonders if anyone else got the Aliens reference | 00:13 | |
fungi | in irc, no one can hear you scream movie quotes | 00:14 |
anteaya | ha ha ha | 00:15 |
clarkb | I am bad at movie quotes | 00:15 |
anteaya | I'm pretty good, if I have seen the movie | 00:15 |
anteaya | I'm really bad at song titles | 00:15 |
davidlenwell | I do some of my best communication with movie qoutes | 00:15 |
*** Ryan_Lane1 has joined #openstack-infra | 00:16 | |
davidlenwell | now that I see refstack in stackforge I am going to go enjoy a birthday beer .. or 7 | 00:16 |
davidlenwell | thanks guys | 00:17 |
fungi | wow. i keep cleaning up and re-running manage projects and am encountering a beautiful menagerie of seemingly random failures | 00:17 |
clarkb | fungi: that log you have for manage projects seems to be missing information | 00:17 |
clarkb | did we not run it at debug level? | 00:17 |
clarkb | it should also include what the actual commands are I thought | 00:17 |
fungi | clarkb: that's at info level and above | 00:17 |
fungi | i can rerun it at debug | 00:17 |
clarkb | but I think it is trying to git add groups which failed | 00:18 |
fungi | i'll do a new one at debug | 00:18 |
clarkb | and for the groups file to be written we need to be able to get the group uuids which if the code still works the way I thought it did would short circuit the git add | 00:18 |
fungi | these were the last two i did (not debug yet)... http://paste.openstack.org/show/70809 http://paste.openstack.org/show/70810 | 00:19 |
anteaya | davidlenwell: happy birthday, enjoy your beer | 00:19 |
fungi | but those are with remote rm instead of remote remove, so that bug seems solved at least | 00:19 |
*** valentinbud has quit IRC | 00:20 | |
clarkb | fungi: are you running some conglomeration of code for jeepyb? is it available anywhere for reading not on review.o.o? | 00:20 |
clarkb | I want to reread the stuff around group creation | 00:20 |
davidlenwell | thanks anteaya | 00:20 |
fungi | clarkb: the only changes are applying 74118, 74130 and 77298 plus my comment in that last patch | 00:21 |
fungi | here's one at debug level... http://paste.openstack.org/show/70811 | 00:21 |
clarkb | gah that cron is still not working | 00:22 |
*** VijayTripathi1 has quit IRC | 00:24 | |
fungi | i'm going to go diving in the gerrit ssh and error logs to see if i can spot anything related to group creation | 00:24 |
anteaya | hmmmmm fatal: pathspec 'groups' did not match any files | 00:24 |
clarkb | anteaya: that means the file didn't exist | 00:26 |
clarkb | which mean jeepyb didn't write one out like it is supposed to | 00:26 |
anteaya | ah ha | 00:26 |
fungi | i find this... http://paste.openstack.org/show/70817/ | 00:27 |
anteaya | create_groups_file seems to be the function in question: http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/manage_projects.py#n203 | 00:28 |
clarkb | most likely uuids is empty | 00:29 |
anteaya | MissingObjectException ? | 00:29 |
clarkb | which may mean the regex isn't matching | 00:29 |
clarkb | no the object exists, its just an empty dict {} | 00:30 |
clarkb | notice that the write out to the groups file is wrapped in if uuids: | 00:30 |
fungi | i'm going to see if anything made it into the db | 00:30 |
*** alexpilotti has quit IRC | 00:31 | |
*** jhesketh__ has joined #openstack-infra | 00:31 | |
*** valentinbud has joined #openstack-infra | 00:31 | |
*** gokrokve_ has quit IRC | 00:32 | |
fungi | no entries for ospurge-core or ospurge-ptl in teh account_groups table | 00:32 |
anteaya | interesting | 00:33 |
*** cody-somerville has joined #openstack-infra | 00:34 | |
clarkb | fungi: shouldn't that have logged at the error level? | 00:34 |
anteaya | might it be an issue with gerritlib createGroups: http://git.openstack.org/cgit/openstack-infra/gerritlib/tree/gerritlib/gerrit.py#n184 | 00:34 |
anteaya | or is it manage-projects create_group_file? | 00:34 |
fungi | clarkb: that last 70811 paste gets debug level and i redirected all stdout/stderr to the file i captured | 00:35 |
fungi | clarkb: unless the logging config is not conveying to gerritlib for some reason | 00:35 |
clarkb | fungi: not gerritlib, http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/manage_projects.py#n217 | 00:36 |
fungi | clarkb: paramiko seems to pick it up, so i have a feeling maybe gerritlib's logging isn't quite right | 00:36 |
fungi | oh | 00:36 |
mordred | fungi: from git remote --help: | 00:36 |
mordred | git remote rename <old> <new> | 00:36 |
mordred | git remote remove <name> | 00:36 |
mordred | git remote rename <old> <new> | 00:36 |
mordred | git remote remove <name> | 00:37 |
mordred | gah | 00:37 |
mordred | double paste | 00:37 |
fungi | clarkb: unless the re.match() is coming up fail | 00:37 |
*** valentinbud has quit IRC | 00:37 | |
fungi | clarkb: crap | 00:38 |
fungi | we've been chasing a wild waterfoul | 00:38 |
fungi | fowl | 00:38 |
anteaya | foul fowl | 00:39 |
anteaya | what do you have? | 00:39 |
fungi | fix incoming | 00:39 |
*** david-lyle has joined #openstack-infra | 00:41 | |
openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct filename for stackforge/ospurge Gerrit ACL https://review.openstack.org/77308 | 00:41 |
*** jcooley_ has joined #openstack-infra | 00:41 | |
*** gokrokve has joined #openstack-infra | 00:41 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Only add upstream remotes if we track upstream https://review.openstack.org/77298 | 00:42 |
*** pmathews has quit IRC | 00:42 | |
fungi | sometimes i think i should take up deep sea fishing | 00:43 |
clarkb | ha I know that feel | 00:44 |
anteaya | tempest patch 73789 just failed in the gate on a timeout error | 00:44 |
fungi | anteaya: what's the name of the slave it ran on? | 00:45 |
*** yamahata has quit IRC | 00:45 | |
fungi | curious if it's our old friend hpcloud-az2 once more | 00:45 |
anteaya | devstack-precise-rax-iad-1899492 | 00:45 |
fungi | oh, not a network timeout, a job timeout | 00:45 |
fungi | nm | 00:45 |
*** nati_ueno has quit IRC | 00:45 | |
anteaya | a job timeout, sorry | 00:45 |
*** jp_at_hp has joined #openstack-infra | 00:47 | |
*** ok_delta has quit IRC | 00:47 | |
mordred | fungi: if you run again - you may also want to include https://review.openstack.org/#/c/75641/ | 00:48 |
mordred | fungi: or, just approve it :) | 00:48 |
fungi | mordred: yep, approved, though i still have to hand-patch it since puppet is stopped while i test this stuff | 00:49 |
mordred | ++ | 00:49 |
fungi | added | 00:50 |
*** protux has quit IRC | 00:52 | |
fungi | so here's the new error of the moment... http://paste.openstack.org/show/70823/ | 00:52 |
openstackgerrit | A change was merged to openstack-infra/config: Correct filename for stackforge/ospurge Gerrit ACL https://review.openstack.org/77308 | 00:52 |
clarkb | I may have cried wolf about the logstash worker cron failing | 00:54 |
clarkb | I think its working maybe | 00:54 |
clarkb | when I run the command direclty it works | 00:54 |
clarkb | and when I hand patch for it to log what it does it seems to be run by cron ok | 00:55 |
*** gokrokve has quit IRC | 00:56 | |
*** mriedem has joined #openstack-infra | 00:56 | |
*** wenlock has quit IRC | 00:56 | |
fungi | ran again, and SUCCESS! http://paste.openstack.org/show/70829 | 00:56 |
anteaya | so http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/manage_projects.py#n131 is failing | 00:57 |
anteaya | sometimes | 00:57 |
*** jp_at_hp has quit IRC | 00:57 | |
*** wenlock has joined #openstack-infra | 00:57 | |
*** Ryan_Lane1 has quit IRC | 00:57 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Support resetting back to a non-master branch https://review.openstack.org/77311 | 00:58 |
mordred | clarkb: you'll enoy that one ^^ | 00:58 |
mordred | fungi: woot! | 00:58 |
mordred | fungi: so, I'm guessing we feel good about landing all of the patches you had applied locally then | 00:58 |
fungi | mordred: so far, yes | 00:59 |
*** CaptTofu has joined #openstack-infra | 00:59 | |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Log logstash watchdog. https://review.openstack.org/77312 | 00:59 |
*** bhuvan__ has joined #openstack-infra | 00:59 | |
*** bhuvan___ has joined #openstack-infra | 00:59 | |
fungi | mordred: 77311 is new feature work, not addressing existing bugs, right? | 01:00 |
mordred | fungi: that's right | 01:00 |
clarkb | fungi: ^ so I am going to put that in place by hand here on 03 and 04 (my current test nodes) but I think we should approve that to get numbers and logs | 01:00 |
mordred | fungi: not urgent | 01:00 |
mordred | fungi: I just put it in because I was in there for the other patch | 01:00 |
openstackgerrit | A change was merged to openstack-infra/jeepyb: Don't split the initial branch list on all whitespace https://review.openstack.org/75641 | 01:00 |
fungi | clarkb: agreed. silently papering over failures is bad form, of course | 01:00 |
clarkb | bah, the cron is still wrong new patchset incoming | 01:00 |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Log logstash watchdog. https://review.openstack.org/77312 | 01:01 |
mordred | fungi: all failures should get paper | 01:02 |
*** hemna_ has quit IRC | 01:02 | |
fungi | that explains all the toilet paper in my yard | 01:02 |
*** rfolco_ has quit IRC | 01:02 | |
mordred | :) | 01:03 |
*** bhuvan has quit IRC | 01:03 | |
*** CaptTofu has quit IRC | 01:03 | |
*** Ryan_Lane1 has joined #openstack-infra | 01:04 | |
*** bhuvan_ has quit IRC | 01:04 | |
*** nati_ueno has joined #openstack-infra | 01:04 | |
*** prad has quit IRC | 01:04 | |
*** jcooley_ has quit IRC | 01:04 | |
clarkb | fungi: for that failure about refs/meta/config not fetching I wodner if that is the new race | 01:04 |
*** jcooley_ has joined #openstack-infra | 01:05 | |
fungi | clarkb: the pack error? | 01:05 |
clarkb | fungi: eg gerrit create project returning before there is a git repo with refs/meta/config in it on disk | 01:05 |
clarkb | fungi: ya | 01:05 |
fungi | clarkb: i think that matches up with the java stacktrace i pasted earlier too | 01:05 |
clarkb | so we create project then immediately try to fetch refs/meta/config which may not be synced to disk | 01:05 |
clarkb | might be useful to add a poll loop there | 01:05 |
anteaya | so create_project should be isolated from fetch? | 01:06 |
clarkb | and fail after a few seconds | 01:06 |
clarkb | anteaya: I think it is, its just we immediately go do the fetch | 01:06 |
clarkb | it would be so nice if these api commands were synchronous | 01:06 |
fungi | okay, so i'm moving ahead to the non-upstream new projects. brace yourselves for new and exciting failures | 01:06 |
fungi | (or not, if we're lucky) | 01:06 |
anteaya | sorry I mean in a separate file, if we were looking to salt it | 01:06 |
clarkb | anteaya: no I don't think it needs to be a separate file | 01:07 |
anteaya | clarkb: k | 01:07 |
clarkb | anteaya: we just need to be careful to wait if it isn't already available | 01:07 |
anteaya | yes | 01:07 |
anteaya | I'm trying to figure out the best way to wait | 01:07 |
*** nati_ueno has quit IRC | 01:07 | |
anteaya | if create was a separate command, that returned something | 01:07 |
anteaya | salt could wait for the something returned | 01:08 |
anteaya | then fire a different file | 01:08 |
clarkb | anteaya: it already is something that we wait for a return on | 01:08 |
anteaya | within the file? | 01:08 |
clarkb | gerrit is just being silly and returning before everything is done (thats my theory at least) | 01:08 |
clarkb | anteaya: ya | 01:08 |
anteaya | oh | 01:08 |
clarkb | we use gerritlib to create a project and that returns prior to the project being created | 01:08 |
anteaya | the returned thing is not a response that indicates the actual state | 01:09 |
anteaya | it is worthless returned thing | 01:09 |
clarkb | rgith at least that is my theory | 01:09 |
*** gyee has quit IRC | 01:09 | |
anteaya | you are probably correct | 01:09 |
anteaya | or gerritlib is silly | 01:09 |
anteaya | http://git.openstack.org/cgit/openstack-infra/gerritlib/tree/gerritlib/gerrit.py#n194 | 01:10 |
anteaya | we say return err | 01:10 |
anteaya | I am not seeing any waiting | 01:10 |
*** openstackgerrit has quit IRC | 01:10 | |
anteaya | should there be waiting in gerritlib/createProject? | 01:11 |
*** openstackgerrit has joined #openstack-infra | 01:11 | |
*** yamahata has joined #openstack-infra | 01:11 | |
clarkb | fungi I will have a patch for you really quick if you can wait a minute | 01:13 |
fungi | clarkb: it's already running | 01:13 |
fungi | but it may fail spectacularly anyway! | 01:13 |
fungi | so you could just hold out hope for that ;) | 01:13 |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/jeepyb: Poll manage projects config fetches. https://review.openstack.org/77314 | 01:14 |
clarkb | fungi: ^ that is a bit hacky and could use better logging and a better commit message but I think that should allow us to confirm what is happenin | 01:14 |
clarkb | and if that is indeed happening it may explain all of the trouble we have had | 01:14 |
clarkb | fungi: mordred https://review.openstack.org/#/c/77312/ it would be great to have that in place before the weekend so that the restarts get logged | 01:15 |
clarkb | it is running on 03 and 04 and seems happy | 01:15 |
openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Trim down gantt check/gate jobs https://review.openstack.org/71648 | 01:15 |
*** zehicle_at_dell has joined #openstack-infra | 01:15 | |
anteaya | for me, if gerritlib/createProject is the place that is supposed to return something useful when the project is created, I think the fix should go there | 01:17 |
SpamapS | is there a quick way I can find jenkins jobs running tripleo-* tests? | 01:17 |
anteaya | then if anything else uses createProject, it won't hit the same error | 01:17 |
fungi | clarkb: lgtm. feel free to wait for mordred or just approve before you knock off | 01:17 |
clarkb | anteaya: but as best as createProject can tell the project has been created | 01:17 |
clarkb | fungi: I will approve because impatient >_> | 01:18 |
*** nati_ueno has joined #openstack-infra | 01:18 | |
fungi | clarkb: as for 77314 i'll patch it in on the next run if we need another | 01:18 |
clarkb | ok | 01:18 |
*** zehicle has quit IRC | 01:18 | |
*** CaptTofu has joined #openstack-infra | 01:18 | |
SpamapS | like, given https://review.openstack.org/#/c/76675/ ... how can I find the in-progress jenkins job to watch? | 01:19 |
anteaya | SpamapS: I did `grep -R "tripleo" modules/openstack_project/files/zuul/layout.yaml` in config | 01:19 |
fungi | SpamapS: can you refine what you mean? the names of the jobs or their contents? | 01:19 |
anteaya | you get more than you need, but it is there | 01:19 |
SpamapS | We've been having hardware issues.. | 01:19 |
SpamapS | we're hoping one is solved.. | 01:19 |
mordred | clarkb: looks good - any reason to not +A? | 01:19 |
SpamapS | I see VMs spinning up.. but.. how to watch.. ?? | 01:19 |
*** krotscheck has quit IRC | 01:20 | |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/jeepyb: Poll manage projects config fetches. https://review.openstack.org/77314 | 01:20 |
clarkb | that is a bit cleaner | 01:20 |
clarkb | mordred: which change? the watchdog logging one? I just approved it | 01:20 |
mordred | clarkb: the jeepyb poll loop | 01:20 |
clarkb | mordred: oh, only that it may not actually solve anything | 01:20 |
anteaya | clarkb: but I thought you said the value createProjects returns is treated as an indication the project has been created | 01:20 |
clarkb | anteaya: yes as far as gerrit is concerned | 01:20 |
anteaya | clarkb: when in actual fact it hasn't | 01:21 |
fungi | SpamapS: jenkins jobs to not spin up vms. nodepoold does that | 01:21 |
clarkb | anteaya: gerrit is asserting the project is created there | 01:21 |
openstackgerrit | A change was merged to openstack-infra/config: Log logstash watchdog. https://review.openstack.org/77312 | 01:21 |
clarkb | other code needs to be defensive | 01:21 |
anteaya | clarkb: how is gerrit asserting the project has been created? | 01:21 |
anteaya | what I see is return err | 01:22 |
anteaya | which to my eyes isn't indicative of the existance of anything | 01:22 |
fungi | SpamapS: but if you want to see the progress of jobs which run on them, start from http://status.openstack.org/zuul/ and look in the check-tripleo or experimental-tripleo pipeline sections for changes which have jobs in progress, then click on the name of one which has a progress bar next to it | 01:22 |
anteaya | unless I don't understand | 01:22 |
clarkb | anteaya: because gerritlib has said the project is created | 01:22 |
SpamapS | fungi: well I'm not so much interested in the vms that are spinning up, but just seeing that jobs are in fact going. | 01:22 |
mordred | anteaya: the gerrit api is returning when it's done | 01:22 |
fungi | SpamapS: right now i see no signs any are running jobs | 01:22 |
mordred | anteaya: but under the covers, gerrit is not actually done | 01:22 |
mordred | anteaya: but the api thinks that it is | 01:23 |
SpamapS | fungi: hm yeah, I think all of the vms I see are left over zombies | 01:23 |
anteaya | clarkb: what I am saying is that gerritlib is wrong | 01:23 |
clarkb | I have cleaned up after myself, puppet is running on 03 and 04 again. going to grab a beer now will touch base later | 01:23 |
mordred | it's not gerritlib | 01:23 |
clarkb | anteaya: but it isn't | 01:23 |
mordred | gerrit is lying to gerritlib | 01:23 |
anteaya | clarkb: what isn't what | 01:23 |
anteaya | gerrit is lying to gerritlib | 01:23 |
clarkb | gerritlib isn't wrong. gerrit told it the project was created | 01:23 |
mordred | but gerritlib doesn't have any context to know that it's being lied to | 01:23 |
anteaya | what is gerrit giving to gerritlib? | 01:23 |
fungi | SpamapS: i'm trying to delete some whichi have been left in various states for more than 3 hours now... looks like we may be timing out reaching your api endpoint though | 01:23 |
clarkb | anteaya: a successful return code | 01:23 |
mordred | so you can't do the check in gerritlib | 01:24 |
SpamapS | fungi: endpoint should be up now | 01:24 |
*** VijayTripathi has joined #openstack-infra | 01:24 | |
anteaya | so gerrit is return success even though there is no success | 01:24 |
mordred | anteaya: yes | 01:24 |
anteaya | ah okay | 01:24 |
fungi | SpamapS: ooh, fun, i'm hitting an sqlalchemy lock wait timeout instead at the moment | 01:24 |
mordred | anteaya: in manage-projects, we have a git repo, so we can do additional git operations that gerritlib does not have the context to perform - so we can loop and check there | 01:24 |
fungi | SpamapS: so i can't confirm just yet | 01:24 |
anteaya | mordred: so we are assuming that if the git repo is success that gerrit is success | 01:25 |
anteaya | and ignore gerrit's return code | 01:25 |
mordred | anteaya: well, kinda | 01:25 |
anteaya | mordred: what part am I missing? | 01:25 |
SpamapS | fungi: mmk. | 01:25 |
mordred | anteaya: actually, no. you're right. we're checking the git repo for success | 01:26 |
mordred | because that's the ultimate source of truth for us | 01:26 |
anteaya | and extrapolating that if the git repo is up that gerrits is up | 01:26 |
anteaya | because we can't trust gerrit's return code | 01:26 |
fungi | mordred: so rerunning a full (not specific projects) manage-projects with your proposed fix for upstream tracking, it still looks like it's checking out all the projects. i'll get a new log pasted shortly, when it eventually finishes running | 01:27 |
*** lttrl has joined #openstack-infra | 01:27 | |
mordred | fungi: it will check out all of the projects - but it should skip the remote update step | 01:27 |
fungi | mordred: here's the log from this run... http://paste.openstack.org/show/70838 | 01:30 |
mordred | fungi: looking | 01:30 |
anteaya | fungi: so it was successful? | 01:30 |
fungi | anteaya: no idea yet... i'm checking | 01:31 |
torandu_ | hi. i have another q re jjb. should this be possible? http://paste.openstack.org/show/70839/ specifically, setting builders on the jobs. the docs seem to suggest that that would be possible, although it's not working for me | 01:31 |
anteaya | so far I see warm in there | 01:32 |
mordred | fungi: you know what ... | 01:32 |
anteaya | and openstack-sdk-dotnet | 01:32 |
anteaya | and openstack-cli-powershell | 01:33 |
fungi | okay, those three projects all seem to have been created successfully | 01:33 |
fungi | their acls are added and i can clone them and list branches from all members of our git server farm | 01:33 |
anteaya | heat-translator | 01:33 |
anteaya | awesome | 01:33 |
*** bhuvan___ has quit IRC | 01:34 | |
*** bhuvan__ has quit IRC | 01:34 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Only add upstream remotes if we track upstream https://review.openstack.org/77298 | 01:34 |
mordred | fungi: updated | 01:34 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Support resetting back to a non-master branch https://review.openstack.org/77311 | 01:34 |
mordred | fungi: moved the remote update into the if track_upstream block | 01:34 |
mordred | fungi: we don't really need to do a remote update otherwise | 01:35 |
anteaya | os-cloud-config | 01:35 |
mordred | fungi: eventually I'd been planning on making an attempt to read setup.cfg in master for description and homepage | 01:35 |
SpamapS | heat-translator? wha? | 01:35 |
fungi | SpamapS: it translates heat into something usable--haven't you heard? ;) | 01:35 |
anteaya | SpamapS: https://review.openstack.org/75988 | 01:36 |
SpamapS | no but... i can haz? such happy. need now. | 01:36 |
SpamapS | nice | 01:36 |
fungi | looks like something ibm must be using internally | 01:36 |
fungi | the readme is succinct... http://git.openstack.org/cgit/stackforge/heat-translator/tree/README.md | 01:37 |
fungi | oh wow. they committed .pyc files in their repo | 01:37 |
* fungi should know better than to peek inside a stackforge project | 01:38 | |
anteaya | the name should be heat-tanslator | 01:38 |
*** thuc has quit IRC | 01:39 | |
fungi | okay, since "new project friday" is over (yaaay! thanks everyone! flailing kermit arms!) i'm going to turn puppet agent back on for review.openstack.org | 01:40 |
anteaya | nice work fungi | 01:40 |
anteaya | mordred: do we have a way forward? | 01:40 |
*** thuc has joined #openstack-infra | 01:40 | |
clarkb | fungi ha | 01:40 |
anteaya | besides what was addressed tonight? | 01:40 |
clarkb | dat pyc | 01:40 |
* fungi snorfles | 01:40 | |
torandu_ | pls disregard my prev question re jjb. i figured it out | 01:42 |
fungi | anteaya: i believe we're continuing "new project fridays" as a weekly exercise until we've worked out the remaining kinks, but yes we identified several actual and theoretical bugs and have patches either proposed or approved so i think we're in good shape | 01:42 |
fungi | and next week, if there are any pending new projects (at least the several in the "needs work" section from this week's etherpad are likely candidates) we'll see what happens | 01:43 |
anteaya | that is good | 01:43 |
anteaya | I just have one week left before I disappear | 01:43 |
*** VijayTripathi has quit IRC | 01:43 | |
anteaya | and want to hold up my end of helping mordred to reach the goal of having this autmated by mid-March | 01:44 |
*** thuc has quit IRC | 01:44 | |
anteaya | I'm not as good yet as picking up the ball and running with it | 01:44 |
*** melwitt has quit IRC | 01:44 | |
anteaya | automated | 01:44 |
fungi | i'm not good with sports analogies either so don't feel bad | 01:44 |
anteaya | okay | 01:45 |
anteaya | when you and clark and jim are talking | 01:45 |
anteaya | I can't tell what I should be doing to help | 01:45 |
anteaya | though it might seem obvious to others | 01:45 |
anteaya | other than answering the phone | 01:45 |
*** gokrokve has joined #openstack-infra | 01:46 | |
* anteaya goes to bed | 01:48 | |
*** SumitNaiksatam has quit IRC | 01:51 | |
*** dcramer__ has joined #openstack-infra | 01:51 | |
*** Ryan_Lane1 has quit IRC | 01:53 | |
*** flashgordon is now known as jogo | 01:54 | |
*** lnxnut has joined #openstack-infra | 02:10 | |
*** Hunner has quit IRC | 02:12 | |
*** Hunner has joined #openstack-infra | 02:18 | |
*** sahumada has quit IRC | 02:18 | |
*** CaptTofu has quit IRC | 02:29 | |
*** CaptTofu has joined #openstack-infra | 02:29 | |
*** SumitNaiksatam has joined #openstack-infra | 02:31 | |
*** CaptTofu has quit IRC | 02:33 | |
openstackgerrit | A change was merged to openstack-infra/devstack-gate: Write DO_NOT_UPGRADE_SERVICES to grenade's localrc not devstacks https://review.openstack.org/76658 | 02:39 |
*** pcm_ has joined #openstack-infra | 02:42 | |
*** mriedem has quit IRC | 02:44 | |
fungi | nodepoold seems to be overlimit in rax-iad (probably the 5000 builds/day again), so it's looping rapidly trying to add nodes there and holding a perpetual lock on the db such that i can't delete the tripleo nodes which have been hung in a building state for over 4 hours | 02:46 |
fungi | i suspect we just need to wait it out | 02:46 |
* fungi is going to watch a movie | 02:47 | |
lifeless | fungi: we can delete tripleo nodes | 02:48 |
lifeless | fungi: will nodepool retry if we delete them from nova ? | 02:49 |
lifeless | SpamapS: oh, probably want to restart all the n-comutes, we have error-deleting state again | 02:49 |
*** thuc has joined #openstack-infra | 02:51 | |
*** rfolco_ has joined #openstack-infra | 02:52 | |
*** lnxnut has quit IRC | 02:52 | |
*** lnxnut has joined #openstack-infra | 02:55 | |
*** thuc has quit IRC | 02:55 | |
*** dolphm is now known as dolphm_503 | 03:00 | |
*** rfolco_ has quit IRC | 03:01 | |
*** jerryz has quit IRC | 03:01 | |
*** pcrews has quit IRC | 03:09 | |
*** annegentle has joined #openstack-infra | 03:10 | |
*** reed has quit IRC | 03:10 | |
*** talluri has joined #openstack-infra | 03:13 | |
*** CaptTofu has joined #openstack-infra | 03:16 | |
*** wenlock has quit IRC | 03:17 | |
*** talluri has quit IRC | 03:21 | |
*** pmathews has joined #openstack-infra | 03:28 | |
*** weshay has quit IRC | 03:28 | |
*** amcrn has quit IRC | 03:29 | |
zaro | pabelanger: i could not repro https://code.google.com/p/gerrit/issues/detail?id=2296 I tested on review-dev.o.o, you can give it a try on there as well. | 03:30 |
*** talluri has joined #openstack-infra | 03:30 | |
zaro | pabelanger: i'm using git ver 1.8.5.1 that i installed from brew repo. | 03:32 |
*** julim has quit IRC | 03:39 | |
*** dolphm_503 is now known as dolphm | 03:39 | |
*** CaptTofu has quit IRC | 03:42 | |
*** dolphm is now known as dolphm_503 | 03:52 | |
*** talluri has quit IRC | 03:58 | |
*** lttrl has quit IRC | 04:07 | |
*** lttrl has joined #openstack-infra | 04:07 | |
*** lnxnut has quit IRC | 04:18 | |
*** pcm_ has quit IRC | 04:20 | |
*** lnxnut has joined #openstack-infra | 04:22 | |
*** harlowja is now known as harlowja_away | 04:31 | |
*** dolphm_503 is now known as dolphm | 04:43 | |
*** chandan_kumar has joined #openstack-infra | 04:48 | |
*** wchrisj has quit IRC | 04:49 | |
*** jcooley_ has quit IRC | 04:50 | |
*** jcooley_ has joined #openstack-infra | 04:50 | |
*** jcooley_ has quit IRC | 04:55 | |
*** dolphm is now known as dolphm_503 | 04:55 | |
*** wchrisj__ has joined #openstack-infra | 05:16 | |
*** CaptTofu has joined #openstack-infra | 05:43 | |
*** VijayTripathi has joined #openstack-infra | 05:45 | |
*** dolphm_503 is now known as dolphm | 05:46 | |
*** dcramer__ has quit IRC | 05:47 | |
*** CaptTofu has quit IRC | 05:48 | |
*** pmathews has quit IRC | 05:50 | |
*** dolphm is now known as dolphm_503 | 05:56 | |
*** dcramer__ has joined #openstack-infra | 06:00 | |
*** chandan_kumar has quit IRC | 06:01 | |
*** lttrl has quit IRC | 06:13 | |
*** chandan_kumar has joined #openstack-infra | 06:13 | |
*** vkozhukalov has joined #openstack-infra | 06:18 | |
*** jcooley_ has joined #openstack-infra | 06:19 | |
*** wchrisj__ has quit IRC | 06:22 | |
*** VijayTripathi has quit IRC | 06:24 | |
*** chandan_kumar has quit IRC | 06:28 | |
*** jcooley_ has quit IRC | 06:34 | |
lifeless | fungi: nodepool recovering from rax? | 06:37 |
*** marun has quit IRC | 06:39 | |
*** chandan_kumar has joined #openstack-infra | 06:40 | |
*** marun has joined #openstack-infra | 06:40 | |
*** dstanek has quit IRC | 06:46 | |
*** dolphm_503 is now known as dolphm | 06:47 | |
*** nati_ueno has quit IRC | 06:47 | |
*** chandan_kumar has quit IRC | 06:48 | |
*** nati_ueno has joined #openstack-infra | 06:48 | |
*** jamespage_ has quit IRC | 06:50 | |
*** jhesketh__ has quit IRC | 06:54 | |
*** valentinbud has joined #openstack-infra | 06:55 | |
*** dolphm is now known as dolphm_503 | 06:56 | |
*** chandan_kumar has joined #openstack-infra | 06:58 | |
*** nati_ueno has quit IRC | 07:02 | |
*** jcooley_ has joined #openstack-infra | 07:08 | |
*** saper has quit IRC | 07:16 | |
*** chandan_kumar has quit IRC | 07:22 | |
*** valentinbud has quit IRC | 07:27 | |
*** yolanda_ has joined #openstack-infra | 07:32 | |
*** ArxCruz_ has quit IRC | 07:37 | |
*** CaptTofu has joined #openstack-infra | 07:44 | |
morganfainberg | lifeless, now i know you can't claim it's Monday today! | 07:47 |
morganfainberg | lifeless, >.> | 07:47 |
*** dolphm_503 is now known as dolphm | 07:47 | |
*** CaptTofu has quit IRC | 07:49 | |
lifeless | morganfainberg: Saturday :P | 07:56 |
morganfainberg | lifeless, it's ... almost saturday here | 07:56 |
*** dolphm is now known as dolphm_503 | 07:57 | |
*** cody-somerville has quit IRC | 07:57 | |
*** ianw has quit IRC | 08:02 | |
openstackgerrit | A change was merged to openstack-infra/jeepyb: Change manage-projects verbose logging to info https://review.openstack.org/74118 | 08:05 |
*** jcooley_ has quit IRC | 08:10 | |
*** davidhadas has joined #openstack-infra | 08:17 | |
*** davidhadas_ has quit IRC | 08:18 | |
*** jdurgin has quit IRC | 08:18 | |
*** NikitaKonovalov_ is now known as NikitaKonovalov | 08:27 | |
*** jdurgin has joined #openstack-infra | 08:32 | |
*** morganfainberg is now known as morganfainberg_Z | 08:33 | |
*** gokrokve has quit IRC | 08:38 | |
*** gokrokve_ has joined #openstack-infra | 08:41 | |
*** gokrokv__ has joined #openstack-infra | 08:43 | |
*** gokrokve_ has quit IRC | 08:45 | |
*** dolphm_503 is now known as dolphm | 08:48 | |
*** sarob_ has joined #openstack-infra | 08:50 | |
*** sarob_ has quit IRC | 08:54 | |
*** NikitaKonovalov is now known as NikitaKonovalov_ | 08:55 | |
*** dolphm is now known as dolphm_503 | 08:58 | |
*** Ryan_Lane has quit IRC | 09:00 | |
*** jcooley_ has joined #openstack-infra | 09:02 | |
*** vkozhukalov has quit IRC | 09:08 | |
*** david_lyle_ has joined #openstack-infra | 09:10 | |
*** david-lyle has quit IRC | 09:13 | |
*** talluri has joined #openstack-infra | 09:14 | |
*** david_lyle_ has quit IRC | 09:15 | |
*** gokrokv__ has quit IRC | 09:17 | |
*** jcooley_ has quit IRC | 09:32 | |
*** Ryan_Lane has joined #openstack-infra | 09:32 | |
*** sdake__ has quit IRC | 09:36 | |
*** Ryan_Lane has quit IRC | 09:41 | |
*** talluri has quit IRC | 09:41 | |
*** talluri has joined #openstack-infra | 09:42 | |
*** CaptTofu has joined #openstack-infra | 09:45 | |
*** gokrokve has joined #openstack-infra | 09:48 | |
*** dolphm_503 is now known as dolphm | 09:49 | |
*** gokrokve_ has joined #openstack-infra | 09:49 | |
*** CaptTofu has quit IRC | 09:50 | |
*** talluri has quit IRC | 09:50 | |
*** chandan_kumar has joined #openstack-infra | 09:50 | |
*** gokrokve has quit IRC | 09:52 | |
*** fifieldt has quit IRC | 09:56 | |
*** jcooley_ has joined #openstack-infra | 09:58 | |
*** dolphm is now known as dolphm_503 | 09:59 | |
*** chandan_kumar has quit IRC | 10:02 | |
*** jhesketh_ has quit IRC | 10:03 | |
*** jhesketh_ has joined #openstack-infra | 10:16 | |
*** valentinbud has joined #openstack-infra | 10:16 | |
*** talluri has joined #openstack-infra | 10:18 | |
*** chandan_kumar has joined #openstack-infra | 10:24 | |
*** jcooley_ has quit IRC | 10:30 | |
*** alexpilotti has joined #openstack-infra | 10:40 | |
openstackgerrit | Lukasz Jernas proposed a change to openstack-infra/config: Add fonts-takao package on the Debian build slave https://review.openstack.org/72287 | 10:42 |
*** davidhadas has quit IRC | 10:45 | |
*** dolphm_503 is now known as dolphm | 10:50 | |
*** sarob_ has joined #openstack-infra | 10:51 | |
*** sarob_ has quit IRC | 10:56 | |
*** jcooley_ has joined #openstack-infra | 10:59 | |
*** ildikov_ has joined #openstack-infra | 10:59 | |
*** dolphm is now known as dolphm_503 | 11:00 | |
*** valentinbud has quit IRC | 11:04 | |
*** NikitaKonovalov_ is now known as NikitaKonovalov | 11:25 | |
*** NikitaKonovalov is now known as NikitaKonovalov_ | 11:26 | |
*** jcooley_ has quit IRC | 11:29 | |
*** ildikov_ has quit IRC | 11:31 | |
*** chandan_kumar has quit IRC | 11:36 | |
*** NikitaKonovalov_ is now known as NikitaKonovalov | 11:36 | |
*** talluri has quit IRC | 11:37 | |
*** NikitaKonovalov is now known as NikitaKonovalov_ | 11:37 | |
*** CaptTofu has joined #openstack-infra | 11:43 | |
*** dolphm_503 is now known as dolphm | 11:50 | |
*** chandan_kumar has joined #openstack-infra | 11:53 | |
*** alexpilotti has quit IRC | 11:54 | |
*** jcooley_ has joined #openstack-infra | 11:55 | |
*** dolphm is now known as dolphm_503 | 12:00 | |
*** ildikov_ has joined #openstack-infra | 12:03 | |
*** HenryG_ has joined #openstack-infra | 12:06 | |
*** HenryG has quit IRC | 12:09 | |
*** lcostantino has joined #openstack-infra | 12:09 | |
*** ryanpetrello has joined #openstack-infra | 12:15 | |
*** CaptTofu has quit IRC | 12:17 | |
*** lcostantino has quit IRC | 12:18 | |
*** yamahata has quit IRC | 12:18 | |
*** jhesketh_ has quit IRC | 12:21 | |
*** e0ne has joined #openstack-infra | 12:24 | |
*** jcooley_ has quit IRC | 12:26 | |
*** dizquierdo has joined #openstack-infra | 12:27 | |
*** ryanpetrello has quit IRC | 12:30 | |
*** yamahata has joined #openstack-infra | 12:32 | |
*** valentinbud has joined #openstack-infra | 12:34 | |
*** ryanpetrello has joined #openstack-infra | 12:36 | |
*** banix has joined #openstack-infra | 12:37 | |
*** valentinbud has quit IRC | 12:37 | |
*** chandan_kumar has quit IRC | 12:38 | |
*** valentinbud has joined #openstack-infra | 12:38 | |
*** dizquierdo has quit IRC | 12:40 | |
*** CaptTofu has joined #openstack-infra | 12:40 | |
banix | Noticed the jobs at the gate for patches being merged takes several hours; Is this how it should be? The tests get run many many times? Just wondering. | 12:41 |
*** alexpilotti has joined #openstack-infra | 12:44 | |
*** yamahata has quit IRC | 12:45 | |
*** ryanpetrello has quit IRC | 12:46 | |
*** valentinbud has quit IRC | 12:48 | |
*** dolphm_503 is now known as dolphm | 12:51 | |
*** sarob_ has joined #openstack-infra | 12:52 | |
*** jcooley_ has joined #openstack-infra | 12:55 | |
*** davidhadas has joined #openstack-infra | 12:56 | |
*** sarob_ has quit IRC | 12:57 | |
*** dolphm is now known as dolphm_503 | 13:01 | |
*** yamahata has joined #openstack-infra | 13:01 | |
*** lcostantino has joined #openstack-infra | 13:02 | |
*** weshay has joined #openstack-infra | 13:03 | |
*** lcostantino has quit IRC | 13:08 | |
*** yamahata has quit IRC | 13:08 | |
*** yamahata has joined #openstack-infra | 13:13 | |
*** lcostantino has joined #openstack-infra | 13:14 | |
*** unicell has joined #openstack-infra | 13:20 | |
*** lcostantino has quit IRC | 13:21 | |
*** talluri has joined #openstack-infra | 13:24 | |
*** jcooley_ has quit IRC | 13:26 | |
*** alexpilotti has quit IRC | 13:27 | |
*** yamahata has quit IRC | 13:28 | |
*** Ryan_Lane has joined #openstack-infra | 13:28 | |
*** Ryan_Lane has quit IRC | 13:33 | |
*** dstanek has joined #openstack-infra | 13:33 | |
*** dolphm_503 is now known as dolphm | 13:35 | |
dhellmann_ | good morning | 13:41 |
*** dhellmann_ is now known as dhellmann | 13:41 | |
*** jcooley_ has joined #openstack-infra | 13:44 | |
*** dhellmann is now known as dhellmann_ | 13:57 | |
*** yamahata has joined #openstack-infra | 13:57 | |
*** yamahata has quit IRC | 14:02 | |
*** rfolco has joined #openstack-infra | 14:03 | |
*** yamahata has joined #openstack-infra | 14:07 | |
*** jcooley_ has quit IRC | 14:16 | |
*** dolphm is now known as dolphm_503 | 14:17 | |
*** dstanek has quit IRC | 14:22 | |
*** yamahata has quit IRC | 14:24 | |
*** e0ne has quit IRC | 14:24 | |
*** zehicle_at_dell has quit IRC | 14:36 | |
*** dstanek has joined #openstack-infra | 14:36 | |
*** dolphm_503 is now known as dolphm | 14:36 | |
*** rfolco has quit IRC | 14:44 | |
*** vkozhukalov has joined #openstack-infra | 14:47 | |
*** ryanpetrello has joined #openstack-infra | 14:48 | |
*** Adri2000 has quit IRC | 14:50 | |
*** ryanpetrello has quit IRC | 14:51 | |
*** jcooley_ has joined #openstack-infra | 14:52 | |
*** ryanpetrello has joined #openstack-infra | 14:52 | |
*** blamar has quit IRC | 14:52 | |
*** sarob_ has joined #openstack-infra | 14:54 | |
*** lnxnut has quit IRC | 14:58 | |
*** sarob_ has quit IRC | 14:58 | |
*** e0ne has joined #openstack-infra | 15:03 | |
*** lnxnut has joined #openstack-infra | 15:04 | |
*** dolphm is now known as dolphm_503 | 15:05 | |
*** davidhadas has quit IRC | 15:10 | |
*** e0ne has quit IRC | 15:12 | |
*** Adri2000 has joined #openstack-infra | 15:14 | |
*** Adri2000 has quit IRC | 15:14 | |
*** Adri2000 has joined #openstack-infra | 15:14 | |
*** jcooley_ has quit IRC | 15:15 | |
*** jcooley_ has joined #openstack-infra | 15:15 | |
*** ryanpetrello has quit IRC | 15:16 | |
*** e0ne has joined #openstack-infra | 15:22 | |
*** jcooley_ has quit IRC | 15:24 | |
*** dolphm_503 is now known as dolphm | 15:25 | |
*** e0ne has quit IRC | 15:26 | |
*** lnxnut has quit IRC | 15:27 | |
*** dstanek has quit IRC | 15:28 | |
*** unicell has quit IRC | 15:34 | |
*** lnxnut has joined #openstack-infra | 15:35 | |
*** alexpilotti has joined #openstack-infra | 15:37 | |
*** dolphm is now known as dolphm_503 | 15:37 | |
*** lnxnut has quit IRC | 15:39 | |
*** jcooley_ has joined #openstack-infra | 15:40 | |
*** e0ne has joined #openstack-infra | 15:43 | |
*** unicell has joined #openstack-infra | 15:43 | |
*** talluri has quit IRC | 15:43 | |
*** ArxCruz has joined #openstack-infra | 15:44 | |
*** sdake__ has joined #openstack-infra | 15:44 | |
*** sdake__ has quit IRC | 15:47 | |
*** sdake_ has joined #openstack-infra | 15:48 | |
*** thuc has joined #openstack-infra | 15:49 | |
*** e0ne has quit IRC | 15:53 | |
*** mestery_ has quit IRC | 15:54 | |
*** mestery has joined #openstack-infra | 15:54 | |
*** lttrl has joined #openstack-infra | 15:54 | |
*** dolphm_503 is now known as dolphm | 15:56 | |
*** blamar has joined #openstack-infra | 15:56 | |
*** e0ne has joined #openstack-infra | 16:02 | |
*** pcm_ has joined #openstack-infra | 16:03 | |
*** sarob_ has joined #openstack-infra | 16:05 | |
*** jcooley_ has quit IRC | 16:07 | |
*** jcooley_ has joined #openstack-infra | 16:07 | |
*** sarob_ has quit IRC | 16:09 | |
*** thuc has quit IRC | 16:20 | |
*** thuc has joined #openstack-infra | 16:20 | |
*** davidhadas has joined #openstack-infra | 16:21 | |
*** e0ne has quit IRC | 16:21 | |
*** alexpilotti has quit IRC | 16:22 | |
*** thuc has quit IRC | 16:24 | |
*** jcooley_ has quit IRC | 16:25 | |
*** pcrews has joined #openstack-infra | 16:27 | |
*** dstanek has joined #openstack-infra | 16:28 | |
*** thuc has joined #openstack-infra | 16:30 | |
*** thuc has quit IRC | 16:34 | |
*** pcrews has quit IRC | 16:34 | |
*** thuc has joined #openstack-infra | 16:34 | |
*** lnxnut has joined #openstack-infra | 16:35 | |
SergeyLukjanov | dhellmann_, hey, what's up? | 16:37 |
*** DinaBelova has joined #openstack-infra | 16:38 | |
*** NikitaKo- has joined #openstack-infra | 16:39 | |
*** ruhe- has joined #openstack-infra | 16:39 | |
*** NikitaKo- is now known as NikitaKonovalov | 16:39 | |
*** NikitaKonovalov_ has quit IRC | 16:39 | |
*** SergeyLukjanov2 has joined #openstack-infra | 16:40 | |
*** ruhe has quit IRC | 16:40 | |
*** ruhe- is now known as ruhe | 16:40 | |
*** lnxnut has quit IRC | 16:40 | |
*** dolphm is now known as dolphm_503 | 16:40 | |
*** SergeyLukjanov has quit IRC | 16:41 | |
*** SergeyLukjanov2 is now known as SergeyLukjanov | 16:41 | |
*** DinaBelova_ has quit IRC | 16:41 | |
*** pmathews has joined #openstack-infra | 16:49 | |
*** valentinbud has joined #openstack-infra | 16:49 | |
*** talluri has joined #openstack-infra | 16:50 | |
*** thuc_ has joined #openstack-infra | 16:50 | |
openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Actually restart services in logstash watchdog https://review.openstack.org/77356 | 16:50 |
*** valentinbud has quit IRC | 16:51 | |
*** valentinbud has joined #openstack-infra | 16:51 | |
*** valentinbud has quit IRC | 16:52 | |
*** thuc has quit IRC | 16:54 | |
*** yolanda_ has quit IRC | 16:54 | |
openstackgerrit | A change was merged to openstack-infra/config: Actually restart services in logstash watchdog https://review.openstack.org/77356 | 16:55 |
*** sarob_ has joined #openstack-infra | 16:55 | |
mordred | banix: yes. it can take several hours, depending on the situation | 16:57 |
clarkb | mordred: I self approved ^ | 16:58 |
clarkb | because I want reliable indexing before I weekend | 16:58 |
*** lnxnut has joined #openstack-infra | 17:00 | |
*** sarob_ has quit IRC | 17:00 | |
*** pmathews has quit IRC | 17:03 | |
banix | mordred: Thanks, do you know for neutron patches, how many times the set of tests are run for merges? Just curious. | 17:05 |
*** thuc has joined #openstack-infra | 17:07 | |
clarkb | they run twice if the check results are older than 24 hours old. First the change goes through the check queue and if that passed it goes through the gate queue. If the check results are recent (less than 24 hours old) it is just once through the gate queue. However anytime the gate resets will cause tests to rerun there which is a variable number | 17:08 |
*** thuc_ has quit IRC | 17:11 | |
*** thuc has quit IRC | 17:12 | |
*** dcramer__ has quit IRC | 17:12 | |
mordred | clarkb: kk | 17:17 |
openstackgerrit | Alex Gaynor proposed a change to openstack-dev/pbr: Declare support for Python versions in setup.cfg https://review.openstack.org/75868 | 17:18 |
*** jcooley_ has joined #openstack-infra | 17:22 | |
*** wchrisj has joined #openstack-infra | 17:22 | |
*** jcooley_ has quit IRC | 17:24 | |
*** thuc has joined #openstack-infra | 17:26 | |
*** wchrisj has quit IRC | 17:26 | |
sdague | clarkb: I salute your work to get logstash back up to speed. Will be really happy to have that up to date again. | 17:26 |
clarkb | me too :) | 17:26 |
clarkb | I think at this point the major issues are addressed or worked around | 17:26 |
clarkb | the change above restarted 4 indexers after it went in and we are digging through the backlog once again | 17:27 |
clarkb | hopefully upstream can address that bug | 17:27 |
*** thuc_ has joined #openstack-infra | 17:27 | |
*** thuc has quit IRC | 17:28 | |
sdague | coolio | 17:28 |
*** wchrisj has joined #openstack-infra | 17:28 | |
*** wchrisj has quit IRC | 17:31 | |
clarkb | then after feature freeze I think we want to grow our index filesystems to 2TB each. 1TB is startin gto feel small | 17:31 |
*** e0ne has joined #openstack-infra | 17:32 | |
*** dolphm_503 is now known as dolphm | 17:35 | |
*** e0ne has quit IRC | 17:37 | |
*** dhellmann_ is now known as dhellmann | 17:41 | |
dhellmann | SergeyLukjanov: just saying hi :-) | 17:42 |
SergeyLukjanov | :) | 17:42 |
*** gokrokve_ has quit IRC | 17:43 | |
*** gokrokve has joined #openstack-infra | 17:43 | |
*** ryanpetrello has joined #openstack-infra | 17:47 | |
*** thedodd has joined #openstack-infra | 17:49 | |
*** talluri has quit IRC | 17:51 | |
*** lnxnut has quit IRC | 17:52 | |
*** thuc_ has quit IRC | 17:52 | |
*** thuc has joined #openstack-infra | 17:53 | |
*** lnxnut has joined #openstack-infra | 17:53 | |
*** ociuhandu has quit IRC | 17:57 | |
*** thuc has quit IRC | 17:57 | |
*** andre__ has quit IRC | 17:59 | |
*** ryanpetrello has quit IRC | 18:00 | |
clarkb | fungi: the shard delete cron isn't working anymore. I think that is because cron isn't installed on the pvhvm images by default. I have deleted a couple shards by hand today to give us breathing room, but we shuold fix that monday | 18:00 |
*** CaptTofu has quit IRC | 18:06 | |
*** CaptTofu has joined #openstack-infra | 18:08 | |
SpamapS | so are people still unable to reach the tripleo ci cloud? | 18:11 |
SpamapS | I don't see nodepooled vms running so I assume yes | 18:11 |
*** VijayTripathi has joined #openstack-infra | 18:16 | |
*** ArxCruz has quit IRC | 18:19 | |
*** nicedice_ has joined #openstack-infra | 18:20 | |
*** lnxnut has quit IRC | 18:20 | |
*** lnxnut has joined #openstack-infra | 18:21 | |
*** lnxnut has quit IRC | 18:21 | |
*** lnxnut has joined #openstack-infra | 18:22 | |
*** nicedice has quit IRC | 18:22 | |
*** lnxnut has quit IRC | 18:26 | |
*** ryanpetrello has joined #openstack-infra | 18:33 | |
*** ryanpetrello has quit IRC | 18:35 | |
*** e0ne has joined #openstack-infra | 18:38 | |
*** wchrisj has joined #openstack-infra | 18:39 | |
*** lttrl has quit IRC | 18:42 | |
*** dcramer__ has joined #openstack-infra | 18:43 | |
*** wchrisj has quit IRC | 18:48 | |
*** andre__ has joined #openstack-infra | 18:50 | |
*** wchrisj has joined #openstack-infra | 18:56 | |
*** sarob_ has joined #openstack-infra | 18:57 | |
*** dolphm is now known as dolphm_503 | 19:00 | |
*** sarob_ has quit IRC | 19:01 | |
*** wchrisj has quit IRC | 19:06 | |
*** Ryan_Lane has joined #openstack-infra | 19:07 | |
*** Ryan_Lane has quit IRC | 19:14 | |
*** dolphm_503 is now known as dolphm | 19:20 | |
banix | sdague: Could you please revisit: https://review.openstack.org/#/c/71359/ You had given a -2 pending the merge of the plugin. the plugin Just got merged. The link to it is in the commit message. | 19:20 |
*** lnxnut has joined #openstack-infra | 19:23 | |
*** milki has quit IRC | 19:24 | |
*** Ryan_Lane has joined #openstack-infra | 19:27 | |
*** lnxnut has quit IRC | 19:27 | |
*** dolphm is now known as dolphm_503 | 19:29 | |
*** lnxnut has joined #openstack-infra | 19:35 | |
*** therve` is now known as therve | 19:37 | |
*** lnxnut has quit IRC | 19:40 | |
*** banix has quit IRC | 19:43 | |
*** banix has joined #openstack-infra | 19:58 | |
*** milki has joined #openstack-infra | 20:00 | |
*** marun has quit IRC | 20:03 | |
*** marun has joined #openstack-infra | 20:05 | |
*** banix has quit IRC | 20:14 | |
*** dolphm_503 is now known as dolphm | 20:20 | |
*** yolanda_ has joined #openstack-infra | 20:26 | |
*** marun has quit IRC | 20:34 | |
*** lnxnut has joined #openstack-infra | 20:35 | |
*** boris-42 has quit IRC | 20:39 | |
*** rwsu has quit IRC | 20:40 | |
*** lnxnut has quit IRC | 20:40 | |
*** torandu_ has quit IRC | 20:43 | |
*** yolanda_ has quit IRC | 20:48 | |
*** dolphm is now known as dolphm_503 | 20:58 | |
*** sarob_ has joined #openstack-infra | 20:58 | |
*** sarob_ has quit IRC | 21:03 | |
*** thedodd has quit IRC | 21:05 | |
*** boris-42 has joined #openstack-infra | 21:11 | |
*** ildikov_ has quit IRC | 21:18 | |
*** markmcclain has joined #openstack-infra | 21:20 | |
*** pmathews has joined #openstack-infra | 21:24 | |
*** wchrisj has joined #openstack-infra | 21:34 | |
*** lnxnut has joined #openstack-infra | 21:35 | |
*** lnxnut has quit IRC | 21:40 | |
*** pmathews has quit IRC | 21:40 | |
*** Sukhdev has joined #openstack-infra | 21:40 | |
*** cody-somerville has joined #openstack-infra | 21:48 | |
*** cody-somerville has quit IRC | 21:48 | |
*** cody-somerville has joined #openstack-infra | 21:48 | |
*** dolphm_503 is now known as dolphm | 21:49 | |
*** thuc has joined #openstack-infra | 21:53 | |
clarkb | I have now written and compiled C on an apple lisa running xenix. /weekend | 21:53 |
*** wchrisj has quit IRC | 21:54 | |
*** pmathews has joined #openstack-infra | 21:54 | |
openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: restore bug 1253482 - keystone ephemeral port conflict https://review.openstack.org/77367 | 21:55 |
*** CaptTofu has quit IRC | 21:56 | |
*** thuc has quit IRC | 21:57 | |
*** dolphm is now known as dolphm_503 | 21:58 | |
sdague | clarkb: nice | 22:00 |
openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add bug for bad tempest test with ec2 az https://review.openstack.org/77368 | 22:08 |
*** dolphm_503 is now known as dolphm | 22:14 | |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: restore bug 1253482 - keystone ephemeral port conflict https://review.openstack.org/77367 | 22:16 |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add bug for bad tempest test with ec2 az https://review.openstack.org/77368 | 22:16 |
*** dolphm is now known as dolphm_503 | 22:17 | |
mordred | clarkb: pvhvm images don't have cron by default? | 22:20 |
clarkb | curl | 22:21 |
clarkb | did I tyop? i meant curl | 22:22 |
*** e0ne has quit IRC | 22:24 | |
*** e0ne has joined #openstack-infra | 22:25 | |
*** dolphm_503 is now known as dolphm | 22:27 | |
sdague | clarkb: you seen this fail in the gate of late - http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiV0FSTklORzogVGhlIGZvbGxvd2luZyBwYWNrYWdlcyBjYW5ub3QgYmUgYXV0aGVudGljYXRlZFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkzNzExNjQxMzMzfQ== ? | 22:28 |
sdague | looks like tons of mirror issues actually | 22:29 |
sdague | there are lots of ways this seems to be failing, I wonder if the mirrors are updating oddly. I've found a couple other bad package issues in the unclassified pile | 22:29 |
sdague | tons might be over stating | 22:30 |
sdague | but more package fails than I expect | 22:30 |
fungi | cron isn't installed on pvhvm images by default? and i was surprised to discover curl and lvm2 missing | 22:33 |
clarkb | no it is | 22:34 |
clarkb | I meant to say curl | 22:34 |
fungi | clarkb: apple lisa? i take it you visited the museum | 22:35 |
clarkb | which is used by the cronjob | 22:35 |
clarkb | I did and am now getting ramen | 22:35 |
fungi | and yeah, i already discovered curl missing, which is why the bootstrap scripts now try wget when curl fails | 22:35 |
*** lnxnut has joined #openstack-infra | 22:35 | |
clarkb | we should put curl in our base manifrst | 22:35 |
clarkb | and install it everywhere | 22:35 |
clarkb | figuring out how to change modes in vi on the lisa keyboard was hard | 22:36 |
openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add query for un-authenticated packages https://review.openstack.org/77371 | 22:36 |
fungi | agreed. in the bootstrap scripts i did a fallback because that's pre-puppet, but yes definitely | 22:37 |
sdague | clarkb: I excitedly see the logstash queue is 0. :) | 22:38 |
fungi | the logstash job queue graph looks... abrupt... though | 22:38 |
*** VijayTripathi has quit IRC | 22:39 | |
clarkb | sdague: ya and hopefully it stays down there | 22:39 |
clarkb | it is much happier with the watchdog working | 22:39 |
fungi | did it clear out that quickly for natural (no incoming volume) reasons, or did it get dumped? | 22:39 |
*** lnxnut has quit IRC | 22:40 | |
clarkb | natural | 22:41 |
anteaya | so yesterday at 13:25 utc neutron ryu was emailed that their system was voting -1 due to their ci failing: http://lists.openstack.org/pipermail/openstack-infra/2014-February/000925.html | 22:45 |
*** cody-somerville has quit IRC | 22:45 | |
*** ociuhandu has joined #openstack-infra | 22:45 | |
anteaya | they replied today at 10:40 utc to say they were working on it: http://lists.openstack.org/pipermail/openstack-infra/2014-March/000928.html | 22:46 |
anteaya | but they haven't said it is fixed and their system is still voting: http://180.37.183.32/jenkins/job/ryuplugin-tempest/9/console | 22:46 |
anteaya | https://review.openstack.org/#/dashboard/9681 | 22:46 |
anteaya | can I get some input? | 22:46 |
anteaya | seeing devstack failed. in their logs indicates their is still a problem on their end | 22:47 |
anteaya | to my eyes | 22:47 |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add query for un-authenticated packages https://review.openstack.org/77371 | 22:53 |
*** alexpilotti has joined #openstack-infra | 22:55 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/meetbot: Reply to lines with urls with shortened urls https://review.openstack.org/77372 | 22:55 |
mordred | clarkb: ^^ that's inspired by a feature in the bot in the solum channel | 22:56 |
*** sarob_ has joined #openstack-infra | 22:59 | |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/meetbot: Reply to lines with urls with shortened urls https://review.openstack.org/77372 | 23:01 |
anteaya | mordred: is that just for meetbot? | 23:04 |
*** sarob_ has quit IRC | 23:04 | |
mordred | anteaya: yeah. although since meetbot watches all of the channels it's in, it should work for any channel meetbot inhabits | 23:07 |
*** vkozhukalov has quit IRC | 23:08 | |
openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: remove the stevedore query, it's almost all success https://review.openstack.org/77373 | 23:08 |
anteaya | really? is meetbot in here? | 23:08 |
mordred | openstack: help | 23:09 |
sdague | mordred: any concerns with the external service ? | 23:09 |
mordred | sdague: I went and read their about page | 23:09 |
mordred | sdague: they seem somewhat sane | 23:09 |
mordred | sdague: we coudl also run our own url shortener I suppose | 23:09 |
sdague | yeh, more concerned about uptime | 23:09 |
sdague | and what happens if it is down | 23:09 |
sdague | and explodes | 23:09 |
mordred | sdague: I believe we just won't get shortened urls | 23:10 |
sdague | how long does requests go before it times out? | 23:11 |
mordred | good question ... | 23:11 |
* mordred looks | 23:11 | |
sdague | I'm not actually in the "we always need to run our own camp", just want to make sure there is graceful handling of network outages | 23:12 |
sdague | because... there are always network outages :) | 23:12 |
mordred | sdague: configurable | 23:12 |
mordred | sdague: what do you think is reasonable? 2 seconds? | 23:12 |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: remove the stevedore query, it's almost all success https://review.openstack.org/77373 | 23:12 |
sdague | 2 seems small. 10 is fine | 23:12 |
sdague | just making sure it wasn't 120 or something | 23:13 |
sdague | and not responding until that completed | 23:13 |
sdague | or timed out | 23:13 |
anteaya | do we want a message posted that the url couldn't be shortened? | 23:13 |
anteaya | or silent failing? | 23:13 |
StevenK | Personally, silent failing | 23:13 |
sdague | yeh, silent failing is fine | 23:14 |
sdague | anyway, time to run off for a bit. Enjoy folks. | 23:14 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/meetbot: Reply to lines with urls with shortened urls https://review.openstack.org/77372 | 23:15 |
mordred | oh - well, I just added an error message... | 23:15 |
mordred | StevenK: you think we should silent fail? | 23:16 |
openstackgerrit | Monty Taylor proposed a change to openstack-infra/meetbot: Reply to lines with urls with shortened urls https://review.openstack.org/77372 | 23:16 |
StevenK | mordred: Yes | 23:16 |
mordred | k. done | 23:17 |
anteaya | neutron ryu ml update: http://lists.openstack.org/pipermail/openstack-infra/2014-March/000931.html | 23:20 |
*** pmathews has quit IRC | 23:26 | |
*** dolphm is now known as dolphm_503 | 23:29 | |
*** dolphm_503 is now known as dolphm | 23:30 | |
mikal | anteaya: would something like http://www.rcbops.com/gerrit/reports/nova-cireport.html for neutron CI systems help you? | 23:31 |
anteaya | mikal: hi | 23:33 |
anteaya | I'm looking at it | 23:34 |
anteaya | lots of good information there | 23:34 |
anteaya | the bit I would like to track is how many failures are reported because the system couldn't stand up a devstack? | 23:34 |
mikal | Hmmmm, I'd have to parse logs to get that though right? | 23:34 |
mikal | That would appear in gerrit comments | 23:35 |
mikal | s/would/wont/ | 23:35 |
mikal | More coffee needed | 23:35 |
anteaya | we are probably looking at a logstash scenario | 23:35 |
anteaya | coffee up | 23:35 |
anteaya | toss in some kermit arms | 23:35 |
*** lnxnut has joined #openstack-infra | 23:35 | |
mikal | Although, you'd see a secondary metric of many many -1 votes in that report | 23:35 |
* anteaya looks again | 23:35 | |
anteaya | Valid patches in report period: 495 | 23:36 |
anteaya | what time span is the report period? | 23:36 |
mikal | Yep, and then each CI system gets a summary row | 23:36 |
mikal | 7 days | 23:36 |
mikal | This is only for nova | 23:36 |
anteaya | right | 23:36 |
*** CaptTofu has joined #openstack-infra | 23:36 | |
mikal | And "valid patches" is a bit complex | 23:36 |
anteaya | yeah | 23:36 |
mikal | Only master, only ones that lasted more than three hours before being superceeded, etc | 23:36 |
*** che-arne has quit IRC | 23:36 | |
anteaya | how is it being consumed in nova? | 23:37 |
mikal | This has only existed for a couple of days | 23:37 |
mikal | The plan is to use it to decide who has met the "Group B" requirements for hypervisors | 23:37 |
mikal | i.e. its nice if you're voting, but if you're always voting +1 that doesn't count | 23:37 |
*** masayukig has joined #openstack-infra | 23:37 | |
anteaya | why is hyperv ci and turbo hipster in black ink | 23:38 |
anteaya | and the others in red ink/print/pixels | 23:38 |
mikal | You get red if you have missed too many votes | 23:38 |
mikal | 95% IIRC | 23:38 |
anteaya | hmmmm | 23:39 |
anteaya | what would cause them to miss votes? | 23:39 |
mikal | My current theory is that the CI systems aren't keeping up with the workload | 23:39 |
mikal | So they get backlogged and then flushed | 23:39 |
*** dolphm is now known as dolphm_503 | 23:39 | |
anteaya | the definition of workload here is difuse | 23:39 |
anteaya | which makes it hard to hit the mark | 23:39 |
mikal | I'd like to add a metric about review latency, but I can't without a gerrit upgrade | 23:40 |
anteaya | the ones that are keeping up are doing so in an outstanding fashion | 23:40 |
*** lnxnut has quit IRC | 23:40 | |
*** Sukhdev has quit IRC | 23:40 | |
anteaya | ah sorry, you were replying to my question 'what would cause them to miss votes' answer being processing the backlog | 23:41 |
mikal | Yeah | 23:41 |
*** CaptTofu has quit IRC | 23:41 | |
anteaya | now I am with you | 23:41 |
anteaya | well certainly having some sort of page to point to would be helpful | 23:42 |
mikal | Ok | 23:42 |
mikal | I shall have a quick hack tomorrow and see what I can do | 23:42 |
anteaya | even if it just manually updated how many devs had come to me to complain about the ci | 23:42 |
*** thuc has joined #openstack-infra | 23:42 | |
mikal | Where can I find a list of the usernames doing neutron CI? | 23:43 |
anteaya | then I could teach them all to look at it | 23:43 |
anteaya | mikal: here is all the gerrit 3rd party accounts: https://review.openstack.org/#/admin/groups/270,members | 23:43 |
anteaya | you will see it contains a sublist | 23:43 |
anteaya | this list is the non-voting accounts and also the sub-list of voting accounts | 23:44 |
anteaya | https://review.openstack.org/#/admin/groups/91,members | 23:44 |
mikal | Cool | 23:44 |
anteaya | but this is everybody, not just neutron | 23:44 |
mikal | Is there a simple way to tell which ones are neutron? | 23:44 |
anteaya | not that I know of | 23:44 |
mikal | I cna just guess worst case | 23:44 |
anteaya | vmware minesweeper is nova and neutron | 23:44 |
anteaya | they don't have to tell us their testing intentions when they register the account | 23:45 |
anteaya | just name, ssh key and hopefully contact info | 23:45 |
anteaya | and thank you | 23:45 |
mikal | Ok, I shall guess | 23:46 |
mikal | And if I miss any we can add them later | 23:46 |
anteaya | sounds great to me | 23:46 |
anteaya | you have my thanks kind sir | 23:46 |
anteaya | mikal: can you look at this log: http://180.37.183.32/jenkins/job/ryuplugin-tempest/9/console | 23:47 |
anteaya | does it appear to you that the ci is still failing to stand up a devstack? | 23:47 |
mikal | Yes, "devstack failed" | 23:48 |
anteaya | looking again I am in correct | 23:48 |
*** che-arne has joined #openstack-infra | 23:49 | |
*** dolphm_503 is now known as dolphm | 23:49 | |
anteaya | mikal: this is the patch: https://review.openstack.org/#/c/74490/ | 23:49 |
mikal | That's a lot of CI checks | 23:50 |
anteaya | and the system says it voted on it at 4:36pm https://review.openstack.org/#/dashboard/9681 (which I think is 4:36pm my time) which would be 21:36 utc | 23:50 |
anteaya | the ci crowd is vast | 23:51 |
anteaya | stack.sh log /var/lib/jenkins/jobs/ryuplugin-tempest/workspace/logs/devstack.2014-02-28-003806 | 23:51 |
anteaya | so what timestamp is that? | 23:52 |
anteaya | February 28 2014, ? | 23:52 |
anteaya | 00:38 timezone of server? | 23:52 |
mikal | I'd say so, yes | 23:52 |
mikal | Yep | 23:52 |
mikal | That's how I'd read it at least | 23:52 |
anteaya | but we don't know what time zone this server is using | 23:52 |
anteaya | we can't assume utc | 23:53 |
anteaya | fungi: are you willing to weigh in? | 23:53 |
mikal | If you go to the front page, it says "Page generated: Mar 2, 2014 8:53:19 AM" | 23:53 |
mikal | Which means two hours behind me I guess | 23:53 |
mikal | Which is approximately the timezone of Hong Kong | 23:53 |
anteaya | Page generated: Mar 2, 2014 7:43:18 AM | 23:54 |
anteaya | is what i have | 23:54 |
mikal | http://180.37.183.32/jenkins/ is where I got that timestamp from | 23:54 |
anteaya | http://180.37.183.32/jenkins/job/ryuplugin-tempest/9/artifact/logs/ | 23:54 |
anteaya | I refreshed the page: Page generated: Mar 2, 2014 8:54:51 AM | 23:55 |
anteaya | so that is just the timestamp for when your browser last rendered the page, it looks like | 23:55 |
mikal | Ryu is based in Japan? | 23:55 |
anteaya | http://lists.openstack.org/pipermail/openstack-infra/2014-March/000928.html | 23:55 |
anteaya | Motoki-san | 23:55 |
anteaya | I'd go with Japan | 23:56 |
mikal | My point is, its a timestamp of the server time when the page was generated | 23:56 |
anteaya | ah | 23:56 |
mikal | And it matches the current time in Tokyo (8:55am) | 23:56 |
anteaya | now I am with you | 23:56 |
mikal | i.e. it helps you work out what timezone the log is in | 23:56 |
anteaya | Tokyo time | 23:56 |
anteaya | about 33 hours ago | 23:57 |
anteaya | do you calcuate the same? | 23:58 |
*** CaptTofu has joined #openstack-infra | 23:58 | |
*** markmcclain has quit IRC | 23:58 | |
*** dolphm is now known as dolphm_503 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!