| *** dcramer_ has quit IRC | 00:00 | |
| *** SumitNaiksatam has joined #openstack-infra | 00:13 | |
| *** sarob has quit IRC | 00:19 | |
| *** matsuhashi has joined #openstack-infra | 00:19 | |
| *** sarob has joined #openstack-infra | 00:20 | |
| *** MarkAtwood has quit IRC | 00:26 | |
| *** pcrews has joined #openstack-infra | 00:33 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Zuul layout: start using templates https://review.openstack.org/63854 | 00:36 |
|---|---|---|
| *** pcrews has quit IRC | 00:38 | |
| cody-somerville | Are projects not allowed to have a stricter requirements file than openstack-infra/requirements? | 00:44 |
| cody-somerville | ex. require a higher version of a python client library than global-requirements | 00:45 |
| *** cyeoh has joined #openstack-infra | 00:47 | |
| *** nosnos has joined #openstack-infra | 00:51 | |
| *** hogepodge has quit IRC | 00:56 | |
| *** praneshp has joined #openstack-infra | 01:00 | |
| clarkb | no | 01:04 |
| clarkb | they must match | 01:04 |
| *** harlowja_ has quit IRC | 01:09 | |
| *** krotscheck has joined #openstack-infra | 01:10 | |
| *** ljjjustin has joined #openstack-infra | 01:16 | |
| *** dkliban has quit IRC | 01:21 | |
| *** yamahata has joined #openstack-infra | 01:24 | |
| *** sarob has quit IRC | 01:27 | |
| *** senk has joined #openstack-infra | 01:29 | |
| * StevenK peers at the error in http://logs.openstack.org/99/59699/5/check/gate-glance-pep8/5d6edaa/console.html | 01:31 | |
| StevenK | Why would it try and reach pypi? I thought we had a mirror. | 01:31 |
| cody-somerville | clarkb: So if a core project needs a bug fix in a client available in a newer version, they can't require it? | 01:33 |
| *** jhesketh has joined #openstack-infra | 01:38 | |
| clarkb | they can but must bump the requirement first | 01:38 |
| clarkb | StevenK: it shouldn't, according to the log it seems to clearly have updated the settings at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/select-mirror.sh#n36 | 01:41 |
| clarkb | StevenK: pretty sure that is mysql-python being "smart" | 01:42 |
| *** ljjjustin has quit IRC | 01:42 | |
| cody-somerville | clarkb: won't that bump it for everyone else? | 01:43 |
| clarkb | cody-somerville: yes | 01:43 |
| clarkb | cody-somerville: that is the idea. we are keeping everyone in sync | 01:43 |
| cody-somerville | clarkb: so if I update requirements, won't that break everyone else until they update their projects to match? | 01:45 |
| clarkb | only if they need to propose a change | 01:45 |
| dstufft | This is mostly a problem because pip doesn't have a real version dependency solver | 01:46 |
| clarkb | new requirements are used in all integration testing | 01:46 |
| dstufft | so sorry about that | 01:46 |
| dstufft | :[ | 01:46 |
| clarkb | and we automatically propose changes to update individual repos | 01:46 |
| clarkb | so things don't break, they just get out of sync until someone bothers to review those changes | 01:46 |
| *** UtahDave has quit IRC | 01:51 | |
| *** matsuhashi has quit IRC | 01:57 | |
| cody-somerville | Ok. Thanks for the info. :) | 01:59 |
| *** matsuhashi has joined #openstack-infra | 02:00 | |
| *** yaguang has joined #openstack-infra | 02:01 | |
| *** guohliu has joined #openstack-infra | 02:05 | |
| *** ladquin has joined #openstack-infra | 02:13 | |
| *** syerrapragada1 has joined #openstack-infra | 02:13 | |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack-infra/devstack-gate: Copy libvirt log file after tempest run https://review.openstack.org/61892 | 02:15 |
| *** ljjjustin has joined #openstack-infra | 02:33 | |
| *** w__ is now known as olaph | 02:40 | |
| *** ljjjustin has quit IRC | 02:46 | |
| *** alexpilotti has quit IRC | 02:46 | |
| *** harlowja has joined #openstack-infra | 02:48 | |
| *** nati_ueno has joined #openstack-infra | 02:50 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Collect list of installed packages at end of run https://review.openstack.org/63551 | 02:54 |
| *** fifieldt has joined #openstack-infra | 02:56 | |
| *** nati_ueno has quit IRC | 02:56 | |
| *** dcramer_ has joined #openstack-infra | 02:59 | |
| *** harlowja has quit IRC | 03:03 | |
| *** harlowja has joined #openstack-infra | 03:04 | |
| *** nati_ueno has joined #openstack-infra | 03:06 | |
| *** rwsu has quit IRC | 03:06 | |
| *** vogxn has joined #openstack-infra | 03:07 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Explicitly document requirements for 3rd party testing https://review.openstack.org/63478 | 03:11 |
| *** nati_ueno has quit IRC | 03:11 | |
| *** krotscheck has quit IRC | 03:12 | |
| *** krotscheck has joined #openstack-infra | 03:14 | |
| *** senk has quit IRC | 03:15 | |
| *** dkliban has joined #openstack-infra | 03:18 | |
| *** vkozhukalov has joined #openstack-infra | 03:27 | |
| *** mgagne has joined #openstack-infra | 03:27 | |
| *** coolsvap has joined #openstack-infra | 03:33 | |
| *** dstanek has quit IRC | 03:40 | |
| *** mgagne has quit IRC | 03:46 | |
| *** vogxn has quit IRC | 03:46 | |
| jog0 | so something is wrong with grenade | 03:48 |
| jog0 | http://status.openstack.org/elastic-recheck/ | 03:48 |
| jog0 | sdague clarkb: assume your both away until new years but ... ^ | 03:48 |
| *** mgagne has joined #openstack-infra | 03:49 | |
| fungi | i was about to say the same--i see it impacting multiple changes | 03:51 |
| fungi | seems to be impacting devstack exercises prior to upgrade | 03:52 |
| jog0 | fungi: yup | 03:53 |
| jog0 | just filed a bug https://bugs.launchpad.net/nova/+bug/1263824 | 03:53 |
| uvirtbot | Launchpad bug 1263824 in nova "grenade is failing on euca and bundle tests" [Undecided,New] | 03:53 |
| jog0 | my guess is either we have an asymetric hole in the gate or this is a dependency issue | 03:54 |
| fungi | well, we have a huge asymmetric hole in grenade until we go back to gating on exercises or until forward grenade is voting | 03:56 |
| jog0 | hmm boto 2.18.0 vs boto 2.21.1 | 03:56 |
| jog0 | http://pypi.openstack.org/openstack/boto/?C=M;O=D | 03:57 |
| jog0 | boto 2.2.1 is brand spaking new | 03:57 |
| jog0 | boto 2.21.1 is brand spaking new | 03:57 |
| jog0 | and euca2ools use boto if I remember coorectly | 03:58 |
| jog0 | yup they do | 03:58 |
| fungi | well, requirements gates on grenade right? i wonder whether we could rule this out by proposing a wip change to cap boto and watching check results | 03:59 |
| jog0 | lets do it | 04:00 |
| jog0 | it does | 04:00 |
| jog0 | I'll push a patch | 04:00 |
| jog0 | so we have boto 2.21.0 but there is a 2.21.1 out | 04:02 |
| *** jerryz has quit IRC | 04:02 | |
| *** dstanek has joined #openstack-infra | 04:02 | |
| fungi | oh, i wonder if .0 was flat broken | 04:03 |
| *** krotscheck has quit IRC | 04:03 | |
| jog0 | http://docs.pythonboto.org/en/latest/releasenotes/v2.21.1.html | 04:03 |
| clarkb | we just changed project selection for grenade | 04:03 |
| jog0 | clarkb: ? | 04:03 |
| *** dkliban has quit IRC | 04:03 | |
| clarkb | but I don't expect that to cause trouble (it removes stuff doesn't add it) | 04:03 |
| fungi | that crossed my mind | 04:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Blacklist boto 2.21.0 https://review.openstack.org/63863 | 04:04 |
| *** krotscheck has joined #openstack-infra | 04:04 | |
| fungi | http://docs.pythonboto.org/en/latest/releasenotes/v2.21.1.html | 04:05 |
| fungi | .0 is 4 days old | 04:05 |
| fungi | oh, jog0 beat me to the release notes link | 04:06 |
| jog0 | checking logstash to see when the issues started | 04:07 |
| jog0 | started at midnight | 04:07 |
| jog0 | message:"Failure bundling image bundle.img" | 04:07 |
| jog0 | brb relo | 04:07 |
| jog0 | hmm the times don't match up .. this may be a red herring | 04:08 |
| fungi | yeah, it doesn't look like 2.21.0 would be at fault... it's been in our mirror since the 20th | 04:09 |
| fungi | most recent success i see was at 2300 utc | 04:10 |
| StevenK | Can someone give me a hint about the failure in http://logs.openstack.org/99/59699/5/check/gate-glance-pep8/5d6edaa/console.html ? | 04:11 |
| fungi | earliest failure i see exhibiting the grenade havana devstack exercises bundle/euca problems started just prior to 2330 utc | 04:12 |
| fungi | https://jenkins01.openstack.org/job/check-grenade-dsvm/3353/consoleFull | 04:13 |
| fungi | StevenK: looks like mysql-python's setup.py for some hairbrained reason wanted to retrieve a package of distribute from pypi.python.org instead of our pypi.openstack.org mirror | 04:16 |
| fungi | and then encountered some connection issue to it | 04:16 |
| fungi | StevenK: looks like it calls use_setuptools() which wants to run some sort of distribute downloader, which may ignore the easy_install/pip override configs | 04:18 |
| StevenK | fungi: Ah, so they are able to talk to pypi, rather than being firewalled off? | 04:21 |
| *** mgagne has quit IRC | 04:22 | |
| fungi | StevenK: we don't prevent slaves from trying to reach the internet, but even if we did that wouldn't have prevented it from trying and failing to reach upstream pypi there | 04:22 |
| fungi | jog0: clarkb: i think we can rule out a dependency change since the window for when it started was 4.5-5 hours ago and the most recent mirror updates were 17 hours ago | 04:23 |
| * fungi checks devstack image updates | 04:24 | |
| StevenK | fungi: Is there a bug for this behavior, or shall I be naughty and recheck no bug? | 04:24 |
| fungi | StevenK: a bug about this would be nice--just a copy of that traceback and a link to your log fileshould be enough to go on, i think | 04:25 |
| jog0 | fungi: this didn't start 5 hours ago | 04:26 |
| jog0 | err yes it did | 04:26 |
| jog0 | UTC math | 04:26 |
| fungi | clarkb: jog0: i don't think nodepool image updates are at fault for the grenade thing either... images were rebuilt a couple hours after it started | 04:27 |
| StevenK | fungi: Under https://launchpad.net/openstack-ci ? | 04:27 |
| fungi | StevenK: yep | 04:27 |
| * fungi starts looking at timeframes from suspect things merging | 04:27 | |
| fungi | would have been something landing between 2300-2330 utc | 04:28 |
| StevenK | fungi: Bug reported, thanks. I'll comment on the review to recheck against it | 04:29 |
| fungi | thanks StevenK! | 04:31 |
| fungi | clarkb: jog0: most recent changes to devstack landed a couple days ago, and devstack-gate last merged a change 2+ hours prior | 04:32 |
| jog0 | https://review.openstack.org/#/q/status:merged+branch:stable/havana,n,z | 04:32 |
| jog0 | all quite on stable merges | 04:32 |
| fungi | i take back what i said about devstack and d-g changes. git logs were showing me the original commit times, not when they merged | 04:35 |
| fungi | https://review.openstack.org/63641 | 04:35 |
| fungi | nevermind, that merged right after the danger zone | 04:35 |
| jog0 | and thats master | 04:35 |
| clarkb | what are the times we are talking about? | 04:35 |
| jog0 | clarkb: search logstash for "Failure bundling image bundle.img" | 04:36 |
| clarkb | I hvae heard 5 hours ago which is 3pm PST which is not midnight UTC | 04:36 |
| jog0 | clarkb: I see midnight lostash time | 04:36 |
| jog0 | logstash | 04:36 |
| clarkb | that is utc | 04:36 |
| fungi | clarkb: last success was 2300, first fail i find was 2329 (utc) | 04:37 |
| fungi | those being job *start* times according to jenkins01 and 02 | 04:37 |
| clarkb | so that was well after the d-g change merged | 04:37 |
| fungi | yeah | 04:37 |
| clarkb | and reasonably after the devstack change | 04:37 |
| fungi | the last d-g change was 21:08 | 04:38 |
| fungi | most recent mirror updates were a dozen hours before it started, ruling out new dependency stuff | 04:38 |
| fungi | nodepool image updates were well after it started, ruling that out | 04:39 |
| clarkb | could it be the d-g change? it isn't failing 100% of the time right? | 04:40 |
| fungi | 100% of the time, yes | 04:40 |
| clarkb | I am looking at openstack/openstack and nothing looks likely | 04:40 |
| fungi | there was one more d-g change to merge after the project selection one, but it didn't merge until shortly after this | 04:40 |
| *** dcramer_ has quit IRC | 04:41 | |
| clarkb | fungi: and it affected the bits that should avoid grenade | 04:42 |
| clarkb | it is in the if not grenade section | 04:42 |
| jog0 | it is boto 2.21 | 04:42 |
| *** krotscheck has quit IRC | 04:43 | |
| jog0 | on trunk devstack I can reproduce thei ssue | 04:43 |
| *** nosnos_ has joined #openstack-infra | 04:44 | |
| jog0 | clarkb fungi: ^ | 04:44 |
| clarkb | is this when we all take a shot? | 04:44 |
| jog0 | yes | 04:44 |
| *** krotscheck has joined #openstack-infra | 04:44 | |
| clarkb | jog0: I am happy to review a requirements change if you just want to pin it, but am also happy to review a fix to whatever it broke | 04:44 |
| *** dripton has joined #openstack-infra | 04:45 | |
| jog0 | clarkb: euca2ools borke | 04:45 |
| jog0 | broke | 04:45 |
| jog0 | I am not touching that | 04:45 |
| jog0 | IMHO we shouldn't gate on it | 04:45 |
| fungi | how is that happening now though? the mirror has had boto 2.21.0 since several days and doesn't have 2.21.1 yet | 04:47 |
| *** nosnos has quit IRC | 04:48 | |
| jog0 | fungi: weekend + ? | 04:48 |
| jog0 | oh and boto 2.21.x is broken I think | 04:48 |
| jog0 | not just 2.21.1 | 04:48 |
| fungi | so do <2.21 i guess | 04:48 |
| clarkb | fungi: it is getting it from pypi.python.org | 04:48 |
| fungi | gah | 04:49 |
| clarkb | fungi: Storing download in cache at /var/cache/pip/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fb%2Fboto%2Fboto-2.21.1.tar.gz | 04:49 |
| clarkb | I have no idea why | 04:49 |
| fungi | why are jobs skipping our mirror now? | 04:49 |
| clarkb | no idea (fwiw StevenK's job did not) | 04:49 |
| jog0 | https://github.com/eucalyptus/euca2ools hasn't been touched in 17 days | 04:49 |
| jog0 | sigh | 04:50 |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Blacklist boto 2.21.0 https://review.openstack.org/63863 | 04:50 |
| fungi | clarkb: well, StevenK's problem job showed mysql-python calling use_setuptools() in its setup.py which seemed to spawn some hardcoded distribute downloader | 04:50 |
| fungi | which very well may ignore pip/easy_install overrides | 04:51 |
| clarkb | ya | 04:51 |
| clarkb | for grenade | 04:51 |
| clarkb | http://logs.openstack.org/03/63003/1/gate/gate-grenade-dsvm/95f1aab/logs/devstack-gate-setup-host.txt.gz seems to do the correct thing | 04:51 |
| clarkb | wait | 04:52 |
| clarkb | sudo is derping | 04:52 |
| clarkb | what is going on there? | 04:52 |
| jog0 | merry Christmas everyone! | 04:52 |
| jog0 | https://review.openstack.org/63865 | 04:54 |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Blacklist boto 2.21.0 and 2.21.1 https://review.openstack.org/63863 | 04:54 |
| jog0 | now to file a bug with euca2ools hmm | 04:55 |
| *** krotscheck has quit IRC | 04:55 | |
| *** jhesketh has quit IRC | 04:55 | |
| *** elasticio has joined #openstack-infra | 04:57 | |
| fungi | so what i'm still wondering is what happened between 2300 and 2330 to suddenly start exposing this issue with euca2ools... ubuntu package update? | 04:57 |
| *** krotscheck has joined #openstack-infra | 04:57 | |
| clarkb | boto released today? | 04:58 |
| clarkb | 2.21.1 looks like ti is frim today | 04:58 |
| clarkb | for fixing the mirror stuff how do you edit /etc/hosts if sudo isn't working | 04:58 |
| jog0 | maybe another dep was updated today | 04:58 |
| jog0 | boto or euca | 04:58 |
| clarkb | oh wait, looking in logs it looks like sudo may work afterall? /me tries a thing to clean it up | 04:59 |
| fungi | jog0: but the mirrors updated 12 hours prior to this issue starting and hadn't updated since | 05:00 |
| jog0 | fungi: I have no idea | 05:00 |
| *** harlowja is now known as harlowja_away | 05:00 | |
| jog0 | well lets wwait the requisite hour to see if this actually fixes it | 05:00 |
| jog0 | and then scratch our heads | 05:01 |
| fungi | at least from what i could see (most recent posts were ~17 hours ago, periodics even longer, for all three interpreter versions) | 05:01 |
| fungi | but it's also entirely possible something was lying to me | 05:01 |
| *** matsuhashi has quit IRC | 05:02 | |
| *** matsuhashi has joined #openstack-infra | 05:03 | |
| jog0 | if I am right this will pass https://jenkins01.openstack.org/job/check-grenade-dsvm/3389/console | 05:04 |
| clarkb | does devstack create a stack group along with the user? | 05:05 |
| StevenK | Hm, why does that pull from github? | 05:05 |
| jog0 | clarkb: I think so yeah | 05:07 |
| *** elasticio has quit IRC | 05:08 | |
| fungi | jog0: https://jenkins02.openstack.org/job/check-grenade-dsvm/3535/console seems to not have worked | 05:08 |
| jog0 | fungi: that was a patch to trunk | 05:08 |
| jog0 | fungi: it shouldn't work | 05:08 |
| fungi | ahh, right. needs to be havana | 05:08 |
| jog0 | yeah I forgot that too | 05:09 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Fix copying of select-mirror outputs. https://review.openstack.org/63869 | 05:09 |
| clarkb | fungi: jog0 ^ that *should* use the mirror | 05:09 |
| clarkb | we can confirm when logs are copied off | 05:09 |
| fungi | oh, good catch! | 05:11 |
| jog0 | fungi: https://jenkins01.openstack.org/job/check-grenade-dsvm/3389/console worked woo | 05:11 |
| fungi | yep, so it did | 05:12 |
| jog0 | what launchpad project do I file this against? | 05:13 |
| jog0 | opesntack-ci? | 05:13 |
| fungi | clarkb: if we want to review/approve jog0's 63865 asap i'll promote it to the head of the gate | 05:13 |
| fungi | jog0: sure, i guess. we don't have one for requirements so wfm | 05:14 |
| clarkb | the requirements change? | 05:14 |
| jog0 | clarkb: yup | 05:14 |
| clarkb | jog0: does 2.21.0 not work? | 05:14 |
| clarkb | 2.21.1 is the new one | 05:14 |
| jog0 | clarkb: 2.2.1.1 doesn't work | 05:14 |
| jog0 | let me confirm 2.21.0 doesn't either | 05:14 |
| jog0 | oh wait that is what pip freeze on gate said we ran | 05:15 |
| fungi | ahh, right, the havana backport needs a stable branch manager to approve, which i am not | 05:15 |
| jog0 | 2.21.0 | 05:15 |
| * fungi could temporarily make himself a stable branch manager to cram this in, if needed | 05:15 | |
| jog0 | so 2.21.1 doesn't work and neither does 2.21.0 | 05:15 |
| clarkb | jog0: ? 2.21.0 has been around for days according to fungi | 05:16 |
| fungi | not something i normally feel good doing, but with this being an urgent fix | 05:16 |
| clarkb | 2.21.1 came out today | 05:16 |
| fungi | clarkb: the last modified time on .0 in the mirror was the 20th | 05:16 |
| fungi | http://pypi.openstack.org/openstack/boto/?C=M;O=A | 05:17 |
| jog0 | oh wait | 05:17 |
| jog0 | 2.21.0 does work | 05:17 |
| *** nicedice has quit IRC | 05:18 | |
| fungi | *could* be preserved from the download and delayed by days, but even then it would have been added to the mirror at a minimum 12 hours before this problem cropped up (most recent mirror updates) | 05:18 |
| *** __amotoki__ is now known as amotoki | 05:18 | |
| fungi | okay, so we just want to blacklist 2.21.1 then, and presume it was being retrieved from pypi because of what clarkb's https://review.openstack.org/63869 is fixing | 05:19 |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Blacklist boto 2.21.1 https://review.openstack.org/63863 | 05:19 |
| clarkb | fungi: yup | 05:20 |
| jog0 | fungi: ahh now it all makes sense | 05:20 |
| fungi | jog0: on havana? | 05:21 |
| jog0 | fungi: pushing ow | 05:21 |
| jog0 | now | 05:21 |
| jog0 | we should blacklist on trunk too I figure | 05:21 |
| fungi | agreed | 05:22 |
| jog0 | anyway both patches are up | 05:22 |
| jog0 | its time to take that shot and call it a day | 05:22 |
| fungi | jog0: awesome | 05:23 |
| jog0 | o/ | 05:23 |
| *** coolsvap has quit IRC | 05:28 | |
| fungi | #status alert gating is currently wedged by consistent grenade job failures--proposed fix is being confirmed now--eta 06:00 utc | 05:31 |
| openstackstatus | NOTICE: gating is currently wedged by consistent grenade job failures--proposed fix is being confirmed now--eta 06:00 utc | 05:31 |
| *** ChanServ changes topic to "gating is currently wedged by consistent grenade job failures--proposed fix is being confirmed now--eta 06:00 utc" | 05:31 | |
| *** yfried has quit IRC | 05:35 | |
| fungi | okay, so https://jenkins01.openstack.org/job/check-grenade-dsvm/3395/console passed euca exercises but failed bundle exercises? wtf? | 05:36 |
| clarkb | existing problem? | 05:36 |
| fungi | trying to figure that out now | 05:39 |
| fungi | [ERROR] /opt/stack/old/devstack/exercises/bundle.sh:61 Image ami-00000001 not available within 15 seconds | 05:39 |
| fungi | seems to maybe be bug 1177134 | 05:40 |
| uvirtbot | Launchpad bug 1177134 in devstack "[ERROR] /opt/stack/new/devstack/exercises/bundle.sh:61 Image ami-00000001 not available within 15 seconds" [Medium,Triaged] https://launchpad.net/bugs/1177134 | 05:40 |
| *** coolsvap has joined #openstack-infra | 05:41 | |
| fungi | seems to be known to e-r as of yesterday, but been around much longer | 05:41 |
| clarkb | fungi: http://logs.openstack.org/69/63869/1/check/check-tempest-dsvm-full/d122a6f/logs/devstack-gate-setup-host.txt.gz much cleaner | 05:53 |
| clarkb | and grepping for pypi in the console log shows it using our mirror | 05:53 |
| fungi | clarkb: agreed. also that permissions mistake seems to have existed since at least september, possibly longer | 05:55 |
| clarkb | ya | 05:55 |
| fungi | the reexec job fail is one known to e-r | 05:56 |
| fungi | and other d-g jobs passed, so approving | 05:56 |
| clarkb | thanks | 05:56 |
| clarkb | is the grenade fix in? | 05:56 |
| clarkb | I am assuming yes because it passed check tests | 05:57 |
| clarkb | or, does using the mirror fix the problem? | 05:57 |
| fungi | using the mirror fixes it for the moment | 05:57 |
| clarkb | gotcha | 05:57 |
| fungi | but the problem package is about to land on the mirror | 05:58 |
| fungi | i think the mirror update periodics start shortlu | 05:58 |
| StevenK | Everything that grenade well, exploded on will be rechecked? | 05:59 |
| fungi | StevenK: not automatically, no. it might get manually rechecked by someone who wants to comb through gerrit results, but that probably won't be me since it's already 2am here | 06:00 |
| fungi | i will mention the bug number in the status update here shortly | 06:01 |
| *** ljjjustin has joined #openstack-infra | 06:01 | |
| clarkb | fungi: neither requirements change seems to be in the gate | 06:02 |
| fungi | clarkb: i hadn't approved them | 06:02 |
| clarkb | oh I see we are waiting to see if it fixes? wfm | 06:02 |
| clarkb | seems like it failed bundle and euca in grenade testing | 06:03 |
| fungi | well the havana one seems to not break euca. failed on bundle for what seems to be a separate known bug | 06:03 |
| clarkb | is this a case where we have to fix havana first? | 06:03 |
| fungi | however, the havana requirements change is testing grizzly->havana and we want to have it fix havana requirements to fix havana->master which isn't tested | 06:04 |
| fungi | so i think it just needs to be approved regardless | 06:05 |
| *** SumitNaiksatam has quit IRC | 06:05 | |
| fungi | it's got probably another half hour on the requirements integration job in check though, so i'm toying with approving without seeing that succeed | 06:06 |
| fungi | opinion? | 06:06 |
| clarkb | fungi: at this point it can't really hurt | 06:07 |
| clarkb | everything is dying in the gate | 06:07 |
| fungi | k | 06:07 |
| zaro | clarkb: hey finally got it up. https://github.com/jenkinsci/scp-plugin/pulls | 06:08 |
| fungi | clarkb: the havana requirements change is in the gate behind the d-g change | 06:09 |
| openstackgerrit | Renat Akhmerov proposed a change to openstack-infra/config: Adding configuration for python-mistralclient project https://review.openstack.org/63876 | 06:11 |
| fungi | #status ok fix for grenade euca/bundle failures is in the gate. changes failing on those issues in the past 7 hours should be rechecked or reverified against bug 1263824 | 06:11 |
| openstackstatus | NOTICE: fix for grenade euca/bundle failures is in the gate. changes failing on those issues in the past 7 hours should be rechecked or reverified against bug 1263824 | 06:11 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 06:11 | |
| uvirtbot | Launchpad bug 1263824 in openstack-ci "grenade is failing on euca and bundle tests" [Undecided,New] https://launchpad.net/bugs/1263824 | 06:11 |
| *** ladquin has quit IRC | 06:13 | |
| fungi | i'll try to keep an eye on the two fixes in the gate until they merge, but may not keep my eyes open that long | 06:14 |
| clarkb | zaro: I have left comments | 06:15 |
| zaro | clarkb: looking.. | 06:15 |
| fungi | btw, we have http://pypi.openstack.org/openstack/boto/boto-2.21.1.tar.gz as of a few minutes ago (first of the mirror periodic updates just finished) | 06:17 |
| *** krotscheck has quit IRC | 06:18 | |
| *** yfried has joined #openstack-infra | 06:21 | |
| *** matsuhashi has quit IRC | 06:35 | |
| *** matsuhashi has joined #openstack-infra | 06:36 | |
| fungi | the neutronclient grenade failure in the gate right now suggests we may need requirements for havana updated for all projects individually? | 06:37 |
| *** matsuhashi has quit IRC | 06:41 | |
| zaro | clarkb: hey, trying to understand your comment about if else entries | 06:41 |
| *** dstanek has quit IRC | 06:41 | |
| zaro | clarkb: i thought the problem with the copy was due to using the same session. | 06:42 |
| *** reed has quit IRC | 06:45 | |
| *** matsuhashi has joined #openstack-infra | 06:45 | |
| clarkb | zaro: it is | 06:47 |
| clarkb | zaro: the other comment is just pointing out the change is inefficient as is | 06:47 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Fix copying of select-mirror outputs. https://review.openstack.org/63869 | 06:48 |
| clarkb | fungi: shouldn't we use the global reqs in devstack, hmm but grenade isn't devstack so maybe? | 06:48 |
| clarkb | fungi: oh I bet those are master changes I bet we now need to update master | 06:52 |
| zaro | clarkb: sorry, your comment is confusing. so are you just pointing out something or suggesting that i should update the patch? | 06:53 |
| zaro | clarkb: it sounds like the latter. | 06:53 |
| *** vkozhukalov has quit IRC | 06:53 | |
| clarkb | yes you need to update the patch | 06:53 |
| fungi | i think maybe the zuul ref for havana reqs isn't being used setting up master grenade jobs | 06:53 |
| clarkb | zaro: if the only file being copied is the console log then two sessions are started | 06:53 |
| clarkb | zaro: it should only start one in that case | 06:53 |
| * StevenK hands fungi some matchsticks | 06:53 | |
| StevenK | Bah, I had this scrollbacked | 06:54 |
| *** simonmcc has quit IRC | 06:54 | |
| zaro | clarkb: ohh yeah. i see. You mean the session in the parent thread is _not_ necessary if only console log is copied? | 06:55 |
| clarkb | yes | 06:56 |
| zaro | gotcha. thanks. | 06:56 |
| *** bnemec has quit IRC | 07:02 | |
| *** bnemec has joined #openstack-infra | 07:02 | |
| *** simonmcc_ has joined #openstack-infra | 07:04 | |
| *** shardy_afk is now known as shardy | 07:09 | |
| *** cody-somerville has quit IRC | 07:19 | |
| *** sarob has joined #openstack-infra | 07:28 | |
| *** boris-42 has quit IRC | 07:37 | |
| *** coolsvap has quit IRC | 07:37 | |
| openstackgerrit | Renat Akhmerov proposed a change to openstack-infra/config: Adding configuration for python-mistralclient project https://review.openstack.org/63876 | 07:38 |
| *** cody-somerville has joined #openstack-infra | 07:39 | |
| *** matsuhashi has quit IRC | 07:40 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:41 | |
| *** matsuhashi has joined #openstack-infra | 07:41 | |
| *** matsuhashi has quit IRC | 07:47 | |
| *** akscram1 is now known as akscram | 07:48 | |
| *** matsuhashi has joined #openstack-infra | 07:49 | |
| enikanorov__ | dims: hi. apparently lazy umount didn't solve the problem | 07:50 |
| *** coolsvap has joined #openstack-infra | 07:54 | |
| *** sarob has quit IRC | 08:12 | |
| *** matsuhashi has quit IRC | 08:16 | |
| *** matsuhashi has joined #openstack-infra | 08:17 | |
| *** bogdando has joined #openstack-infra | 08:20 | |
| *** coolsvap has quit IRC | 08:22 | |
| *** vkozhukalov has joined #openstack-infra | 08:30 | |
| *** nati_ueno has joined #openstack-infra | 08:38 | |
| *** sarob has joined #openstack-infra | 08:43 | |
| openstackgerrit | Yair Fried proposed a change to openstack-infra/config: Adds neutron-isolated to Tempest gate as experimental https://review.openstack.org/63889 | 08:46 |
| *** yassine has joined #openstack-infra | 08:47 | |
| *** dizquierdo has joined #openstack-infra | 08:49 | |
| *** nati_ueno has quit IRC | 08:50 | |
| *** nati_ueno has joined #openstack-infra | 08:50 | |
| *** guohliu has quit IRC | 09:00 | |
| *** yongli is now known as yongli_away | 09:06 | |
| *** nosnos has joined #openstack-infra | 09:08 | |
| *** nosnos_ has quit IRC | 09:09 | |
| *** nati_ueno has quit IRC | 09:10 | |
| openstackgerrit | Renat Akhmerov proposed a change to openstack-infra/config: Adding configuration for python-mistralclient project https://review.openstack.org/63876 | 09:11 |
| *** sarob has quit IRC | 09:15 | |
| *** avishayb has joined #openstack-infra | 09:21 | |
| *** yaguang has quit IRC | 09:29 | |
| *** yamahata has quit IRC | 09:30 | |
| *** simonmcc_ has quit IRC | 09:30 | |
| *** simonmcc_ has joined #openstack-infra | 09:31 | |
| *** matsuhashi has quit IRC | 09:40 | |
| *** ruhe has joined #openstack-infra | 09:52 | |
| *** matsuhashi has joined #openstack-infra | 09:58 | |
| *** ljjjustin has quit IRC | 10:05 | |
| *** sarob has joined #openstack-infra | 10:13 | |
| *** avishayb has quit IRC | 10:16 | |
| *** nosnos has quit IRC | 10:31 | |
| *** gilliard has joined #openstack-infra | 10:39 | |
| *** ruhe has quit IRC | 10:39 | |
| openstackgerrit | Vladimir Kozhukalov proposed a change to openstack-infra/devstack-gate: Tiny improvement in paths https://review.openstack.org/63902 | 10:42 |
| *** sarob_ has joined #openstack-infra | 10:44 | |
| *** sarob has quit IRC | 10:46 | |
| *** yamahata has joined #openstack-infra | 10:47 | |
| *** dpyzhov has joined #openstack-infra | 10:59 | |
| dpyzhov | Hello. I'm trying to create stable branches in our stackforge/fuel-* repos. All repos are fine. Except fuel-web and fuel-ostf. For some reason I see "Application Error, Server Error, LOCK_FAILURE" message | 11:02 |
| *** ruhe has joined #openstack-infra | 11:09 | |
| *** boris-42 has joined #openstack-infra | 11:11 | |
| *** avishayb has joined #openstack-infra | 11:14 | |
| *** matsuhashi has quit IRC | 11:15 | |
| *** sarob_ has quit IRC | 11:16 | |
| dpyzhov | I've created a bug about our issue: https://bugs.launchpad.net/openstack-ci/+bug/1263899 | 11:18 |
| uvirtbot | Launchpad bug 1263899 in openstack-ci "Branch creation on review.openstack.org is broken for some repos" [Undecided,New] | 11:18 |
| *** michchap_ has quit IRC | 11:21 | |
| *** dpyzhov_ has joined #openstack-infra | 11:21 | |
| *** michchap has joined #openstack-infra | 11:21 | |
| *** dpyzhov has quit IRC | 11:22 | |
| *** dpyzhov_ is now known as dpyzhov | 11:22 | |
| *** heyongli has joined #openstack-infra | 11:22 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: don't use ec2 devstack exercises for validation https://review.openstack.org/63907 | 11:24 |
| *** ruhe has quit IRC | 11:34 | |
| *** avishayb has quit IRC | 11:34 | |
| *** fifieldt has quit IRC | 11:34 | |
| *** ruhe has joined #openstack-infra | 11:36 | |
| *** sarob has joined #openstack-infra | 11:43 | |
| *** sarob has quit IRC | 11:48 | |
| *** ruhe has quit IRC | 12:02 | |
| *** ruhe has joined #openstack-infra | 12:04 | |
| chmouel | I am getting some reccurent errors with the grenade test | 12:06 |
| chmouel | on different patches (for swift) | 12:06 |
| sdague | chmouel: a bad boto was released | 12:06 |
| sdague | https://review.openstack.org/63907 is the fix | 12:07 |
| chmouel | sdague: ah yeah that's what i was going to say it had to be with ec2 :) | 12:07 |
| sdague | however it's a d-g fix, and no +2ers on that are awake yet | 12:07 |
| sdague | the scrollback looks like they were working on it about 5hrs ago, so unclear when anyone will be awake to handle it | 12:07 |
| chmouel | humm shame, should we sent an email at least? | 12:08 |
| dpyzhov | Could anyone delete branches from our stackforge repos? | 12:08 |
| sdague | chmouel: if you like. I'm officially on vacation. :) | 12:08 |
| chmouel | sdague: hey, will do | 12:08 |
| sdague | I just figured I'd work up that fix | 12:08 |
| chmouel | sdague: thanks | 12:10 |
| sdague | man, freenode is no fun. I want utf8 nicks | 12:11 |
| chmouel | sdague: by the way you still have your old work email in that review https://review.openstack.org/#/c/63907/ | 12:16 |
| sdague | no, that's my new work email | 12:16 |
| chmouel | sdague: oh okay :) well congrats then | 12:17 |
| sdague | yeh, I left IBM for Samsung in Nov | 12:17 |
| sdague | the week after summit | 12:17 |
| chmouel | cool | 12:17 |
| *** ruhe has quit IRC | 12:19 | |
| *** ruhe has joined #openstack-infra | 12:22 | |
| *** avishayb has joined #openstack-infra | 12:27 | |
| *** ruhe has quit IRC | 12:28 | |
| *** ruhe has joined #openstack-infra | 12:30 | |
| *** dpyzhov has quit IRC | 12:31 | |
| *** dpyzhov has joined #openstack-infra | 12:32 | |
| *** ruhe has quit IRC | 12:33 | |
| *** dpyzhov has quit IRC | 12:36 | |
| *** ruhe has joined #openstack-infra | 12:42 | |
| *** sarob has joined #openstack-infra | 12:43 | |
| *** sarob has quit IRC | 12:48 | |
| *** dpyzhov has joined #openstack-infra | 12:53 | |
| *** flaper87|afk is now known as flaper87 | 12:55 | |
| *** vkozhukalov has quit IRC | 12:59 | |
| *** CaptTofu has joined #openstack-infra | 12:59 | |
| *** rahmu has left #openstack-infra | 13:00 | |
| *** rahmu has joined #openstack-infra | 13:02 | |
| *** dstanek has joined #openstack-infra | 13:03 | |
| yfried | dkranz: ping | 13:05 |
| *** dstanek has quit IRC | 13:08 | |
| *** tma996 has joined #openstack-infra | 13:13 | |
| openstackgerrit | Vladimir Kozhukalov proposed a change to openstack-infra/devstack-gate: Tiny improvement in paths https://review.openstack.org/63902 | 13:15 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Move tests to the zuul dir https://review.openstack.org/63920 | 13:15 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Enable hacking, ignore only failing checks https://review.openstack.org/63921 | 13:15 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Fix and enable H2XX (py3k compatibility issues) https://review.openstack.org/63922 | 13:16 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Fix and enable H4XX (docstrings) and H902 ('not in') https://review.openstack.org/63923 | 13:16 |
| *** jishaomei has joined #openstack-infra | 13:17 | |
| ArxCruz | fungi: jeblair hey, can you help me and jishaomei in one question regarding zuul yaml configuration? | 13:19 |
| ArxCruz | We would like to know how to configure zuul to not vote | 13:20 |
| ArxCruz | using this pipeline works? http://paste.openstack.org/show/56650/ | 13:20 |
| ruhe | ArxCruz: would this example help you? https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n124 | 13:27 |
| ArxCruz | ruhe: so, just add a voting: false in the job ? | 13:27 |
| SergeyLukjanov | ArxCruz, it looks correct | 13:27 |
| SergeyLukjanov | ArxCruz, which kind of non-voting do you want? | 13:28 |
| ArxCruz | SergeyLukjanov: we just don't want to report back voting +1 or -1 | 13:28 |
| SergeyLukjanov | ArxCruz, so, to post only comment | 13:29 |
| ArxCruz | SergeyLukjanov: here's my entire layout.yaml http://paste.openstack.org/show/56651/ | 13:29 |
| SergeyLukjanov | ArxCruz, experimental pipeline only posts comments - https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n83 | 13:29 |
| jishaomei | SergeyLukjanov, we want to configure non-voting pipeline to community and only just reporting message to community. | 13:30 |
| ArxCruz | SergeyLukjanov: but is that correct? Use the pipeline experimental ? | 13:30 |
| SergeyLukjanov | ArxCruz, I mean that you can configure pipeline in your zuul using experimental pipeline as example | 13:31 |
| ArxCruz | SergeyLukjanov: can you review the yaml i post? http://paste.openstack.org/show/56651/ | 13:33 |
| SergeyLukjanov | ArxCruz, sure | 13:33 |
| ArxCruz | thanks | 13:33 |
| ruhe | SergeyLukjanov, ArxCruz, you also can set voting=false for the job | 13:34 |
| ArxCruz | ruhe: got it | 13:34 |
| SergeyLukjanov | ruhe, nope, it'll make zuul always comment +1 | 13:34 |
| ruhe | SergeyLukjanov, hmm, i wouldn't expect that. but you know it better :) | 13:36 |
| *** dizquierdo has quit IRC | 13:37 | |
| ArxCruz | SergeyLukjanov: ruhe so, add voting = false make zuul comment +1 always ? | 13:38 |
| jishaomei | SergeyLukjanov, do we need to configure gerrit: verified: 0 in zull configure for non-voting pipeline? | 13:38 |
| SergeyLukjanov | ruhe, I'm not really sure now, but it seems like if all jobs are non-voting in pipeline than nothing can change mark | 13:40 |
| SergeyLukjanov | ruhe, probably it'll be 0 as the default value, but using vote: false doesn't seems good for me | 13:40 |
| SergeyLukjanov | ArxCruz, so, you'd like to run several jobs only for comment 'validate db2' for nova and migrate | 13:40 |
| SergeyLukjanov | ArxCruz, you have tons of unneeded lines that are sets some jobs to be non-voting | 13:40 |
| SergeyLukjanov | jishaomei, I think yes | 13:40 |
| ArxCruz | SergeyLukjanov: we want to run several jobs and report that in review.openstack.org without voting +1 or -1 just post the result | 13:41 |
| ArxCruz | btw would be nice have a yaml sample in documentation | 13:42 |
| gsamfira | hey guys. Just out of curiosity. If you remove the "success" and "failure" fields from the pipeline, would that not cause the pipeline to be non voting? (https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout-dev.yaml) | 13:42 |
| *** sarob has joined #openstack-infra | 13:43 | |
| SergeyLukjanov | gsamfira, I think that in this case nothing will be published to gerrit | 13:44 |
| gsamfira | got it. Thanks :) | 13:44 |
| SergeyLukjanov | btw I'd like to note that I'm not a zuul guru ;) | 13:45 |
| ArxCruz | SergeyLukjanov: you don't? :) | 13:45 |
| ArxCruz | sorry, you're not? :) | 13:46 |
| ArxCruz | bad english here :D | 13:46 |
| gsamfira | no worries. Neither are we :D. | 13:46 |
| *** sarob has quit IRC | 13:47 | |
| *** dpyzhov has quit IRC | 13:47 | |
| *** dstanek has joined #openstack-infra | 13:48 | |
| gsamfira | but your input was enough to put me on the right track. I don't want to generate any more bogus reports :|. So thanks loads :) | 13:48 |
| *** samuelbercovici has joined #openstack-infra | 13:48 | |
| SergeyLukjanov | :) | 13:48 |
| *** mancdaz has quit IRC | 13:49 | |
| samuelbercovici | maybe unrelated to this channel. who can assist with the login credentails for https://www.openstack.org/profile/? | 13:49 |
| ruhe | SergeyLukjanov, btw unit test for non-voting jobs has this comment "Test that non-voting jobs don't vote." | 13:49 |
| SergeyLukjanov | ruhe, it only checks that non-voting failed job doesn't breaks merge | 13:52 |
| ArxCruz | SergeyLukjanov: so, my yaml is wrong? can you point me where? | 13:52 |
| *** b3nt_pin has joined #openstack-infra | 13:54 | |
| *** salv-orlando_ has joined #openstack-infra | 13:54 | |
| *** simonmc| has joined #openstack-infra | 13:55 | |
| *** yfried has quit IRC | 13:56 | |
| *** mestery_ has joined #openstack-infra | 13:56 | |
| SergeyLukjanov | ArxCruz, I think it should looks like - http://paste.openstack.org/show/56659/ | 13:57 |
| *** simonmcc_ has quit IRC | 13:58 | |
| *** yhe_ has joined #openstack-infra | 13:58 | |
| *** jpeeler1 has joined #openstack-infra | 13:58 | |
| *** yhe_ is now known as yongli | 13:59 | |
| *** yassine has quit IRC | 13:59 | |
| *** bknudson has quit IRC | 13:59 | |
| *** jd__` has joined #openstack-infra | 13:59 | |
| SergeyLukjanov | ArxCruz, I've removed non needed lines and set success/failure conf like in the experimental pipeline | 14:00 |
| *** yassine has joined #openstack-infra | 14:01 | |
| ArxCruz | SergeyLukjanov: I'm your fan! Thanks a lot!!! | 14:01 |
| fungi | woke up to discover the boto requirements caps didn't merge, so have reinserted them in the gate. sdague: i see your d-g exercises patch now, which also lgtm | 14:01 |
| SergeyLukjanov | ArxCruz, np, I just hope that it'll work ok for you | 14:02 |
| sdague | fungi: honestly, the d-g change is probably the better one | 14:02 |
| sdague | because we really shouldn't be doing ec2 compat in the sanity check | 14:02 |
| ArxCruz | SergeyLukjanov: so, the trigger is only comment-added ? | 14:02 |
| fungi | sdague: okay, i'll kill the requirements changes and approve/promote the d-g ceanup | 14:02 |
| fungi | cleanup | 14:02 |
| *** bknudson has joined #openstack-infra | 14:02 | |
| sdague | cool | 14:03 |
| ArxCruz | SergeyLukjanov: I though it should have patchset-created too | 14:03 |
| SergeyLukjanov | fungi, morning! | 14:03 |
| SergeyLukjanov | ArxCruz, if you'd like to trigger it for each change | 14:03 |
| sdague | I'm puttering on my computer, so if something goes wrong, ping me. Though I'm mostly trying to ignore irc :) | 14:03 |
| ArxCruz | SergeyLukjanov: got it | 14:04 |
| ArxCruz | jishaomei: did you get that? | 14:04 |
| *** milki_ has joined #openstack-infra | 14:05 | |
| *** samuelbercovici has quit IRC | 14:06 | |
| *** heyongli has quit IRC | 14:06 | |
| *** mestery has quit IRC | 14:06 | |
| *** gsamfira has quit IRC | 14:06 | |
| *** jpeeler has quit IRC | 14:06 | |
| *** ilyashakhat_ has quit IRC | 14:06 | |
| *** beagles has quit IRC | 14:06 | |
| *** salv-orlando has quit IRC | 14:06 | |
| *** gema has quit IRC | 14:06 | |
| *** ekarlso has quit IRC | 14:06 | |
| *** StevenK has quit IRC | 14:06 | |
| *** pabelanger has quit IRC | 14:06 | |
| *** jd__ has quit IRC | 14:06 | |
| *** Grishkin has quit IRC | 14:06 | |
| *** milki has quit IRC | 14:06 | |
| jishaomei | ArxCruz,SergeyLukjanov: I got it. thanks for your help. | 14:06 |
| *** jd__` is now known as jd__ | 14:06 | |
| *** salv-orlando_ is now known as salv-orlando | 14:06 | |
| jishaomei | SergeyLukjanov. I am your fans too. | 14:06 |
| fungi | SergeyLukjanov: hi! | 14:10 |
| *** gema has joined #openstack-infra | 14:10 | |
| *** mancdaz has joined #openstack-infra | 14:10 | |
| *** gsamfira has joined #openstack-infra | 14:10 | |
| *** ilyashakhat_ has joined #openstack-infra | 14:10 | |
| *** ekarlso has joined #openstack-infra | 14:10 | |
| *** StevenK has joined #openstack-infra | 14:10 | |
| *** pabelanger has joined #openstack-infra | 14:10 | |
| *** Grishkin has joined #openstack-infra | 14:10 | |
| * fungi is still trying to wake up and caffeinate | 14:11 | |
| *** SergeyLukjanov has quit IRC | 14:11 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:14 | |
| *** dstanek has quit IRC | 14:14 | |
| SergeyLukjanov | fungi, o/ | 14:14 |
| SergeyLukjanov | fungi, fungi, first catch for check-projects-yaml-alphabetized :) https://review.openstack.org/#/c/63876/1 | 14:14 |
| * SergeyLukjanov can't understand why internet connection is too bad :( | 14:15 | |
| fungi | SergeyLukjanov: mine too. but it's because i'm visiting family in a very rural place with no cell phone coverage, extremely old copper lines and very limited options for internet service providers | 14:16 |
| SergeyLukjanov | fungi, :) | 14:17 |
| fungi | also, awesome alphabetization catch. seems to have properly identified the culprit entry | 14:18 |
| fungi | if samuelbercovici shows back up asking about foundation profile authentication problems again, the contact info for the site admins is https://wiki.openstack.org/wiki/Website | 14:20 |
| * SergeyLukjanov adds https://wiki.openstack.org/wiki/Website to notes | 14:21 | |
| *** rossella_s has joined #openstack-infra | 14:23 | |
| *** dstanek has joined #openstack-infra | 14:23 | |
| *** ruhe is now known as ruhe_ | 14:23 | |
| *** ruhe_ has quit IRC | 14:23 | |
| fungi | ArxCruz: jishaomei: did you still have zuul configuration questions, or are you up to speed now? | 14:23 |
| ArxCruz | fungi: thanks to SergeyLukjanov we think we have everything covered :) | 14:24 |
| fungi | great! | 14:24 |
| ArxCruz | fungi: is possible you restore jishaomei account so he can test? You can review his zuul yaml to ensure everything is correct | 14:24 |
| *** ruhe has joined #openstack-infra | 14:26 | |
| fungi | ArxCruz: jishaomei: of course. i've put that ibm db2 test account back in the external testing tools group so it should be able to add -1/+1 vrfy votes again | 14:26 |
| *** yamahata has quit IRC | 14:26 | |
| ArxCruz | jishaomei: please when you have you yaml done, send to fungi so he can review and restore your account :) | 14:27 |
| *** yamahata has joined #openstack-infra | 14:27 | |
| fungi | oh, i already did | 14:27 |
| *** annegentle_ has quit IRC | 14:27 | |
| fungi | (sorry, i probably wasn't clear enough when i said it the first time) | 14:28 |
| fungi | we didn't completely block that db2 account anyway, we had only removed it from the group which has verify voting privileges, but it's back in there now (as of a couple minutes ago) | 14:29 |
| *** rossella_s has quit IRC | 14:30 | |
| jishaomei | fungi, please set our account for non-voting privileges, I am not sure everything is right and I will check it more carefully. | 14:32 |
| *** russellb is now known as russellb_see_you | 14:33 | |
| *** russellb_see_you is now known as russellb_gonetil | 14:33 | |
| *** russellb_gonetil is now known as russellb_vacatio | 14:33 | |
| *** russellb_vacatio is now known as russellb_gone | 14:34 | |
| fungi | jishaomei: oh, okay. well it had the ability to leave comments as long as it didn't also try to leave -1/+1 vrfy votes. i'll put it back that way for now | 14:35 |
| *** ruhe is now known as ruhe_ | 14:35 | |
| *** ruhe_ has quit IRC | 14:36 | |
| fungi | ibm db2 test has been removed from the external testing tools group again, so it won't be able to -1/+1 vrfy until you're ready to try that feature | 14:36 |
| fungi | but it can still leave comments on changes as long as it does not try to vote | 14:36 |
| *** dstanek has quit IRC | 14:38 | |
| *** dims has quit IRC | 14:43 | |
| *** sarob has joined #openstack-infra | 14:43 | |
| *** ruhe has joined #openstack-infra | 14:45 | |
| *** dizquierdo has joined #openstack-infra | 14:45 | |
| jishaomei | fungi, could you help to review the zuul yaml configuration http://paste.openstack.org/show/56659/, if I configure our zuul like that, it would implement non-voting to community but just report message to commutity? | 14:46 |
| *** sarob has quit IRC | 14:48 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: don't use ec2 devstack exercises for validation https://review.openstack.org/63907 | 14:50 |
| fungi | if dpyzhov comes back, i've done the branch removals he requested in bug 1263899 | 14:50 |
| uvirtbot | Launchpad bug 1263899 in openstack-ci "Branch creation on review.openstack.org is broken for some repos" [Wishlist,Incomplete] https://launchpad.net/bugs/1263899 | 14:50 |
| fungi | jishaomei: i'm taking a look now | 14:51 |
| sdague | grenade should be back to normal now | 14:52 |
| *** SergeyLukjanov has quit IRC | 14:53 | |
| fungi | sdague: thanks! i'll keep an eye on the gate for a bit and mark that bug resolved if i don't see it continue to be broken | 14:54 |
| *** dims has joined #openstack-infra | 14:55 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:56 | |
| fungi | jishaomei: ArxCruz: you have a job name pattern of ^.*devstack-vm.*$ set to apply single-use node behavior, but you have no job names in your layout matching that pattern | 14:58 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: comparison to stable/grizzly is not numeric https://review.openstack.org/63934 | 14:59 |
| sdague | fungi: also, just noticed that while watching that log | 14:59 |
| fungi | sdague: yep, saw that last night and bookmarked that line in the script so i'd remember to fix it. thanks! | 15:04 |
| *** dstanek has joined #openstack-infra | 15:05 | |
| *** yfried has joined #openstack-infra | 15:06 | |
| *** yongli has quit IRC | 15:06 | |
| jishaomei | fungi, thx.I fix it in http://paste.openstack.org/show/56671/ | 15:06 |
| *** che-arne has quit IRC | 15:07 | |
| *** boris-42 has quit IRC | 15:10 | |
| *** senk has joined #openstack-infra | 15:12 | |
| jishaomei | @fungi,Could you please review the http://paste.openstack.org/show/56671/, Thanks so much | 15:15 |
| *** senk has quit IRC | 15:15 | |
| fungi | jishaomei: ArxCruz: the only other mistake i see there is the "^[\s\S]*validate db2[\s\S]*$" pattern, since "[\s\S]" (literally, any space or non-space character) is the same as "." | 15:16 |
| ArxCruz | fungi: and you compile the regex in your head... ;) | 15:17 |
| *** coolsvap has joined #openstack-infra | 15:20 | |
| fungi | jishaomei: ArxCruz: i'd recommend using "(?i)^\s*check db2\s*$" like we do with the experimental pipeline | 15:22 |
| fungi | http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n90 | 15:23 |
| *** yassine has quit IRC | 15:24 | |
| *** yassine has joined #openstack-infra | 15:24 | |
| jishaomei | @fungi, thanks for your review, I'll revise comment_filter as "(?i)^\s*check db2\s*$" | 15:26 |
| *** dstanek has quit IRC | 15:29 | |
| *** senk has joined #openstack-infra | 15:32 | |
| *** dstanek has joined #openstack-infra | 15:35 | |
| *** avishayb has quit IRC | 15:36 | |
| jeblair | fungi: agreed about the "check foo" syntax (if it's needed) | 15:37 |
| jeblair | jishaomei: also, note that will _only_ run when you leave a comment of "check db2". that's probably a good idea while you're testing; i assume when it's working you'll want to run it on all new patchsets. | 15:37 |
| *** harlowja_away is now known as harlowja | 15:38 | |
| *** samuelbercovici has joined #openstack-infra | 15:40 | |
| jeblair | fungi: gracefully restarting zuul (to upgrade to version with the latest template changes). eta 40 mins. | 15:41 |
| *** ruhe is now known as ruhe_ | 15:42 | |
| *** sarob has joined #openstack-infra | 15:43 | |
| *** samuelbercovici has quit IRC | 15:44 | |
| *** rahmu has quit IRC | 15:46 | |
| *** sarob has quit IRC | 15:48 | |
| *** dstanek has quit IRC | 15:48 | |
| *** rahmu has joined #openstack-infra | 15:49 | |
| *** flaper87 is now known as flaper87|afk | 15:50 | |
| fungi | jeblair: okay, great--thanks! | 15:58 |
| *** krotscheck has joined #openstack-infra | 16:04 | |
| *** ruhe has joined #openstack-infra | 16:07 | |
| krotscheck | So, most of the build boxes are running either centos or ubuntu raring, correct? | 16:07 |
| jeblair | krotscheck: centos (for python2.6 jobs) or precise | 16:07 |
| krotscheck | Precise. Check. | 16:07 |
| krotscheck | I'm trying to figure out a way to get the javascript toolchain onto a box in a way that doesn't make our eyes bleed. | 16:09 |
| krotscheck | Any suggestions? The big elephant in the room is the lack of offical recent nodejs package. | 16:09 |
| fungi | the primary reasons for centos and ubuntu lts is long-term security support. usually shiny new versions of things run counter to stable, carefully backported fixes | 16:10 |
| jeblair | krotscheck: i guess you're concerned about unit tests, right? not so much a production deployment? | 16:11 |
| krotscheck | jeblair: Correct. I don't need a server, I need the commandline script execution to build and test a binary. | 16:11 |
| krotscheck | s/binary/distribution/ | 16:11 |
| krotscheck | I have a reference project on github, am finishing the readme now. | 16:11 |
| jeblair | krotscheck: for the production deployment of etherpad, we install it from github; however i'm not sure anything being _tested_ targets a later version than what's in precise | 16:12 |
| jeblair | on purpose | 16:12 |
| *** jishaomei has quit IRC | 16:13 | |
| *** dstanek has joined #openstack-infra | 16:15 | |
| krotscheck | Well, thankfully I don't need a production server. | 16:15 |
| krotscheck | For that, apache will do. | 16:15 |
| krotscheck | Also... I dunno- installing from github directly feels a little bit "Oh hey your history just got rewritten" | 16:15 |
| jeblair | it's not something we like to do. | 16:16 |
| krotscheck | I don't blame you | 16:16 |
| krotscheck | So the options are "Install from source", "roll our own package", ... or... have a build box spin up a vm? | 16:16 |
| krotscheck | Any other ideas? | 16:16 |
| jeblair | krotscheck: what project is this for? | 16:17 |
| ruhe | would it be possible to install needed version of nodejs from ppa and restrict tests to ubuntu nodes? | 16:17 |
| krotscheck | ruhe: yes, assuming I can get xvfb and firefox. | 16:18 |
| krotscheck | Here's the repo for now- doc updates and bootstrapper on the way: https://github.com/krotscheck/storyboard-webclient | 16:19 |
| *** ruhe has quit IRC | 16:20 | |
| *** chandankumar has quit IRC | 16:20 | |
| *** dizquierdo has quit IRC | 16:21 | |
| krotscheck | Ugh- precise doesn't have a recent version of compass- I guess we'll just use LESS instead | 16:21 |
| krotscheck | Fewer dependencies++ though | 16:22 |
| jeblair | krotscheck: install from source (in the test workspace) is probably the best answer for now. the primary purpose of the unit test nodes is to test openstack, so we can't install a node.js there that isn't in precise. | 16:24 |
| krotscheck | Seems fair. You're referring to the jenkins workspace, yes? | 16:26 |
| jeblair | yeah | 16:26 |
| krotscheck | Got it. Alright, I'll see if I can get a bootstrap script up and running that'll build it and add it to the runtime PATH | 16:27 |
| krotscheck | Do we blow away those workspaces after every build? | 16:28 |
| jeblair | krotscheck: before every build, but yes | 16:28 |
| krotscheck | kk. Clean working dir. | 16:28 |
| *** dstanek has quit IRC | 16:32 | |
| *** boris-42 has joined #openstack-infra | 16:34 | |
| *** reed has joined #openstack-infra | 16:39 | |
| *** sarob has joined #openstack-infra | 16:43 | |
| *** harlowja is now known as harlowja_away | 16:44 | |
| *** sarob has quit IRC | 16:48 | |
| *** dstanek has joined #openstack-infra | 16:51 | |
| *** prad has joined #openstack-infra | 16:53 | |
| jeblair | wow. the devstack-gate checkout logic for branches has been broken since late october. | 16:54 |
| jeblair | so most of those 21 tests that we run on devstack-gate changes aren't actually testing anything. :( | 16:55 |
| *** dstanek has quit IRC | 16:55 | |
| *** coolsvap has quit IRC | 16:55 | |
| fungi | they were working (could tell by the way they were failing) when i first added them :/ | 16:56 |
| jeblair | fungi: yeah, but i think that was a long time ago | 16:57 |
| jeblair | i'm going to try to straighten this out, but it's going to take a while | 16:57 |
| jeblair | fungi: note this change is not self-testing: https://review.openstack.org/#/c/63934/ and it's pretty much exactly the sort of change that we have all those tests there for | 16:58 |
| fungi | argh | 17:00 |
| jeblair | fungi: i'm going to start by attempting to write functional tests for the checkout code. | 17:01 |
| *** alexpilotti has joined #openstack-infra | 17:06 | |
| fungi | this may explain why grenade tests on master changes last night were failing even though a stable/havana requirements change ahead of them should have fixed that failure | 17:08 |
| jeblair | :( | 17:08 |
| *** nicedice has joined #openstack-infra | 17:09 | |
| fungi | it seemed like the zuul ref for the havana change wasn't being incorporated into the tests on master, but it was also very late and i was having trouble seeing straight at that point, so i thought i may have just been misunderstanding the checkout code there | 17:09 |
| *** prad has quit IRC | 17:09 | |
| *** prad has joined #openstack-infra | 17:10 | |
| *** senk has quit IRC | 17:15 | |
| SergeyLukjanov | will we have an infra meeting today? | 17:20 |
| jeblair | SergeyLukjanov: no, next one is next year | 17:20 |
| SergeyLukjanov | jeblair, k thx | 17:21 |
| *** reed has quit IRC | 17:21 | |
| *** senk has joined #openstack-infra | 17:22 | |
| *** senk has quit IRC | 17:27 | |
| *** nati_ueno has joined #openstack-infra | 17:28 | |
| *** senk has joined #openstack-infra | 17:28 | |
| *** sarob has joined #openstack-infra | 17:31 | |
| *** senk1 has joined #openstack-infra | 17:31 | |
| *** senk has quit IRC | 17:33 | |
| *** sarob_ has joined #openstack-infra | 17:35 | |
| *** sarob has quit IRC | 17:35 | |
| *** yassine has quit IRC | 17:38 | |
| *** dcramer_ has joined #openstack-infra | 17:42 | |
| *** sarob_ has quit IRC | 17:43 | |
| *** krotscheck has quit IRC | 17:52 | |
| *** gsamfira has quit IRC | 17:59 | |
| *** tma996 has quit IRC | 18:00 | |
| *** senk1 has quit IRC | 18:01 | |
| *** ruhe has joined #openstack-infra | 18:02 | |
| *** reed has joined #openstack-infra | 18:07 | |
| *** ruhe has quit IRC | 18:15 | |
| *** nicedice has quit IRC | 18:16 | |
| *** nicedice has joined #openstack-infra | 18:17 | |
| *** nicedice_ has joined #openstack-infra | 18:19 | |
| *** nicedice has quit IRC | 18:22 | |
| *** salv-orlando has quit IRC | 18:25 | |
| clarkb | morning | 18:28 |
| clarkb | also :( d-g checkout logic | 18:29 |
| jeblair | clarkb: yeah. i have failing tests now though. | 18:29 |
| clarkb | jeblair: is CRM114 Revealed the book you referred to earlier about crm114 stuff? | 18:31 |
| jeblair | clarkb: yes | 18:32 |
| *** senk has joined #openstack-infra | 18:32 | |
| clarkb | thanks, I am doing my best to understand the crm program today | 18:33 |
| jeblair | clarkb: k. lemme know if you have questions. | 18:33 |
| *** senk has quit IRC | 18:36 | |
| *** sarob has joined #openstack-infra | 18:48 | |
| *** dcramer_ has quit IRC | 18:50 | |
| *** sarob has quit IRC | 18:52 | |
| *** senk has joined #openstack-infra | 19:02 | |
| *** gothicmindfood has joined #openstack-infra | 19:04 | |
| *** gothicmindfood has quit IRC | 19:05 | |
| clarkb | jeblair: "CRM114 does NOT have a perfect solution for the Halting Problem!" this book is amusing | 19:06 |
| jeblair | clarkb: yeah, i had a lot of fun hacking in it. :) | 19:07 |
| *** senk has quit IRC | 19:07 | |
| clarkb | the language is pretty neat too | 19:07 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Move workspace setup to functions.sh https://review.openstack.org/63956 | 19:18 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Move workspace setup to functions.sh https://review.openstack.org/63956 | 19:20 |
| jeblair | clarkb, fungi: when you have a minute, i'd love feedback on whether https://review.openstack.org/#/c/63956/2/test-functions.sh hits all the test scenarios we need it to (and that the assertions about final state are correct) | 19:21 |
| jeblair | it's safe to run in a local checkout of devstack-gate. it mocks out all of the git actions so it's fully simulated. | 19:21 |
| jeblair | (i mostly did that for speed; early versions actually did all the git actions that were possible, but it's really slow) | 19:22 |
| fungi | i just spotted it on the zuul status page and started looking at it. i'll let you know | 19:23 |
| clarkb | sure taking a look now. for the crm stuff http://git.openstack.org/cgit/openstack-infra/config/tree/modules/log_processor/files/classify-log.crm#n53 seems to copy a variable that we never use again and should all of the matches use the <nomultiline> option? | 19:23 |
| clarkb | jeblair: ^ | 19:23 |
| jeblair | clarkb: yeah, i think that was left over from an earlier version of the script that echoed the line back, but it turns out we didn't need it (the way i ended up writing the worker-filter just expected a number and associated it with the line which it already knew) | 19:25 |
| jeblair | clarkb: probably some of the other vars (:prob: :pr: :result:, maybe even :stats:) don't need to be isolated either | 19:26 |
| clarkb | ya, I don't expect a major performance improvement by not copying those things, but I suppose every little bit counts | 19:26 |
| clarkb | jeblair: also, apparently crm114 can run as a daemon to preserve JIT knowledge. Maybe we should look into doing that as well? | 19:27 |
| jeblair | oh, good idea | 19:27 |
| *** nicedice_ has quit IRC | 19:32 | |
| *** nicedice_ has joined #openstack-infra | 19:33 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Move workspace setup to functions.sh https://review.openstack.org/63956 | 19:35 |
| fungi | jeblair: the list of scenarios on 63956 looks sufficient to me. i'm still looking deeper into the rest of the change | 19:35 |
| jeblair | fungi: cool. hoping ^ will make it actually run for real. :) | 19:35 |
| fungi | though i guess i'll switch to the next patchset to continue ;) | 19:35 |
| jeblair | fungi, clarkb: that patchset was a change to the wrap script, not to the testing infrastructure | 19:36 |
| fungi | seems to run clean when i try it myself | 19:39 |
| clarkb | jeblair: https://review.openstack.org/#/c/63956/2/test-functions.sh line 213 why is that wrong? ZUUL_BRANCH is set to master shouldn't it checkout master/ZC instead of stable/havana/ZC? | 19:40 |
| jeblair | okay, that's running. there shouldn't be any logic changes, so after it's done in an hour, we'll just want to check that it still does what we expect | 19:40 |
| clarkb | oh I see because setup project sets up on the old version first | 19:41 |
| jeblair | clarkb: yep; since the intent is to set up the predicted future state of havana branches, it should grab change #2 (the havana change to glance) | 19:41 |
| clarkb | lgtm | 19:42 |
| fungi | i guess the idea is that we peel back the commented asserts one or more at a time with accompanying fixes which make them pass? | 19:44 |
| *** ryanpetrello has joined #openstack-infra | 19:44 | |
| jeblair | fungi: yep | 19:45 |
| jeblair | oh, it looks like you can += without needing to assign first; so i think i'll do that real quick for all those array assignments | 19:48 |
| fungi | jeblair: on 244 (and maybe i'm just asking the inverse of clarkb's question but) shouldn't that be ZUUL_REF='refs/zuul/stable/havana/ZC' there? | 19:48 |
| *** mancdaz is now known as mancdaz_away | 19:49 | |
| jeblair | fungi: i think you're right about that; it should be the info for the change at the bottom of the queue | 19:49 |
| *** sarob has joined #openstack-infra | 19:49 | |
| fungi | okay, that's what it seemed to me as well. good ;) | 19:49 |
| jeblair | that fixes the 2nd test failure in there (but not the first) | 19:51 |
| jeblair | (nor the 3rd or 4th) | 19:52 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Move workspace setup to functions.sh https://review.openstack.org/63956 | 19:53 |
| jeblair | fungi, clarkb: ^ changes to test-functions.sh | 19:53 |
| *** sarob has quit IRC | 19:53 | |
| *** ryanpetrello has quit IRC | 19:57 | |
| *** alexpilotti has quit IRC | 20:07 | |
| *** krtaylor has quit IRC | 20:07 | |
| *** krtaylor has joined #openstack-infra | 20:08 | |
| *** markmcclain has quit IRC | 20:19 | |
| *** alexpilotti has joined #openstack-infra | 20:34 | |
| fungi | looks like it's passing its last test now | 20:41 |
| *** sarob has joined #openstack-infra | 20:43 | |
| clarkb | jeblair: can http://git.openstack.org/cgit/openstack-infra/config/tree/modules/log_processor/files/classify-log.crm#n46 use bychunk instead of bychar to potentially go faster | 20:47 |
| *** sarob has quit IRC | 20:48 | |
| akscram | hello, fungi and clarkb | 21:00 |
| akscram | fungi, clarkb: how can I receive an approval on this https://review.openstack.org/#/c/61215/? | 21:02 |
| clarkb | akscram: we need to merge https://review.openstack.org/#/c/63490/ first | 21:10 |
| clarkb | akscram: jeblair hasn't reviewed that change yet | 21:10 |
| clarkb | jeblair: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/log_processor/files/classify-log.crm#n75 why is pR always positive in that case? | 21:20 |
| jeblair | clarkb: the worker is line-oriented (that is, it sends one line, then waits for a response to that line before sending the next) | 21:36 |
| jeblair | clarkb: i tried writing the whole file then reading all the data in, and that was such a marginal improvement (i don't recall the numbers) that i thought this was easier and good enough | 21:38 |
| clarkb | jeblair: gotcha wfm | 21:38 |
| clarkb | jeblair: for the pR always being positive statement. Is that because you give two stats files and pR will be negative for one and positive for the other | 21:38 |
| jeblair | clarkb: so i that's why it's bychar; it may not be the best, but i don't think there's a big gain to be had with a simple change | 21:38 |
| jeblair | clarkb: if you look at the example stats output (in the comments starting on line 66) | 21:40 |
| jeblair | clarkb: each file gets a pR indicating the probability that file matches or doesn't match | 21:40 |
| jeblair | clarkb: in practice the winning pR is always positive (i'm not certain that's absolutely true, but it's what seems to happen) | 21:41 |
| jeblair | clarkb: so we're basically encoding an extra bit of information in the sign field: which file matched | 21:42 |
| jeblair | clarkb: so if it matched the failure class, we return the negative of the pR for the winning (failure) class | 21:43 |
| *** sarob has joined #openstack-infra | 21:43 | |
| clarkb | I see so it may happen that both pRs are negative with FAILURE being the best match which would be negated turning positive flipping the result however that didn't happen in practice | 21:43 |
| jeblair | clarkb: and if it matched the success class, we return the pR as is | 21:43 |
| jeblair | clarkb: right, that seems like a theoretical possibility, but i didn't explore deeply enough to confirm. | 21:44 |
| clarkb | jeblair: I think that is the only case where the sign flipping breaks | 21:44 |
| jeblair | clarkb: oh, i did leave a (0) comment on https://review.openstack.org/#/c/63490/ | 21:45 |
| clarkb | jeblair: I see it now thanks | 21:45 |
| jeblair | clarkb: i think i got the name of the column in elasticsearch backwards; it's not really error_pr, it's non_error_pr. | 21:46 |
| clarkb | I feel much more up to speed on the CRM stuff now. Last question, do we think train everything is working because jenkins is preclassifying for us? | 21:47 |
| *** sarob_ has joined #openstack-infra | 21:47 | |
| clarkb | that gives the corpus quit a bit more info | 21:47 |
| *** sarob has quit IRC | 21:47 | |
| jeblair | clarkb: yeah, there's a huge opportunity to try other systems (and you'll note the book describes a lot of training methods). but yeah... | 21:48 |
| jeblair | clarkb: the pre-classification as well as the fact that most lines show up in good and bad runs and we're looking for lines that more stongly correlate to one or the other... | 21:49 |
| clarkb | jeblair: for https://review.openstack.org/#/c/63490/1/jeepyb/cmd/manage_projects.py basically move line 177 out of the loop? | 21:50 |
| jeblair | clarkb: that is to say, that we're really interested in the frequency, seems to make this system work well enough -- at least, slightly better than a TOE system on my very small data set. | 21:50 |
| jeblair | clarkb: yes, i think so (line 177) | 21:51 |
| clarkb | I will test that in a bit on review-dev | 21:51 |
| jeblair | clarkb: i think the reason i didn't leave a vote is because i knew at least your current solution was tested; if you have time to give it another pass, that would be cool, or if you just want to merge to fix, i'm okay with that. | 21:52 |
| *** sarob_ has quit IRC | 21:52 | |
| *** nicedice_ has quit IRC | 21:52 | |
| *** nicedice_ has joined #openstack-infra | 21:52 | |
| clarkb | I should have time a little later, I need to make lunch now | 21:55 |
| fungi | clarkb: if you want to review/approve https://review.openstack.org/63204 then you can use puppet agent again and not need to apply by hand | 21:55 |
| fungi | mmmlunch | 21:55 |
| clarkb | fungi: I have actually got a small test script that just does group creation and uuid retrieval because using manage-projects directly was too much extra work :) | 21:56 |
| clarkb | but I will review that change | 21:56 |
| *** dims has quit IRC | 21:58 | |
| *** sarob has joined #openstack-infra | 21:59 | |
| clarkb | fungi: approved | 22:00 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "upgrade review-dev from gerrit ver 2.4.x to ver 2.8" https://review.openstack.org/63204 | 22:00 |
| *** sarob has quit IRC | 22:03 | |
| jeblair | clarkb, fungi: mordred added OVERRIDE_ZUUL_BRANCH to devstack-gate; but I don't see it used anywhere; do you recall it every being set? | 22:03 |
| jeblair | (note, this is not the same thing as clarkb's BRANCH_OVERRIDE jjb var; that just sets ZUUL_BRANCH) | 22:03 |
| *** nicedice_ has quit IRC | 22:07 | |
| *** nicedice_ has joined #openstack-infra | 22:07 | |
| *** jaypipes has quit IRC | 22:07 | |
| clarkb | yes it is for the client testing that hasn't happened yet | 22:11 |
| *** senk has joined #openstack-infra | 22:11 | |
| *** dims has joined #openstack-infra | 22:12 | |
| jeblair | clarkb: i think that may be the same logic we want to use with the devstack-gate branch check jobs | 22:14 |
| clarkb | good point | 22:15 |
| *** senk has quit IRC | 22:15 | |
| *** prad has quit IRC | 22:21 | |
| *** reed has quit IRC | 22:27 | |
| *** nicedice_ has quit IRC | 22:33 | |
| *** nicedice_ has joined #openstack-infra | 22:33 | |
| openstackgerrit | Dmitry Pyzhov proposed a change to openstack-infra/config: Allow fuel-core team manage stable branches https://review.openstack.org/63970 | 22:33 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/jeepyb: Use MySQLDB transactions properly. https://review.openstack.org/63490 | 22:37 |
| clarkb | that was tested on review-dev it seems to work | 22:37 |
| *** sarob has joined #openstack-infra | 22:43 | |
| *** reed has joined #openstack-infra | 22:47 | |
| SergeyLukjanov | guys, is it theoretically possible to add a comment in gerrit to the approved/merged change? | 22:48 |
| *** sarob has quit IRC | 22:48 | |
| jeblair | SergeyLukjanov: yes; if you use the ssh api, you have to give --force-message (or something like that) | 22:49 |
| SergeyLukjanov | jeblair, interesting, I just thinking about how to make 'post' jobs results easily accessible | 22:50 |
| SergeyLukjanov | like coverage reports and etc. | 22:50 |
| jeblair | SergeyLukjanov: the unfortunate thing is that the post jobs don't know what change they are for. | 22:50 |
| SergeyLukjanov | jeblair, oh, that's the thing I've missed | 22:51 |
| jeblair | SergeyLukjanov: they run based on the ref-updated event; if gerrit creates a merge commit, that information isn't supplied in the change-merged event | 22:51 |
| SergeyLukjanov | jeblair, yup ,got it, thanks | 22:51 |
| jeblair | SergeyLukjanov: so if "change foo" merges, but gerrit makes a merge commit "Merge 'change foo'", "Merge 'change foo'" is what we want to actually use, but we don't know that. | 22:52 |
| jeblair | sorry, i had typed most of that already. :) | 22:52 |
| jeblair | i think it's possible to patch gerrit to include that information though. | 22:53 |
| jeblair | then we could switch | 22:53 |
| SergeyLukjanov | jeblair, interesting, I have never look before at gerrit's code | 22:54 |
| SergeyLukjanov | should try to do it after completing my patches for zuul | 22:54 |
| SergeyLukjanov | jeblair, are you guys planning to take day offs for xmas holidays? | 22:55 |
| jeblair | SergeyLukjanov: i'm planning on taking tomorrow off | 22:56 |
| SergeyLukjanov | jeblair, heh, it seems like it'll OpenStack shutdown if all infra folks will take a week off ;) | 22:58 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Fix workspace setup https://review.openstack.org/63972 | 22:59 |
| SergeyLukjanov | it'll be* | 22:59 |
| jeblair | clarkb, fungi: ^ a fix! | 22:59 |
| clarkb | looking | 22:59 |
| clarkb | SergeyLukjanov: I will take tomorrow and probably friday off | 22:59 |
| clarkb | I need to write me LCA talk which is why I dove into CRM today. I think I grok enough of it to get through much of the talk on thursday \o/ | 23:00 |
| jeblair | clarkb: cool! | 23:00 |
| clarkb | jeblair: BRANCH_OVERRIDE is used in a lot of jobs | 23:02 |
| clarkb | jeblair: is that a different override or will they need updating? | 23:02 |
| clarkb | er ZUUL_BRANCH | 23:04 |
| jeblair | clarkb: i _think_ that any time zuul_branch is set, we should set override_zuul_branch instead | 23:04 |
| clarkb | ok | 23:04 |
| jeblair | clarkb: for non-periodic jobs, at least. | 23:05 |
| jeblair | clarkb: i _think_ the current set of jobs that use a branch override that are not periodic is exactly equal to the set of devstack-gate alternate branch check jobs... | 23:06 |
| jeblair | clarkb: and i'm fairly certain that's the right thing to do for them. | 23:06 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use OVERRIDE_ZUUL_BRANCH in alternate-branch jobs https://review.openstack.org/63973 | 23:09 |
| jeblair | there's that ^ | 23:09 |
| clarkb | jeblair: waiting to see how check tests do, but my eyeballs don't see anything wrong | 23:20 |
| jeblair | clarkb: i don't think the alternate-branch jobs will quite do the right thing until the other change lands; once it does we can recheck the change i -2'd this morning to see. | 23:24 |
| clarkb | jeblair: order should be d-g change, config change, then rechecks? | 23:24 |
| jeblair | yep | 23:24 |
| jeblair | clarkb: i'm going to copy the css files from logstash-worker16 to 01-08, then make it so that worker01-04 processes A-D are the only ones running | 23:28 |
| jeblair | clarkb: (that's the same count as now, but 4 per host instead of 2, and on hosts 1-8 since, well, they are more likely to not be deleted than the higher numbers which are currently running) | 23:29 |
| jeblair | (sorry, 1-4 in that last ^) | 23:29 |
| clarkb | ok | 23:30 |
| *** SergeyLukjanov has quit IRC | 23:31 | |
| jeblair | clarkb, fungi: okay, puppet is running on all the logstash workers and is up to date; workers 01-04 are running 4 processes each (A-D). | 23:39 |
| jeblair | clarkb, fungi: if the individual hosts become overloaded, stop one or two of those processes on them. and if the system gets behind overall, start processes on more hosts | 23:39 |
| clarkb | ok | 23:40 |
| *** sarob has joined #openstack-infra | 23:43 | |
| *** boris-42 has quit IRC | 23:44 | |
| clarkb | jeblair: I left a comment on the config change | 23:45 |
| jeblair | clarkb: the first of the d-g changes has failed tempest twice with the same error (but in different jobs) | 23:45 |
| jeblair | looks like there's a bug for it; bug 1263019 | 23:46 |
| uvirtbot | Launchpad bug 1263019 in nova "tempest.api.volume.test_volume_transfers.VolumesTransfersTest.test_create_list_delete_volume_transfer failed" [Undecided,New] https://launchpad.net/bugs/1263019 | 23:46 |
| *** sarob has quit IRC | 23:48 | |
| *** xchu has joined #openstack-infra | 23:55 | |
| *** nicedice_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!