| *** dstanek has quit IRC | 00:01 | |
| gyee_ | is there a way to restore a patch on behalf of someone else? | 00:01 |
|---|---|---|
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/config: Add compass project to stackforge Partial-bug: 1265229 Change-Id: I29bcffc64e7f3ea1dbd50ffae87206fe5b1229fe https://review.openstack.org/65178 | 00:01 |
| fungi | gyee_: in our current version of gerrit, only a gerrit global administrator can do that--if you give me the change url i'll be happy to | 00:01 |
| gyee_ | fungi, https://review.openstack.org/#/c/57811/ | 00:01 |
| gyee_ | fabio's on vacation and asked me to finish up the patch for him | 00:02 |
| fungi | gyee_: once we get done upgrading to a more recent gerrit, we should be able to grant core teams that right via an acl | 00:02 |
| *** mkerrin1 has joined #openstack-infra | 00:02 | |
| fungi | gyee_: it's restored now. enjoy! | 00:02 |
| *** lifeless_ has joined #openstack-infra | 00:02 | |
| *** david-lyle_ has quit IRC | 00:02 | |
| *** rwsu has quit IRC | 00:02 | |
| *** krotscheck has quit IRC | 00:02 | |
| *** dims_ has quit IRC | 00:02 | |
| *** UtahDave has quit IRC | 00:02 | |
| *** salv-orlando has quit IRC | 00:02 | |
| *** krtaylor has quit IRC | 00:02 | |
| *** nati_ueno has quit IRC | 00:02 | |
| *** herndon has quit IRC | 00:02 | |
| *** dkliban has quit IRC | 00:02 | |
| *** yidclare has quit IRC | 00:02 | |
| *** branen__ has quit IRC | 00:02 | |
| *** rcleere has quit IRC | 00:02 | |
| *** weshay has quit IRC | 00:02 | |
| *** mkerrin has quit IRC | 00:02 | |
| *** medberry has quit IRC | 00:02 | |
| *** jcooley has quit IRC | 00:02 | |
| *** lifeless has quit IRC | 00:02 | |
| *** rockstar has quit IRC | 00:02 | |
| *** rcleere_ is now known as rcleere | 00:02 | |
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/config: Add compass project to stackforge https://review.openstack.org/65178 | 00:02 |
| *** salv-orlando_ is now known as salv-orlando | 00:03 | |
| morganfainberg | fungi, ooooh AWESOME if a core can do that w/ an ACL when we get a new verison | 00:03 |
| gyee_ | fungi, thanks! | 00:03 |
| morganfainberg | fungi, that would make me super happy (as a core member) | 00:03 |
| gyee_ | ++ | 00:03 |
| morganfainberg | and means i don't need to tell people to come bug you guys for keystone reviews that need life breathed back into them | 00:03 |
| fungi | morganfainberg: yep, been on our wishlist for ages. i for one will be happy to no longer need to restore changes for people | 00:03 |
| morganfainberg | :) | 00:04 |
| morganfainberg | well, i look forward to new version of gerrit then! | 00:04 |
| *** weshay has joined #openstack-infra | 00:04 | |
| fungi | as much as i like playing the hero, there's already plenty of other broken thigs which need my time | 00:04 |
| *** rockstar has joined #openstack-infra | 00:04 | |
| *** rockstar has joined #openstack-infra | 00:04 | |
| *** dims_ has joined #openstack-infra | 00:04 | |
| fungi | er, things | 00:04 |
| *** krotscheck has joined #openstack-infra | 00:04 | |
| *** krtaylor has joined #openstack-infra | 00:04 | |
| *** rwsu has joined #openstack-infra | 00:04 | |
| *** yamahata has quit IRC | 00:05 | |
| *** UtahDave has joined #openstack-infra | 00:06 | |
| *** dkliban has joined #openstack-infra | 00:06 | |
| *** jcooley has joined #openstack-infra | 00:06 | |
| sdague | fungi: so we're getting those mysterious lost time at the beginning of runs again - https://jenkins02.openstack.org/job/gate-tempest-dsvm-full/4101/consoleFull | 00:07 |
| sdague | that was a git gc issue the last time, right? | 00:07 |
| fungi | sdague: i honestly am not sure. i think we've also seen it when jenkins masters are hammered (like now) | 00:07 |
| fungi | which is one of the reasons we're in the process of standing up jenkins02 and 04 at this moment | 00:08 |
| sdague | ok, cool | 00:08 |
| fungi | er, 03 and 04 | 00:08 |
| sdague | I'll let you get back to it. Dinner time anyway. will ping for a post mortem tomorrow when things are mellow, as I missed what happened | 00:09 |
| *** pmathews has quit IRC | 00:11 | |
| *** fifieldt has quit IRC | 00:13 | |
| *** fifieldt has joined #openstack-infra | 00:13 | |
| *** oubiwann has quit IRC | 00:13 | |
| fungi | corvus: /var/lib/jenkins/plugins/{scp,zmq-event-publisher}.jpi exist on 03 and 04 now, copied from 02 | 00:16 |
| *** rcleere has quit IRC | 00:16 | |
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/config: Add compass project to stackforge https://review.openstack.org/65178 | 00:18 |
| *** dstanek has joined #openstack-infra | 00:18 | |
| fungi | not sure what else is required (normally i've uploaded them via the webui instead, so no idea whether those help and maven metadata files in the similarly-named directories are needed) | 00:19 |
| *** kraman has joined #openstack-infra | 00:20 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add jobs for cliff integration tests https://review.openstack.org/65180 | 00:21 |
| fungi | corvus: went ahead and added those too. hopefully they don't embed anything host-specific | 00:23 |
| *** senk has quit IRC | 00:24 | |
| *** UtahDave has quit IRC | 00:25 | |
| *** fungi has quit IRC | 00:30 | |
| devananda | something seems odd with jenkins on 48109 | 00:34 |
| *** eharney has quit IRC | 00:34 | |
| devananda | gate started 10hr ago, no response. zuul status page shows "unknown" for the job and skipped for all the tests | 00:36 |
| *** dstanek has quit IRC | 00:36 | |
| *** rnirmal has quit IRC | 00:36 | |
| *** dims_ is now known as dims | 00:37 | |
| *** lifeless_ is now known as lifeless | 00:40 | |
| *** salv-orlando has quit IRC | 00:41 | |
| *** salv-orlando has joined #openstack-infra | 00:42 | |
| *** ^d has quit IRC | 00:45 | |
| *** jerryz is now known as jerryzhao | 00:51 | |
| *** jerryzhao is now known as jerryz | 00:51 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: make scm test as the examples https://review.openstack.org/65186 | 00:52 |
| *** krotscheck has quit IRC | 00:55 | |
| *** prad has quit IRC | 00:58 | |
| corvus | clarkb2: let's start jenkins on 3 and 4 now | 00:59 |
| corvus | clarkb2: i'll do 3 | 00:59 |
| *** clarkb2 is now known as clarkb | 01:00 | |
| clarkb | corvus: starting 4 now | 01:00 |
| clarkb | corvus: will need to update the logstash client to consume events from those jenkinses as well | 01:01 |
| corvus | clarkb: yes, we'll want to add them to everything except nodepool config in one change, then nodepool in a second | 01:02 |
| corvus | clarkb: i think authn isn't right | 01:02 |
| corvus | clarkb: i think there's some xml that needs editing; i'll do ti | 01:04 |
| clarkb | corvus: ok | 01:04 |
| corvus | clarkb: done, restarting both | 01:06 |
| clarkb | corvus: which file was edited? | 01:06 |
| * clarkb is curious what was needed to fix it | 01:06 | |
| corvus | hudson.tasks.Mailer.xml jenkins.model.JenkinsLocationConfiguration.xml org.codefirst.SimpleThemeDecorator.xml | 01:07 |
| corvus | clarkb: want to run "jenkins-jobs update /etc/jenkins_jobs/config" ? | 01:08 |
| clarkb | corvus: sure | 01:08 |
| clarkb | corvus: there was an auth failure, I assume we either update xml to change the key or change the jjb ini to use a different key? | 01:10 |
| *** rhsu has quit IRC | 01:10 | |
| corvus | clarkb: yeah, they have different api keys | 01:11 |
| *** praneshp has quit IRC | 01:11 | |
| corvus | clarkb: i think we should stop them again and change the xml file for the gerrig user to match what jjb is sending | 01:11 |
| clarkb | ok | 01:11 |
| clarkb | stopping 4 | 01:11 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Add different envs for different sqlalchemy versions https://review.openstack.org/65135 | 01:11 |
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/config: Add compass project to stackforge https://review.openstack.org/65178 | 01:12 |
| corvus | clarkb: oh, i think we have to copy the gerrig xml file from another host | 01:12 |
| clarkb | I see | 01:12 |
| corvus | clarkb: /var/lib/jenkins/users/gerrig/config.xml | 01:13 |
| *** jasondotstar has joined #openstack-infra | 01:13 | |
| *** fungi has joined #openstack-infra | 01:14 | |
| corvus | clarkb: oh, i think i botched the xml extraction; | 01:17 |
| corvus | clarkb: we're missing too many files | 01:17 |
| corvus | clarkb: let me try again | 01:17 |
| clarkb | corvus: k | 01:17 |
| fungi | okay, i'm back | 01:18 |
| fungi | hopefully that extended outage was my isp fixing flow issues | 01:18 |
| *** praneshp has joined #openstack-infra | 01:18 | |
| fungi | i'll check the logs in a sec | 01:18 |
| *** weshay has quit IRC | 01:18 | |
| clarkb | fungi: basically corvus found that we were missing xml stuff that we needed so jenkins processes are stopped and he is redoing the xml extraction | 01:19 |
| *** weshay has joined #openstack-infra | 01:19 | |
| fungi | okay, caught the tail end of that already. thanks | 01:19 |
| fungi | sounds like i missed very little then | 01:19 |
| clarkb | ya not much, we were poking at it then corvus realized what had happened | 01:19 |
| *** ^d has joined #openstack-infra | 01:20 | |
| hub_cap | so we just added our first core member (post the initial seed of members), and im really not sure what to do after the email approval :) | 01:20 |
| *** nosnos has joined #openstack-infra | 01:20 | |
| clarkb | hub_cap: log into review.o.o, under admin groups find your core group and click on it | 01:21 |
| clarkb | hub_cap: on the left there should be a members tab, go there, and add the user | 01:21 |
| hub_cap | o snap thats easy | 01:21 |
| hub_cap | thx clarkb | 01:21 |
| clarkb | np | 01:21 |
| *** yaguang has joined #openstack-infra | 01:23 | |
| corvus | clarkb, fungi: the correct tarballs are in ~corvus on 03 and 04 (i hope) | 01:25 |
| clarkb | corvus: just extract in place? | 01:25 |
| *** jerryz is now known as jerryzhao | 01:25 | |
| corvus | clarkb: yes, then change those 3 files | 01:26 |
| fungi | corvus: okay. i originally looked at the 3 tarballs in your homedir on jenkins.o.o | 01:27 |
| corvus | fungi: same ones | 01:28 |
| fungi | awesome | 01:28 |
| corvus | still missing something though; the api keys are different | 01:28 |
| corvus | secret.key is different | 01:30 |
| *** moted has joined #openstack-infra | 01:30 | |
| *** vijay has joined #openstack-infra | 01:30 | |
| *** CaptTofu has quit IRC | 01:31 | |
| vijay | Can someone help creating a gerrit service account, it will be used by the TestInfraStructure at citrix | 01:33 |
| *** melwitt has quit IRC | 01:33 | |
| *** vipul is now known as vipul-away | 01:34 | |
| *** dkranz has joined #openstack-infra | 01:35 | |
| *** weshay has quit IRC | 01:36 | |
| fungi | vijay: sure, have you seen http://ci.openstack.org/third_party.html#requesting-a-service-account | 01:36 |
| fungi | vijay: that lists the information we'll need to be able to set it up for you | 01:36 |
| vijay | Yes. I have the details withe me right now | 01:36 |
| vijay | Shall i post it here? | 01:37 |
| fungi | vijay: the ssh key might be a little large (you could post it to http://paste.openstack.org/ and provide a link to it here if it's not published somewhere already) | 01:37 |
| fungi | vijay: the full name, username and e-mail address should be easy to communicate directly in channel though | 01:38 |
| *** boris-42 has quit IRC | 01:39 | |
| corvus | fungi, clarkb: ugh; looks like secrets.tar.gz is missing a path component | 01:39 |
| corvus | fixing | 01:40 |
| vijay | ssh-key: http://paste.openstack.org/show/60544/ | 01:40 |
| vijay | UserName: NetScalerAts | 01:40 |
| vijay | Display Name: NetScaler TestingSystem | 01:40 |
| vijay | Email-id: #Networking-CloudIntegDev@citrix.com | 01:41 |
| *** dkranz has quit IRC | 01:41 | |
| fungi | vijay: i've added it | 01:42 |
| fungi | vijay: test it out and let us know if you have any trouble with it | 01:42 |
| vijay | The email-id will be enabled only in another 3 hours. I have requested for creation. | 01:42 |
| clarkb | fungi: have you followed the ability to vote for third party testers discussion? | 01:42 |
| SpamapS | when I get this, what bug should I reverify against: error: Failed to connect to 2001:4800:7813:516:3bc3:d7f6:ff04:aacb: Network is unreachable while accessing https://git.openstack.org/openstack/tempest/info/refs | 01:42 |
| clarkb | fungi: I thin kwe want to defailt to not allowing voting | 01:43 |
| SpamapS | new bug on openstack-infra ? | 01:43 |
| SpamapS | looks entirely transient | 01:43 |
| fungi | vijay: i guess that was your e-mail to the openstack-infra mailing list earlier today. i'll follow up to it too | 01:43 |
| *** sarob has joined #openstack-infra | 01:43 | |
| clarkb | SpamapS: ya, bug against openstack-infra with note that it appares to be a transient network problem will work | 01:43 |
| *** talluri has quit IRC | 01:44 | |
| corvus | okay the api keys match | 01:44 |
| fungi | clarkb: i've been following, but looked like we hadn't arrived at and officially published a decision on it yet, so i'd been doing business as usual and disabling voting for specific accounts on request in the meantime | 01:44 |
| corvus | clarkb: you should be able to run jjb now | 01:44 |
| clarkb | corvus: running now | 01:44 |
| *** CaptTofu has joined #openstack-infra | 01:45 | |
| *** ^d has quit IRC | 01:45 | |
| clarkb | corvus: did we make it official in that bofh last night and just fail at communicating it? | 01:46 |
| fungi | clarkb: part of the challenge is that some of the third-party testing systems are implemented specifically to provide feedback on one (or a specific group of) stackforge project(s), so i'm not sure under which circumstances we evaluate that | 01:46 |
| *** pcrews has quit IRC | 01:46 | |
| SpamapS | clarkb: ty, https://bugs.launchpad.net/openstack-ci/+bug/1266616 filed | 01:46 |
| corvus | clarkb: i don't think we can make something like that official at a bof | 01:46 |
| *** ryanpetrello has joined #openstack-infra | 01:46 | |
| corvus | clarkb: however, no one has objected to that portion of the proposed requirements in review | 01:46 |
| clarkb | fungi: I think with ACLs? we can restrict which accounts can vote more precisely | 01:46 |
| vijay | fungi: I was not able to vote "--verified" with my account. But the newly created account "NetScalerATS" should be able to vote "verified" right? | 01:47 |
| clarkb | corvus: thats fair, the change needs reviwe then? | 01:47 |
| fungi | vijay: yes | 01:47 |
| corvus | clarkb: yes | 01:47 |
| lifeless | fungi: did you check whether your mirror is working atm? | 01:47 |
| clarkb | however, please please please, make sure you are ready to vote verifiend when you start doing that | 01:48 |
| *** yamahata has joined #openstack-infra | 01:48 | |
| fungi | vijay: though it's worth noting that there are some consensus decisions coming to a head about additional requirements before voting on changes is appropriate | 01:48 |
| corvus | fungi: until we have a better plan, i think 'ask the relevant ptl' if a system is ready to vote is a reasonable plan to start with. | 01:48 |
| fungi | vijay: we'll update documentation accordingly once that information has been finalized | 01:48 |
| fungi | corvus: sounds sane, though we need to collect additional information to determine how to map a third-party testing system to a particular program, i suppose | 01:49 |
| vijay | Ok. Currently it does LBaaS API testing along with NetScaler and then votes. It listens to changes and tests only if the change submitted is relevant to LBaaS plugin or NetScaler or Tempest LBaaS Tests | 01:50 |
| clarkb | jog0: any chance you can push a new patchset that addresses the current comments in the third party docs/ | 01:50 |
| jog0 | clarkb: actually about to step out for dinner, you want to push a new version? | 01:51 |
| fungi | lifeless: i've been fighting broken internet service today, but have just about managed to get enough of a web proxy up to be able to get a vm launched where i can try running pypi-mirror. the job logs didn't indicate the usual signs we saw from pip 1.5/virtualenv 1.11 use, but i saw other tracebacks and so want to try running it myself | 01:51 |
| clarkb | jog0: not really, I am conferencing :) | 01:52 |
| *** prad has joined #openstack-infra | 01:52 | |
| jog0 | clarkb: oh right enjoy LCA | 01:52 |
| *** michchap_ has joined #openstack-infra | 01:52 | |
| *** dcramer_ has joined #openstack-infra | 01:52 | |
| *** prad has quit IRC | 01:52 | |
| fungi | lifeless: also, i'm going to try to run it with newer pip so i can hopefully find other externally-hosted packages which haven't already turned up on unit test failures people have reporte | 01:52 |
| anteaya | vijay: it isn't what you are testing, it is if your system is stable (is it functioning) and do you have logs and contact information | 01:52 |
| fungi | d | 01:52 |
| clarkb | corvus: JJB is still going on 04, I have it in screen so will just detach if necessary | 01:52 |
| *** michchap has quit IRC | 01:53 | |
| jog0 | clarkb: I'll push a new rev in the AM -- but if someone else wants to push one instead by all means they can | 01:53 |
| clarkb | jog0: ok | 01:53 |
| fungi | lifeless: today has been full of more urgent issues which mostly killed my progress on that | 01:53 |
| anteaya | vijay: if your system is voting -1 because your system has fallen over, people will try to contact you and thenn your permissions will be revoked if you are not responsive | 01:53 |
| corvus | the only contentios thing in the patchset is the bit about the recheck commands. the rest of the comments are about things people want to add. | 01:53 |
| corvus | so i'm okay merging as is and continuing to revise. | 01:53 |
| lifeless | fungi: ack | 01:54 |
| jog0 | anteaya: btw since I got sick and couldn't make LCA I am seeing if I can make it to cananda | 01:54 |
| corvus | or a new patchset. either way. | 01:54 |
| anteaya | jog0: woohoo! | 01:54 |
| corvus | clarkb: jjb on 3 is done | 01:54 |
| anteaya | jog0: get better fast, lots of soup and sleep, we need you | 01:54 |
| clarkb | corvus: done on 4 too | 01:54 |
| corvus | writing config chnages | 01:55 |
| jog0 | anteaya: I am now on medication and starting to feel better | 01:55 |
| anteaya | fungi: also I see two new requests for 3rd party test accounts, please make them non-verifying by default | 01:55 |
| anteaya | jog0: awesome, keep me posted, health first, flying sick into cold weather won't help you, so feel good first | 01:56 |
| vijay | anteaya: ok. Sure. To start with, to make sure our system is stable enuf. We have triggerred the system to vote -1 only after we manually trigger the workflow. It will send an internal mail if it is "-1" and only after we double check the system will post -1. | 01:56 |
| fungi | anteaya: new requests where? | 01:56 |
| harlowja | clarkb i think corvus was thinking u might know about how https://github.com/openstack-infra/config/blob/master/modules/jenkins/manifests/slave.pp#L220 gets exposed to say nova (or other test users), do u know if there are any ENV variables set (or something similar to let the test know that this can be used) | 01:57 |
| clarkb | harlowja: there aren't any set. Intead you can do what nova does which is test the connection | 01:57 |
| harlowja | ah | 01:57 |
| clarkb | harlowja: I will get a link | 01:57 |
| anteaya | vijay: it sounds like a good approach, also make sure that muliple peopole check the email inbox for your testing system so you are responsive to inquires | 01:57 |
| *** vipul-away is now known as vipul | 01:57 | |
| anteaya | fungi: on the infra ml | 01:57 |
| vijay | antaeya: it is an email alias | 01:58 |
| harlowja | clarkb thx! | 01:58 |
| vijay | antaeya: Also, the logs are uploaded to an external system for download, and we will available through email for comments | 01:58 |
| anteaya | vijay: sounds like you are positioning yourself for success, thank you | 01:58 |
| clarkb | harlowja: http://git.openstack.org/cgit/openstack/nova/tree/nova/tests/db/test_migrations.py the docstring for that file is good as well as line 410 | 01:59 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add jenkins03 and 04 to most places https://review.openstack.org/65191 | 01:59 |
| corvus | clarkb, fungi: ^ | 01:59 |
| fungi | anteaya: if there are any new ones to the infra ml, then my internet breakage has delayed me receiving them. more likely they're requests i've already completed and replied to | 01:59 |
| *** AaronGr is now known as AaronGr_Zzz | 02:00 | |
| *** ryanpetrello has quit IRC | 02:00 | |
| clarkb | corvus: lgtm | 02:00 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Start using jenkins03 and 04 https://review.openstack.org/65192 | 02:02 |
| lifeless | fungi: alembic might be in the same boat | 02:03 |
| lifeless | No distributions at all found for alembic>=0.4.1 (from -r /tmp/tmpcfUzJx (line 1)) | 02:03 |
| corvus | fungi, clarkb: we'll want to merge the 1st change, then restart the gearman log client, check firewall rules, then merge the 2nd one | 02:03 |
| fungi | lifeless: sounds probable | 02:03 |
| fungi | lifeless: all fodder for a "pypi wall of shame" | 02:03 |
| *** sarob has quit IRC | 02:04 | |
| vijay | anteaya, fungi: Currently only the screen logs and the tempest logs from the devstack are uploaded by the test infrastructure. Is there a script which can be used to package more logs? | 02:04 |
| *** senk has joined #openstack-infra | 02:04 | |
| clarkb | vijay: devstack-gate copies the logs, and it copies a lot more than just screen and tempest logs | 02:05 |
| *** ryanpetrello has joined #openstack-infra | 02:06 | |
| *** talluri has joined #openstack-infra | 02:07 | |
| corvus | fungi: i'm going away for a bit; what to do with jenkins3/4 is up to you for now (merging that 1st change should be safe) | 02:08 |
| clarkb | vijay: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n334 is tehre | 02:08 |
| corvus | fungi: i'll be back in maybe 1.5-2 hours | 02:09 |
| vijay | clarkb: Wonderful. Thanks will have a look. | 02:09 |
| *** oubiwann has joined #openstack-infra | 02:09 | |
| anteaya | fungi: you are correct, I believe you have responded already, can we implement this going forward please? | 02:10 |
| anteaya | mikal harlowja do you know of a script that vijay can use for packaging more logs from third party testing? | 02:12 |
| harlowja | hmmm, not sure | 02:12 |
| anteaya | fair enough, I have to stand up and walk away now, vijay you can ping me in -neutron if you hit an obstacle or need more | 02:13 |
| *** senk has quit IRC | 02:13 | |
| anteaya | and thank you for being responsive | 02:13 |
| vijay | anteaya: sure. | 02:14 |
| *** ryanpetrello has quit IRC | 02:14 | |
| *** CaptTofu has quit IRC | 02:23 | |
| *** CaptTofu has joined #openstack-infra | 02:23 | |
| fungi | anteaya: i haven't had time to read the proposal yet for revamping the process, but based on what i've heard it sounds like there may be some missing pieces to the picture still, and we'll need to update/announce those changes to the process | 02:25 |
| *** dstanek has joined #openstack-infra | 02:35 | |
| anteaya | fungi: fair enough | 02:36 |
| *** nati_uen_ has quit IRC | 02:37 | |
| *** malini is now known as malini_afk | 02:39 | |
| *** amotoki has joined #openstack-infra | 02:41 | |
| *** vijay has quit IRC | 02:41 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add jenkins03 and 04 to most places https://review.openstack.org/65191 | 02:45 |
| *** weshay has joined #openstack-infra | 02:46 | |
| anteaya | for those interested in the 3rd party verification voting conversation, be sure to follow and weigh in here https://review.openstack.org/#/c/63478/5 | 02:54 |
| *** sandywalsh has quit IRC | 02:55 | |
| *** rhsu has joined #openstack-infra | 03:00 | |
| *** yamahata has quit IRC | 03:03 | |
| *** rhsu has left #openstack-infra | 03:03 | |
| harlowja | clarkb thx, making something similar @ https://review.openstack.org/#/c/65199/ | 03:09 |
| *** wenlock has quit IRC | 03:12 | |
| lifeless | fungi: was pinning just virtual env sufficient? | 03:13 |
| lifeless | fungi: and is there a bug for this? | 03:13 |
| harlowja | does https://review.openstack.org/#/c/65135/ make sense to u guys, it adds alot of venvs :( | 03:16 |
| openstackgerrit | lifeless proposed a change to openstack-infra/pypi-mirror: Pin virtualenv as 1.11 is bust. https://review.openstack.org/65201 | 03:19 |
| lifeless | SpamapS: ^ is there a bug ? | 03:20 |
| *** kraman has quit IRC | 03:23 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 03:29 |
| *** mlipchuk has quit IRC | 03:30 | |
| corvus | fohi | 03:30 |
| corvus | gah | 03:31 |
| corvus | fungi: hi | 03:31 |
| *** corvus is now known as jeblair | 03:31 | |
| jeblair | fungi: i think i can continue now | 03:31 |
| fungi | jeblair: hey | 03:32 |
| clarkb | jeblair: you are you again, I am up acpi says almost 1.5 hours of battery | 03:33 |
| clarkb | reality is probably closer to 1 | 03:33 |
| fungi | lifeless: yeah, i opened a bug today to collect the various affected projects in. just a sec and i can find it | 03:33 |
| jeblair | fungi: i see the 1st change merged; has the log gearman client restarted? | 03:33 |
| *** CaptTofu has quit IRC | 03:33 | |
| *** CaptTofu has joined #openstack-infra | 03:34 | |
| *** dcramer_ has quit IRC | 03:34 | |
| fungi | lifeless: https://launchpad.net/bugs/1266513 | 03:34 |
| clarkb | jeblair: ps indicates that it may have been | 03:35 |
| fungi | jeblair: i haven't checked yet, but i'll figure out where/what to determine | 03:35 |
| clarkb | jeblair: possible puppet did it for us, I didn't think it would beacuse you don't like that :) | 03:35 |
| jeblair | heh | 03:35 |
| openstackgerrit | lifeless proposed a change to openstack-infra/pypi-mirror: Pin virtualenv as 1.11 is bust. https://review.openstack.org/65201 | 03:35 |
| fungi | jenkins-log-client.yaml ends up consumed by a process on logstash.o.o, or deeper into the worker farm? | 03:36 |
| clarkb | fungi: on logstash.o.o, the gearman client daemon | 03:37 |
| clarkb | the -worker stuff happens on the farm | 03:37 |
| fungi | yeah, restarted 24 minutes after the config change merged, looks like | 03:38 |
| *** weshay has quit IRC | 03:38 | |
| fungi | Jan 7 03:09:53 logstash puppet-agent[3836]: (/Stage[main]/Log_processor::Client/Service[jenkins-log-client]) Triggered 'refresh' from 1 events | 03:39 |
| clarkb | that matches ps, so I think we are good on that front | 03:39 |
| * fungi has been trying in futility to get at least a little of the review backlog handled before passing out for the night | 03:40 | |
| clarkb | fungi: good luck (I feel bad for not helping much) | 03:40 |
| *** praneshp has quit IRC | 03:40 | |
| jeblair | the firewall on zuul has updated | 03:40 |
| jeblair | i reckon we should merge change#2 now | 03:41 |
| clarkb | sounds good | 03:42 |
| jeblair | done | 03:42 |
| *** nati_ueno has joined #openstack-infra | 03:43 | |
| *** krotscheck has joined #openstack-infra | 03:43 | |
| fifieldt | thank you fungi :) | 03:45 |
| fifieldt | now I wonder how badly this will break :D | 03:45 |
| *** dstanek has quit IRC | 03:47 | |
| clarkb | jeblair: I think we need to clean up the slave list on 03 and possibly 04 (checking that one now) | 03:47 |
| *** yamahata has joined #openstack-infra | 03:47 | |
| *** krotscheck has quit IRC | 03:47 | |
| clarkb | jeblair: it seems to have been pulled in with the xml copy | 03:47 |
| *** nati_ueno has quit IRC | 03:47 | |
| fungi | fifieldt: well, since it's called as a separate hook, it at least shouldn't break anything else | 03:47 |
| fifieldt | I hope so :) | 03:48 |
| fifieldt | hey, where's the best place to see the logs it spams to>? | 03:48 |
| jeblair | clarkb: ack, thx | 03:48 |
| clarkb | 04 has the same problem | 03:48 |
| clarkb | jeblair: should I manually delete the slaves or do we want to shutdown, then clean the xml? | 03:49 |
| jeblair | clarkb: if it's just a few, then manually delet sholud be fine | 03:49 |
| clarkb | jeblair: its a bunch, basically on slaves on the source jenkins | 03:49 |
| jeblair | clarkb: ok, let's shut down then | 03:50 |
| jeblair | clarkb: can you do that for both? | 03:50 |
| clarkb | yup | 03:50 |
| clarkb | jenkins is stopped, hacking xml on 04 now | 03:51 |
| *** yamahata has quit IRC | 03:52 | |
| fungi | fifieldt: it'll wind up dumped in the gerrit service log, which is only visible to those of us with admin control over the server. i can spot-check for failures/hits periodically and post them to paste.o.o until we're comfortable enough to undo the --dryrun | 03:54 |
| fifieldt | coolo | 03:55 |
| fifieldt | that would be nice | 03:55 |
| fifieldt | I really want to check that it's not giving false postives or negatives as much as posible | 03:55 |
| fifieldt | so a grep of the log for the appropriate bits in a couple days time would be nice | 03:55 |
| clarkb | jeblair: ithink the css may not have been correcteither, the webuis looked funny | 03:56 |
| clarkb | jeblair: can you look at that? | 03:56 |
| jeblair | clarkb: yes | 03:56 |
| jeblair | clarkb: oh it's not running | 03:57 |
| jeblair | clarkb: i'll look when you're done | 03:57 |
| *** fallenpegasus has joined #openstack-infra | 03:58 | |
| jeblair | nodepool is not running | 03:59 |
| clarkb | jeblair: :( I have claered the slaves lists on 03 and 04 | 04:00 |
| clarkb | jeblair: should I start 3 and 4? | 04:01 |
| jeblair | clarkb: yes | 04:01 |
| *** CaptTofu has quit IRC | 04:01 | |
| jeblair | clarkb: i think it shut down gracefully, maybe puppet did it | 04:01 |
| clarkb | 3 and 4 are starting | 04:02 |
| *** CaptTofu has joined #openstack-infra | 04:02 | |
| *** noorul has joined #openstack-infra | 04:02 | |
| noorul | fungi: hi | 04:03 |
| clarkb | jeblair: they are up | 04:03 |
| fungi | hi noorul | 04:03 |
| jeblair | clarkb: what's wrong with the css? | 04:03 |
| noorul | fungi: I see 2 +2s on https://review.openstack.org/#/c/64226/ | 04:03 |
| lifeless | fungi: alembic was a false positive due to a dead dns server | 04:03 |
| noorul | fungi: Did you miss approving it or something else is required? | 04:03 |
| fungi | lifeless: great | 04:03 |
| clarkb | jeblair: it actually lokos better now | 04:03 |
| clarkb | jeblair: no idea why it looked different before. perhaps local network issues loading css? | 04:04 |
| jeblair | maybe | 04:04 |
| fungi | noorul: yep! sorry, i missed hitting the approve | 04:04 |
| fungi | approved now | 04:05 |
| noorul | fungi: Thank you! | 04:05 |
| *** rcleere has joined #openstack-infra | 04:08 | |
| *** praneshp has joined #openstack-infra | 04:09 | |
| jeblair | i'm running a bunch of nodepool deletes in screen to deal with things nodepool may have missed during its downtime | 04:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Start using jenkins03 and 04 https://review.openstack.org/65192 | 04:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Add welcome_message.py to patchset-created trigger https://review.openstack.org/61898 | 04:11 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 04:13 |
| openstackgerrit | A change was merged to openstack-infra/config: Alter zookeeper params for Fedora. https://review.openstack.org/62645 | 04:14 |
| jeblair | the new config file is in place on nodepool | 04:14 |
| jeblair | and it's just started talking to jenkins03 | 04:14 |
| openstackgerrit | A change was merged to openstack-infra/config: Update solum devstack job to install packages from test-requirements.txt https://review.openstack.org/64226 | 04:14 |
| clarkb | jeblair: col | 04:15 |
| clarkb | *cool | 04:15 |
| *** fallenpegasus has quit IRC | 04:16 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 04:16 |
| *** fallenpegasus has joined #openstack-infra | 04:17 | |
| jeblair | it's launching nodes for them now | 04:18 |
| *** noorul has left #openstack-infra | 04:19 | |
| jeblair | 04 is going to get the first ones | 04:19 |
| jeblair | looks like credentials problem | 04:21 |
| *** fallenpegasus has quit IRC | 04:21 | |
| openstackgerrit | A change was merged to openstack-infra/config: Install tempest to re-use REST client code in solum functional tests https://review.openstack.org/64062 | 04:24 |
| openstackgerrit | A change was merged to openstack-infra/config: Add niceness gate for operations-guide https://review.openstack.org/64567 | 04:24 |
| *** dcramer_ has joined #openstack-infra | 04:25 | |
| *** fallenpegasus has joined #openstack-infra | 04:27 | |
| *** talluri has quit IRC | 04:27 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Add Storyboard puppet module https://review.openstack.org/65017 | 04:34 |
| *** ArxCruz has quit IRC | 04:36 | |
| jeblair | fungi, clarkb: jenkins03 and jenkins04 are both running jobs now | 04:37 |
| *** fallenpegasus has quit IRC | 04:38 | |
| * fungi cheers | 04:41 | |
| fungi | hopefully this bodes for greater stability the rest of the week | 04:42 |
| *** harlowja is now known as harlowja_away | 04:42 | |
| jeblair | fungi: yeah, should get nodes/master closer to 100, which is i think where performance starts to degrade | 04:42 |
| jeblair | fungi: clarkb was suggesting we sholud also move rax performance nodes into the gate (and increase our usage up to quota there) | 04:43 |
| jeblair | fungi: we might want to add one more jenkins master. :) | 04:43 |
| fifieldt | fungi, did anything break crazily in the first hour? :D | 04:43 |
| jeblair | fungi: anyway, lunch time here | 04:43 |
| fungi | fifieldt: not sure--i've been trying to knock out more reviews, and am about to pass out for the night. haven't heard anyone scream about anything broken yet though | 04:44 |
| fifieldt | oh, sorry fungi | 04:44 |
| fifieldt | please sleep well :) | 04:45 |
| fifieldt | no screams is good | 04:45 |
| fifieldt | btw - thanks to infra for git.openstack.org ... performs so much faster than github (now I have a connection fast enough to tell the difference) | 04:46 |
| fungi | fifieldt: that's great to hear | 04:49 |
| fifieldt | Receiving objects: 100% (41222/41222), 65.49 MiB | 9.35 MiB/s, done | 04:51 |
| fifieldt | again and again and again ;) | 04:51 |
| openstackgerrit | A change was merged to openstack-infra/config: Add gerritbot for #openstack-anvil channel https://review.openstack.org/53974 | 04:52 |
| openstackgerrit | A change was merged to openstack-infra/config: Add oslo.rootwrap to the integrated gate https://review.openstack.org/64677 | 04:52 |
| *** coolsvap has joined #openstack-infra | 04:59 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add libvirt-dev in anticipation of python-libvirt https://review.openstack.org/64611 | 05:00 |
| *** openstackgerrit has quit IRC | 05:01 | |
| *** openstackgerrit has joined #openstack-infra | 05:01 | |
| *** jasondotstar has quit IRC | 05:10 | |
| *** harlowja_away is now known as harlowja | 05:12 | |
| jog0 | dumb rootwrap question: rootwrap's first arg is its config file. doesn't that mean that someone could use rootwrap as a local privilage escillation by making up there own rootwrap.conf? | 05:13 |
| zaro | clarkb, jeblair, fungi : i think we are all set with review-dev.o.o. is there something you guys do for testing it before rolling to prod? | 05:15 |
| jog0 | dumb question answered: the config file arg is explicitly listed in the sudoers file | 05:18 |
| *** nati_ueno has joined #openstack-infra | 05:18 | |
| *** yamahata has joined #openstack-infra | 05:19 | |
| *** dstufft_ has joined #openstack-infra | 05:20 | |
| fungi | clarkb: jeblair: i shouldn't have looked at zuul's status... jobs on 03 and 04 failing with "java.lang.NoClassDefFoundError: Could not initialize class hudson.Util" https://jenkins03.openstack.org/job/gate-tempest-dsvm-neutron/46/console | 05:21 |
| *** dstufft has quit IRC | 05:22 | |
| jeblair | zaro: yes, quite a bit more testing | 05:23 |
| *** pmathews has joined #openstack-infra | 05:23 | |
| fungi | also "java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer" | 05:23 |
| *** blamar has quit IRC | 05:24 | |
| jeblair | fungi: hrm; we had to stop and restart jenkins04; i wonder if that could have caused something | 05:24 |
| jeblair | fungi: oh but that's on 03 | 05:24 |
| jeblair | it's not all jobs | 05:25 |
| fungi | i see similar issues for jobs run on both of them | 05:25 |
| fungi | https://jenkins03.openstack.org/job/gate-tempest-dsvm-postgres-full/57/console was about 8 minutes ago | 05:26 |
| clarkb | could my removal of the slaves list have acused that? | 05:27 |
| clarkb | I removed the stuff between <slaves></slaves> pretty sure the master computer is special in jenkins though and you just get it | 05:27 |
| clarkb | rather than needing it listed explicitly | 05:27 |
| jeblair | a lot of jobs ran on that one slave | 05:29 |
| jeblair | possibly many/all the failed ones | 05:29 |
| *** vipul has quit IRC | 05:29 | |
| jeblair | i wonder if the offline function failed | 05:29 |
| *** vipul has joined #openstack-infra | 05:30 | |
| *** cody-somerville has quit IRC | 05:30 | |
| *** zul has quit IRC | 05:30 | |
| *** saper_ has joined #openstack-infra | 05:31 | |
| *** medberry has joined #openstack-infra | 05:31 | |
| *** medberry has joined #openstack-infra | 05:31 | |
| *** zul has joined #openstack-infra | 05:31 | |
| *** cody-somerville has joined #openstack-infra | 05:32 | |
| *** devanand1 has joined #openstack-infra | 05:33 | |
| *** w_ has joined #openstack-infra | 05:33 | |
| zaro | jeblair: i think i've pretty much covered the basics. not sure what to target next but can get started on testing something else if you like. | 05:34 |
| jeblair | zaro: we need answers to at least all of the things in the etherpad | 05:35 |
| *** kraman has joined #openstack-infra | 05:35 | |
| zaro | clarkb: i've noticed that lots of methods to get master computer have been deprecated in the latest jenkins api. | 05:35 |
| *** cody-somerville has quit IRC | 05:36 | |
| *** oubiwann has quit IRC | 05:36 | |
| *** salv-orlando has quit IRC | 05:36 | |
| *** rwsu has quit IRC | 05:36 | |
| *** rockstar has quit IRC | 05:36 | |
| *** lifeless has quit IRC | 05:36 | |
| *** harlowja has quit IRC | 05:36 | |
| *** ociuhandu has quit IRC | 05:36 | |
| *** luis_ has quit IRC | 05:36 | |
| *** andreaf has quit IRC | 05:36 | |
| *** milki has quit IRC | 05:36 | |
| *** crank has quit IRC | 05:36 | |
| *** SEJeff_work has quit IRC | 05:36 | |
| *** crank has joined #openstack-infra | 05:36 | |
| *** rockstar has joined #openstack-infra | 05:36 | |
| *** luis_ has joined #openstack-infra | 05:36 | |
| *** milki has joined #openstack-infra | 05:36 | |
| *** lifeless has joined #openstack-infra | 05:36 | |
| *** kraman has quit IRC | 05:36 | |
| *** andreaf has joined #openstack-infra | 05:36 | |
| *** SEJeff_work has joined #openstack-infra | 05:36 | |
| *** cody-somerville has joined #openstack-infra | 05:36 | |
| *** rockstar has quit IRC | 05:36 | |
| *** rockstar has joined #openstack-infra | 05:36 | |
| *** rwsu has joined #openstack-infra | 05:36 | |
| *** cody-somerville has quit IRC | 05:37 | |
| *** cody-somerville has joined #openstack-infra | 05:37 | |
| clarkb | jeblair: ^ | 05:37 |
| jeblair | fungi: it looks like a bad node, possibly due to zuul having abroted its first build | 05:37 |
| jeblair | hrm, more than one | 05:38 |
| zaro | jeblair: ok. most have already been answered, rest just seem like decisions we need to make like which plugins to install, etc.. | 05:38 |
| zaro | jeblair: i'll dig into it some more and try to answer all the questions that need answering. | 05:39 |
| *** pleia2 has quit IRC | 05:39 | |
| *** sbadia has quit IRC | 05:39 | |
| *** olaph has quit IRC | 05:39 | |
| *** vladan has quit IRC | 05:39 | |
| *** saper has quit IRC | 05:39 | |
| *** akscram has quit IRC | 05:39 | |
| *** shardy has quit IRC | 05:39 | |
| *** sirushti has quit IRC | 05:39 | |
| *** devananda has quit IRC | 05:39 | |
| *** malini_afk has quit IRC | 05:39 | |
| *** _sirushti has joined #openstack-infra | 05:39 | |
| *** _sirushti is now known as sirushti | 05:40 | |
| *** vladan has joined #openstack-infra | 05:40 | |
| zaro | jeblair, clarkb : didn't know if you noticed: https://review-dev.openstack.org/monitoring | 05:40 |
| *** sbadia has joined #openstack-infra | 05:41 | |
| clarkb | zaro: I had not noticed, thanks | 05:42 |
| clarkb | zaro: that should be behind authentication thoug | 05:42 |
| jeblair | i think huge numbers of jobs are failing | 05:42 |
| clarkb | and only be visible to admins if that isn't already the case | 05:42 |
| jeblair | let's worry about gerrit later | 05:42 |
| clarkb | jeblair: because of the change? | 05:42 |
| zaro | clarkb: i'll look into that. | 05:43 |
| *** pleia2 has joined #openstack-infra | 05:43 | |
| jeblair | clarkb: i don't understand why, i'm talking about the thing fungi pointed out and i've been reporting on my investigations for the past 20 minutes | 05:43 |
| clarkb | jeblair: ya, I just wasn't sure if you had correlated the two | 05:44 |
| *** chandankumar has joined #openstack-infra | 05:44 | |
| *** rcleere has quit IRC | 05:45 | |
| *** fallenpegasus has joined #openstack-infra | 05:45 | |
| clarkb | https://issues.jenkins-ci.org/browse/JENKINS-19453 claims to have fixed something similar in 1.542 | 05:45 |
| *** malini has joined #openstack-infra | 05:46 | |
| fungi | i'm also seeing a jenkins-related failure on https://jenkins01.openstack.org/job/gate-config-layout/1825/console which ran on 01 | 05:46 |
| *** shardy has joined #openstack-infra | 05:46 | |
| *** akscram has joined #openstack-infra | 05:47 | |
| fungi | "Caused by: hudson.remoting.ChannelClosedException: channel is already closed" | 05:47 |
| jeblair | fungi: are you sure that wasn't an aborted job? | 05:47 |
| fungi | jeblair: shouldn't have been aborted. was at the head of its queue | 05:47 |
| jeblair | the gearman-plugin md5sum is different from jenkins02 and jenkins03 | 05:48 |
| clarkb | o_O /me looks at what versions are reported by jenkins | 05:48 |
| *** Ryan_Lane has joined #openstack-infra | 05:48 | |
| fungi | i only copied over the scp and zmq plugins from 02 to 03/04... i assume the gearman plugin was obtained elseways | 05:49 |
| clarkb | jeblair: 03 has 0.0.3 installed, 01 has 0.0.4.ad75b7e | 05:49 |
| clarkb | jeblair: I think we put 03 and 04 in shutdown mode, then update the gearman plugin to match 01/02 | 05:50 |
| jeblair | clarkb: i guess it needs a release | 05:50 |
| *** dkliban_ has joined #openstack-infra | 05:50 | |
| clarkb | then swing back around and possibly release the plugin | 05:50 |
| clarkb | ya that | 05:50 |
| jeblair | clarkb: sounds good, you want to shutdown? | 05:50 |
| clarkb | sure, doing that now | 05:50 |
| *** oubiwann has joined #openstack-infra | 05:50 | |
| jeblair | clarkb: i'll copiy | 05:51 |
| *** jcooley_ has joined #openstack-infra | 05:51 | |
| clarkb | 03 and 04 are now in shutdown mode | 05:51 |
| clarkb | there is one job on each server | 05:51 |
| clarkb | nope 3 is idle now, I am stopping it | 05:51 |
| *** sirushti has quit IRC | 05:52 | |
| *** lifeless has quit IRC | 05:52 | |
| *** rockstar has quit IRC | 05:52 | |
| *** dcramer_ has quit IRC | 05:52 | |
| *** dkliban has quit IRC | 05:52 | |
| *** jcooley has quit IRC | 05:52 | |
| *** jcooley_ is now known as jcooley | 05:52 | |
| clarkb | jeblair: 3 is stopped, feel free to restart when you have updated the plugin | 05:52 |
| fungi | the jobs were failing rapidly enough i would expect 03 and 04 to quiesce quickly | 05:53 |
| jeblair | 03 starting | 05:53 |
| clarkb | 04 is still running a job | 05:54 |
| *** dcramer_ has joined #openstack-infra | 05:54 | |
| *** llu has joined #openstack-infra | 05:54 | |
| jeblair | 03 reports 0.0.4.2.ad75b7e | 05:55 |
| clarkb | jeblair: that looks correct, I tyoped the .2. earleier and left it out | 05:56 |
| *** sirushti has joined #openstack-infra | 05:57 | |
| *** rockstar has joined #openstack-infra | 05:57 | |
| jeblair | clarkb: we can make sure the node running https://jenkins03.openstack.org/job/gate-config-layout/3/console is offlined and deleted correctly | 05:58 |
| *** gyee_ has quit IRC | 05:58 | |
| *** lifeless has joined #openstack-infra | 05:59 | |
| jeblair | clarkb: don't shut down 04 yet | 05:59 |
| clarkb | the node for the job you linked is offline | 05:59 |
| llu | hi, anyone knows the failure at http://logs.openstack.org/12/63912/1/gate/gate-tempest-dsvm-full/1cd8253/console.html? | 05:59 |
| clarkb | will hold off on 04, it is still running a job | 05:59 |
| *** talluri has joined #openstack-infra | 05:59 | |
| openstackgerrit | A change was merged to openstack-infra/config: Give Zuul file descriptor breathing room. https://review.openstack.org/64621 | 05:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Add requirements jobs for rally https://review.openstack.org/63743 | 06:00 |
| jeblair | clarkb: Offline due to Gearman request | 06:00 |
| jeblair | excellent | 06:00 |
| jeblair | clarkb: and now that node is gone | 06:00 |
| *** sandywalsh has joined #openstack-infra | 06:01 | |
| jeblair | clarkb: i'm doen with 04; feel free to restart whet it's ready | 06:02 |
| fungi | llu: that looks like the issue jeblair and clarkb are working on | 06:02 |
| clarkb | jeblair: ok | 06:02 |
| clarkb | jeblair: it seems to actually be running tempest properly, I guess I will just watch it and restart it when it is done | 06:02 |
| jeblair | clarkb: i was getting a list of offline nodes on 04 so i could nodepool delete them (i think they are from our earlier thing. | 06:03 |
| *** fallenpegasus has quit IRC | 06:04 | |
| *** fallenpegasus has joined #openstack-infra | 06:05 | |
| *** fallenpegasus has quit IRC | 06:05 | |
| *** fallenpegasus has joined #openstack-infra | 06:05 | |
| llu | fungi: ok. what should I do now? Give another approval to put it into the gate queue since there is no reverify no bug now, or wait until it is fixed? | 06:05 |
| *** fallenpegasus has quit IRC | 06:06 | |
| fungi | llu: i suspect it's safe to reapprove now, but you may want to wait a few more minutes while they confirm | 06:06 |
| dstufft_ | fungi: clarkb mordred whoever, how hard would it be to try a 1.11.1 RC in the next day or two to make sure everything is kosher with it before we release it? | 06:06 |
| dstufft_ | for virtualenv that is | 06:06 |
| *** dstufft_ is now known as dstufft | 06:07 | |
| lifeless | dstufft_: of virtualenv? | 06:07 |
| *** fallenpegasus has joined #openstack-infra | 06:07 | |
| lifeless | heh | 06:07 |
| clarkb | dstufft_: not too terrible, we should be able to grab a slave, upgrade to the rc then manually run a job that won't report to gerrit | 06:07 |
| jeblair | fungi: llu: should be ok now; | 06:08 |
| dstufft | assuming nobody objects in the pypa-dev mailing list, we're going to release 1.11.1 without pip 1.5.1 (normally they get released together) to solve the regressions that occured in virtualenv with 1.11 | 06:08 |
| dstufft | it'll still have 1.5 bundled that is | 06:08 |
| lifeless | dstufft: UI'm having to rollback to both 1.4.1 and 1.10.1 to work. | 06:08 |
| lifeless | dstufft: It's not clear to me that 1.5 works at all | 06:08 |
| clarkb | dstufft: though fungi may have the most bandwidth to help as the rest of us are conferencing | 06:08 |
| llu | jungi: jeblair: thanks for the info | 06:08 |
| clarkb | and that means he doesn't have much bw... | 06:08 |
| dstufft | lifeless: can you verify that https://github.com/pypa/pip/issues?milestone=6&state=open captures whatever issues you're having with pip? | 06:09 |
| fungi | dstufft: yeah, i may be spread thin, but the pip/virtualenv patches are part of what i'm trying to get nailed down anyway | 06:09 |
| fungi | dstufft: #1428 is the only one there i'm aware we're hitting | 06:11 |
| lifeless | fungi: and netaddtr | 06:11 |
| dstufft | lifeless: FWIW we can't really roll back to 1.4.1 in virtualenv 1.11.1, so the options are basically hold up 1.11.1 until 1.5.1 is ready (which we want to get the issues in that milestone fixed in that), or release a 1.11.1 w/o 1.5.1 and then do a 1.11.2 w/ 1.5.1 | 06:11 |
| lifeless | also we've found another bug with --enable-site-packages | 06:11 |
| *** nicedice has quit IRC | 06:11 | |
| fungi | lifeless: the netaddr et a issues aren't a bug, just us having to realign our requirements with the "pypi wall of shame" | 06:11 |
| clarkb | jeblair: version of gearman plugin on 04 looks correct | 06:11 |
| dstufft | lifeless: netaddr is externally hosted yea? That won't change | 06:11 |
| dstufft | lifeless: can you file a virtualenv issue for it? | 06:12 |
| *** nicedice has joined #openstack-infra | 06:12 | |
| jeblair | fungi: 03 and 04 are running the correct gearman plugin now; one job on 03 finished and the node was offlined and delted correctly | 06:12 |
| lifeless | dstufft: where are teh virtualenv issues ? | 06:12 |
| jeblair | fungi: so i think we've corrected the problem; we'll have more evidence is ~40mins | 06:12 |
| dstufft | github.com/pypa/virtualenv | 06:12 |
| lifeless | dstufft: do you have one about binaries being created in /usr/loval/bin rather than the venv (because the venv is missnig usr/bin/pip ? | 06:13 |
| *** oubiwann has quit IRC | 06:13 | |
| dstufft | lifeless: I think that's probably the pip/setupptols problem | 06:13 |
| dstufft | we switched how setuptools/pip got installed in 1.11 | 06:14 |
| dstufft | and when you use --system-site-packages, it erroneously sees setuptools and/or pip installed in the system site packages, and doesn't install it in the venv | 06:14 |
| jeblair | clarkb, fungi: i'll file a bug for reverifies | 06:14 |
| clarkb | jeblair: ok | 06:14 |
| fungi | jeblair: thanks | 06:14 |
| fungi | dstufft: lifeless: yep, that's the one we're seeing--the fix has been merged to trunk for a couple days, just waiting on the 1.11.1 release for it | 06:15 |
| lifeless | dstufft: https://github.com/pypa/virtualenv/issues/531 | 06:16 |
| fungi | i've personally tested the patch for the sitepackages issue and confirmed it fixes it for us where we have than enabled under tox | 06:16 |
| lifeless | dstufft: oh, ok ! ;) | 06:16 |
| dstufft | lifeless: oh https://github.com/pypa/virtualenv/issues/531 is a pip thing, and it's a purposeful change. warnings were emitted in 1.4, and in 1.5 the behavior changed | 06:17 |
| dstufft | netaddr doesn't host its files on PyPI | 06:17 |
| *** fallenpegasus has quit IRC | 06:17 | |
| lifeless | dstufft: yes, I would like to see them all copied into PyPI before disabling it | 06:18 |
| fungi | lifeless: the fix for netaddr is to talk them into uploading their releases to pypi (and the workaround is to whitelist them with cli options to pip) | 06:18 |
| lifeless | so we don't break /all the tooling/ | 06:18 |
| dstufft | lifeless: we can't copy anything for netaddr, they have to do it themselves | 06:18 |
| dstufft | it looks like they might have done it though | 06:18 |
| lifeless | dstufft: I don't understand why | 06:18 |
| dstufft | https://pypi.python.org/simple/netaddr/ | 06:18 |
| lifeless | dstufft: I'm getting that error right now... | 06:18 |
| dstufft | oh | 06:18 |
| dstufft | they don't have 0.7.6 uploaded | 06:18 |
| dstufft | 0.7.3 is the last version they uploaded to PyPI | 06:19 |
| fungi | netaddr moved *off* pypi for their newer releases | 06:19 |
| jeblair | bug 1266649 for rechecks | 06:19 |
| lifeless | dstufft: so has pypi disablined registering new releases that are broken like this? | 06:19 |
| fungi | i think it coincided with a change in upstream maintainers | 06:19 |
| jeblair | https://bugs.launchpad.net/openstack-ci/+bug/1266649 | 06:19 |
| lifeless | dstufft: I'm super worried about a moving target where we keep having to whitelist everything | 06:19 |
| lifeless | dstufft: or should we just disable the new 1.5 stuff so that we're not chasing a moving target? | 06:19 |
| dstufft | lifeless: so there's 3 classes of "things pip can download" | 06:20 |
| dstufft | there's A) stuff hosted on PyPI | 06:20 |
| clarkb | see https://github.com/drkjam/netaddr/issues/57 maybe we should all make noise on that issue | 06:20 |
| dstufft | B) Stuff linked directly from a /simple/foo/ page with a hash in the url like https://example.com/foo-1.0.tar.gz#md5=asdadasd | 06:21 |
| dstufft | C) Stuff either linked directly from /simple/foo/ page without a hash in the url, or stuff linked from a page linked from the /simple/foo/ page | 06:21 |
| fungi | lifeless: there's no global "disable" flag for the equivalent of --allow-unverified/--allow-insecure... you have to pass it for each package you want to whitelist (or drop back to running pip 1.4.1) | 06:21 |
| dstufft | netaddr falls in the C case for recent releases | 06:21 |
| lifeless | noise made | 06:21 |
| dstufft | which is a massive security problem | 06:21 |
| dstufft | because pip has no way to verify the file you're getting isn't being MITM'd | 06:22 |
| *** boris-42 has joined #openstack-infra | 06:22 | |
| lifeless | dstufft: yes, I get that. I object to pip making the change to require an option while PyPI still permits class C registrations to be made. | 06:22 |
| dstufft | (C is the case that --allow-unverified, or it's older name --allow-insecure allows) | 06:22 |
| lifeless | dstufft: because we're dealing with layers like tox -> pip or cron -> run-mirror -> pip | 06:22 |
| dstufft | B isn't a security issue, but a reliability issue, which is what --allow-external // --allow-all-external handles | 06:23 |
| dstufft | lifeless: so | 06:23 |
| dstufft | PyPI allowing them | 06:23 |
| dstufft | I fought really hard to disable them | 06:23 |
| lifeless | and having to mpoke trough all those layers whenever someone does C by mistake (or deliberately) is a massive PITA | 06:23 |
| dstufft | but distutils-sig wouldn't let me | 06:23 |
| dstufft | so the way pip works is the middle ground, authors can upload them, but even they won't be able to install them by default (assuming a pip >= 1.5) so they'll be incentivzed to upload | 06:23 |
| dstufft | but not forced | 06:23 |
| lifeless | dstufft: I'm there, I saw IIRC two peopel whinging | 06:23 |
| lifeless | :) | 06:24 |
| lifeless | anyhow, I'm whinging now, because the predictable outcome sucks | 06:24 |
| lifeless | dstufft: could pip be much more aggressive about complaining? E.g. say 'you need to file a bug with the upstream, what they are doing is insecure'. | 06:24 |
| fungi | _needs more shame_ | 06:25 |
| lifeless | I think we should do a fork of netaddr like we were going to do for mysql, TBH. | 06:25 |
| lifeless | it's simpler than poking holes in all the infra to override this | 06:25 |
| dstufft | lifeless: there's some pain in the short term, because we're purposely removing or making things opt in that used to just work, so there's some subset of packages that this will fail for. But long term this will be a lot better.. it just sucks short term :( | 06:25 |
| dstufft | lifeless: we can adjust the message sure! | 06:26 |
| dstufft | I wrote the message, I suck at wording things | 06:26 |
| fungi | lifeless: it's not just netaddr. so far i've also had to whitelist netifaces and lazr.restful in some projects too, and i'm sure we have more | 06:26 |
| lifeless | dstufft: doesn't C require specific effort though? So there muust be some broken tooling or docs somewhere, or this would be a class B situation instead. | 06:26 |
| *** fallenpegasus has joined #openstack-infra | 06:27 | |
| dstufft | lifeless: for *new* projects uploaded to PyPI (brand new, not new versions of an existing project) C requires specific effort -- it requires the author to either manually register an url through the PyPI web interface, or to purposely set the mode to the legacy option | 06:27 |
| dstufft | for existing projects, if we detected there was any versions that could only be isntalled externally, we left them in legacy mode | 06:28 |
| dstufft | for them, often times the authors don't realize it, because if they type ``setup.py register`` and don't type ``python setup.py upload`` they can get in this case | 06:28 |
| dstufft | or if they publish a version to their homepage, which is linked from their /simple/foo/ page but didn't upload it PyPI, then that falls in the C option again | 06:29 |
| fungi | yeah, i remember the -sig thread... it was disheartening. "we can't disable them, ever, because we can't be guaranteed we can ever reach them to let them know" | 06:29 |
| lifeless | yeah | 06:29 |
| lifeless | I'm worried about folk not realising | 06:29 |
| lifeless | netaddr has been broken for 3 months | 06:29 |
| dstufft | that last one was particularly troublesome before pip 1.4 when pip would install pre-releases by default | 06:30 |
| dstufft | because people would upload a pre-release to their project homepage for people to try, and pip's spider algorithm would find it on their homepage and install it | 06:30 |
| dstufft | fungi: FWIW Warehouse (PyPI 2.0) is going to eventually introduce a new API that isn't scraping the HTML pages of /simple/, and it will not support external links at all | 06:30 |
| fungi | looking forward to future sanity | 06:31 |
| *** fallenpegasus has quit IRC | 06:31 | |
| dstufft | lifeless: yea, We tried emailing what addresses we could find automatically, out of 30k projects or so there was about 5k total that we had to keep in legacy mode | 06:31 |
| lifeless | fungi: so I think we need an insecure-requirements.txt | 06:32 |
| lifeless | fungi: or something like that | 06:32 |
| dstufft | a good number of them were only for a "dev" requirement that was a git checkout or similar | 06:32 |
| dstufft | or for one old version that they forgot to upload | 06:32 |
| fungi | lifeless: i was thinking magic comment tags in global-requirements.txt, but i guess a separate file could work | 06:32 |
| lifeless | fungi: needs to be in requirements.txt itself | 06:33 |
| lifeless | fungi: doesn't it? Because we'll need pbr to auto enable this | 06:33 |
| dstufft | FWIW you might find my pypi-externals tool useful | 06:33 |
| dstufft | http://pypi-externals.caremad.io/netaddr/ | 06:33 |
| lifeless | fungi: otherwise every single one of our devs will eventually end up asking | 06:33 |
| dstufft | tells you what versions of a project are only available externally | 06:33 |
| fungi | lifeless: at least for the tox use case, it's enough to just whitelist them in the tox.ini | 06:33 |
| lifeless | fungi: my assumption is we want to keep 'pip install .' working | 06:33 |
| lifeless | fungi: we have tox, we have tripleo, we have devstack, we have the mirror | 06:34 |
| dstufft | FWIW --allow-insecure//--allow-unverified//--allow-external work in a requirements.txt file | 06:34 |
| lifeless | fungi: + folk doing 'pip install ... | 06:34 |
| lifeless | dstufft: what does that look like? | 06:34 |
| lifeless | netaddr>0.7.6 --allow-insecure --allow-external | 06:35 |
| lifeless | ? | 06:35 |
| fungi | lifeless: good point--i hadn't tried that yet to see if it was also breaking on a straight pip install. i'll need to sleep on it though (approaching 2am here) | 06:35 |
| lifeless | fungi: SHOOO | 06:35 |
| dstufft | lifeless: https://gist.github.com/dstufft/8295462 | 06:35 |
| fungi | that syntax may be easy enough to implement | 06:36 |
| lifeless | ok | 06:36 |
| lifeless | so we need the updater to look for -- lines too | 06:37 |
| *** fallenpegasus has joined #openstack-infra | 06:37 | |
| *** pmathews has quit IRC | 06:37 | |
| fungi | though the fact that pip lets a transitive dependency override your download security policy that way is a little odd | 06:37 |
| dstufft | fungi: what do you mean? | 06:38 |
| fungi | i ask pip to install package A which then pulls in unverified dependency B... somewhat surprising | 06:38 |
| dstufft | pip doesn't process requirements.txt files | 06:39 |
| *** harlowja has joined #openstack-infra | 06:39 | |
| dstufft | not unless you point to them with -r | 06:39 |
| dstufft | that's an openstackism that does that with pbr | 06:39 |
| fungi | ahh, good point | 06:39 |
| fungi | normally you'd have to explicitly ask pip to -r the file | 06:39 |
| clarkb | (because easy_install is broken for other reasons :) ) | 06:40 |
| fungi | anyway | 06:40 |
| * fungi sleeps | 06:40 | |
| dstufft | there's no way for a setup.py (which is all pbr really is, it just reads the requirements.txt in, which I hate but it's really just setup.py still) to force an unverified dependency | 06:40 |
| dstufft | (this is actually why 1.5 turned off dependency links too, because a random package from PyPI shouldn't be able to control where you get files from) | 06:41 |
| dstufft | fungi: night! | 06:41 |
| lifeless | dstufft: so we can't make pip install nova Just Work anymore? | 06:44 |
| jeblair | clarkb: nodepool has error: [Errno 24] Too many open files | 06:45 |
| dstufft | lifeless: don't use dependencies that aren't hosted on PyPI and it'll just work. You can't use stuff hosted elsewhere and have it just work without the user passing additional flags in a pip 1.5+ world | 06:46 |
| clarkb | jeblair: :/ | 06:46 |
| lifeless | dstufft: ok, so this is rather huge for us I think. | 06:46 |
| clarkb | jeblair: I propose we do the same thing I proposed to zuul | 06:46 |
| clarkb | jeblair: and ride it out until we can properly fix it | 06:46 |
| *** fallenpegasus has quit IRC | 06:46 | |
| jeblair | clarkb: we should bump mysql max conn too if needed | 06:47 |
| dstufft | lifeless: FWIW the process for deprecating things not hosted on PyPI went though was documented http://www.python.org/dev/peps/pep-0438/ | 06:47 |
| dstufft | documented here* | 06:47 |
| clarkb | jeblair: want me to write the init script change? | 06:47 |
| jeblair | clarkb: yes, thanks | 06:47 |
| lifeless | dstufft: yes, no problem there - again, I'm in the -sig | 06:48 |
| *** fallenpegasus has joined #openstack-infra | 06:48 | |
| lifeless | dstufft: my complaint is that we, as the nova authors, can't decide about our dependencies. | 06:49 |
| lifeless | dstufft: so we can't shield our many users from problems *we didn't create* | 06:49 |
| jeblair | nodepol just passed the 1M node mark | 06:50 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Bump nodepool's file descriptor limit. https://review.openstack.org/65216 | 06:50 |
| clarkb | jeblair: ^ | 06:50 |
| clarkb | jeblair: woah | 06:50 |
| * clarkb looks into puppet-mysql for conn limits | 06:50 | |
| dstufft | lifeless: well you decide what you depend on, so the future looking solution is don't depend on things that don't host their releases on PyPI. That doesn't exactly help the current situation but I don't see a better way currently to move to the desired outcome | 06:53 |
| *** reed has quit IRC | 06:53 | |
| lifeless | so we're back at forking some set of projects | 06:54 |
| clarkb | jeblair: what value do you want for mysqld max_connections? | 06:56 |
| dstufft | lifeless: Of course we're open to suggestions, but anything I can think of just kicks the can down the road, either we provide a way for projects like nova to remove the pain of hosting externally, and projects like netaddr have little incentive to change (the netaddr issue was opened 3 months ago with no movement) in which case we're just shuffling deck chairs on the titantic | 06:57 |
| dstufft | until the point we actually do cut it off, at which point we're basically where we are today, sans maybe a few extra libraries | 06:57 |
| dstufft | forking might be the best solution for openstack, until netaddr (and any other projects) start hosting on PyPI | 06:57 |
| dstufft | or if they don't start hosting on PyPI, you may wish to their maintain it yourself, or find a new library | 06:58 |
| jeblair | clarkb: == to open file limit | 06:58 |
| clarkb | jeblair: ok | 06:58 |
| dstufft | to either maintain* | 06:59 |
| lifeless | dstufft: so, I would really like to reopen the -sig thing and make class C things flat impossible. Now. | 06:59 |
| lifeless | dstufft: that provides plenty of incentive for folk to register. | 06:59 |
| jeblair | lifeless: ++ the alternative sounds untenable | 07:00 |
| lifeless | folk that want to host separately can advise their users to add whatever extra pip index url their stuff *is* available at | 07:00 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Bump mysql connection limit for nodepool. https://review.openstack.org/65218 | 07:01 |
| dstufft | lifeless: I would be supportive of that, I hate the fact that class C exists at all, class B annoys me but I can see _some_ use case for it (it still requires a flag, but there's a --allow-all-external flag so it's not separate for each package) | 07:01 |
| lifeless | either way users need to add some options to their pip invocation | 07:01 |
| clarkb | jeblair: ^ there is the change | 07:01 |
| dstufft | lifeless: an extra index can be as simple as a directory of tarballs if used with --find-links, so it's not like setting up your own mirror is hard | 07:01 |
| dstufft | "own index" | 07:01 |
| clarkb | lifeless: ya, I think forking is probably the most direct and sane path to working | 07:02 |
| dstufft | lifeless: really PyPI could just allow the existing "legacy mode" applications to stay legacy, and just prevent anyone from switching to legacy mode ever again | 07:02 |
| clarkb | or, we try to advertise the state of broken, apologize profusely and actively work to change deps | 07:03 |
| lifeless | dstufft: yes, I'm very worried about accidental switches to this mode for deps we choose | 07:03 |
| clarkb | should be simple to register netaddr-working on pypi and upload an sdist whenever upstream releases a new version >_> | 07:03 |
| *** praneshp has quit IRC | 07:04 | |
| dstufft | lifeless: http://d.stufft.io/image/1u3v1W3S000e | 07:04 |
| dstufft | first number is the number of projects set to the legacy mode, second number is the total number of projects | 07:05 |
| dstufft | (if I remember correctly, a signifcant portion of the first number are projects whose only external version is a ==dev version) | 07:06 |
| lifeless | clarkb: we could even automate it - openstack-$name-working | 07:06 |
| clarkb | ya | 07:06 |
| clarkb | we definitely could with some of our existing jobs even as they just wget sdist, push sdist | 07:07 |
| *** fallenpegasus has quit IRC | 07:07 | |
| lifeless | bbiab | 07:07 |
| dstufft | lifeless: clarkb I am sorry for this pain though :( I promise things will be much more stable in the long run once we clear out the techincal debt | 07:08 |
| *** masayukig has joined #openstack-infra | 07:11 | |
| *** AJaeger has joined #openstack-infra | 07:18 | |
| *** noorul has joined #openstack-infra | 07:19 | |
| noorul | hi | 07:19 |
| noorul | Is there any infra changes going on now? | 07:20 |
| *** talluri has quit IRC | 07:20 | |
| *** rcarrillocruz has joined #openstack-infra | 07:25 | |
| *** loq_mac has joined #openstack-infra | 07:26 | |
| *** jerryzhao has quit IRC | 07:26 | |
| anteaya | noorul: yes, more jenkins are coming online | 07:31 |
| anteaya | noorul: why do you ask? | 07:31 |
| *** luis_ has quit IRC | 07:31 | |
| *** StevenK has quit IRC | 07:32 | |
| *** jheiss_ has quit IRC | 07:34 | |
| noorul | anteaya: I think I found the problem | 07:38 |
| noorul | anteaya: http://logs.openstack.org/06/65206/1/check/gate-solum-devstack-dsvm/3a53145/logs/screen-c-vol.txt.gz | 07:38 |
| noorul | anteaya: I asked because I see that feedback from Jenkins is taking too much of time | 07:39 |
| *** StevenK has joined #openstack-infra | 07:39 | |
| *** loq_mac has quit IRC | 07:40 | |
| *** loq_mac has joined #openstack-infra | 07:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Bump nodepool's file descriptor limit. https://review.openstack.org/65216 | 07:41 |
| openstackgerrit | A change was merged to openstack-infra/config: Bump mysql connection limit for nodepool. https://review.openstack.org/65218 | 07:41 |
| *** loq_mac has quit IRC | 07:43 | |
| *** harlowja is now known as harlowja_away | 07:45 | |
| *** loq_mac has joined #openstack-infra | 07:45 | |
| *** mlipchuk has joined #openstack-infra | 08:05 | |
| *** mlipchuk has left #openstack-infra | 08:06 | |
| *** chandankumar has quit IRC | 08:06 | |
| StevenK | Should I be concerned that my new patch hasn't hit zuul's status page? | 08:07 |
| clarkb | StevenK: not really, zuul is bogged down dealing with the existing queues | 08:07 |
| clarkb | StevenK: notice the event queue on the status page number | 08:07 |
| clarkb | StevenK: it is non zero which means zuul has unprocessed events | 08:08 |
| StevenK | clarkb: Status page number? | 08:08 |
| clarkb | once that goes to 0 your change should be in the status page and if it isn't then you worry | 08:08 |
| clarkb | StevenK: under queue lengths on the status page | 08:08 |
| StevenK | clarkb: Oooh. '38 events' meaning zuul is behind by that much. | 08:09 |
| *** fallenpegasus has joined #openstack-infra | 08:10 | |
| clarkb | StevenK: yup I bet one of those is the event your change generated | 08:10 |
| StevenK | clarkb: I think that's a fairly safe bet, so I'll keep my money. | 08:10 |
| *** AJaeger has left #openstack-infra | 08:11 | |
| *** yolanda has joined #openstack-infra | 08:11 | |
| *** talluri has joined #openstack-infra | 08:12 | |
| *** mlipchuk has joined #openstack-infra | 08:13 | |
| *** jamielennox has joined #openstack-infra | 08:17 | |
| *** llu has left #openstack-infra | 08:19 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use performance rax nodes in the gate https://review.openstack.org/65236 | 08:22 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove devstack-precise-check rax images https://review.openstack.org/65237 | 08:22 |
| anteaya | noorul: okay, I looked at the file you linked and I don't see where it is saying Jenins is taking too much time | 08:22 |
| *** hashar has joined #openstack-infra | 08:23 | |
| *** chandankumar has joined #openstack-infra | 08:23 | |
| jeblair | clarkb: ^ | 08:23 |
| anteaya | noorul: but if this answers why the test is failing for you, that is fine | 08:23 |
| *** afazekas has joined #openstack-infra | 08:25 | |
| clarkb | jeblair: those changes lgtm | 08:25 |
| *** afazekas has quit IRC | 08:26 | |
| *** afazekas has joined #openstack-infra | 08:27 | |
| *** flaper87|afk is now known as flaper87 | 08:29 | |
| *** chandankumar has quit IRC | 08:29 | |
| jeblair | clarkb: we have about another hour, i think that's pushing it for watching the first image build, etc, so i think we can wait to approve later | 08:31 |
| *** afazekas has quit IRC | 08:31 | |
| clarkb | jeblair: wfm | 08:32 |
| *** llu has joined #openstack-infra | 08:36 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:37 | |
| *** dizquierdo has joined #openstack-infra | 08:38 | |
| *** fallenpegasus has quit IRC | 08:39 | |
| *** fallenpegasus has joined #openstack-infra | 08:40 | |
| *** coolsvap has quit IRC | 08:41 | |
| *** chandankumar has joined #openstack-infra | 08:42 | |
| *** blamar has joined #openstack-infra | 08:44 | |
| *** jcoufal has joined #openstack-infra | 08:44 | |
| *** afazekas has joined #openstack-infra | 08:46 | |
| *** saschpe_ has joined #openstack-infra | 08:49 | |
| *** saschpe has quit IRC | 08:50 | |
| *** nati_ueno has quit IRC | 08:52 | |
| *** rossella_s has joined #openstack-infra | 08:55 | |
| noorul | Any idea why am I getting this error | 08:55 |
| noorul | http://logs.openstack.org/06/65206/1/check/gate-solum-devstack-dsvm/d1cbb27/logs/screen-c-vol.txt.gz | 08:55 |
| *** loq_mac has quit IRC | 08:56 | |
| *** coolsvap has joined #openstack-infra | 08:59 | |
| clarkb | StevenK: https://bugs.launchpad.net/nova/+bug/1228977 | 09:03 |
| clarkb | its filed against nova, cloud archive, and the libvirt package | 09:04 |
| *** derekh has joined #openstack-infra | 09:04 | |
| clarkb | noorul: no idea | 09:05 |
| *** jpich has joined #openstack-infra | 09:05 | |
| anteaya | is this a pbr issue? http://logs.openstack.org/06/65206/1/check/gate-solum-devstack-dsvm/d1cbb27/console.html#_2014-01-07_08_48_06_780 | 09:06 |
| *** mancdaz_away is now known as mancdaz | 09:06 | |
| *** johnthetubaguy has joined #openstack-infra | 09:10 | |
| *** fallenpegasus has quit IRC | 09:10 | |
| *** yassine has joined #openstack-infra | 09:18 | |
| anteaya | noorul: if I understand correctly lifeless and clarkb think the issue is an import error, for some reason oslo.config is not available: http://logs.openstack.org/06/65206/1/check/gate-solum-devstack-dsvm/d1cbb27/console.html#_2014-01-07_08_53_23_195 | 09:18 |
| lifeless | http://logs.openstack.org/06/65206/1/check/gate-solum-devstack-dsvm/d1cbb27/console.html#_2014-01-07_08_50_00_727 reckons it installed it | 09:19 |
| *** fallenpegasus has joined #openstack-infra | 09:22 | |
| openstackgerrit | sahid proposed a change to openstack/requirements: Tox fails to build environment because of MySQL-Python https://review.openstack.org/62028 | 09:23 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Increase hpcloud ssh timeout to 180 https://review.openstack.org/65246 | 09:23 |
| *** fallenpegasus2 has joined #openstack-infra | 09:25 | |
| *** fallenpegasus has quit IRC | 09:25 | |
| *** pblaho has joined #openstack-infra | 09:33 | |
| *** fallenpegasus2 has quit IRC | 09:34 | |
| *** Ryan_Lane has quit IRC | 09:37 | |
| noorul | anteaya, lifeless Thank yo for looking | 09:41 |
| hashar | good morning | 09:41 |
| *** michchap_ has quit IRC | 09:46 | |
| noorul | Could this (https://review.openstack.org/#/c/64226/) be the problem ? | 09:46 |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 09:47 | |
| *** michchap has joined #openstack-infra | 09:47 | |
| *** pblaho has left #openstack-infra | 09:48 | |
| *** pblaho has joined #openstack-infra | 09:48 | |
| *** lcestari has joined #openstack-infra | 09:49 | |
| noorul | I don't think so. Strange that import is failing | 09:50 |
| *** fifieldt has quit IRC | 09:51 | |
| *** chandankumar_ has joined #openstack-infra | 10:01 | |
| *** jooools has joined #openstack-infra | 10:06 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 10:07 | |
| *** matel has joined #openstack-infra | 10:22 | |
| *** masayukig has quit IRC | 10:24 | |
| *** saschpe_ has quit IRC | 10:32 | |
| *** tma996 has joined #openstack-infra | 10:46 | |
| *** rossella_s_ has joined #openstack-infra | 10:49 | |
| *** CaptTofu has quit IRC | 10:50 | |
| *** CaptTofu has joined #openstack-infra | 10:50 | |
| *** jroovers has joined #openstack-infra | 10:52 | |
| *** rossella_s has quit IRC | 10:53 | |
| *** rossella_s_ is now known as rossella_s | 10:53 | |
| *** haypo has joined #openstack-infra | 10:54 | |
| *** yassine has quit IRC | 10:55 | |
| haypo | hi. it's still not possible to specific Python 3-only dependencies in requirements.txt. jd__ proposed a patch: https://review.openstack.org/#/c/58770/ | 10:55 |
| haypo | can someone please look at its patch? | 10:55 |
| *** coolsvap has quit IRC | 10:55 | |
| haypo | FYI i updated the https://wiki.openstack.org/wiki/Python3 page to add more recent information | 10:56 |
| jd__ | mordred: ^^^ | 10:57 |
| jd__ | dhellmann: ^^^ | 10:57 |
| *** yaguang has quit IRC | 10:58 | |
| *** yamahata has quit IRC | 11:08 | |
| *** jroovers has quit IRC | 11:09 | |
| chmouel | haypo: it would be nice to be able to specify a supported distro syntax directly in pip {test-}requirements.txt | 11:10 |
| haypo | chmouel: what is a "distro syntax"? | 11:10 |
| chmouel | haypo: sorry i mean python version | 11:11 |
| haypo | chmouel: i asked pip authors if they plan to support distlib labels like "[os.name == 'nt']" and the answer is no | 11:11 |
| haypo | chmouel: pip doesn't want to support that | 11:11 |
| chmouel | haypo: and python versions? | 11:11 |
| haypo | chmouel: neither | 11:11 |
| chmouel | haypo: ok | 11:12 |
| chmouel | haypo: i guess the tox.inis would need to be updated to support python3-requirements as well, right? | 11:13 |
| dstufft | FWIW (as a pip developer) that was a soft no from me. I don't personally think it makes a ton of sense for projects other than pbr based one because pbr is crazy and reads requirements.txt files inside of a setup.py :) Marcus or Paul might feel differently | 11:14 |
| dstufft | (Marcus and Paul being the other major pip devs) | 11:14 |
| haypo | dstufft: ah, you're here :-) hi | 11:14 |
| haypo | dstufft: i was trying to transfer you emails to chmouel | 11:14 |
| dstufft | haypo: I'm everywhere! | 11:14 |
| haypo | dstufft: the PEP 426 is still a draft, i don't think that pip is going to support markers in a near future, but requirements.txt is blocking me right now | 11:15 |
| haypo | dstufft: jd__'s option is simple and you may use something else later if pip evolves | 11:15 |
| haypo | i don't understand why https://review.openstack.org/#/c/58770/ is still stuck since the end of november, the latest patch is just one line !? | 11:16 |
| haypo | jd__: i don't understand how a single line is enough to support python3 dependencies. you also modified pbr? | 11:17 |
| jd__ | haypo: the original patch was a bit more invasive, and I split it into what it is now and https://review.openstack.org/#/c/63236/ | 11:17 |
| jd__ | I already added you to these reviews | 11:17 |
| haypo | jd__: ah ok :) | 11:17 |
| *** coolsvap has joined #openstack-infra | 11:17 | |
| dstufft | haypo: pip supports markers inside of a Wheel right now, and has since pip gained Wheel support, setuptools doesn't really (it sort of does) but it supports python so you can replicate it in there. afaik openstack needs this because you don't write a setup.py really, you use pbr to extract from a requirements.txt. | 11:19 |
| dstufft | So the pbr patch would likely implement markers itself, process them, and dynamically add things to install_requires as if you wrote some code in setup.py to dynamically add them based on python version (or whatever) | 11:19 |
| dstufft | What I'd personally need for markers in a requirements.txt is a solid use case outside of pbr, because pbr is the wierdo and the one "doing it wrong" IMO -- unless I misunderstand how you're planning on using them :) | 11:20 |
| haypo | dstufft: for markers, it was suggested to write "mox [sys.version_info < (3,)]" and "mox3 [sys.version_info >= (3,)]" in requirements.txt | 11:21 |
| openstackgerrit | Soren Hansen proposed a change to openstack-infra/jenkins-job-builder: Make it possible to pass an SSH agent to jobs https://review.openstack.org/65259 | 11:21 |
| *** CaptTofu has quit IRC | 11:21 | |
| haypo | dstufft: (i don't know if it's the correct syntax for markers to check for python major version, python 2 and python 3) | 11:22 |
| haypo | dstufft: "pip supports markers inside of a Wheel right now" i need something different. i don't want to install mox on Python 3, because it only supports Python 2 | 11:23 |
| haypo | dstufft: in setuptools, it's the install_requires parameter of the setup() function | 11:25 |
| haypo | i don't needs markers, just the ability to tag "this dependency is only for python 2" and "this one is only for python 3" | 11:25 |
| haypo | (others are for python 2 & python 3) | 11:25 |
| dstufft | haypo: Can you open an issue with github.com/pypa/pip ? I'm going to have to go soonish to take my daughter to the bus stop, and I'd like to get Marcus's and Paul's opinion too | 11:26 |
| *** rfolco has joined #openstack-infra | 11:26 | |
| haypo | jd__, dstufft : https://github.com/pypa/pip/issues/1433 | 11:31 |
| dstufft | haypo: thanks | 11:33 |
| dstufft | I guess I'm not leaving heh, went outside and it's too damn cold, we're keeping her home | 11:33 |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/nodepool: Support nodes with launch condition https://review.openstack.org/65261 | 11:34 |
| *** CaptTofu has joined #openstack-infra | 11:37 | |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/nodepool: Support nodes with launch condition https://review.openstack.org/65261 | 11:37 |
| *** jroovers has joined #openstack-infra | 11:42 | |
| jd__ | haypo: great | 11:44 |
| *** roeyc has joined #openstack-infra | 11:44 | |
| *** SergeyLukjanov has quit IRC | 11:44 | |
| *** yolanda has quit IRC | 11:53 | |
| *** yolanda has joined #openstack-infra | 11:53 | |
| *** coolsvap has quit IRC | 11:58 | |
| *** CaptTofu has quit IRC | 12:06 | |
| *** pblaho has quit IRC | 12:09 | |
| *** SergeyLukjanov has joined #openstack-infra | 12:14 | |
| *** hashar has quit IRC | 12:14 | |
| *** talluri has quit IRC | 12:19 | |
| chmouel | isn't that the gate a bit broken right now? | 12:25 |
| chmouel | or just super slow | 12:25 |
| sdague | chmouel: so all nova py 27 unit tests are failing | 12:25 |
| sdague | there was a bug for that on the list | 12:26 |
| chmouel | sdague: oh thanks didn't see that | 12:26 |
| *** yassine has joined #openstack-infra | 12:26 | |
| sdague | I don't know the other issues in there | 12:26 |
| chmouel | sdague: is somebody on https://bugs.launchpad.net/nova/+bug/1266711 ? | 12:27 |
| sdague | not as far as I know | 12:29 |
| sdague | I just woke up and was just making sure the bug had all the details that I'd seen people post in IRC | 12:30 |
| chmouel | sdague: okay, i can try to look at it but i haven't touched to nova since i think the bexar version | 12:31 |
| *** b3nt_pin has joined #openstack-infra | 12:33 | |
| *** b3nt_pin is now known as beagles | 12:33 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add bug for nova libvirt failure https://review.openstack.org/65266 | 12:35 |
| *** salv-orlando has joined #openstack-infra | 12:35 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add bug for nova libvirt failure https://review.openstack.org/65266 | 12:40 |
| chmouel | sdague: it would be nice to have the elastic recheck page http://status.openstack.org/elastic-recheck/ directly in http://status.openstack.org but not sure who takes care of this page | 12:47 |
| *** heyongli has joined #openstack-infra | 12:47 | |
| sdague | chmouel: agreed, did you see my whole thread on that? :) | 12:47 |
| chmouel | sdague: oh no,i haven't will read it | 12:48 |
| sdague | honestly we need the e-r page a bit more functional first, it's top of my list, which is why I'm trying to not actually chase gate bugs as it will keep me from doing that | 12:48 |
| chmouel | sdague: yeah agreed | 12:49 |
| *** weshay has joined #openstack-infra | 12:52 | |
| *** kruskakli has joined #openstack-infra | 12:52 | |
| *** smarcet has joined #openstack-infra | 12:53 | |
| anteaya | fungi once you are around can you advise kruskakli if there is any way he can manually sign into his tail-f account and manually clean up the -1 votes still outstanding? | 12:55 |
| kruskakli | also, is this the place to ask about this error when using tox: https://bitbucket.org/pypa/setuptools/issue/129/assertionerror-egg-info-pkg-info-is-not-a | 12:56 |
| *** yaguang has joined #openstack-infra | 12:56 | |
| kruskakli | it started to occur 2-January | 12:57 |
| anteaya | kruskakli: let's focus on one thing at a time please | 12:57 |
| anteaya | kruskakli: these patches need the -1 votes removed: http://paste.openstack.org/show/60626/ | 12:57 |
| kruskakli | anteaya: sure, but that error caused the -1 votes in the end... :-) | 12:57 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Mark resolved bugs https://review.openstack.org/65157 | 12:58 |
| *** lukego has joined #openstack-infra | 12:59 | |
| anteaya | kruskakli: no, the -1 votes occured because your system was unstable and shouldn't have had voting turned on in the first place | 12:59 |
| anteaya | also when posting to the mailing list, it is helpful if one topic is addressed per thread | 12:59 |
| anteaya | so addressing an error is best done in a support channel for that error | 13:00 |
| anteaya | in this case it is a development error and asking in #openstack-dev is a good first step | 13:00 |
| kruskakli | anteaya: sure, I'm trying to find my ways around here | 13:00 |
| anteaya | kruskakli: I'm glad to hear that | 13:00 |
| *** nosnos has quit IRC | 13:00 | |
| anteaya | and hope that what I am offering is useful information | 13:01 |
| kruskakli | anteaya: absolutely, I'm very greatful :-) | 13:01 |
| *** nosnos has joined #openstack-infra | 13:01 | |
| anteaya | I wonder if we should see a level of famialiryt with openstack and its processes prior to granting 3rd party testing accounts | 13:01 |
| *** afazekas has quit IRC | 13:02 | |
| anteaya | since having a third party testing account randomly voting because it is broken has a very bad effect on the community as a whole | 13:02 |
| anteaya | and if you aren't familiar with the openstack processes, you wouldn't know that | 13:02 |
| kruskakli | mm...the openstack-environment has been kind of overwhelming | 13:02 |
| anteaya | just thinking out loud here | 13:02 |
| anteaya | kruskakli: yes, I can understand that | 13:03 |
| anteaya | it is for most people | 13:03 |
| anteaya | kruskakli: do you mind if I ask, what is the motivation for having a 3rd party testing account if you aren't familiar with openstack processes? | 13:03 |
| kruskakli | I for one could settle for manual voting, based on the outcome of my tests | 13:04 |
| *** dkranz has joined #openstack-infra | 13:04 | |
| *** CaptTofu has joined #openstack-infra | 13:04 | |
| anteaya | kruskakli: some 3rd party testing accounts have the results reported internally with a manual process to forward those results to gerrit | 13:04 |
| kruskakli | anteaya: we had a 'driver' for our NCS product developed, that we want to maintain, however we are a small company and it landed in my lap... | 13:05 |
| anteaya | did your company develop the driver? | 13:05 |
| *** nosnos has quit IRC | 13:06 | |
| *** hashar has joined #openstack-infra | 13:07 | |
| kruskakli | anteaya: a contractor (a former colleque) | 13:07 |
| kruskakli | + some inhouse work | 13:07 |
| anteaya | who wrote the tests you are running? | 13:08 |
| anteaya | as in how do you know what to test? | 13:08 |
| kruskakli | the test was written by the contractor, I wrote the automation harness to make it fit with Jenkins | 13:09 |
| *** yamahata has joined #openstack-infra | 13:10 | |
| anteaya | this just sounds really fragile | 13:11 |
| anteaya | what happens if something changes and the tests you are running need to be adjusted to fit the change? | 13:11 |
| *** pblaho has joined #openstack-infra | 13:11 | |
| kruskakli | anteaya: then I have to dig into it, I also have easy ways of contacting the contractor if necessary | 13:12 |
| kruskakli | it's an old friend | 13:12 |
| anteaya | just so you know what you are going to have to expect | 13:12 |
| anteaya | so this landed on your lap | 13:13 |
| kruskakli | anteaya: yup | 13:13 |
| anteaya | and you have no prior knowledge of openstack | 13:13 |
| *** johnthetubaguy has quit IRC | 13:13 | |
| anteaya | is setting up the third party testing the first time you have interacted with anything openstack? | 13:13 |
| kruskakli | anteaya: I've played around with it a year ago or so | 13:13 |
| sdague | fungi: something seems odd in the zuul allocator, where when it's allocating dsvm nodes it's not going top to bottom. So you'll have jobs towards the tail of the gate that have all their resources, but towards the top that are still waiting on nodes | 13:14 |
| kruskakli | but i'm really just running the unit tests of the driver, but toward an NCS system instead of a dummy sink | 13:14 |
| anteaya | kruskakli: from your perspective that is how it will seem | 13:15 |
| anteaya | what you are doing is conveying information that the rest of the openstack community needs to know is trustworthy | 13:15 |
| anteaya | since the information is relied upon by those making decisions | 13:16 |
| sdague | kruskakli: so one other thing to realize, because of the amount of upstream code openstack depends on, that Jan 2nd event (which I think was the new pip release) is something you'll see at least once a month, where some upstream piece breaks an interface on us | 13:16 |
| anteaya | and -1 verification votes means that patch might not get reviewed | 13:16 |
| sdague | so best to code defensively around things like that | 13:16 |
| kruskakli | sdague: sure, so that's I would be fine doing manual voting | 13:17 |
| sdague | kruskakli: yep | 13:17 |
| sdague | that's what smokestack did for a long time | 13:17 |
| sdague | it queued a bunch of votes on their side, and dan prince would manually verify them and put them through | 13:17 |
| *** thomasem has joined #openstack-infra | 13:18 | |
| *** dcramer_ has quit IRC | 13:19 | |
| *** amotoki has quit IRC | 13:19 | |
| kruskakli | I guess I just can clear out the 'gerrit approve' commands in the gerrit config in the jenkins plugin to disable any voting? | 13:19 |
| *** talluri_ has joined #openstack-infra | 13:19 | |
| *** CaptTofu has quit IRC | 13:20 | |
| *** CaptTofu has joined #openstack-infra | 13:20 | |
| fungi | kruskakli: anteaya: just skimming scrollback at the moment (was up seriously late working), but you should be able to leave a vote of 0 to remove the -1... just has to be done via the ssh api, can't log into that account using the webui | 13:20 |
| anteaya | fungi: very good | 13:21 |
| fungi | sdague: it wasn't doing that last night. very odd | 13:21 |
| anteaya | kruskakli: that might be a start, but keep in mind this account now has voting disabled | 13:21 |
| sdague | fungi: yeh, looks like it started about 6 or 7 hours ago | 13:21 |
| anteaya | fungi: and yes, I understand, thanks for the new jenkinses | 13:21 |
| kruskakli | fungi: thanks, any example of how to use ssh for this purpose? | 13:22 |
| anteaya | kruskakli: I can help you | 13:22 |
| fungi | sdague: it also doesn't seem to be doing it now--did you catch it in the middle of recalculating a reset | 13:22 |
| *** dkliban_ has quit IRC | 13:22 | |
| kruskakli | anteaya: aha, so I can't clear out those votes by voting 0 then? | 13:22 |
| sdague | fungi: yes | 13:22 |
| anteaya | kruskakli: have you read this page? http://ci.openstack.org/third_party.html | 13:22 |
| fungi | kruskakli: same way you voted -1 with it (same basic api syntax) | 13:22 |
| sdague | oh, sorry, that first comment was unrelated | 13:23 |
| anteaya | fungi: will it work if voting is disabled for that account? | 13:23 |
| fungi | anteaya: i don't know | 13:23 |
| kruskakli | anteaya: yes, but will re-read it since I've forgotten what it says... :-) | 13:23 |
| anteaya | fungi: very good | 13:23 |
| *** ArxCruz has joined #openstack-infra | 13:23 | |
| anteaya | kruskakli: thank you | 13:23 |
| kruskakli | anteaya: thanks for your help, much appreciated! | 13:24 |
| sdague | fungi: honest, if there was an easy way to do it, all the keystone and nova patches should be kicked from the queue | 13:24 |
| sdague | as they are both failing unit tests, a lot | 13:24 |
| anteaya | kruskakli: np | 13:24 |
| anteaya | kruskakli: thanks for being available to learn | 13:24 |
| anteaya | kruskakli: so after that page, read this one: http://gerrit-documentation.googlecode.com/svn/Documentation/2.3/cmd-review.html | 13:24 |
| kruskakli | anteaya: I'll be around in these channels from now on | 13:25 |
| anteaya | kruskakli: thank you, that will make a big difference to the success of your testing | 13:25 |
| anteaya | please join #openstack-dev if you haven't already | 13:25 |
| *** jroovers has quit IRC | 13:26 | |
| kruskakli | anteaya: joined! | 13:26 |
| anteaya | kruskakli: thanks | 13:27 |
| fungi | sdague: short of uploading trivial updates to all those patchsets, nothing springs to mind | 13:27 |
| smarcet | fungi: morning , i just approved this change, https://review.openstack.org/#/c/65152/, but seems that is stuck, i mean , merge didnt happen | 13:29 |
| chmouel | anteaya: fyi that tool https://github.com/pandemicsyn/fgerrit/ can batch bunch of gerrit votes if needed | 13:30 |
| fungi | smarcet: the gating system is going a bit slowly at the moment... it should pick it up shortly | 13:30 |
| *** oubiwann has joined #openstack-infra | 13:30 | |
| fungi | smarcet: apparently everyone is now back from their holidays and approving changes, so we're dealing with some unprecedented load | 13:31 |
| smarcet | fungi: oh got it, thought that i need clicking and viewing all files on commit to marked them as reviewed | 13:31 |
| smarcet | fungi: i understand np :), thx for the info | 13:31 |
| anteaya | chmouel: thanks | 13:31 |
| anteaya | I will admit I am having a hard time identifying the feature set of the tool amoungst all the agressive language | 13:32 |
| *** talluri_ has quit IRC | 13:32 | |
| anteaya | chmouel: we actually were looking for a tool this morning (Australia time) to do just that - batch gerrit votes | 13:33 |
| fungi | anteaya: usually we just stick a bunch of ssh api calls in a temporary shell script and run it | 13:34 |
| *** jroovers has joined #openstack-infra | 13:34 | |
| *** dstanek has joined #openstack-infra | 13:35 | |
| anteaya | fungi: makes sense | 13:35 |
| anteaya | mikal had offered to do up a script this morning | 13:35 |
| anteaya | but he had things to do like run his miniconf | 13:35 |
| sdague | fungi: yeh, I should write a tool to bulk snip like that | 13:35 |
| fungi | or a for loop in shell over a list of changes or whatever | 13:36 |
| kruskakli | anteaya: trying to vote 0 on this: https://review.openstack.org/#/c/64802 doing this: ssh -p 29418 ncsopenstack gerrit review -m '"Clearing out wrong -1 vote"' --verified=0 c1d53fd4 | 13:36 |
| fungi | smarcet: your change finally made it into a test queue according to http://status.openstack.org/zuul/ so it should return in just a bit | 13:36 |
| kruskakli | gives me: error: Change is closed\n one or more approvals failed; review output above | 13:36 |
| smarcet | fungi: tnx sire for the info | 13:36 |
| sdague | kruskakli: the change is abandoned | 13:36 |
| sdague | so no voting can happen on it | 13:37 |
| kruskakli | sdague: aha, ok so for those I don't need to clear out the vote? | 13:37 |
| kruskakli | ok, I see | 13:37 |
| anteaya | kruskakli: you can't clear out the vote, gerrit won't let you since it is abandoned | 13:37 |
| *** michchap has quit IRC | 13:38 | |
| kruskakli | and the same goes for status:Merged I assume? | 13:38 |
| *** michchap has joined #openstack-infra | 13:38 | |
| anteaya | kruskakli: yes, if the patch is merged you don't need to change the vote | 13:38 |
| anteaya | not sure what gerrit will give you if you try | 13:39 |
| anteaya | but I do appreciate you taking the time to do this, and learn the process | 13:40 |
| kruskakli | anteaya: ok, I looked at all those changesets they were all either Abandoned or Merged | 13:40 |
| *** _ruhe is now known as ruhe | 13:40 | |
| kruskakli | should I ask about that tox-traceback error in #openstack-dev ? | 13:41 |
| anteaya | kruskakli: okay, thanks for looking at them | 13:42 |
| anteaya | kruskakli: can you put the error in a paste please? | 13:42 |
| anteaya | paste.openstack.org | 13:42 |
| anteaya | and bring the url back here, I will take a look | 13:42 |
| anteaya | but normally yes, -dev is a good place to start | 13:43 |
| dstufft | fungi: I don't know if it's a problem you guys hit or not, but pip 1.5.1 will work with verisons of setuptools < 0.8 | 13:43 |
| kruskakli | anteaya: http://paste.openstack.org/show/60635/ | 13:44 |
| anteaya | kruskakli: okay thank you | 13:47 |
| anteaya | now let's go back the sdague's comment | 13:48 |
| anteaya | about a pip version release that broke us | 13:48 |
| *** mrodden has joined #openstack-infra | 13:48 | |
| anteaya | kruskakli: so you are running neutron in a vagrant, correct? | 13:50 |
| fungi | dstufft: good to know... i think that may be the issue which had forced us to use get-pip on all of our systems because we couldn't get in-place upgrades to work (various setuptools and pkg-config exceptions) | 13:51 |
| *** CaptTofu has quit IRC | 13:52 | |
| *** CaptTofu has joined #openstack-infra | 13:52 | |
| *** dprince has joined #openstack-infra | 13:53 | |
| dstufft | fungi: techincally pip 1.5.1 won't require setuptools to install from a Wheel (but will from a sdist) at all, with the ceaveat that if you install pip from a sdist that the setuptools entrypoint wrapper requires some version of setuptools installed | 13:53 |
| kruskakli | anteaya: yes | 13:53 |
| kruskakli | anteaya: perhaps I should paste my setup script? | 13:55 |
| fungi | dstufft: great--that simplifies things a good deal | 13:56 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add bug for nova libvirt failure https://review.openstack.org/65266 | 13:56 |
| anteaya | kruskakli: sure | 13:56 |
| anteaya | kruskakli: the more information, the better | 13:57 |
| kruskakli | anteaya: at the end of http://paste.openstack.org/show/60636/ | 13:58 |
| kruskakli | you'll find the Vagrantfile and the setup.sh script | 13:58 |
| *** dhellmann is now known as dhellmann_ | 13:59 | |
| *** rcleere has joined #openstack-infra | 14:00 | |
| kruskakli | anteaya: sorry this contains the correct Vagrantfile http://paste.openstack.org/show/60637/ | 14:01 |
| *** pblaho has quit IRC | 14:02 | |
| anteaya | kruskakli: okay for starters, our infrastructure uses precise, not quantal | 14:03 |
| anteaya | and we use pip for installation of python packages not easy_install | 14:03 |
| kruskakli | anteaya: aha, good | 14:03 |
| anteaya | these might seem like small points, but the closer you replicate our set-up the easier it will be for you | 14:03 |
| kruskakli | anteaya: I'll try precise and pip and see if that changes anything | 14:04 |
| anteaya | kruskakli: awesome, thank you | 14:05 |
| anteaya | also do a fresh clone/install of devstack/neutron | 14:06 |
| anteaya | I'm signing off for sleep purposes | 14:06 |
| kruskakli | anteaya: ok, thanks | 14:06 |
| anteaya | you are welcome to ping me in -neutron and keep me informed of your progress | 14:06 |
| anteaya | thanks for taking an interest | 14:06 |
| kruskakli | anteaya: will do | 14:06 |
| *** jasondotstar has joined #openstack-infra | 14:07 | |
| anteaya | it will take a while but hopefully we can help you get what you need so you have a better sense of how to achieve your goals | 14:07 |
| anteaya | kruskakli: thanks | 14:07 |
| * anteaya shuts down for my night | 14:07 | |
| *** saschpe has joined #openstack-infra | 14:09 | |
| *** dcramer_ has joined #openstack-infra | 14:11 | |
| *** saschpe has quit IRC | 14:14 | |
| *** mfer has joined #openstack-infra | 14:14 | |
| *** noorul has left #openstack-infra | 14:15 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: force removal of libvirt-dev https://review.openstack.org/65278 | 14:15 |
| *** dims has quit IRC | 14:17 | |
| *** changbl has quit IRC | 14:17 | |
| *** tma996 has quit IRC | 14:18 | |
| *** dims has joined #openstack-infra | 14:18 | |
| *** dkliban has joined #openstack-infra | 14:19 | |
| *** tma996 has joined #openstack-infra | 14:19 | |
| *** dims is now known as Guest71835 | 14:19 | |
| *** Guest71835 has quit IRC | 14:19 | |
| *** Guest71835 has joined #openstack-infra | 14:20 | |
| sdague | ok, here | 14:22 |
| fungi | yep | 14:22 |
| *** julim has joined #openstack-infra | 14:22 | |
| fungi | so to recap... | 14:22 |
| *** tma996 has quit IRC | 14:23 | |
| fungi | when we reverted the most recent change to add ubuntu cloud archive on our ubuntu slaves, we apparently neglected to ensure /etc/apt/sources.list.d/cloudarchive.list got removed, and so upon installing libvirt-dev we ended up with a bunch of packages updated... http://paste.openstack.org/show/60640/ | 14:24 |
| fungi | i'll rework that list to show the versions they started at | 14:24 |
| sdague | fungi: ok, so how do we recover? | 14:24 |
| sdague | because that seems like we need to rebuild all those unit test nodes from scratch | 14:24 |
| *** Guest71835 has quit IRC | 14:24 | |
| sdague | as rollback isn't really supported | 14:24 |
| sdague | what is the unit test node build model? | 14:25 |
| *** dims_ has joined #openstack-infra | 14:25 | |
| fungi | rollback generally works for most stuff. packages that keep data around or which make one-way updates to their configuration are an exception to that | 14:26 |
| *** saschpe has joined #openstack-infra | 14:27 | |
| fungi | sdague: we build unit test nodes by hand with the help of the launch.py script in openstack-infra/config:launch | 14:27 |
| sdague | ok | 14:28 |
| fungi | okay, so here are the package versions which changed today... http://paste.openstack.org/show/60645 | 14:28 |
| fungi | we may only need to downgrade those first three | 14:28 |
| *** tma996 has joined #openstack-infra | 14:29 | |
| fungi | the others are probably normal updates to precise, but i'll check them in just a sec | 14:29 |
| sdague | yeh, probably | 14:29 |
| sdague | fungi: so how would you orchestrate the rollback? | 14:31 |
| *** yassine has quit IRC | 14:31 | |
| *** yassine has joined #openstack-infra | 14:32 | |
| fungi | sdague: thankfully we have all the slaves configured as salt minions | 14:32 |
| *** sandywalsh has quit IRC | 14:32 | |
| sdague | ok, but you still need to provide the puppet policy, no? | 14:32 |
| fungi | sorry, still trying to get up to speed--it's my fault for looking at irc when i first woke up, rather than after coffee/shower/et cetera. i have a full day and it's clearly not going as planned | 14:32 |
| fungi | sdague: yeah, i'm proposing a change to remove that file | 14:33 |
| fungi | then get it applied, then use salt to downgrade package versions everywhere | 14:33 |
| *** w_ is now known as olaph | 14:34 | |
| sdague | ok | 14:34 |
| sdague | so the downgrade is manual | 14:34 |
| *** saschpe has quit IRC | 14:34 | |
| fungi | yeah, but i can tell salt to do it on all 58 unit test slaves at once | 14:35 |
| sdague | cool | 14:35 |
| russellb | thanks a bunch guys :) | 14:36 |
| *** AJaeger has joined #openstack-infra | 14:36 | |
| *** homeless has joined #openstack-infra | 14:37 | |
| fungi | russellb: thank me for the unscheduled ubuntu cloud archive test ;) | 14:37 |
| sdague | isn't there someone from cannonical around we can blame for that :) | 14:38 |
| fungi | sdague: don't anger zul. we need him | 14:38 |
| *** afazekas has joined #openstack-infra | 14:38 | |
| *** tma996 has quit IRC | 14:40 | |
| *** alexpilotti has joined #openstack-infra | 14:40 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Really remove Ubuntu Cloud Archive for now https://review.openstack.org/65281 | 14:44 |
| fungi | ninja-approving that ^ now | 14:44 |
| *** sandywalsh has joined #openstack-infra | 14:45 | |
| annegentle_ | hey all, what's the latest on releasing clouddocs-maven-plugin? We'll need a release soon for a pair of new features for the API reference page, and dcramer_ and I were wondering where we're at. | 14:46 |
| *** eharney has joined #openstack-infra | 14:46 | |
| *** wenlock has joined #openstack-infra | 14:46 | |
| *** oubiwann has quit IRC | 14:46 | |
| mriedem | this looks like a total infra fail when something was down, so not sure if there was a bug to reverify against or not? anyone know? | 14:47 |
| mriedem | http://logs.openstack.org/16/60716/4/gate/gate-tempest-dsvm-postgres-full/c99e3f8/console.html | 14:47 |
| *** wenlock has quit IRC | 14:47 | |
| mriedem | nm, looks like bug https://bugs.launchpad.net/openstack-ci/+bug/1266578 | 14:47 |
| *** tma996 has joined #openstack-infra | 14:47 | |
| fungi | mriedem: different issue, but basically the same traceback | 14:48 |
| mriedem | fungi: do i need a different bug? | 14:48 |
| fungi | i think someone opened a bug for that event last night | 14:48 |
| mriedem | oh | 14:48 |
| mriedem | looking | 14:48 |
| mriedem | here we go https://bugs.launchpad.net/openstack-ci/+bug/1266692 | 14:48 |
| fungi | yep, basically we brought two more jenkins masters online last night to deal with the increased load, but accidentally started them with an outdated gearman plugin which didn't know how to properly offline used slaves | 14:49 |
| *** ruhe is now known as _ruhe | 14:49 | |
| mriedem | ok, i should hold off on reverifying nova patches right now though right? | 14:49 |
| mriedem | due to the libvirt issue | 14:49 |
| fungi | mriedem: yes, until i get the libvirt on the slaves downgraded to an older, more usable (for us) version | 14:50 |
| fungi | i'm guessing about 30 minutes | 14:50 |
| mriedem | k, thanks | 14:50 |
| *** ociuhandu has joined #openstack-infra | 14:50 | |
| fungi | #status alert working on undoing the accidental libvirt upgrade which is causing nova and keystone unit test failures (ETA 15:30 UTC) | 14:51 |
| openstackstatus | NOTICE: working on undoing the accidental libvirt upgrade which is causing nova and keystone unit test failures (ETA 15:30 UTC) | 14:51 |
| *** ChanServ changes topic to "working on undoing the accidental libvirt upgrade which is causing nova and keystone unit test failures (ETA 15:30 UTC)" | 14:51 | |
| openstackgerrit | Malini Kamalambal proposed a change to openstack-infra/config: Add Job for marconi-tempest integration https://review.openstack.org/65140 | 14:52 |
| annegentle_ | wah emergencies always take over my doc build needs :) /whine | 14:54 |
| fungi | annegentle_: manufacturing emergencies is my job | 14:55 |
| annegentle_ | fungi: nicely done :) | 14:55 |
| AJaeger | annegentle_, should we help with producing some doc build emergencies? ;) | 14:55 |
| *** coolsvap has joined #openstack-infra | 14:56 | |
| AJaeger | fungi, thanks for approving the operations-guide niceness gate working with openstack-doc-tools - everything looks fine. Now time to use this for api-site ;) | 14:56 |
| openstackgerrit | A change was merged to openstack-infra/config: Really remove Ubuntu Cloud Archive for now https://review.openstack.org/65281 | 14:57 |
| *** burt has joined #openstack-infra | 14:57 | |
| *** che-arne has joined #openstack-infra | 15:00 | |
| *** heyongli has quit IRC | 15:01 | |
| sdague | fungi: so... with more digging, I actually think nova code is wrong | 15:01 |
| *** marun has quit IRC | 15:02 | |
| *** mancdaz is now known as mancdaz_away | 15:03 | |
| fungi | sdague: quite possibly, but that's why we've had a bug open for ages asking nova devs to dig into why tempest fails with new libvirt | 15:03 |
| fungi | i hadn't even checked to see if the unit tests also exhibited issues, but apparently they do | 15:04 |
| *** mancdaz_away is now known as mancdaz | 15:04 | |
| *** dhellmann_ is now known as dhellmann | 15:09 | |
| *** luisg has joined #openstack-infra | 15:10 | |
| *** yaguang has quit IRC | 15:13 | |
| *** dkranz has quit IRC | 15:14 | |
| fungi | okay, manually downgraded precise40, confirmed on precise39 that i could do the same via salt.run... http://paste.openstack.org/show/60652/ | 15:17 |
| fungi | (that's the resulting downgrades in the dpkg log) | 15:17 |
| fungi | doing the same en masse for the rest now | 15:17 |
| chmouel | sdague: is there some sort of regexp on the elastic search queries? | 15:19 |
| *** oubiwann has joined #openstack-infra | 15:19 | |
| *** haypo has left #openstack-infra | 15:19 | |
| fungi | i'll do a pass on them once that completes to confirm they're all showing libvirt0 0.9.8-2ubuntu17.16 and then we can recheck a previously failed job | 15:19 |
| *** alexpilotti has quit IRC | 15:20 | |
| sdague | chmouel: no | 15:20 |
| sdague | search engines don't really work with regex | 15:20 |
| *** alexpilotti has joined #openstack-infra | 15:21 | |
| sdague | because in order to search that much data efficiently you need to precompute all the token location indexes | 15:21 |
| russellb | fungi: if you want to add openstack-ci to the bug ... https://bugs.launchpad.net/nova/+bug/1266711 | 15:22 |
| russellb | document the infra side of this | 15:22 |
| lifeless | sdague: there is a regex search engine that is efficient | 15:23 |
| lifeless | sdague: based on a google paper about google codesearch | 15:23 |
| *** oubiwann_ has joined #openstack-infra | 15:23 | |
| lifeless | sdague: search for 'regex search index code' in google | 15:24 |
| sdague | lifeless: sure. I can be more specific - lucene based search engines | 15:24 |
| fungi | russellb: thanks | 15:24 |
| lifeless | sdague: ah heh :) | 15:24 |
| sdague | which are the basis of all the major open source search engines | 15:24 |
| *** oubiwann has quit IRC | 15:25 | |
| *** kraman has joined #openstack-infra | 15:25 | |
| *** alexpilotti has quit IRC | 15:25 | |
| *** rcleere has quit IRC | 15:26 | |
| mriedem | sdague: chmouel: there is some kind of regex capability in logstash, clarkb linked me to a page on the syntax once | 15:27 |
| mriedem | but i don't think e-r queries were honoring it if i remember correctly... | 15:27 |
| *** ^d has joined #openstack-infra | 15:27 | |
| chmouel | mriedem, sdague: ah ok thanks | 15:27 |
| *** mancdaz is now known as mancdaz_away | 15:27 | |
| mriedem | chmouel: i think this was the one i had which wasn't working: https://review.openstack.org/#/c/62067/4/queries/1258682.yaml | 15:28 |
| sdague | mriedem: so even if it was supported, it would be effectively like going from an index scan to a table scan on our data | 15:28 |
| mriedem | i think it works in logstash but not e-r | 15:28 |
| openstackgerrit | Chmouel Boudjnah proposed a change to openstack-infra/elastic-recheck: Add message about failures on hpcloud. https://review.openstack.org/65292 | 15:29 |
| sdague | mriedem: so that's globbing not regex :) | 15:29 |
| sdague | and I think that doesn't work because you need to put the string in quotes | 15:29 |
| sdague | if it has a glob | 15:29 |
| chmouel | ah yeah "some sort of" regexp i was including globbing in there :) | 15:29 |
| chmouel | let me know if that submission looks good ^ | 15:30 |
| fungi | some day i will write a paper about the various different kinds of regex implementations and their quirks, and call it "regular expressions are anything but regular" | 15:30 |
| chmouel | fungi: heh | 15:30 |
| sdague | chmouel: is there really a reason to put the devstack-precise- in there at all? | 15:31 |
| mriedem | sdague: there were two different bugs with the same trace | 15:31 |
| * mriedem gets them | 15:31 | |
| fungi | for an out-in-left-field kind of regex, i give you "cisco ios bgp prefix regular expressions" | 15:31 |
| mriedem | sdague: https://bugs.launchpad.net/bugs/+bugs?field.searchtext=java.io.IOException%3A+Remote+call+on&search=Search+Bug+Reports&field.scope=all&field.scope.target= | 15:32 |
| *** jpich has quit IRC | 15:33 | |
| *** saschpe has joined #openstack-infra | 15:33 | |
| sdague | mriedem: so they probably should be merged | 15:33 |
| fungi | chmouel: mriedem: sdague: i question the usefulness (or accuracy) of adding jenkins agent java exceptions to e-r. those are individual isolated events, and usually fixed within hours. by the time they make it into e-r's field of view, it's time to remove them again so that they don't point people at the bug for the wrong incident | 15:33 |
| *** mancdaz_away is now known as mancdaz | 15:34 | |
| fungi | unless you really just want a blanket pattern for all manner of incidents infra-side which impact jenkins operation or connectivity, ever | 15:34 |
| sdague | fungi: I disagree for the following reasons | 15:35 |
| fungi | that's like having a bug for "sometimes python generates a traceback" | 15:35 |
| mriedem | fungi: that hp one has 61 hits in the last 7 days | 15:35 |
| mriedem | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiamF2YS5pby5JT0V4Y2VwdGlvbjogUmVtb3RlIGNhbGwgb24gZGV2c3RhY2stcHJlY2lzZS1ocGNsb3VkXCIgQU5EIG1lc3NhZ2U6XCJmYWlsZWRcIiBBTkQgZmlsZW5hbWU6XCJjb25zb2xlLmh0bWxcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4OTEwODc5OTkzNH0= | 15:35 |
| openstackgerrit | Chmouel Boudjnah proposed a change to openstack-infra/elastic-recheck: Add message about failures on hpcloud. https://review.openstack.org/65292 | 15:35 |
| sdague | 1) we can't compute our classification rates if we have some nebulus amount of gate fails which aren't binned | 15:35 |
| *** saschpe has quit IRC | 15:36 | |
| sdague | 2) it is important to figure out what percentage of gate resets are infra vs. inherent races | 15:36 |
| fungi | sdague: so i guess this brings us back to rechecks on specific bugs for isolated infra events aren't really useful, and we might as well just have a catch-all bug which makes no pretense about what it is | 15:37 |
| fungi | i can see the logic in that | 15:37 |
| sdague | if you want to handle infra issues that way (i.e. you see no value in trying to bin them out), I'm fine with it | 15:38 |
| *** _ruhe is now known as ruhe | 15:38 | |
| sdague | we do need to be able to account bugs against infra though | 15:39 |
| *** wenlock has joined #openstack-infra | 15:40 | |
| *** alexpilotti has joined #openstack-infra | 15:40 | |
| mriedem | sdague: for this one: https://review.openstack.org/#/c/62067/4/queries/1258682.yaml , the logstash query wasn't working when there were quotes around the lines with the wildcard, and removing them made the logstash query hit, | 15:41 |
| mriedem | not sure if that would be different with e-r though? | 15:41 |
| fungi | yeah, agreed. there just may not be a lot of benefit to rechecking on individual bugs since e-r isn't going to be able to tell those incidents apart so it's mostly going to be pointing devs at catchall bugs anyway | 15:41 |
| sdague | fungi: some times. the pypi mirror was an example exactly the opposite | 15:44 |
| sdague | I think think that infra has the same 2 kinds of bugs that other projects have | 15:44 |
| *** lukego has quit IRC | 15:44 | |
| fungi | probably so | 15:44 |
| sdague | intermittent issues | 15:44 |
| sdague | and flat out breaks that nothing moves forward until they are solved | 15:44 |
| sdague | maybe more of the second camp | 15:45 |
| sdague | because things like the jenkins slave changes don't get self tested | 15:45 |
| sdague | so there are a few more ways to land a break | 15:45 |
| sdague | anyway, it's probably worth a discussion on -dev or -infra list about how we should approach it | 15:46 |
| sdague | because it does end up also as a useful metric of "what could someone do to help the gates" the most. | 15:46 |
| fungi | yep--totally merits a wider audience | 15:47 |
| *** nati_ueno has joined #openstack-infra | 15:48 | |
| fungi | okay, libvirt downgrades completed... there was some delay as we had a broken version of salt-minion on some of them which was failing on the apt-get commands i was trying to pass in (apparently unattended upgrades from the salt deb package repository still aren't working, i think because it's hitting config file changes) | 15:51 |
| fungi | i'm going to recheck a previously failed nova change | 15:51 |
| *** blamar has quit IRC | 15:51 | |
| *** senk has joined #openstack-infra | 15:51 | |
| *** rnirmal has joined #openstack-infra | 15:53 | |
| dhellmann | I know (think?) there have been some changes to the recheck feature of gerrit, is it still possible to 'recheck no bug' for an old changeset that hasn't been through the test suite in a while? | 15:54 |
| fungi | dhellmann: yep | 15:54 |
| mriedem | dhellmann: yeah, i think sdague actually wants that, there was a ML thread on it a few weeks back | 15:55 |
| dhellmann | I rechecked https://review.openstack.org/#/c/56093/14 about 15 minutes ago, maybe I'm just not waiting long enough -- the queue looks fairly long in zuul | 15:55 |
| dhellmann | mriedem: right, that's what I thought I remembered | 15:55 |
| mriedem | i did it this morning on a nova patch that didn't run through jenkins since november | 15:55 |
| sdague | dhellmann: yes, it's all just very busy right now | 15:55 |
| fungi | dhellmann: it's only "reverify no bug" which was removed | 15:55 |
| dhellmann | fungi: cool, that's what I remembered | 15:55 |
| fungi | dhellmann: yeah, zuul is pretty bogged down handling results from resets and recalculating the dependent queue in the gate pipeline | 15:55 |
| * dhellmann had 2 weeks off and forgot how to computer | 15:56 | |
| sdague | heh | 15:56 |
| mriedem | dhellmann: http://lists.openstack.org/pipermail/openstack-dev/2013-December/022797.html | 15:56 |
| sdague | fungi: how's the libvirt downgrade? | 15:56 |
| dhellmann | fungi: I'll go review another and be more patient, I just wanted to make sure I had done it properly -- thanks! | 15:56 |
| fungi | sdague: i've filed a recheck on 64893, just waiting for the job to start | 15:56 |
| sdague | fungi: well there are plenty of nova changes still in the queue | 15:57 |
| mriedem | do e-r bugs get opened against openstack-ci and tagged with elastic-recheck? | 15:57 |
| mriedem | not seeing a bug tracking system for elastic-recheck in launchpad | 15:57 |
| sdague | mriedem: we aren't using a formal bug tracker for e-r | 15:57 |
| fungi | mriedem: yeah, that's what i've been doing anyway | 15:57 |
| sdague | -ETOOMANYTRACKERS | 15:58 |
| fungi | mriedem: for the couple infra bugs which got filed for e-r i've simply added an "elastic-recheck" tag on them | 15:58 |
| mriedem | fungi: ok, that's what i saw and will do, thanks | 15:58 |
| sdague | mriedem: so if there is an e-r issue better to just stick it out here somewhere | 15:58 |
| *** dkranz has joined #openstack-infra | 15:58 | |
| mriedem | sdague: the issue is my busted e-r query with wildcards | 15:58 |
| sdague | gotcha | 15:58 |
| mriedem | i see that it's not reporting, so gonna report the bug and try to fix it | 15:58 |
| mriedem | shit gets crazy in here and lost quickly, so want to make sure i have a bug filed so i don't forget it | 15:59 |
| kraman | mordred: ping | 15:59 |
| sdague | ok, cool | 16:00 |
| fungi | kraman: mordred's in australia this week, so if he's awake he probably shouldn't be | 16:01 |
| sdague | we need bug management via irc | 16:01 |
| annegentle_ | fungi: ok, tagging 0.2 for the openstack-doc-tools, and I don't recall doing a git pull previously, but the instructions on https://wiki.openstack.org/wiki/GerritJenkinsGit say to do so... is that just to make sure you've got the latest? | 16:01 |
| annegentle_ | fungi: this tagging thing isn't innate to me yet :) | 16:01 |
| annegentle_ | fungi: and we don't have milestone-proposed | 16:02 |
| *** rcleere has joined #openstack-infra | 16:03 | |
| annegentle_ | fungi: basically I just took the master, git checkout -b release-0.2 and then expect to tag that. I hope that's right... | 16:03 |
| fungi | annegentle_: yes. sometimes your changes don't merge in exactly the same order as they exist in your local repository, in which case there's a merge commit or the change got a different sha when it was cherry-picked in. if you tag your local repo without first pulling from the public one, you might tag the wrong sha | 16:03 |
| *** eharney_ has joined #openstack-infra | 16:04 | |
| annegentle_ | fungi: ok so for the not-changing-often repo, my method should be ok? | 16:04 |
| *** eharney has quit IRC | 16:05 | |
| fungi | annegentle_: it seemed to work okay last time, but it's not really safe to tag without first making sure you have the same commits the rest of the world is seeing | 16:05 |
| fungi | annegentle_: in other words, you got lucky, but i wouldn't make a habit of tempting fate that way | 16:07 |
| *** rnirmal has quit IRC | 16:07 | |
| *** rnirmal has joined #openstack-infra | 16:07 | |
| *** eharney_ is now known as eharney | 16:08 | |
| fungi | annegentle_: this is also one of the reasons to develop your changes in topic branches and leave your local master clean so you can pull into it safely with --ff-only | 16:08 |
| *** nati_ueno has quit IRC | 16:08 | |
| *** nati_ueno has joined #openstack-infra | 16:08 | |
| david-lyle | fungi: can you update https://review.openstack.org/#/admin/groups/146,members to me? I need to release a new version of django-openstack-auth to support django 1.6 | 16:09 |
| fungi | david-lyle: sure thing. usually the outgoing ptl should add the incoming ptl, but in your case i can see that might not have happened as intended | 16:09 |
| david-lyle | there are a few things like that | 16:10 |
| david-lyle | I'm not admin in launchpad either | 16:10 |
| david-lyle | for horizon | 16:10 |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 16:10 | |
| fungi | david-lyle: it's also a good reason to designate one or more release delegates from your core team to handle things like that in your absence or during transitions | 16:10 |
| *** dims_ is now known as dims | 16:10 | |
| david-lyle | fungi: good suggestion, I'll do that | 16:11 |
| fungi | david-lyle: i've added you to the group in gerrit... i'll let you remove gabriel or not as you see fit | 16:11 |
| david-lyle | fungi: thx | 16:11 |
| *** reed has joined #openstack-infra | 16:11 | |
| fungi | yw | 16:12 |
| *** jooools has quit IRC | 16:12 | |
| fungi | grr... 64893 is still waiting for a precise node to run the py27 job | 16:12 |
| *** jooools has joined #openstack-infra | 16:13 | |
| fungi | sdague: though this looks promising... https://jenkins02.openstack.org/job/gate-nova-python27/14699/console | 16:13 |
| *** johnthetubaguy has joined #openstack-infra | 16:13 | |
| *** MarkAtwood has joined #openstack-infra | 16:13 | |
| fungi | (that's the earliest non-failing nova change in the gate right now) | 16:13 |
| *** jgrimm has joined #openstack-infra | 16:14 | |
| *** nati_ueno has quit IRC | 16:16 | |
| *** yamahata has quit IRC | 16:16 | |
| annegentle_ | fungi: so how is milestone-proposed made? | 16:16 |
| *** marun has joined #openstack-infra | 16:16 | |
| *** nati_ueno has joined #openstack-infra | 16:16 | |
| *** johnthetubaguy1 has joined #openstack-infra | 16:16 | |
| *** chandankumar has quit IRC | 16:16 | |
| *** blamar has joined #openstack-infra | 16:17 | |
| *** johnthetubaguy has quit IRC | 16:17 | |
| annegentle_ | fungi: I want to be "right" :) | 16:17 |
| fungi | annegentle_: a release manager creates a new branch from master when the milestone is in sight, and then changes from master get backported to the milestone-proposed branch | 16:17 |
| annegentle_ | fungi: oh ok | 16:18 |
| annegentle_ | fungi: that's overkill for this for now, seems like. | 16:18 |
| fungi | annegentle_: that is the model we use for server projects, but for libraries and utilities we generally just tag sequentially on master and don't make branches | 16:19 |
| annegentle_ | fungi: ok,t hat's more our model, the libraries | 16:19 |
| fungi | yep | 16:19 |
| annegentle_ | fungi: still, locally I'd not change master, I'd checkout -b based on master, I think | 16:19 |
| fungi | annegentle_: just tag the current state of the repository when you want to release, and tag again when it's time for another release | 16:19 |
| annegentle_ | fungi: don't you love how I have to "talk aloud" through this ? Hope I'm not driving you crazy | 16:19 |
| annegentle_ | fungi: sounds good | 16:20 |
| annegentle_ | fungi: tag on master? | 16:20 |
| fungi | annegentle_: you can do that... checkout origin/master though, not your local master | 16:20 |
| fungi | annegentle_: yes, tag only on master | 16:20 |
| annegentle_ | fungi: okie. git checkout master; git remote update; git pull origin master is the same thing? | 16:21 |
| fungi | annegentle_: for the server projects which use stable branches and milestones, we have special jobs to merge those tags back into master, but it's not necessary on a non-branching project | 16:21 |
| fungi | annegentle_: your example is of using a local master branch and pulling into it from the origin remote's master branch, but if you don't pull --ff-only you risk mixing local changes with the state of the published repository | 16:22 |
| annegentle_ | fungi: makes sense, and I've never --ff-only, but I can see what to avoid now... thanks! | 16:23 |
| *** pmathews has joined #openstack-infra | 16:23 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Fix the e-r query for bug 1258682 https://review.openstack.org/65303 | 16:24 |
| fungi | annegentle_: if you want to do it via checkout, use 'git remote update && git checkout -b foo origin/master' which should create a local branch "foo" tracking the master branch from the origin remote | 16:24 |
| annegentle_ | fungi: okie. | 16:25 |
| annegentle_ | fungi: also how many -infra folks are in Australia? | 16:25 |
| annegentle_ | fungi: mostly just idle curiousity | 16:25 |
| fungi | then you can tag on that throwaway branch and push the tag, and it will be a clean copy of the published master branch you're tagging on | 16:26 |
| fungi | annegentle_: depends on how you #define "folks" but all the core/root admins besides me, and also several of the non-core team members | 16:27 |
| fungi | but infra has tentacles which reach far and wide, so "-infra folks" could be a ton of people (there are currently 231 lurking on the irc channel, for example, many of which are likely there for lca too) | 16:28 |
| fungi | er, many of whom | 16:28 |
| * fungi hates when he grammarfails | 16:28 | |
| * dstufft is not in australia, but is also mostly here to whine about things | 16:28 | |
| annegentle_ | fungi: nice grammar :) | 16:28 |
| annegentle_ | dstufft: I whined earlier, probably hit my quota | 16:29 |
| fungi | dstufft: you can start by whining about how you're not in australia right now (otherwise you'd have written it whinging, i think) | 16:29 |
| *** gyee_ has joined #openstack-infra | 16:29 | |
| dstufft | fungi: lol | 16:30 |
| fungi | sdague: i've seen three nova changes pass python27 unit tests in the gate, so i think i'll sound the all-clear. objections? | 16:31 |
| *** AaronGr_Zzz is now known as AaronGr | 16:32 | |
| *** nati_ueno has quit IRC | 16:33 | |
| *** nati_uen_ has joined #openstack-infra | 16:33 | |
| fungi | sdague: four more on track to finish in the next 10 minutes too and haven't failed yet | 16:33 |
| *** SergeyLukjanov has joined #openstack-infra | 16:34 | |
| mfer | folks, anyone have a little time to chat about https://bugs.launchpad.net/openstack-ci/+bug/1242569 | 16:35 |
| mfer | This is the new project creation bug | 16:35 |
| mfer | as I understand it, there is a fix that just needs to be all test... the fix has been merged in | 16:35 |
| *** chandankumar_ has quit IRC | 16:35 | |
| *** nati_uen_ has quit IRC | 16:36 | |
| *** nati_ueno has joined #openstack-infra | 16:36 | |
| fungi | mfer: yes, i'm hoping to try pushing one through today (i saw a couple which didn't have -2 votes on them) and will then ask the other cores to remove their -2 votes if it works | 16:38 |
| fungi | but they're all on the opposite side of the globe at the moment, so won't be on until later | 16:38 |
| *** mrmartin has joined #openstack-infra | 16:38 | |
| mfer | fungi mine is at https://review.openstack.org/#/c/62069/. jeblair marked it down due to the bug after first passing it. I think this is fine. | 16:39 |
| mfer | i did have to push a second change due to changes in config | 16:39 |
| *** nati_ueno has quit IRC | 16:40 | |
| *** nati_uen_ has joined #openstack-infra | 16:40 | |
| *** nati_uen_ has quit IRC | 16:41 | |
| fungi | mfer: yep, hopefully we can catch up on those outstanding project additions in the next 12-24 hours if nothing else major breaks and diverts my attention | 16:41 |
| *** dkranz has quit IRC | 16:41 | |
| *** UtahDave has joined #openstack-infra | 16:41 | |
| *** mrmartin has quit IRC | 16:41 | |
| mfer | fungi if there is anything I can do please feel free to ping me. | 16:42 |
| *** mancdaz is now known as mancdaz_away | 16:42 | |
| mfer | I'm happy to help and learn more | 16:42 |
| *** hashar has quit IRC | 16:43 | |
| *** marun has quit IRC | 16:44 | |
| *** hashar has joined #openstack-infra | 16:45 | |
| fungi | sdague: those other four passed as well, and so did the recheck on 64893 | 16:46 |
| fungi | #status ok unit tests seem to be passing consistently after the upgrade. use bug 1266711 for related rechecks | 16:47 |
| openstackstatus | NOTICE: unit tests seem to be passing consistently after the upgrade. use bug 1266711 for related rechecks | 16:47 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 16:47 | |
| *** johnthetubaguy1 has quit IRC | 16:48 | |
| *** ruhe is now known as ruhe_away | 16:49 | |
| *** johnthetubaguy has joined #openstack-infra | 16:49 | |
| *** rossella_s_ has joined #openstack-infra | 16:49 | |
| *** johnthetubaguy has quit IRC | 16:49 | |
| *** johnthetubaguy has joined #openstack-infra | 16:49 | |
| mriedem | sdague: looks like elasticsearch doesn't analyze wildcards by default: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_wildcards | 16:50 |
| fungi | mriedem: they're disabled in the config except on one field, i think. if memory serves that was for performance reasons | 16:51 |
| *** julim has quit IRC | 16:51 | |
| mriedem | fungi: that's what the es docs say, it's disabled by default for performance | 16:51 |
| fungi | jesusaurus may remember more | 16:51 |
| mriedem | memory consumption | 16:51 |
| *** MarkAtwood has quit IRC | 16:52 | |
| mriedem | fungi: which config? | 16:52 |
| *** rossella_s has quit IRC | 16:52 | |
| *** rossella_s_ is now known as rossella_s | 16:52 | |
| *** flaper87 is now known as flaper87|afk | 16:52 | |
| *** dkranz has joined #openstack-infra | 16:53 | |
| *** Ryan_Lane has joined #openstack-infra | 16:53 | |
| fungi | the kibana or elasticsearch config templates in openstack-infra/config. i don't remember which one exactly, i just remember spending a while figuring that out a couple months back | 16:53 |
| mriedem | ok, i'll look | 16:53 |
| mriedem | i might be able to hack around it with a regex but not sure | 16:53 |
| mriedem | and not sure if that would even be acceptable | 16:54 |
| *** ruhe_away is now known as ruhe | 16:54 | |
| *** marun has joined #openstack-infra | 16:55 | |
| sdague | mriedem: yeh, I think that in general we probably want to err away from doing anything to impact es performance | 16:55 |
| *** MarkAtwood has joined #openstack-infra | 16:55 | |
| *** herndon has joined #openstack-infra | 16:55 | |
| *** mancdaz_away is now known as mancdaz | 16:55 | |
| *** roeyc has quit IRC | 16:56 | |
| mriedem | the wildcard in the one query is just to glob on build_name, looks like there are at least 7 of them that could be hit | 16:58 |
| mriedem | maybe build_name doesn't even need to be in the query, but i think it was to avoid false negatives | 16:58 |
| *** ruhe is now known as ruhe_away | 16:58 | |
| *** SergeyLukjanov has quit IRC | 16:59 | |
| jesusaurus | mriedem: if you want to analyzed on wildcards, i think you would set that in the file /etc/elastcisearch/default-mapping.json (not sure where in -infra/config thats managed) | 16:59 |
| mriedem | jesusaurus: here? https://git.openstack.org/cgit/openstack-infra/config/tree/modules/elasticsearch/files/elasticsearch.mapping.json | 16:59 |
| mriedem | anyway, i don't think we need/want the wildcard analysis, we should doc it though | 16:59 |
| *** hashar has quit IRC | 17:00 | |
| mriedem | i ran this w/o the wildcards and it has actually more hits on jobs that weren't in the original query, e.g. the rally one, all fails: | 17:00 |
| mriedem | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQnVpbGQgdGltZWQgb3V0IChhZnRlclwiIEFORCBtZXNzYWdlOlwibWludXRlcykuIE1hcmtpbmcgdGhlIGJ1aWxkIGFzIGZhaWxlZC5cIiBBTkQgZmlsZW5hbWU6XCJjb25zb2xlLmh0bWxcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4OTExMzkyODY5Nn0= | 17:00 |
| *** dkranz has quit IRC | 17:00 | |
| jesusaurus | mriedem: yeah thats the file | 17:01 |
| jesusaurus | i actually just stopped analyzing some fields in my cluster setup because the analysis does stuff like split strings on '-' | 17:02 |
| *** markmcclain has quit IRC | 17:02 | |
| *** ruhe_away is now known as ruhe | 17:03 | |
| *** Ryan_Lane has quit IRC | 17:04 | |
| *** MarkAtwood has quit IRC | 17:04 | |
| *** jasondotstar has left #openstack-infra | 17:05 | |
| *** jasondotstar has joined #openstack-infra | 17:06 | |
| *** DennyZhang has joined #openstack-infra | 17:06 | |
| *** tma996 has quit IRC | 17:07 | |
| *** markmcclain has joined #openstack-infra | 17:07 | |
| *** rwsu has quit IRC | 17:08 | |
| *** andreaf has quit IRC | 17:08 | |
| *** hashar has joined #openstack-infra | 17:08 | |
| *** marun has quit IRC | 17:09 | |
| mrodden | are there any docs for using nodepool? | 17:10 |
| fungi | mrodden: nothing substantial yet--jeblair is still drafting them | 17:10 |
| *** prad has joined #openstack-infra | 17:10 | |
| *** alexpilotti has quit IRC | 17:10 | |
| mrodden | ok | 17:10 |
| fungi | mrodden: but others have gotten it to work by looking at what little it comes with and maybe some code comments, plus asking a handful of questions in here | 17:11 |
| portante | fungi: this grenade job timed out after 65 minutes, http://logs.openstack.org/35/64835/2/gate/gate-grenade-dsvm/c39e0db/ | 17:11 |
| portante | no logs were saved | 17:11 |
| portante | :( | 17:11 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Fix the e-r query for bug 1258682 https://review.openstack.org/65303 | 17:11 |
| portante | should I file a bug for that, and then what is the best way forward to get this job reverified? | 17:11 |
| mriedem | portante: see the patch i just pushed ^ | 17:12 |
| mriedem | excellent timing | 17:12 |
| fungi | portante: that's going to be one of those edge cases where 5 minutes apparently wasn't enough. maybe we should consider bumping the jenkins timeouts to 15 minutes more than the devstack timeouts just to be safe (we saw log uploads going very slowly yesterday when the jenkins masters were heavily loaded) | 17:12 |
| *** dkranz has joined #openstack-infra | 17:12 | |
| portante | fungi: perhaps we need to insert a heartbeat into the jobs? | 17:13 |
| fungi | portante: i should be able to dig up the console log for it at least | 17:13 |
| portante | instead of just doing a timeout alone? | 17:13 |
| portante | fungi: great | 17:13 |
| fungi | portante: perhaps... i think that's more or less the idea with the per-test timeouts in testr though | 17:13 |
| fungi | we should make sure that each individual test can run in a reasonable amount of time, and refactor the ones which take too long | 17:14 |
| portante | fungi: agreed | 17:14 |
| fungi | and then we can set aggressive per-test timeouts | 17:14 |
| fungi | portante: oh, the log link you provided already includes the console log | 17:16 |
| *** jorisroovers has joined #openstack-infra | 17:16 | |
| portante | mriedem: thanks, though it does not appear to be the exact same mode of failure ... | 17:16 |
| portante | fungi: yes, that appears to be the only thing that was captured aside from the setup .txt files | 17:16 |
| fungi | portante: oh, it looks like maybe devstack-gate didn't kill that job at the 60-minute mark, so it kept running until jenkins killed it at 65. maybe we have a mismatch in the set durations... digging deeper | 17:17 |
| portante | fungi: looks torgomatic reverified it using bug 1254270 | 17:18 |
| *** senk has quit IRC | 17:18 | |
| *** mancdaz is now known as mancdaz_away | 17:19 | |
| fungi | looks like the job passed d-g the envvar to kill it at 60... http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#n654 | 17:19 |
| *** jroovers has quit IRC | 17:20 | |
| fungi | yeah, can see it at http://logs.openstack.org/35/64835/2/gate/gate-grenade-dsvm/c39e0db/console.html#_2014-01-07_15_48_36_312 as well | 17:20 |
| *** jorisroovers has quit IRC | 17:20 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add jobs for cliff integration tests https://review.openstack.org/65180 | 17:24 |
| fungi | i wonder if the gate_hook function never gets called for grenade runs | 17:25 |
| fungi | nevermind. i'm crackers. it has to since that's how grenade.sh gets reached | 17:25 |
| *** marun has joined #openstack-infra | 17:27 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add some documentation on wildcard limitations in queries https://review.openstack.org/65326 | 17:28 |
| *** yassine has quit IRC | 17:28 | |
| sdague | fungi: note it was 46 minutes from job kick off, until that job actually started | 17:29 |
| sdague | http://logs.openstack.org/35/64835/2/gate/gate-grenade-dsvm/c39e0db/console.html#_2014-01-07_15_48_36_341 | 17:30 |
| fungi | sdague: yeah, i think the d-g gate hook probably was not reached until well after 5 minutes into the job, which explains the behavior we saw with jenkins killing it before teh wrap script | 17:31 |
| *** rossella_s has quit IRC | 17:32 | |
| *** pcrews has joined #openstack-infra | 17:32 | |
| sdague | honestly, it's probably worth providing an early kill timeout at job start | 17:32 |
| mordred | morning all | 17:32 |
| sdague | if we don't get to job start in 10 minutes, kill the job | 17:32 |
| fungi | mordred: wow, you're up early | 17:32 |
| sdague | because it's going to cause a cascading fail | 17:32 |
| mordred | fungi: I'm in Palo Alto :) | 17:32 |
| sdague | mordred: wait, I thought you were at lca? | 17:32 |
| *** marun has quit IRC | 17:33 | |
| *** marun has joined #openstack-infra | 17:33 | |
| *** jerryz has joined #openstack-infra | 17:33 | |
| *** dizquierdo has quit IRC | 17:34 | |
| *** oubiwann has joined #openstack-infra | 17:34 | |
| *** dstanek has quit IRC | 17:35 | |
| *** johnthetubaguy1 has joined #openstack-infra | 17:35 | |
| fungi | the palo alto just outside of perth? | 17:36 |
| *** DennyZhang has quit IRC | 17:36 | |
| *** senk has joined #openstack-infra | 17:36 | |
| *** johnthetubaguy has quit IRC | 17:37 | |
| *** oubiwann_ has quit IRC | 17:38 | |
| sdague | he's just trying to trick us to not have to work on infra issues :) | 17:39 |
| *** Hefeweizen has quit IRC | 17:41 | |
| *** senk1 has joined #openstack-infra | 17:41 | |
| *** senk has quit IRC | 17:41 | |
| *** DennyZhang has joined #openstack-infra | 17:41 | |
| *** mgagne1 has joined #openstack-infra | 17:42 | |
| *** hashar has quit IRC | 17:44 | |
| *** zigo_ has joined #openstack-infra | 17:45 | |
| *** derekh has quit IRC | 17:46 | |
| *** persia__ has joined #openstack-infra | 17:46 | |
| *** cyeoh_ has joined #openstack-infra | 17:47 | |
| *** briancli1e has joined #openstack-infra | 17:47 | |
| *** lcestari has quit IRC | 17:47 | |
| mordred | bah. like I do work anyway | 17:48 |
| *** rnirmal has quit IRC | 17:50 | |
| *** blamar has quit IRC | 17:50 | |
| *** mgagne has quit IRC | 17:50 | |
| *** bknudson has quit IRC | 17:50 | |
| *** morganfainberg has quit IRC | 17:50 | |
| *** Adri2000 has quit IRC | 17:50 | |
| *** briancline has quit IRC | 17:50 | |
| *** Alex_Gaynor has quit IRC | 17:50 | |
| *** persia has quit IRC | 17:51 | |
| *** cyeoh has quit IRC | 17:51 | |
| *** zigo has quit IRC | 17:51 | |
| *** morganfainberg has joined #openstack-infra | 17:51 | |
| *** Alex_Gaynor_ has joined #openstack-infra | 17:51 | |
| *** morganfainberg has quit IRC | 17:51 | |
| *** morganfainberg has joined #openstack-infra | 17:51 | |
| *** bknudson has joined #openstack-infra | 17:51 | |
| *** Alex_Gaynor_ is now known as Alex_Gaynor | 17:51 | |
| *** rnirmal has joined #openstack-infra | 17:51 | |
| *** lcestari has joined #openstack-infra | 17:52 | |
| *** Ryan_Lane has joined #openstack-infra | 17:53 | |
| *** Adri2000 has joined #openstack-infra | 17:53 | |
| *** Ryan_Lane has quit IRC | 17:53 | |
| *** Ryan_Lane has joined #openstack-infra | 17:53 | |
| *** Adri2000 has quit IRC | 17:53 | |
| *** Adri2000 has joined #openstack-infra | 17:53 | |
| *** atiwari has joined #openstack-infra | 17:57 | |
| *** DennyZhang has quit IRC | 17:58 | |
| *** oubiwann_ has joined #openstack-infra | 17:59 | |
| *** Ryan_Lane has quit IRC | 18:00 | |
| *** oubiwann has quit IRC | 18:02 | |
| *** harlowja_away is now known as harlowja | 18:06 | |
| *** morganfainberg has quit IRC | 18:09 | |
| *** bknudson has quit IRC | 18:09 | |
| *** bknudson has joined #openstack-infra | 18:10 | |
| *** morganfainberg has joined #openstack-infra | 18:11 | |
| *** harlowja has joined #openstack-infra | 18:11 | |
| *** esker has joined #openstack-infra | 18:12 | |
| *** nati_ueno has joined #openstack-infra | 18:12 | |
| *** dims has quit IRC | 18:13 | |
| *** talluri has joined #openstack-infra | 18:17 | |
| *** dims has joined #openstack-infra | 18:18 | |
| *** julim has joined #openstack-infra | 18:19 | |
| markmcclain | ugh the gate just reset again with missing logs | 18:19 |
| markmcclain | https://jenkins01.openstack.org/job/gate-grenade-dsvm/3065/console | 18:19 |
| markmcclain | oops sorry skipped over a line in the logs.. actually hit the 65 min limit discussed earlier | 18:20 |
| fungi | markmcclain: yep, probably took too long to get started and didn't have enough remaining time to finish | 18:21 |
| markmcclain | yeah.. there's a 50 min gap | 18:21 |
| fungi | without timestamps in the setup logs, it's hard to tell where it was lagging | 18:23 |
| *** Ajaeger1 has joined #openstack-infra | 18:24 | |
| *** CaptTofu has quit IRC | 18:25 | |
| *** CaptTofu has joined #openstack-infra | 18:25 | |
| *** dstufft_ has joined #openstack-infra | 18:26 | |
| *** dstanek has joined #openstack-infra | 18:26 | |
| *** dstufft has quit IRC | 18:27 | |
| jerryz | fungi: ^ https://review.openstack.org/#/c/65178/ please take a look if you have time. and also https://bugs.launchpad.net/openstack-ci/+bug/1266603 , thanks! | 18:28 |
| fungi | we're definitely pegging zuul's cpu today... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=389&rra_id=all | 18:28 |
| fungi | wonder whether it's related | 18:28 |
| fungi | load average spiking up to 140 at times... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=388&rra_id=all | 18:29 |
| *** SergeyLukjanov has joined #openstack-infra | 18:29 | |
| fungi | it's on an 8g rackspace flavor right now, so maybe it's time to consider a bigger zuul | 18:31 |
| mordred | fungi: perhaps a performance flavor | 18:32 |
| fungi | like rocky road | 18:32 |
| *** changbl has joined #openstack-infra | 18:34 | |
| *** mlipchuk has quit IRC | 18:34 | |
| jlk | go perf, and you'll never go back | 18:35 |
| *** CaptTofu_ has joined #openstack-infra | 18:36 | |
| *** talluri has quit IRC | 18:38 | |
| *** CaptTofu has quit IRC | 18:38 | |
| *** smarcet has left #openstack-infra | 18:42 | |
| *** praneshp has joined #openstack-infra | 18:43 | |
| fungi | looking at graphs for the various central systems involved in setup for that job (jenkins01, review, git.*, zuul) that's the only likely resource starvation i'm seeing | 18:44 |
| *** dstufft_ is now known as dstufft | 18:44 | |
| fungi | unless it was the performance of the slave itself or something external like a network issue | 18:44 |
| *** CaptTofu has joined #openstack-infra | 18:45 | |
| *** CaptTofu_ has quit IRC | 18:45 | |
| *** dims has quit IRC | 18:47 | |
| *** jooools has quit IRC | 18:47 | |
| *** dims has joined #openstack-infra | 18:48 | |
| *** senk1 has quit IRC | 18:48 | |
| fungi | checked static too, and didn't see anything too out of the ordinary either | 18:48 |
| *** jooools has joined #openstack-infra | 18:48 | |
| fungi | some load average spikes up around 5 but very brief | 18:48 |
| jgriffith | fungi: correct me if I'm wrong on: https://bugs.launchpad.net/tempest/+bug/1265623 | 18:50 |
| jgriffith | fungi: seems it can be closed as "upgrade to newer virtualenv version" ? | 18:51 |
| *** julim_ has joined #openstack-infra | 18:51 | |
| *** krtaylor has quit IRC | 18:51 | |
| fungi | jgriffith: i think the install_venv_common.py bit relevant to cinder is actually an oslo fix? | 18:52 |
| sdague | yes, it's an oslo bug | 18:52 |
| jgriffith | fungi: I believe so yes | 18:52 |
| jgriffith | sdague: thanks | 18:52 |
| fungi | i'd consider switching the cinder bugtask to oslo since that's where it would be worked around, if at all | 18:52 |
| jgriffith | fungi: will do | 18:52 |
| jgriffith | fungi: sdague thanks | 18:52 |
| *** oubiwann_ is now known as oubiwann | 18:53 | |
| *** dcramer_ has quit IRC | 18:53 | |
| *** che-arne has quit IRC | 18:53 | |
| *** senk has joined #openstack-infra | 18:53 | |
| *** resker has joined #openstack-infra | 18:54 | |
| *** jcoufal-mob has joined #openstack-infra | 18:54 | |
| fungi | mordred: so, if you're around, last night dstufft mentioned something which might be our ticket to working around the current external packages issue we have... we can put the options in requirements files like https://gist.github.com/dstufft/8295462 | 18:59 |
| sdague | fungi: zuul leaking memory? | 18:59 |
| sdague | that seems like a really high load average for an environment that's not swapping to death | 18:59 |
| fungi | mordred: i think we'd need to get pbr to grok that too, as well as the global-requirements enforcement/proposal jobs? | 18:59 |
| fungi | sdague: no swap utilization, memory seems reasonable | 18:59 |
| fungi | sdague: the git merge calculations are what kills it, i think | 19:00 |
| fungi | anybody want an infra meeting? | 19:00 |
| *** julim has quit IRC | 19:01 | |
| *** bknudson has quit IRC | 19:01 | |
| *** changbl has quit IRC | 19:01 | |
| *** morganfainberg has quit IRC | 19:01 | |
| *** esker has quit IRC | 19:01 | |
| *** changbl has joined #openstack-infra | 19:02 | |
| *** jcoufal-mob has quit IRC | 19:05 | |
| *** bknudson has joined #openstack-infra | 19:06 | |
| *** morganfainberg has joined #openstack-infra | 19:06 | |
| *** jcoufal-m has joined #openstack-infra | 19:06 | |
| *** Ryan_Lane has joined #openstack-infra | 19:06 | |
| *** hashar has joined #openstack-infra | 19:06 | |
| *** hashar has quit IRC | 19:07 | |
| *** hashar has joined #openstack-infra | 19:07 | |
| *** blamar has joined #openstack-infra | 19:07 | |
| *** morganfainberg is now known as Guest73162 | 19:07 | |
| *** Ryan_Lane is now known as Guest36676 | 19:07 | |
| *** Guest73162 has quit IRC | 19:07 | |
| *** Guest73162 has joined #openstack-infra | 19:07 | |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Added rbenv-env wrapper https://review.openstack.org/65340 | 19:08 |
| *** krtaylor has joined #openstack-infra | 19:08 | |
| *** Guest73162 is now known as needscoffee | 19:08 | |
| *** _david_ has joined #openstack-infra | 19:09 | |
| *** Guest36676 has quit IRC | 19:10 | |
| *** atiwari has quit IRC | 19:12 | |
| *** dcramer_ has joined #openstack-infra | 19:12 | |
| *** needscoffee is now known as not_stevemar | 19:15 | |
| dstufft | fungi: note, there's no way for pbr to pass those options via the setup.py | 19:15 |
| *** not_stevemar has quit IRC | 19:15 | |
| dstufft | fungi: those only work for requirements files read via ``pip install -r requirements.txt`` | 19:16 |
| *** praneshp has quit IRC | 19:16 | |
| fungi | dstufft: good point, that hadn't dawned on me | 19:16 |
| fungi | i think pbr imports pip as a module, but no idea what it can actually do about it from that end | 19:17 |
| *** praneshp has joined #openstack-infra | 19:17 | |
| dstufft | fungi: I'm pretty sure that's for when someone unpacks the tarball and calls ``python setup.py install`` manually | 19:17 |
| fungi | yeah, it is | 19:17 |
| dstufft | and that gets disabled when pip is installing it | 19:17 |
| fungi | right, okay | 19:18 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add e-r query for bug 1266611 https://review.openstack.org/65344 | 19:21 |
| *** jcoufal-m has quit IRC | 19:21 | |
| *** praneshp has quit IRC | 19:22 | |
| *** saper_ is now known as saper | 19:25 | |
| *** dcramer_ has quit IRC | 19:25 | |
| *** mgagne1 is now known as mgagne | 19:26 | |
| *** dcramer_ has joined #openstack-infra | 19:28 | |
| *** CaptTofu_ has joined #openstack-infra | 19:28 | |
| *** jroovers has joined #openstack-infra | 19:30 | |
| hashar | zaro: hey ! :-) Wikimedia Zuul got switched to Gearman successfully ! Thank you so much for your patch to jenkins gearman plugin :-] | 19:30 |
| *** morganfainberg has joined #openstack-infra | 19:30 | |
| *** morganfainberg has quit IRC | 19:30 | |
| *** morganfainberg has joined #openstack-infra | 19:30 | |
| *** CaptTofu has quit IRC | 19:30 | |
| fungi | hashar: also, congrats on having a scalable ci. now you can add more jenkins masters, or replace them with turbo-hipster ;) | 19:34 |
| zaro | hashar: good to hear. | 19:34 |
| hashar | took me a while :-D | 19:35 |
| hashar | our last upgrade was back in May 2013... | 19:35 |
| *** CaptTofu has joined #openstack-infra | 19:36 | |
| *** CaptTofu_ has quit IRC | 19:36 | |
| hashar | fungi: what the hell is turbo-hipster ? :D | 19:36 |
| *** mrodden has quit IRC | 19:36 | |
| *** mrodden has joined #openstack-infra | 19:37 | |
| *** praneshp has joined #openstack-infra | 19:37 | |
| reed | slightly off topic: what's your choice for spam prevention systems, free(dom) software, for website comments? | 19:37 |
| hashar | volunteers :-D | 19:38 |
| fungi | hashar: that should be the title in http://git.openstack.org/cgit/stackforge/turbo-hipster/plain/README.md | 19:38 |
| hashar | fungi: yeah that is exactly what I wanted to do | 19:38 |
| hashar | tiny gearman client that would run tasks instead of having Jenkins in between | 19:38 |
| hashar | I thought about a basic linting task that would fetch repo then run flake8 | 19:39 |
| fungi | hashar: you should talk to jhesketh (whenever he's around next) and mikal about it | 19:39 |
| hashar | reed: more seriously we have a bunch of very lame measure such as captcha / IP blocking | 19:40 |
| fungi | hashar: right now they have it running timed nova database migration tests and reporting results for them on gerrit changes | 19:40 |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Added rbenv-env wrapper https://review.openstack.org/65352 | 19:40 |
| reed | hashar, thanks... any experience with http://bad-behavior.ioerror.us/ ? | 19:40 |
| hashar | reed: some private measures on the backend to detect well known spammers. And finally we have tons of volunteers | 19:40 |
| *** CaptTofu has quit IRC | 19:41 | |
| reed | hashar, makes sense | 19:41 |
| hashar | reed: and we have some folks that wrote some kind of AI which does roughly 10k reverts per minutes : https://en.wikipedia.org/wiki/User:ClueBot_NG | 19:41 |
| reed | I was looking at Ask OpenStack, we're incredibly lucky that so far we haven't had much trouble | 19:41 |
| hashar | fungi: I am pretty sure that can be generalized to other jobs as well. Afterall you barely use jenkins anymore. | 19:42 |
| *** CaptTofu has joined #openstack-infra | 19:43 | |
| fungi | hashar: yes, we're incrementally reducing the jenkins-specific features we rely on in our systems, so that day will almost certainly eventually arrive | 19:44 |
| *** hogepodge has joined #openstack-infra | 19:44 | |
| fungi | hashar: right now the biggest hurdle is probably just making sense of the various jenkinsisms in our jjb yaml and turning it into something we can feed to a generalized worker | 19:46 |
| fungi | (or a fairly lengthy migration/translation project) | 19:47 |
| Shrews | fungi: So, was a change made to pbr recently to not allow pip to download externally hosted packages? | 19:47 |
| dstufft | That was a change in pip | 19:47 |
| fungi | Shrews: the release of pip 1.5 on january 2nd | 19:47 |
| Shrews | oh? uh.. hrm | 19:48 |
| dstufft | http://www.python.org/dev/peps/pep-0438/ | 19:48 |
| dstufft | ^ More details about the general deprecating external hosted stuff | 19:48 |
| *** resker has quit IRC | 19:48 | |
| *** yolanda has quit IRC | 19:48 | |
| *** esker has joined #openstack-infra | 19:48 | |
| Shrews | thx. believe i can work around this for now with the --allow-external option | 19:49 |
| fungi | Shrews: we're still working through solutions for our end to deal with it until those remaining packages end up on pypi proper. tracking with https://launchpad.net/bugs/1266513 and https://etherpad.openstack.org/p/pip1.5Upgrade | 19:49 |
| Shrews | thx x 2 | 19:50 |
| dstufft | fungi: there's https://github.com/pypa/pip/issues/1268 though I'm opposed to it, but in the interest of being ~transparent~ and what not :] | 19:50 |
| fungi | dstufft: yeah, i'm fairly opposed to it as well. it will just turn into conventional wisdom that you always run pip with that flag and nobody will fix the remaining semi-popular libraries which are the cause | 19:51 |
| *** praneshp has quit IRC | 19:52 | |
| fungi | at least forcing people to enumerate the broken ones leads to a sort of "wall of shame" situation, and guilts the developers into fixing their problem | 19:52 |
| dstufft | fungi: I added it to your etherpad, but just fyi if you all need --allow-insecure to stick around past 1.7 just let me know and we'll make it happen. It's mostly getting removed just to keep our list of hidden options down to a minimum | 19:53 |
| *** esker has quit IRC | 19:53 | |
| *** flaper87|afk is now known as flaper87 | 19:53 | |
| dstufft | fungi: yea, that was part of the original goal, I wanted people to know which libraries were making them type this crap out so they shamed developers into doing it :V | 19:54 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add some documentation on wildcard limitations in queries https://review.openstack.org/65326 | 19:54 |
| fungi | dstufft: i figured, bit i also don't see any problem with transitioning after 1.6 is released. as long as we have an option which works across 2-3 releases i think it's fine, to give people a way to run it before and after upgrading pip | 19:54 |
| dstufft | fungi: cool | 19:54 |
| *** afazekas has quit IRC | 19:54 | |
| fungi | i'm just leery of forcing options right now which only work with the very shiniest, newest pip release and leaving everyone out in the cold until they upgrade | 19:55 |
| dstufft | ya | 19:55 |
| dstufft | that's why we kept --allow-insecure around when we renamed it to --allow-unverified | 19:55 |
| hashar | fungi: ah I forgot about jjb :/ | 19:55 |
| fungi | and it's very much appreciated! | 19:55 |
| hashar | fungi: at least the jobs definitions are abstracted out. | 19:55 |
| dstufft | I kinda want to convince folks to switch pip to semver tbh | 19:56 |
| dstufft | maybe after i'm done breaking stuff :V | 19:56 |
| *** sarob has joined #openstack-infra | 19:56 | |
| fungi | hashar: right. we're in a better position to replace jenkins since everything is in jjb yaml, as opposed to if we were stuck trying to translate it all out of jenkins xml | 19:56 |
| *** praneshp has joined #openstack-infra | 19:56 | |
| fungi | hashar: also, the yaml as it stands now provides us with a pretty good view of what we're still missing non-jenkins coverage for | 19:57 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add some documentation on wildcard limitations in queries https://review.openstack.org/65326 | 19:57 |
| fungi | the closer everything gets to being a reference to something else in jjb or a shell block, the closer we are to being able to stop doing it in jenkins | 19:58 |
| hashar | fungi: do you know whether turbo-hipster has a mechanism to clone the repos with the proper changes? | 19:58 |
| fungi | hashar: no idea whether they're doing that or using our gerrit-git-prep.sh | 19:59 |
| mordred | hashar: I believe so - I believe it's cloning them directly from zuul | 19:59 |
| fungi | i haven't looked at it closely yet | 19:59 |
| hashar | yeah code reuse! | 19:59 |
| mordred | me either | 19:59 |
| hashar | I wanted to write a tiny zuul-cloner client that would be given ZUUL_ variables + repositories to clone | 19:59 |
| hashar | and would do everything for me just like your devstack shell script | 20:00 |
| hashar | except, written in python :-D | 20:00 |
| fungi | mordred: planning to be on the defcore call? if so, it's starting nopw | 20:00 |
| ttx | conveniently placed at the same time as the TC meeting, yay | 20:01 |
| *** jcoufal has quit IRC | 20:01 | |
| *** harlowja is now known as harlowja_away | 20:01 | |
| *** harlowja_away is now known as harlowja | 20:02 | |
| mordred | fungi: no, I'm not. I'm planning on being at teh TC meeting which has been scheduled for a few years now | 20:02 |
| fungi | mordred: and #refstsack | 20:02 |
| *** jcoufal has joined #openstack-infra | 20:02 | |
| fungi | er stack | 20:02 |
| *** sarob has quit IRC | 20:02 | |
| *** dkliban has quit IRC | 20:03 | |
| *** sarob has joined #openstack-infra | 20:06 | |
| *** praneshp has quit IRC | 20:06 | |
| *** krotscheck has joined #openstack-infra | 20:06 | |
| *** gyee_ has quit IRC | 20:10 | |
| *** jergerber has joined #openstack-infra | 20:10 | |
| *** jergerber has quit IRC | 20:10 | |
| *** praneshp has joined #openstack-infra | 20:12 | |
| *** praneshp has quit IRC | 20:14 | |
| krotscheck | dhellmann: Hey, mordred just forwarded a question of mine to you regarding wsme/pecan. I don't suppose there's a project somewhere that I can reference as a starting point? | 20:17 |
| dhellmann | krotscheck: ceilometer, ironic, and tuskar all use pecan and wsme | 20:17 |
| krotscheck | Awesome, thanks. | 20:18 |
| *** sarob has quit IRC | 20:18 | |
| *** esker has joined #openstack-infra | 20:18 | |
| ruhe | krotscheck: fyi there is a patch with intends to add pecan to storyboard - https://review.openstack.org/#/c/63118/ | 20:19 |
| ruhe | *which intends | 20:19 |
| krotscheck | ruhe: Thanks, I'll take a look. | 20:21 |
| krotscheck | I'm technically on vacation, it might be a bit. | 20:21 |
| *** sarob has joined #openstack-infra | 20:24 | |
| *** sarob has quit IRC | 20:24 | |
| *** CaptTofu has quit IRC | 20:24 | |
| jog0 | whoa gate is way backed up | 20:24 |
| *** sarob has joined #openstack-infra | 20:24 | |
| *** CaptTofu has joined #openstack-infra | 20:25 | |
| jog0 | sdague: 74 patches, and top is from 22 hours ago | 20:25 |
| *** senk has quit IRC | 20:26 | |
| sdague | yeh, well nova vs. libvirt was a culprit | 20:26 |
| jog0 | gah libvirt | 20:26 |
| jog0 | it used to work | 20:26 |
| jog0 | I like the graph http://status.openstack.org/elastic-recheck/ | 20:27 |
| jog0 | lol | 20:27 |
| krotscheck | I don't suppose someone from infra can take a look at this patch? https://review.openstack.org/#/c/64761/ | 20:28 |
| *** sarob_ has joined #openstack-infra | 20:28 | |
| *** esker has quit IRC | 20:29 | |
| fungi | krotscheck: i already have it starred. hopefully i'll get to it in the next day or so | 20:29 |
| *** melwitt has joined #openstack-infra | 20:29 | |
| fungi | maybe today depending on what else crops up | 20:29 |
| *** sarob has quit IRC | 20:29 | |
| *** esker has joined #openstack-infra | 20:29 | |
| krotscheck | fungi: Oh, thanks. THat'd be appreciated :) | 20:29 |
| krotscheck | fungi: In particular I'm curious about whether the test reporting that I'm doing is appropriate - I've got manual tarball uploads for some of the test output. | 20:31 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Add openstack-doc-tools https://review.openstack.org/65361 | 20:33 |
| *** esker has quit IRC | 20:33 | |
| *** dkliban has joined #openstack-infra | 20:34 | |
| *** sarob_ has quit IRC | 20:35 | |
| *** sarob has joined #openstack-infra | 20:35 | |
| *** praneshp has joined #openstack-infra | 20:35 | |
| *** banix has joined #openstack-infra | 20:39 | |
| *** sarob has quit IRC | 20:40 | |
| mriedem | chmouel: i think you can abandon this: https://review.openstack.org/#/c/65292/ | 20:41 |
| mriedem | it's a duplicate | 20:41 |
| dkranz | fungi: The periodic-tempest-dsvm-all-havana builds have been failing for at least two weeks http://logs.openstack.org/periodic-qa/periodic-tempest-dsvm-all-havana/aeece8b/console.html.gz | 20:43 |
| chmouel | mriedem: yeah sounds good but mine was catching explicitely the cloud provider (i.e: HP here) so we can track those yours look a bit more greedy right? | 20:43 |
| dkranz | fungi: I could not see anything interesting in the logs | 20:43 |
| dkranz | fungi: The error is "timeout: failed to run command `/opt/stack/new/devstack-gate/devstack-vm-gate.sh': No such file or directory" | 20:44 |
| mriedem | chmouel: if you look at the elastic-recheck status page, it should already be catching those failures with the existing query | 20:44 |
| fungi | dkranz: interesting. i knew devstack exercises on grizzly were breaking havana grenade | 20:44 |
| mriedem | so yes, it's greedy and generic | 20:44 |
| mriedem | chmouel: that particular issue is going to be an ongoing infra problem so i'd say let the existing query always handle it | 20:44 |
| dkranz | fungi: This error seems to happen before devstack is even installed | 20:45 |
| fungi | dkranz: oh, that looks like a broken job--i'll see what's gone wrong there | 20:45 |
| dkranz | fungi: logstash shows many hits on that same job over the last two weeks | 20:45 |
| chmouel | mriedem: what if we want to detect a particuliar cloud providerr? | 20:45 |
| dkranz | fungi: thanks | 20:45 |
| *** sarob has joined #openstack-infra | 20:45 | |
| fungi | dkranz: are other periodic devstack-gate jobs failing with a similar error? | 20:45 |
| dkranz | fungi: Yes, the periodic stress job | 20:46 |
| mriedem | chmouel: my guess is elastic-recheck will report back on the patch with multiple bugs and the author will recheck on one of them | 20:46 |
| dkranz | fungi: http://logs.openstack.org/periodic-qa/periodic-tempest-dsvm-stress-havana/419b810 | 20:46 |
| chmouel | mriedem: ok | 20:46 |
| dkranz | fungi: Always the same two fail | 20:46 |
| fungi | k, that may help me narrow down a common code path. thanks dkranz | 20:47 |
| mriedem | chmouel: either way, i think it's going to always be whack a mole with that failure | 20:47 |
| *** _david_ has left #openstack-infra | 20:48 | |
| chmouel | mriedem: yeah agreed, it's the "sh*** happens" catch all bug :) | 20:48 |
| *** smarcet has joined #openstack-infra | 20:49 | |
| *** jroovers has quit IRC | 20:54 | |
| *** SergeyLukjanov has quit IRC | 20:56 | |
| *** arosen has joined #openstack-infra | 20:56 | |
| *** SergeyLukjanov has joined #openstack-infra | 20:56 | |
| *** krotscheck has quit IRC | 21:00 | |
| *** dizquierdo has joined #openstack-infra | 21:01 | |
| *** jroovers has joined #openstack-infra | 21:02 | |
| *** jorisroovers has joined #openstack-infra | 21:06 | |
| *** jroovers has quit IRC | 21:06 | |
| *** jcoufal has quit IRC | 21:10 | |
| *** ryanpetrello has joined #openstack-infra | 21:10 | |
| *** smarcet has left #openstack-infra | 21:11 | |
| *** senk has joined #openstack-infra | 21:13 | |
| *** jorisroovers has quit IRC | 21:13 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: install gerrit core plugins https://review.openstack.org/65368 | 21:13 |
| *** Hefeweizen has joined #openstack-infra | 21:16 | |
| *** pballand has joined #openstack-infra | 21:16 | |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Add local-branch option https://review.openstack.org/65369 | 21:16 |
| *** jooools has quit IRC | 21:17 | |
| *** julim_ has quit IRC | 21:17 | |
| *** mrodden1 has joined #openstack-infra | 21:18 | |
| *** julim has joined #openstack-infra | 21:18 | |
| *** mrodden has quit IRC | 21:19 | |
| openstackgerrit | Peter Balland proposed a change to openstack-infra/config: Remove gates from Congress https://review.openstack.org/64730 | 21:19 |
| SpamapS | what is the long term plan for netifaces / netaddr? Are the authors going to move them onto pypi? | 21:21 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: install gerrit core plugins https://review.openstack.org/65368 | 21:22 |
| fungi | SpamapS: perhaps you can ask them (i think someone posted a link to a bug opened against netaddr asking them to switch releases back to pypi) | 21:23 |
| *** julim has quit IRC | 21:23 | |
| *** ociuhandu has quit IRC | 21:25 | |
| SpamapS | fungi: Yeah I'm sending David Moss an email now | 21:26 |
| SpamapS | https://github.com/drkjam/netaddr/issues/57 ftr | 21:26 |
| pballand | can someone point me to a doc/howto for getting a project ready for pep8 tests (configure project for tox?) | 21:29 |
| SpamapS | pballand: there is a repo you can copy | 21:30 |
| Ajaeger1 | pballand: cookiecutter is the repo | 21:30 |
| SpamapS | pballand: https://git.openstack.org/cgit/openstack-dev/cookiecutter/ | 21:30 |
| pballand | SpamapS: That's exactly what I was looking for, thanks! | 21:32 |
| SpamapS | pballand: no, thank you :) | 21:34 |
| arosen | Hi, is there a way to push to review.openstack.org over http instead of ssh? | 21:36 |
| *** ociuhandu has joined #openstack-infra | 21:36 | |
| * arosen on a network where i'm only able to make an outbound connection over http today :( | 21:37 | |
| *** marun has quit IRC | 21:40 | |
| fungi | arosen: there is not, no | 21:40 |
| *** CaptTofu has quit IRC | 21:42 | |
| *** CaptTofu has joined #openstack-infra | 21:42 | |
| arosen | fungi: ah I found out a way to do it. Gerrit seems to provide this functionality. | 21:42 |
| arosen | git push http://review.openstack.org:/openstack/nova.git HEAD:refs/for/master | 21:42 |
| arosen | you have to get an http password and use that in gerrit though | 21:42 |
| fungi | arosen: i suspect that is not a feature in gerrit 2.4 (what we're currently running) | 21:43 |
| arosen | I'll go ahead and document this on: https://wiki.openstack.org/wiki/GerritJenkinsGit | 21:43 |
| arosen | fungi: nah it worked :) | 21:43 |
| fungi | really? i'm surprised | 21:43 |
| fungi | neat! | 21:44 |
| arosen | fungi: proof :P http://codepad.org/WkHZwfwq | 21:44 |
| * fungi has been totally schooled | 21:44 | |
| jog0 | sdague: looks like you were right, now that libvirt is fixed gate is moving a bit | 21:44 |
| *** johnthetubaguy1 has quit IRC | 21:49 | |
| *** sarob has quit IRC | 21:49 | |
| *** sarob has joined #openstack-infra | 21:50 | |
| *** mfer has quit IRC | 21:50 | |
| *** jamielennox is now known as jamielennox|away | 21:51 | |
| *** jpich has joined #openstack-infra | 21:54 | |
| jog0 | jeblair: http://status.openstack.org/zuul/ the top of the gate queue is stuck | 21:54 |
| *** sarob has quit IRC | 21:54 | |
| jeblair | so we need zuul to push git refs to a git server farm and stop serving them itself | 21:56 |
| jeblair | but no one has had a chance to work on that yet afaik | 21:56 |
| *** jcoufal-m has joined #openstack-infra | 21:56 | |
| jeblair | in the mean time, replacing zuul with a sizable performance node would probably be helpful. | 21:57 |
| jog0 | jeblair: thats related to gate-tempest-dsvm-large-ops:queued at the top of the gate | 21:57 |
| jeblair | jog0: i can't see what you're talking about because zuul is so slow | 21:57 |
| jog0 | jeblair: pastebin from top of gate http://paste.openstack.org/show/60694/ | 21:58 |
| jog0 | note the queued statuses | 21:58 |
| jeblair | jog0: it should get to it; that usually happens when jenkins completely fails to run a job | 21:58 |
| jeblair | jog0: we used to error out there, now we detect and re-enqueue | 21:59 |
| jog0 | jeblair: ahh that makes sense thanks | 21:59 |
| jog0 | silly jenkins | 21:59 |
| *** syerrapragada1 has joined #openstack-infra | 22:01 | |
| *** syerrapragada1 has quit IRC | 22:01 | |
| *** Ajaeger1 has quit IRC | 22:01 | |
| *** praneshp has quit IRC | 22:04 | |
| *** gyee has quit IRC | 22:05 | |
| *** jamielennox|away is now known as jamielennox | 22:06 | |
| *** ruhe is now known as _ruhe | 22:08 | |
| *** sandywalsh has quit IRC | 22:09 | |
| dstufft | fungi: setuptools released, so we'll have a 1.11.1rc1 in the next couple hours at the most | 22:09 |
| fungi | dstufft: awesome | 22:10 |
| *** CaptTofu has quit IRC | 22:10 | |
| *** CaptTofu has joined #openstack-infra | 22:11 | |
| sdague | so a good zuul web ui enhancement would be the # of hrs a change has been in the gate | 22:11 |
| sdague | as that seems to be the best indicator of reset at this point | 22:11 |
| sdague | and today it's all manual to figure that out | 22:11 |
| jeblair | sdague: yeah, would be trivial (zuul tracks that; item.enqueue_time) | 22:12 |
| *** jasondotstar has quit IRC | 22:12 | |
| *** sparkycollier has joined #openstack-infra | 22:12 | |
| *** CaptTofu has quit IRC | 22:13 | |
| *** CaptTofu has joined #openstack-infra | 22:13 | |
| *** hashar has quit IRC | 22:13 | |
| *** jroovers has joined #openstack-infra | 22:14 | |
| *** jroovers has quit IRC | 22:14 | |
| *** jroovers has joined #openstack-infra | 22:15 | |
| fungi | mordred: so, here's a problem statement for pbr... | 22:15 |
| fungi | mordred: this fails... http://paste.openstack.org/show/60699 | 22:15 |
| fungi | mordred: this works... http://paste.openstack.org/show/60700 | 22:16 |
| fungi | mordred: do you have any ideas on how we might take advantage of the latter to solve the former? | 22:16 |
| mordred | fungi: I don't think the former is a thing we want to solve | 22:16 |
| mordred | fungi: I believe that pbr is behaving appropriately - the user has installed pip 1.5 and pip 1.5 does not allow installing insecure things | 22:17 |
| fungi | mordred: okay, so pip installs from source using pip 1.5 are not intended. got it | 22:17 |
| mordred | well, I mean, nova currently depends on an insecure package | 22:17 |
| *** gyee has joined #openstack-infra | 22:17 | |
| fungi | well, pip installs of current nova with pip 1.5 i mean | 22:17 |
| mordred | right. I think that is a bug somewhere, but I don't think it's a pbr bug | 22:17 |
| mordred | as in - I do not want to add an "allow insecure" to pbr, because that would defeat much of the point of having secure checking | 22:18 |
| sdague | is build_uuid fully unique in es metadata? | 22:19 |
| sdague | or is it unique per build_job? | 22:19 |
| jeblair | sdague: it's fully unique | 22:20 |
| *** sandywalsh has joined #openstack-infra | 22:21 | |
| sdague | ok, that makes certain things way easier for me :) | 22:22 |
| sdague | which, I didn't realize before | 22:22 |
| sdague | jeblair: thanks | 22:22 |
| *** fungi has quit IRC | 22:22 | |
| *** fungi has joined #openstack-infra | 22:23 | |
| jog0 | sdague: thats the first u in uuid | 22:24 |
| jeblair | jog0: heh | 22:25 |
| fungi | ugh, my isp needs to stop breaking whatever it's decided to break this week | 22:25 |
| *** dprince has quit IRC | 22:26 | |
| jeblair | fungi: if we wanted to resize zuul, we could do a host swap | 22:26 |
| jeblair | fungi: the only firewall that would have to be reloaded is graphite | 22:26 |
| jeblair | everything else connects _to_ zuul | 22:27 |
| *** malini is now known as malini_afk | 22:27 | |
| *** dizquierdo has quit IRC | 22:27 | |
| jeblair | fungi: or we could work on pushing zuul refs to git.o.o | 22:28 |
| jeblair | or a brand new git farm | 22:28 |
| jeblair | (if we push them to git.o.o, i'd want to push them to an alternate set of trees (eg git.o.o/zuul/openstack/foo) | 22:29 |
| *** fungi has quit IRC | 22:30 | |
| *** rnirmal has quit IRC | 22:31 | |
| *** weshay is now known as weshay_afk | 22:31 | |
| *** marun has joined #openstack-infra | 22:33 | |
| *** dstanek has quit IRC | 22:33 | |
| *** thomasem has quit IRC | 22:36 | |
| *** dkranz has quit IRC | 22:39 | |
| *** sarob has joined #openstack-infra | 22:40 | |
| *** sarob has quit IRC | 22:40 | |
| *** sarob has joined #openstack-infra | 22:41 | |
| *** jcoufal-m has quit IRC | 22:43 | |
| *** sarob_ has joined #openstack-infra | 22:43 | |
| *** jcoufal-m has joined #openstack-infra | 22:43 | |
| *** ArxCruz has quit IRC | 22:44 | |
| *** fungi has joined #openstack-infra | 22:44 | |
| fungi | hopefully that's the last of my connectivity issues. not at all conducive to discussion | 22:45 |
| *** praneshp has joined #openstack-infra | 22:45 | |
| *** burt has quit IRC | 22:45 | |
| *** sarob has quit IRC | 22:45 | |
| sdague | jog0: sure... I just mistook that it was tied to the changeset | 22:46 |
| sdague | vs. the jobs | 22:46 |
| sdague | my lack of understanding | 22:46 |
| jeblair | fungi: i think that we can get zuul pushing to git.o.o with a reasonable amount of effort, probably not much more than a zuul server resize | 22:46 |
| jeblair | fungi: so maybe that's what we should pursue to relieve the load on z.o.o | 22:46 |
| fungi | jeblair: agreed, just had a look at eavesdrop to see what i'd missed :/ | 22:47 |
| jroovers | hi fungi, can I ask a quick question related to bug rechecks? | 22:47 |
| jeblair | fungi: it would need a small-moderate patch to zuul, and a scheme to set up the repos on the git servers | 22:47 |
| fungi | jeblair: what sort of alternate trees were you thinking? | 22:47 |
| fungi | jroovers: sure | 22:47 |
| jeblair | fungi: we have something (is it manage-projects) create the repos on git.o.o, right? | 22:48 |
| jroovers | fungi, I accidently did a recheck with a wrong bug number, I have found the correct one now. Can I check the recheck comment? What is the right way to approach this? | 22:48 |
| fungi | jeblair: just an additional set of shadow repositories, maybe not indexed by cgit? | 22:48 |
| fungi | jeblair: right, we do | 22:48 |
| jeblair | fungi: yes | 22:48 |
| jroovers | fungi, "*change the recheck number" | 22:48 |
| fungi | jroovers: sure, you can leave an additional recheck comment. zuul will ignore it, but the rechecks stream will see and add it | 22:48 |
| fungi | jroovers: it's additive though... you can't really remove old references, but there's enough noise in them anyway that it's expected | 22:49 |
| jroovers | fungi, ok thanks | 22:49 |
| fungi | jeblair: so, i think we could quite easily init a second set of repos no problem | 22:49 |
| fungi | jeblair: but we'd also need to see if gerrit's replication config supports mirroring to a different repo name | 22:50 |
| fungi | jeblair: or maybe just a different path but the same name | 22:50 |
| *** flaper87 is now known as flaper87|afk | 22:50 | |
| fungi | the latter should work in gerrit i think? but serving those might get more complicated | 22:51 |
| fungi | i'll have a look at the replication config syntax and see | 22:51 |
| jroovers | fungi, thanks! | 22:52 |
| jeblair | fungi: actually i don't think gerrit needs to replicate | 22:52 |
| jeblair | fungi: i think we just need to have zuul push to these repos | 22:52 |
| mriedem | anyone ever gotten a regex query to work in logstash? | 22:53 |
| fungi | jeblair: oh, that might simplify things... right, zuul creates refs and pushes whatever objects are necessary to build them | 22:53 |
| fungi | jeblair: so no need for replication, agreed | 22:53 |
| jeblair | fungi: yep. i can work on adding the feature to zuul to push to configurable locations; if you have time to figure out how to get another set of repos created and served, that would be helpful | 22:54 |
| jeblair | fungi: (and if not, i'll pick that up when i'm done with zuul) | 22:54 |
| jeblair | i don't think i'm walking through the park today :( | 22:55 |
| mriedem | mrodden1: have you written logstash queries with regex? | 22:56 |
| mriedem | mrodden1: trying to get this to work (check|gate)-(tempest|grenade)-[a-z\-]+ | 22:56 |
| jeblair | mriedem: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html | 22:57 |
| mriedem | jeblair: yeah, been looking through that one for awhile | 22:57 |
| jeblair | mriedem: i think it's elasticsearch syntax, so maybe that will help | 22:57 |
| jeblair | oh sorry | 22:57 |
| * jeblair fails to be helpful | 22:58 | |
| *** fungi has quit IRC | 22:58 | |
| mriedem | well this says most regex should work as long as it's in forward slashes | 22:58 |
| mriedem | http://lucene.apache.org/core/4_0_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#Regexp_Searches | 22:58 |
| *** lcestari has quit IRC | 22:58 | |
| *** jcoufal-m has quit IRC | 22:58 | |
| *** jcoufal-m has joined #openstack-infra | 22:58 | |
| *** fungi has joined #openstack-infra | 22:59 | |
| *** senk has quit IRC | 23:00 | |
| *** jpich has quit IRC | 23:00 | |
| *** jcoufal-m has quit IRC | 23:00 | |
| *** melwitt has quit IRC | 23:01 | |
| *** melwitt has joined #openstack-infra | 23:01 | |
| *** melwitt has quit IRC | 23:01 | |
| *** melwitt1 has joined #openstack-infra | 23:01 | |
| mrodden1 | mriedem: i haven't done any really advanced regex-y things with elasticsearch so idk | 23:03 |
| mriedem | might be a bug or limitation | 23:03 |
| mriedem | https://github.com/elasticsearch/kibana/issues/631 | 23:03 |
| sdague | mriedem: so I think what that actually means is you can't use it for message | 23:04 |
| mriedem | sdague: trying to use it for build_name | 23:04 |
| sdague | mriedem: so you'll have to bypass the query builder to get er to create that kind of query | 23:05 |
| sdague | or extend query builder | 23:05 |
| *** melwitt1 has quit IRC | 23:07 | |
| sdague | mriedem: https://github.com/openstack-infra/elastic-recheck/blob/master/elastic_recheck/query_builder.py#L22 | 23:07 |
| *** melwitt has joined #openstack-infra | 23:07 | |
| sdague | that's what an actual es query gets created as | 23:07 |
| mriedem | i'm just going to brute force the query and list the current check|gate/tempest|grenade builds | 23:07 |
| sdague | so you'd need a regex leaf in there I think | 23:08 |
| *** dcramer_ has quit IRC | 23:08 | |
| *** fungi has quit IRC | 23:08 | |
| *** ^d has quit IRC | 23:08 | |
| *** fungi has joined #openstack-infra | 23:11 | |
| *** senk has joined #openstack-infra | 23:12 | |
| *** hogepodge has quit IRC | 23:13 | |
| *** hogepodge has joined #openstack-infra | 23:14 | |
| mriedem | sdague: do you know if the regexp query will get intersected with the query entry? | 23:14 |
| mriedem | *multiple entries in the dict i mean | 23:14 |
| sdague | mriedem: so to use it in er would be a deeper patch | 23:14 |
| sdague | because we try to simplify it for people | 23:14 |
| mriedem | yeah, like i'd have to add a regexp field to the yaml files right? | 23:14 |
| sdague | yep | 23:15 |
| mriedem | sdague: so would you be against me just brute-forcing the build_name in this query? https://review.openstack.org/#/c/65303/ | 23:16 |
| mriedem | until we had regexp support in e-r? | 23:16 |
| pballand | is there a way to bypass verification (as a core-reviewer) while I get the tests passing? | 23:18 |
| sdague | honestly, I'd be +2 with that bug as is | 23:18 |
| mriedem | sdague: feel free to comment in the review :) | 23:18 |
| dstufft | fungi: mordred sdague clarkb whoever else https://gist.github.com/dstufft/ad130655c81e45098bff | 23:20 |
| *** dims has quit IRC | 23:20 | |
| dstufft | that should fix the --system-site-packages problem | 23:20 |
| mordred | cool! | 23:21 |
| dstufft | If it's at all possible to test that in the openstack infra and let me know if it fixes i for you that'd be awesome | 23:21 |
| *** ryanpetrello has quit IRC | 23:21 | |
| fungi | pballand: non-working jobs can be set to non-voting as long as you have at least one voting job unique to your project | 23:22 |
| pballand | fungi: do I set that in layout.yaml? | 23:24 |
| pballand | (perhaps instead of https://review.openstack.org/#/c/64730/) | 23:24 |
| *** senk1 has joined #openstack-infra | 23:25 | |
| fungi | pballand: yes, you can set a job name pattern to non-voting, or you can just make your jobs always return 0 (success) and rely on looking at the logs for them to tune them up | 23:26 |
| *** sparkycollier has quit IRC | 23:26 | |
| fungi | pballand: there is one catch with non-voting jobs... you need at least one voting job, and you can't use the shared gate-noop job if you run other jobs for your project | 23:27 |
| fungi | pballand: so if you need something like that, you'd want a clone of the -noop job specific to your project | 23:27 |
| *** senk has quit IRC | 23:27 | |
| *** CaptTofu has quit IRC | 23:27 | |
| pballand | is there a quick tox hack to make pep8 and py27 return 0? | 23:28 |
| *** CaptTofu has joined #openstack-infra | 23:28 | |
| fungi | pballand: probably, though i'm not sure exactly... i'd have to play around with it | 23:29 |
| clarkb | sdague: why does er need to explicitly support regexes? | 23:30 |
| clarkb | isnt the query just passed through mostly as is? | 23:30 |
| pballand | fungi: thanks, I'll play around with the targets in tox.ini to get a working set | 23:35 |
| fungi | pballand: also, you can run tox locally and just not enable jobs for your project until you're ready to start using them | 23:35 |
| *** dims has joined #openstack-infra | 23:36 | |
| fungi | pballand: and simply start with only the gate-noop job | 23:36 |
| *** jroovers has quit IRC | 23:36 | |
| pballand | fungi: I started there, but that's been stuck in review for 5 days - probably better to keep in the project and not bother the infra codebase | 23:37 |
| *** oubiwann has quit IRC | 23:37 | |
| *** rcleere has quit IRC | 23:38 | |
| fungi | pballand: yes, sorry, our review bandwidth has been insufficient to keep up with requests. a lot of holiday backlog still | 23:38 |
| * mordred looks away | 23:38 | |
| mriedem | clarkb: query_string and regexp are different things in elasticsearch | 23:38 |
| pballand | fungi: no worries - if there is a way I can stay out of the way (like faking out tests) I'll do it | 23:39 |
| mriedem | clarkb: see my comment here: https://review.openstack.org/#/c/65303/ | 23:39 |
| fungi | pballand: your patience is appreciated | 23:39 |
| mriedem | clarkb: and right now elastic-recheck only sends the query_string to ES | 23:39 |
| pballand | fungi: your build system services for a stackforge project are appreciated :) | 23:40 |
| mriedem | ah infra, when i spend my day in here i feel like i spent 8 hours figuring out why something won't work o-) | 23:40 |
| fungi | pballand: glad to hear it! part of our lack of review bandwidth is that we spend a lot of time keeping those running :/ | 23:41 |
| *** senk1 has quit IRC | 23:42 | |
| *** senk has joined #openstack-infra | 23:43 | |
| fungi | mriedem: that sounds like every day to me ;) | 23:44 |
| mriedem | fungi: i know, it's amazing, always appreciative of the work you guys do | 23:45 |
| *** changbl has quit IRC | 23:46 | |
| *** mriedem has quit IRC | 23:46 | |
| *** cyeoh_ is now known as cyeoh | 23:48 | |
| *** prad has quit IRC | 23:50 | |
| pballand | fungi: FYI, replacing the tox commands with 'true' was sufficient for my needs - I can leave the existing gates and reenable from my project once things are cleaned up | 23:53 |
| zaro | clarkb: gerrit monitoring is already setup to be viewable by gerrit admins. we are both admins so we both get access. | 23:56 |
| *** _ruhe is now known as ruhe | 23:57 | |
| *** melwitt has quit IRC | 23:58 | |
| *** melwitt has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!