| fungi | hey whaddyaknow, ironic py27 passes and runs db migrations for mysql and postgresql | 00:01 |
|---|---|---|
| clarkb | woot | 00:01 |
| fungi | i guess i _could_ just drop the db_name param from the mysql-db and pgsql-db macros in that case | 00:01 |
| fungi | simplifies things nicely | 00:02 |
| fungi | also tested neutron and cinder. they check out without needing anything additional beyond what i install for nova | 00:02 |
| fungi | i expect ceilo to be a little on the special side tho | 00:02 |
| fungi | running it now | 00:03 |
| fungi | i should probably try some random clients too | 00:03 |
| fungi | though those don't need the db | 00:03 |
| fungi | but would be a good exercise of teh package set | 00:03 |
| clarkb | ya clients shouldn't need the db, just lxml | 00:03 |
| *** yamamoto has joined #openstack-infra | 00:04 | |
| *** camunoz has quit IRC | 00:05 | |
| *** dangers is now known as dangers_away | 00:07 | |
| fungi | yep, ceilo wants mongod | 00:08 |
| clarkb | we don't install that for them do we? | 00:08 |
| clarkb | apparently we do | 00:08 |
| clarkb | also zookeeper is there for tooz/taskflow | 00:08 |
| fungi | for their unit tests, looks like we preinstall mongodb-server | 00:08 |
| clarkb | ya | 00:09 |
| *** patrickeast has quit IRC | 00:10 | |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Respect NODEPOOL_SSH_KEY in prepare-node for dib https://review.openstack.org/157205 | 00:13 |
| *** stevemar has quit IRC | 00:16 | |
| fungi | okay, mongodb-server got ceilo passing | 00:17 |
| mordred | Woot | 00:17 |
| mordred | This is exciting | 00:18 |
| jhesketh | clarkb: so I think we need to restart zuul to get the update swift timeout... is this somethign we want to do? Alternatively I should be able to increase the timeout on a per-job basis in the layout | 00:18 |
| clarkb | jhesketh: ya we probably should | 00:19 |
| *** markvoelker has quit IRC | 00:19 | |
| jhesketh | clarkb: as in should restart or do the more graceful layout.yaml? | 00:19 |
| clarkb | jhesketh: sorry, should restart zuul | 00:19 |
| clarkb | jogo: I think that centos7 glusterfs job is broken | 00:20 |
| fungi | i guess the tooz unit tests opportunistically check for zookeeper? | 00:20 |
| clarkb | jogo: at least on that tempest change, I am watching a couple others now | 00:20 |
| clarkb | fungi: yes iirc | 00:20 |
| *** stevemar has joined #openstack-infra | 00:22 | |
| *** ddieterly has joined #openstack-infra | 00:22 | |
| *** camunoz has joined #openstack-infra | 00:22 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: xUnit publisher: add test-time-margin extra config option https://review.openstack.org/134922 | 00:22 |
| clarkb | harlowja_: ^ you would know care to full us in | 00:22 |
| harlowja_ | ?? | 00:23 |
| clarkb | harlowja_: 00:20:12 fungi | i guess the tooz unit tests opportunistically check for zookeeper? | 00:23 |
| harlowja_ | correcto | 00:24 |
| harlowja_ | tooz/taskflow do at least | 00:24 |
| *** dkliban is now known as dkliban_afk | 00:24 | |
| clarkb | thanks (we are working on doing package installs at job runtime rather than during image build time, so that we can reduce the number of images we carry and better document/manage system deps) | 00:25 |
| fungi | harlowja_: thanks, i just managed to confirm that for myself too | 00:25 |
| harlowja_ | cool | 00:25 |
| harlowja_ | redis also | 00:25 |
| harlowja_ | and probably memcache (at least by tooz) | 00:26 |
| clarkb | we would cache the packages on the images so installs should be pretty quick | 00:26 |
| *** ddieterly has quit IRC | 00:26 | |
| fungi | yep | 00:26 |
| *** sarob has quit IRC | 00:27 | |
| harlowja_ | does 'test-time-margin' affect these kind of probings (aka the tests try to connect to redis, and timeout if can't get there...) | 00:27 |
| * harlowja_ pretty sure that the fails pretty quick, but just want to check | 00:27 | |
| fungi | cool, i'll check those too. thanks! | 00:28 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: xUnit publisher: add option to skip if there are no tests files https://review.openstack.org/134923 | 00:28 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: add support for EnvInject mask password params option https://review.openstack.org/148724 | 00:28 |
| clarkb | harlowja_: I don't understand the test-time-margin question | 00:29 |
| clarkb | harlowja_: where is test-time-margin from? | 00:29 |
| harlowja_ | nm, probably a bad up pointer, lol | 00:29 |
| harlowja_ | https://review.openstack.org/134922 | 00:29 |
| clarkb | we don't use xunit | 00:30 |
| harlowja_ | ya, nm, my confusion | 00:30 |
| clarkb | thats just a feature that JJB supports but not something we use. that plugin has a global lock in the jenkins master so each job is serially processed one after the other | 00:31 |
| clarkb | harlowja_: create a major backup for jobs that run quickly when they bump up against jobs that run slowly | 00:31 |
| harlowja_ | kk | 00:31 |
| * harlowja_ thought that was connected to this discussion, but i think its not, ha | 00:31 | |
| *** rmcall has joined #openstack-infra | 00:31 | |
| *** stevemar has quit IRC | 00:32 | |
| *** bhuvan has quit IRC | 00:32 | |
| *** zul has quit IRC | 00:32 | |
| fungi | harlowja_: yeah, this is just working on figuring out how to make unit and static analysis jobs run successfully (and completely) on the same relatively raw servers where we also run devstack-based jobs | 00:32 |
| harlowja_ | cool | 00:32 |
| *** melwitt has quit IRC | 00:33 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add 'last-completed' to copyartifact 'which-build' values https://review.openstack.org/154418 | 00:33 |
| *** tmcpeak has quit IRC | 00:33 | |
| *** ghostpl_ has joined #openstack-infra | 00:33 | |
| fungi | harlowja_: interesting, so with zookeeperd, redis-server and memcached installed, the tooz py27 job still says Ran 295...skips=243 | 00:33 |
| clarkb | fungi: maybe the opportunistic stuff isn't working, should cross check against previous jobs on bare-* | 00:34 |
| harlowja_ | probably missed out on a few more oppurinities fungi | 00:34 |
| harlowja_ | https://github.com/openstack/tooz/blob/master/tooz/tests/test_coordination.py#L31 | 00:34 |
| fungi | yep, doing | 00:34 |
| harlowja_ | kazoo (zookeeper), memcached, redis should all proceed | 00:34 |
| clarkb | harlowja_: well we only care about maintaining those that are tested today | 00:34 |
| clarkb | and if none are tested today then this gets simpler :) | 00:35 |
| *** tmcpeak has joined #openstack-infra | 00:35 | |
| harlowja_ | :) | 00:35 |
| harlowja_ | fair enough, ha | 00:35 |
| fungi | our motto is "no regressions" | 00:35 |
| fungi | (or as few as possible anyway) | 00:35 |
| clarkb | should be simpleish to fix once we move too since this makes it a lot more direct in the job(s) | 00:35 |
| fungi | even moreso if i do the bindep integration i want next | 00:36 |
| clarkb | jhesketh: did you see my comment on 98306? | 00:36 |
| fungi | ahh, yep, same quantity of tests run and skipped on a bare-trusty node too, so this matches anyway | 00:36 |
| clarkb | yup confirmed the glusterfs centos7 job is borked | 00:36 |
| clarkb | the nodes just seem to stop talking to jenkins after a bit then zuul restarts the jobs | 00:37 |
| clarkb | I am going to push up a change to stop running that test | 00:37 |
| jhesketh | clarkb: nope, that's an old one... I'll need to refresh my state but will reply shortly :-) | 00:37 |
| *** ghostpl_ has quit IRC | 00:38 | |
| *** dprince has quit IRC | 00:38 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Remove the tempest glusterfs job https://review.openstack.org/157213 | 00:39 |
| clarkb | jogo: fungi ^ | 00:39 |
| clarkb | jhesketh: ya I sat down with fresh eyes and had new questions :) | 00:39 |
| clarkb | and the author of that test isn't in here anymore | 00:40 |
| *** david-lyle is now known as david-lyle_afk | 00:41 | |
| fungi | clarkb: who was shepherding that job? | 00:41 |
| clarkb | BharatK | 00:41 |
| jhesketh | clarkb: you mean since the first patchset, or you have more since your latest comment (and I should wait before replying) | 00:41 |
| *** bdpayne has quit IRC | 00:42 | |
| *** gordc has quit IRC | 00:42 | |
| clarkb | jhesketh since the land before time. my comment thwre is current | 00:42 |
| *** zul has joined #openstack-infra | 00:42 | |
| jhesketh | cool | 00:43 |
| *** tmcpeak has quit IRC | 00:43 | |
| *** arosen1 has quit IRC | 00:45 | |
| clarkb | fungi: https://jenkins04.openstack.org/job/check-tempest-dsvm-full-glusterfs-centos7/30/consoleFull thats an example of a job that just dropped off | 00:46 |
| fungi | nice. my bet is on network reconfiguration | 00:46 |
| clarkb | ya but gluster shouldn't do any of that but its a long time since I played with gluster | 00:46 |
| fungi | though i suppose it could be flat out panicking the instance | 00:47 |
| clarkb | it is in the object storage tests I don't know if glusterfs is being used as a backend for swift there but ya it may be causing the fs to go belly up | 00:47 |
| clarkb | and kernel panic | 00:47 |
| *** tkelsey has quit IRC | 00:49 | |
| *** ddieterly has joined #openstack-infra | 00:51 | |
| *** hyakuhei has quit IRC | 00:51 | |
| *** markvoelker has joined #openstack-infra | 00:52 | |
| *** ddieterl_ has joined #openstack-infra | 00:52 | |
| *** ddieterly has quit IRC | 00:53 | |
| *** pcrews has quit IRC | 00:55 | |
| ianw | clarkb: i guess on reflection i want to walk "providers->images" not "labels->provider" since the logger is setup logging.getLogger("nodepool.image.build.%s.%s" % (self.provider.name, self.image.name)) | 00:55 |
| clarkb | ianw: yup | 00:55 |
| clarkb | ianw: I think one of my comments addresses that (and using a set as there may be duplicate entries in that list) | 00:56 |
| *** dannywilson has quit IRC | 00:56 | |
| *** nelsnelson has joined #openstack-infra | 00:56 | |
| *** amitgandhinz has joined #openstack-infra | 00:56 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool: Remove the need for zmq in proper demand calcs. https://review.openstack.org/98306 | 00:56 |
| jhesketh | clarkb: ^ updated the commit message to help a little (and commented) | 00:57 |
| *** otter768 has joined #openstack-infra | 00:57 | |
| ianw | clarkb: don't think there are duplicates in provider->images['name'] ? | 00:57 |
| clarkb | jhesketh: thank you | 00:57 |
| *** ZZelle_ has quit IRC | 00:57 | |
| *** sdake__ has joined #openstack-infra | 00:58 | |
| *** bknudson has quit IRC | 00:58 | |
| clarkb | ianw: well you want to map label -> image -> provider to only use images that are in use otherwise its just noise (though maybe that won't hurt here) | 00:58 |
| clarkb | ianw: and you can have mutiple entries for image -> provider from multiple labels. one example is the case of our 2 node envs they use the same images as our single node envs | 00:58 |
| *** markvoelker has quit IRC | 00:59 | |
| ianw | clarkb: right, but i think really the simplest thing to do is just create loggers for provider.image['name'] . even if min-ready is 0 it's still there as a image build | 01:00 |
| *** amitgandhinz has quit IRC | 01:00 | |
| *** sdake_ has quit IRC | 01:01 | |
| clarkb | ianw: not talking about min ready here, we don't build images if they are not associated to a label regardless of the min ready | 01:01 |
| *** otter768 has quit IRC | 01:01 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 01:02 | |
| clarkb | since nodepool only acts on images that end up attached to jenkins with a label | 01:02 |
| clarkb | but you are probably right, that since its just logging config not a runtime optimization we can ignore that | 01:02 |
| ianw | i even called the fn _get_providers_and_images so i'm not sure what happened in between. too much staring at yaml | 01:04 |
| *** yamamoto has quit IRC | 01:05 | |
| *** vishy has quit IRC | 01:07 | |
| *** vishy has joined #openstack-infra | 01:08 | |
| *** patrickeast has joined #openstack-infra | 01:10 | |
| jeblair | oh, i'm an hour late, but happy utc lunar new year | 01:12 |
| *** sdake__ has quit IRC | 01:13 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: Add non-voting job for Nova V2.1 API with neutron https://review.openstack.org/151564 | 01:16 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set custom cpu_model for live_migrate https://review.openstack.org/141530 | 01:16 |
| *** krotscheck has quit IRC | 01:18 | |
| *** MarkAtwood has joined #openstack-infra | 01:19 | |
| *** sputnik13 has quit IRC | 01:21 | |
| asselin | jeblair, happy lunar year :) | 01:22 |
| *** zz_dimtruck is now known as dimtruck | 01:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add generate-log-config command https://review.openstack.org/153904 | 01:23 |
| *** mayurig has joined #openstack-infra | 01:25 | |
| *** ivar-laz_ has quit IRC | 01:25 | |
| *** mayurig has quit IRC | 01:25 | |
| *** mtanino has quit IRC | 01:25 | |
| *** ivar-lazzaro has joined #openstack-infra | 01:26 | |
| *** ivar-lazzaro has quit IRC | 01:28 | |
| *** markvoelker has joined #openstack-infra | 01:29 | |
| *** ivar-lazzaro has joined #openstack-infra | 01:29 | |
| *** nelsnelson has quit IRC | 01:29 | |
| *** doug-fish has quit IRC | 01:31 | |
| *** EmilienM is now known as EmilienM|afk | 01:36 | |
| *** stevemar has joined #openstack-infra | 01:37 | |
| *** ivar-lazzaro has quit IRC | 01:40 | |
| *** crc32 has joined #openstack-infra | 01:41 | |
| *** ivar-lazzaro has joined #openstack-infra | 01:41 | |
| zaro | HAPPY LUNAR NEW YEAR! | 01:41 |
| clarkb | zaro: I just saw that space jam dunk | 01:41 |
| *** kgiusti has left #openstack-infra | 01:43 | |
| *** tiswanso has joined #openstack-infra | 01:47 | |
| *** tiswanso has quit IRC | 01:49 | |
| *** tiswanso has joined #openstack-infra | 01:50 | |
| anteaya | zaro: and to you | 01:53 |
| * anteaya is a sheep | 01:53 | |
| anteaya | zaro: what are you? | 01:53 |
| *** patrickeast has quit IRC | 01:54 | |
| *** devlaps has quit IRC | 01:55 | |
| *** claudiub has joined #openstack-infra | 02:00 | |
| *** krotscheck has joined #openstack-infra | 02:03 | |
| *** crc32 has quit IRC | 02:03 | |
| *** vilobhmm has quit IRC | 02:03 | |
| *** krtaylor has joined #openstack-infra | 02:08 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/nodepool: Reset the client object before creating the image https://review.openstack.org/74943 | 02:10 |
| *** krotscheck has quit IRC | 02:12 | |
| *** pcrews has joined #openstack-infra | 02:13 | |
| *** krotscheck has joined #openstack-infra | 02:13 | |
| anteaya | my goodness, two weeks of eating: https://en.wikipedia.org/wiki/Chinese_New_Year | 02:18 |
| *** arosen1 has joined #openstack-infra | 02:19 | |
| arosen1 | Hi I was wondering if anyone could tell me why my CI is going to pypi.python.org and pypi.dfw.opentack.org here: http://logs2.aaronorosen.com/01/157201/4/check/dsvm-tempest-full-congress-nodepool/ab67d5f/logs/devstacklog.txt.gz#_2015-02-19_02_11_21_174 | 02:20 |
| arosen1 | In my nodepool configurations I don't see any more references to pypi.dfw | 02:20 |
| clarkb | arosen1: depending on how old your images are puppet sets up pypi.dfw as a default | 02:21 |
| clarkb | I don't know why pypi.python.org is also used yet | 02:21 |
| *** ghostpl_ has joined #openstack-infra | 02:22 | |
| clarkb | arosen1: look in /etc/pip.conf $HOME/.pip/pip.conf | 02:22 |
| arosen1 | clarkb: the only place I have it set is: scripts/configure_mirror.sh:NODEPOOL_PYPI_MIRROR=${NODEPOOL_PYPI_MIRROR:-https://pypi.python.org/simple/} | 02:22 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Fix if check in neutron lib forward testing definition https://review.openstack.org/157235 | 02:22 |
| *** FallenPegasus has joined #openstack-infra | 02:23 | |
| arosen1 | clarkb: i should look on one of the jenkin slaves? | 02:23 |
| mtreinish | jhesketh, clarkb, fungi, anteaya: ^^^ That should be super simple | 02:23 |
| clarkb | arosen1: yes | 02:23 |
| clarkb | mtreinish: I have used my discretion to fast approve | 02:24 |
| mtreinish | clarkb: cool, thanks | 02:24 |
| arosen1 | clarkb: ah okay i see: [global] | 02:24 |
| arosen1 | index-url = http://pypi.dfw.openstack.org/simple/ there | 02:24 |
| *** MarkAtwood has quit IRC | 02:25 | |
| *** FallenPegasus is now known as MarkAtwood | 02:25 | |
| arosen1 | clarkb: seems weird because: http://paste.openstack.org/show/177303/ | 02:26 |
| clarkb | arosen1: images up to date? | 02:26 |
| arosen1 | clarkb: hrm one thing i'm wondering is why i only have one dib image here : http://paste.openstack.org/show/177304/ | 02:26 |
| *** ghostpl_ has quit IRC | 02:27 | |
| arosen1 | clarkb: when I added this new provider it didn't create a dib image just a normal image though it's the same config as the other provider | 02:27 |
| clarkb | arosen1: ya the idea is you can use a single dib image | 02:28 |
| arosen1 | clarkb: okay let me delete the dib image and have it rebuild it and see if that fixes this. | 02:29 |
| clarkb | arosen1: this way you don' do a couple hundred image builds a day like us :) also you get better consistency across providers | 02:29 |
| *** armax has quit IRC | 02:30 | |
| arosen1 | clarkb: ah okay i see so you don't build a new dib image for multiple providers. | 02:31 |
| clarkb | nope, it does mean a few things get a little weird but overall its been a big win | 02:31 |
| arosen1 | i went to update the image but i created a new provider to do so and i guess it pushed the same image. | 02:31 |
| arosen1 | I added a new provider only because there is this gluster bug in our cloud | 02:32 |
| clarkb | ya it will upload the same qcow2 | 02:32 |
| arosen1 | cool thanks hopefully this will fix it. | 02:32 |
| *** achanda has quit IRC | 02:33 | |
| *** otter768 has joined #openstack-infra | 02:38 | |
| *** salv-orlando has quit IRC | 02:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix if check in neutron lib forward testing definition https://review.openstack.org/157235 | 02:40 |
| *** achanda has joined #openstack-infra | 02:42 | |
| *** koolhead17 has joined #openstack-infra | 02:45 | |
| *** achanda has quit IRC | 02:46 | |
| arosen1 | clarkb: quick question. Rebuilding the image im seeing this error: INFO nodepool.image.build.devstack-trusty-dib: [ERROR] /tmp/image.ZsWcsDJr/mnt/opt/git/openstack-dev/devstack/stackrc:665 Could not determine host ip address. See local.conf for suggestions on setting HOST_IP. | 02:50 |
| clarkb | arosen1: I think there is a patch up to devstack to avoid that problem | 02:50 |
| clarkb | arosen1: otherwise check the HOST_IP stuff that devstack is looking for | 02:51 |
| arosen1 | clarkb: how do i set that for nodepool dib? | 02:51 |
| clarkb | arosen1: it will depend on what DEVSTACK is looking for | 02:51 |
| clarkb | er devstack | 02:51 |
| *** dims_ has quit IRC | 02:52 | |
| clarkb | its just a chroot so its your build node's interfaces that it iwll be looking at | 02:52 |
| arosen1 | clarkb: sorry i don't follow. This error is showing in the nodepool log. | 02:52 |
| *** mriedem has joined #openstack-infra | 02:53 | |
| clarkb | arosen1: yes, one of the things nodepool does is ask devstack for a list of files to cache. When it does that devstack expects to be able to determine your ip address | 02:53 |
| *** cody-somerville has joined #openstack-infra | 02:53 | |
| clarkb | arosen1: this is unnecessary for the functionality so there is a patch to devstack to stop requiring that, but as a workaround you just need to have devstack be able to determine the host's ip address | 02:53 |
| arosen1 | clarkb: okay let me see want i need to do to make that happen. | 02:54 |
| arosen1 | thanks. | 02:54 |
| *** rkukura has quit IRC | 02:54 | |
| *** rkukura has joined #openstack-infra | 02:55 | |
| *** hyakuhei has joined #openstack-infra | 03:02 | |
| *** achanda has joined #openstack-infra | 03:02 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/elastic-recheck: Add query for bug 1423427 https://review.openstack.org/157245 | 03:03 |
| openstack | bug 1423427 in OpenStack Compute (nova) "os-baremetal-node ext relies on presence of specific ironic node properties " [Undecided,New] https://launchpad.net/bugs/1423427 | 03:03 |
| *** spzala has joined #openstack-infra | 03:10 | |
| *** harlowja_ is now known as harlowja_away | 03:15 | |
| *** amitgandhinz has joined #openstack-infra | 03:15 | |
| arosen1 | clarkb: do you know why you guys don't hit this bug ? | 03:18 |
| clarkb | devstack can figure out what our hosts ip is | 03:19 |
| *** rmcall has quit IRC | 03:21 | |
| *** nelsnelson has joined #openstack-infra | 03:36 | |
| *** unicell1 has quit IRC | 03:37 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Cap irc < 11.1 due to ImportError https://review.openstack.org/157248 | 03:38 |
| *** salv-orlando has joined #openstack-infra | 03:41 | |
| *** achanda has quit IRC | 03:45 | |
| arosen1 | clarkb: ah i figured it out. It's because my host_ip is in the range of 10.0.0.0/24 as $FIXED_IPS | 03:47 |
| arosen1 | clarkb: i'm guessing there is no way to inject my own localrc or setting HOST_IP through nodepool. | 03:48 |
| *** yamamoto has joined #openstack-infra | 03:48 | |
| arosen1 | i'd rather not have to re ip the box unless i don't have to . | 03:48 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Cap irc < 11.1 due to ImportError https://review.openstack.org/157248 | 03:49 |
| *** Vivek_ has quit IRC | 03:49 | |
| *** spzala has quit IRC | 03:50 | |
| *** achanda has joined #openstack-infra | 03:52 | |
| *** dims__ has joined #openstack-infra | 03:52 | |
| *** koolhead17 has quit IRC | 03:52 | |
| *** yamahata has quit IRC | 03:54 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/elastic-recheck: Add query for bug 1423427 https://review.openstack.org/157245 | 03:54 |
| openstack | bug 1423427 in OpenStack Compute (nova) "os-baremetal-node ext relies on presence of specific ironic node properties " [Undecided,New] https://launchpad.net/bugs/1423427 | 03:54 |
| *** dims__ has quit IRC | 03:57 | |
| *** hyakuhei has left #openstack-infra | 03:58 | |
| *** tkelsey has joined #openstack-infra | 03:59 | |
| *** amotoki has joined #openstack-infra | 04:04 | |
| *** koolhead17 has joined #openstack-infra | 04:09 | |
| *** koolhead17 has joined #openstack-infra | 04:09 | |
| *** ghostpl_ has joined #openstack-infra | 04:11 | |
| *** otter768 has quit IRC | 04:11 | |
| *** teran has quit IRC | 04:13 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Uncap some test requirements https://review.openstack.org/157251 | 04:14 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Uncap some test requirements https://review.openstack.org/157251 | 04:15 |
| *** ghostpl_ has quit IRC | 04:16 | |
| *** achanda has quit IRC | 04:16 | |
| mordred | clarkb: gahhhhhh!!!!!!! | 04:16 |
| clarkb | indeed | 04:19 |
| *** claudiub has quit IRC | 04:19 | |
| *** yamamoto has quit IRC | 04:20 | |
| arosen1 | clarkb: doh i figured it out. I just needed to export HOST_IP="1..." in the process that nodepoold was running to get it to use that value instead. | 04:22 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for bug 1423427 https://review.openstack.org/157245 | 04:23 |
| openstack | bug 1423427 in OpenStack Compute (nova) "os-baremetal-node ext relies on presence of specific ironic node properties " [Undecided,New] https://launchpad.net/bugs/1423427 | 04:23 |
| *** ivar-laz_ has joined #openstack-infra | 04:25 | |
| *** coolsvap_ is now known as coolsvap | 04:28 | |
| *** btran has quit IRC | 04:28 | |
| *** ivar-lazzaro has quit IRC | 04:29 | |
| *** nelsnelson has quit IRC | 04:30 | |
| *** ivar-laz_ has quit IRC | 04:30 | |
| *** salv-orlando has quit IRC | 04:31 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for bug 1423427 https://review.openstack.org/157245 | 04:34 |
| openstack | bug 1423427 in OpenStack Compute (nova) "os-baremetal-node ext relies on presence of specific ironic node properties " [High,Confirmed] https://launchpad.net/bugs/1423427 | 04:34 |
| *** koolhead17 has quit IRC | 04:36 | |
| *** changbl has joined #openstack-infra | 04:38 | |
| mordred | clarkb: +++++++++ | 04:40 |
| *** woodster_ has quit IRC | 04:40 | |
| mordred | (We miss you in the bar) | 04:40 |
| clarkb | so I hear | 04:41 |
| clarkb | mordred I appointed crinkle my proxy | 04:41 |
| clarkb | and instructed on proper cloud init stance | 04:41 |
| *** markvoelker has quit IRC | 04:42 | |
| *** miqui has joined #openstack-infra | 04:42 | |
| *** markvoelker has joined #openstack-infra | 04:42 | |
| *** markvoelker has quit IRC | 04:46 | |
| mordred | clarkb: I may be coming around .. You haven't taken the right stance on a different topic | 04:47 |
| *** dimtruck is now known as zz_dimtruck | 04:47 | |
| clarkb | whats the other thing | 04:47 |
| clarkb | the ansible stuff? I think I have come around. facter does the right thing | 04:51 |
| *** BharatK has joined #openstack-infra | 04:52 | |
| *** amitgandhinz has quit IRC | 04:52 | |
| *** miqui has quit IRC | 04:52 | |
| *** miqui has joined #openstack-infra | 04:53 | |
| mordred | clarkb: I'm not telling, I'm assuming we are in complete agreement | 04:56 |
| mordred | :) | 04:56 |
| clarkb | ha | 04:56 |
| *** bhunter71 has quit IRC | 04:58 | |
| *** asselin_ has joined #openstack-infra | 05:00 | |
| *** camunoz has quit IRC | 05:01 | |
| mordred | #goduke | 05:02 |
| *** vigneshvar has joined #openstack-infra | 05:02 | |
| *** asselin_ has quit IRC | 05:02 | |
| *** mriedem has quit IRC | 05:05 | |
| *** coolsvap is now known as coolsvap_ | 05:05 | |
| *** yamamoto has joined #openstack-infra | 05:07 | |
| *** garyh has quit IRC | 05:09 | |
| *** mriedem has joined #openstack-infra | 05:10 | |
| *** mriedem has quit IRC | 05:11 | |
| *** koolhead17 has joined #openstack-infra | 05:11 | |
| *** markvoelker has joined #openstack-infra | 05:13 | |
| *** camunoz has joined #openstack-infra | 05:14 | |
| *** coolsvap_ is now known as coolsvap | 05:15 | |
| *** achanda has joined #openstack-infra | 05:16 | |
| *** tkelsey has quit IRC | 05:17 | |
| *** markvoelker has quit IRC | 05:17 | |
| *** achanda has quit IRC | 05:18 | |
| *** achanda has joined #openstack-infra | 05:18 | |
| *** Sukhdev has joined #openstack-infra | 05:19 | |
| *** yamahata has joined #openstack-infra | 05:19 | |
| dstufft | mordred: what dates are you in philly again | 05:19 |
| dstufft | I forget | 05:20 |
| dstufft | march something | 05:20 |
| mordred | dstufft: march 9-10 | 05:20 |
| dstufft | mordred: cool thanks | 05:23 |
| *** koolhead_ has joined #openstack-infra | 05:24 | |
| *** koolhead17 has quit IRC | 05:27 | |
| *** mfink_ has joined #openstack-infra | 05:28 | |
| *** ParsectiX has quit IRC | 05:30 | |
| *** ParsectiX has joined #openstack-infra | 05:31 | |
| openstackgerrit | Khai Do proposed stackforge/gertty: Disable InsecureRequestWarning https://review.openstack.org/157259 | 05:33 |
| *** gyee has quit IRC | 05:36 | |
| *** vigneshvar has quit IRC | 05:36 | |
| *** vigneshvar has joined #openstack-infra | 05:37 | |
| *** sandywalsh has joined #openstack-infra | 05:37 | |
| *** sandywalsh_ has quit IRC | 05:40 | |
| *** amotoki_ has joined #openstack-infra | 05:44 | |
| *** achanda has quit IRC | 05:45 | |
| openstackgerrit | Igor Belikov proposed openstack-infra/jenkins-job-builder: Add 'readable-message' parameter to gerrit trigger https://review.openstack.org/152641 | 05:46 |
| *** tkelsey has joined #openstack-infra | 05:47 | |
| *** amotoki has quit IRC | 05:47 | |
| *** zigo has quit IRC | 05:54 | |
| *** zigo has joined #openstack-infra | 05:55 | |
| *** achanda has joined #openstack-infra | 05:56 | |
| *** ParsectiX has quit IRC | 05:58 | |
| *** ParsectiX has joined #openstack-infra | 05:58 | |
| *** ghostpl_ has joined #openstack-infra | 05:59 | |
| *** woodster_ has joined #openstack-infra | 06:00 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 06:01 | |
| *** ghostpl_ has quit IRC | 06:04 | |
| *** hamzy has quit IRC | 06:10 | |
| *** cnesa1 has quit IRC | 06:10 | |
| *** garyh has joined #openstack-infra | 06:10 | |
| *** otter768 has joined #openstack-infra | 06:12 | |
| *** Sukhdev has quit IRC | 06:12 | |
| *** markvoelker has joined #openstack-infra | 06:13 | |
| *** hamzy has joined #openstack-infra | 06:14 | |
| *** mfink_ has quit IRC | 06:15 | |
| *** salv-orlando has joined #openstack-infra | 06:15 | |
| *** otter768 has quit IRC | 06:17 | |
| *** mfink_ has joined #openstack-infra | 06:17 | |
| *** markvoelker has quit IRC | 06:18 | |
| *** Sukhdev has joined #openstack-infra | 06:21 | |
| *** tkelsey has quit IRC | 06:22 | |
| *** signed8bit has joined #openstack-infra | 06:22 | |
| *** signed8bit_ZZZzz has quit IRC | 06:23 | |
| *** belmoreira has joined #openstack-infra | 06:26 | |
| *** miqui has quit IRC | 06:26 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: Allow for git base override https://review.openstack.org/151034 | 06:28 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: post_test_hook is not executed when the gate test returns an error https://review.openstack.org/148863 | 06:32 |
| *** dannywilson has joined #openstack-infra | 06:34 | |
| *** camunoz has quit IRC | 06:38 | |
| *** dannywilson has quit IRC | 06:41 | |
| *** dannywilson has joined #openstack-infra | 06:42 | |
| *** ildikov has quit IRC | 06:47 | |
| *** tkelsey has joined #openstack-infra | 06:49 | |
| *** yamahata has quit IRC | 06:51 | |
| *** yamahata has joined #openstack-infra | 06:52 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: xUnit publisher: add option to skip if there are no tests files https://review.openstack.org/134923 | 06:52 |
| *** harlowja_at_home has joined #openstack-infra | 06:52 | |
| *** tkelsey has quit IRC | 07:03 | |
| *** mrunge has joined #openstack-infra | 07:03 | |
| *** mrunge has quit IRC | 07:03 | |
| *** mrunge has joined #openstack-infra | 07:03 | |
| *** Sukhdev has quit IRC | 07:04 | |
| *** mrmartin has joined #openstack-infra | 07:06 | |
| *** afazekas has joined #openstack-infra | 07:08 | |
| *** hdd has joined #openstack-infra | 07:08 | |
| *** Ryan_Lane has joined #openstack-infra | 07:11 | |
| *** harlowja_at_home has quit IRC | 07:13 | |
| *** markvoelker has joined #openstack-infra | 07:14 | |
| *** Sukhdev has joined #openstack-infra | 07:16 | |
| *** doobi-sham-95717 has quit IRC | 07:17 | |
| *** chlong has quit IRC | 07:19 | |
| *** markvoelker has quit IRC | 07:19 | |
| *** arosen1 has quit IRC | 07:20 | |
| *** MarkAtwood has quit IRC | 07:20 | |
| *** MarkAtwood has joined #openstack-infra | 07:21 | |
| *** Ryan_Lane has quit IRC | 07:22 | |
| *** salv-orlando has quit IRC | 07:26 | |
| *** ildikov has joined #openstack-infra | 07:29 | |
| *** dims__ has joined #openstack-infra | 07:30 | |
| *** zz_dimtruck is now known as dimtruck | 07:31 | |
| *** ZZelle has quit IRC | 07:32 | |
| *** ZZelle has joined #openstack-infra | 07:33 | |
| *** Sukhdev has quit IRC | 07:34 | |
| *** dims__ has quit IRC | 07:35 | |
| *** hdd has quit IRC | 07:37 | |
| *** cipcosma has joined #openstack-infra | 07:39 | |
| *** sputnik13 has joined #openstack-infra | 07:48 | |
| *** ghostpl_ has joined #openstack-infra | 07:48 | |
| *** sdake_ has joined #openstack-infra | 07:49 | |
| *** cinerama has quit IRC | 07:49 | |
| *** ghostpl_ has quit IRC | 07:53 | |
| *** vponomaryov has quit IRC | 07:53 | |
| *** vponomaryov has joined #openstack-infra | 07:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Bandit is a stackforge project https://review.openstack.org/156774 | 07:54 |
| *** jgallard__ has joined #openstack-infra | 07:55 | |
| openstackgerrit | Merged openstack-infra/project-config: VPNaaS: make functional test voting https://review.openstack.org/156281 | 07:59 |
| *** harlowja_away has quit IRC | 08:03 | |
| *** dimtruck is now known as zz_dimtruck | 08:07 | |
| *** doude has quit IRC | 08:08 | |
| *** arxcruz has joined #openstack-infra | 08:08 | |
| *** cnesa has joined #openstack-infra | 08:09 | |
| *** sergsh has joined #openstack-infra | 08:09 | |
| *** doude has joined #openstack-infra | 08:13 | |
| *** otter768 has joined #openstack-infra | 08:13 | |
| *** markvoelker has joined #openstack-infra | 08:15 | |
| *** otter768 has quit IRC | 08:17 | |
| *** Ala has joined #openstack-infra | 08:19 | |
| *** markvoelker has quit IRC | 08:20 | |
| *** mpavone has joined #openstack-infra | 08:21 | |
| *** stevemar has quit IRC | 08:22 | |
| *** HeOS has quit IRC | 08:23 | |
| *** scheuran has joined #openstack-infra | 08:23 | |
| *** yamamoto has quit IRC | 08:24 | |
| *** jcoufal has joined #openstack-infra | 08:25 | |
| *** salv-orlando has joined #openstack-infra | 08:25 | |
| *** achanda has quit IRC | 08:29 | |
| *** andreykurilin_ has joined #openstack-infra | 08:29 | |
| *** achanda has joined #openstack-infra | 08:29 | |
| *** sdake__ has joined #openstack-infra | 08:29 | |
| *** achanda has quit IRC | 08:30 | |
| *** woodster_ has quit IRC | 08:30 | |
| *** sdake_ has quit IRC | 08:33 | |
| *** koolhead_ has quit IRC | 08:35 | |
| *** luqas___ has quit IRC | 08:36 | |
| *** dtantsur|afk is now known as dtantsur | 08:41 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for 'All Nodes For Label Factory' https://review.openstack.org/149359 | 08:41 |
| *** jistr has joined #openstack-infra | 08:42 | |
| *** sergsh has quit IRC | 08:43 | |
| *** hamzy has quit IRC | 08:43 | |
| *** hamzy has joined #openstack-infra | 08:44 | |
| *** zz_avozza is now known as zz_zz_avozza | 08:44 | |
| *** sergsh has joined #openstack-infra | 08:45 | |
| *** luqas___ has joined #openstack-infra | 08:47 | |
| *** cinerama has joined #openstack-infra | 08:48 | |
| *** markus_z has joined #openstack-infra | 08:48 | |
| *** MaxV has joined #openstack-infra | 08:50 | |
| *** deepakcs has joined #openstack-infra | 08:52 | |
| deepakcs | clarkb, ping | 08:52 |
| *** salv-orlando has quit IRC | 08:52 | |
| *** salv-orlando has joined #openstack-infra | 08:52 | |
| AJaeger | deepakcs, I expect clarkb to sleep... | 08:52 |
| *** teran has joined #openstack-infra | 08:52 | |
| deepakcs | AJaeger, sure :) I wanted to see if someone has more details on https://review.openstack.org/#/c/157213/ | 08:52 |
| deepakcs | AJaeger, you seem to +2 it.. myself and BharatK are working on the glusterfs job. We are not clear on what 'partway' in the commit msg mean ? | 08:53 |
| AJaeger | deepakcs, just check the experimental queue at http://status.openstack.org/zuul/ | 08:53 |
| deepakcs | AJaeger, ok.. checking now | 08:53 |
| *** teran_ has joined #openstack-infra | 08:53 | |
| AJaeger | there'S a job running for 40+ hours - and that because of glusterfs | 08:54 |
| deepakcs | AJaeger, is it bcos of 'check experimental' in https://review.openstack.org/152401 ? | 08:55 |
| * deepakcs still trying to understand why it would take 40+ hours. local tempest runs go fine | 08:56 | |
| AJaeger | deepakcs, also scrollback or read the public logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-02-19.log for details | 08:56 |
| *** jp_at_hp has joined #openstack-infra | 08:56 | |
| *** teran has quit IRC | 08:57 | |
| AJaeger | deepakcs, as said in the commit message: This comes from "check experimental" and somehow the glusterfs job stops in the middle and then gets restarted - in an endless loop | 08:57 |
| AJaeger | We can't have that ;( Please talk with clarkb or fungi at US time again on how to fix this. | 08:57 |
| deepakcs | AJaeger, i read the logs, it says its failign in the object tests | 08:58 |
| *** andreykurilin_ has quit IRC | 08:58 | |
| AJaeger | deepakcs, I can't help further here... | 08:58 |
| deepakcs | AJaeger, any idea how to add a job to run tempest volume tests only ? | 08:59 |
| AJaeger | deepakcs, there're ways to limit what tempest tests are run, check what other jobs do - in project-config/jenkins/jobs/* | 09:00 |
| deepakcs | AJaeger, thanks, will do | 09:00 |
| *** xyang1 has quit IRC | 09:03 | |
| *** jlibosva has joined #openstack-infra | 09:03 | |
| *** jcoufal_ has joined #openstack-infra | 09:05 | |
| AJaeger | deepakcs, the top most experimental job just got into the next iteration ;( | 09:07 |
| AJaeger | and I now approved that change to get it stopped | 09:07 |
| AJaeger | deepakcs, once you're sure everything is working as it should, we can enable the test again | 09:08 |
| *** jcoufal has quit IRC | 09:08 | |
| deepakcs | AJaeger, you mean https://review.openstack.org/#/c/141424/ is running for 40+ hours. I am still trying to understand how is that related to glusterfs job | 09:08 |
| AJaeger | deepakcs, look at http://status.openstack.org/zuul/ | 09:08 |
| AJaeger | 141424 is the first change | 09:08 |
| AJaeger | If you monitor the glusterfs job, you see it'S the only job that has not passed | 09:09 |
| AJaeger | And if you look at it over time, you see that it runs for a long time - and then starts again, as just now | 09:09 |
| *** dizquierdo has joined #openstack-infra | 09:10 | |
| AJaeger | I can't help you debug why this happens - but what happens is that the glusterfs job suddenly dies and then zuul starts it again and again and again - now since 40 hours. | 09:10 |
| deepakcs | AJaeger, i see 141424 change, but what do I need to do to monitor glusterfs job ? The zuul links points me back to the patch | 09:11 |
| AJaeger | deepakcs, check the link of the glusterfs job, it's currently: https://jenkins07.openstack.org/job/check-tempest-dsvm-full-glusterfs-centos7/36/ | 09:11 |
| AJaeger | Or check one of the others - each glusterfs job is a link | 09:11 |
| *** mpaolino has joined #openstack-infra | 09:11 | |
| *** ghostpl_ has joined #openstack-infra | 09:12 | |
| *** vigneshvar has quit IRC | 09:12 | |
| deepakcs | AJaeger, clicking on check-tempest-dsvm-full-glusterfs-centos7 #36: None gets me to the same page again :) | 09:13 |
| openstackgerrit | Merged openstack-infra/project-config: Remove the tempest glusterfs job https://review.openstack.org/157213 | 09:13 |
| deepakcs | let me check the console link | 09:13 |
| AJaeger | deepakcs, yes, in jenkins you need to click on console output | 09:13 |
| *** vigneshvar has joined #openstack-infra | 09:14 | |
| deepakcs | AJaeger, thanks, looking at that.. new to jenkins/infra... so trying to make my way to the error. I still haven't been able to see the actual error on the node that causes zuul to think its gone and hence restartsit | 09:14 |
| AJaeger | deepakcs, there are more experimental glusterfs jobs running if you need more data ;) | 09:15 |
| deepakcs | AJaeger, yeah i am still a bit lost on what exactly is happening inside the job. Shall i just start from the jenkins dashboard to get a list of all experimental glustefs jobs ? | 09:16 |
| AJaeger | deepakcs, you might need to discuss with clarkb and fungi later today (US time - so UTC 15:00 or later). | 09:16 |
| deepakcs | AJaeger, ok will do | 09:16 |
| AJaeger | start from the zuul page - we have several jenkins running and zuul is the best way to guide you | 09:16 |
| *** markvoelker has joined #openstack-infra | 09:16 | |
| *** skolekonov has joined #openstack-infra | 09:18 | |
| *** dannywilson has quit IRC | 09:18 | |
| *** derekh has joined #openstack-infra | 09:19 | |
| *** markvoelker has quit IRC | 09:21 | |
| *** ildikov has quit IRC | 09:27 | |
| *** salv-orlando has quit IRC | 09:29 | |
| *** yamamoto has joined #openstack-infra | 09:31 | |
| *** salv-orlando has joined #openstack-infra | 09:33 | |
| *** ssam2 has joined #openstack-infra | 09:34 | |
| *** HeOS has joined #openstack-infra | 09:35 | |
| *** sdake_ has joined #openstack-infra | 09:36 | |
| *** yamahata has quit IRC | 09:39 | |
| *** ildikov has joined #openstack-infra | 09:40 | |
| *** sdake__ has quit IRC | 09:40 | |
| *** mpaolino has quit IRC | 09:43 | |
| *** viktors|afk is now known as viktors | 09:47 | |
| *** yamamoto has quit IRC | 09:50 | |
| *** zz_zz_avozza is now known as avozza | 09:52 | |
| *** hashar has joined #openstack-infra | 09:54 | |
| mescanef | hi there, ihve created a patch for adding new repository on stackforge for fuel plugin (https://review.openstack.org/#/c/153173/) | 09:55 |
| mescanef | would like to be added to the core-group | 09:55 |
| mescanef | please add me :-) | 09:55 |
| *** mpaolino has joined #openstack-infra | 09:56 | |
| *** yamamoto has joined #openstack-infra | 10:00 | |
| AJaeger | mescanef, which repository exactly and what's your email? | 10:01 |
| mescanef | AJaeger: fuel-plugin-contrail | 10:02 |
| mescanef | AJaeger: mmatuszkowiak@mirantis.com | 10:02 |
| mescanef | "stackforge/fuel-plugin-contrail" ;-) | 10:02 |
| AJaeger | mescanef, Let's ask fungi or clarkb to do this later today (US time). They normally scroll back and should see the above lines. | 10:03 |
| mescanef | :-)) ok TY | 10:03 |
| *** teran_ has quit IRC | 10:08 | |
| *** yamamoto has quit IRC | 10:10 | |
| *** hashar_ has joined #openstack-infra | 10:13 | |
| *** amotoki_ has quit IRC | 10:14 | |
| *** otter768 has joined #openstack-infra | 10:14 | |
| *** hashar has quit IRC | 10:17 | |
| *** markvoelker has joined #openstack-infra | 10:17 | |
| *** amotoki has joined #openstack-infra | 10:18 | |
| cody-somerville | Hey folks. | 10:18 |
| cody-somerville | Are you seeing timeouts on sshing into nodes in hp cloud on spawn? | 10:19 |
| *** otter768 has quit IRC | 10:19 | |
| *** dtantsur is now known as dtantsur|bbl | 10:21 | |
| *** ihrachyshka has joined #openstack-infra | 10:21 | |
| *** koolhead17 has joined #openstack-infra | 10:21 | |
| *** e0ne has joined #openstack-infra | 10:21 | |
| *** yamamoto has joined #openstack-infra | 10:21 | |
| *** markvoelker has quit IRC | 10:22 | |
| *** briancline has quit IRC | 10:22 | |
| *** hogepodge has quit IRC | 10:22 | |
| *** briancline has joined #openstack-infra | 10:23 | |
| *** teran has joined #openstack-infra | 10:24 | |
| *** hogepodge has joined #openstack-infra | 10:24 | |
| *** teran has quit IRC | 10:28 | |
| *** yamamoto has quit IRC | 10:31 | |
| *** nikil89_ has joined #openstack-infra | 10:35 | |
| *** nikil89_ is now known as naggappan | 10:35 | |
| *** rlandy has joined #openstack-infra | 10:38 | |
| *** e0ne is now known as e0ne_ | 10:39 | |
| *** e0ne_ is now known as e0ne | 10:39 | |
| *** yamamoto has joined #openstack-infra | 10:39 | |
| *** ldnunes has joined #openstack-infra | 10:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Updates training guides jobs https://review.openstack.org/152118 | 10:40 |
| *** rfolco has joined #openstack-infra | 10:42 | |
| *** jcoufal_ has quit IRC | 10:45 | |
| *** jcoufal has joined #openstack-infra | 10:46 | |
| *** yfried_ is now known as yfried|afk | 10:46 | |
| *** pelix has joined #openstack-infra | 10:46 | |
| *** yamamoto has quit IRC | 10:47 | |
| *** yfried|afk is now known as yfried_ | 10:48 | |
| *** dims__ has joined #openstack-infra | 10:50 | |
| *** hashar_ has quit IRC | 10:52 | |
| *** e0ne is now known as e0ne_ | 10:53 | |
| *** e0ne_ is now known as e0ne | 10:54 | |
| *** dims__ has quit IRC | 10:54 | |
| *** teran has joined #openstack-infra | 11:03 | |
| *** yfried_ is now known as yfried|afk | 11:04 | |
| *** mpaolino has quit IRC | 11:05 | |
| *** avozza is now known as zz_avozza | 11:07 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Uncap some test requirements https://review.openstack.org/157251 | 11:08 |
| *** salv-orlando has quit IRC | 11:11 | |
| *** salv-orlando has joined #openstack-infra | 11:12 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add hasExtension method to check cloud capabilities https://review.openstack.org/156954 | 11:13 |
| *** hashar has joined #openstack-infra | 11:14 | |
| *** koolhead17 has quit IRC | 11:14 | |
| *** zz_avozza is now known as avozza | 11:16 | |
| openstackgerrit | Sergey Lukjanov proposed openstack-infra/project-config: Clone using git.o.o instead of directly from gerrit https://review.openstack.org/157330 | 11:17 |
| *** kashyap has left #openstack-infra | 11:17 | |
| *** markvoelker has joined #openstack-infra | 11:18 | |
| *** mpaolino has joined #openstack-infra | 11:19 | |
| *** hichihara has quit IRC | 11:19 | |
| *** jedimike has joined #openstack-infra | 11:20 | |
| *** mpaolino has quit IRC | 11:21 | |
| *** markvoelker has quit IRC | 11:23 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/infra-specs: Spec for nodepool image update retries https://review.openstack.org/155305 | 11:23 |
| *** yfried|afk is now known as yfried_ | 11:24 | |
| *** salv-orlando has quit IRC | 11:31 | |
| *** aysyd has joined #openstack-infra | 11:31 | |
| *** salv-orlando has joined #openstack-infra | 11:31 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/infra-specs: Add nodepool image testing spec. https://review.openstack.org/139598 | 11:32 |
| *** e0ne is now known as e0ne_ | 11:34 | |
| *** deepakcs has quit IRC | 11:38 | |
| *** pc_m has joined #openstack-infra | 11:40 | |
| *** hashar_ has joined #openstack-infra | 11:40 | |
| *** chlong has joined #openstack-infra | 11:41 | |
| *** hashar has quit IRC | 11:43 | |
| *** nicodemos has joined #openstack-infra | 11:44 | |
| *** vigneshvar has quit IRC | 11:49 | |
| *** marcusvrn1 has quit IRC | 11:49 | |
| *** enikanorov_ has quit IRC | 11:49 | |
| *** enikanorov has joined #openstack-infra | 11:50 | |
| *** yamamoto has joined #openstack-infra | 11:50 | |
| *** marcusvrn has joined #openstack-infra | 11:50 | |
| *** alexpilotti has joined #openstack-infra | 11:51 | |
| *** marcusvrn1 has joined #openstack-infra | 11:52 | |
| *** yamamoto has quit IRC | 11:55 | |
| *** marcusvrn has quit IRC | 11:56 | |
| *** dtantsur|bbl is now known as dtantsur | 11:56 | |
| *** adalba has quit IRC | 11:59 | |
| *** adalba has joined #openstack-infra | 11:59 | |
| *** agentle_ has quit IRC | 12:06 | |
| *** koolhead17 has joined #openstack-infra | 12:08 | |
| *** oomichi has joined #openstack-infra | 12:08 | |
| *** oomichi has quit IRC | 12:09 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add 'readable-message' parameter to gerrit trigger https://review.openstack.org/152641 | 12:09 |
| *** mwagner_lap has quit IRC | 12:10 | |
| *** e0ne_ is now known as e0ne | 12:10 | |
| *** otter768 has joined #openstack-infra | 12:15 | |
| *** ihrachyshka has quit IRC | 12:16 | |
| *** yamamoto has joined #openstack-infra | 12:16 | |
| *** dannywilson has joined #openstack-infra | 12:19 | |
| *** markvoelker has joined #openstack-infra | 12:19 | |
| *** otter768 has quit IRC | 12:20 | |
| *** yfried_ is now known as yfried|afk | 12:21 | |
| *** woodster_ has joined #openstack-infra | 12:22 | |
| *** dannywilson has quit IRC | 12:23 | |
| *** markvoelker has quit IRC | 12:24 | |
| *** hashar_ has quit IRC | 12:28 | |
| *** doude has quit IRC | 12:30 | |
| *** salv-orlando has quit IRC | 12:31 | |
| *** hashar has joined #openstack-infra | 12:31 | |
| *** salv-orlando has joined #openstack-infra | 12:31 | |
| *** koolhead17 has quit IRC | 12:33 | |
| *** doude has joined #openstack-infra | 12:33 | |
| *** salv-orlando has quit IRC | 12:40 | |
| *** cdent has joined #openstack-infra | 12:40 | |
| *** salv-orlando has joined #openstack-infra | 12:41 | |
| *** avozza is now known as zz_avozza | 12:42 | |
| *** psedlak has quit IRC | 12:43 | |
| *** amotoki has quit IRC | 12:43 | |
| *** dims__ has joined #openstack-infra | 12:44 | |
| *** psedlak has joined #openstack-infra | 12:45 | |
| *** yfried|afk is now known as yfried_ | 12:45 | |
| *** salv-orlando has quit IRC | 12:49 | |
| *** tiswanso has quit IRC | 12:53 | |
| *** dkranz has quit IRC | 12:54 | |
| *** spredzy|afk is now known as spredzy | 12:57 | |
| *** Hal has joined #openstack-infra | 12:58 | |
| *** EmilienM|afk is now known as EmilienM | 12:58 | |
| *** kashyap has joined #openstack-infra | 12:58 | |
| *** Hal is now known as Guest43224 | 12:59 | |
| *** marcusvrn has joined #openstack-infra | 13:01 | |
| *** yfried_ is now known as yfried|afk | 13:02 | |
| *** markvoelker has joined #openstack-infra | 13:02 | |
| kashyap | rfolco, Redirecting discussion from #openstack-dev, | 13:03 |
| *** marcusvrn1 has quit IRC | 13:04 | |
| kashyap | Yeah - for minimal installs - I can't recommend enough `virt-builder`, I personally think it is a lot more secure and convenient than 'disk-image-builder' | 13:04 |
| kashyap | Here's some convenient CLI on usage - https://kashyapc.fedorapeople.org/virt/virt-builder-usage.txt | 13:04 |
| *** alexpilotti has quit IRC | 13:05 | |
| *** sputnik13 has quit IRC | 13:05 | |
| rfolco | kashyap, hmm perhaps its a good time for moving to f21 ? | 13:06 |
| kashyap | rfolco, Sure. I've been testing/developing (and DevStack too) on F21 for about 6 months already | 13:06 |
| kashyap | rfolco, Especially, for a server/minimal-core style stuff, it should not be much disruptive at all | 13:07 |
| kashyap | s/much/any | 13:07 |
| *** vigneshvar has joined #openstack-infra | 13:08 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/storyboard: Session per request https://review.openstack.org/155751 | 13:09 |
| rfolco | kashyap, nice, the reason for rebuilding my base image is that f20 updates repo is broken for perl... and I had pre-cached many packages from the updates repo so I can't downgrade all them now...are your aware of that problem ? | 13:09 |
| *** zz_avozza is now known as avozza | 13:10 | |
| *** koolhead17 has joined #openstack-infra | 13:11 | |
| *** koolhead17 has joined #openstack-infra | 13:11 | |
| kashyap | rfolco, 1 sec, having a parallel discussion elsewhere on #openstack-nova | 13:11 |
| *** BharatK has quit IRC | 13:12 | |
| rfolco | kashyap, sure :) | 13:13 |
| BobBall | Does Zuul have this magic way of finding changesets that it should check but it hasn't voted on? (i.e. missed changesets)... My external CI seems to have queued up a whole bunch of old jobs from openstack-dev/sandbox without there being resubmits of the jobs or comments | 13:16 |
| openstackgerrit | Merged openstack-infra/release-tools: Skip tag and push if tag exists https://review.openstack.org/151282 | 13:17 |
| BobBall | oh - ignore me... it was a "depends on" chain. Ouch. | 13:19 |
| *** mbacchi has joined #openstack-infra | 13:21 | |
| *** Guest43224 has quit IRC | 13:21 | |
| *** yfried|afk is now known as yfried_ | 13:22 | |
| *** cnesa has quit IRC | 13:23 | |
| *** dkliban_afk is now known as dkliban | 13:23 | |
| *** cnesa has joined #openstack-infra | 13:24 | |
| *** bswartz has quit IRC | 13:27 | |
| *** cnesa has quit IRC | 13:28 | |
| *** kgiusti has joined #openstack-infra | 13:29 | |
| openstackgerrit | Merged openstack-infra/storyboard: Removed unsupported python versions from setup.cfg https://review.openstack.org/154582 | 13:31 |
| *** yfried_ is now known as yfried|afk | 13:32 | |
| *** nicodemos has quit IRC | 13:34 | |
| *** jistr_ has joined #openstack-infra | 13:37 | |
| kashyap | rfolco, I don't know about any Perl broken updates | 13:37 |
| *** ociuhandu has joined #openstack-infra | 13:37 | |
| rfolco | kashyap, not sure if it affects ppc64 only... | 13:38 |
| *** jistr has quit IRC | 13:39 | |
| *** markvoelker_ has joined #openstack-infra | 13:42 | |
| *** markvoelker has quit IRC | 13:42 | |
| *** yfried|afk is now known as yfried_ | 13:45 | |
| *** Hal has joined #openstack-infra | 13:45 | |
| *** Hal is now known as Guest21555 | 13:46 | |
| *** koolhead17 has quit IRC | 13:46 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Updated library versions. https://review.openstack.org/155942 | 13:46 |
| *** hogepodge has quit IRC | 13:47 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Added root state https://review.openstack.org/155943 | 13:47 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/python-storyboardclient: Improved exception handling https://review.openstack.org/153217 | 13:47 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Session checks resolution against session info service. https://review.openstack.org/155944 | 13:48 |
| *** briancline has quit IRC | 13:48 | |
| *** jistr_ is now known as jistr | 13:48 | |
| *** briancline has joined #openstack-infra | 13:49 | |
| *** hogepodge has joined #openstack-infra | 13:49 | |
| *** salv-orlando has joined #openstack-infra | 13:50 | |
| *** yamamoto has quit IRC | 13:51 | |
| openstackgerrit | Merged openstack-infra/python-storyboardclient: Python34 Fixes. https://review.openstack.org/157192 | 13:52 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:52 | |
| openstackgerrit | Merged openstack-infra/python-storyboardclient: Removed py33, pypy from python-storyboardclient https://review.openstack.org/157193 | 13:52 |
| *** ihrachyshka has joined #openstack-infra | 13:52 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens https://review.openstack.org/153591 | 13:55 |
| *** mkerrin has quit IRC | 13:55 | |
| *** jklare has quit IRC | 13:56 | |
| *** hashar has quit IRC | 13:57 | |
| *** BharatK has joined #openstack-infra | 13:57 | |
| *** jklare has joined #openstack-infra | 13:58 | |
| *** claudiub has joined #openstack-infra | 13:59 | |
| *** bhunter71 has joined #openstack-infra | 14:00 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens https://review.openstack.org/153591 | 14:01 |
| *** bswartz has joined #openstack-infra | 14:01 | |
| *** jklare has quit IRC | 14:02 | |
| BobBall | How do I specify credentials for zuul-swift-upload-logs? Seems I need to set environment variables like a HMAC but I can't find in the infra repos where this is created | 14:02 |
| *** sdake_ has quit IRC | 14:03 | |
| *** jklare has joined #openstack-infra | 14:03 | |
| *** vigneshvar has quit IRC | 14:04 | |
| *** mpaolino has joined #openstack-infra | 14:04 | |
| *** pblaho has quit IRC | 14:07 | |
| *** BharatK has quit IRC | 14:08 | |
| *** naggappan has quit IRC | 14:08 | |
| *** dkranz has joined #openstack-infra | 14:09 | |
| *** weshay has joined #openstack-infra | 14:11 | |
| *** koolhead17 has joined #openstack-infra | 14:11 | |
| *** ddieterl_ has quit IRC | 14:12 | |
| *** xyang1 has joined #openstack-infra | 14:15 | |
| *** otter768 has joined #openstack-infra | 14:16 | |
| *** dustins has joined #openstack-infra | 14:20 | |
| *** BharatK has joined #openstack-infra | 14:20 | |
| *** otter768 has quit IRC | 14:20 | |
| anteaya | wznoinsk: are you in the nova meeting? | 14:23 |
| wznoinsk | I am | 14:23 |
| anteaya | okay I hadn't seen your hand up, so I thought I would check | 14:23 |
| *** mriedem has joined #openstack-infra | 14:27 | |
| *** gulic has joined #openstack-infra | 14:27 | |
| *** ildikov has quit IRC | 14:28 | |
| *** woodster_ has quit IRC | 14:30 | |
| openstackgerrit | Max Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper https://review.openstack.org/156681 | 14:30 |
| openstackgerrit | Merged openstack/requirements: Remove configobj https://review.openstack.org/144363 | 14:31 |
| *** EmilienM is now known as EmilienM|afk | 14:32 | |
| openstackgerrit | Merged openstack/requirements: remove python-openstackclient https://review.openstack.org/155991 | 14:34 |
| *** hogepodge has quit IRC | 14:36 | |
| *** briancline has quit IRC | 14:37 | |
| *** markvoelker_ has quit IRC | 14:38 | |
| *** briancline has joined #openstack-infra | 14:38 | |
| *** hogepodge has joined #openstack-infra | 14:39 | |
| *** salv-orlando has quit IRC | 14:39 | |
| *** markvoelker has joined #openstack-infra | 14:39 | |
| *** sabeen has joined #openstack-infra | 14:39 | |
| *** yfried_ is now known as yfried|afk | 14:40 | |
| *** salv-orlando has joined #openstack-infra | 14:40 | |
| *** yfried|afk is now known as yfried_ | 14:40 | |
| *** woodster_ has joined #openstack-infra | 14:41 | |
| *** koolhead17 has quit IRC | 14:42 | |
| *** tiswanso has joined #openstack-infra | 14:42 | |
| *** dustins has quit IRC | 14:43 | |
| *** bswartz has quit IRC | 14:43 | |
| *** marcusvrn1 has joined #openstack-infra | 14:45 | |
| *** marcusvrn has quit IRC | 14:46 | |
| *** marcusvrn has joined #openstack-infra | 14:46 | |
| anteaya | wznoinsk: were you going to introduce yourself? | 14:47 |
| *** marcusvrn has quit IRC | 14:47 | |
| *** marcusvrn has joined #openstack-infra | 14:47 | |
| *** zz_dimtruck is now known as dimtruck | 14:49 | |
| *** marcusvrn1 has quit IRC | 14:49 | |
| *** deepakcs has joined #openstack-infra | 14:51 | |
| *** MaxV has quit IRC | 14:51 | |
| deepakcs | clarkb, fungi ping | 14:51 |
| anteaya | deepakcs: best to state what you need | 14:52 |
| anteaya | others may be available to help | 14:52 |
| deepakcs | anteaya, sure | 14:52 |
| deepakcs | clarkb, fungi reg: https://review.openstack.org/#/c/157213/ | 14:52 |
| *** ildikov has joined #openstack-infra | 14:52 | |
| deepakcs | anteaya, ^^ | 14:52 |
| anteaya | it is merged | 14:53 |
| anteaya | so what do you need? | 14:53 |
| deepakcs | anteaya, :) yeah, it is removing the glusterfs tempest job, bcos in recent past the job was getting restarted by zuul | 14:53 |
| anteaya | that is fine | 14:53 |
| *** wenlock has joined #openstack-infra | 14:53 | |
| anteaya | what you do need? | 14:53 |
| deepakcs | anteaya, i need that job back so that i can claim my CI to be working. I don't see those failures in the local setup | 14:54 |
| deepakcs | anteaya, https://jenkins07.openstack.org/job/check-tempest-dsvm-full-glusterfs-centos7/35/consoleFull | 14:54 |
| anteaya | see right there we have a problem | 14:54 |
| deepakcs | anteaya, as you can see, the job is getting killed bcos of some java hudson excp, i have no idea why thats happening | 14:54 |
| anteaya | since I am not interersted in helping you to _claim_ your CI is working | 14:55 |
| *** yfried_ is now known as yfried|afk | 14:55 | |
| anteaya | I will however support you in getting your CI working | 14:55 |
| anteaya | can you see the difference? | 14:55 |
| deepakcs | anteaya, yeah the latter :) i see it | 14:55 |
| anteaya | great | 14:55 |
| *** mwagner_lap has joined #openstack-infra | 14:55 | |
| deepakcs | I did some research | 14:55 |
| anteaya | so you are asking how to get your CI working? | 14:55 |
| deepakcs | anteaya, and it looks like most of the glusterfs tempest jobs are failing with java excp | 14:55 |
| *** sabeen has quit IRC | 14:55 | |
| anteaya | that is good information to have | 14:56 |
| deepakcs | anteaya, which looks like zuul <-> jenkins issue ? ( I am not a expert in infra) | 14:56 |
| deepakcs | anteaya, and some of the jobs are failing with "Build timeout" | 14:56 |
| anteaya | have you a stacktrace of the exceptions? | 14:56 |
| anteaya | that might be a place to begin | 14:56 |
| deepakcs | anteaya, the link above, scroll to end, it has the java trace | 14:57 |
| *** carl_baldwin has joined #openstack-infra | 14:57 | |
| *** yamahata has joined #openstack-infra | 14:57 | |
| *** ddieterly has joined #openstack-infra | 14:58 | |
| anteaya | 2015-02-19 08:40:36.696 | FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel | 14:58 |
| deepakcs | anteaya, yes and i have no idea why thats happening. so need help from infra folks | 14:59 |
| *** sabeen has joined #openstack-infra | 14:59 | |
| *** bswartz has joined #openstack-infra | 14:59 | |
| anteaya | so are you asking for information on what that exception means? | 14:59 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Uncap irc, block on 11.1 https://review.openstack.org/157395 | 15:00 |
| deepakcs | anteaya, well it says IO Excp and below the original excp seems to be EOF, but not sure why or how is it related to glusterfs devstack job | 15:00 |
| deepakcs | anteaya, wondering in infra has seen such things before for any other job/scenario from where we can get some clues | 15:00 |
| anteaya | well since it occurs due to the addition of your job are you willing to consider that it might in fact be a result of your job and start there? | 15:00 |
| deepakcs | anteaya, there are around 30+ glustefs experimental jobs that have failed and they don't fail at the same testcase, its random, so can't tie it to the testcase running at failure too | 15:01 |
| *** amitgandhinz has joined #openstack-infra | 15:03 | |
| sdague | so did someone look at the melody stats? | 15:04 |
| *** dimtruck is now known as zz_dimtruck | 15:04 | |
| *** dustins has joined #openstack-infra | 15:04 | |
| *** dannywilson has joined #openstack-infra | 15:05 | |
| *** yfried|afk is now known as yfried_ | 15:05 | |
| anteaya | I think melody stats were mentioned the other day | 15:06 |
| anteaya | I haven't seen them mentioned today | 15:06 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: WIP: Not even close to finished - use shade https://review.openstack.org/156322 | 15:07 |
| sdague | right, gerrit is quite slow again | 15:07 |
| sdague | it would be nice to see if there is a thing to be done to make it not so, as it's definitely impacting review time | 15:08 |
| anteaya | ack | 15:09 |
| *** dannywilson has quit IRC | 15:09 | |
| openstackgerrit | Max Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper https://review.openstack.org/156681 | 15:09 |
| BobBall | deepakcs: I suspect that error means the node lost SSH connectivity ? | 15:10 |
| deepakcs | BobBall, ok, so does it mean something in the job is causing it to lose SSH connectivity ? | 15:11 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Uncap irc, block on 11.1 https://review.openstack.org/157395 | 15:11 |
| deepakcs | BobBall, anteaya couple of days back the job was running fine, with 2 testcases failing, it was never broken, until now | 15:11 |
| BobBall | If it's reproducible, perhaps, yes. Otherwise might be some random cloudy thing that caused that node to die | 15:11 |
| deepakcs | BobBall, https://jenkins07.openstack.org/job/check-tempest-dsvm-full-glusterfs-centos7/ | 15:12 |
| AJaeger | anteaya, sdague: I did some hands-on help with the training-guides team, could you review my updated patch https://review.openstack.org/#/c/152118/ this week, please? | 15:12 |
| deepakcs | BobBall, most of the jobs there are aborted due to this java excp | 15:12 |
| deepakcs | BobBall, and few due to build timeout error | 15:12 |
| fungi | deepakcs: so, there are two prime possibilities for what was happening with that job | 15:12 |
| openstackgerrit | Max Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper https://review.openstack.org/156681 | 15:12 |
| deepakcs | fungi, ah, u r there :) | 15:12 |
| *** nelsnelson has joined #openstack-infra | 15:13 | |
| fungi | deepakcs: either something is making network configuration changes (routing, interface reconfiguration, firewall rules, et cetera) which are causing the jenkins master to no longer be able to reach teh worker, or something is causing a kernel panic (maybe something filesystem-related?) | 15:13 |
| fungi | deepakcs: can you attempt to recreate the situation on virtual machines of your own to identify the problem? | 15:14 |
| *** nelsnelson has quit IRC | 15:14 | |
| deepakcs | fungi, we don't touch neutron / networking, its a cinder-glusterfs job. Do u remember where/when u saw the kernel panic.. most of the failed jobs I saw either had java excp or build timeout as the reason for abortion | 15:14 |
| *** MaxV has joined #openstack-infra | 15:15 | |
| fungi | deepakcs: we see nothing. we just stop getting a response from the vm | 15:15 |
| *** nelsnelson has joined #openstack-infra | 15:15 | |
| *** mpaolino has quit IRC | 15:15 | |
| *** koolhead17 has joined #openstack-infra | 15:15 | |
| deepakcs | fungi, BharatK runs the tempest volume only and full tests in VM locally, and they are running fine. | 15:15 |
| fungi | deepakcs: those java exceptions are the jenkins master saying "i am no longer getting a response from the remote worker so i'm giving up now" | 15:15 |
| *** cdent has quit IRC | 15:15 | |
| *** cdent has joined #openstack-infra | 15:16 | |
| deepakcs | fungi, 2 days back the job was running fine (~2 hours to complete as its a full test), but were failing 2 testcases, we added a change to allow nova policy for online snap, post which we are seeing this | 15:16 |
| fungi | deepakcs: another possibility is an out-of-memory condition which is causing the kernel to kill lots of processes on the worker and also happens to blow out the jenkins slave agent in the process... is it possible this job would require more than 8gb ram? | 15:16 |
| deepakcs | fungi, with the same change on a local VM, we don't see that issue | 15:16 |
| *** hyakuhei has joined #openstack-infra | 15:17 | |
| anteaya | AJaeger: slides will be posted to docs after the patch merges? | 15:17 |
| deepakcs | BharatK, how much is ur local VM RAM where u test the tempest ? | 15:17 |
| *** dims__ has quit IRC | 15:17 | |
| AJaeger | anteaya, slides will not be posted at the moment for master | 15:17 |
| BharatK | deepakcs, locally I am using 4GB RAM | 15:17 |
| AJaeger | They're in complete reorganization mode and have nothing to publish from master for now... | 15:18 |
| anteaya | AJaeger: ah okay just created in the check and gate jobs, okay | 15:18 |
| deepakcs | fungi, since it says EOF excp (if u follow the caused by in java excp) i was wondering if its running out of disk space ? | 15:18 |
| fungi | so that's probably not the issue then | 15:18 |
| anteaya | AJaeger: ah okay thanks | 15:18 |
| *** dims__ has joined #openstack-infra | 15:18 | |
| AJaeger | anteaya, correct | 15:18 |
| fungi | deepakcs: do you need more than 20gb disk? | 15:18 |
| deepakcs | BharatK, locally whats ur disk size ? | 15:18 |
| BharatK | fungi, in the local environment I am using 40 GB | 15:18 |
| clarkb | also note we run the "full" tempest run not a subset | 15:19 |
| BharatK | fungi, while job is running I have seen the 34GB free space in "df -h" output | 15:19 |
| deepakcs | BharatK, fungi but the max we do is create a loop device of 8G using the disk file to be used as a brick for glusterfs storage, so 20G should be sufficient | 15:19 |
| clarkb | so you may not see this locally if running a subset | 15:19 |
| deepakcs | clarkb, hi | 15:19 |
| fungi | jenkins will automatically offline the worker if its root filesystem fills up, but it also often misreports a full filesystem as the reason for offlining when in actuality it loses contact with the worker, so that tends to be a false error | 15:19 |
| fungi | deepakcs: do you create it in /opt or somewhere else? we do have hundreds of gb in /opt generally but / can be as small as 20gb | 15:20 |
| BharatK | clarkb, In local environment we ran full tempest | 15:20 |
| deepakcs | BharatK, u did run "full" tempest too, and it ran fine, right ? | 15:20 |
| BharatK | deepakcs, just one hour back I ran full tempest without any issues. | 15:20 |
| deepakcs | BharatK, can u answer fungi 's q on where we create the 8G disk file ? | 15:21 |
| deepakcs | fungi, looking at the console logs, before the tests start, df -h shows 32G as free space for root | 15:21 |
| clarkb | BobBall you put your swift creds in zuul's ini file then zuul generates the job specific creds | 15:21 |
| deepakcs | fungi, /dev/xvda1 40G 5.9G 32G 16% / | 15:21 |
| BharatK | fungi, we are creating loopback device in /var/lib/glance | 15:22 |
| BobBall | oh brilliant clarkb *looks* thanks. | 15:22 |
| fungi | deepakcs: yeah, some have 40 and some 20 depending on what provider they end up running in (we have virtual machines from a couple different service providers) | 15:22 |
| BharatK | fungi, /var/lib/glusterfs/ | 15:22 |
| clarkb | BobBall those creds are scoped to the container and path and have anexpiry iirc | 15:22 |
| deepakcs | fungi, i don't see /opt as a diff mnt in the df -h | 15:23 |
| deepakcs | so /opt is part of / itself | 15:23 |
| clarkb | that df likely runs before the mount then | 15:23 |
| clarkb | see the setup host log | 15:23 |
| fungi | we might want to see if we can fix the ordering there at some point | 15:23 |
| clarkb | we carve out the ephemeral drive and mount swap and /opt | 15:24 |
| deepakcs | clarkb, ok, setup host log can be seen only if the job completed gracefully, right ? | 15:24 |
| *** timcline has joined #openstack-infra | 15:24 | |
| clarkb | deepakcs oh right because the node goes away in this case | 15:24 |
| fungi | good point, we won't have other logs/files on the aborted or timed out jobs | 15:24 |
| deepakcs | clarkb, yes :) | 15:24 |
| BobBall | clarkb: Yes - I saw that. Now looking at the zuul config file it seems easy enough. Will test and see if/where I get errors... | 15:25 |
| fungi | clarkb: any idea if we were we seeing this happen in both hpcloud and rax? | 15:25 |
| deepakcs | clarkb, fungi since we are not able to recreate this in local VM is it possible to somehow enable this job, but not come in the queue for others, so that we can debug with some ideas ? | 15:25 |
| clarkb | fungi no I didnt get that far. it was ever job running yesterday though so I would be suprised if it wasnt both | 15:27 |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens https://review.openstack.org/153591 | 15:27 |
| fungi | deepakcs: i can try to rerun the job by hand as long as i know which provider i'm likely to trigger the failure scenario in | 15:27 |
| deepakcs | clarkb, fungi Other than the java excp , the only other error i saw was "build timeout / aborted" - what does that signify ? jenkins server timedout connecting to slave ? | 15:27 |
| clarkb | deepakcs I think in both cases that is what happens | 15:28 |
| *** koolhead17 has quit IRC | 15:28 | |
| deepakcs | clarkb, fungi just wondering if there is any clue hidden in there when both the diff types of aborts are taken into consideration ? | 15:28 |
| clarkb | jenkins just doesnt express it the same depending on how it notices | 15:28 |
| *** mtanino has joined #openstack-infra | 15:29 | |
| fungi | deepakcs: i suspect i'll need to manually run the same steps the job does, and then when i (likely) lose access to the virtual machine, try to log in through an out-of-band console to see what's gone wrong | 15:29 |
| *** yamamoto has joined #openstack-infra | 15:29 | |
| deepakcs | clarkb, fungi by provider u mean the cloud provider (hpcloud Vs rax) ? | 15:29 |
| clarkb | yes | 15:29 |
| deepakcs | clarkb, and where in the console log i can figure that ? | 15:29 |
| clarkb | at the beginning, it is encoded into the hostname | 15:29 |
| *** tim_o has quit IRC | 15:30 | |
| sdague | clarkb: so, was there any resolution from the melody graphs on gerrit the other day? Gerrit still seems like it's dragging quite a bit | 15:30 |
| fungi | i suspect anyone here running a lenovo laptop is installing linux anyway, but this sets a disturbing precedent: http://www.zdnet.com/article/lenovo-accused-of-pushing-superfish-self-signed-mitm-proxy/ | 15:31 |
| fungi | i wonder what hardware backdoors they're also installing in those | 15:31 |
| clarkb | sdague no I think it likely need to be restarted again, but jeblair wasnt convinced? should see if the graph helped swing him over to restarting | 15:31 |
| deepakcs | clarkb, 2015-02-19 06:14:56.918 | Available disk space on this host: | 15:31 |
| deepakcs | 2015-02-19 06:14:56.920 | Filesystem Size Used Avail Use% Mounted on | 15:31 |
| deepakcs | 2015-02-19 06:14:56.920 | /dev/xvda1 40G 5.9G 32G 16% / | 15:31 |
| deepakcs | 2015-02-19 06:14:56.920 | devtmpfs 3.7G 0 3.7G 0% /dev | 15:31 |
| deepakcs | 2015-02-19 06:14:56.920 | tmpfs 3.7G 0 3.7G 0% /dev/shm | 15:31 |
| deepakcs | 2015-02-19 06:14:56.920 | tmpfs 3.7G 8.3M 3.7G 1% /run | 15:31 |
| deepakcs | 2015-02-19 06:14:56.921 | tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup | 15:31 |
| deepakcs | Oops.. sorry! | 15:31 |
| clarkb | fungi ya never booted windows at all :) | 15:31 |
| deepakcs | clarkb, https://jenkins07.openstack.org/computer/devstack-centos7-rax-iad-912561 | 15:31 |
| deepakcs | so its rax provider | 15:31 |
| clarkb | depoakcs please use a paste service | 15:31 |
| deepakcs | clarkb, that was by mistake | 15:32 |
| deepakcs | apologies | 15:32 |
| *** AJaeger has quit IRC | 15:32 | |
| sdague | so I saw you send him a graph, but then the conversation stopped. Anyway, lots of "people nagios" are reporting in that it's slow, including me. | 15:32 |
| deepakcs | clarkb, fungi is it a bad idea to add the job again and we do some debug (Like limit the tests to volume only, reduce the size of loop device etc )? | 15:33 |
| fungi | deepakcs: i can grab a centos7 machine in rax-iad and try to recreate the failure | 15:33 |
| mugsie | anteaya: ping? | 15:33 |
| fungi | clarkb: when did we get confirmation that those centos7 workers were actually operational? | 15:33 |
| *** achanda has joined #openstack-infra | 15:33 | |
| deepakcs | clarkb, fungi or add it to a different expt pipeline so that it doesn't affect others ? | 15:33 |
| clarkb | fungi a while back, just devstack/temlest failed on them but the nodes themselves were ok | 15:34 |
| fungi | oh, got it. so as long as they're not running devstack/tempest on centos7 their job should work i guess? | 15:34 |
| deepakcs | fungi, i just randonly checked 4-5 jobs and all had 'rax' in the hostname | 15:34 |
| fungi | because it sort of sounds like they're trying to set up devstack and run tempest against it on centos7 there | 15:35 |
| clarkb | fungi it is a tempest job but the other centos7 jobs dont die like this they fail normally | 15:35 |
| *** frickler has joined #openstack-infra | 15:35 | |
| deepakcs | fungi, yes thats what we are doing there | 15:35 |
| fungi | oh, got it. so other jobs failing on centos7 weren't tanking the entire worker | 15:35 |
| *** avozza is now known as zz_avozza | 15:36 | |
| clarkb | correct | 15:36 |
| fungi | i'll hold a ready devstack-centos7 worker in rax and see if i can recreate the failure by hand | 15:36 |
| openstackgerrit | Max Rasskazov proposed openstack-infra/jenkins-job-builder: Fix broken link on documentation https://review.openstack.org/157416 | 15:36 |
| deepakcs | fungi, let me or BharatK know if u need any help | 15:37 |
| *** yamamoto has quit IRC | 15:37 | |
| *** zz_dimtruck is now known as dimtruck | 15:37 | |
| deepakcs | fungi, clarkb Do we in parallel also start looking at a F20/f21 based gate job, if centos7 has issues ? | 15:37 |
| *** ddieterly has quit IRC | 15:38 | |
| anteaya | mugsie: how can I help? | 15:38 |
| *** yamamoto has joined #openstack-infra | 15:39 | |
| mugsie | about https://review.openstack.org/#/c/155788/ | 15:39 |
| anteaya | what about it? | 15:39 |
| clarkb | deepakcs no I think we figure this one out | 15:39 |
| mugsie | is there a problem allowing us to have the third party ci? | 15:39 |
| anteaya | do you need to have third party ci? | 15:40 |
| mugsie | we don;t have a massive immediate need, but I want to try running a few other job against our patches | 15:40 |
| mugsie | jobs* | 15:40 |
| anteaya | I'm feeling is is a fashion statement rather than something folks need | 15:40 |
| deepakcs | clarkb, fungi thanks in advance :) Do we shar eour email IDs with you so that you can keep us in the loop (given the TZ diff, we will be asleep soon) :) | 15:40 |
| mugsie | and we will eventually need it (in the next cycle or so) | 15:40 |
| anteaya | mugsie: you can run a few jobs against your patches | 15:40 |
| anteaya | and you can have the ci systems comment on your patches | 15:40 |
| mugsie | but not vote | 15:40 |
| anteaya | all without needing a gerrit group | 15:40 |
| anteaya | right | 15:41 |
| fungi | mugsie: you need to test things against designate that involve special hardware or non-free software? curious why you can't just run the jobs in our system | 15:41 |
| *** achanda has quit IRC | 15:41 | |
| *** EmilienM|afk is now known as EmilienM | 15:41 | |
| mugsie | i am not sure if they are usful, its a sonar job | 15:41 |
| openstackgerrit | James Graf proposed openstack-infra/jenkins-job-builder: Add support for Job DSL plugin for Groovy-based job and view creation and management https://review.openstack.org/150112 | 15:41 |
| fungi | mugsie: that' | 15:42 |
| fungi | s coverage right? | 15:42 |
| mugsie | I want to see if having something that votes helps us with code complexity | 15:42 |
| mugsie | code complexity / syntax etc | 15:42 |
| fungi | ahh, code complexity | 15:42 |
| deepakcs | clarkb, fungi ^^ | 15:42 |
| mugsie | it can take code coverage as well | 15:42 |
| mugsie | but its mainly a static analyis tool | 15:42 |
| *** skolekonov has quit IRC | 15:43 | |
| mugsie | we will need to start testing backend that talk to APIs / comercial software as well, and 3rd party testing is must for that | 15:43 |
| fungi | mugsie: a commercial/proprietary tool? | 15:43 |
| mugsie | nope, there is a GPL version | 15:43 |
| clarkb | there are free/open tools to do this | 15:43 |
| *** abhirc has joined #openstack-infra | 15:44 | |
| mugsie | http://www.sonarqube.org/ | 15:44 |
| fungi | mugsie: because you can probably just install and run it on our workers without needing to set up a separate entire system for it | 15:44 |
| *** Sukhdev has joined #openstack-infra | 15:44 | |
| fungi | deepakcs: if you can't leave your irc client connected while you sleep, then i guess start a brief mailing list thread and one of us will follow up to it with what we find | 15:44 |
| deepakcs | fungi, ok, will start a thread on -infra, hope that is fine ? | 15:45 |
| clarkb | yes that would be good | 15:45 |
| fungi | deepakcs: i would say start one on openstack-dev@lists.openstack.org instead | 15:45 |
| fungi | but infra will work if you'd rather | 15:45 |
| clarkb | or there | 15:45 |
| mugsie | fungi: yeah. but if left as a long running thing it gives interesting trends etc | 15:45 |
| clarkb | actually ya -dev will get more eyes on it and maybe other will recognize the behavior | 15:46 |
| deepakcs | i thought infra was more appropriate for this | 15:46 |
| fungi | mugsie: oh, so it's not a testing tool, it's a trending tool? | 15:46 |
| fungi | deepakcs: infra is more appropriate for topics which are not of interest to openstack, but you're testing openstack with glusterfs, right? | 15:46 |
| mugsie | it can do both. Its can give an incremental report, and then keep track of results over time | 15:46 |
| deepakcs | fungi, yes | 15:46 |
| deepakcs | fungi, i can start on -dev too, not a issue | 15:47 |
| *** mrunge has quit IRC | 15:47 | |
| *** cipcosma has quit IRC | 15:48 | |
| fungi | mugsie: out of curiosity, what trending is of interest across _proposed_ changes? wouldn't you want to trend the state of the codebase itself? | 15:48 |
| mugsie | fungi: yeah, was going to trigger the trending jobs on post-merge | 15:49 |
| fungi | mugsie: in that case as anteaya says you don't need a -ci group for that. it's only to be able to control what accounts can leave code verification votes on proposed changes | 15:49 |
| fungi | mugsie: any account can listen to the event stream, which is all you need to be able to see the change-merged events | 15:50 |
| *** hashar has joined #openstack-infra | 15:50 | |
| mugsie | yeah, and as i said earlier - we have some backends that aren't fully tested currently, and will need this as well | 15:51 |
| mugsie | (commercial API driven dns vendors etc) | 15:51 |
| *** jaypipes has quit IRC | 15:51 | |
| mugsie | but that req is not immediate | 15:51 |
| fungi | ahh, and for that yes you would need something like this. we wouldn't be able to interface with your commercial vendor dns server devices | 15:51 |
| *** imcsk8 has quit IRC | 15:51 | |
| *** yamamoto has quit IRC | 15:51 | |
| *** imcsk8 has joined #openstack-infra | 15:52 | |
| *** ddieterly has joined #openstack-infra | 15:52 | |
| * fungi assumes things like infoblox, bluecoat, et cetera | 15:52 | |
| mugsie | so, shoiuld I back out the reuest until we need it asap? I figured getting it in early means its one less thjing to worry about | 15:52 |
| mugsie | yup, and Dyn, and as few others | 15:52 |
| mugsie | a* | 15:52 |
| anteaya | mtreinish: can you indicate you are okay with this new non-voting job being added to tempest: https://review.openstack.org/#/c/151564/6 | 15:54 |
| fungi | it's fine if you forsee a need, just be wary not to run things externally which you could instead run upstream | 15:54 |
| anteaya | mugsie: if you recall my question was why do you need it | 15:54 |
| anteaya | mugsie: if you have a need that is fine | 15:54 |
| mtreinish | anteaya: sure I'll take a look | 15:55 |
| anteaya | but many folks feel they have to have something because other projects do and then don't know what to do with it | 15:55 |
| anteaya | mtreinish: thanks | 15:55 |
| *** bogdando has quit IRC | 15:55 | |
| *** ddieterl_ has joined #openstack-infra | 15:55 | |
| *** ddieterly has quit IRC | 15:56 | |
| *** yolanda has quit IRC | 15:57 | |
| *** bcourt has joined #openstack-infra | 15:57 | |
| *** Sukhdev has quit IRC | 15:58 | |
| *** yolanda has joined #openstack-infra | 16:00 | |
| *** ParsectiX has quit IRC | 16:00 | |
| *** ParsectiX has joined #openstack-infra | 16:01 | |
| *** hyakuhei has quit IRC | 16:01 | |
| *** jaypipes has joined #openstack-infra | 16:01 | |
| *** armax has joined #openstack-infra | 16:02 | |
| *** jlibosva has quit IRC | 16:04 | |
| *** annegentle has joined #openstack-infra | 16:04 | |
| *** david-lyle_afk is now known as david-lyle | 16:04 | |
| fungi | mescanef: i've added you as the initial member of https://review.openstack.org/#/admin/groups/599,members sorry for the wait | 16:05 |
| *** denis_makogon has quit IRC | 16:05 | |
| *** afazekas has quit IRC | 16:05 | |
| *** jlibosva has joined #openstack-infra | 16:06 | |
| mescanef | fungi: thank you | 16:06 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: DO NOT MERGE COPY ALL OF .TESTREPOSITORY to log server https://review.openstack.org/149719 | 16:06 |
| mtreinish | anteaya: lgtm | 16:07 |
| mugsie | anteaya: yeap, I understand that | 16:07 |
| *** tmcpeak has joined #openstack-infra | 16:07 | |
| anteaya | mtreinish: thanks | 16:07 |
| anteaya | mugsie: great, so can you add a reply as a comment on the patch | 16:07 |
| mugsie | it just seems everytime we go to get something that is standard for other projects we get push back | 16:07 |
| anteaya | so others can read it too? | 16:07 |
| mugsie | anteaya: doing it now | 16:07 |
| anteaya | mugsie: you believe having a -ci group is standard for other projects? | 16:08 |
| mugsie | no, but the ability to have it is standard | 16:08 |
| anteaya | that is a perception that is a shame if that is what you believe | 16:08 |
| *** hashar_ has joined #openstack-infra | 16:08 | |
| anteaya | you have the ability to have it | 16:08 |
| anteaya | I simply asked a question | 16:08 |
| anteaya | part of my reviewing process is to ask questions | 16:08 |
| *** arxcruz has quit IRC | 16:09 | |
| fungi | mugsie: other projects that have it don't have it just because they're able to have it. they have it because they have ci systems leaving comments on their changes and want to be able to control whether those can vote | 16:09 |
| *** hashar has quit IRC | 16:10 | |
| fungi | mugsie: we simply ask that you don't speculatively request support for features you might exercise in the future without some certainty of that, because otherwise it adds unnecessary workload on our team, which is already stretched extremely thin (hundreds and hundreds of outstanding changes/support requests we have to prioritize) | 16:11 |
| fungi | mugsie: so it's not just your project. pretty much any project which requests something should expect to have to justify their request | 16:12 |
| *** marun has joined #openstack-infra | 16:12 | |
| *** deepakcs has quit IRC | 16:12 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: DO NOT MERGE COPY ALL OF .TESTREPOSITORY to log server https://review.openstack.org/149719 | 16:13 |
| *** cody-somerville has quit IRC | 16:13 | |
| *** sergsh has quit IRC | 16:13 | |
| *** stevemar has joined #openstack-infra | 16:14 | |
| anteaya | fungi clarkb any problem with raising a stable job timeout from 40 to 90? python26: https://review.openstack.org/#/c/156734/1 | 16:16 |
| *** otter768 has joined #openstack-infra | 16:17 | |
| clarkb | pcrews: ^ any chance we can bump that timeout a little less and still get things to go through? | 16:17 |
| clarkb | anteaya: my only concern is that that is a huge jump and may hide a larger underlying issue if 90 minutes is really necessary | 16:17 |
| *** dannywilson has joined #openstack-infra | 16:19 | |
| *** dangers_away is now known as dangers | 16:19 | |
| *** dannywilson has quit IRC | 16:19 | |
| pcrews | clarkb: sure - just shot for a high value to ensure it would pass (or if it doesn't run in 90 min, it needs more tlc imo) - do you have an opinion / recommendation for time? | 16:19 |
| anteaya | clarkb: thanks | 16:19 |
| clarkb | pcrews: no immediate opinion, but usually if jobs start timing out over time we are straddling the boundary and don't need a large jump. maybe just bump to 50-60 minutes? | 16:20 |
| *** otter768 has quit IRC | 16:21 | |
| fungi | pcrews: the general rule of thumb is that if you allow that job to run for 90 minutes instead of only 40 then it will _very_soon_ start taking 90 minutes instead of only 40 | 16:21 |
| pcrews | fungi: ack and noted ;) | 16:22 |
| fungi | because the only pushback on whether tests are poorly-written and take too long is when they start causing the job to time out | 16:22 |
| pcrews | clarkb: ack - omw to office then will submit a new patch | 16:22 |
| *** marcusvrn1 has joined #openstack-infra | 16:22 | |
| *** zz_avozza is now known as avozza | 16:23 | |
| *** dannywilson has joined #openstack-infra | 16:23 | |
| fungi | clarkb: the experimental glusterfs job has gotten as far as starting tempest now. still up and running at the moment | 16:23 |
| anteaya | pcrews: ack, I 1'd for now | 16:23 |
| anteaya | -1'd | 16:23 |
| fungi | no signs yet of resource exhaustion or disconnection | 16:23 |
| *** med_` is now known as medberry | 16:24 | |
| *** ddieterl_ has quit IRC | 16:24 | |
| *** tmcpeak has quit IRC | 16:24 | |
| *** ssam2 has quit IRC | 16:24 | |
| *** medberry is now known as medberry2 | 16:24 | |
| clarkb | fungi: ya it seems to go under during object storage tests | 16:24 |
| clarkb | fungi: so hopefully you will see something soon :) | 16:24 |
| *** marcusvrn has quit IRC | 16:24 | |
| *** krtaylor has quit IRC | 16:25 | |
| fungi | clarkb: running on 104.130.229.191 if you get curious | 16:25 |
| sdague | pcrews: do you have any idea why this regressed? | 16:25 |
| *** ddieterly has joined #openstack-infra | 16:26 | |
| fungi | pcrews: also, note that we have some problem on centos6 nodes where sometimes git remote actions take an indefinite amount of time and cause jobs to time out. if it's hitting that, then upping the timeout won't help at all | 16:26 |
| *** pcrews has quit IRC | 16:27 | |
| clarkb | oh right this is the python26 job | 16:27 |
| fungi | i've so far been entirely unable to recreate the condition to be able to nail down a root cause, but seemed to start right after the most recent big security fix scramble | 16:27 |
| mordred | clarkb: I remembered why I did the puppet ansible thign in python rather than pure ansible yaml | 16:28 |
| mordred | clarkb: it's because there is a need to parse the stdout string frmo puppet because it double-uses 2 return codes for different things | 16:29 |
| clarkb | mordred: I want to cry right now | 16:29 |
| mordred | clarkb: and although the primitives for respoding to error codes are good - once you get to expressing string parsing logic in yaml | 16:29 |
| mordred | you know | 16:29 |
| mordred | it's just crazypants | 16:29 |
| *** rcarrillocruz has quit IRC | 16:29 | |
| mordred | sorry it took me so long to remember the reasoning that led to that | 16:29 |
| clarkb | mordred: thats ok, maybe we should tack it into a docstring/comment for the python file that says "No really there are reasons and they are:" | 16:30 |
| mordred | ++ | 16:30 |
| *** spredzy is now known as spredzy|afk | 16:30 | |
| mordred | I will do that - as soon as I write the patch I owe you :) | 16:30 |
| openstackgerrit | Merged openstack-infra/project-config: Remove retired repos from channel notification https://review.openstack.org/156211 | 16:32 |
| *** rcarrillocruz has joined #openstack-infra | 16:32 | |
| *** ssam2 has joined #openstack-infra | 16:32 | |
| *** hashar_ has quit IRC | 16:32 | |
| *** openstackgerrit has quit IRC | 16:36 | |
| *** openstackgerrit has joined #openstack-infra | 16:36 | |
| *** jlibosva has quit IRC | 16:37 | |
| *** cnesa has joined #openstack-infra | 16:37 | |
| *** annegentle has quit IRC | 16:39 | |
| *** yamahata has quit IRC | 16:39 | |
| BobBall | So apparently I'm missing something obvious... Trying to add a new parameter I added swift_authurl to the class block mimicking other parameters but I get an error saying the parameter is invalid. I'd appreciate help on two levels - 1) why is it invalid, 2) where is this line 114 it's complaining about :) http://paste.openstack.org/show/177740/ | 16:40 |
| *** annegentle has joined #openstack-infra | 16:40 | |
| clarkb | BobBall: I think the os_ext_testing puppet class doesn't take that parameter | 16:41 |
| BobBall | I added it at https://github.com/citrix-openstack/os-ext-testing/blob/master/puppet/modules/os_ext_testing/manifests/master.pp#L49 | 16:42 |
| clarkb | BobBall: so likely need to update that puppet to pass through swift_authurl similarly to https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/zuul_prod.pp | 16:42 |
| BobBall | I may have needed to add it somewhere else too, but that's the bit I don't understand currently... | 16:42 |
| BobBall | yes, that's exactly what I tried to do :/ | 16:43 |
| *** mpavone has quit IRC | 16:43 | |
| clarkb | BobBall: that appears to be the right file, is it updated in your puppet path? | 16:43 |
| krotscheck | So, when using storyboard, can y'all keep an eye on that page-not-loading-when-direct-link issue you were seeing? A patch that went in recently is supposed to fix that. | 16:43 |
| *** ddieterly has quit IRC | 16:43 | |
| clarkb | krotscheck: noted, will try to track if I notice it | 16:43 |
| krotscheck | clarkb: Thanks. | 16:43 |
| BobBall | Yup ... root@jenkins:~/os-ext-testing/puppet# grep -r swift_authurl /root/os-ext-testing/puppet/os-ext-testing/puppet/modules: /root/os-ext-testing/puppet/os-ext-testing/puppet/modules/logging/manifests/master.pp: $swift_authurl = '', | 16:44 |
| *** yfried_ is now known as yfried|afk | 16:45 | |
| *** ddieterly has joined #openstack-infra | 16:45 | |
| BobBall | hang on... | 16:47 |
| * BobBall is an idiot. | 16:47 | |
| BobBall | Thanks :) | 16:47 |
| *** dmsimard_away is now known as dmsimard | 16:47 | |
| *** krtaylor has joined #openstack-infra | 16:47 | |
| clarkb | BobBall: oh what was it? I had traced it down through the zuul manifest too and it seemed to be there | 16:48 |
| clarkb | the only odd thing I noticed was the line number if complains about doesn't seem right based on your github link | 16:48 |
| *** mestery has quit IRC | 16:49 | |
| clarkb | krotscheck: fwiw I just ran through a couple things that normally triggered it for me and it was fine (open bug, login, get bug back instead of blank page) | 16:50 |
| *** hyakuhei has joined #openstack-infra | 16:50 | |
| BobBall | I had a nested version of the repository which was messing things up | 16:50 |
| BobBall | "os-ext-testing/puppet/os-ext-testing/puppet" from above | 16:50 |
| BobBall | I need a rubber duck. | 16:50 |
| *** koolhead17 has joined #openstack-infra | 16:50 | |
| *** tkelsey has joined #openstack-infra | 16:51 | |
| *** salv-orlando has quit IRC | 16:52 | |
| *** yamamoto has joined #openstack-infra | 16:52 | |
| *** scheuran has quit IRC | 16:52 | |
| fungi | git clean -dfx | 16:53 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add hasExtension method to check cloud capabilities https://review.openstack.org/156954 | 16:54 |
| *** gondoi is now known as zz_gondoi | 16:55 | |
| *** zz_gondoi is now known as gondoi | 16:57 | |
| *** yamamoto has quit IRC | 16:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Python34 Jobs for StoryBoard are now voting. https://review.openstack.org/157190 | 16:57 |
| *** coolsvap is now known as coolsvap_ | 16:57 | |
| *** dprince has joined #openstack-infra | 16:58 | |
| *** tiswanso has quit IRC | 16:58 | |
| openstackgerrit | Andrew Lazarev proposed openstack/requirements: Bump sahara client version https://review.openstack.org/155428 | 16:59 |
| *** annegentle has quit IRC | 17:01 | |
| *** jlibosva has joined #openstack-infra | 17:02 | |
| *** carl_baldwin has quit IRC | 17:02 | |
| *** ihrachyshka has quit IRC | 17:03 | |
| ctlaugh | Are there plans / another recommendation to move away from using scp-plugin? No new release since 2011? | 17:03 |
| openstackgerrit | patrick-crews proposed openstack-infra/project-config: Increase py26 timeouts for nova-stable https://review.openstack.org/156734 | 17:04 |
| zaro | ctlaugh: we use master | 17:04 |
| krotscheck | clarkb: Awesome, thanks | 17:05 |
| ctlaugh | zaro: master, as in manually built it from master? | 17:05 |
| anteaya | krotscheck: ack | 17:05 |
| zaro | ctlaugh: yes. | 17:05 |
| *** ParsectiX has quit IRC | 17:05 | |
| clarkb | ctlaugh: zaro: and we are moving to using swift uploads instead | 17:06 |
| clarkb | all of the infra jobs use swift uploaded logs exclusively now | 17:06 |
| * anteaya hands BobBall a rubber duck | 17:06 | |
| *** ParsectiX has joined #openstack-infra | 17:06 | |
| clarkb | or a vast majority at least | 17:06 |
| ctlaugh | zaro: yeah, I did that too... just doesn't feel right relying on it. | 17:06 |
| *** tmcpeak has joined #openstack-infra | 17:06 | |
| ctlaugh | clarkb: any more details/pointers to info on how that works? | 17:07 |
| *** tmcpeak has quit IRC | 17:07 | |
| *** tmcpeak1 has joined #openstack-infra | 17:07 | |
| clarkb | ctlaugh: zuul hands out upload urls to jobs that are scoped to specific locations and have an expiry, then the job has a script that uploads files with that info | 17:07 |
| fungi | the glusterfs job is _still_ going | 17:07 |
| clarkb | ctlaugh: http://ci.openstack.org/zuul/zuul.html#swift | 17:08 |
| clarkb | fungi: is tempest still running new tests or has it hung? | 17:08 |
| *** dtantsur is now known as dtantsur|afk | 17:08 | |
| fungi | clarkb: still going through tests | 17:08 |
| ctlaugh | clarkb: thank you! | 17:08 |
| BobBall | thanks anteaya! | 17:09 |
| clarkb | fungi: it is much slower than the non glusterfs job so this may be normal | 17:09 |
| fungi | yeah, the devstack setup phase took 20+ minutes (i didn't wall-clock it but a while) | 17:10 |
| anteaya | BobBall: let me know if it helps | 17:10 |
| *** yamahata has joined #openstack-infra | 17:10 | |
| * anteaya has to leave soon for errands which will take most of the afternoon | 17:11 | |
| *** markus_z has quit IRC | 17:12 | |
| fungi | clarkb: btw i'm testing on master tip of everything... is it possible the experimental job loop you saw was specific to a particular change? | 17:12 |
| clarkb | fungi: I thought so initially so I made sure to check all of the glusterfs runs that were going and every one did it | 17:13 |
| *** teran_ has joined #openstack-infra | 17:13 | |
| *** afazekas has joined #openstack-infra | 17:13 | |
| clarkb | fungi: so I think it is unlikely related to a specific unmerged change | 17:13 |
| fungi | k | 17:13 |
| clarkb | there were ~5 running across 4 projects iirc | 17:13 |
| fungi | ahh, yeah then must have been endemic to the job | 17:14 |
| fungi | right now it's on tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_set_empty_name | 17:14 |
| *** teran has quit IRC | 17:14 | |
| *** ihrachyshka has joined #openstack-infra | 17:15 | |
| fungi | i've got free and df looping once a second in a separate ssh session to the worker, and am also trying to maintain an active virtual console session to it in case there's a kernel panic | 17:16 |
| *** pradk has joined #openstack-infra | 17:19 | |
| *** Ala has quit IRC | 17:20 | |
| *** teran has joined #openstack-infra | 17:20 | |
| *** cdent has quit IRC | 17:21 | |
| *** pcrews has joined #openstack-infra | 17:21 | |
| fungi | okay, it's on object_storage tests now | 17:21 |
| fungi | so hopefully rsn | 17:22 |
| clarkb | fungi: this is where I saw the others stop | 17:22 |
| *** teran__ has joined #openstack-infra | 17:22 | |
| fungi | it's running very close to 8gb active ram, about to dip into swap i think | 17:23 |
| krotscheck | I'm pondering removing the apache2 vhost from puppet-storyboard, because it presumes to know how you want to set things up. | 17:23 |
| krotscheck | Any opinions? | 17:23 |
| *** teran_ has quit IRC | 17:23 | |
| clarkb | krotscheck: what do you have in mind to replace it with? | 17:23 |
| fungi | krotscheck: we could move that into system-config's openstack_project module instead | 17:23 |
| clarkb | oh I see | 17:24 |
| clarkb | ya you could decouple storyboard deployment from deploment of a proxy for storyboard | 17:24 |
| fungi | but i sort of like the idea that installing the module gives you a working _something_ you can then proceed to extend as needed | 17:24 |
| clarkb | or maybe ship a manifest that you can include that uses some relatively sane defaults but otherwise doesn't force you into using it | 17:24 |
| fungi | yeah, i think move the vhost handling into a separate optional class? | 17:24 |
| *** cdent has joined #openstack-infra | 17:24 | |
| clarkb | fungi: ya | 17:24 |
| *** teran has quit IRC | 17:24 | |
| krotscheck | Yeah, what fungi just said makes the most sense to me. | 17:25 |
| *** jcoufal has quit IRC | 17:25 | |
| fungi | then a deployer can choose to use it or substitute their own | 17:25 |
| clarkb | +1 | 17:25 |
| krotscheck | That also neatly solves my http/https problem. | 17:25 |
| *** bknudson has joined #openstack-infra | 17:26 | |
| *** cdent has quit IRC | 17:26 | |
| fungi | clarkb: actually the https://jenkins07.openstack.org/job/check-tempest-dsvm-full-glusterfs-centos7/35/consoleFull example failure got to volume snapshot and instance resize tests | 17:26 |
| *** teran_ has joined #openstack-infra | 17:27 | |
| *** teran__ has quit IRC | 17:27 | |
| *** jistr has quit IRC | 17:28 | |
| nibalizer | +1 | 17:29 |
| *** MaxV has quit IRC | 17:30 | |
| *** teran__ has joined #openstack-infra | 17:30 | |
| *** dansmith has quit IRC | 17:30 | |
| fungi | krotscheck: though if you want to make http vs https configurable, have a look at my proposed puppet-zuul change (not quite the same, it allows you to enable/disable https in addition to http, but the mechanisms you need to switch that stuff are similar) | 17:30 |
| *** dansmith has joined #openstack-infra | 17:30 | |
| BobBall | clarkb: So I've got the values set in zuul/zuul.conf now (http://paste.openstack.org/show/177758/), I've restarted zuul and rebuild the jobs but I'm getting Environment variable 'SWIFT_logs_LOGSERVER_PREFIX' not found | 17:31 |
| BobBall | Where should I be looking to debug why zuul isn't setting this up like you expect? | 17:31 |
| *** bdpayne has joined #openstack-infra | 17:31 | |
| fungi | and now it's past the cli and scenario tests and into the thirdparty boto tests | 17:31 |
| clarkb | BobBall: oh I think you ma actually need to enable it in layout.yaml per job as well (or with a regex that matches everything) | 17:31 |
| *** teran_ has quit IRC | 17:32 | |
| *** jlibosva has quit IRC | 17:32 | |
| clarkb | BobBall: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n527 sorry I forgot about that | 17:32 |
| BobBall | ahhh perfect! Many thanks again :) | 17:32 |
| * BobBall is getting there... | 17:33 | |
| fungi | clarkb: hrm, i guess tempest test order is nondeterministic so trying to track progress there is likely not tractable | 17:33 |
| clarkb | fungi: ya | 17:33 |
| *** teran has joined #openstack-infra | 17:33 | |
| *** teran__ has quit IRC | 17:34 | |
| *** unicell has joined #openstack-infra | 17:34 | |
| fungi | been running for ~95 minutes now | 17:34 |
| *** derekh has quit IRC | 17:34 | |
| *** Shrews has quit IRC | 17:34 | |
| *** patrickeast has joined #openstack-infra | 17:34 | |
| mtreinish | fungi: heh, well normally you can since the preseeding is a crapshoot on whether it does anything or not | 17:35 |
| jogo | anyone else having issues with gerrit's responsiveness? | 17:35 |
| mtreinish | then it's pseudo alphabetical order | 17:35 |
| fungi | mtreinish: nice! | 17:35 |
| *** ldnunes has quit IRC | 17:35 | |
| *** krotscheck has quit IRC | 17:36 | |
| clarkb | anteaya: on 157330 would you like to see those changes enough to -1 instead of +2? otherwise I think I will approve it | 17:36 |
| mtreinish | fungi: do you have a console output somewhere? | 17:38 |
| *** ihrachyshka has quit IRC | 17:38 | |
| *** hdd has joined #openstack-infra | 17:38 | |
| fungi | mtreinish: in my ssh session | 17:38 |
| mtreinish | ah, ok | 17:38 |
| zaro | clarkb: i don't see how asselin change (https://review.openstack.org/#/c/155154/) helps with getting '/p/' in review-dev.o.o URL to work. would you be able to enlighten me? | 17:39 |
| *** krotscheck has joined #openstack-infra | 17:39 | |
| sigmavirus24 | Is anyone else noticing that review.o.o is sluggish today? | 17:40 |
| fungi | sigmavirus24: sdague was noticing it | 17:40 |
| *** teran_ has joined #openstack-infra | 17:40 | |
| mtreinish | fungi: fwiw, 95 min is a very long time (if you're running in parallel) | 17:40 |
| BobBall | OK - one more question... When running a job kicked by zuul, jenkins does offline my node, but nodepool doesn't delete it. It doesn't seem to go into the Used state either, just sits there "Ready". Any ideas? | 17:40 |
| zaro | sigmavirus24: me too | 17:40 |
| clarkb | zaro: by default /p is served by gerrit itself so the normal proxying to gerrit makes /p work. However for us we want /p to bypass gerrit and hit the git repo replica on /var/lib/git. This requirse running a git cgi script so apache needs to be told that it is ok to do that | 17:40 |
| * sigmavirus24 wonders if we'll ever find the memory leak causing this | 17:40 | |
| fungi | mtreinish: this is the glusterfs job we turned off last night | 17:40 |
| clarkb | zaro: for 155208, should we build 2.8 and 2.9 instead of just 2.9? | 17:41 |
| *** teran has quit IRC | 17:41 | |
| *** Shrews has joined #openstack-infra | 17:41 | |
| fungi | sigmavirus24: no signs of packet loss or excessive latency between your system and review.o.o? | 17:41 |
| *** bhuvan has joined #openstack-infra | 17:41 | |
| sigmavirus24 | fungi: everything else seems to be just fine I haven't done a lot of investigation yet though | 17:41 |
| clarkb | fungi: no I am pretty sure its related to gerrits JVM heap size | 17:41 |
| clarkb | fungi: let me link the melody graph again | 17:41 |
| *** carl_baldwin has joined #openstack-infra | 17:42 | |
| *** gondoi is now known as zz_gondoi | 17:42 | |
| fungi | log-start-timestamp.txt says i started running it at 16:01:57 utc | 17:43 |
| *** markus_z has joined #openstack-infra | 17:43 | |
| zaro | fungi: can you check the gerrit queue to see if there are errors? | 17:43 |
| fungi | i want it to hurry up and break so i can go get lunch | 17:43 |
| fungi | zaro: 107 tasks, lots of pending push/upload | 17:44 |
| *** ParsectiX has quit IRC | 17:44 | |
| *** ParsectiX has joined #openstack-infra | 17:44 | |
| * StevenK peers at zuul | 17:44 | |
| clarkb | I think the whole thing gums up as it runs into memory pressure | 17:44 |
| *** timcline has quit IRC | 17:44 | |
| clarkb | https://review.openstack.org/monitoring?part=graph&graph=usedMemory is the week graph | 17:44 |
| clarkb | mean use has been in steady increase since monday | 17:45 |
| *** timcline has joined #openstack-infra | 17:45 | |
| *** unicell has quit IRC | 17:46 | |
| *** unicell has joined #openstack-infra | 17:46 | |
| *** markus_z has quit IRC | 17:46 | |
| krotscheck | Hrm, this puppet module is starting to look like it needs a more well developed conf manifest :/ | 17:46 |
| *** markus_z has joined #openstack-infra | 17:47 | |
| zaro | clarkb: good point about keeping the javamelody 2.8 build. but i don't see any development on it after the upgrade. | 17:47 |
| zaro | fungi: that doesn't seem too bad. are those mostly from replication? | 17:48 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Revamp README file https://review.openstack.org/157463 | 17:49 |
| clarkb | zaro: I am more thinking to that we have it if necessary (rollback etc) | 17:49 |
| zaro | clarkb: also wouldn't it get a little unwiedly if we start adding more plugins? | 17:49 |
| clarkb | zaro: then we remove 2.8 when we are happily on 2.9 | 17:49 |
| clarkb | zaro: I don't think so | 17:49 |
| fungi | zaro: the git push ones are (that's about half of them) but not sure if the git-upload-pack tasks (roughly the other half) are replication as well | 17:49 |
| *** BobBall is now known as BobBall_AWOL | 17:49 | |
| *** timcline has quit IRC | 17:50 | |
| fungi | though now the queue has dropped to 53 tasks so it does seem to be working through them | 17:50 |
| clarkb | https://review.openstack.org/monitoring?part=graph&graph=usedMemory&period=mois that graph is a lot more telling | 17:50 |
| clarkb | zaro: just instantiate the jobs multiple times with different parameters | 17:50 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Added new OAuth Parameters https://review.openstack.org/157464 | 17:51 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Added a Vagrant module and manifest for easier development. https://review.openstack.org/157465 | 17:51 |
| zaro | clarkb: i don't see why you would need it for rollback since the built version is still gonna be on tarballs | 17:51 |
| fungi | clarkb: score! no kernel panic (yet) but the console is now unresponsive to carriage returns and the df i had running in a loop is hanging | 17:51 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Revamp README file https://review.openstack.org/157463 | 17:52 |
| krotscheck | nibalizer: I'm trying to find some docs on best practice for module development - like, how to break out all the various config options into something shared. | 17:52 |
| krotscheck | nibalizer: Any pointers? | 17:52 |
| clarkb | fungi: I have never been this excited for brokeness before :) | 17:52 |
| clarkb | zaro: its more a matter of, we haev it today, so it is probably useful, therefore why remove it until we don't need it anymore | 17:52 |
| fungi | clarkb: unfortunately, i have no additional detail, so this doesn't help a bunch | 17:52 |
| *** HeOS has quit IRC | 17:52 | |
| *** salv-orlando has joined #openstack-infra | 17:52 | |
| clarkb | fungi: at least it is reproduceable :) | 17:53 |
| zaro | clarkb: ok, i have no arguement against that except that it adds code where not really necessary and requires slightly more maintenance going forward. | 17:53 |
| fungi | clarkb: the ip address still responds to icmp echo but attempting to open a tcp socket to port 22 times out | 17:54 |
| zaro | clarkb: i'm happy to update the change if you feel we should keep it around though. | 17:54 |
| fungi | clarkb: and both my existing ssh sessions seem to have hung | 17:54 |
| clarkb | fungi: anything on the console? eg panic? | 17:54 |
| mkoderer | hi folks, could someone add steveg and me to telcowg-usecases-core? | 17:54 |
| fungi | nope, though it's back to responding to carriage returns ...very...slowly | 17:55 |
| *** SumitNaiksatam has joined #openstack-infra | 17:55 | |
| nibalizer | krotscheck: eh not really | 17:55 |
| clarkb | mkoderer: sure, give me a minute | 17:55 |
| nibalizer | krotscheck: you're in seattle tho | 17:55 |
| mkoderer | clarkb: thx :) | 17:55 |
| nibalizer | so sit with crinkle | 17:55 |
| fungi | clarkb: it's too bad i can't send sysrq through rackspace's virtual console | 17:55 |
| *** Ryan_Lane has joined #openstack-infra | 17:56 | |
| clarkb | mkoderer: you are added but there is no 'steveg' account | 17:56 |
| krotscheck | Who? | 17:56 |
| clarkb | mkoderer: you should be able to add whomever you like at this point though | 17:56 |
| mkoderer | clarkb: ah great :) | 17:56 |
| fungi | clarkb: i'll have to try again to recreate this after lunch and leave myself a console backdoor open. this vm isn't likely to give us much else useful :/ | 17:57 |
| clarkb | fungi: ok thanks for the help | 17:57 |
| mkoderer | clarkb: ok worked thx again | 17:57 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Added a Vagrant module and manifest for easier development. https://review.openstack.org/157465 | 17:57 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Added new OAuth Parameters https://review.openstack.org/157464 | 17:57 |
| zaro | fungi: are you referring to ssh hang to review.o.o? | 17:57 |
| fungi | i lacked the foresight to create a working password login so i could do more than just play with the virtual console login prompt | 17:57 |
| clarkb | zaro: no | 17:57 |
| fungi | zaro: no, completely unrelated problem we've been trying to identify all morning | 17:58 |
| clarkb | zaro: the review.o.o slowness appears to be a memory leak in gerrit | 17:58 |
| clarkb | zaro: over time we use up the entire 8GB heap and things grind to a halt | 17:58 |
| fungi | okay, i'm going out to the burrito bar. back in a while | 17:58 |
| krotscheck | fungi: I hope that's a bar made out of burrito's | 17:59 |
| fungi | thankfully the weather has finally subsided to where we can leave the house again | 17:59 |
| StevenK | Haha | 17:59 |
| zaro | clarkb: nothing's changed for a while. so memory leak must be caused by something right? | 17:59 |
| fungi | krotscheck: it's a burrito made out of bars. steel bars | 17:59 |
| StevenK | fungi: Sounds rather hard to digest, really. | 17:59 |
| * krotscheck passes fungi the tums. | 17:59 | |
| clarkb | zaro: this has been going on for a while too | 17:59 |
| zaro | clarkb: any info from javamelody? | 17:59 |
| fungi | StevenK: it's an australian recipe | 18:00 |
| StevenK | Woo, zuul queue length is now >1,000 | 18:00 |
| clarkb | zaro: yes javamelody is how I have narrowed it down to memory pressure but haven't been able to narrow it from there | 18:00 |
| * fungi will bbiaw | 18:00 | |
| StevenK | fungi: Surely Australians would be using vegemite jars and beer cans? | 18:00 |
| clarkb | also cpu pegs but I think that is because the garbage collector is doing more work | 18:00 |
| *** bswartz has quit IRC | 18:01 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:03 | |
| clarkb | zaro: also restarting the service corrects the problem | 18:04 |
| *** bswartz has joined #openstack-infra | 18:04 | |
| *** AJaeger has joined #openstack-infra | 18:05 | |
| *** timcline has joined #openstack-infra | 18:07 | |
| *** olaph has joined #openstack-infra | 18:07 | |
| *** e0ne has quit IRC | 18:08 | |
| *** avozza is now known as zz_avozza | 18:09 | |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: Require openstack-doc-tools 0.22 https://review.openstack.org/157472 | 18:09 |
| *** harlowja has joined #openstack-infra | 18:10 | |
| *** jgallard__ has quit IRC | 18:10 | |
| *** doug-fish has joined #openstack-infra | 18:10 | |
| *** belmoreira has quit IRC | 18:11 | |
| *** sdake_ has joined #openstack-infra | 18:12 | |
| clarkb | javamelody reports that we are spending 10% of our time garbage collecitng now | 18:13 |
| clarkb | and that is a recent change | 18:13 |
| clarkb | will link that graph for people that have access shortly | 18:13 |
| clarkb | https://review.openstack.org/monitoring?part=graph&graph=gc | 18:14 |
| asselin | clarkb,^^ " Forbidden access" for me | 18:15 |
| *** teran has joined #openstack-infra | 18:15 | |
| *** teran has quit IRC | 18:15 | |
| olaph | same here | 18:15 |
| clarkb | asselin: ya melody is restricted to gerrit admins because it does a few things that are potentially scary | 18:15 |
| *** teran has joined #openstack-infra | 18:15 | |
| clarkb | though I can probably screenshot the graph and share that | 18:15 |
| *** ParsectiX has quit IRC | 18:16 | |
| *** ParsectiX has joined #openstack-infra | 18:16 | |
| *** sdake_ has quit IRC | 18:17 | |
| *** otter768 has joined #openstack-infra | 18:18 | |
| *** teran_ has quit IRC | 18:18 | |
| *** otter768 has quit IRC | 18:22 | |
| clarkb | http://i.imgur.com/9c3ZVzr.png | 18:22 |
| clarkb | zaro: olaph asselin ^ | 18:22 |
| clarkb | zaro: any other ideas on how to debug this? I am gessing we will need a dump of the heap to figure out where the memory is being used | 18:25 |
| zaro | clarkb: yeah, dump might help | 18:27 |
| *** andreaf_ has joined #openstack-infra | 18:29 | |
| *** prad has quit IRC | 18:30 | |
| clarkb | greghaynes: commented on your ssh key in dib change. needs small puppet fix | 18:31 |
| *** achanda has joined #openstack-infra | 18:31 | |
| *** pradk is now known as prad | 18:31 | |
| *** cnesa1 has joined #openstack-infra | 18:35 | |
| *** cnesa has quit IRC | 18:35 | |
| *** sputnik13 has joined #openstack-infra | 18:35 | |
| bcourt | this might be a silly question, how does nodepool validate that an image built correctly? Is it checking the return code of the prepare node script? | 18:37 |
| clarkb | bcourt: it has two bits that do that. First is image building where it checks the return code of the prepare node script. Second is the ready node script which it runs post boot and also checks the return code on | 18:39 |
| clarkb | bcourt: so if nodeool cannot ssh into a node and subsequently run whatever ready script it has successfully that node will be deleted and a new one booted | 18:39 |
| bcourt | clarkb, thanks | 18:40 |
| *** EmilienM is now known as EmilienM|afk | 18:40 | |
| *** hdd has quit IRC | 18:41 | |
| *** ashleighfarnham has quit IRC | 18:41 | |
| *** dizquierdo has quit IRC | 18:42 | |
| *** ashleighfarnham has joined #openstack-infra | 18:42 | |
| greghaynes | clarkb: im guessing puppet doesnt mind a trailing end comma? | 18:42 |
| greghaynes | because I also have that going on | 18:42 |
| clarkb | greghaynes: it does not (at least not in puppet3 it did way back when) | 18:42 |
| greghaynes | kk | 18:42 |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Respect NODEPOOL_SSH_KEY in prepare-node for dib https://review.openstack.org/157205 | 18:43 |
| *** melwitt has joined #openstack-infra | 18:43 | |
| *** BharatK has quit IRC | 18:43 | |
| nibalizer | ogod greghaynes is writing poopit? | 18:44 |
| * nibalizer watches for other signs of the apocalypes | 18:44 | |
| *** Sukhdev has joined #openstack-infra | 18:47 | |
| *** sdake_ has joined #openstack-infra | 18:47 | |
| tchaypo | jeblair: i *finally* managed to invoke gertty’s search functionality | 18:47 |
| greghaynes | nibalizer: most certainly not | 18:48 |
| tchaypo | nibalizer: have you looked at the weather map of the eastern seaboard? | 18:49 |
| tchaypo | my hometown had an earthquake last week and is expecting a hurricane tomorrow | 18:49 |
| *** teran has quit IRC | 18:50 | |
| *** MaxV has joined #openstack-infra | 18:50 | |
| dstufft | sdague: do you just ant a commit message like "stevedore depends on this but doesn't declare it, so let's declare it" ? | 18:52 |
| *** ashleighfarnham has quit IRC | 18:53 | |
| openstackgerrit | Dan Prince proposed openstack-infra/project-config: Remove check job from tripleo-puppet-elements https://review.openstack.org/157486 | 18:54 |
| sdague | dstufft: sure | 18:54 |
| *** ssam2 has quit IRC | 18:54 | |
| *** yolanda has quit IRC | 18:57 | |
| openstackgerrit | James Polley proposed stackforge/gertty: Expand sample keymaps to ameliorate OSX features https://review.openstack.org/157488 | 18:57 |
| *** salv-orlando has quit IRC | 18:57 | |
| *** yolanda has joined #openstack-infra | 18:57 | |
| openstackgerrit | Donald Stufft proposed openstack/requirements: Add setuptools to global requirements https://review.openstack.org/155498 | 18:57 |
| dstufft | sdague: ^^ | 18:58 |
| *** dims_ has joined #openstack-infra | 18:59 | |
| dstufft | dhellmann: ^^ | 18:59 |
| openstackgerrit | Dan Prince proposed openstack-infra/project-config: Update prepare_tripleo.sh to use git.openstack.org https://review.openstack.org/157490 | 19:00 |
| *** dims__ has quit IRC | 19:02 | |
| openstackgerrit | James Polley proposed stackforge/gertty: Print keymap command names correctly https://review.openstack.org/157492 | 19:03 |
| *** bswartz has quit IRC | 19:03 | |
| *** yfried|afk has quit IRC | 19:04 | |
| *** e0ne has joined #openstack-infra | 19:06 | |
| *** ghostpl_ has quit IRC | 19:06 | |
| *** r-daneel has joined #openstack-infra | 19:10 | |
| *** achanda has quit IRC | 19:11 | |
| *** mestery has joined #openstack-infra | 19:12 | |
| *** TravT_ has joined #openstack-infra | 19:12 | |
| *** Guest21555 has quit IRC | 19:14 | |
| *** Hal has joined #openstack-infra | 19:14 | |
| *** Hal is now known as Guest42102 | 19:15 | |
| zaro | clarkb: Just a guess, it might be sshd causing memory leak, similar to https://code.google.com/p/gerrit/issues/detail?id=2003 , dump might narrow it down. | 19:15 |
| *** dustins has quit IRC | 19:15 | |
| TravT_ | Hello infra: if anybody has some time to look at a couple of requirements patches we have up. Would appreciate it. | 19:17 |
| TravT_ | we're running a "virtual sprint" in horizon | 19:17 |
| TravT_ | where everybody comes together for several hours on code reviews | 19:17 |
| TravT_ | and the patches that need a couple of requirements are coming up. | 19:18 |
| *** MarkAtwood has quit IRC | 19:18 | |
| TravT_ | https://review.openstack.org/#/c/147362/ | 19:18 |
| TravT_ | https://review.openstack.org/#/c/155625/ | 19:18 |
| *** r-daneel has quit IRC | 19:19 | |
| zaro | clarkb: Gerrit 2.8 is using sshd ver 0.11.0, Gerrit 2.9 reverted to using ver 0.9.0 | 19:19 |
| *** Guest42102 has quit IRC | 19:19 | |
| *** sarob has joined #openstack-infra | 19:19 | |
| AJaeger | TravT_: infra is just the fallback for requirements, see here for list of cores: https://review.openstack.org/#/admin/projects/openstack/requirements,access | 19:19 |
| TravT_ | thanks, also figured dropping a note in here might help. | 19:20 |
| *** HeOS has joined #openstack-infra | 19:23 | |
| *** MaxV has quit IRC | 19:23 | |
| *** markus_z has quit IRC | 19:23 | |
| *** bswartz has joined #openstack-infra | 19:24 | |
| *** dims_ has quit IRC | 19:26 | |
| greghaynes | huh, my loal nodepool dib image is failing to build because the 90-venv-swift-logs install.d needs virtualenv but this isnt installed | 19:26 |
| greghaynes | do we not have an element that does this? | 19:26 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Added a Vagrant module and manifest for easier development. https://review.openstack.org/157465 | 19:27 |
| asselin | anyone seen this nodepool exception deleting a node? http://paste.openstack.org/show/177836/ | 19:28 |
| *** vigneshvar has joined #openstack-infra | 19:28 | |
| otherwiseguy | So, all neutron patches seem to be failing on gate-neutron-pylint all of a sudden. | 19:29 |
| fungi | otherwiseguy: new version of pylint maybe? | 19:29 |
| otherwiseguy | ala https://jenkins04.openstack.org/job/gate-neutron-pylint/1383/console | 19:29 |
| *** tiswanso has joined #openstack-infra | 19:30 | |
| *** dustins has joined #openstack-infra | 19:30 | |
| fungi | hrm, no new pylint for a month | 19:30 |
| fungi | clarkb: interesting update, i left my hung ssh sessions going while lunching. on returning i found they'd recovered | 19:30 |
| *** abhirc has quit IRC | 19:31 | |
| fungi | clarkb: devstack-gate _did_ eventually successfully timeout and kill tempest | 19:31 |
| otherwiseguy | Looks like a change in novaclient or something maybe. | 19:31 |
| * otherwiseguy investigates | 19:31 | |
| *** BobBall_AWOL is now known as BobBall | 19:32 | |
| fungi | clarkb: oom | 19:32 |
| fungi | clarkb: Killed process 6448 (mysqld) total-vm:4797460kB, anon-rss:228456kB, file-rss:0kB | 19:32 |
| *** salv-orlando has joined #openstack-infra | 19:32 | |
| fungi | otherwiseguy: i have to wonder why neutron pylint jobs would be checking novaclient source | 19:34 |
| openstackgerrit | James Graf proposed openstack-infra/jenkins-job-builder: Add support for Job DSL plugin for Groovy-based job and view creation and management https://review.openstack.org/150112 | 19:34 |
| *** achanda has joined #openstack-infra | 19:34 | |
| otherwiseguy | neutron has a notifier that uses novaclient | 19:34 |
| BobBall | What should be setting nodepool nodes to USED? Zuul successfully runs a job on a node, but it's not transitioning to USED then to deleting - presumably because the onStarted messages aren't sent and/or received, but I don't see where they are sent from and why they might not be being sent. | 19:35 |
| *** mpavone has joined #openstack-infra | 19:35 | |
| fungi | otherwiseguy: oh, is pylint failing in some way other than pointing out lint-specific issues? | 19:35 |
| otherwiseguy | No name 'client' in module 'novaclient.v1_1 | 19:35 |
| fungi | like hitting an import failure or something? | 19:35 |
| fungi | yeah, that seems likely then | 19:35 |
| otherwiseguy | No name 'contrib' in module 'novaclient.v1_1 | 19:35 |
| fungi | so the pylint run isn't so much failing ans completely breaking | 19:36 |
| otherwiseguy | from lines: | 19:36 |
| otherwiseguy | 18 import novaclient.v1_1.client as nclient | 19:36 |
| otherwiseguy | 19 from novaclient.v1_1.contrib import server_external_events | 19:36 |
| fungi | probably was a novaclient release today | 19:36 |
| fungi | yep, see https://pypi.python.org/pypi/python-novaclient and -dev ml thread | 19:37 |
| fungi | 2.21.0 release today | 19:37 |
| otherwiseguy | why would the v1_1 stuff change with a new release? | 19:37 |
| * otherwiseguy sighs | 19:37 | |
| otherwiseguy | So close to getting everything merged. No more 20-patch-long chains for me. :p | 19:38 |
| *** krtaylor has quit IRC | 19:38 | |
| *** tiswanso has quit IRC | 19:38 | |
| fungi | otherwiseguy: i would suggest asking the novaclient devs for details (though a quick hack through git diff 2.20.0..2.21.0 and some selective blaming would get you a commit message and possibly bug/spec link with details) | 19:38 |
| *** vigneshvar has quit IRC | 19:38 | |
| *** tiswanso has joined #openstack-infra | 19:39 | |
| *** TravT_ has left #openstack-infra | 19:39 | |
| otherwiseguy | https://github.com/openstack/python-novaclient/blob/master/novaclient/v1_1/__init__.py | 19:39 |
| otherwiseguy | "Rename v1_1 to v2" for a patch name sounds ominous. | 19:40 |
| fungi | heh | 19:40 |
| fungi | indeed! | 19:40 |
| fungi | if that was a public api, then leaving v1_1 aliased to that would have been nice | 19:41 |
| *** mrunge has joined #openstack-infra | 19:41 | |
| otherwiseguy | fungi: yes, yes it would have been. | 19:41 |
| fungi | otherwiseguy: might be a quick patch and 2.21.1 brown-bag release away | 19:42 |
| fungi | i suggest following up to the release announcement on the -dev ml | 19:42 |
| openstackgerrit | Dan Prince proposed openstack-infra/project-config: Actually test tripleo-puppet-elements https://review.openstack.org/157503 | 19:43 |
| *** pelix has quit IRC | 19:44 | |
| *** mestery has quit IRC | 19:44 | |
| fungi | also, mikal ^ if you're around, see otherwiseguy's shiny new brokenness | 19:44 |
| clarkb | fungi interesting | 19:46 |
| clarkb | I am running errands for sick wife should be able to help debug in a while | 19:46 |
| fungi | clarkb: take your time. it will keep | 19:46 |
| clarkb | BobBall nodepool listens to the jenkins zmq event publisher | 19:46 |
| BobBall | *sigh* somehow that had been disabled. That'll be the issue then. Thanks again clarkb. | 19:47 |
| fungi | BobBall: sounds like you need to take away someone's mouse finger | 19:48 |
| mordred | fungi: wait - what? | 19:48 |
| mordred | they renamed v1_1 to v2? | 19:48 |
| BobBall | *grin* | 19:48 |
| fungi | mordred: what the what what? | 19:48 |
| * mordred reading scrollback | 19:48 | |
| fungi | mordred: yeah, but not in anything widely used, just you know novaclient | 19:48 |
| mordred | AWESOME | 19:48 |
| fungi | and apparently at least some feature of neutron (master at a minimum) was importing from it | 19:49 |
| jroll | o.o | 19:49 |
| fungi | i guess not in devstack-deployed scenarios, so only tripped on neutron's pylint job | 19:50 |
| fungi | hence not noticed until a new novaclient release happened | 19:50 |
| *** erlon has quit IRC | 19:51 | |
| dhellmann | dstufft: +2a | 19:52 |
| mordred | fungi: well, also - I USE THAT CODE PATH | 19:52 |
| fungi | mordred: not any more you don't | 19:52 |
| mordred | mikal, sdague, jogo: ^^^ I would like to complain about your breakage of library | 19:52 |
| fungi | (well, until they fix it hopefully) | 19:53 |
| mordred | https://github.com/openstack-infra/shade/blob/master/shade/__init__.py#L27 | 19:54 |
| mordred | https://github.com/openstack-infra/nodepool/blob/master/nodepool/provider_manager.py#L24 | 19:54 |
| mordred | https://github.com/openstack-infra/system-config/blob/master/launch/launch-node.py#L50 | 19:54 |
| otherwiseguy | fungi: is it possible to temporarily disable neutron's pylint test voting? | 19:54 |
| mordred | mikal, sdague, jogo: ^^^ it turns out that infra uses that code path | 19:54 |
| *** yamamoto has joined #openstack-infra | 19:55 | |
| mordred | and I find removing it from a widely used library yuck | 19:55 |
| otherwiseguy | mordred: it looked like they just modify sys.modules directly to alias it, which pylint doesn't see? | 19:55 |
| fungi | otherwiseguy: yeah, you could override it in neutron's tox.ini to just always return true, or submit a patch to openstack-infra/project-config to override it to voting:false in zuul/layout.yaml | 19:55 |
| mordred | otherwiseguy: oh, how clever | 19:56 |
| otherwiseguy | mordred: https://github.com/openstack/python-novaclient/blob/master/novaclient/v1_1/__init__.py#L43 | 19:56 |
| mordred | otherwiseguy: sigh | 19:58 |
| *** yamamoto has quit IRC | 19:59 | |
| openstackgerrit | Terry Wilson proposed openstack-infra/project-config: Disable voting on pylint because of novaclient breakage https://review.openstack.org/157507 | 19:59 |
| otherwiseguy | fungi: ^ like that? | 19:59 |
| * otherwiseguy has no idea what he is doing | 20:00 | |
| fungi | it's entirely possible that this could also be considered a http://xkcd.com/1172 on pylint's part | 20:00 |
| mordred | ok - I will scream less - I have tested that the code paths work | 20:00 |
| mordred | although now they throw deprecation warnings | 20:01 |
| mordred | it would be great if someone had an example of the "right" way | 20:01 |
| *** dims__ has joined #openstack-infra | 20:01 | |
| fungi | otherwiseguy: also further down in the openstack/neutron entry for the projects list take it out of the gate pipeline | 20:01 |
| *** bswartz has quit IRC | 20:01 | |
| *** crc32 has joined #openstack-infra | 20:02 | |
| fungi | otherwiseguy: this way it runs non-voting in the check pipeline and not at all in the gate pipeline | 20:02 |
| openstackgerrit | Terry Wilson proposed openstack-infra/project-config: Disable voting on pylint because of novaclient breakage https://review.openstack.org/157507 | 20:03 |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 20:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Import from v2 instead of v1_1 https://review.openstack.org/157509 | 20:04 |
| vipul | question about global-requirements.. is there a reason why python-openstackclient isn't there? noticed most of the other clients exist | 20:05 |
| otherwiseguy | mordred: https://github.com/openstack/python-novaclient/blob/master/novaclient/v1_1/__init__.py#L32 | 20:05 |
| otherwiseguy | :) | 20:05 |
| fungi | vipul: what other projects use python-openstackclient? | 20:05 |
| otherwiseguy | mordred: looks like https://github.com/openstack/python-novaclient/blob/master/novaclient/v1_1/__init__.py#L32 | 20:05 |
| otherwiseguy | doh | 20:05 |
| *** dims__ has quit IRC | 20:05 | |
| fungi | vipul: that is, what is it a requirement of? | 20:05 |
| otherwiseguy | mordred: I'm guessing https://github.com/openstack/python-novaclient/blob/master/novaclient/client.py#L778 get_client_class() | 20:05 |
| vipul | fungi: https://github.com/stackforge/python-cueclient/blob/master/requirements.txt | 20:06 |
| *** crc32 has quit IRC | 20:06 | |
| vipul | fungi: also... https://github.com/stackforge/python-congressclient/blob/master/requirements.txt | 20:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Change references from v1_1 to v2 https://review.openstack.org/157512 | 20:07 |
| mordred | otherwiseguy: awsome | 20:07 |
| fungi | vipul: weird. what do they use openstackclient for? just avoiding depending on the python sdk (novaclient, neutronclient, et cetera)? | 20:07 |
| *** r1chardj0n3s has left #openstack-infra | 20:07 | |
| vipul | fungi: they are implemented as 'plugins' to python-openstackclient rather than introducing their own shell | 20:08 |
| *** dprince has quit IRC | 20:08 | |
| vipul | fungi: https://github.com/stackforge/python-congressclient/blob/master/setup.cfg#L27 | 20:08 |
| fungi | vipul: that seems like it gets the dependency backwards then... you're saying they don't use openstackclient, but openstackclient uses them? | 20:09 |
| mordred | no - they're right | 20:09 |
| mordred | if you want to install cueclient | 20:09 |
| mordred | you install it | 20:09 |
| *** ghostpl_ has joined #openstack-infra | 20:09 | |
| mordred | and it installs an _additional_ optional plugin for python-openstackclient | 20:09 |
| *** MaxV has joined #openstack-infra | 20:09 | |
| fungi | oh, so it's still invoked ac cueclient, not as a plugin to openstackclient | 20:09 |
| mordred | that is then consumed by running openstackclient | 20:09 |
| mordred | nope | 20:09 |
| mordred | you run openstackclient | 20:09 |
| vipul | essentially it allows you to run 'openstack <your plugin command>' | 20:09 |
| fungi | er | 20:09 |
| mordred | but if you install the cueclient plugin, you get more commands in openstackclient | 20:09 |
| mordred | so openstackclient doesn't depend on it | 20:09 |
| mordred | but cueclient is useless without openstackclient | 20:10 |
| fungi | i mean, the same could be said of any optional dependency i guess | 20:10 |
| mordred | yes | 20:10 |
| mordred | it's like "apt-get install firefox-plugin-lightning" which you'd expect to wind up also installing firefox if it's not installed | 20:10 |
| vipul | mordred: great example ! | 20:11 |
| fungi | yep, that analogy works | 20:12 |
| vipul | shall i submit a patch? fungi | 20:12 |
| fungi | so this goes back to the "should we include dependencies only requested by stackforge projects in the openstack projects global requirements list?" debate | 20:13 |
| mordred | fungi: I honestly think python-openstackclient is fair game for requirements | 20:13 |
| fungi | vipul: feel free to submit a patch. i don't generally vote on requirements patches though so my opinion should be taken with a grain of salt | 20:13 |
| mordred | without having to go down the rabbithole | 20:13 |
| vipul | should stackforge projects be allowed to plugin to devstack? | 20:13 |
| *** ghostpl_ has quit IRC | 20:13 | |
| *** mestery has joined #openstack-infra | 20:13 | |
| *** ghostpl_ has joined #openstack-infra | 20:13 | |
| vipul | this is where this dependency is needed.. when setting up devstack | 20:13 |
| vipul | fungi: will do | 20:14 |
| *** aysyd has quit IRC | 20:14 | |
| fungi | vipul: we have soft requirements where unofficial projects can set a flag that lets them include requirements which aren't in the official global requirements list | 20:14 |
| fungi | in devstack-based jobs | 20:14 |
| vipul | something i can set in my local.conf? | 20:15 |
| fungi | it basically turns off devstack's complaint about using a package which isn't in global requirements | 20:15 |
| * fungi gets the details | 20:15 | |
| *** krtaylor has joined #openstack-infra | 20:16 | |
| *** dprince has joined #openstack-infra | 20:16 | |
| fungi | REQUIREMENTS_MODE=soft | 20:16 |
| fungi | looking for an example job definition to point you at now | 20:16 |
| vipul | fungi: awesome i'll try that.. thx for the help | 20:16 |
| *** otter768 has joined #openstack-infra | 20:19 | |
| fungi | yeah, REQUIREMENTS_MODE defaults to "strict" in devstack-gate but can be overridden from the calling environment, and then passes that into devstack which also defaults to "strict" but bypasses that extra requirements check if it's "soft" instead | 20:19 |
| fungi | however i don't see any current jobs passing that into devstack-gate, in theory it's an implemented and working feature | 20:19 |
| *** tkelsey has quit IRC | 20:20 | |
| *** rfolco has quit IRC | 20:22 | |
| *** ociuhandu has quit IRC | 20:23 | |
| *** otter768 has quit IRC | 20:23 | |
| *** abhirc has joined #openstack-infra | 20:25 | |
| *** achanda has quit IRC | 20:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Disable voting on pylint because of novaclient breakage https://review.openstack.org/157507 | 20:28 |
| AJaeger | otherwiseguy: once pylint passes again, please propose a revert of the patch | 20:29 |
| otherwiseguy | AJaeger: will do. Also have a patch that should adapt to the new novaclient change pending in the neutron queue. | 20:29 |
| AJaeger | otherwiseguy: great! | 20:30 |
| otherwiseguy | I haven't run even basic testing on it, and have no idea if I did things right because I pretty much just wrote and submitted to see what would happen. | 20:30 |
| otherwiseguy | (the neutron way?) /js | 20:30 |
| otherwiseguy | er, 's | 20:30 |
| * otherwiseguy gives up | 20:31 | |
| *** ihrachyshka has joined #openstack-infra | 20:32 | |
| *** achanda has joined #openstack-infra | 20:32 | |
| jogo | mordred: looking into it | 20:37 |
| *** MaxV has quit IRC | 20:37 | |
| *** hdd has joined #openstack-infra | 20:37 | |
| otherwiseguy | weird, disconnected right as I sent that message. Guest79? | 20:39 |
| *** tkelsey has joined #openstack-infra | 20:42 | |
| *** bswartz has joined #openstack-infra | 20:42 | |
| openstackgerrit | mbitard proposed openstack-infra/jenkins-job-builder: add slack notification support, largely inspired by hipchat module https://review.openstack.org/136627 | 20:43 |
| *** ddieterly has quit IRC | 20:44 | |
| *** Sukhdev has quit IRC | 20:46 | |
| *** e0ne has quit IRC | 20:47 | |
| jogo | mordred: we didn't break anything novaclient.v1_1 still works https://review.openstack.org/#/c/149006/13/novaclient/v1_1/__init__.py,cm | 20:47 |
| *** jedimike has quit IRC | 20:47 | |
| jogo | mordred: its just pylint isn't smart | 20:48 |
| fungi | jogo: yeah, it turns out it just wasn't backward-compatible in a way which pylint could follow | 20:48 |
| mikal | mordred: I find the tone of your conversation intriguing | 20:48 |
| clarkb | ok mostly back now | 20:48 |
| jogo | fungi: silly pylint | 20:48 |
| *** ZZelle_ has joined #openstack-infra | 20:48 | |
| fungi | jogo: i like to anagram it as "pintly" | 20:49 |
| *** ddieterly has joined #openstack-infra | 20:49 | |
| jogo | haha | 20:49 |
| *** arosen has quit IRC | 20:50 | |
| *** yamahata has quit IRC | 20:51 | |
| fungi | jogo: a cursory review of novaclient commit subject lines since the prior release turned up a change which looked like it was moving v1_1 to v2 which would have explained the pylint error, but closer examination revealed that it was still aliased to the new name and pylint was merely a mendicant | 20:51 |
| *** mpavone has quit IRC | 20:52 | |
| *** w_ has joined #openstack-infra | 20:52 | |
| *** dkranz has quit IRC | 20:52 | |
| *** gyee has joined #openstack-infra | 20:53 | |
| mikal | jogo: but we still managed to make it sound like the nova team is incompetent, just to keep it real | 20:53 |
| mordred | mikal: I'd like to sell you some real estate | 20:53 |
| *** sdake_ has quit IRC | 20:54 | |
| clarkb | hrm if I were to make a heap dump of gerrit I doubt I could share it because secrets | 20:54 |
| clarkb | also I doubt that I will have the patience to dig through it to find a memory leak | 20:55 |
| fungi | clarkb: yeah we'd probably need to spend quite a lot of time to filter than down and sanitize/redact things | 20:55 |
| mriedem | mordred | 20:55 |
| mriedem | oops, disregard | 20:56 |
| clarkb | so I am leaning towards just restarting it, but can figure out jmap if someone really wants that heap data and its safe to share it with them | 20:56 |
| clarkb | fungi: jeblair mordred jhesketh SergeyLukjanov pleia2 ^ | 20:56 |
| *** sdake_ has joined #openstack-infra | 20:56 | |
| fungi | clarkb: i wouldn't know what to do with it, but if it's easy to dump and stash for now then we can always trot it out later to answer more directed questions if there's a gerritite willing to help us troubleshoot post-mortem | 20:57 |
| clarkb | wait people willingly use pylint? | 20:57 |
| clarkb | fungi: well I think it will make an 8GB binary file | 20:57 |
| fungi | clarkb: yes, neutron (recentlyish i believe) added it back | 20:57 |
| clarkb | fungi: which would then have to be sanitized | 20:58 |
| *** arosen has joined #openstack-infra | 20:58 | |
| *** ZZelle__ has joined #openstack-infra | 20:58 | |
| *** ZZelle__ has quit IRC | 20:58 | |
| clarkb | and jmap isn't in my path on review.o.o | 20:58 |
| clarkb | zaro: ^ is jmap what I should be using? the internet says it is but I am a noob at jvm debugging tools | 20:59 |
| fungi | clarkb: if memory serves the reasoning was that pyflakes' static analysis didn't cover some of the problems deeper inspection via pylint pointed out | 20:59 |
| clarkb | fungi: ya but pylint also points out a lot of things that aren't issues | 20:59 |
| clarkb | its false positive rate is incredibly high | 20:59 |
| BobBall | fungi: Did you get to the bottom of the EOF issue from earlier? I'm hitting it too on my Ubuntu nodes - testing with Xen. During the devstack setup script it dies | 20:59 |
| dansmith | clarkb: yeah I was thinking "holy crap, voting pylint in a gate?" | 20:59 |
| fungi | and also to achieve that deeper analysis it has to do things like import modules outside of normal python mechanisms | 21:00 |
| fungi | hence... this | 21:00 |
| fungi | BobBall: eof problem what now? | 21:00 |
| *** crc32 has joined #openstack-infra | 21:00 | |
| anteaya | clarkb: nah, you can go ahead on https://review.openstack.org/#/c/157330 | 21:00 |
| BobBall | fungi: SSH session just dying causing an exception log in jenkins | 21:01 |
| clarkb | anteaya: thanks | 21:01 |
| BobBall | fungi: it was deepakcs problem with the glusterfs jobs | 21:01 |
| fungi | BobBall: for glusterfs? | 21:01 |
| fungi | BobBall: the held worker i ran it manually on eventually recovered, though the kernel oom killer did shoot mysqld in the head along the way | 21:01 |
| openstackgerrit | mbitard proposed openstack-infra/jenkins-job-builder: add slack notification support, largely inspired by hipchat module https://review.openstack.org/136627 | 21:02 |
| BobBall | Hmmmz - interesting... | 21:02 |
| *** sdake_ has quit IRC | 21:02 | |
| otherwiseguy | clarkb: yeah, neutron gates on pylint...after disabling at least 50 tests. | 21:02 |
| otherwiseguy | :) | 21:02 |
| BobBall | Also interesting I just noticed that nodepool failed to kill the nodes that seem to be dying... saying they don't exist. Which sound highly fatal and interesting to me... | 21:02 |
| *** sdake_ has joined #openstack-infra | 21:03 | |
| clarkb | supposedly jmap comes from the jdk, we must only install jre. I am confirming that. There is an option to have signal 3 create a heap dump which I may enable post restart for next time this happens | 21:03 |
| fungi | i started devstack-gate at 16:01:57 and the kernel killed mysqld at 18:47:08 so that may have been unrelated breakage after the fact | 21:03 |
| BobBall | ah no - I misread the log - it was the node in jenkins that had disappeared | 21:03 |
| *** unicell has quit IRC | 21:04 | |
| *** unicell has joined #openstack-infra | 21:04 | |
| * BobBall holds a node to try and get some debug info out | 21:04 | |
| clarkb | ya we install the openjdk-7-jre-headless package so no jmap. My vote is for not doing a heap dump today, but add the flag to the jvm that will do a heap dump on signal 3 when we restart gerrit | 21:04 |
| clarkb | I will write the puppet change for that now | 21:04 |
| fungi | ahh, nope, timing works out. mysqld oom'd about 4 seconds before the devstack-gate-cleanup-host.txt log starts | 21:05 |
| BobBall | Ah - yes. I bet mine is an OOM too. | 21:06 |
| jogo | mordred: btw did you ever re-fix the pbr/setuptools issue with creating binaries | 21:06 |
| BobBall | *fixes dom0 RAM*... | 21:06 |
| *** achanda has quit IRC | 21:06 | |
| BobBall | I wonder if we should give the OOMkiller some hints not to kill SSH | 21:06 |
| *** sabeen2 has joined #openstack-infra | 21:06 | |
| jogo | mordred: a release with https://review.openstack.org/#/c/150803/ in it that is | 21:07 |
| BobBall | but perhaps to kill the d-g-wrap script first which should let us extract logs perhaps? | 21:07 |
| mordred | No. Thanks for reminder | 21:07 |
| clarkb | BobBall: at one point we did use cgroups for that. not sure if those are still in use | 21:07 |
| *** sabeen has quit IRC | 21:07 | |
| clarkb | BobBall: basically set a lower memory threshhold for the safe processes allowing ssh to survive | 21:08 |
| *** mpavone has joined #openstack-infra | 21:08 | |
| *** sdake_ has quit IRC | 21:09 | |
| *** doude has quit IRC | 21:10 | |
| *** doude has joined #openstack-infra | 21:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Clone using git.o.o instead of directly from gerrit https://review.openstack.org/157330 | 21:11 |
| *** mpavone has quit IRC | 21:13 | |
| *** abhirc has quit IRC | 21:13 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerrit: Collect Gerrit heap dumps on signal 3 https://review.openstack.org/157527 | 21:14 |
| clarkb | zaro: fungi ^ something like that | 21:14 |
| clarkb | I should probably test that on review dev first though | 21:14 |
| clarkb | zaro: is it ok for me to use review-dev for some testing of that? | 21:15 |
| AJaeger | clarkb, fungi: If either of you have some spare review cycles, I would like to unblock the training-guides folks: https://review.openstack.org/#/c/152118/ | 21:15 |
| AJaeger | could you review it, please? | 21:15 |
| clarkb | AJaeger: sure | 21:16 |
| AJaeger | There's on +2 already - and I did the last change, so don't want to +2 myself ;) | 21:16 |
| AJaeger | thanks, clarkb! | 21:16 |
| sarob | \o/ | 21:16 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Add a job for Gerrit javamelody plugin ver 2.9 https://review.openstack.org/155208 | 21:17 |
| *** abhirc has joined #openstack-infra | 21:17 | |
| jogo | wee cells broke the gate | 21:18 |
| *** markmcclain has quit IRC | 21:18 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting https://review.openstack.org/157530 | 21:19 |
| jogo | mriedem alaski: ^ | 21:19 |
| jogo | fungi clarkb: ^ that should unwedge the world | 21:20 |
| *** mbacchi has quit IRC | 21:20 | |
| clarkb | hahahahahahahaha I figured out why my puppet env is borken | 21:20 |
| clarkb | I fetched puppet-gerrit into system-config | 21:21 |
| clarkb | such shame | 21:21 |
| clarkb | jogo: why not revert the nova change that broke cells? | 21:21 |
| jogo | clarkb: we don't know what it is yet | 21:21 |
| jogo | plus that job should really go bye bte | 21:21 |
| jogo | plus that job should really go bye bye | 21:21 |
| *** timcline has quit IRC | 21:21 | |
| *** ghostpl_ has quit IRC | 21:22 | |
| *** mbacchi has joined #openstack-infra | 21:22 | |
| clarkb | then delete the job don't make it non voting... | 21:22 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting https://review.openstack.org/157530 | 21:22 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting https://review.openstack.org/157530 | 21:22 |
| clarkb | zaro: ok I don't hear any nays to me restarting review-dev with those jvm options | 21:23 |
| clarkb | zaro: I am going to restart it shortly if I don't hear otherwise | 21:23 |
| alaski | jogo: why do you want to remove that job? It works as a good smoke test for cells | 21:23 |
| jogo | alaski: first things first | 21:23 |
| jogo | alaski: lets make it non-voting to unwedge everything | 21:23 |
| jogo | then we can have the to remove or not discussion | 21:24 |
| *** achanda has joined #openstack-infra | 21:24 | |
| *** signed8bit has quit IRC | 21:24 | |
| *** signed8bit has joined #openstack-infra | 21:24 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for cells bug 1423695 https://review.openstack.org/157532 | 21:25 |
| openstack | bug 1423695 in OpenStack Compute (nova) "gate-devstack-dsvm-cells fails attaching volume" [Undecided,New] https://launchpad.net/bugs/1423695 | 21:25 |
| *** AJaeger has quit IRC | 21:25 | |
| openstackgerrit | Merged openstack-infra/project-config: Updates training guides jobs https://review.openstack.org/152118 | 21:26 |
| *** mrmartin has quit IRC | 21:26 | |
| * clarkb restarts review-dev | 21:27 | |
| pleia2 | oof, between conference and related events, and the time change, I've been totally not here all week | 21:27 |
| anteaya | pleia2: how goes the long sleeves in hot weather set? | 21:28 |
| *** mbacchi has quit IRC | 21:28 | |
| pleia2 | anteaya: they have amazing air conditioning, and it's a dry heat, so when we do go outside it's ok | 21:28 |
| *** ddieterly has quit IRC | 21:28 | |
| anteaya | hehehe | 21:28 |
| anteaya | awesome | 21:28 |
| anteaya | how is the food? | 21:28 |
| *** ddieterly has joined #openstack-infra | 21:28 | |
| clarkb | woo my change totally makes gerrit not start :) | 21:29 |
| pleia2 | very indian-influenced, so it's good | 21:29 |
| clarkb | zaro: any idea why? | 21:29 |
| jogo | andreaf: ping, see my comment on https://review.openstack.org/#/c/157530/3 | 21:29 |
| anteaya | pleia2: sounds yummy | 21:29 |
| anteaya | pleia2: when do you fly again? | 21:29 |
| pleia2 | anteaya: it is :) | 21:29 |
| *** dims__ has joined #openstack-infra | 21:29 | |
| pleia2 | my first flight is in 4 hours | 21:29 |
| anteaya | you are in an airport then, yes? | 21:29 |
| anteaya | how are the dates and figs there? | 21:30 |
| pleia2 | nah, just wrapping up at my hotel, heading over in an hour | 21:30 |
| anteaya | cool | 21:30 |
| pleia2 | dates and figs are good! | 21:30 |
| anteaya | I bet | 21:30 |
| anteaya | that would be what I would consider a highlight | 21:30 |
| pleia2 | and they drink a lot of juice here | 21:30 |
| anteaya | do they | 21:30 |
| anteaya | what kind? | 21:30 |
| pleia2 | mango, orange, pineapple | 21:30 |
| anteaya | sounds tasty | 21:30 |
| pleia2 | it's a dry country, so most places where we'd have alcohol are replaced with juice, no complaints from me :) | 21:31 |
| *** MaxV has joined #openstack-infra | 21:31 | |
| anteaya | I have no problem with that | 21:31 |
| *** mbacchi has joined #openstack-infra | 21:31 | |
| pleia2 | anyway, I shall finish packing and will be back next week to catch up on things | 21:31 |
| anteaya | safe flights | 21:31 |
| pleia2 | thanks :) | 21:32 |
| *** morganfainberg is now known as needscoffee | 21:32 | |
| nibalizer | pleia2: don't forget your charger this time | 21:33 |
| nibalizer | :P | 21:33 |
| jogo | clarkb: can you take a second look at https://review.openstack.org/#/c/157530 | 21:35 |
| jogo | as this is hurting a lot of projects | 21:35 |
| clarkb | jogo: sure, ajaegers -1 applies still | 21:35 |
| *** sabeen2 has quit IRC | 21:35 | |
| jogo | clarkb: kk will update | 21:36 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting https://review.openstack.org/157530 | 21:38 |
| *** BobBall is now known as BobBall_AWOL | 21:39 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for cells bug 1423695 https://review.openstack.org/157532 | 21:40 |
| openstack | bug 1423695 in OpenStack Compute (nova) "gate-devstack-dsvm-cells fails attaching volume" [Undecided,New] https://launchpad.net/bugs/1423695 | 21:40 |
| *** achanda has quit IRC | 21:40 | |
| *** doude has quit IRC | 21:40 | |
| *** doude has joined #openstack-infra | 21:40 | |
| clarkb | jogo: almost one small change | 21:40 |
| *** ivar-laz_ has joined #openstack-infra | 21:41 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Stop making devstack-dsvm-cells voting https://review.openstack.org/157530 | 21:41 |
| anteaya | sdague: any objection to this change merging? the stable python26 jobs going from 40 to 60: https://review.openstack.org/#/c/156734/ | 21:42 |
| *** bswartz has quit IRC | 21:42 | |
| clarkb | jogo: +2 | 21:43 |
| *** ivar-lazzaro has quit IRC | 21:43 | |
| openstackgerrit | Dan Prince proposed openstack-infra/project-config: Remove wasteful tripleo workspace-cache cloning https://review.openstack.org/157490 | 21:43 |
| *** ihrachyshka has quit IRC | 21:44 | |
| jogo | clarkb: thanks! | 21:44 |
| *** andreykurilin_ has joined #openstack-infra | 21:45 | |
| *** ivar-laz_ has quit IRC | 21:46 | |
| *** achanda has joined #openstack-infra | 21:46 | |
| *** ivar-lazzaro has joined #openstack-infra | 21:46 | |
| *** sarob has quit IRC | 21:46 | |
| jhesketh | Morning | 21:47 |
| *** ivar-lazzaro has quit IRC | 21:47 | |
| *** jp_at_hp has quit IRC | 21:48 | |
| clarkb | ohai | 21:48 |
| *** ivar-lazzaro has joined #openstack-infra | 21:48 | |
| clarkb | jhesketh: I would totally try to make zuul things happen but I am wrestling java and gerrit | 21:48 |
| jhesketh | clarkb: no worries, all good :-). Maybe there's something I could help with? | 21:49 |
| clarkb | jhesketh: it appears that gerrit 2.8 has a memory leak. I have discovered that there are in theory options to the jvm that will cause it to dump a heap dump on signal 3 | 21:50 |
| clarkb | but applying those options to review-dev makes gerrit daemon not start | 21:50 |
| clarkb | my current plan is to figure out how to run gerrit with those options so that after we restart to kill the current leak we have debugging options fr next time | 21:50 |
| anteaya | jhesketh: morning | 21:51 |
| *** dkranz has joined #openstack-infra | 21:52 | |
| *** wenlock has quit IRC | 21:53 | |
| *** jaypipes has quit IRC | 21:53 | |
| jhesketh | clarkb: okay. I don't know much about java but if there is something specific I can do let me know :-). | 21:53 |
| *** oomichi has joined #openstack-infra | 21:54 | |
| clarkb | jhesketh: will do, maybe I should recheck the dg jobs right now and see if you can get in there with the 2 hour window? | 21:54 |
| clarkb | jhesketh: since its relatively low overhead for me to start a job and hold a node | 21:54 |
| clarkb | ok HeapDumpOnCtrlBreak is an unrecognized option | 21:55 |
| *** wenlock has joined #openstack-infra | 21:55 | |
| clarkb | apparently its an old option and no longer supported | 21:56 |
| jhesketh | clarkb: up to you. Maybe keep your current state and I'll deal with email and other stuff instead of distracting you | 21:56 |
| clarkb | jhesketh: well I think I am about to give up on this. I think our only other option is to install the jdk instead of jre to get jmap | 21:57 |
| clarkb | fungi: what do you think about just restarting gerrit now? then we can worry about switching to a jdk install later? | 21:59 |
| clarkb | jeblair: mordred jhesketh ^ | 21:59 |
| fungi | clarkb: i'm fine with it. need help? | 22:00 |
| fungi | i'm mostly just tweaking and rerunning random non-devstack jobs on held devstack nodes to improve my baseline for the image convergence | 22:00 |
| clarkb | fungi: I shouldn't just want to make sure that others are on board. Will restart it now | 22:01 |
| *** zz_gondoi is now known as gondoi | 22:01 | |
| clarkb | fungi: maybe do a statusbot alert? | 22:02 |
| *** patrickeast_ has joined #openstack-infra | 22:02 | |
| anteaya | gerrit is down for me | 22:02 |
| greghaynes | hrmmm, gerrit 503 | 22:02 |
| anteaya | did you restart? | 22:02 |
| clarkb | you guys crack me up | 22:02 |
| greghaynes | Im glad we can entertain :) | 22:02 |
| *** dannywil_ has joined #openstack-infra | 22:02 | |
| anteaya | why do a status bot alert then | 22:03 |
| anteaya | after the restart | 22:03 |
| clarkb | anteaya: because I just restarted it | 22:03 |
| *** MaxV has quit IRC | 22:03 | |
| anteaya | status that 503? that was me | 22:03 |
| fungi | it's working for me! ;) | 22:03 |
| clarkb | anteaya: I am confused | 22:03 |
| greghaynes | Yes, it has returned | 22:03 |
| clarkb | anyways /me digs up statusbot syntax | 22:04 |
| *** patrickeast has quit IRC | 22:04 | |
| *** patrickeast_ is now known as patrickeast | 22:04 | |
| *** dannywilson has quit IRC | 22:04 | |
| anteaya | well usually for me a status bot is effective before the restart | 22:04 |
| *** marun has quit IRC | 22:04 | |
| anteaya | not sure of the value after the restart | 22:04 |
| clarkb | anteaya: to let people know why it happened so that I don't get a flood of "is gerrit down?" questions | 22:05 |
| anteaya | okay | 22:05 |
| clarkb | looks like #status notice Gerrit was restarted to correct memory pressure due to what appears to be a memory leak | 22:06 |
| clarkb | how does that sound? | 22:06 |
| anteaya | I can live with that | 22:06 |
| fungi | clarkb: lgtm | 22:06 |
| fungi | and yeah, you want notice so that it's a non-stateful message with no topic editing | 22:07 |
| *** carl_baldwin has quit IRC | 22:07 | |
| clarkb | #status notice Gerrit was restarted to correct memory pressure due to what appears to be a memory leak | 22:07 |
| *** vponomaryov has quit IRC | 22:07 | |
| *** enikanorov_ has joined #openstack-infra | 22:07 | |
| clarkb | no statusbot? | 22:07 |
| *** viktors has quit IRC | 22:08 | |
| *** rlandy has quit IRC | 22:08 | |
| clarkb | or is it #notice? this bot doesn't have docs and can never remember what to execute /me should fix this | 22:08 |
| fungi | no, openstackstatis is mia apparently | 22:09 |
| *** enikanorov has quit IRC | 22:09 | |
| fungi | er, openstackstatus | 22:09 |
| *** carl_baldwin has joined #openstack-infra | 22:09 | |
| fungi | also it's been long enough now i'd say just fix statusbot and no bother sending the notice ;) | 22:09 |
| openstackgerrit | Spencer Krum proposed openstack-infra/infra-specs: Spec to refactor puppet code in openstack_project https://review.openstack.org/137471 | 22:10 |
| jhesketh | clarkb: restarting the dev Gerrit or live? | 22:10 |
| anteaya | live | 22:11 |
| anteaya | and done | 22:11 |
| *** vponomaryov has joined #openstack-infra | 22:11 | |
| jhesketh | Cool :-) | 22:11 |
| openstackgerrit | Merged openstack-infra/project-config: Remove check job from tripleo-puppet-elements https://review.openstack.org/157486 | 22:11 |
| clarkb | fungi: statusbot is running on eavesdrop I guess it just left this channel? restart the whole daemon? | 22:11 |
| clarkb | seems like you tpically kick that bot | 22:11 |
| *** bswartz has joined #openstack-infra | 22:12 | |
| *** viktors has joined #openstack-infra | 22:12 | |
| fungi | yeah | 22:12 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Skip volume devstack exercise https://review.openstack.org/157553 | 22:12 |
| fungi | i just use the initscript to restart it | 22:12 |
| fungi | on eavesdrop | 22:12 |
| *** mrunge has quit IRC | 22:13 | |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Allow ability to prevent static dns resolver https://review.openstack.org/157554 | 22:13 |
| fungi | 2015-02-07 22:50:33 <-- openstackstatus (~openstack@eavesdrop.openstack.org) has quit (Ping timeout: 245 seconds) | 22:13 |
| fungi | so it's been gone... a while | 22:13 |
| anteaya | it should send postcards | 22:13 |
| clarkb | fungi: kk doing that now | 22:13 |
| anteaya | I try to when I am gone that long | 22:13 |
| fungi | even my vacation next week won't rival that bot's time off | 22:14 |
| anteaya | fungi: where are you next week? | 22:14 |
| anteaya | or would you prefer not to say? | 22:14 |
| fungi | florida | 22:14 |
| anteaya | nice | 22:14 |
| *** koolhead17 has quit IRC | 22:14 | |
| fungi | get away from some of the ice and wind here | 22:14 |
| anteaya | beach sand sun | 22:14 |
| anteaya | good call | 22:14 |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Allow ability to prevent static dns resolver https://review.openstack.org/157554 | 22:14 |
| anteaya | shark's teeth? | 22:14 |
| clarkb | fungi: https://review.openstack.org/monitoring?part=graph&graph=usedMemory&period=jour you can see the drop in memory use post restart thee | 22:14 |
| *** openstackstatus has joined #openstack-infra | 22:15 | |
| *** ChanServ sets mode: +v openstackstatus | 22:15 | |
| anteaya | i remember collecting shark's teeth in florida as a kid | 22:15 |
| fungi | clarkb: yep, classic leak (or cache, but if that's a cache it's a terrible one) | 22:15 |
| fungi | anteaya: yeah, they have those. also sharks | 22:15 |
| anteaya | okay maybe leave the sharks alone? | 22:15 |
| clarkb | fungi: not just the teeth? :P | 22:15 |
| *** dims__ has quit IRC | 22:15 | |
| fungi | clarkb: oh, they have both | 22:16 |
| *** tmcpeak1 has quit IRC | 22:16 | |
| *** sabeen has joined #openstack-infra | 22:16 | |
| fungi | definitely | 22:16 |
| clarkb | ok Ithink gerrit is a happier camper now | 22:16 |
| clarkb | jhesketh: ok where were we on swift logs? | 22:16 |
| *** hdd has quit IRC | 22:17 | |
| clarkb | jhesketh: I'm thinking maybe we just change the timeout on the jobs for now? that avoids a zuul restart and the gate is thrashing so I would like to avoid making that worse | 22:18 |
| openstackgerrit | Jamie Lennox proposed openstack/requirements: Update version of neutronclient https://review.openstack.org/142657 | 22:18 |
| jhesketh | clarkb: okay, fair enough, give me a minute | 22:19 |
| *** otter768 has joined #openstack-infra | 22:19 | |
| jogo | clarkb: an alternate unbreak the world patch https://review.openstack.org/#/c/157553/ | 22:20 |
| clarkb | jogo: looks like they are narrowing in on an actual fix though | 22:20 |
| *** tmcpeak has joined #openstack-infra | 22:20 | |
| jogo | clarkb: hopefully. at this point its a race | 22:20 |
| jogo | clarkb: ideally we get a fix out first. but just in case | 22:21 |
| anteaya | jogo: I can't weigh in on this patch until dansmith and alaski get behind it: https://review.openstack.org/#/c/157530/ | 22:21 |
| jogo | the clock is ticking on a half dozen projects are blocked | 22:22 |
| anteaya | or at least change their -1's to 0's | 22:22 |
| clarkb | they are actively debugging so I am happy to let them do that and probably remove the job only from ceilometer | 22:22 |
| clarkb | (based on what dansmith said) | 22:22 |
| dansmith | clarkb: thanks | 22:22 |
| jogo | anteaya: there is an alternate proposal | 22:22 |
| *** ghostpl_ has joined #openstack-infra | 22:22 | |
| dansmith | clarkb: from what we can tell, it's an actual novaclient bug that we can't hit any other way than this exercise | 22:22 |
| anteaya | 157553? | 22:22 |
| dansmith | clarkb: just happens that we only run those on the cells job | 22:22 |
| clarkb | dansmith: neat | 22:22 |
| dansmith | clarkb: so leaving it is the right thing to do for the moment, IMHO | 22:22 |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting job for Nova V2.1 API with neutron https://review.openstack.org/151564 | 22:24 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Update swift upload expiry for all jobs https://review.openstack.org/157561 | 22:24 |
| jhesketh | clarkb: ^ | 22:24 |
| *** otter768 has quit IRC | 22:24 | |
| jogo | dansmith: well whoever gets there first the fix or this skip passes check https://review.openstack.org/#/c/157553/ | 22:24 |
| clarkb | jhesketh: +2, maybe we can get fungi behind that too then I can go about triggering and holding nodes and all that | 22:24 |
| *** ociuhandu has joined #openstack-infra | 22:25 | |
| *** oomichi has quit IRC | 22:26 | |
| *** signed8bit has quit IRC | 22:26 | |
| *** ghostpl_ has quit IRC | 22:27 | |
| *** signed8bit has joined #openstack-infra | 22:28 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix broken link on documentation https://review.openstack.org/157416 | 22:30 |
| *** mbacchi has quit IRC | 22:31 | |
| harlowja | jeblair dhellmann was i think wanting to change the owner of https://launchpad.net/debtcollector/ i think u can do that right? | 22:31 |
| * harlowja not sure why i set that as the owner (guess i did, ha) | 22:31 | |
| harlowja | owner/maintainer | 22:31 |
| clarkb | harlowja: what should it be? | 22:32 |
| harlowja | clarkb i think 'Oslo Drivers' | 22:33 |
| *** mriedem has quit IRC | 22:33 | |
| clarkb | harlowja: done | 22:33 |
| harlowja | thx:) | 22:33 |
| *** ghostpl_ has joined #openstack-infra | 22:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable non-voting keystoneclient functional tests https://review.openstack.org/156123 | 22:35 |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Add openstack/debtcollector to projects.txt https://review.openstack.org/157566 | 22:36 |
| *** medberry2 is now known as med_ | 22:37 | |
| *** med_ has quit IRC | 22:37 | |
| *** med_ has joined #openstack-infra | 22:37 | |
| *** signed8bit has quit IRC | 22:38 | |
| openstackgerrit | Douglas Mendizábal proposed openstack-infra/project-config: Add devstack gate to barbicanclient https://review.openstack.org/157568 | 22:40 |
| openstackgerrit | Merged openstack-infra/project-config: Add pypi jobs for stackforge/networking-cisco https://review.openstack.org/156948 | 22:42 |
| *** pc_m has quit IRC | 22:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Update swift upload expiry for all jobs https://review.openstack.org/157561 | 22:44 |
| anteaya | clarkb: am I reading your comment correctly? the value that currently is 115 should be 125, yes? https://review.openstack.org/#/c/155100/1..2/jenkins/jobs/devstack-gate.yaml | 22:45 |
| *** Alexandra_ has joined #openstack-infra | 22:45 | |
| *** hyakuhei has quit IRC | 22:45 | |
| hogepodge | fungi irc operator error | 22:46 |
| zaro | clarkb: did you get review-dev working with special VM args yet? | 22:46 |
| *** MarkAtwood has joined #openstack-infra | 22:47 | |
| *** camunoz has joined #openstack-infra | 22:47 | |
| clarkb | zaro no they arent supported so I reverted | 22:47 |
| openstackgerrit | Vipul Sabhaya proposed openstack/requirements: Add python-openstackclient to global-requirements https://review.openstack.org/157569 | 22:48 |
| zaro | clarkb: not supported on jvm version? | 22:48 |
| clarkb | anteaya yes | 22:48 |
| *** hyakuhei has joined #openstack-infra | 22:48 | |
| clarkb | zaro ya according to my testing java didnt like those args | 22:48 |
| zaro | clarkb: where did you find those args? | 22:50 |
| ekarlso | what's up with stackforge projects now being enforced with globla requirements ? | 22:50 |
| fungi | hogepodge: i consider defcore/refstack concerns to be equally on-topic in #-qa, so either was fine by me ;) | 22:50 |
| anteaya | clarkb: thank you, I will comment | 22:50 |
| fungi | ekarlso: what do you mean? | 22:51 |
| clarkb | zaro on a blog that took me to oracle docs | 22:51 |
| *** thingee has joined #openstack-infra | 22:51 | |
| clarkb | but we dont use oracle jvm and I think the args were removed in favor of jmap | 22:51 |
| fungi | ekarlso: stackforge projects can both be global requirements and can have global requirements forced on them, but whether they can request additions to the global requirements list is still debated i think | 22:52 |
| *** kgiusti has left #openstack-infra | 22:52 | |
| clarkb | yup | 22:53 |
| *** hyakuhei has quit IRC | 22:54 | |
| *** jgrimm is now known as zz_jgrimm | 22:55 | |
| clarkb | fungi can you review jheskeths change above? | 22:56 |
| *** hyakuhei has joined #openstack-infra | 22:56 | |
| clarkb | will help the swift log upload performance debugging | 22:56 |
| *** yamamoto has joined #openstack-infra | 22:57 | |
| *** jedimike has joined #openstack-infra | 22:58 | |
| *** claudiub has quit IRC | 22:58 | |
| *** andreaf_ has quit IRC | 23:00 | |
| fungi | clarkb: the one that already merged, or something else? | 23:00 |
| clarkb | oh it merged ignore me | 23:01 |
| clarkb | too much going on | 23:01 |
| *** marun has joined #openstack-infra | 23:01 | |
| fungi | yeah, i approved it a while back | 23:01 |
| fungi | so, um, done! ;) | 23:01 |
| anteaya | ha ha ha | 23:01 |
| *** yamamoto has quit IRC | 23:02 | |
| openstackgerrit | Douglas Mendizábal proposed openstack-infra/project-config: Add devstack-gate job to barbicanclient https://review.openstack.org/157568 | 23:02 |
| openstackgerrit | Douglas Mendizábal proposed openstack-infra/project-config: Add a devstack-gate job to barbicanclient's tests https://review.openstack.org/157568 | 23:04 |
| *** dannywil_ has quit IRC | 23:04 | |
| *** abhirc has quit IRC | 23:04 | |
| ekarlso | fungi: funky that bit, cause it didn't use to be like that | 23:05 |
| fungi | ekarlso: which bit? | 23:06 |
| *** marun has quit IRC | 23:06 | |
| ekarlso | the hard requirement enforcement :p | 23:06 |
| clarkb | its always been that way... | 23:06 |
| ekarlso | it use to work like a few days or weeks back | 23:06 |
| ekarlso | rly with devstack ? | 23:06 |
| clarkb | ekarlso: I think we are talking past each other on what "it" is | 23:06 |
| fungi | ekarlso: yeah, i'm not following | 23:07 |
| clarkb | global requirements has always been applied to all projects in its project list | 23:07 |
| clarkb | and requirements have been enforced | 23:07 |
| *** hichihara has joined #openstack-infra | 23:07 | |
| ekarlso | 'python-openstackclient' is not in global-requirements.txt | 23:07 |
| *** sarob has joined #openstack-infra | 23:07 | |
| *** crc32 has quit IRC | 23:07 | |
| ekarlso | when python-cueclient is depending on it | 23:07 |
| fungi | ekarlso: why would it be? | 23:07 |
| ekarlso | that used to work before :/ | 23:07 |
| dtroyer | it was just removed in the last day or so. | 23:07 |
| ekarlso | dtroyer: rly, why ? | 23:08 |
| dtroyer | because nothing 'should' depend on it | 23:08 |
| clarkb | jhesketh: I just rechecked a change and will hold a couple nodes from that | 23:08 |
| clarkb | jhesketh: should runwith the 4 hour expiry | 23:08 |
| dtroyer | apparently a couple of clients with OSC plugins do though. I'd argue that they should not so that as it is really optional | 23:08 |
| ekarlso | dtroyer: isn't openstackclient the way to go for cli forwards kinda ? | 23:08 |
| jhesketh | clarkb: thanks :-) | 23:09 |
| fungi | ekarlso: if python-cueclient is a stackforge project, it can use the REQUIREMENTS_MODE=soft option to devstack-gate/devstack allowing it to install additional requirements which aren't in the global requirements list | 23:09 |
| dtroyer | if you just want the congressclient libs, you would otherwise have to install OSC and _all_ of its dependencies, which may not be what you want | 23:09 |
| fungi | dtroyer: the counterargument from congressclient and cueclient is "i want to pip install this thing that's a plugin to openstackclient, so it should automatically install openstackclient for me when i do" | 23:10 |
| dtroyer | IOW, OSC and the various client lins with OSC plugins should not have each other in their requirements as they are both useful without the other | 23:10 |
| clarkb | jhesketh: do you think I/we should add better logging to the threadpull change? we should be able to use python's logging lib to get synchronized log writes across the threads | 23:10 |
| *** MarkAtwood has quit IRC | 23:11 | |
| needscoffee | mordred, you know if I stick with this name instead of morganfainberg, no more tab-doh,wrongperson | 23:11 |
| *** bkero has quit IRC | 23:12 | |
| needscoffee | for "mor-<tab>" | 23:12 |
| *** MarkAtwood has joined #openstack-infra | 23:12 | |
| clarkb | fungi: does nodepool look funny to you? | 23:12 |
| dtroyer | ekarlso, fungi: I'm not going to get in their way from putting OSC into their requirements.txt, but I think it the tradeoff of having to install two pacakges with pi vs being required to install 23 deps is a better user experience | 23:12 |
| clarkb | fungi: the graph indicates we aren't building nodes. I am tailing logs now | 23:12 |
| fungi | clarkb: no, looks like zuul stopped running jobs though? | 23:12 |
| ekarlso | dtroyer: +1 | 23:12 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/devstack-gate: Add debtcollector to devstack-vm-gate-wrap.sh https://review.openstack.org/157580 | 23:13 |
| fungi | clarkb: i think nodepool "caught up" now that zuul seems to have ceased starting new jobs | 23:13 |
| *** Guest936 has joined #openstack-infra | 23:13 | |
| clarkb | oh good point | 23:13 |
| clarkb | jhesketh: so is it possible that config change broke teh zuul? | 23:13 |
| *** Guest936 is now known as annegentle | 23:13 | |
| anteaya | fungi: do you consider this patch mergable or are you still working on it? https://review.openstack.org/#/c/156698/ | 23:13 |
| clarkb | AttributeError: 'int' object has no attribute 'format' | 23:14 |
| jhesketh | clarkb: possibly not a bad idea, although any errors should raise up to the main thread | 23:14 |
| clarkb | http://paste.openstack.org/show/177964/ | 23:14 |
| fungi | anteaya: it's mergeable, though if nobody approves it i may extend it to test more broadly. otherwise i'll extend with new changes | 23:14 |
| *** hichihara has quit IRC | 23:14 | |
| anteaya | very good | 23:15 |
| clarkb | jhesketh: I wonder if that string should be quoted | 23:15 |
| jhesketh | clarkb: :-( | 23:15 |
| clarkb | jhesketh: let me try that really quick by hand | 23:15 |
| jhesketh | clarkb: should we revert? | 23:15 |
| anteaya | needscoffee: yes easier for tab complete, keep this one | 23:15 |
| needscoffee | anteaya, haha | 23:15 |
| needscoffee | :P | 23:15 |
| anteaya | #vote or | 23:15 |
| needscoffee | anteaya, troooooooollll ;) | 23:15 |
| *** yamamoto has joined #openstack-infra | 23:15 | |
| *** prad has quit IRC | 23:16 | |
| anteaya | :) | 23:16 |
| clarkb | jhesketh: no I think quotes fixes it | 23:16 |
| anteaya | you opened the door | 23:16 |
| fungi | jhesketh: clarkb: yep, timing and the error message indicate that it is indeed that | 23:16 |
| clarkb | jhesketh: so just '14400' unless the swift stuff needs it as an int | 23:16 |
| jhesketh | clarkb: looking at swift.py there is int(time() + settings['expiry']), so I'm not sure if a string will screw that up | 23:16 |
| jhesketh | or if the layout stuff reads it in as an int | 23:16 |
| fungi | yeah, i think without wrapping like int(settings['expiry']) that will asplode | 23:17 |
| clarkb | oh then ya we should revert | 23:17 |
| jhesketh | clarkb: I'll revert the change to layout | 23:17 |
| clarkb | so I think a better way to handle this is where the exception is thrown coerce to a string first | 23:18 |
| clarkb | I will write that change | 23:18 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Revert "Update swift upload expiry for all jobs" https://review.openstack.org/157585 | 23:19 |
| jhesketh | clarkb: ^ | 23:19 |
| jhesketh | clarkb: okay so you're going to look into fixing zuul? Otherwise I can | 23:19 |
| *** crc32 has joined #openstack-infra | 23:19 | |
| *** mjturek1 has quit IRC | 23:19 | |
| clarkb | ya I think I see a simple fix already | 23:19 |
| fungi | i'm mildly surprised the layout parser doesn't hate that | 23:19 |
| fungi | seems like it would be opinionated about what datatype it's getting there | 23:20 |
| *** weshay has quit IRC | 23:20 | |
| clarkb | I think the layout may expect an int | 23:20 |
| fungi | aha | 23:20 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Parameterized working directories. https://review.openstack.org/157587 | 23:21 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Added storyboard working_dir config option to correct block. https://review.openstack.org/157588 | 23:21 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Moved all module parameters into one manifest. https://review.openstack.org/157589 | 23:21 |
| fungi | well, except doesn't the voluptuous check run on config reloading too? | 23:21 |
| fungi | so i expect you'd have seen it complain when you manually edited the file too | 23:21 |
| jhesketh | yeah the layout expects an int | 23:21 |
| fungi | weird | 23:21 |
| fungi | i need to spend some time rereading that i guess | 23:22 |
| jhesketh | it's more that the code tries to substitute things on all variables rather than just the strings | 23:22 |
| *** carl_baldwin has quit IRC | 23:22 | |
| *** gondoi is now known as zz_gondoi | 23:23 | |
| clarkb | ya but I think coersion to string is what we want anyways since they end up as env vars right? | 23:23 |
| *** yamamoto has quit IRC | 23:24 | |
| clarkb | anyways waiting for tox to finish to avoid any embarassing format issues but will push up what I think is the simple fix | 23:24 |
| jhesketh | clarkb: I think for the expiry we want to coerce it at the hmac | 23:24 |
| jhesketh | otherwise adding time.time() to it won't work | 23:24 |
| anteaya | fungi: what is the reason the new job is experimental in nova? https://review.openstack.org/#/c/156698/2 | 23:24 |
| jhesketh | see lib/swift.py line 151 | 23:24 |
| anteaya | as opposed to an infra repo | 23:25 |
| fungi | anteaya: because it's a nova job? | 23:25 |
| clarkb | jhesketh: I see | 23:25 |
| fungi | anteaya: selecting nova as probably one of the more complex/needy projects from a unit testing perspective (exercises database configuration on the worker as well as some system package expectations) | 23:26 |
| anteaya | fungi: but that is just the title of it, right? there is nothing else that is specific to nova, is there? | 23:26 |
| *** bhuvan has quit IRC | 23:26 | |
| anteaya | fungi: oh okay, yes that makes sense | 23:26 |
| jhesketh | clarkb: although expires is substituted into the string with %s which is surprising | 23:26 |
| anteaya | I just wasn't seeing anything nova specific | 23:26 |
| jhesketh | if you're fixing that up, it should probably be %d | 23:26 |
| fungi | anteaya: it checks out nova source code and runs tox -e py27 basically | 23:26 |
| anteaya | it just installs packages and sets up the dbs | 23:27 |
| jhesketh | clarkb: let me know if you'd prefer I made the changes | 23:27 |
| clarkb | jhesketh: %s basically just does str() on the thing first iirc | 23:27 |
| jogo | clarkb: so we broke icehouse which means d-g is wedged too | 23:27 |
| jogo | clarkb: so your welcome :) | 23:27 |
| jhesketh | clarkb: ah, okay | 23:27 |
| fungi | anteaya: the next step (whether in that same change or in a new change) will be to extend that so i can add it as several experimental jobs to basically all projects | 23:27 |
| clarkb | jhesketh: so I think we don't coerce in launchers/gearman then, instead maybe just only do the format() call if v is a string type? | 23:27 |
| clarkb | jhesketh: since there shouldn't be a reason for expiry to have that format called on it | 23:28 |
| jhesketh | clarkb: that was my thinking | 23:28 |
| jhesketh | +1 | 23:28 |
| anteaya | fungi: okay yes in the experimental-workers.yaml | 23:28 |
| fungi | anteaya: but nova provides a good initial proving ground that the mechanics of the macros work (i've already tested the scripts they contain so more worried about making sure parameter substitution and worker node label selection work correctly) | 23:28 |
| clarkb | jhesketh: ok I have things open should have this done soon | 23:28 |
| anteaya | fungi: right that is what I was expecting, hence my question about nova | 23:28 |
| anteaya | right | 23:28 |
| anteaya | okay thanks | 23:28 |
| anteaya | I understand now | 23:29 |
| krotscheck | nibalizer: So, working on puppet-storyboard, I'm trying to figure out a good way to notify an optional module. Have you seen something like that before? | 23:29 |
| anteaya | I'll wait until -nova has fixed their fire | 23:29 |
| anteaya | then bug them | 23:29 |
| anteaya | the look close | 23:29 |
| fungi | okay, so having built a _full_ package install list from our thick slave class manifest, installing those (mostly locally cached) at job run time adds ~1m11s. once i add bindep hooks, projects can bypass that by providing a more specific bindep manifest to just install the packages they really need | 23:29 |
| anteaya | they | 23:29 |
| anteaya | cool | 23:30 |
| *** dannywilson has joined #openstack-infra | 23:30 | |
| fungi | but once i extend it to cover more than just ubuntu-trusty and also get the same package lists unioned to the lists we're pre-caching via nodepool prep scripts/dib elements, we'll be almost there | 23:31 |
| nibalizer | krotscheck: the optional thing can subscribe to the required thing | 23:31 |
| krotscheck | nibalizer: BUT THAT WOULD MAKE SENSE. | 23:31 |
| krotscheck | nibalizer: :) | 23:31 |
| krotscheck | So clearly I can't do it. | 23:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul: Only format swift string parameters https://review.openstack.org/157592 | 23:31 |
| clarkb | jhesketh: ^ something like that? | 23:31 |
| krotscheck | nibalizer: How would I then tell service['http'] (from an extrnal module) to subscribe to something that I'm doing inside of mine? | 23:32 |
| *** yamamoto has joined #openstack-infra | 23:32 | |
| anteaya | fungi: makes sense | 23:32 |
| clarkb | oh wait the fix won't hrm | 23:32 |
| openstackgerrit | Miro Hroncok proposed openstack-infra/project-config: Add Python 3 template for openstack/heat-cfntools https://review.openstack.org/157593 | 23:33 |
| clarkb | fungi: jhesketh so zuul stopped running jobs so our fix won't merge :) I will stop puppet on zuul and apply by hand then reenable puppet once it goes through | 23:33 |
| greghaynes | clarkb: do we care about python3 in that location? | 23:33 |
| clarkb | greghaynes: no zuul isn't py3k friendly | 23:34 |
| greghaynes | ok | 23:34 |
| jhesketh | clarkb: if zuul stopped running jobs won't it need a manual restart? | 23:34 |
| clarkb | jhesketh: well its "stopped" basically was in a tight loop on that error, removing the error allowed it to continue | 23:35 |
| clarkb | jobs are starting now | 23:35 |
| clarkb | but will keep an eye on it just in case it does need a restart for some reason | 23:35 |
| jhesketh | oh right, puppet kept putting the bad config back in | 23:35 |
| clarkb | jhesketh: right | 23:35 |
| jhesketh | okay, sounds good then | 23:36 |
| clarkb | so puppet is disabled, I removed the line, and HUP'd the service | 23:36 |
| clarkb | once your change gets in I will reenable puppet and we should be good | 23:36 |
| openstackgerrit | Douglas Mendizábal proposed openstack-infra/project-config: Add a devstack-gate job to barbicanclient's tests https://review.openstack.org/157568 | 23:36 |
| clarkb | Deficit: devstack-trusty: 0 (start: 293 min: 293 ready: 312 capacity: 8) looks much better on the nodepool side | 23:36 |
| clarkb | jhesketh: I think this would happen with the global change as well | 23:37 |
| clarkb | so good thing we caught it just with the simple config change :) | 23:37 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Add bandit jobs https://review.openstack.org/157595 | 23:38 |
| *** thingee has quit IRC | 23:38 | |
| *** salv-orlando has quit IRC | 23:39 | |
| *** bdpayne has quit IRC | 23:39 | |
| clarkb | jogo: ok now that I have caught up. we intend on making the icehouse and juno jobs for dg non voting | 23:39 |
| clarkb | jogo: pleia2 was working on that I thought I had reviewed the change too | 23:39 |
| jogo | clarkb: cool, I spoke too soon -- neutron broke it not nova | 23:39 |
| jogo | clarkb: a better upper limit is needed in icehouse for neutronclient | 23:40 |
| clarkb | jogo: also it would be good if you could expand in your email. What you linked does not explain anything as far as how these things are related | 23:40 |
| clarkb | python-keystone is the issue | 23:40 |
| *** andreykurilin_ has quit IRC | 23:40 | |
| jogo | clarkb: yeah I should explain but I am sick of cleaning up after people | 23:40 |
| *** salv-orlando has joined #openstack-infra | 23:40 | |
| clarkb | jogo: but no one else knows how to clean up bceause there is no info... | 23:41 |
| clarkb | jogo: also this sort of makes my point about capping vs pinning :) | 23:41 |
| jhesketh | clarkb: actually I'm not sure it would because the defaults are looked up in lib/swift.py and aren't formatted | 23:41 |
| clarkb | we aren't any better so meh | 23:41 |
| *** amitgandhinz has quit IRC | 23:41 | |
| clarkb | jhesketh: but those don't appear to be the defaults from the ini file? | 23:42 |
| * clarkb double checks | 23:42 | |
| jhesketh | clarkb: line 149 it fetches from the ini file | 23:42 |
| clarkb | oh nevermind you are right | 23:42 |
| clarkb | ya | 23:42 |
| *** carl_baldwin has joined #openstack-infra | 23:43 | |
| jogo | they broke semver | 23:43 |
| fungi | i guess the good semver solution would be to revert that patch, make a new micro point release, then restore it and have a version bump? | 23:44 |
| clarkb | jogo: I don't think they did | 23:44 |
| clarkb | jogo: requiring a different version of keystoneclient isn't a backward incompatible change | 23:44 |
| jogo | clarkb: wild dependency jumps | 23:44 |
| clarkb | its only becuse you have decided to pin everthing :) | 23:45 |
| clarkb | but I don't think that makes it a backward incompat change. Its just not compatible with how you ahve chosen to do requirements | 23:45 |
| *** carl_baldwin has quit IRC | 23:45 | |
| jogo | clarkb: actually no. it was from this Id5a53eaf3da86932999d62e51d07f7015b0ab53a | 23:46 |
| *** tmcpeak has quit IRC | 23:47 | |
| fungi | i tried to embark on a similar solution for folsom/grizzly and didn't get nearly this far because we didn't have good requirements sync across projects, but could forsee even then that trying to do that given how pypi handles dep resolution was going to be fraught with peril | 23:47 |
| clarkb | jogo: I don't follow the leap there either | 23:47 |
| *** dimtruck is now known as zz_dimtruck | 23:47 | |
| jogo | clarkb: that is where OSC was pinned. which is what is breaking | 23:47 |
| clarkb | right but its breaking because ou have pinned keystone clietn and neutronclient did a release that didn't respect that pin | 23:48 |
| clarkb | which would be fine if the pins in general weren't in place | 23:48 |
| clarkb | anyways this is why I have been wary of the pins. I think it will lead to just as much work and pain | 23:49 |
| *** markvoelker has quit IRC | 23:49 | |
| *** hyakuhei has quit IRC | 23:49 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Apache Config is now an optional module. https://review.openstack.org/157600 | 23:49 |
| clarkb | caps on the other hand give you enough flexibility to work around with things when they shift within reason | 23:49 |
| jogo | clarkb: well neutron client has a cap <2.4 | 23:49 |
| jogo | not a pin | 23:49 |
| jogo | same with OSC | 23:49 |
| krotscheck | nibalizer: ^^ is the relevant code. I'm not certain where I can fit the subscription though. Wouldn't mind your feedback | 23:50 |
| jogo | its not a hard '==' | 23:50 |
| clarkb | jogo: what about keystoneclient? | 23:50 |
| clarkb | keystoneclient is the thing at fault here | 23:50 |
| jogo | python-keystoneclient>=0.7.0,<0.12 | 23:50 |
| jogo | clarkb: I think its safe to say there is no easy answer for any of this | 23:50 |
| clarkb | jogo: yes that is what I have said :) | 23:50 |
| tchaypo | ah, this is still going, excellent | 23:50 |
| clarkb | jogo: which is why I have had a preference to just deal with status quo since its all bad and least effort | 23:50 |
| jogo | clarkb: capped,uncapped,pinned or other | 23:50 |
| *** Alexandra_ has quit IRC | 23:51 | |
| clarkb | but am open to trying other things, we just have to realize there is pain in every direction we turn | 23:51 |
| jogo | clarkb: this was a a status quo break. my cap all stable/juno didn't impact this | 23:51 |
| fungi | when i started to try, i concluded that it's really not particularly tractable even with requirements sync (which we thankfully now have) unless someone has available time to help get a smart dep solver into pip | 23:52 |
| clarkb | jogo: you didn't also cap/pin icehouse? | 23:52 |
| clarkb | jogo: I thought you did both | 23:52 |
| *** dustins has quit IRC | 23:53 | |
| lifeless | fungi: harlowja wrote one, does it have to be /in/ pip ? | 23:53 |
| jogo | clarkb: nope. I didn't realize it was voting on anything besides stable/icehouse and juno. | 23:53 |
| *** mriedem has joined #openstack-infra | 23:53 | |
| jogo | clarkb: forgot about devstack-gate | 23:53 |
| *** yamamoto has quit IRC | 23:53 | |
| harlowja | i start writing one, its still sorta WIP :-P | 23:53 |
| fungi | lifeless: i suppose not. i guess we can replace pip | 23:53 |
| *** bknudson has quit IRC | 23:53 | |
| harlowja | https://github.com/harlowja/pippin has a few idea(s) | 23:54 |
| fungi | harlowja: my understanding is it just turns pip into a downloader and handles the dep resolution on its own? | 23:54 |
| harlowja | basically | 23:54 |
| fungi | sounds reasonable, though i can already see the ensuing shitstorm from "here's how you pip install openstack projects _now_" | 23:55 |
| harlowja | the current code in that repo uses pip for downloading, expands the full dependency graph, then creates 'requirement' trees (where each path in that tree is one solution for a requirement) | 23:55 |
| clarkb | jhesketh: ssh jenkins@104.239.149.53 for https://jenkins01.openstack.org/job/check-dg-tempest-dsvm-full/329/ and ssh jenkins@104.239.162.27 for https://jenkins07.openstack.org/job/check-dg-tempest-dsvm-full-reexec/231/ they are running anwys so may as well try to use them? | 23:55 |
| *** oomichi has joined #openstack-infra | 23:55 | |
| *** tkelsey has quit IRC | 23:55 | |
| harlowja | fungi ya, anythings possible :-P | 23:55 |
| harlowja | https://github.com/harlowja/pippin/blob/future/pippin.py#L449 (the main routine); feel free to hack on that as much as u want | 23:56 |
| harlowja | and https://github.com/harlowja/pippin/blob/future/pippin.py#L489 | 23:56 |
| clarkb | honestly the right place to fix this *is* pip | 23:56 |
| clarkb | because you will install other python things too | 23:56 |
| fungi | i mean, we're already partway there with declaring that we just don't support easy_install, so saying you can't use straight pip to install this stuff either is only likely to alienate the remaining parts of the python community we haven't already | 23:56 |
| jhesketh | clarkb: awesome, thanks! | 23:56 |
| lifeless | fungi: we don't need to change anyone elses behaviour to fix the gate | 23:56 |
| *** abhirc has joined #openstack-infra | 23:56 | |
| *** ddieterly has quit IRC | 23:57 | |
| fungi | lifeless: only those people who wish to have test results similar to ours, or installations similar to what we tested | 23:57 |
| clarkb | lifeless: you do if ou expect to map your tests onto what people are doing real world | 23:57 |
| fungi | you're right about that | 23:57 |
| clarkb | ya what fungi just said | 23:57 |
| fungi | but honestly, pip doesn't guarantee much of that to start with | 23:57 |
| jhesketh | clarkb: I have to wait for the job to finish to get the build params | 23:57 |
| harlowja | u can all join https://groups.google.com/forum/#!topic/pypa-dev/YqgoE1ZLpqg :-P | 23:58 |
| clarkb | jhesketh: and to have it generate all the artifacts | 23:58 |
| harlowja | 'Dependency solver (working group?)' (my post) | 23:58 |
| *** dannywilson has quit IRC | 23:58 | |
| dstufft | oh goody | 23:58 |
| dstufft | what did pipd o now | 23:58 |
| fungi | harlowja: i applaud this effort ;) | 23:58 |
| fungi | dstufft: it convinced people to start pitching in on the lack of a smart dep solver | 23:58 |
| fungi | dstufft: positive stuff | 23:58 |
| clarkb | looks like matthew has outlined exactly what needs to be done | 23:59 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Refactor Gerrit plugin build jobs https://review.openstack.org/157603 | 23:59 |
| harlowja | ya, and i think its also fine to have a POC somewhere that someone can get to work (whether thats pippin or not) | 23:59 |
| harlowja | i've got the full dep graph | 23:59 |
| clarkb | harlowja: POC doesn't hurt but I don't think we should use it for the reasons fungi described | 23:59 |
| lifeless | pip doesn't guarantee same as what we have anyway :) | 23:59 |
| harlowja | sure | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
